I think the birth of this legend tends to spring from the porn industry -- and particularly the inanimate side. If we limit our investigation to that part of the industry -- photgraphs or stories, it tends to be true. Or at least, a straw poll of my male and female friends indicates that it is true.
Not that women aren't turned on by images, or men by text, just the inclination has tended to be textual amongst the females, and visual amongst males.
This may be partially because the 'perfect' male body, unclothed, tends to be less aesthetically pleasing than the 'perfect' female, so often such stimulation as the women I know experience in visual erotica results from pictures of women and not men. It may also be connected to the fact that visual erotica in magazines/still pictures is largely divorced from emotional context, and women have tended to be brought up associating sexual stimulation with emotional connection.
However, my discussions with female friends on the subject have led to one,all-overriding complaint with the visual erotica they have seen, and it's simply this -- it just looks so damn fake to us. Not the acts -- not the "bits moving within bits", but the faces of the participants. Either they have this glossy, model-style smile, or some expression that tries, and fails, to convey smouldering passion (it usually looks more like a child in a sulk). Suspension of disbelief is not achieved, and therefore titillation is minimal. With well written textual erotica, since the imagination provides the images, disbelief is more easily suspended -- although certain cliches and key phrases will lead to that suspension of disbelief being broken, and the story being put quickly aside.