Monday, August 07, 2006

It's the Research Methods Stupid

My good friend is a newspaper graphics editor, and we occasionally get into it about the MSM and it's inherent stupidity. He'd just sent me this adorable math cheat sheet showing me that some journalists do indeed care about math, when I started reading a simply FABululous article on CNN.com, entitled:
Well there goes my attempt to make peace with the media. I can tell by the title of this article that I will not be able to maintain my ceasefire.
CHICAGO, Illinois (AP) -- Teens whose iPods are full of music with raunchy, sexual lyrics start having sex sooner than those who prefer other songs, a study found.

Whether it's hip-hop, rap, pop or rock, much of popular music aimed at teens contains sexual overtones. Its influence on their behavior appears to depend on how the sex is portrayed, researchers found.

Songs depicting men as "sex-driven studs," women as sex objects and with explicit references to sex acts are more likely to trigger early sexual behavior than those where sexual references are more veiled and relationships appear more committed, the study found.
So finally there is proof, it is in the MUSIC, we have something we can BLAME! YAY! Parents of the world rejoice, researchers of the world sigh with resignation.
This study cannot conclusively prove that the music causes kids to behave a certain way. All it shows is that there is a significant link* between sexy music and kids who have sex early. (But that doesn't sound nearly as sexy.)

How do I know this? Well let us look at their methodology:
The study, based on telephone interviews with 1,461 participants aged 12 to 17, appears in the August issue of Pediatrics, being released Monday.

Most participants were virgins when they were first questioned in 2001. Follow-up interviews were done in 2002 and 2004 to see if music choice had influenced subsequent behavior.
So what we're saying is, teens who like sexy music had sex. I bet we could show the same relationship between a thousand different types of music. Teens who listen to music about drugs, do drugs, teens who listen to music about anarchy, like anarchy, teens who listen to music about racism, like racism, teens who listen to music about Jesus, like Jesus. Is it because of the music? Or is it because they like X and therefore listen to music that they can relate to?
It's a chicken and the egg problem. And this research cannot prove the link because it did not control for outside factors like views that were already formed. If they truly wanted to test this they would have to randomly assign kids to different groups and force these groups to listen to randomly determined types of music for an assigned period of time and see if their preferences changed.
All of us humans have choices about the music that we store on or iPods or we tolerate on our radio, and these choices could actually reflect our personal views more than change them. I am not saying that music can't affect our life choices, I am saying that this study in no way proves that because of the way it was conducted.
The problem I have with the MSM is that they don't understand research methods and they report irresponsibly on things they clearly don't understand. Sorry Pat.
*Significant Link- This is NOT NOT NOT the same thing as saying X causes Y. It just means that X is related to Y. People who prefer Y are more likely to prefer X, people who prefer X are more likely to prefer Y, it works both ways. But neither necessarily determines the other, it could even be caused by force Z that isn't even included in the analysis.
UPDATE: My journalist friend just sent me a PDF or the article from the front page of his paper, my e-mailed response used the f word a lot. I wish someone was actually going to read this post, then maybe I'd feel better having told at least one person that this study is completely full of shit. Alas, too bad my traffic is actually in negative numbers now.
UPDATE II: The guys from Freakonomics posted on this, so did Jeff Jarvis. And thanks to Flashman for the link! I'm very glad this story is getting some visibility!

7 comments:

Anonymous said...

Thank goodness I'm not the only person who thinks this research is full of shit. It's like the violence/video games "link": tenuous at best.

In the short term, kids who play violent video games become more aggressive. Because they're stimulated from just having played a violent game.

In the long term, violence and video games are more probably linked because kids who are already aggressive prefer to play violent games, not because video games made them violent.

Correlation don't make causality, bub.

AliceAyres said...

I think journalists often write irresponsibly: probably without knowing how to actually read a research paper, they skim the abstract and make wild claims.

That said, it is also irresponsibility on the part of the researchers who conduct studies like these. Contrary to what we were taught in the third grade, it is not enough to come up with a hypothesis question and then find out whether you can answer it. As you so clearly and cleverly explained, it is necessary to control for randomness and outside factors, which is often impossible in questions of sociology.

Conclusion: I hate everyone.

Kav said...

Journalists and science. Hah!

Just don't get me started on how they report on climate change science...

Lisa said...

Well put. (You got linked in today's edition of Crikey too). I have thoguht this so many times, only now see it defined in four words "correlation doesn't mean causation".

Anonymous said...

I wish someone was actually going to read this post, then maybe I'd feel better having told at least one person that this study is completely full of shit. Alas, too bad my traffic is actually in negative numbers now.

Cheer up Shinobi, I, and lots of other Crikey.com readers in OZ read your post....very well put indeed, and I agree, the article was FULL of shit..like most of the stuff peddled in the popular press and network TV....
"correlation doesn't mean causation"... this phrase should be tattooed on every researchers arse...
tootee
PS: Mmm, on the other hand they could be onto something...Sexy music worked for me...still does!!!

Anonymous said...

Might I ask, have you actually read the full report from Rand Corp, or just the journalistic interpretation of it? The account in the papers is terrible, the actual paper is not so bad. I personally have wanted to work for Rand corp in the past (they are one of the more famous and sexy private sociological analysis companies), so I went and checked out what they had done. Turns out it DOES take into account things like preconcieved ideas as much as possible (they looked into things like where they were developmentally, asking questions like have you kissed anyone yet, do you expect to have sex in the next few years, etc).

That's not to say it is perfect, the report admits openly that the researchers couldn't prove causality, but they could show a strong statistical link with listening to music that contained a very specific message of "men as studs who sought after sex" and "women as things to be attained" and an increased likelihood that the person would have sex sooner than people in a similar social, cultural and sexual maturity position that didn't listen to such music.

Shinobi said...

Oh yay! Look at all the comments! Thank you all for stopping by.

I didn't read the entire Rand Study, and I'm sure that they did an excellent job as they usually do. The problem I have isn't really with the study itself but the reporting of it. I am sure the Rand organization realizes that they cannot prove causality in a study like this, and I bet they even mentioned this to journalists (pure conjecture).

But the headlines will soon have overprotective parents deleting songs from their kids iPods (if they can figure out how). And that I have a problem with.