The Swamp

A serious idea

September 3rd, 2008 at 9:06 am (Science!, Ideas)

So I’m thinking about starting something science-y. A friend of mine and I wanted to start a science blog last year, partly as a way of staying in touch and partly because we thought it would be fun, but it never came together. Now I’m back on that idea, only slightly modified. I’ve been thinking about how the fundamental principals of science are based on replication. You don’t get any theories until you’ve repeated the experiment many times. In psychology, though, you don’t see that happening very often. Everyone is so interested in coming up with the new idea, so you see lots of people making chimps listen to rap music in an fMRI machine, or having people play video games while on a unicycle in an fMRI machine, or giving mice tiny little lightsabres to fight each other inside a miniature fMRI machine. You don’t, however, see a second person verifying the newly famous Stanford Giving Mice Tiny Little Lightsabres to Fight Each Other Inside a Miniature fMRI Machine Experiment. I think it’s very important to do so, and it’s falling by the wayside.

Failure to replicate or attempt to replicate means that some experiments can produce an effect because of a quirk of the testing environment, and then that effect can become an assumption for someone else’s experiment and never be questioned. Things like this can happen.

Another big problem with this is that people use fMRI experiments to localize high-level brain functions, and then other folks use those results to figure out what else is going on in even higher levels of brain function. Lets say you play a bunch of tones at people and you figure out where the representation for the diminished fifth is. You get a bunch of people to do this, do a bunch of stats, and report that it’s in the parietal lobe between the area that controls scratching yourself and the area that thinks about Jessica Alba*. You publish, and 200 people read your paper. Of those 200 people, 20 get past the abstract, and 4 of those people aren’t just reading it to look at pretty brain pictures. One of those people decides to do a study about music and uses your paper as a reference. So now we’ve got someone looking in the parietal lobe for music interpretation. They isolate the parietal lobe for experimentation to save time, and they run some stats using a program someone else designed. Of the 1000 ways they could have interpreted the massively complex data they’ve collected, they decide to go with the one that says that when you play Black Sabbath to a macaque in an fMRI machine, their parietal lobe gets really excited when you also give them a ham sandwich to eat. And lo, Parietal Activation of Macaques Eating Ham Sandwiches and Tripping Balls To Black Sabbath is published in a reputable journal, and now we have two papers confirming the location of music in the parietal lobe. This continues indefinitely, and somewhere in Latvia, a little orphan cries himself to sleep, not knowing why.

Anyhoo, my idea is that I want to run a blog where I pick one recent non-fMRI-related study each week to turn into a web experiment using the program I wrote over the past two years. It’s quite sophisticated and surprisingly easy to set up, so it would be able to handle some pretty complicated experiments with ease. I would collect responses from about 100 people each time (if it manages to become wildly famous, that is), and then post the results a week later. Either that, or they’d last for a few weeks and overlap. In general, though, I’d want to start and end a study every week. I’d aim for about 20% completely new studies that I’d be curious about, and 80% recycled studies to replicate. People would visit the site if they were bored, and they’d be able to sign in under anonymous usernames and never have to type in their age and gender and all that other crap more than once. Data could be connected from study to study, and mined ruthlessly for all sorts of interesting effects. People would waste hours and hours on this site, all in the name of Science. Stats would be calculated on the fly for me to quickly post, and every study would be catalogued and rated based on whether their results were confirmed. Best of all, I would do it in the style of a Mythbusters kind of thing, and I’d get to slam BUSTED on any theory that didn’t hold up to Internet muster. I think this would be very cool. Because Science–as we all know via Bill Nye–RULES.

I’m currently welcoming names for this blog.

* The Parieto-occipital Jessica Alba Sulcus is, however, widely confirmed by many neuroanatomical studies.