Substack cometh, and lo it is good. (Pricing)

Science before the replication crisis

I’m still broadly supportive of the heuristics and biases program. I still think Noise: A Flaw in Human Judgment and Thinking, Fast and Slow are worth reading. This sort of stuff is based on deep-seated elements of human cognitive neurological architecture. They’re not fabricated out of whole cloth.

But, the replication crisis has been a total disaster for huge swaths of science, and that’s probably a good thing.

BuzzFeed came out with a review of l’affaire Ariely and there’s not much there beyond what you could find in social media. Many of the researchers did not respond to calls for interviews, which I think is reasonable, since they are speaking directly in their own words on Twitter and their websites. This is better than talking to the media, which is going to twist their comments to fit some narrative. The interesting thing about BuzzFeed’s piece is it reminds us of the period in the late 2000’s when pop-social science was huge and massive inferences were generated by small (or now we know fraudulent) data. Perhaps we’ve at least moved on beyond that?

And yet let me point you to my previous post on testosterone. On Twitter, a correspondent argued that Matt Walker’s Why We Sleep: Unlocking the Power of Sleep and Dreams is just as bad. So perhaps it didn’t get better?

5 thoughts on “Science before the replication crisis

  1. It’d be nice if some actual structural changes came out of it. There’s still going to be stuff like this as long as the incentives are all towards Big New Results, and there’s comparatively no funding in duplication efforts.

  2. Re Walker’s Why We Sleep, thought Guzey’s comment on his very long post digging into flaws problems with that book was interesting.

    From twitter: Probably the craziest thing about all of this for me is that my piece on the book has gotten >250k views by now and still not a single neuroscientist or sleep scientist commented meaningfully on the merits of my accusations. Only Andrew Gelman (a statistician) picked them up.

    FWIW, believe we’ve *always* had some high status lying in science every era. That is to say, the replication crisis has helped improve some areas, but perhaps it’s too much to hope (and historically implausible) that it eliminates all of them.

  3. When I was in graduate school for my PhD, I was frustrated for my low yield on outcome. Meanwhile, many fellow graduate students in the same school were highly productive with a lot of publications. Even though in classroom, none of them could measures up to my level of “test taking ability”.

    Some of those published results might be useful for my own research project. So I needed to replicate them before incorporating into my own researches. Most of them are not replicable in our own lab. I was frustrated and thought my lousy hands for lab research.

    Years later most those results turned out to be either false or downright frauds. But these questionable PhD graduates were able to advance through all academic ranks before these problematic results starting caughting up with them. Most of them are able to keep their jobs.

    So fraud does pay in human society.

    Now I am making living in business world and have far better reward than academic field. My private charity trust for science and education is my own way to return wealth to society and continue my contribution in the academic field.

Comments are closed.