Replication in Science

The internet is abuzz with the recent publication of an article in Science which replicated 100 studies in psychology, but only found positive results for a third of them. A lot of people are freaking out over this — both scientists and lay people — and I sure wouldn’t want to be one of those scientists whose study couldn’t be successfully replicated. But for those people, it isn’t the end of the world. Most of them didn’t screw up or do anything fraudulent. This is just how science works. A lot of people have talked about what this means about the state of science, or the state of psychology, or the state of whatever else, and I don’t have much to add to this conversation. But this does make me think about a few things about science in general, and it is those things that I want to say a few words about.

1) The publication of a study is not the end of the conversation. I think the media is guilty of misleading people about this. The truth is that you don’t sell many newspapers with headlines like, “scientists discover that X may be true.” What I see instead are headlines like, “scientists prove X is true,” or, “study disproves theory about X.” This isn’t how scientists think about things. If you have never had the pleasure of being in the presence of scientists when they hear the results of a new paper, it is a wonderful thing. More often than not, they will try to find problems with it. Maybe the author conceptualized their problem in the wrong way. Maybe they didn’t think of a confounding variable in their experiment. Maybe their statistics were misapplied. Maybe their mathematical model didn’t describe reality. To the lay person, this might make scientists sound petty for trying to take away a colleague’s accomplishment, but this is just part of the process.

Replication is not what you do to show that scientists are terrible, it is just a part of the process. When a new study comes out, scientists often say, “This is really interesting, but I’d like to see more work done on it.” What they are calling for, is more studies to see whether the effect can be reproduced in a different way. This is replication and is the process.

A few years ago, one of my own studies was subjected to a replication by another research team. They believed that we had failed to take a particular variable into account, and may have made a type I error as a result. I will admit that I figuratively bit my nails when I heard that this study was coming out. Nobody wants their study to be wrong. But it happened to turn out that even with the new variable included in the analysis, our hypothesis was still supported. It could have gone the other way.

This is part of why science is great — it is self-correcting. If one scientist draws the wrong conclusion, either innocently or maliciously, other scientists will be there to fix it. A scientific publication is not the end of the conversation. It is the beginning. When a scientist publishes a paper, they are saying, “This is what we think is going on, and here is some evidence that we are right. What do you think?”

2) When making conclusions in science, there are two types of possible errors: Type I errors and Type II errors. Type I errors are when you conclude that something is true, but it is not. Maybe your study concludes that watering plants with Brawndo (it’s got what plants crave) makes them grow better, when in reality it does not — type I error. A type II error is when you conclude that something is not true, but in reality it is. Maybe you conclude that monkeys are not actually primates, when they really are — type II error. A lot of the conversation around this replication study is about how scientists might be making too many type I errors. This could be true, but it is not my purpose here to comment on that. But I will point out that when you make it harder to make a type I error, you necessarily make it easier to make a type II error. Conversely, when you make it harder to make a type II error, you make it easier to make a type I error. Both types of errors are still errors, and you don’t want to make either one.

Here’s how it works: In science, we often use p-values as a tool for figuring out what happened. When you analyze your data, the p-value measures how likely it is that whatever trend you found in the data is the result of random chance and not a relationship between your variables. a p-value of 1.0 means that there is a 100% chance that your data is random numbers, and a p-value of 0.0 means that it is impossible for random chance to create those numbers (in reality, p-values are never exactly 0.0 or 1.0, but somewhere in between). Scientists use a cutoff of 0.05 for drawing conclusions — that is, you cannot tell people that you made a discovery unless there is a 5% chance or less of your data being meaningless noise. If the chance is 6% or higher, you cannot make any claims. Ideally, most things you find with a p-value of 0.05 or less are true, and most things that you find with a p-value higher that 0.05 are not true. But sometimes you have a real effect with a p-value higher than 0.05, which leads you to think it’s false (type II error), and sometimes you have an effect that is not real with a p-value that is lower than 0.05, and you think it’s real (type I error). If you lower the cutoff value to, say, 0.01, you will get fewer false positives (type I errors), but more false negatives (type II errors). If you raise the cutoff value to 0.1, the opposite happens — type II errors become less common, but type I errors become more common.

3) Science is hard, and we don’t know all of the answers. When we take science classes in school, there is always a right answer. You know that the substance is supposed to turn blue when you add the right chemicals, and if it doesn’t, you know you made a mistake. When I was a teacher, I would frequently have students come to me with a test tube full of orange fluid and ask, “is this right?” I would resist a straight answer as much as I could, because the purpose of the experiment was not to teach them how to make the fluid change colors, but to think like a scientist. The domain of scientists is the edge of human knowledge. There is no one to whom a scientist can ask, “is this right?” Because if there was, they wouldn’t be doing science. No one knows the answers to the questions that scientists ask, which is why scientists are trying to figure out the answers to those questions. This means that we will sometimes get it wrong. It is for this reason that I praise the scientists who do replication studies, as well as the scientists who did the original research. It is all part of the process.

Have a topic you want me to cover? Let me know in the comments or on twitter @cgeppig

Follow me on Facebook

Advertisement

Testing a Claim: Ceramic Knives

This post is going to read a little bit like an episode of MythBusters. Testing claims is not something I plan to do regularly in this blog, but this particular one concerns something I’m very interested in (knives) and something I’ve written about in the past. Yes, this is going to be my third post on what makes food turn brown. See my other two here and here. This blog is about science, and science is fundamentally about testing claims; either claims made by yourself or claims made by someone else. Science typically deals with testing a hypothesis — a formal, scientific claim made by a scientist — but the scientific method can be used just as well to test less formal claims. 

 On to the topic at hand…

A friend of mine gave me a ceramic paring knife for christmas (Thanks, Jack!). In researching the properties of ceramic knives, I came across a claim that seems to be made by almost everyone advocating their use: using ceramic knives prevents food from turning brown after it has been cut. See, for example, this video. 

One common explanation given for this purported phenomenon is that contact with the food causes steel blades to corrode and leave brown residue on on the food. Ceramic does not corrode, so it cannot cause food to turn brown for this reason. The video I linked to mentions “ion transfer” as the mechanism of browning. I don’t know what that is, but its predictions are basically the same as the corrosion mechanism.

I have written before about one factor that makes food turn brown (here and here). You will understand this post better if you read my previous ones, but I will summarize them briefly in case you’re short on time: When a fruit or vegetable is damaged (either by cutting or bruising), it produces a brown substance that prevents bacterial infection. This process is triggered by exposure to atmospheric oxygen. 

But let’s pretend for a minute that we don’t know anything about enzymatic browning. What we think will happen will only get us so far — let’s do some science and see what actually happens. 

If the stories are true — that food will turn brown faster with steel blades because of the corrosion of the steel — then a blade made of non-stainless steel will cause a stronger effect of food browning than a blade made of stainless steel, and both steel blades will cause a stronger effect of food browning than a blade made of ceramic. 

I will use three paring knives in this experiment:

Top: The ceramic knife that was given to me. There are no markings on the blade and I threw away the package, so I have no idea what brand it is, but I know it was made in China. I’m pretty sure it is made of zirconium dioxide.

Middle: This is a handmade knife made of 52100 steel. This steel has no corrosion resistance to speak of. 

Bottom: A J.A. Henckels Zwillinge Pro S paring knife. The steel is X50CrMoV15. It is very resistant to rust and other corrosion.

 

I have selected three foods to test these knives on: apples, potatoes and onions. I have already discussed the process of browning in apples and potatoes, so they will be good for this experiment. The naturally-occurring chemicals in onions can be corrosive to steel, so this is another good food to use in this experiment. I cut each of these three foods with each of the three knives to see what would happen. 

Here is the initial setup for this experiment. The column on the left was cut only by the ceramic knife, the column in the middle was cut only by the non-stainless steel knife, and the column on the right was cut only with the stainless steel knife:

 

After four hours, not much has happened. There is a little bit of browning in the apples, but nothing in the potatoes or onions:

 

After a total of six hours, the apples are continuing to brown, but the potatoes and onions don’t look any different: 

 

After a total of nine hours, all of the potatoes are starting to brown very slightly: 

 Twenty four hours into this experiment, the apples and potatoes are quite brown and dried up. The onions have dried a little bit but haven’t changed color: 

At the end of the experiment, and at each documented time, there is no discernible difference in brownness between the foods cut with the ceramic, stainless, or non-stainless blades. 

The point of this is not just to test this particular claim, but that it can be very easy to test claims that people make. While this particular claim turned out to be false, other claims are true. The purpose of science is not to find a particular answer, but to find out what the correct answer is. 

This experiment also highlights the importance of comparing different experimental conditions. A poor way to test this claim would have been to cut food only with a ceramic knife and to leave it out. I am used to potatoes turning brown in just a couple of hours, but in this experiment they were still white after nine hours. (It was fairly cold in my kitchen when I did this experiment which probably explains why it took so long for the potatoes to turn brown.) If I had only used potatoes and only cut them with a ceramic knife, I might have incorrectly concluded after nine hours that the ceramic knife prevented the browning. But by comparing the three knives simultaneously, I found that the potatoes all took longer than I expected to turn brown and not just the one I cut with the ceramic blade. 

A scientist doesn’t have to be someone with a fancy degree working in a laboratory. A scientist is just someone who makes discoveries about the world using the scientific method. Go be a scientist. 

 

 

Have a topic that you want me to cover? Let me know in the comments section.

Follow me on twitter @CGEppig

Only a Theory

With the recent debate between Bill Nye and Ken Ham, there has been a resurgence of people claiming that “evolution is just a theory.” This is something that really needs to stop. 

In common usage, the word “theory” means a guess. You might have a theory about who will play in the super bowl next year or what will happen later this season on American Horror Story. In this respect, your “theory” is as good as anybody else’s. This is not what we mean when we say “theory” as scientists.

In science, our level of certainty about an idea can be summed up with three words: speculation, hypothesis and theory.

A speculation is the lowest level of certainty. When we suspect that something is true but we haven’t done enough research on the topic to be confident in it, we use this word. This is like having a puzzle that is 10% completed and making a guess about what the missing pieces look like.

When a scientist forms a hypothesis, it means that they have learned everything or nearly everything that humans know about a subject, and they are confident in making a claim about something that is unknown. This is like having a puzzle that is completed but for a couple of pieces and making a claim about what the remaining pieces look like. My students will often define “hypothesis” as “an educated guess.” This is technically true, but it is not a particularly good definition because it implies a larger degree of uncertainty than there really is. While a particular hypothesis may not turn out to be correct, a scientist is confident enough in its accuracy to spend years of their life and tens of thousands of dollars determining whether or not it is true.

In scientific parlance, the word “theory” is synonymous with “fact.” It is an idea for which we have a great deal of evidence. A hypothesis can graduate to a theory once enough evidence is collected in support of it. Having a new theory is like adding a piece to a puzzle. When we know a new piece of information, it give us more insight into the things we do not yet know. It allows us to form new hypotheses that we couldn’t have before. So when someone says “x is only a theory,” they are saying “x is only a fact,” which doesn’t make much sense.

For reference, here are some ideas that are considered theories by scientists: The earth is spherical. The earth orbits the sun. General Relativity. Some diseases are caused by microorganisms. Plate tectonics. Electricity. The diversity of life on earth is the result of evolution by natural selection. Of these, the theory of evolution by natural selection is one of the most strongly supported — about as well-supported as the theory of the heliocentric solar system. It is a benchmark against which we can compare our certainty in other ideas. When something is supported as well as evolutionary theory, we know we’ve got it in the bag.

But when something is a theory, doesn’t that mean it hasn’t been proven? In a sense, yes. Outside of mathematics, scientists do not use the word “proof.” This convention is to constantly remind ourselves that we are not finished learning about things and that there is always a chance, however slim it may be, that we are wrong about something. Within mathematics, a statement can be proven within certain, carefully-established bounds. Some mathematically-proven statements are called “theorems.” Math aside, “theory” means as close to proven as we’re willing to say.

So what is a law? Isn’t a law higher than a theory? Not at all. As it is used, one could define it as a robust and generalizable observation. Unlike with a theory, there is no demand that we understand how it works. For example, Kleiber’s Law is the observation that the body size and metabolic rate of animals are related through the function y = x^3/4. There are hypotheses to explain why this is true, but these explanations are not a part of Kleiber’s law. Despite what many people think, theories do not graduate into laws once we get enough evidence for them. Rather, when we do have laws, we need to develop theories to explain them.

Whenever someone is trying to convince you that a particular scientific idea is wrong by claiming that it is “only a theory,” they are either deliberately trying to mislead you, or they are profoundly ignorant of science. These are the only two options.

 

Have a topic that you want me to cover? Let me know in the comments section.

Follow me on twitter @CGEppig