Watch or intervene?

There are two basic ways to collect data about whether one thing affects another.

• In an observational study you measure what is already happening and do not change anything. You might record how long students sleep and what grades they get.
• In an experiment you assign a treatment to the units and then measure what happens. You might tell some students to sleep eight hours and others to sleep six.

Both can find patterns. Only an experiment, done well, can say much about cause and effect. Here is why.

Say students who sleep more also get better grades in an observational study. Maybe sleep helps. But maybe students who sleep more also have fewer jobs, calmer homes or more organised study habits. Any of these could be the real reason, and they travel together with sleep. Such a hidden variable is called a confounder: it is linked to both the thing you study and the outcome, and it can mimic or hide an effect. World 5 spends four lessons on it.

An experiment fixes this with random assignment. If a coin toss (or a computer) decides who gets which treatment, then the groups differ only by chance in everything else: sleep habits, jobs, home life, things you never thought to measure. With enough units, the groups will be alike on average, so if the outcomes differ afterwards, the treatment is the only systematic difference left.

A good experiment usually has:

• A treatment group that gets the thing being tested and a control group that does not (or gets the standard alternative).
• Random assignment to those groups.
• A placebo when people's expectations matter: a harmless look-alike, so everyone believes they might be getting the treatment.
• Blinding: people do not know which group they are in (single blind), and the people measuring the results do not know either (double blind), so hopes cannot tilt the measurements.

An example. A gardener wants to know whether a plant food works. A weak design: give the food to the plants by the window, nothing to the shady ones, and compare. Light is now mixed up with the food. A better design: number 40 pots, let a random draw choose 20 for the food, keep the other 20 as the control, and put them all on the same shelf. Now light, water and pot size are balanced by chance, and a clear difference in growth points to the food. (A made-up example, for the idea only.)

Experiments are not always possible. We cannot randomly make people smoke for thirty years, or randomly assign a city to have a flood. For these questions we rely on observational studies, careful design, and several lines of evidence. We should just be honest that the evidence is weaker for causes.