How Much Evidence Is Enough?
Read the transcript of “How Much Evidence Is Enough?”
Film transcript: How Much Evidence Is Enough?
Chapter one used matched controls to identify an opsin-linked effect. When that effect looks large, how much evidence is enough? An experiment can record hundreds of responses without having hundreds of independent units. Repeated trials share each animal's expression, fibre placement and state. More trials sharpen that animal's mean. They do not create more animals. Eight mice each complete 40 trials. Does the study have 8 independent units, or 320?
In this design the independent unit is the animal. Keep its trials nested inside it during analysis. Treating every trial as independent erases shared variation within mice. The result then looks more precise than the experiment really is. For a two-group planning example, a standardized effect divides the difference between the means by the pooled spread. The same raw shift looks larger when outcomes cluster tightly. It looks smaller when the animals vary widely. Planning asks which effect is worth detecting, not which effect a small pilot happened to show. On a computed sample-size curve, smaller target effects demand much larger groups. The curve rises sharply because noisy overlap is harder to distinguish.
Small pilots give noisy effect estimates. The pilots that overshoot are also the most tempting to follow. If planning copies that lucky estimate, the confirmatory study can be too small before it begins. The same discipline applies to human imaging, where many brain-wide associations among people are small. A blood-oxygenation signal is not a spike count. A group average also does not guarantee stable rankings of individuals. Precision and reliability are different checks. A larger sample narrows uncertainty around an association.
Repeating the measurement asks whether people keep similar relative scores. A strong group average cannot substitute for either check when the goal is an individual biomarker. Keep three claims separate. Association says two measures vary together in a sample. Held-out prediction asks whether a model travels to new people. Causation asks what changes after the proposed cause is changed. Prediction alone does not supply that intervention. So size the study for the effect that matters, keep observations inside their independent units, and report uncertainty. Then match the final verb to the design. A precise association is still an association.
A controlled intervention can support a causal claim within its tested scope. Match the claim to the contrast and the sample.
Light alone changed firing and turning behaviour in mice with no opsin at all. What does your light-on versus light-off comparison actually show?
Continue in the lesson: Designing a Causal Circuit Experiment →
3:25 · Free, with captions and a transcript. The lesson has the full text, the questions and the sources.
More films from this lesson
How this film was made. Narration: an AI narrator voice (Kokoro, af_heart) reading narration written from this lesson's text. Animation: AI-generated in code (Manim) by Claude, a schematic with no drawn molecular structures. Captions come from the narration script. See something wrong? Report it.