Do twenty experiments and roughly one "significant" result is a fluke

Statistics has an uncomfortable rule. Run twenty independent tests at the usual significance threshold and about one will look significant by pure chance, even when nothing is happening. A lab that publishes findings without accounting for this will, sooner or later, publish a confident result that is just noise. We run a growing programme of GEO experiments, so we built a guard against it. Here is how it works in plain terms.

The many-tests problem

Significance at the common 0.05 threshold means "there is a 1 in 20 chance of seeing this result if nothing real is going on." That is a fine bar for one test. Run many tests and the flukes accumulate: with enough experiments, a false positive is not unlucky, it is expected. This is why a single eye-catching p-value, taken alone, is weak evidence in any research programme.

What we built: a programme-wide ledger

After every experiment, we now run a Benjamini-Hochberg false-discovery-rate correction (what that is) across the primary result of every experiment we have ever run, not just the latest one. The output is a q-value next to each finding. In plain terms: the p-value asks "is this one result significant?" while the q-value asks "does it still hold up once we account for every other test we have run?" A finding earns a confident place on the dashboard only if it survives that programme-wide check.

We ran it across our experiments so far, and most of the significant findings survived. The ones that did not are demoted, not deleted, because a result that fails the stricter bar is still data. It is just not something we will state as fact.

Honesty all the way to the public page

There is one more guard. Our public findings page is hand-written, and hand-written pages drift from the underlying data over time. So we added a build test that fails if the public page ever claims a result is "supported" when the committed experiment does not survive the false-discovery check. In other words, the system cannot let the marketing overstate the science. If they disagree, the build breaks until we fix the page.

Why bother

Because the entire pitch of L8EntSpace is that you can trust the numbers. "Every claim on our dashboard is corrected for false discovery across our whole research programme" is a sentence almost no competitor can say, and it is only worth saying if it is enforced rather than promised. The treadmill of running experiments builds trust only if the experiments are scored honestly.

What to do next

When you read one of our findings, look for the q-value, not just the p-value. If a result survives false-discovery control, it has cleared a high bar. If it is marked preliminary, treat it as a lead, not a law. For how a single experiment is made trustworthy before it ever reaches the ledger, read How We Tried to Break Our Own GEO Findings.

Key Takeaways

• Run about twenty tests at p < 0.05 and roughly one false positive is expected by chance.
• We apply a Benjamini-Hochberg false-discovery correction across every experiment, producing a q-value per finding.
• A finding must survive the programme-wide check to earn a confident place on the dashboard.
• A build test stops the public page from claiming "supported" for anything that fails the check.
• Read the q-value, not just the p-value: it accounts for every test we have run.