Reproducibility

Reproducible research – the fact that results can be independently reproduced by anyone is a hallmark of science. In practice, it is muddier. Not all studies have been reproduced; some studies are simply impossible to reproduce because of special dataset, circumstances, etc.

There was a big effort to reproduce results in psychological science¹, which spurred lots of controversies.

Researcher degree of freedom is another complication. Even with the exactly same data and hypothesis, researchers may produce very different results. See Breznau2022observing.

With generation capacity, the bottleneck shifts to verification and evaluation. We need more “shields”. We need to embrace AI to automatically verify as much as possible. See Xu2026scaling for instance.