36 - Commentary to Part VI

Summary

Publishing is an implicit contract in empirical design. One essay illustrates the challenge in using incomplete designs that may not adhere to convention, but may be more ethical (less wasteful of subjects, either animal or human). Educating reviewers and other readers does burden the authors – and stretches word limits – but advancing our collective methodological and statistical practices is an often-unacknowledged but still important part of the contract.

Another feature of the implicit contract is full disclosure. Norms around this have varied from tell-the-best-story-without-lying manuscripts to provide-the-autobiography-of-the-idea manuscripts. Most often, the field falls between these extremes, although at present, the norms are shifting to recommend disclosure of all conditions, all participants, all measures, and all analyses, explaining how their inclusion or exclusion affects the reported results, and the rationale for the decisions taken. This potentially long-winded account can be available online as supplemental material, without jeopardizing the coherence of the scientific narrative. After all, coherence is another part of the implicit contract.

However, few of the current proposals for greater transparency recommend describing each and every failed pilot study. As noted, the reasons for failures to produce a given result are multiple, and supporting the null hypothesis is only one explanation. Deciding when one has failed to replicate is a matter of persistence and judgment.