17 - On journal editing as a probabilistic process (Co-authored with Richard Ofshe)

Summary

The process of evaluating papers for journal publication is a measurement process, similar to the other measurement processes of qualitative material in the social sciences. Let us assume that there are no biases in this process, and that it has about the same level of measurement efficiency as other measurement processes based on coding qualitative materials.

A good qualitative measurement technique rarely has a reliability coefficient of more than about ·50 (i.e., a correlation of ·50 of one measurement with another measurement by the same technique). Let us assume that all the covariation between two measures is due to the true variable we are trying to measure. Then the square root of the reliability of a measure is the upper limit of its validity. If we assume that journal editing as a measurement process is as good as good qualitative measurement can possibly be, given our experience in the social sciences, we would estimate the validity of a judgement of article quality to be about ·70.

Then the situation would be about as diagramed in the accompanying figure. The two major journals in sociology accept about 16 per cent of the papers submitted to them. If the quality of papers judged is approximately normal, we can say that papers judged to be 1 standard deviation or more above the mean of papers submitted will be accepted.