Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-05T08:05:25.075Z Has data issue: false hasContentIssue false
  • English
  • Français

Magic and Science in Multivariate Sentencing Models: Reflections on the Limits of Statistical Methods*

Published online by Cambridge University Press:  04 July 2014

David Weisburd
Get access

Extract

In the modern age, magic and science are seen to have little in common. Magic is the art of making effects seem real that aren't. Science is concerned with explaining effects that are real through observation and experimentation. Magic is a slight of hand, a manipulation that produces baffling outcomes that we know are illusions, even though we generally have no idea how the magician produces them. Science seeks to understand the underlying principles that govern events using methods that are clearly defined and that encourage replication. There would seem to be little ambiguity as to where magic ends and science begins.

Type
Theoretical Issues and Methodological Problems
Information
Israel Law Review , Volume 35 , Issue 2-3 , Summer-Autumn 2001 , pp. 225 - 248
Copyright
Copyright © Cambridge University Press and The Faculty of Law, The Hebrew University of Jerusalem 2001

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1 See Brehem, E., “Roger Bacon's Place in the History of Alchemy,” (1973) 23 AMBIX: The Journal of the Society of the History of Alchemy and Chemistry 1Google Scholar.

2 See Weisburd, David, “Good For What Purpose: Social Science, Race and Proportionality Review in New Jersey,” in Ewick, P., et al. eds., Social Science and Law (New York, Russell Sage, 1999) 258–288Google Scholar.

3 See Bienen, L.B., “The Proportionality Review of Capital Cases by State High Courts after Gregg: On the Appearance of Justice,” (1996) 87 Journal of Criminal Law and Criminology 130CrossRefGoogle Scholar.

4 Baldus, D.C. et al. , Equal Justice and The Death Penalty: A Legal and Empirical Analysis (Boston, Northwestern University Press, 1990)Google Scholar.

5 Baldus, D.C. and Woodworth, G.G., “Proportionality: The View of the Special Master,” (1993) 6 Chance 3 9–17CrossRefGoogle Scholar.

6 State v. Robert Marshall 130 N.J. 109 (1992).

7 D.C. Baldus, New Jersey Administrative Office of the Courts, Death Penalty Proportionality Review Project Final Report to the New Jersey Supreme Court 93 (1991).

8 Ibid., at 94.

9 Baldus, supra n. 5.

10 Baldus, supra n. 7, at 95.

11 D.C. Baldus, New Jersey Administrative Office of the Courts, Methodology Appendix, Death Penalty Proportionality Review Project Final Report to the New Jersey Supreme Court (1991) 1.

12 Weisburd, David and Britt, Chester, Statistics in Criminal Justice (Belmont, CA, West/Wadsworth, 2003) 508Google Scholar.

13 Baldus, supra n. 11.

14 David Weisburd and Joseph Naus, New Jersey Administrative Office of the Courts, Assessment of the Index of Outcomes Approach for Use in Proportionality Review, Report to Special Master Baime (1999).

15 Baldus addresses this problem as well in his technical appendix. After noting that the probability interpretation of this coefficient is “unreasonable on its face,” he goes on to suggest another interpretation — “that a case with a 4H finding and no other aggravating characteristics is as likely to receive the death penalty as a case with a combination of aggravating factors whose coefficients sum to about 13.” See Baldus, supra n. 11, at 6. Baldus and his colleagues then conducted a series of “bootstrap” replications that led them to question the reliability of their results. (It was noted in this regard that only 4 cases were found to have the 4H factor). Nonetheless, Baldus argues: “However, it is important to understand that for the cases which occurred (as opposed to hypothetical cases which might have occurred), it is statistically impossible to explain the dispositions of those six cases with 4D or 4H without including those factors in the model.” See Baldus, supra n. 11, at 8.

16 David Baime, Administrative Office of the Courts, Report to the New Jersey Supreme Court (1999) 76. Importantly, the court did not abandon social science methods more generally, as indicated by its use of regression modeling to answer other questions regarding death penalty sentencing. The court continues to use multivariate methods in its assessment of systematic proportionality review of possible racial impacts on death penalty sentencing. See David Weisburd and Joseph Naus, New Jersey Administrative Office of the Courts, Report to Special Master Baime, Re Systematic Proportionality Review (2001).

17 See Baime, supra n. 16, at 92.

18 See State v. Marshall, supra n. 6, at 147-148.

19 See Baime, supra n. 16, at 76.

20 Baldus himself uses this term in discussing the probability coefficient associated with the 4H factor discussed earlier (see Baldus, supra n. 11, at 5).

21 See Weisburd and Britt, supra n. 12 §16. See also Beck, M.S.L., Applied Regression: An Introduction (Newbury Park, CA, Sage, 1990)Google Scholar. It is also assumed that the variables included are measured correctly.

22 The assumption here in linear regression is that the error term and the included independent or predictor variables are independent. When a relevant predictor is excluded that is related to an included independent variable its effect is found in the error term which thus becomes correlated with the independent variable of interest. For a discussion of this assumption in regression see Pedhazur, E.J., Multiple Regression in Behavioral Research: Explanation and Prediction, § 2 (New York. Holt, Rinehart and Winston, 1982)Google Scholar. See also Weisburd and Britt, supra n. 12 § 16.

23 The example here assumes that an Ordinary Least Squares regression is being calculated. The same logic however applies to other types of regression approaches. Standardized regression coefficients are used to simplify the presentation.

24 See Pedhazur, supra n. 23, at 36.

25 See Baldus, supra n. 11, at 1.

26 J.W. Tukey, in R.S. Cohen and New Jersey Administrative Office of the Courts, Report to the Special Master (1997).

27 Fishman, Gideon and Rattner, Aryeh, “The Israeli Criminal Justice System in Action: Is Justice Administrated Differentially?” (1997) 13 Journal of Quantitative Criminology 7–28CrossRefGoogle Scholar.

28 For a discussion of the computation of standardized effect coefficients see Lipsey, M. and Wilson, D., Practical Meta-Analysis (California, Sage, 2001)Google Scholar.

29 Definition of moderate and large effects is drawn from Cohen, J., Statistical Power Analysis For The Behavioral Sciences (Hillsdale, Erlbaum Associates, 1988)Google Scholar. As noted by Cohen, these are only rough estimates developed with common sense in mind.

30 See Boruch, Robert et al. , “The Importance of Randomized Field Trials,” (2000) 44 Crime and DelinquencyGoogle Scholar; Farrington, David, Randomized Experiments in Crime and Justice (Morris, Norval and Tonry, Michael eds., Chicago, University of Chicago Press, 1983)Google Scholar; Kunz, Regina and Oxman, Andy, “The Unpredictability Paradox: Review of Empirical Comparisons of Randomized and Non-Randomized Clinical Trials,” (1998) 317 British Medical Journals 1185–90CrossRefGoogle Scholar; Weisburd, David and Lum, Cynthia, “Does Research Design Affect Study Outcome in Criminal Justice?” (2002) 578 The Annals 50–70Google Scholar.

31 This point was made more than a quarter century ago by Borhrnstedt and Carter in a review of problems of measurement and specification errors in regression analyses. Their conclusion then seems to me to still be relevant to much multivariate sentencing research: “We can only come to the sobering conclusion … that many of the published results based on regression analysis … are possible distortions of whatever reality may exist” See Bohrnstedt, G.W. and Carter, T.M., in Costner, H.L. ed., Robustness in Regression Analysis (San Francisco, Jossey-Bass, 1971) 43Google Scholar.