Updating PAJID Scores for State Supreme Court Justices (1970–2019)

Abstract

We build upon Brace, Langer, and Hall’s (2000, The Journal of Politics 62: 387–413) original measure of American state supreme court justice ideology – the PAJID scores. To do so, we gather new data on 1,666 state supreme court justices who served between 1970 and 2019 and update the PAJID scores throughout this period. Testing indicates that PAJID scores are a valid measure of state supreme court justices’ policy preferences and compare favorably, though less efficiently, to others such as Bonica and Woodruff (2015, The Journal of Law, Economics, and Organization 31: 472–98) and Windett, Harden, and Hall (2015, Political Analysis 23: 461–9). Given limited data availability for other ideological measures pre-1990 and post-2010, we conclude that these updated PAJID scores should prove attractive to scholars studying state courts during these periods and among those who desire additional state supreme court ideological data for robustness checks.

Introduction

Judicial ideology is a cornerstone of the public law literature and plays an important role in models of behavior. Scholars first measured judicial attitudes among members of the US Supreme Court but have since measured ideology in the lower federal courts as well as state supreme courts. Importantly, the first systematic effort to measure the political attitudes of American state supreme court justices emerged with Brace, Langer, and Hall’s (2000) party-adjusted surrogate judge ideology (PAJID) scores.

The intuition underlying PAJID scores is that, unlike federal jurists who are Executive-nominated and Senate-confirmed, state supreme court justices are selected by both appointment and election methods. Therefore, by using ideological information about the preferences of those who choose judges – voters in states that elect judges and political elites in states that appoint them – in addition to information about the party of a given judge, it is possible to create a surrogate measure for a given judge’s political preferences.

In recent years, newer methods have emerged that use different sources of information to measure judicial attitudes. Bonica and Woodruff (2015) use campaign donations/receipts to model ideology, while Windett, Harden, and Hall (2015) use state supreme court case votes to estimate temporally dynamic estimates. The Bonica and Woodruff (2015) estimates provide data for most state supreme court justices after 1990, while the Windett, Harden, and Hall (2015) estimates provide data for justices between 1995 and 2010. These estimates have proven to be reliable indicators of judicial attitudes and have helped advance the study of state judicial politics.

Despite the development of these newer ideological measures, a persistent problem remains in the state courts literature regarding how to account for judicial ideology for datasets that pre-date the 1990s and post-date the 2010s (e.g., Curry and Hurwitz 2016). The original PAJID scores cover the years prior to the early 1990s, while the other measures collectively cover more recent years from 1990 up to about 2010. To capture ideology over the full range of time, however, a PAJID update is the most practical approach.

For one thing, Bonica and Woodruff’s (2015) measure of state supreme court preferences relies upon campaign finance data for state actors that often do not exist in a digital format for years pre-dating 1990. For example, the National Institute on Money in State Politics, which was one key source for mapping Bonica’s (2014) ideological marketplace of state actors, currently does not report state supreme court campaign finance data prior to the year 2000, and it does not report gubernatorial or state legislative campaign finance data prior to the year 1990. ¹ One encounters similar difficulties when searching for digital campaign finance reports from state secretaries of states offices. ²

For another thing, Windett, Harden, and Hall’s (2015) measure of judicial ideology is similarly constrained by time. True enough, state supreme court justices’ votes are more easily retrieved compared to state campaign finance reports. Nevertheless, Windett, Harden, and Hall (2015) rely upon Bonica and Woodruff’s (2015) measure of judicial ideology in order to map their own dynamic estimates within each state court into a common space. Hence, the same problem emerges with respect to the availability of campaign finance reports. By comparison, PAJID estimates rely upon three pieces of information: (1) A justice’s political partisanship, (2) The method by which justices are selected, and (3) The ideology of those tasked with choosing state supreme court justices. Because these data are readily available, a reasonable approach for scholars in need of state supreme court ideological data pre-1990 or post-2010 is to update the PAJID measures until advances in data availability allow them to pursue these other types of ideological measures.

Given all this, we provide updated PAJID scores for state supreme court justices serving between 1970 and 2019. Our efforts yield a dataset with a total of 17,092 unique justice-year observations with complete PAJID data for 96.2 percent of all observations. Statistical testing indicates that these updated PAJID scores compare favorably, though less efficiently, to more recent measures. Nevertheless, we believe that these updated estimates will prove attractive to scholars who either want to perform robustness checks with other measures of state supreme court ideology or study state high courts outside of the years 1990 to 2010.

Updating the PAJID scores

To update PAJID scores, we replicate Brace, Langer, and Hall’s (2000) original methodology using more recent data. ³ First, we identified 1,666 unique individuals who worked on each state supreme court between 1970 and 2019 in the 50 American states. We then coded a dichotomous variable equal to “1” if a justice was a Democrat at the time of their selection, “0” otherwise. We also identified whether a state supreme court selects its members via popular election or elite appointment at the time a justice was selected.

The final variable in our PAJID update relies upon Berry et al.’s (2010) measure of state citizen and elite ideology in a given state and year (“Berry scores”). ⁴ Berry scores are measured on a scale from 0 to 100, where smaller values represent conservatism and larger values represent liberalism. For states using elective judicial selection methods, we incorporate Berry et al.’s (2010) citizen ideology value, and for states using appointive selection methods, we incorporate Berry et al.’s (2010) elite ideology value. We label either of these measures as a “preferences” indicator.

The first step in calculating the PAJID scores is to estimate a logistic regression that models the likelihood a given judge, $ j $ , is a Democrat given the preferences of their selectors:

(1) $$ \hat{\mathit{\Pr}}\left( Democra{t}_j=1\right)={\Lambda}^{-1}\left({\hat{\beta}}_0+{\hat{\beta}}_1 Preference{s}_j\right). $$

Using Equation (1), we then calculate the predicted probability a given judge is a Democrat, $ {\hat{p}}_j=\hat{\mathit{\Pr}}\left( Democra{t}_j=1\right) $ . Next, using $ \hat{p} $ , we calculate a pseudo-residual that is the difference between a justice’s partisanship and the predicted probability they are a Democrat:

(2) $$ {\tilde{u}}_j= Democra{t}_j-{\hat{p}}_j. $$

Equation (2) simply measures the degree to which the preferences of a given judge’s selectors fail to account for their partisanship.

Finally, a justice’s PAJID score is calculated accordingly:

(3) $$ PAJI{D}_j=\left( Preference{s}_j\hskip0.35em \ast \hskip0.35em {\tilde{u}}_j\right)+ Preference{s}_j. $$

The logic of Equation (3) is as follows. If the preferences of selectors in Equation (1) perfectly predicted partisanship, we would arrive at a value of $ {\tilde{u}}_j=0 $ , and no adjustment would be necessary to an individual’s PAJID score given the preferences of their selectors. Now, provided we find some $ {\tilde{u}}_j>0 $ , this would mean that we have a Democratic judge, but our model under-predicted the likelihood of them being a Democrat. Equation (3) would then add to a judge’s preferences an amount proportional to the size of the error in $ {\tilde{u}}_j $ . A similar logic holds for calculating the preferences of Republican judges with $ {\tilde{u}}_j<0 $ .

Assessing the validity of the updated PAJID scores

In Figure 1, we present median PAJID scores for each state supreme court between 1970 and 2019. Geographic and temporal trends speak to the face validity of the updated scores. For example, southern justices are approximately 21.6 percent less liberal compared to their counterparts in northern states ( $ t=7.53\Big) $ . And across all states and years, Democratic justices have a mean PAJID score of 68.0 compared to just 19.2 among Republicans – a 254.7 percent difference ( $ t=68.2 $ ). Given that the updated PAJID scores comport with one’s general expectations and knowledge of state politics and partisan power, we conclude they are facially valid measures of state supreme court ideology.

Figure 1. Median state supreme court PAJID scores by state (1970–2019).

Beyond facial validity, we also examine convergence validity in our updated PAJID estimates. Convergence validity assesses whether a given measure of a concept is associated with other common measures of that concept. To do so, we examine how PAJID scores compare with (1) Partisanship, (2) Ideology as measured by Bonica and Woodruff (2015), and (3) Ideology as measured by Windett, Harden, and Hall (2015). We present the results of this analysis in Table 1, which contains correlation coefficients among the variables of interest.

Table 1. Convergence validity analysis of state supreme court justice preferences

Note: Table entries represent Pearson’s correlation coefficients.

First, the results demonstrate a strong, positive correlation between a justice’s PAJID score and partisanship. This is intuitive given that PAJID scores are calculated using a judge’s partisanship. Next, we find that that PAJID scores are negatively associated with Bonica and Woodruff (BW) and Windett, Harden, and Hall’s (WHH) estimates. This is also expected, given that PAJID is measured on a conservative-to-liberal scale, while the other two are measured on a liberal-to-conservative scale. It is worth noting that the strength of association between Bonica and Woodruff (BH) and Windett, Harden, and Hall’s (WHH) scores are relatively higher compared to either’s association with PAJID. This likely reflects the differences in how these measures are estimated compared to PAJID. Even still, the degree of strength that PAJID correlates with these more recent ideological scores indicates moderate convergence.

Our final validity check examines the construct validity of the updated PAJID scores. Construct validity assesses whether a given measure is associated with outcomes in a theoretically related concept. Drawing upon literature related to the attitudinal and strategic models which hold that judges’ votes in cases are a function of their policy preferences, we examine whether updated PAJID scores are reliable predictors of judicial behavior. We also consider how favorably they compare to other, more recent ideological indicators.

For this analysis, we used a novel dataset of state supreme court cases related to abortion and capital punishment. To populate our sample, we searched Westlaw using keycite terms related to abortion and death penalty cases heard in the state supreme courts between 1970 and 2018. Our data include 213 cases (139 abortion and 74 death penalty), with a total of 1,511 judge-votes. Each justice’s vote is coded as either liberal (pro-abortion or anti-death penalty) or conservative (anti-abortion and pro-death penalty). Among all votes, 52.3 percent were in a conservative direction, while 47.8 percent were in a liberal direction.

Next, we employed logistic regression to model the likelihood a state supreme court justice cast a liberal vote in a case. Our primary focus is the effect of a justice’s policy preference upon their vote. We estimate four separate logistic regressions using a different preference measure in each to predict the directionality of a justice’s vote. ⁵ The results from these models appear in Table 2.

Table 2. Construct validity analysis of justice voting across four preference measures

Note: Table entries represent logistic regression coefficients (standard errors in parentheses). The dependent variable is whether a justice cast a liberal vote in a given decision (“1” if yes, “0” otherwise).

* denote statistical significance ( $ p<0.05 $ ).

Our results demonstrate that the updated PAJID scores, in addition to each of the other three measures, are significantly associated with judicial voting behavior – a good indication of construct validity. There are comparable proportional reductions in error (PRE) across all four models, which would seem to indicate that each preference measure captures a similar phenomenon. Given heterogeneous sample sizes, we also examine the change in the predicted probability of a liberal vote across each model. For the sake of comparison, we examine changes in the predicted probability given a shift in a preference measure from its minimum to its maximum.

The simplest measure under consideration is partisanship. A shift from minimum to maximum partisanship (Republican to Democrat) is associated with a 15.0 percent predicted change in the probability that a state supreme court justice casts a liberal vote, ceteris paribus. Next, a change from PAJID’s minimum to maximum is associated with a 23.1 percent predicted change in the probability of a liberal vote. A similar shift in Bonica and Woodruff’s (BW) measure is associated with a 99.6 percent predicted change in vote choice. Finally, a similar change in Windett, Harden,651 and Hall’s (WHH) measure is associated with a 409.0 percent change in vote choice.

From the above results, we can reaffirm Windett, Harden, and Hall’s (2015) conclusion regarding the efficiency of those scores compared to other alternatives. Nevertheless, WHH scores are only available for 25.6 percent of the observations in Table 2. While Bonica and Woodruff’s (2015) measure is the next most discriminating, it also has limits given sparse availability prior to 1990. Consequently, we conclude that if either of these former measures is of limited availability, PAJID scores are more efficient compared to rote partisanship and are of sufficient predictive power to offer additional robustness checks on more sophisticated measures.

Discussion

In this work, we have updated Brace, Langer, and Hall’s (2000) PAJID measure of state supreme court justice ideology between 1970 and 2019. While the state courts literature has provided newer estimates for state supreme court justices in recent years, these measures often only cover limited spans of time. Of the 17,092 justice-year observations we identified, updated PAJID scores are available for 96.2 percent of justice-years. By comparison, Bonica and Woodruff (2015) measures are available for 65.9 percent of all justice-years, while Windett, Harden, and Hall’s (2015) measure is available for only 31.7 percent. While we find that the updated PAJID estimates do not perform as efficiently as newer estimates, these new data should interest scholars who examine state supreme courts pre-1990 or post-2010, or who desire additional data for robustness checks.