Estimating Parties’ Policy Positions in Uruguay: Comparing Scaling Methods Based on Legislative Speeches and Roll-Call Votes

Abstract

This research note takes advantage of a novel dataset to analyze legislators’ behavior in Uruguay’s Parliament. Comparing the positions of legislators based on floor speeches and roll-call voting, it discusses the relationship between discourse and voting among individual legislators and parties. The dataset contains more than 57,000 speeches from more than 1,000 Uruguayan legislators between 1985 and 2015 and its related R package. The study estimates the parties’ policy positions on the basis of two data sources, roll-call votes and floor speeches, and then compares both results. Contrary to expectations, no clear association appears between the two scaling methods, demonstrating that vote and legislative speech may reflect the behavior of individual legislators with potentially conflicting goals. Strategic calculations or party discipline may be plausible explanations for the divergent results obtained from text and roll-call scaling methods.

Political scientists frequently attribute a relevant role to actors’ policy preferences in the explanation of a wide range of political phenomena and have sought to estimate them as accurately as possible. In recent years, text analysis has appeared as a low-cost and reliable alternative to more restrictive data-gathering strategies based on roll-call voting in legislatures, and as a matter of fact, it has been suggested as a reasonable substitute where roll-call data are simply not available (Slapin and Proksch 2014). The underlying presumption is that the use of different scaling techniques using archival, documental, or speech data is convergent with ideal point estimation using roll-call data.

This research note asks if this is the case. It uses a novel dataset gathering more than 57,000 speeches from about 1,000 Uruguayan legislators between 1985 and 2015. As a way of illustrating the potential uses of the dataset, we address a theoretically relevant question on legislative behavior applied to the Uruguayan Parliament: How is legislative speech related to voting? The study takes advantage of the functionalities included in the R package related to the database to collect all roll-call votes during the period, and then it compares voting patterns with text analysis among the same legislators, in line with some previous studies for advanced democracies (Schwarz et al. 2017).

We find an overall low level of convergence between the speech and the voting patterns of individual legislators, contrary to the underlying presumption but in line with the findings from previous studies like Proksch and Slapin 2012, Schwarz et al. 2017, and Izumi and Medeiros 2021. While both techniques allow for intraparty differentiation, the analysis of roll calls exhibits greater potential to capture differences between parties, showing that individual legislators may use speech and vote to pursue different goals.

Uruguay is an interesting case to address this issue because of its political stability, as well as the longevity and institutionalization of its main political parties (Mainwaring 2018; Piñeiro and Rosenblatt 2020). Additionally, it is a case of high reelection rates among both deputies and senators (Chasquetti 2014). This allows us to systematically compare individual legislators and parties across a long period. Uruguayan political parties also diverge in their policy positions and programmatic profiles, allowing us to analyze the relationship between voting and speech in Parliament.

This research provides a twofold contribution to the study of text analysis in social science and legislative studies. The original dataset overcomes a series of limitations in text analysis, making available flexible and ready-to-use data. This dataset is the result of applying different computational techniques implemented on the statistical software R (R Core Team 2017) to solve the trade-offs between extension and depth in dealing with large volumes of written content (Welbers et al. 2017). Likewise, this work enables the treatment of a particular type of content, related to collective actors (like parliaments), which comprise a large number of individual speakers who belong to other collective actors, such as parties, factions, or legislative committees.

The second contribution is to discuss the relationship between vote and speech in the Uruguayan Parliament. For this purpose, the data were used to collect every roll-call vote in a legislature that marginally uses this kind of voting rule. ¹ That process allowed for the estimation of W-nominate scores for each available vote during the period and further comparison of voting behavior with Wordscores as a standard scaling method for text analysis.

The article proceeds as follows. First it introduces the basic traits of Uruguayan politics and the set of techniques involving the dataset and briefly describes the data. Then it outlines two scaling methods for estimating the positions of individual legislators and parties, based on two types of data: roll-call votes and floor speeches. This application illustrates the potential utility of the dataset and its related R package for students interested in the ideological location of parties in the policy space. To conclude, it reviews the main findings of the research and suggests a research agenda for the study of a large array of political phenomena based on legislative speeches.

Gathering Legislative Speeches in the Uruguayan Parliament

Uruguay is one of the oldest and stablest democratic regimes in Latin America, leading, along with Costa Rica, most rankings measuring the quality of democracy in the region (Coppedge et al. 2017; Polity I 2012; Freedom House 2014). Its highly institutionalized party system (Mainwaring 2018), established by Nationalists and Colorados (PN and PC) during the nineteenth century and lately the Frente Amplio (FA, born in 1971), is considered the backbone of Uruguayan democracy.

To a large extent, such persistence of parties in Uruguayan politics is rooted in the ability of partisan elites to preserve clear programmatic and ideological stances in party competition. For this reason, Uruguay is an interesting case to reconstruct the preferences of parties and legislators over time. Given this opportunity, the main concern of this research note is whether the positions of Uruguayan parties are identifiable and differentiable across legislatures using text analysis. And more important: are different scaling methods used to estimate the preferences of parties convergent or divergent over time? Because these questions have received little attention in the research, particularly among Latin American countries (for an exception see Kim et al. 2018; and Izumi and Medeiros 2021 for the Brazilian case), the goal here is to contribute to current and future research on these topics, comparing two scaling methods that make use of a different type of data to address our research questions.

Traditionally, studies of political behavior have used legislative surveys, expert surveys, and party manifestos to reconstruct the preferences of political actors (Benoit and Laver 2006, 2007; Knutsen 1998; Laver and Hunt 1992). However, despite their widespread use and undeniable virtues, these sources have their own shortcomings (Slapin and Proksch 2014). On the one hand, there are temporal limitations to rebuilding the preferences of actors, due to generations of deceased voters, extinct political parties, or disappeared economic elites, among many other limitations, especially in third wave democracies. On the other hand, spatial variation is also challenging, as a great number of countries and other subnational institutions have made only limited advances in data availability.

More recently, scholars have developed new techniques to complement attitudinal studies, with data coming from roll-call votes in legislatures (Poole and Rosenthal 2000; Carroll and Poole 2014), social networks (Bond and Messing 2015), archival data, and public speeches (Kim et al. 2018; Laver and Garry 2000; Huber and Inglehart 1995; Dinas and Gemenis 2010; Laver et al. 2003; Welbers et al. 2017; Ramey et al. 2019; Slapin and Proksch 2014). In all cases, the goal has been to collect and systematize data to infer the preferences of different individual and collective actors over time and under different political and institutional contexts. Among these alternatives, W-nominate scores (Poole et al. 2011) and other ideal point estimation techniques (Jackman 2001; Martin and Quinn 2002; Kellermann 2012) based on roll calls have become one of the most reliable pieces of evidence to capture the preferences of political actors. Yet roll-call data availability is very limited beyond most advanced democracies.

In this context, text analysis has emerged as a reasonable option to bypass the limitations of roll-call records in most democracies around the world. As a matter of fact, text analysis has been considered a viable alternative to solve the limitations of time and space usually attributed to roll-call data (Curini et al. 2018; Lauderdale and Herzog 2016). However, the content analysis techniques (of examining speeches, documents, or written records of events) also presents some weaknesses that have captured the attention of experts (Gabel and Huber 2000; Klemmensen et al. 2007). On the one hand, the method requires a considerable amount of resources, particularly when it is applied to important volumes of documents or speeches. The need for reliable and systematic coding leads to significant demand for manual work. On the other hand, the ability to extend this technique to analyze content across time and space has important technical limitations. Coding and systematizing large volumes of text from manual work is onerous for one or even several encoders. Due to these limitations, extant applications have mainly focused on industrialized democracies. Our new dataset aims to expand those limits.

We built on extant techniques to develop an original dataset implemented on R (R Core Team 2017). The sequence starts with web scraping to obtain the Actas de Sesión; that is, the formal registry of all the activities involved in an individual legislative session. Some of these available documents on the Uruguayan legislative website are in HTML and some are in PDF format. The remaining documents that were not available on the web were manually scanned and transformed into PDF format.

The data generation process is flexible, giving researchers different options to compile the resulting dataset according to their interests. Depending on the research goal, scholars could be interested in different portions or contents of legislative speeches. The package includes a function allowing users to compile and recompile the dataset with different structures. For example, if someone is interested in scaling the position of parties, the dataset can be compiled using this unit of analysis. Also, the dataset can be recompiled with individual legislators as the unit of analysis, which enables adding new variables like factions, electoral districts, and district size, among many others that might interest students of legislative bodies and other research fields. Additionally, it is possible to track specific debates within and among legislatures (e.g., abortion, human rights, budgeting) using functions included in packages like quanteda (Benoit et al. 2018).

Figure 1 displays the building process of our dataset. First, the resulting structure of the text-mining process considers that some or several legislators speak at different times during the same session. Notice that this is a dataset of speeches performed by legislators who invariably belong to a certain political party. This rule excludes some individuals who participate in legislative sessions, like cabinet ministers, and staff members who are frequently invited to participate in committee and general sessions by request. The main function of the package has a search algorithm based on regular expressions that eliminate these cases.

Figure 1. Sequence of the Data Generation Process

Formally, figure 2 shows the resulting structure of the dataset. In order to work with individual speeches, we extract and cluster all the interventions for each legislator who speaks in a single session. Then we check whether the legislators’ names are correctly written and correct the multiple typing errors in the original document to cluster the interventions for each legislator. Compiling the dataset by individual legislators, the first variable is the legislator’s name, followed by other variables like the legislature number, the date and content of the speech, the legislator’s party, and the document type. ² Depending on the research interest, the dataset can be recompiled using the legislator´s party as the unit of analysis or any other added variable of interest, such as gender, district magnitude, or others.

Figure 2. Resulting Structure of the DGP

The resulting dataset contains more than 57,000 observations, covering more than 1,000 unique legislators from 7 political parties across 6 legislative periods for 3 different chambers: the Chamber of Representatives, the Senate, and the Asamblea General (General Assembly) (as the fusion of the upper and lower chambers). One of the main features of the dataset and the speech package is the easy integration with available packages and tools for computational text analysis, such as quanteda (Benoit et al. 2018) and tm (Feinerer and Hornik 2018; Feinerer et al. 2008). The structure of the dataset was conceived for that purpose, allowing researchers to compute scores, obtain summary statistics and frequency tables, and generate word clouds and other descriptive measures of interest. Also, it is possible to estimate scores for an individual legislator across different legislative periods, or for a specific set of legislators, parties, or any other combination in which the researcher could be interested. Table 1 shows the variables included in the dataset. In addition to the content of legislative speeches, several variables allow working with auxiliary information, like the legislator’s party, gender, or electoral district.

Table 1. Variables Included in the Dataset

Table 2 and figure 3 present descriptive statistics related to the speeches and the number of legislators in the dataset by legislative period. For example, in the first legislature after the transition to democracy (1985–90), a total of 236 legislators (between incumbents and alternates) spoke on the floor, delivering 7,645 speeches in 424 sessions. The average length of those speeches was 1,156 words. Figure 3 shows the distribution of legislative speeches among parties and across legislative periods. There is some systematic variation, suggesting that government parties tend to deliver shorter speeches on the floor while opposition parties tend to do the opposite.

Table 2. Descriptive Statistics of Legislative Speeches in the Uruguayan Parliament, 1985–2015

Figure 3. Number of Words in Individual Legislators’ Speeches in Uruguayan Parliament, 1985–2015, by party

During the first legislative period (1985–90), the legislators of the governing party (PC) delivered speeches with a lower median compared to the opposition parties (PN and FA). During the following three legislative periods, the PC and the PN governed together as a coalition, and both parties’ legislators made shorter speeches compared to the main opposition party, the FA. This was reversed in 2005, when the FA became the governing party. During the following two legislative periods (2005–15), the FA’s legislators delivered shorter legislative speeches than legislators of both opposition parties, the PN and the PC. This is a reasonable finding; time is important for legislative work, so governing parties are mainly concerned with approving legislation, while opposition parties may be more interested in delivering speeches for position-taking purposes.

Estimating Policy Positions of Uruguayan Parties Using Roll-Call Votes and Legislative Speeches

Given the importance of its political parties, Uruguayan democracy has been characterized as a partidocracia (Caetano et al. 1987). The continuity of the main party labels, as well as the salience of their ideological profiles, is a propitious context to address the main goal of this research note. That is, to estimate their policy positions through different scaling methods based on individual legislative speeches and roll-call voting, and compare them to find whether systematic differences emerge.

Although estimating ideological or policy positions from text and roll calls raises some concern because of the limitations that individual legislators might face when voting, both sources of data continue to be the central piece for estimating parties’ and individuals’ preferences (Carroll and Poole 2014, Slapin and Proksch 2014). For example, previous studies focused on the Uruguayan Senate and the Brazilian Congress have shown that ideal point estimates may be reflecting latent dimensions different from ideology (Izumi 2016; Zucco 2009, 2013; Izumi and Medeiros 2021). Also, according to some studies, legislative speeches can be restricted for strategic reasons in a way similar to voting (Proksch and Slapin 2012). However, other studies argue that legislative speeches are less constrained than voting, as long as legislators vote according to the party line (Schwarz et al. 2017).

Therefore, we ask whether Uruguayan legislators and parties are more or less constrained by these factors. We are aware that under some circumstances the government-opposition cleavage or party discipline can influence legislators’ behavior, and that is one additional reason to perform the comparison between speech and vote. If there were no restrictions on the behavior of individual legislators, one would expect to find congruence between what they say on the floor and how they vote. But precisely because it is expected that those restrictions are operative, we seek to systematically compare parties’ positions estimated from legislative speeches and roll calls.

To address our main concern, we first estimated parties’ positions from their legislators’ floor speeches. One of the most widespread methods to infer the parties’ policy positions from political texts is Wordscores (Laver et al. 2003; Lowe 2008; Lowe et al. 2011; Klemmensen et al. 2007). Based on all available floor speeches during the period 1985–2005, figure 4 shows the Wordscores for the three main political parties in the General Assembly. ³ Since Wordscores requires setting reference texts with known scores in the relevant dimension (Slapin and Proksch 2014), we used the speeches of individual legislators known for their ideological positions. ⁴ As the figure shows, the comparison among parties shows little ideological differentiation at the party level. Nevertheless, figure 4 also reveals an important intraparty variation in policy preferences. This variation reflects the well-known factional nature of Uruguayan parties, showing the differences that may be associated with the policy positions of factions (Buquet et al. 1998; Moraes 2008; Magar and Moraes 2012). Thus, based on text scaling, Uruguayan parties display a very similar ideological outlook, with an internal differentiation that we attribute to their traditionally factionalized structure, in line with the literature on Uruguayan parties.

Figure 4. Wordscores in Uruguayan General Assembly, 1985–2005

Second, we proceeded to estimate parties’ positions from roll-call data. Nominal votes are exceptional in Latin American politics, and Uruguay is not the exception (Carey 2009; Zucco 2013; Ainsley et al. 2020). Because the dataset allows for a specific search of words and text patterns, we searched for all the available roll calls during the period. In line with Ainsley et al. (2020), we identified roll-call votes from an automatized search for regular expressions. ⁵ However, unlike Ainsley et al. (2020) or Zucco (2013), we decided to keep only the roll calls associated with the treatment of executive vetoes that take place in the Asamblea General, since the vast majority of nominal votes in the lower or upper chambers refer to specific issues on administrative decisions of the chamber (e.g., the election of authorities and other administrative personnel) that cannot be coerced into yes or no, which is crucial to running the W-nominates algorithm.

Additionally, the treatment of executive vetoes is the only legislative instance in which nominal voting is mandatory, and thus it constitutes the most reliable and comparable source of roll calls in the Uruguayan Parliament. Therefore, limiting the analysis of roll calls to the executive vetoes allows us to deal with a more homogeneous corpus of nominal votes, restricted to substantive and divisive issues that emerge from potentially conflicting interests between legislative and executive powers. ⁶ Thus, we collected 77 roll-call votes during the 4 legislative periods under consideration, which were used to calculate another standard measure of ideal point estimation: W-nominate scores. ⁷

Figure 5 displays the W-nominate scores for the three main Uruguayan parties for the period 1985–2005, using all available roll calls. ⁸ Unlike Wordscores, W-nominate scores display a substantive level of party differentiation. Such differentiation at the party level is largely consistent with the programmatic or ideological differences observed by domestic and comparative students of parties who rely on elite survey data or public opinion polls. Furthermore, our W-nominate data also show a great deal of within-party differentiation, which resembles our estimations in Wordscores. Therefore, parties differentiate not only from each other but also among themselves, as students have observed using different sources of empirical evidence (Buquet et al. 1998; Moraes 2008; Magar and Moraes 2012; Chasquetti 2014). Overall, W-nominate scores are closer to the findings observed in the specialized literature.

Figure 5. W-nominate Scores in Uruguayan General Assembly, 1985–2005

We went on to compare the results of W-nominate scores with Wordscores. Interestingly, a large body of literature in recent years has pointed out that political discourse is an appropriate way to capture the policy preferences of different political agents (Benoit and Herzog 2017). This means that if there is an essential difference among parties (and alternatively among factions), it should be captured by analyzing political speeches, as well as other political expressions using documents, newspapers, and other sources. Yet our results using Wordscores do not reveal such differentiation at the party level. Instead, the only observed differentiation occurs within parties, as can be seen in figure 5.

Regarding the systematic differences among scaling methods, figure 6 shows that there is no clear association. The overall correlation is low (r = 0.22), and it is statistically significant at 99 percent confidence. When we look at partial correlations across legislative periods, some differences emerge. It can be seen that for the periods 1985–90 and 2000–2005, there is no clear association between speech and roll-call votes. But for the periods 1990–95 and 1995–2000. there is a moderate to high correlation (0.64 and 0.76, respectively). Not surprisingly, these two legislative periods were characterized by the functioning of governing coalitions between the PC and the PN that reconfigured Uruguayan politics and laid the foundations for the electoral reform of 1997, motivated by the electoral rise of the FA.

Figure 6. Comparing Parties’ Policy Positions Through Wordscores and W-nominate in Uruguayan General Assembly, 1985–2005

According to our estimations using speech, legislators seem to agree on a similar set of terms, using a highly homogeneous vocabulary that does not allow us to make a clear differentiation among parties with legislative representation. Such lack of differentiation contrasts with the voting patterns of parties in the legislature, where they are distinct from each other. Arguably, legislators may speak with a very similar jargon or set of words and expressions, from which we cannot identify different policy orientations. However, differentiation appears when parties and factions step up to vote.

These results are somewhat surprising, given scholars’ observations on the nature of programmatic parties in Uruguay, which would lead us to expect more congruence between speech and voting patterns. The divergence we found between the two scaling methods, based on a different type of data, does not let us confirm this expectation. While estimating the parties’ positions through roll-call data leads to differentiation among parties (as well as inside parties), the estimation using floor speeches does not confirm that differentiating pattern. These results are similar to those of Schwarz et al. (2017) for the Swiss legislature, who also find that text scaling from legislative speeches reveals intraparty differentiation more clearly than roll calls.

The divergence among scaling methods may be due to the methods themselves, or to the fact that both data sources are the product of different goals on the part of individual legislators. In other words, it is possible that when an individual legislator speaks on the floor, he or she intends to maximize the expressive function of legislative work (Andeweg 2014). But when the same legislator votes, he or she may be maximizing party goals, especially among disciplined parties like those in Uruguay. In this case, we expect to find some divergence among the policy positions based on legislative speeches compared with those coming from roll-call data, in line with what Izumi and Medeiros (2021) found for the Brazilian Senate.

To check the robustness of our results, we replicated the analysis using two alternative, unsupervised scaling methods based on both speech and roll calls. We estimated the parties’ policy positions through Wordfish (Slapin and Proksch 2008) to overcome the issues that may arise by selecting the reference texts. Wordfish is an unsupervised scaling algorithm designed to estimate policy positions based on word frequencies that, unlike Wordscores, does not require the identification of reference texts (Slapin and Proksch 2008). Our results using Wordfish show a very similar picture in terms of differentiation within and among parties. We also estimated parties’ positions from roll-call votes through IDEAL (Jackman 2001; Clinton et al. 2004), a Bayesian scaling method that allows us to overcome the problems that may arise as a consequence of the low number of roll calls, and perhaps more important, to avoid identifying the “polar” legislators, as in W-nominates. As with Wordfish, the estimations from IDEAL show very similar results to those obtained using W-nominates. ⁹

Both robustness tests allow us to be confident that our results are not heavily dependent on the decisions we made by selecting the reference texts and the “polar” legislators when scaling parties’ positions through Wordscores and W-nominates. In turn, and most important, they also suggest that the divergence we found between speech and vote is not dependent on the scaling method but on the fact that they are reflecting different behavior by individual legislators, factions, and parties. It seems that floor speeches in the General Assembly do not allow for capturing the variation in parties’ policy positions. Unlike roll calls, legislative discourse does not seem to be enough to differentiate among parties, but it does allow for intraparty differentiation, which is a key factor for the study of Uruguayan parties, given their high levels of fractionalization.

In sum, legislative speeches are not necessarily a good proxy to substitute for legislative voting using roll-call data, if we are to identify the ideological basis of political parties (Kim et al. 2018). Instead, the fact that speech and votes are not highly correlated indicates that partisan structures are strong enough to impose certain decisions beyond the preferences or political discourse of individual legislators. If this is the case, the role of leadership within parties and factions must be subject to further research.

Concluding Remarks

This research note introduces a novel dataset and its related R package, resulting from a data generation process that involved web scraping, cleaning, and organizing more than 57,000 floor speeches in Uruguayan Parliament between 1985 and 2015. Taken as a whole, the dataset constitutes an effort to address the difficulties involved in analyzing a large amount of data coming from text and discourse, especially when researchers have to deal with multiple speakers in deliberative bodies. The first contribution this research aims to make for researchers interested in these issues is the accessibility of the data, as well as the replicability of the entire data generation process. A set of new and preexisting techniques were used to build our dataset, which we expect will create incentives to explore new applications for a wide variety of research fields, based on specific needs.

Regarding the main concern of this research note, our results indicate that quantitative text analysis reveals diverging outcomes compared to scaling methods using roll-call data. More specifically, voting patterns in the Uruguayan case seem to be a more effective method to capture the ideological preferences of political parties, compared to legislative speeches that do not reveal differentiation among parties. The divergence between scaling parties’ positions from legislative speech and vote could be due to strategic reasons, as suggested by previous works. As Zucco (2013) and Izumi (2016) have demonstrated, the government-opposition dimension can alter the way legislators vote regardless of their ideological or policy preferences.

Additionally, as some studies have shown for advanced and developing democracies, legislative speeches may be a more reliable source for determining the sincere policy positions of individual legislators and parties, since they are less constrained by party leaders, compared to voting behavior (Proksch and Slapin 2012; Schwarz et al. 2017; Izumi and Medeiros 2021). This research note has shown that for the Uruguayan case, legislative speech shows intraparty differentiation but does not allow for differentiation between parties. Roll-call voting, in turn, shows greater potential to capture differences in party positions while also revealing important intraparty differences. Whether these differences are due to strategic reasons or to the government-opposition dimension is a relevant question that could be addressed in detail in future research.

From a methodological standpoint, our results show that scaling techniques based on text, like Wordscores or Wordfish, may not substitute for those based on roll calls, like W-nominate or other ideal point estimation techniques, and vice versa; instead, they complement each other in finding relevant differences between what legislators say and how they vote. This can be seen as our second contribution to the research on these topics, to the extent that the gap may be explained by the different goals that individual legislators face during the legislative work. These partially conflicting goals may lead legislators to balance speech and vote in a way that maximizes the benefits of party unity (e.g., belonging to a united party that constitutes a useful label for electoral purposes) while they signal differential positions to specific constituencies that are important for the legislator individually (e.g., they can help to define a personal profile for legislators seeking to send a signal to specific social groups that form the legislator’s basis of electoral support). Indeed, the divergence among scaling methods has to be seen as a puzzle open to future research, as it may indicate the role played by partisan structures, as well as leadership, in determining party discipline.