4 - Alternative Measures of the Media Policy Signal

Summary

Chapter 3 laid out the building blocks for our measures of the media policy signal and presented a preliminary version of that signal across newspapers, television, and social media content. We now turn to a series of refinements and robustness tests, critical checks on the accuracy of our media policy signal measures. We begin with some comparisons between crowdsourced codes and those produced by trained student coders. Assessing the accuracy of crowdsourced data is important for the dictionary-based measures in the preceding chapter and for the comparisons with machine-learning-based measures introduced in this chapter. We then turn to crowdsourced content analyses of the degree to which extracted content reflects past, present, or future changes in spending. Our measures likely reflect some combination of these spending changes, and understanding the balance of each will be important for analyses in subsequent chapters. Finally, we present comparisons of dictionary-based measures and those based on machine-learning, using nearly 30,000 human-coded sentences and random forest models to replicate that coding across our entire corpus.