Book contents
- Frontmatter
- Dedication
- Contents
- Preface and Acknowledgments
- Introduction
- Section I Thinking Like a Data Scientist
- Section II Communicating Like a Data Scientist
- Section III Applying the Tools of Data Science to Education
- 12 Waiting for Achilles
- 13 How Much Is Tenure Worth?
- 14 Detecting Cheating Badly: If It Could Have Been, It Must Have Been
- 15 When Nothing Is Not Zero: A True Saga of Missing Data, Adequate Yearly Progress, and a Memphis Charter School
- 16 Musing about Changes in the SAT: Is the College Board Getting Rid of the Bulldog?
- 17 For Want of a Nail: Why Worthless Subscores May Be Seriously Impeding the Progress of Western Civilization
- Section IV Conclusion: Don't Try Th is at Home
- Bibliography
- Sources
- Index
17 - For Want of a Nail: Why Worthless Subscores May Be Seriously Impeding the Progress of Western Civilization
from Section III - Applying the Tools of Data Science to Education
Published online by Cambridge University Press: 05 December 2015
- Frontmatter
- Dedication
- Contents
- Preface and Acknowledgments
- Introduction
- Section I Thinking Like a Data Scientist
- Section II Communicating Like a Data Scientist
- Section III Applying the Tools of Data Science to Education
- 12 Waiting for Achilles
- 13 How Much Is Tenure Worth?
- 14 Detecting Cheating Badly: If It Could Have Been, It Must Have Been
- 15 When Nothing Is Not Zero: A True Saga of Missing Data, Adequate Yearly Progress, and a Memphis Charter School
- 16 Musing about Changes in the SAT: Is the College Board Getting Rid of the Bulldog?
- 17 For Want of a Nail: Why Worthless Subscores May Be Seriously Impeding the Progress of Western Civilization
- Section IV Conclusion: Don't Try Th is at Home
- Bibliography
- Sources
- Index
Summary
How's your wife?
Compared to what?
Henny YoungmanStandardized tests, whether to evaluate student performance in coursework, to choose among applicants for college admission, or to license candidates for various professions, are often marathons. Tests designed to evaluate knowledge of coursework typically use the canonical hour, admissions tests are usually two to three hours, and licensing exams can take days. Why are they as long as they are? The first answer that jumps immediately to mind is the inexorable relationship between a test's length and its reliability. And so to know how long a test should be we must first ask, “How much reliability is needed?” Which brings us to Henny Youngman's quip, compared to what? There is a long answer to this, but let us begin with a short one, “a shorter test.”
Although a test score always gets more reliable as the test generating it gets longer, ceteris paribus, the law of diminishing returns sets in very quickly and so the marginal gain in reliability with increases in test length gets small in a big hurry. In Figure 17.1 we show the reliability of a typical professionally prepared test as a function of its length. It shows that the marginal gain of moving from a thirty-item test to a sixty- or even ninety-item one is not worth the trouble unless such additional reliability is required. As we will show shortly, it is hard to think of a situation where that would be the case, so why are tests as long as they are? What is to be gained through the excessive length? What is lost?
A Clarifying Example – the U.S. Census
Our intuitions can be clarified with an example, the Decennial U.S. Census. On midnight of January 1, 2010, the year of the last Census, it was estimated that there were 308,745,538 souls within the borders of the United States. The error bounds were ± 31,000. The budget of the 2010 Census was $13 billion, or approximately $42 per person. Is it worth this amount of money to just get a single number? Before answering this question, note that a single clerk, with access to an adding machine in a minute or two, could estimate the change from the previous Census (to an accuracy of ±0.1 percent).
- Type
- Chapter
- Information
- Truth or TruthinessDistinguishing Fact from Fiction by Learning to Think Like a Data Scientist, pp. 175 - 186Publisher: Cambridge University PressPrint publication year: 2015