Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-745bb68f8f-s22k5 Total loading time: 0 Render date: 2025-01-26T08:50:24.878Z Has data issue: false hasContentIssue false

28 - Measuring Motivation in Educational Settings

A Case for Pragmatic Measurement

from Part VI - Methods, Measures, and Perspective

Published online by Cambridge University Press:  15 February 2019

K. Ann Renninger  and
Suzanne E. Hidi
By
Jeff J. Kosovich ,
Chris S. Hulleman  and
Kenneth E. Barron
K. Ann Renninger
Affiliation:
Swarthmore College, Pennsylvania
Suzanne E. Hidi
Affiliation:
University of Toronto
Get access

Summary

A reality of conducting motivation research in educational settings is that there are tensions between technical standards of research and practical constraints of a given situation. Although adherence to standards for high-quality measurement is critical for good-quality data to be collected, measurement also requires substantial resources to ensure quality. In the current chapter, we discuss several examples of real data collected in different educational settings using a pragmatic measurement framework. Based on contemporary measurement perspectives, the pragmatic measurement framework emphasizes building evidence-based arguments to support the use and interpretation of a measure. Example 1 explores college students’ attitudes toward general education classes. Example 2 tracks students’ classroom motivation over several time points. Example 3 assesses experimental differences from an online motivation intervention. Together the three examples cover a range of possible research questions that researchers may encounter. As a whole, this chapter demonstrates that important and meaningful insights can be gained using pragmatic approaches to measurement. Importantly, we discuss the trade-offs that researchers or other measure users must consider when adopting a pragmatic approach to measurement.

Type
Chapter
Information
The Cambridge Handbook of Motivation and Learning , pp. 713 - 738
Publisher: Cambridge University Press
Print publication year: 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

AERA, APA, & NCME. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
Atkinson, J. W. (1964). An introduction to motivation. Princeton, NJ: D. Van Nostrand Company.Google Scholar
Bandura, A. & Schunk, D. H. (1981). Cultivating competence, self-efficacy, and intrinsic interest through proximal self-motivation. Journal of Personality and Social Psychology, 41(3), 586–98. doi: 10.1037/0022-3514.41.3.586.Google Scholar
Barron, K. E., Grays, M., & Hulleman, C. S. (2014). Assessing motivation in general education. Poster presented at the Annual Meeting of the American Educational Research Association, Philadelphia, PA.Google Scholar
Barron, K. E. & Hulleman, C. S. (2015). Expectancy-value-cost model of motivation. In J. D. Wright (Ed.) International encyclopedia of the social & behavioral sciences (Vol. 8, pp. 503–9). Elsevier. doi: 10.1016/B978-0-08-097086-8.26099-6.Google Scholar
Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87. doi: 10.1080/10627197.2012.715014.Google Scholar
Bill and Melinda Gates Foundation. (2013). Ensuring fair and reliable measures of effective teaching. Retrieved from www.edweek.org/media/17teach-met1.pdf.Google Scholar
Bowman, N. A. (2010). Can 1st-year college students accurately report their learning and development? American Educational Research Journal, 47(2), 466–96. doi: 10.3102/0002831209353595.Google Scholar
Byrne, B. M. (2012). Structural equation modeling with Mplus: Basic concepts, applications, and programming. Multivariate applications series. New York, NY: Routledge.Google Scholar
Cronbach, L. J. & Meehl, P. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.Google Scholar
Duckworth, A. L. & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–51. doi: 10.3102/0013189X15584327.Google Scholar
Eccles, J. S. (1983). Expectancies, values, and academic behaviors. In Spence, J. T. (Ed.), Achievement-related motives and behaviors (pp. 119–46). San Francisco, CA: W. H. Freeman.Google Scholar
Eccles, J. S., Adler, T., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In Spence, J. T. (Ed.), Achievement and achievement motivation (pp. 75–146). San Francisco, CA: W. H. Freeman.Google Scholar
Embretson, S. E. & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Fredricks, J. A. & McColsky, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In Christenson, S. L., Reschly, A. L., & Wylie, C. (Eds.), Handbook of research on student engagement (pp. 763–82). New York, NY: Springer. doi: 10.1007/978-1-4614-2018-7_37.Google Scholar
Gaspard, H., Dicke, A., Flunger, B., Brisson, B. M., Häfner, I., Nagengast, B., & Trautwein, U. (2015). Fostering adolescents’ value beliefs for mathematics with a relevance intervention in the classroom. Developmental Psychology, 51(9), 1226–40. doi: 10.1037/dev0000028.Google Scholar
Goldstein, J. & Flake, J. K. (2016). Towards a framework for the validation of early childhood assessment systems. Educational Assessment, Evaluation and Accountability, 28(3), 273–93. doi: 10.1007/s11092-015-9231-8.Google Scholar
Harackiewicz, J. M., Canning, E. A., Tibbetts, Y., Giffen, C. J., Blair, S. S., Rouse, D. I., & Hyde, J. S. (2014). Closing the social class achievement gap for first-generation students in undergraduate biology. Journal of Educational Psychology, 106(2), 375–89. doi: 10.1037/a0034679.Google Scholar
Hektner, J. M. & Csikszentmihalyi, M. (1996). A longitudinal exploration of flow and intrinsic motivation in adolescents. In Annual Meeting of the American Educational Research Association, NYC, New York.Google Scholar
Hidi, S. & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–27. doi: 10.1207/s15326985ep4102_4.Google Scholar
Holstermann, N., Ainley, M., Grube, D., Roick, T., & Bögeholz, S. (2012). The specific relationship between disgust and interest: Relevance during biology class dissections and gender differences. Learning and Instruction, 22(3), 185–92. doi: 10.1016/j.learninstruc.2011.10.005.Google Scholar
Hulleman, C. S. (2007). The role of utility value in the development of interest and achievement. University of Wisconsin-Madison. Retrieved from http://files.eric.ed.gov/fulltext/ED498264.pdf.Google Scholar
Hulleman, C. S., Barron, K. E., Kosovich, J. J., & Lazowski, R. A. (2016). Expectancy-value models of achievement motivation in education. In Lipnevich, A. A., Preckel, F., & Robers, R. D. (Eds.), Psychosocial skills and school systems in the twenty-first century: Theory, research, and applications. (pp. 241–78). Basel, Switzerland: Springer. doi: 10.1007/978-3-319-28606-8.Google Scholar
Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology 102, 880–95. doi: 10.1037/a0019506.Google Scholar
Hulleman, C. S. & Harackiewicz, J. M. (2009). Promoting interest and performance in high school science classes. Science, 326(5958), 1410–12. doi: 10.1126/science.1177067.Google Scholar
Hulleman, C. S., Kosovich, J. J., Barron, K. E., & Daniel, D. B. (2016). Making connections: Replicating and extending the utility value intervention in the classroom. Journal of Educational Psychology. 109(3), 387–404. doi: 10.1037/edu0000146.Google Scholar
Hulleman, C. S., Schrager, S. M., Bodmann, S. M., & Harackiewicz, J. M. (2010). A meta-analytic review of achievement goal measures: Different labels for the same constructs or different constructs with similar labels? Psychological Bulletin, 136(3), 422–49. doi: 10.1037/a0018947.Google Scholar
Jacobs, J. E., Lanza, S., Osgood, D. W., Eccles, J. S., & Wigfield, A. (2002). Changes in children's self-competence and values: Gender and domain differences across grades one through twelve. Child Development, 73(2), 509–27. doi: 10.1111/1467-8624.00421.Google Scholar
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–35.Google Scholar
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. doi: 10.1111/jedm.12000.Google Scholar
Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York: The Guilford Press.Google Scholar
Kosovich, J. J. (2017). Pragmatic measurement for education science: A method-substance synergy of validation and motivation. Charolottesville, VA: University of Virginia.Google Scholar
Kosovich, J. J., Flake, J. K., & Hulleman, C. S. (2017). Short-term motivation trajectories: A parallel process model of expectancy-value. Contemporary Educational Psychology. 49(3), 130–139. doi: 10.1016/j.cedpsych.2017.01.004.Google Scholar
Kosovich, J. J., Hulleman, C. S., Barron, K. E., & Getty, S. (2015). A practical measure of student motivation: Establishing validity evidence for the expectancy-value-cost scale in middle school. The Journal of Early Adolescence, 35(5–6), 790–816. doi: 10.1177/0272431614556890.Google Scholar
Kosovich, J. J., Hulleman, C. S., & Flake, J. K. (2017). Practical measurement: An argument-based approach to exploring alternative psychometric validity evidence. Poster presented at the annual Society for Research on Educational Effectiveness, Washington, DC.Google Scholar
Krumm, A. E., Beattie, R., Takahashi, S., D'Angelo, C., Feng, M., & Cheng, B. (2016). Practical measurement and productive persistence: Strategies for using digital learning system data to drive improvement. Journal of Learning Analytics, 3(2), 116–38. doi: 10.18608/jla.2016.32.6.Google Scholar
Lazowski, R. A. & Hulleman, C. S. (2016). Motivation interventions in education: A meta-analytic review. Review of Educational Research, 86(2), 602–40. doi: 10.3102/0034654315617832.Google Scholar
Lewin, K., Dembo, T., Festinger, L., & Sears, P. S. (1944). Level of aspiration. In Hunt, J. M. (Ed.), Personality and the behavior disorders (pp. 333–78). New York, NY: Ronal Press. doi: 10.1037/10319-006.Google Scholar
Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11.Google Scholar
Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598–609. doi: 10.1037//0022-3514.46.3.598.Google Scholar
Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–86. doi: 10.1037/0022-0663.95.4.667.Google Scholar
Raynor, J. O. (1982). Future orientation, self-evaluation, and achievement motivation: Use of an expectancy × value theory of personality functioning and change. In Feather, N. T. (Ed.), Expectations and actions: Expectancy-value models in psychology (pp. 97–124). Hillsdale, NJ: Erlbaum.Google Scholar
Renninger, K. A. & Bachrach, J. E. (2015). Studying triggers for interest and engagement using observational methods. Educational Psychologist, 50(March), 58–69. doi: 10.1080/00461520.2014.999920.Google Scholar
Renninger, K. A. & Hidi, S. (2016). The power of interest for motivation and engagement. Abingdon: Routledge. Retrieved from http://works.swarthmore.edu/fac-education/91.Google Scholar
Rosenzweig, E. Q., Hulleman, C. H., Barron, K. E., et al. (in press). Promises and pit- falls of adapting utility value interventions for online mathematics courses. Journal of Experimental Education. doi: 10.1080/00220973.2018.1496059.Google Scholar
Rosenzweig, E. Q. & Wigfield, A. (2016). STEM motivation interventions for adolescents: A promising start, but further to go. Educational Psychologist, 51(2), 146–63. doi: 10.1080/00461520.2016.1154792.Google Scholar
Rotgans, J. I. & Schmidt, H. G. (2011). The role of teachers in facilitating situational interest in an active-learning classroom. Teaching and Teacher Education, 27(1), 37–42. doi: 10.1016/j.tate.2010.06.025.Google Scholar
Schmeiser, C. B. & Welch, C. J. (2006). Test development. In Brennan, R. L. (Ed.), Educational measurement (4th ed., pp. 307–53). Westport, CT: Praeger Publishers.Google Scholar
Schwarz, N. & Oyserman, D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. American Journal of Evaluation, 22(2), 127–60. doi: 10.1177/109821400102200202.Google Scholar
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74(1), 107–20. doi: 10.1007/s11336-008-9101-0.Google Scholar
Tibbetts, Y., Harackiewicz, J. M., Canning, E. A., Boston, J. S., Priniski, S. J., & Hyde, J. S. (2016). Affirming independence: Exploring mechanisms underlying a values affirmation intervention for first-generation students. Journal of Personality and Social Psychology, 110, 635–59. doi: 10.1037/pspa0000049.Google Scholar
Traub, R. E. & Rowley, G. L. (1991). Understanding reliability. Educational Measurement: Issues and Practice, 10(1), 37–45. doi: 10.1111/j.1745-3992.1991.tb00183.x.Google Scholar
Vroom, V. H. (1964). Work and motivation: Classic readings in organizational behavior. New York, NY: John Wiley & Sons.Google Scholar
Wise, S. L. & DeMars, C. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17.Google Scholar
Yeager, D. S., Bryk, A., Muhich, J., Hausman, H., & Morales, L. (2013). Practical measurement. Palo Alto, CA: Carnegie Foundation for the Advancement of Teaching. www.carnegiefoundation.org/resources/publications/practical-measurement/.Google Scholar
Yeager, D. S. & Walton, G. M. (2011). Social-psychological interventions in education: They're not magic. Review of Educational Research, 81(2), 267–301. doi: org/10.3102/0034654311405999.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×