Hostname: page-component-cd9895bd7-dzt6s Total loading time: 0 Render date: 2024-12-27T20:24:13.341Z Has data issue: false hasContentIssue false
  • English
  • Français

Investigating the promise and pitfalls of pulse surveys

Published online by Cambridge University Press:  29 March 2022

Matt I. Brown Open the ORCID record for Matt I. Brown [Opens in a new window]
Matt I. Brown*
Affiliation:
Geisinger Health System, Autism and Developmental Medicine Institute
*
Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Despite the growing popularity and marketing of pulse surveys, there is little research concerning this practice. To this end, this practice forum reports the results of a four-wave pulse survey that was conducted in a health care organization. Pulse surveys provided reliable estimates of overall engagement, but scores remained stable across 8 months. Practically no differences in group scores or trends could be found despite high participation (≍ 50%). Item responses displayed little differences between groups, ICC(1) ranging from .03 to .18, and poor discriminant validity. Based on these results, pulse surveys may be adequate for estimating overall employee sentiment but not useful for detecting change over time or differences between groups. These limitations should be considered when designing or implementing pulse surveys.

Keywords

pulse surveysemployee engagementresearch–practice gapsurvey measurement
Type
Practice Forum
Information
Industrial and Organizational Psychology , Volume 15 , Issue 1 , March 2022 , pp. 137 - 149
Copyright
© The Author(s), 2022. Published by Cambridge University Press on behalf of the Society for Industrial and Organizational Psychology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Special thanks to Rick Flynn for his leadership in working with the hospital leadership team to initiate the pulse survey program.

References

Bakker, A. B. (2014). Daily fluctuations in work engagement: An overview and current directions. European Psychologist, 19, 227–236. https://doi.org/10.1027/1016-9040/a000160 CrossRefGoogle Scholar
Balzer, W. K., Brodke, M. H., Kluse, C., & Zickar, M. J. (2019). Revolution or 30-year fad? A role for I-O psychology in Lean management. Industrial and Organizational Psychology: Perspectives on Science and Practice, 12, 215–233. https://doi.org/10.1017/iop.2019.23 CrossRefGoogle Scholar
Bliese, P. D., Maltarich, M. A., Hendricks, J. L., Hofmann, D. A., & Adler, A. B. (2019). Improving the measurement of group-level constructs by optimizing between-group differentiation. Journal of Applied Psychology, 104, 293–302. https://doi.org/10.1037/apl0000349 CrossRefGoogle ScholarPubMed
Boswell, W. R., Shipp, A. J., Payne, S. C., & Culbertson, S. S. (2009). Changes in newcomer job satisfaction over time: Examining the pattern of honeymoons and hangovers. Journal of Applied Psychology, 94, 844–858. https://doi.org/10.1037/a0014975 CrossRefGoogle ScholarPubMed
Bowling, N. A., Beehr, T. A., Wagner, S. H., & Libkuman, T. M. (2005). Adaptation-level theory, opponent process theory, and dispositions: An integrated approach to the stability of job satisfaction. Journal of Applied Psychology, 90, 1044–1053. https://doi.org/10.1037/0021-9010.90.6.1044 CrossRefGoogle Scholar
Brown, M. I. (2020). Comparing the validity of net promoter and benchmark scoring to other commonly used employee engagement metrics. Human Resource Development Quarterly, 3(4), 355–370. https://doi.org/10.1002/hrdq.21392 CrossRefGoogle Scholar
Cappelli, P. (2020, August 1). It’s time to get rid of employee surveys. Wall Street Journal. https://www.wsj.com/articles/its-time-to-get-rid-of-employee-surveys-11596246683 Google Scholar
Cappelli, P., & Tavis, A. (2018). HR goes agile. Harvard Business Review, 96(2), 46–52.Google Scholar
Church, A. H., & Oliver, D. H. (2006). The importance of taking action, not just sharing survey feedback. In Kraut, A. (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 102–130). Jossey-Bass.Google Scholar
Church, A. H., & Waclawski, J. (2020). Rediscovering the art of surveys for organizational change. In Macey, W. H. & Fink, A. A. (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 357–373). Oxford University Press. CrossRefGoogle Scholar
Colihan, J., & Waclawski, J. (2006). Pulse surveys: A limited approach with some unique advantages. In Kraut, A. (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 264–293). Jossey-Bass.Google Scholar
Cucina, J. M., Burtnick, S. K., Byle, K. A., Hayes, T. L., & Pillion, H. K. (2020, April 23–25). A 14-year analysis of unit-level job satisfaction/employee engagement score stability [Poster presentation]. The 35th Annual Conference of the Society for Industrial and Organizational Psychology, virtual.Google Scholar
Delmhorst, F. (2018). What if any science will do? Industrial and Organizational Psychology: Perspectives on Science and Practice, 11, 236–240. https://doi.org/10.1017/iop.2018.11 CrossRefGoogle Scholar
Dutta, S., & O’Rourke, E. M. (2020). Open-ended questions: The role of natural language processing and text analytics. In W. H. Macey & A. A. Fink (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 202–218). Oxford University Press.Google Scholar
Fisher, G. G., Matthews, R. A., & Gibbons, A. M. (2016). Developing and investigating the use of single-item measures in organizational research. Journal of Occupational Health Psychology, 21, 3–23. https://doi.org/10.1037/a0039139 CrossRefGoogle ScholarPubMed
Greenhaus, J. H., Parasuraman, A., & Wormley, W. M. (1990). Effects of race on organizational experiences, job performance evaluations, and career outcomes. Academy of Management Journal, 33, 64–86. https://doi.org/10.5465/256352 Google Scholar
Hackman, J. R. (2003). Learning more by crossing levels: Evidence from airplanes, hospitals, and orchestras. Journal of Organizational Behavior, 24, 905–922. https://doi.org/10.1002/job.226 CrossRefGoogle Scholar
Hahn, E., Gottschling, J., Konig, C. J., & Spinath, F. M. (2016). The heritability of job satisfaction reconsidered: Only unique environmental influences beyond personality. Journal of Business and Psychology, 31, 217–231. https://doi.org/10.1007/s10869-015-9413-x CrossRefGoogle Scholar
Harrison, M. (2018, November 29). How a U.S. health care system uses 15-minute huddles to keep 23 hospitals aligned. Harvard Business Review. https://hbr.org/2018/11/how-a-u-s-health-care-system-uses-15-minute-huddles-to-keep-23-hospitals-aligned Google Scholar
Harter, J. K., & Schmidt, F. L. (2006). Connecting employee satisfaction to business unit performance. In Kraut, A. (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 33–52). Jossey-Bass.Google Scholar
Harter, J. K., Schmidt, F. L., Asplund, J. W., Killham, E. A., & Agrawal, S. (2010). Causal impact of employee work perceptions on the bottom line of organizations. Perspectives on Psychological Science, 5, 378–389. https://doi.org/10.1177/1745691610374589 CrossRefGoogle ScholarPubMed
Ilies, R., Liu, X.-Y., Liu, Y., & Zheng, X. (2017). Why do employees have better family lives when they are highly engaged at work? Journal of Applied Psychology, 102, 956–970. https://doi.org/10.1037/apl0000211 CrossRefGoogle ScholarPubMed
Jolton, J. A., & Klein, C. (2020). Exploring the universe of pulse surveys and continuous listening opportunities. In Macey, W. H. & Fink, A. A. (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 53–67). Oxford University Press.CrossRefGoogle Scholar
Jones, F. F., Scarpello, V., & Bergmann, T. (1999). Pay procedures—What makes them fair? Journal of Occupational and Organizational Psychology, 72, 129–145. https://doi.org/10.1348/096317999166554 CrossRefGoogle Scholar
Kim, E. (2018, March 30). Amazon employees start their day by answering a simple question about work. CNBC. https://www.cnbc.com/2018/03/30/amazon-employee-reaction-to-hr-programs-connections-forte.html Google Scholar
Kraut, A. I. (2006). Moving the needle: Getting action after a survey. In Kraut, A. (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 1–32). Jossey-Bass.Google Scholar
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P V. Marsden & J. D. Wright (Eds.), Handbook of survey research (pp. 263–313). Emerald.Google Scholar
Kumar, A., & Paul, A. (2016). Mastering text mining with R. Packt Publishing.Google Scholar
Kunin, T. (1955). The construction of a new type of attitude measure. Personnel Psychology, 8, 65–77. https://doi.org/10.1111/j.1744-6570.1955.tb01189.x CrossRefGoogle Scholar
LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852. https://doi.org/0.1177/1094428106296642 CrossRefGoogle Scholar
Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority-majority differences and validity. Journal of Applied Psychology, 104, 715–726. https://doi.org/10.1037/apl0000367 CrossRefGoogle ScholarPubMed
Macey, W. H., & Fink, A. A. (2020). Surveys and sensing: Realizing the promise of listening to employees. In Macey, W. H. & Fink, A. A. (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 1–19). Oxford University Press.CrossRefGoogle Scholar
Macey, W. H., & Schneider, B. (2008). The meaning of employee engagement. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 3–30. https://doi.org/10.1111/j.1754-9434.2007.0002.x CrossRefGoogle Scholar
Mastrangelo, P. M. (2020). Improving the design and interpretation of sample surveys in the workplace. In Macey, W. H. & Fink, A. A. (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 38–52). Oxford University Press.CrossRefGoogle Scholar
Newman, D. A., & Harrison, D. A. (2008). Been there, bottled that: Are state and behavioral work engagement new and useful construct “wines”? Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 31–35. https://doi.org/10.1111/j.1754-9434.2007.00003.x CrossRefGoogle Scholar
Owen, D. (2018, January 29). Customer satisfaction at the push of a button. New Yorker. https://www.newyorker.com/magazine/2018/02/05/customer-satisfaction-at-the-push-of-a-button Google Scholar
Phan, W. M. J., Amrhein, R., Rounds, J., & Lewis, P. (2019). Contextualizing interest scales with emojis: Implications for measurement and validity. Journal of Career Assessment, 27, 114–133. https://doi.org/10.1177/1069072717748647 CrossRefGoogle Scholar
Power, B. (2016, May 24). Why John Deere measures employee morale every two weeks. Harvard Business Review. Retrieved from https://hbr.org/2016/05/why-john-deere-measures-employee-morale-every-two-weeks Google Scholar
Rigby, D. K., Elk, S., & Berez, S. (2020, May/June). The agile C-suite. Harvard Business Review. Retrieved from https://hbr.org/2020/05/the-agile-c-suite Google Scholar
Riza, S. D., Ganzach, Y., & Lui, Y. (2018). Time and job satisfaction: A longitudinal study of the differential roles of age and tenure. Journal of Management, 44, 2558–2579. https://doi.org/10.1177/0149206315624962 CrossRefGoogle Scholar
Rotolo, C. T., Church, A. H., Adler, S., Smither, J. W., Colquitt, A. L., Shull, A. C., Paul, K. B., & Foster, G. (2018). Putting an end to bad talent management: A call to action for the field of industrial and organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 11, 176–219. https://doi.org/10.1017/iop.2018.6 CrossRefGoogle Scholar
Rotolo, C. T., Fleck, C. R., & Shepherd, B. (2020). Current and future trends in employee survey practice: A view from employee survey practitioners. In Macey, W. H. & Fink, A. A. (Eds.), Employee surveys and sensing: Challenges and opportunities (pp. 425–442). Oxford University Press.CrossRefGoogle Scholar
Schiemann, W. A., & Seibert, J. H. (2017). Winning the HRM evidence-based impact award—lessons learned: A conversation with key stakeholders to the process. Industrial and Organizational Psychology: Perspectives on Science and Practice, 10, 314–326. https://doi.org/10.1017/iop.2017.25 CrossRefGoogle Scholar
Schleicher, D. J., Smith, T. A., Casper, W. J., Watt, J. D., & Greguras, G. J. (2015). It’s all in the attitude: The role of job attitude strength in job attitude–outcome relationships. Journal of Applied Psychology, 100, 1259–1274. https://doi.org/10.1037/a0038664 CrossRefGoogle ScholarPubMed
Schneider, B., Ashworth, S. D., Higgs, A. C., & Carr, L. (1996). Design, validity, and use of strategically focused employee attitude surveys. Personnel Psychology, 49, 695–705. https://doi.org/10.1111/j.1744-6570.1996.tb01591.x CrossRefGoogle Scholar
Schneider, B., Ehrhart, M. G., & Macey, W. H. (2013). Organizational climate and culture. Annual Review of Psychology, 64, 361–388. https://doi.org/10.1146/annurev-psych-113011-143809 CrossRefGoogle ScholarPubMed
Schneider, B., Yost, A. B., Kropp, A., Kind, C., & Lam, H. (2018). Workforce engagement: What is it, what drives it, and why it matters for organizational performance. Journal of Organizational Behavior, 39, 462–480. https://doi.org/10.1002/job.2244 CrossRefGoogle Scholar
Shuck, B., Nimon, K., & Zigarmi, D. (2017). Untangling the predictive nomological validity of employee engagement: Partitioning variance in employee engagement using job attitude measures. Group & Organization Management, 42, 79–112. https://doi.org/10.1177/1059601116642364 CrossRefGoogle Scholar
Silverman, R. E. (2014, December 2). Are you happy at work? Bosses push weekly polls. Wall Street Journal. http://www.wsj.com/articles/more-bosses-use-short-frequent-polls-to-measure-morale-1417550446 Google Scholar
Speer, A. B. (2020). Scoring dimension-level job performance from narrative comments: Validity and generalizability when using natural language processing. Organizational Research Methods. https://doi.org/10.1177/1094428120930815 CrossRefGoogle Scholar
Staw, B. M., & Cohen-Charash, Y. (2005). The dispositional approach to job satisfaction: More than a mirage, but not yet an oasis. Journal of Organizational Behavior, 26, 59–78. https://doi.org/10.1002/job.299 CrossRefGoogle Scholar
Steel, P., Schmidt, J., Bosco, F., & Uggerslev, K. (2019). The effects of personality on job satisfaction and life satisfaction: A meta-analytic investigation accounting for bandwidth-fidelity and commensurability. Human Relations, 72, 217–247. https://doi.org/10.1177/0018726718771465 CrossRefGoogle Scholar
Steel, P., Schmidt, J., & Shultz, J. (2008). Refining the relationship between personality and subjective well-being. Psychological Bulletin, 134, 138–161. https://doi.org/10.1037/0033-2909.134.1.138 CrossRefGoogle ScholarPubMed
Thorsen, C. J., Kaplan, S. A., Barsky, A. P., Warren, C. R., & de Chermont, K. (2003). The affective underpinnings of job perceptions and attitudes: A meta-analytic review and integration. Psychological Bulletin, 129, 914–945. https://doi.org/10.1037/0033-2909.129.6.914 Google Scholar
TINYpulse. (n.d.). The advanced guide to employee pulsing. https://www.tinypulse.com/#advanced-pulsing-guide Google Scholar
Van Rooy, D. L., Whitman, D. S., Hart, D., & Caleo, S. (2011). Measuring employee engagement during a financial downturn: Business imperative or nuisance? Journal of Business and Psychology, 26(2), 147–152. https://doi.org/10.1007/s10869-011-9225-6 CrossRefGoogle Scholar
Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82, 247–252. https://doi.org/10.1037/0021-9010.82.2.247 CrossRefGoogle ScholarPubMed
Welbourne, T. M. (2016). The potential of pulse surveys: Transforming surveys into leadership tools. Employment Relations Today, 43, 33–39. https://doi.org/10.1002/ert.21548 CrossRefGoogle Scholar
Winton, S., & Palmer, S. N. (2018). It’s about time: Measuring the pulse of engagement. Strategic HR Review, 17, 164–165. https://doi.org/10.1108/SHR-01-2018-0010 CrossRefGoogle Scholar
Young, H. R., Glerum, D. R., Wang, W., & Joseph, D. L. (2018). Who are the most engaged at work? A meta-analysis of personality and employee engagement. Journal of Organizational Behavior, 39, 1330–1346. https://doi.org/10.1002/job.2303 CrossRefGoogle Scholar