Considering the Effects of Nonresponse on Correlations Between Surveyed Variables
A Simulation Study to Provide Context to Evaluate Survey Results
Abstract
Nonresponse to organizational surveys is a vexing problem. Although scholars have a reasonable understanding of how systematic nonresponse influences estimates of population means, they are likely to have less context for understanding how it impacts correlations. We present the results of a simulation study designed to provide a frame of reference for understanding the extent to which systematic nonresponse can bias observed correlations between surveyed variables. Although there were cases where notable levels of bias were observed in the correlations, the majority of situations simulated resulted in mean observed correlations that were within .05 of the true correlation. The variety of situations simulated allows researchers to evaluate the external validity of observed correlations in their data – a critical goal for survey researchers in practice and academics.
References
(1986). Towards a general model of non-random sampling and the impact on population correlation: Generalizations of Berkson’s Fallacy and range restriction. The British Journal of Mathematical and Statistical Psychology, 39, 90–105.
(2013). Perceived organizational support and embeddedness as key mechanisms for connecting socialization tactics to commitment and turnover among new employees. Journal of Organizational Behavior, 34, 350–369.
(2010). Response rates in organizational science, 1995–2008: A meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25, 335–349.
(2010). An introduction to modern missing data analyses. Journal of School Psychology, 48, 5–37.
(1988). Statistical power analysis for the behavioral sciences. Mahwah, NJ: Erlbaum.
(2008). Internet, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: Wiley.
(2010). Applied missing data analysis. New York, NY: Guilford Press.
(2012). Nonresponse in employee attitude surveys: A group-level analysis. Journal of Business and Psychology, 28, 1–16. doi: 10.1007/s10869-012-9260-y
(1996). Assessing the non-random sampling effects of subject attrition in longitudinal research. Journal of Management, 22, 627–652.
(2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.
(2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72, 167–189.
(1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin’s causal model. Psychological Methods, 3, 147–159.
(1966). A note on nonresponse in a mail survey. Public Opinion Quarterly, 30, 639–646.
(2011). On the relationship between job satisfaction and non-response in employee attitude surveys: A longitudinal field study. Journal of Occupational and Organizational Psychology, 84, 780–798.
(1965). A table for determining the increase in mean criterion score obtained by using a selection device. Journal of Industrial Psychology, 3, 33–42.
(2013). Consequences of survey nonresponse. The ANNALS of the American Academy of Political and Social Science, 645, 88–111.
(2002). Perceived organizational support: A review of the literature. The Journal of Applied Psychology, 87, 698–714.
(2003). Profiling active and passive nonrespondents to an organizational survey. The Journal of Applied Psychology, 88, 1104–1114.
(1998). Nonresponse to mailed surveys: A review and guide. Current Directions in Psychological Science, 7, 60–65.
(2000). Employee attitude surveys: Examining the attitudes of noncompliant employees. The Journal of Applied Psychology, 85, 284–293.
(2006). Understanding response behavior to an online special topics organizational satisfaction survey. Personnel Psychology, 59, 903–923.
(2007). Introduction: Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195–209.
(1998). Response rates in HRM/OB survey research: Norms and correlates, 1990–1994. Journal of Management, 24, 97–117.
(2000). Correction for range restriction: An expanded typology. The Journal of Applied Psychology, 85, 112–118.
(2001). The relationship between response rate and effect size in occupational health psychology research. Journal of Occupational Health Psychology, 6, 160–163.
(2006). “If you treat me right, I reciprocate”: Examining the role of exchange in survey response. Journal of Organizational Behavior, 27, 19–35.
(2007). Survey nonrespondents as bad soldiers: Examining the relationship between organizational citizenship and survey response behavior. International Journal of Selection and Assessment, 15, 449–459.
(2007). How may nonresponse affect findings in organizational surveys? The tendency-to-the-positive effect. International Journal of Stress Management, 14, 249–259.
(2007). Employee surveys administered online: Attitudes toward the medium, nonresponse, and data representativeness. Organizational Research Methods, 10, 241–261.
(1947). Research problems and techniques. (Report No. 3). Washington, DC: US Government Printing Office.
(2006). Oversurveying: Causes, consequences, and cures. In A. KrautEd., Getting action from organizational surveys: New concepts technologies and applications. San Francisco, CA: Jossey-Bass.
(2003, April). An exploratory look at response rates of sample, census, special topic and broad-based surveys. Paper presented at the annual convention of the Society for Industrial and Organizational Psychology, Chicago, IL.
