Abstract
Abstract. Most web surveys collect data through nonprobability or opt-in online panels, which are characterized by self-selection. A concern in online research is the emergence of professional respondents, who frequently participate in surveys and are mainly doing so for the incentives. This study investigates if professional respondents can be distinguished in online panels and if they provide lower quality data than nonprofessionals. We analyzed a data set of the NOPVO (Netherlands Online Panel Comparison) study that includes 19 panels, which together capture 90% of the respondents in online market research in the Netherlands. Latent class analysis showed that four types of respondents can be distinguished, ranging from the professional respondent to the altruistic respondent. A profile of professional respondents is depicted. Professional respondents appear not to be a great threat to data quality.
References
2010). AAPOR report on online panels. Deerfield, IL: AAPOR Executive Council. doi: 10.1093/poq/nfq048
. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38, 143–156.
(2012). Handbook of web surveys. New York, NY: Wiley.
(2011). Handbook of nonresponse in household surveys. New York, NY: Wiley.
(2008). Testing the stability of an acquiescence style factor behind two interrelated substantive variables in a panel design. Sociological Methods & Research, 36, 542–562.
(2014).
(Online panel research: History, concepts, applications and a look at the future . In M. CallegaroR. BakerJ. BethlehemA. S. GöritzJ. A. KrosnickP. J. LavrakasEds., Online panel research: A data quality perspective (pp. 1–22). New York, NY: Wiley.2009). National surveys via RDD telephone interviewing versus the Internet. Comparing sample representativeness and response quality. Public Opinion Quarterly, 73, 641–678.
(2005). The effects of survey frequency on panelists’ responses. Paper presented at the ESOMAR Conference on Panel Research, Budapest, Hungary. SSI white paper. Retrieved from http://www.surveysampling.com/ssi-media/Corporate/White%20Paper%202012/the-effects-of-survey-frequency.image
(1988). Statistical power analysis for the behavioral sciences. Hillsdale, NY: Erlbaum.
(2005). Understanding the online panelist. Worldwide panel research: Developments and progress. Amsterdam, The Netherlands: ESOMAR.
(2010). Professional web respondents and data quality. Paper presented at the 2010 AAPOR Conference, Chicago, IL.
(2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464–494.
(2008). Web survey methods: Introduction. Public Opinion Quarterly, 72, 831–835.
(2009). The devil is in the data. A respondent classification analysis comparing the best and worst survey respondents. Retrieved from http://www.usamp.com/objects/pdf/pr/DMS/DMS_Devil_in_the_Data_Whitepaper.pdf
(2012). The quality of Internet surveys. Bulletin de Methodologie Sociologique (BMS), 114, 68–78.
(1998). Detecting aberrant response patterns in multi-item scales: Some nonparametric indices and a computer program. Bulletin de Methodologie Sociologique (BMS), 58, 68–78.
(1978). Mail and telephone surveys. New York, NY: Wiley.
(2014). Internet, phone, mail, and mixed-mode surveys (4th ed.). New York, NY: Wiley.
(2009). Using non-probability samples for confusion surveys-mall intercepts and the Internet. Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/194/html
(2014). Statistics explained. Information society statistic. Retrieved from http://epp.eurostat.ec.europa.eu/statistics_explained/index.php/Information_society_statistics_-_households_and_individuals
. (2005). An experimental comparison of web and telephone surveys. Public Opinion Quarterly, 69, 370–392.
(2005). The “Professional Respondent” problem in online survey panels today. Presentation at the MRA Annual Conference, June 2005.
(2009). Variance between purchasing behaviour profiles in a wide spectrum of online sample sources. Retrieved from http://www.mktginc.com/pdf/Short_%20Variance.pdf
(2006). Incentives in web surveys: Methodological issues and a review. International Journal of Internet Science, 1, 58–70.
(1992). Measuring extreme response style. Public Opinion Quarterly, 56, 328–351.
(2014).
(The impact of speeding on data quality in nonprobability and freshly recruited probability-based online panels . In M. CallegaroR. BakerJ. BethlehemA. S. GöritzJ. A. KrosnickP. J. LavrakasEds., Online panel research: A data quality perspective (pp. 239–262). New York, NY: Wiley.1989). Survey errors and survey costs. New York, NY: Wiley.
(2014).
(Professional respondents in nonprobability online panels . In M. CallegaroR. BakerJ. BethlehemA. S. GöritzJ. A. KrosnickP. J. LavrakasEds., Online panel research: A data quality perspective (pp. 220–237). New York, NY: Wiley.2012). Speedsters, straigthliners, clickhappies and other fraudsters. Presented at the 2012 General Online Conference, Mannheim, Session A: Responsive behaviour. Retrieved from http://conftool.gor.de/conftool12/index.php?page=browseSessions&presentations=show&search=fraudsters
(2008).
(Acquiescence response bias . In P. J. LavrakasEd., Encyclopedia of survey research methods (pp. 3–4). Thousand Oakes, CA: Sage.1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.
(1987). An evaluation of a cognitive theory of response order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.
(1988). A test of the form-resistant correlation hypothesis. Public Opinion Quarterly, 52, 526–538.
(2008).
(Internet surveys . In E. D. de LeeuwJ. J. HoxD. A. DillmanEds., International handbook of survey methodology (pp. 264–284). New York, NY: Erlbaum.2000). The measurement of personal values in survey research. Public Opinion Quarterly, 64, 271–298.
(1987). Latent class analysis. Newbury Park, CA: Sage.
(2011). Internet coverage and coverage bias in countries across Europe and over time: Background, methods, question wording and bias tables. Retrieved from http://www.joophox.net/publist/pubenjh.htm
(2013). Internet coverage and coverage bias in Europe: Developments across countries and over time. Journal of Official Statistics, 29, 1–15.
(2007). A latent class analysis of positive psychosis symptoms based on the British Psychiatric Morbidity Survey. Personality and Individual Differences, 42, 1491–1502.
(1998–2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén and Muthén.
(2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569.
(2012a). Gamification 101-from theory to practice-part I. Quirk’s marketing research media. Retrieved from http://www.quirks.com/articles
(2012b). Gamification 101-from theory to practice-part II. Quirk’s marketing research media. Retrieved from http://www.quirks.com/articles
(1984).
(Cognitive science and survey methods . In T. JabineM. StrafJ. TanurR. TourangeauEds., Cognitive aspects of survey design: Building a bridge between disciplines (pp. 149–162). New York, NY: Springer.2013). The science of web surveys. Oxford, UK: Oxford University Press.
(2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
(2009).
(Rotte appels? Controle op kwaliteit van antwoordgedrag in het Infomart GfK online panel . In A. E. BronnerP. DekkerE. de LeeuwL. J. PaasK. de RuyterA. SmidtsJ. E. WieringaEds., Ontwikkelingen in het Marktonderzoek: Jaarboek MOA (pp. 6378). Haarlem, The Netherlands: Spaar en Hout [An English version “Do rotten apples spoil the barrel?” was presented at the 2007 ESOMAR panel research conference, see http://www.esomar.org/web/research_papers/Web-Panel_1656_Do-rotten-apples-spoil-the-whole-barrel.php].2004).
(Latent class analysis . In E. S. Lewis-BeckA. BrymanT. F. LiaoEds., The Sage encyclopedia of social sciences research methods (pp. 549–553). Thousand Oaks, CA: Sage.2006). The effects of panel recruitment and management on research results. A study across 19 panels ESOMAR. Panel Research, 2006, 79–100.
(2013). Understanding frequent survey responders on online panels. Nera Economic Consulting, Working Paper. Retrieved from http://www.nera.com/publications/archive/2013/understanding-frequent-survey-responders-on-online-panels.html
(