Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

The Validity of Crowdsourcing Data in Studying Anger and Aggressive Behavior

A Comparison of Online and Laboratory Data

Published Online:December 30, 2015https://doi.org/10.1027/1864-9335/a000256

Abstract

Abstract. Crowdsourcing platforms provide an affordable approach for recruiting large and diverse samples in a short time. Past research has shown that researchers can obtain reliable data from these sources, at least in domains of research that are not affectively involving. The goal of the present study was to test if crowdsourcing platforms can also be used to conduct experiments that incorporate the induction of aversive affective states. First, a laboratory experiment with German university students was conducted in which a frustrating task induced anger and aggressive behavior. This experiment was then replicated online using five crowdsourcing samples. The results suggest that participants in the online samples reacted very similarly to the anger manipulation as participants in the laboratory experiments. However, effect sizes were smaller in crowdsourcing samples with non-German participants while a crowdsourcing sample with exclusively German participants yielded virtually the same effect size as in the laboratory.

References

  • Andover, M. S. (2014). Non-suicidal self-injury disorder in a community sample of adults. Psychiatry Research, 219, 305–310. doi: 10.1016/j.psychres.2014.06.001 First citation in articleCrossref, Google Scholar

  • Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. A., Fiedler, K., … Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. doi: 10.1002/per.1919 First citation in articleCrossref, Google Scholar

  • Baumeister, R. F., Bratslavsky, E., Muraven, M. & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74, 1252–1265. doi: 10.1037/0022-3514.74.5.1252 First citation in articleCrossref, Google Scholar

  • Berkowitz, L. (1989). Frustration-aggression hypothesis: Examination and reformulation. Psychological Bulletin, 106, 59–73. doi: 10.1037/0033-2909.106.1.59 First citation in articleCrossref, Google Scholar

  • Bizer, G. Y., Hart, J. & Jekogian, A. M. (2012). Belief in a just world and social dominance orientation: Evidence for a mediational pathway predicting negative attitudes and discrimination against individuals with mental illness. Personality and Individual Differences, 52, 428–432. doi: 10.1016/j.paid.2011.11.002 First citation in articleCrossref, Google Scholar

  • Boynton, M. H. & Richman, L. S. (2014). An online daily diary study of alcohol use using Amazon’s Mechanical Turk. Drug and Alcohol Review, 33, 456–461. doi: 10.1111/dar.12163 First citation in articleCrossref, Google Scholar

  • Bradley, M. M. & Lang, P. J. (1994). Measuring emotion: The Self-Assessment Manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49–59. doi: 10.1016/0005-7916(94)90063-9 First citation in articleCrossref, Google Scholar

  • Buhrmester, M., Kwang, T. & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi: 10.1177/1745691610393980 First citation in articleCrossref, Google Scholar

  • Burns, D. J., Hart, J., Kramer, M. E. & Burns, A. D. (2014). Dying to remember, remembering to survive: Mortality salience and survival processing. Memory, 22, 36–50. doi: 10.1080/09658211.2013.788660 First citation in articleCrossref, Google Scholar

  • Buss, A. H. (1963). Physical aggression in relation to different frustrations. Journal of Abnormal and Social Psychology, 67, 1–7. doi: 10.1037/h0040505 First citation in articleCrossref, Google Scholar

  • Campbell, T. H. & Kay, A. C. (2014). Solution aversion: On the relation between ideology and motivated disbelief. Journal of Personality and Social Psychology, 107, 809–824. doi: 10.1037/a0037963 First citation in articleCrossref, Google Scholar

  • Casler, K., Bickel, L. & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160. doi: 10.1016/j.chb.2013.05.009 First citation in articleCrossref, Google Scholar

  • Chandler, J., Mueller, P. & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. doi: 10.3758/s13428-013-0365-7 First citation in articleCrossref, Google Scholar

  • Chandler, J., Paolacci, G., Peer, E., Mueller, P. & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 1131–1139. doi: 10.1177/095679761558511 First citation in articleCrossref, Google Scholar

  • Chopik, W. J. & Edelstein, R. S. (2014). Death of a salesman: Webpage-based manipulations of mortality salience. Computers in Human Behavior, 31, 94–99. doi: 10.1016/j.chb.2013.10.022 First citation in articleCrossref, Google Scholar

  • Chow, J. T. & Lau, S. (2015). Nature gives us strength: Exposure to nature counteracts ego-depletion. The Journal of Social Psychology, 155, 70–85. doi: 10.1080/00224545.2014.972310 First citation in articleCrossref, Google Scholar

  • Costa, S., Ntoumanis, N. & Bartholomew, K. J. (2015). Predicting the brighter and darker sides of interpersonal relationships: Does psychological need thwarting matter? Motivation and Emotion, 39, 11–24. doi: 10.1007/s11031-014-9427-0 First citation in articleCrossref, Google Scholar

  • Davis, J. R. & Reyna, C. (2015). Seeing red: How perceptions of social status and worth influence hostile attributions and endorsement of aggression. The British Journal of Social Psychology. Advance online publication. doi: 10.1111/bjso.12109 First citation in articleCrossref, Google Scholar

  • DeJoy, D. M. (1985). Information input rate, control over task pacing, and performance during and after noise exposure. Journal of General Psychology, 112, 229–242. doi: 10.1080/00221309.1985.9711008 First citation in articleCrossref, Google Scholar

  • DeWall, C. N. & Anderson, C. A. (2011). The general aggression model. In P. R. ShaverM. MikulincerEds., Human aggression and violence: Causes, manifestations, and consequences (pp. 15–33). Washington, DC: American Psychological Association. doi: 10.1037/12346-001 First citation in articleCrossref, Google Scholar

  • Dynamo. (2014). Guidelines for academic requesters (Version 1.1). Retrieved from http://guidelines.wearedynamo.org/ First citation in articleGoogle Scholar

  • Faul, F., Erdfelder, E., Buchner, A. & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160. First citation in articleCrossref, Google Scholar

  • Geen, R. G. (1968). Effects of frustration, attack, and prior training in aggressiveness upon aggressive behavior. Journal of Personality and Social Psychology, 9, 316–321. doi: 10.1037/h0026054 First citation in articleCrossref, Google Scholar

  • Gentile, D. A., Anderson, C. A., Yukawa, S., Ihori, N., Saleem, M., Ming, L. K., … Sakamoto, A. (2009). The effects of prosocial video games on prosocial behaviors: International evidence from correlational, longitudinal, and experimental studies. Personality and Social Psychology Bulletin, 35, 752–763. doi: 10.1177/0146167209333045 First citation in articleCrossref, Google Scholar

  • Giancola, P. R. & Zeichner, A. (1995). Construct validity of a competitive reaction-time aggression paradigm. Aggressive Behavior, 21, 199–204. doi: 10.1002/1098-2337(1995)21:3<199::AID-AB2480210303>3.0.CO;2-Q First citation in articleCrossref, Google Scholar

  • Goldin, G. & Darlow, A. (2013). TurkGate (Version 0.4.0) [Software]. Available from http://gideongoldin.github.com/TurkGate/ First citation in articleGoogle Scholar

  • Goodman, J. K., Cryder, C. E. & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224. doi: 10.1002/bdm.1753 First citation in articleCrossref, Google Scholar

  • Harwell, M. R., Rubinstein, E. N., Hayes, W. S. & Olds, C. C. (1992). Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational Statistics, 17, 315–339. doi: 10.2307/1165127 First citation in articleCrossref, Google Scholar

  • Hauser, D. J. & Schwarz, N. (2015). Attentive Turkers: MTurk participants perform better on online attention checks than subject pool participants. Behavior Research Methods. Advance online publication. doi: 10.3758/s13428-015-0578-z First citation in articleCrossref, Google Scholar

  • Hirth, M., Hoßfeld, T. & Tran-Gia, P. (2011). Anatomy of a crowdsourcing platform – Using the example of Microworkers.com. Paper presented at the Workshop on Future Internet and Next Generation Networks (FINGNet), Seoul, Korea. Retrieved from http://www3.informatik.uni-wuerzburg.de/staff/hossfeld/Transparentia/pubs/conf_410.pdf First citation in articleGoogle Scholar

  • Horton, J. J., Rand, D. G. & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399–425. doi: 10.1007/s10683-011-9273-9 First citation in articleCrossref, Google Scholar

  • Jonason, P. K., Wee, S., Li, N. P. & Jackson, C. (2014). Occupational niches and the dark triad traits. Personality and Individual Differences, 69, 119–123. doi: 10.1016/j.paid.2014.05.024 First citation in articleCrossref, Google Scholar

  • Krahé, B. & Bieneck, S. (2012). The effect of music‐induced mood on aggressive affect, cognition, and behavior. Journal of Applied Social Psychology, 42, 271–290. doi: 10.1111/j.1559-1816.2011.00887.x First citation in articleCrossref, Google Scholar

  • Krahé, B. & Fenske, I. (2002). Predicting aggressive driving behavior: The role of macho personality, age, and power of car. Aggressive Behavior, 28, 21–29. doi: 10.1002/ab.90003 First citation in articleCrossref, Google Scholar

  • Lickel, B., Kushlev, K., Savalei, V., Matta, S. & Schmader, T. (2014). Shame and the motivation to change the self. Emotion, 14, 1049–1061. doi: 10.1037/a0038235 First citation in articleCrossref, Google Scholar

  • Lieberman, J. D., Solomon, S., Greenberg, J. & McGregor, H. A. (1999). A hot new way to measure aggression: Hot sauce allocation. Aggressive Behavior, 25, 331–348. doi: 10.1002/(SICI)1098-2337(1999)25:5<331::AID-AB2>3.0.CO;2-1 First citation in articleCrossref, Google Scholar

  • Martin, M. (1990). On the induction of mood. Clinical Psychology Review, 10, 669–697. doi: 10.1016/0272-7358(90)90075-L First citation in articleCrossref, Google Scholar

  • Mason, W. & Suri, S. (2012). Conducting behavioral research on Amazon’s mechanical Turk. Behavior Research Methods, 44, 1–23. doi: 10.3758/s13428-011-0124-6 First citation in articleCrossref, Google Scholar

  • Paolacci, G. & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188. doi: 10.1177/0963721414531598 First citation in articleCrossref, Google Scholar

  • Paolacci, G., Chandler, J. & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419. First citation in articleGoogle Scholar

  • Peer, E., Vosgerau, J. & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031. doi: 10.3758/s13428-013 First citation in articleCrossref, Google Scholar

  • Phillips, L. H., Henry, J. D., Hosie, J. A. & Milne, A. B. (2006). Age, anger regulation and well-being. Aging & Mental Health, 10, 250–256. doi: 10.1080/13607860500310385 First citation in articleCrossref, Google Scholar

  • Reips, U. (2002). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256. doi: 10.1026/1618-3169.49.4.243 First citation in articleLink, Google Scholar

  • Rick, S. I., Pereira, B. & Burson, K. A. (2014). The benefits of retail therapy: Making purchase decisions reduces residual sadness. Journal of Consumer Psychology, 24, 373–380. doi: 10.1016/j.jcps.2013.12.004 First citation in articleCrossref, Google Scholar

  • Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J. & Vukovic, M. (2011). An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. Paper presented at the fifth International AAAI Conference on Weblogs and Social Media), Barcelona, Spain. Retrieved from http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2778/3295 First citation in articleGoogle Scholar

  • Routledge, C., Ostafin, B., Juhl, J., Sedikides, C., Cathey, C. & Liao, J. (2010). Adjusting to death: The effects of mortality salience and self-esteem on psychological well-being, growth motivation, and maladaptive behavior. Journal of Personality and Social Psychology, 99, 897–916. doi: 10.1037/a0021431 First citation in articleCrossref, Google Scholar

  • Saleem, M., Anderson, C. A. & Barlett, C. P. (2015). Assessing helping and hurting behaviors through the Tangram help/hurt task. Personality and Social Psychology Bulletin, 41, 1345–1362. doi: 10.1177/0146167215594348 First citation in articleCrossref, Google Scholar

  • Saleem, M., Anderson, C. A. & Gentile, D. A. (2012). Effects of prosocial, neutral, and violent video games on children’s helpful and hurtful behaviors. Aggressive Behavior, 38, 281–287. doi: 10.1002/ab.21428 First citation in articleCrossref, Google Scholar

  • Schmider, E., Ziegler, M., Danay, E., Beyer, L. & Bühner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6, 147–151. doi: 10.1027/1614-2241/a000016 First citation in articleLink, Google Scholar

  • Severance, L., Bui-Wrzosinska, L., Gelfand, M. J., Lyons, S., Nowak, A., Borkowski, W., … Yamaguchi, S. (2013). The psychological structure of aggression across cultures. Journal of Organizational Behavior, 34, 835–865. doi: 10.1002/job.1873 First citation in articleCrossref, Google Scholar

  • Shapiro, D. N., Chandler, J. & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220. doi: 10.1177/2167702612469015 First citation in articleCrossref, Google Scholar

  • Simcox, T. & Fiez, J. A. (2014). Collecting response times using Amazon Mechanical Turk and Adobe Flash. Behavior Research Methods, 46, 95–111. doi: 10.3758/s13428-013-0345-y First citation in articleCrossref, Google Scholar

  • Spielberger, C. D., Sydeman, S. J., Owen, A. E. & Marsh, B. J. (1999). Measuring anxiety and anger with the State-Trait Anxiety Inventory (STAI) and the State-Trait Anger Expression Inventory (STAXI). In M. E. MaruishEd., The use of psychological testing for treatment planning and outcomes (2nd ed. pp. 993–1021). Mahwah, NJ: Erlbaum. First citation in articleGoogle Scholar

  • Summerville, A. & Chartier, C. R. (2013). Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk. Behavior Research Methods, 45, 116–124. doi: 10.3758/s13428-012-0250-9 First citation in articleCrossref, Google Scholar

  • Williams, K. D. & Nida, S. A. (2011). Ostracism: Consequences and coping. Current Directions in Psychological Science, 20, 71–75. doi: 10.1177/0963721411402480 First citation in articleCrossref, Google Scholar

  • Wolf, W., Levordashka, A., Ruff, J. R., Kraaijeveld, S., Lueckmann, J. M. & Williams, K. D. (2015). Ostracism online: A social media ostracism paradigm. Behavior Research Methods, 47, 361–373. doi: 10.3758/s13428-014-0475-x First citation in articleCrossref, Google Scholar