Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

On the Psychology of the Psychology Subject Pool

An Exploratory Test of the Good Student Effect

Published Online:July 09, 2020https://doi.org/10.1027/1614-0001/a000327

Abstract

Abstract. Many psychology researches are performed through “psychology subject pools” which give participants considerable flexibility when they participate. This “participant degree-of-freedom” has led to concern that the characteristics of subject pool participants may change with time, with the most engaged students signing up at the start of the semester and the least engaged students leaving it all to the end. In this paper, we performed an exploratory analysis to look for evidence of this “good student effect.” Consistent with previous work, we find support for the good student effect with earlier participants scoring higher on the Big-Five subscales of Achievement-Striving and Cooperation, as well as Grit and Empathic-Concern. In addition, we found a non-linear effect of time-of-semester on Sensation-Seeking, with this measure peaking in the middle of the semester as well as the end. However, the vast majority of the measures we tested, including measures of personality, cognition, decision-making, and social interaction, did not correlate with time-of-semester or time-of-day at all. Thus, we conclude that, while some studies directly related to measures of Grit and Sensation-Seeking would do well to recruit throughout the semester, in most cases any bias introduced by the good student effect is likely to be small.

References

  • Adair, J. (1973). The human subject: The social psychology of the psychological experiment. Boston, MA: Little Brown. First citation in articleGoogle Scholar

  • Arnett, J. (1994). Sensation seeking: A new conceptualization and a new scale. Personality and Individual Differences, 16, 289–296. https://doi.org/10.1016/0191-8869(94)90165-1 First citation in articleCrossref, Google Scholar

  • Aviv, A. L., Zelenski, J. M., Rallo, L., & Larsen, R. J. (2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33, 487–496. https://doi.org/10.1016/S0191-8869(01)00199-4 First citation in articleCrossref, Google Scholar

  • Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131. https://doi.org/10.1037/0022-3514.42.1.116 First citation in articleCrossref, Google Scholar

  • Cassidy, R. N., & Kangas, B. D. (2014). Impulsive students participate later: Delay discounting in a research subject pool. Experimental Analysis of Human Behavior Bulletin, 30, 1–5. First citation in articleGoogle Scholar

  • Cooper, H., Baumgardner, A. H., & Strathman, A. (1991). Do students with different characteristics take part in psychology experiments at different times of the semester? Journal of Personality, 59, 109–127. https://doi.org/10.1111/j.1467-6494.1991.tb00770.x First citation in articleCrossref, Google Scholar

  • Credé, M., Tynan, M. C., & Harms, P. D. (2017). Much ado about grit: A meta-analytic synthesis of the grit literature. Journal of Personality and Social Psychology, 113, 492. https://doi.org/10.1037/pspp0000102 First citation in articleCrossref, Google Scholar

  • Davis, M. H. (1983). Measuring individual differences in empathy: Evidence for a multi-dimensional approach. Journal of Personality and Social Psychology, 44, 113–126. https://doi.org/10.1037/0022-3514.44.1.113 First citation in articleCrossref, Google Scholar

  • Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101. https://doi.org/10.1037/0022-3514.92.6.1087 First citation in articleCrossref, Google Scholar

  • Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., … Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. https://doi.org/10.1016/j.jesp.2015.10.012 First citation in articleCrossref, Google Scholar

  • Evans, R., & Donnerstein, E. (1974). Some implications for psychological research of early versus late term participation by college subjects. Journal of Research in Personality, 8, 102–109. https://doi.org/10.1016/0092-6566(74)90050-6 First citation in articleCrossref, Google Scholar

  • Everett, J. A. (2013). The 12 item Social and Economic Conservatism Scale (SECS). PLoS One, 8, e82131. https://doi.org/10.1371/journal.pone.0082131 First citation in articleCrossref, Google Scholar

  • Figueredo, A. J., Vásquez, G., Brumbach, B. H., Schneider, S. M., Sefcek, J. A., Tal, I. R., … Jacobs, W. J. (2006). Consilience and life history theory: From genes to brain to reproductive strategy. Developmental Review, 26, 243–275. https://doi.org/10.1016/j.dr.2006.02.002 First citation in articleCrossref, Google Scholar

  • Findley, M., & Cooper, H. (1981). Introductory social psychology textbook citations: A comparison in five research areas. Personality and Social Psychology Bulletin, 7, 173–176. https://doi.org/10.1177/014616728171025 First citation in articleCrossref, Google Scholar

  • Grable, J., & Lytton, R. H. (1999). Financial risk tolerance revisited: The development of a risk assessment instrument. Financial Services Review, 8, 163–181. https://doi.org/10.1016/S1057-0810(99)00041-4 First citation in articleCrossref, Google Scholar

  • Greene, W. A., & Sutor, L. T. (1971). Stimulus control of skin resistance responses on an escape-avoidance schedule. Journal of the experimental analysis of behavior, 16, 269–274. https://doi.org/10.1901/jeab.1971.16-269 First citation in articleCrossref, Google Scholar

  • Groppe, D. (2020). Bonferroni-Holm correction for multiple comparisons. [MATLAB Central File Exchange]. Retrieved from https://www.mathworks.com/matlabcentral/fileexchange/28303-bonferroni-holm-correction-for-multiple-comparisons First citation in articleGoogle Scholar

  • Harris, P. B., & Houston, J. M. (2010). A reliability analysis of the Revised Competitiveness Index. Psychological Reports, 106, 870–874. https://doi.org/10.2466/pr0.106.3.870-874 First citation in articleCrossref, Google Scholar

  • Hegarty, M., Montello, D. R., Richardson, A. E., Ishikawa, T., & Lovelace, K. (2006). Spatial abilities at different scales: Individual differences in aptitude-test performance and spatial-layout learning. Intelligence, 34, 151–176. https://doi.org/10.1016/j.intell.2005.09.005 First citation in articleCrossref, Google Scholar

  • Higbee, K. L., Millard, R. J., & Folkman, J. R. (1982). Social psychology research during the 1970 s: Predominance of experimentation and college students. Personality and Social Psychology Bulletin, 8, 180–183. https://doi.org/10.1177/014616728281028 First citation in articleCrossref, Google Scholar

  • Holden, R. R., & Reddon, J. R. (1987). Temporal personality variations among participants from a university subject pool. Psychological Reports, 60, 1247–1254. https://doi.org/10.2466/pr0.1987.60.3c.1247 First citation in articleCrossref, Google Scholar

  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70. Retrieved from https://www.jstor.org/stable/4615733 First citation in articleGoogle Scholar

  • Jackson, J. J. M., Procidano, M. M. E., & Cohen, C. J. C. (1989). Subject pool sign-up procedures: A threat to external validity. Social Behavior and Personality: An International Journal, 17, 29–42. https://doi.org/10.2224/sbp.1989.17.1.29 First citation in articleCrossref, Google Scholar

  • Johnson, J. A. (2014). Measuring thirty facets of the Five Factor Model with a 120-item public domain inventory: Development of the IPIP-NEO-120. Journal of Research in Personality, 51, 78–89. https://doi.org/10.1016/j.jrp.2014.05.003 First citation in articleCrossref, Google Scholar

  • Kangas, B. D., & Hackenberg, T. D. (2009). On reinforcing human behavior in the laboratory: A brief review and some recommendations. Experimental Analysis of Human Behavior Bulletin, 27, 21–26. First citation in articleGoogle Scholar

  • Kashdan, T. B., Gallagher, M. W., Silvia, P. J., Winterstein, B. P., Breen, W. E., Terhar, D., & Steger, M. F. (2009). The Curiosity and Exploration Inventory-II: Development, factor structure, and psychometrics. Journal of Research in Personality, 43, 987–998. https://doi.org/10.1016/j.jrp.2009.04.011 First citation in articleCrossref, Google Scholar

  • Kirby, K. N., Petry, N. M., & Bickel, W. K. (1999). Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls. Journal of Experimental Psychology: General, 128, 78–87. https://doi.org/10.1037/0096-3445.128.1.78 First citation in articleCrossref, Google Scholar

  • Landrum, R. E., & Chastain, G. (1995). Experiment spot-checks: A method for assessing the educational value of undergraduate participation in research. IRB, 17, 4–6. https://doi.org/10.2307/3564152 First citation in articleCrossref, Google Scholar

  • Landrum, R. E., & Chastain, G. D. (1999). Subject pool policies in undergraduate-only departments: Results from a nationwide survey. In G. D. ChastainR. E. LandrumEds., Protecting human subjects: Departmental subject pools and institutional review boards (pp. 25–42). Washington, DC: American Psychological Association. https://doi.org/10.1037/10322-001 First citation in articleGoogle Scholar

  • Leary, M. R. (1983). A brief version of the fear of negative evaluation scale. Personality and Social Psychology Bulletin, 9, 371–375. https://doi.org/10.1177/0146167283093007 First citation in articleCrossref, Google Scholar

  • Litman, J. A. (2008). Interest and deprivation factors of epistemic curiosity. Personality and Individual Differences, 44, 1585–1595. https://doi.org/10.1016/j.paid.2008.01.014 First citation in articleCrossref, Google Scholar

  • McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. American Psychologist, 52, 509–516. https://doi.org/10.1037/0003-066X.52.5.509 First citation in articleCrossref, Google Scholar

  • Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., … Reed, G. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues. American Psychologist, 56, 128–165. https://doi.org/10.1037/0003-066X.56.2.128 First citation in articleCrossref, Google Scholar

  • Miller, A. (1981). A survey of introductory psychology subject pool practices among leading universities. Teaching of Psychology, 8, 211–214. https://doi.org/10.1207/s15328023top0804_4 First citation in articleCrossref, Google Scholar

  • Mulry, R., & Dunbar, D. (n.d.). Homogeneity of a subject sample: A study of subject motivation (Unpublished manuscript). Bloomington, IN: University of Indiana. First citation in articleGoogle Scholar

  • Neuberg, S. L., & Newsom, J. T. (1993). Personal Need for Structure: Individual differences in the desire for simple structure. Journal of Personality and Social Psychology, 65, 113–131. https://doi.org/10.1037/0022-3514.65.1.113 First citation in articleCrossref, Google Scholar

  • Patton, J. H., Stanford, M. S., & Barratt, E. S. (1995). Factor structure of the Barrett Impulsiveness Scale. Journal of Clinical Psychology, 51, 768–774. https://doi.org/10.1002/1097-4679(199511)51:6<768::AID-JCLP2270510607>3.0.CO;2-1 First citation in articleCrossref, Google Scholar

  • Richert, A. J., & Ward, E. F. (1976). Experimental performance and self-evaluation of subjects sampled early, middle, and late in an academic term: Sex and task. Psychological Reports, 39, 135–142. https://doi.org/10.2466/pr0.1976.39.1.135 First citation in articleCrossref, Google Scholar

  • Richter, D., Wilson, S., Milner, M., & Senter, R. (1981). Some differences among students volunteering as research subjects. Bulletin of the Psychonomic Society, 17, 261–263. https://doi.org/10.3758/BF03333734 First citation in articleCrossref, Google Scholar

  • Roman, R. J., Moskowitz, G. B., Stein, M. I., & Eisenberg, R. F. (1995). Individual differences in experiment participation: Structure, autonomy, and the time of the semester. Journal of Personality, 63, 113–138. https://doi.org/10.1111/j.1467-6494.1995.tb00804.x First citation in articleCrossref, Google Scholar

  • Rosenthal, R., & Rosnow, R. L. (1975). Primer of methods for the behavioral sciences. New York, NY: John Wiley & Sons. First citation in articleGoogle Scholar

  • Rotter, J. B. (1966). Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs: General and Applied, 80, 1. https://doi.org/10.1037/h0092976 First citation in articleCrossref, Google Scholar

  • Sadeghiyeh, H. (2019). Social influence, temporal discounting and active-passive gap in explore-exploit dilemma (Doctoral dissertation). Tucson, AZ: University of Arizona. First citation in articleGoogle Scholar

  • Sadeghiyeh, H., Wang, S., & Wilson, R. C. (2018, November 12). Lessons from a “failed” replication: The importance of taking action in exploration. https://doi.org/10.31234/osf.io/ue7dx First citation in articleGoogle Scholar

  • Sadeghiyeh, H., Wang, S., Alberhasky, M. R., Kyllo, H. M., Shenhav, A., & Wilson, R. C. (2020). Temporal discounting correlates with directed exploration but not with random exploration. Scientific Reports, 10, 1–10. https://doi.org/10.1038/s41598-020-60576-4 First citation in articleCrossref, Google Scholar

  • Scheier, M. F., Carver, C. S., & Bridges, M. W. (1994). Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): A reevaluation of the life orientation test. Journal of Personality and Social Psychology, 67, 1063–1078. https://doi.org/10.1037/0022-3514.67.6.1063 First citation in articleCrossref, Google Scholar

  • Shiota, M. N., Keltner, D., & John, O. P. (2006). Positive emotion dispositions differentially associated with Big Five personality and attachment style. Journal of Positive Psychology, 1, 61–71. https://doi.org/10.1080/17439760500510833 First citation in articleCrossref, Google Scholar

  • Sieber, J. E., & Saks, M. J. (1989). A census of subject pool characteristics and policies. American Psychologist, 44, 1053–1061. https://doi.org/10.1037/0003-066X.44.7.1053 First citation in articleCrossref, Google Scholar

  • Stanley Budner, N. Y. (1962). Intolerance of ambiguity as a personality variable. Journal of Personality, 30, 29–50. https://doi.org/10.1111/j.1467-6494.1962.tb02303.x First citation in articleCrossref, Google Scholar

  • Strube, M. J., & Ota, S. (1982). Type A coronary-prone behavior pattern: Relationship to birth order and family size. Personality and Social Psychology Bulletin, 8, 317–323. https://doi.org/10.1177/0146167282082020 First citation in articleCrossref, Google Scholar

  • Triandis, H. C., & Gelfand, M. J. (1998). Converging measurement of horizontal and vertical individualism and collectivism. Journal of Personality and Social Psychology, 74, 118. https://doi.org/10.1037/0022-3514.74.1.118 First citation in articleCrossref, Google Scholar

  • Wang, A. Y., & Jentsch, F. G. (1998). Point-of-time effects across the semester: Is there a sampling bias? Journal of Psychology: Interdisciplinary and Applied, 132, 211–219. https://doi.org/10.1080/00223989809599160 First citation in articleCrossref, Google Scholar

  • Zelenski, J. M., Rusting, C. L., & Larsen, R. J. (2003). Consistency in the time of experiment participation and personality correlates: A methodological note. Personality and Individual Differences, 34, 547–558. https://doi.org/10.1016/S0191-8869(01)00218-5 First citation in articleCrossref, Google Scholar