Skip to main content
Review Article

Aptitude Requirements for Human Operators in Human–Automation Interaction

A Meta-Analytic Review

Published Online:https://doi.org/10.1027/2192-0923/a000210

Abstract. The purpose of this work was to identify individual differences that affect aptitude requirements for jobs involving autonomous systems and human–automation interaction (HAI). This was addressed in two stages. First, we conducted a literature review of task demands and operator states relevant to HAI. On the basis of this review, we formed a model for understanding performance as a composite of operator states, operator behaviors, and distal outcomes. Second, we conducted a meta-analysis of correlations between individual differences and criteria reflecting job demands of an HAI environment. Results suggest cognitive skills such as working memory are important to performance in an HAI context. Inconsistent findings for personality across studies underscore the need for more research. Measurement challenges and research gaps are identified.

References

  • Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection & Assessment, 9(1), 9–30. https://doi.org/10.1111/1468-2389.00160 First citation in articleCrossrefGoogle Scholar

  • Carretta, T. R. (2008). Predictive validity of the Air Force Officer Qualifying Test for USAF air battle manager training performance, AFRL-RH-WP-TR-2009-0007, Air Force Research Laboratory, Human Effectiveness Directorate, Crew Systems Interface Division, Supervisory Control Interfaces Branch. First citation in articleGoogle Scholar

  • Carretta, T. R., & Ree, M. J. (2003). Pilot selection methods. In P. S. TsangM. A. VidulichEds., Human factors in transportation. Principles and practice of aviation psychology (pp. 357–396). Erlbaum. First citation in articleGoogle Scholar

  • Carretta, T. R., Ree, M. J., & Teachout, M. S. (2016). Training affects variability in training performance both within and across jobs. International Journal of Selection and Assessment, 24(1), 71–76. https://doi.org/10.1111/ijsa.12131 First citation in articleCrossrefGoogle Scholar

  • Carretta, T. R., Rose, M. R., & Bruskiewicz, K. T. (2017). Selection methods for operators of remotely piloted aircraft systems. In N. J. CookeL. RoweW. BennettD. Q. JoralmonEds., Remotely piloted aircraft: A human systems integration approach (pp. 137–162). Wiley. https://doi.org/10.1002/9781118965900.ch6 First citation in articleGoogle Scholar

  • Casner, S. M., & Schooler, J. W. (2015). Vigilance impossible: Diligence, distraction, and daydreaming all lead to failures in a practical monitoring task. Consciousness and Cognition, 35, 33–41. https://doi.org/10.1016/j.concog.2015.04.019 First citation in articleCrossrefGoogle Scholar

  • Chen, G., Whiteman, J.-A., Gully, S. M., & Kilcullen, R. N. (2000). Examination of relationships among trait-like individual differences, state-like individual differences, and learning performance. Journal of Applied Psychology, 85(6), 835–847. https://doi.org/10.1037/0021-9010.85.6.835 First citation in articleCrossrefGoogle Scholar

  • Chen, J. Y. C., & Barnes, M. J. (2012). Supervisory control of multiple robots: Effects of imperfect automation and individual differences. Human Factors, 54, 157–174. https://doi.org/10.1177/0018720811435843 First citation in articleCrossrefGoogle Scholar

  • Colom, R., Martínez-Molina, A., Shih, P., & Santacreu, J. (2010). Intelligence, working memory, and multitasking performance. Intelligence, 38, 543–551. https://doi.org/10.1016/j.intell.2010.08.002 First citation in articleCrossrefGoogle Scholar

  • Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) manual. Psychological Assessment Resources. https://doi.org/10.1037/t07553-000 First citation in articleGoogle Scholar

  • Credé, M., Harms, P., Niehorster, S., & Gaye-Valentine, A. (2012). An evaluation of the consequences of using short measures of the Big Five personality traits. Journal of Personality and Social Psychology, 102(4), 874–888. https://doi.org/10.1037/a0027403 First citation in articleCrossrefGoogle Scholar

  • Cummings, M. L., Mastracchio, C., Thornburg, K. M., & Mkrtchyan, A. (2013). Boredom and distraction in multiple unmanned vehicle supervisory control. Interacting with Computers, 25(1), 34–47. https://doi.org/10.1093/iwc/iws011 First citation in articleCrossrefGoogle Scholar

  • Dillard, M. B., Warm, J. S., Funke, G. J., Nelson, W. T., Finomore, V. S., McClernon, C. K., Eggemeier, F. T., Tripp, L. D., & Funke, M. E. (2019). Vigilance tasks: Unpleasant, mentally demanding, and stressful even when time flies. Human Factors, 61, 225–242. https://doi.org/10.1177/0018720818796015 First citation in articleCrossrefGoogle Scholar

  • Endler, N., & Parker, J. (1990). Multidimensional assessment of coping: A critical review. Journal of Personality and Social Psychology, 58, 844–854. https://doi.org/10.1037/0022-3514.58.5.844 First citation in articleCrossrefGoogle Scholar

  • Endsley, M. R. (2015). Autonomous horizons: System autonomy in the Air Force – a path to the future Technical report. US Air Force Chief Scientist. First citation in articleGoogle Scholar

  • Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. 10.1518/001872095779064555 First citation in articleCrossrefGoogle Scholar

  • Fahey, K. M., & Miller, M. J. (2017). Unmanned systems integrated roadmap, 2017–2042 (Technical report). Department of Defense. https://doi.org/10.21236/ada522247 First citation in articleGoogle Scholar

  • Farmer, R., & Sundberg, N. D. (1986). Boredom proneness: The development and correlates of a new scale. Journal of Personality Assessment, 50, 4–17. https://doi.org/10.1207/s15327752jpa5001_2 First citation in articleCrossrefGoogle Scholar

  • Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37, 504–528. https://doi.org/10.1016/S0092-6566(03)00046-1 First citation in articleCrossrefGoogle Scholar

  • Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537. https://doi.org/10.1038/nature01647 First citation in articleCrossrefGoogle Scholar

  • Guznov, S., Matthews, G., Funke, G., & Dukes, A. (2011). Use of RoboFlag synthetic task environment to investigate workload and stress responses in UAV operation. Behavior Research Methods, 43, 771–780. First citation in articleCrossrefGoogle Scholar

  • Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E., & Parasuraman, R. (2011). A meta-analysis of factors affecting trust in human–robot interaction. Human Factors, 53, 517–527. https://doi.org/10.1177/0018720811417254 First citation in articleCrossrefGoogle Scholar

  • Held, J. D., & Carretta, T. R. (2013). Evaluation of tests of processing speed, spatial ability, and working memory for use in military occupational classification. NPRST in Work Report. Navy Personnel, Research, Studies and Technology. https://apps.dtic.mil/sti/citations/ADA589951 First citation in articleGoogle Scholar

  • Hunter, A., & Eastwood, J. D. (2018). Does state boredom cause failures of attention? Examining the relations between trait boredom, state boredom, and sustained attention. Experimental Brain Research, 236, 2483–2492. https://doi.org/10.1007/s00221-016-4749-7 First citation in articleCrossrefGoogle Scholar

  • Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis. Sage. First citation in articleCrossrefGoogle Scholar

  • Hurtz, G. M., & Donovan, J. J. (2000). Personality and job performance: The Big Five revisited. Journal of Applied Psychology, 85, 869–879. https://doi.org/10.1037/0021-9010.85.6.869 First citation in articleCrossrefGoogle Scholar

  • Kaber, D. B., & Endsley, M. R. (1997). The combined effect of level of automation and adaptive automation on human performance with complex, dynamic control systems. In HFESEd., Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (pp. 205–209). Human Factors and Ergonomics Society. https://doi.org/10.1177/107118139704100147 First citation in articleGoogle Scholar

  • Kanfer, R. (1990). Motivation theory and industrial and organizational psychology. In M. D. DunnetteL. HoughEds., Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 75–170). Consulting Psychologists Press. First citation in articleGoogle Scholar

  • Karpinsky, N. D., Chancey, E. T., Palmer, D. B., & Yamani, Y. (2018). Automation trust and attention allocation in multitasking workspace. Applied Ergonomics, 70, 194–201. https://doi.org/10.1016/j.apergo.2018.03.008 First citation in articleCrossrefGoogle Scholar

  • König, C. J., Bühner, M., & Mürling, G. (2005). Working memory, fluid intelligence, and attention are predictors of multitasking performance, but polychronicity and extraversion are not. Human Performance, 18(3), 243–266. https://doi.org/10.1207/s15327043hup1803_3 First citation in articleCrossrefGoogle Scholar

  • Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50.30392 First citation in articleCrossrefGoogle Scholar

  • Lin, J. (2017). The Impact of automation and stress on human performance in UAV operation (Unpublished doctoral dissertation). University of Central Florida. https://stars.library.ucf.edu/etd/5710 First citation in articleGoogle Scholar

  • Lyons, J. B., Ho, N. T., Koltai, K., Masequesmay, G., Skoog, M., Cacanidin, A., & Johnson, W. W. (2016). A trust-based analysis of an Air Force collision avoidance system: Test pilots. Ergonomics in Design, 24, 9–12. https://doi.org/10.1177/1064804615611274 First citation in articleCrossrefGoogle Scholar

  • Lyons, J. B., Wynne, K. T., Mahoney, S., & Roebke, M. A. (2019). Trust and human-machine teaming: A qualitative study. In W. LawlessR. MittuD. SofgeL. MoskowitzS. RussellEds., Artificial intelligence for the Internet of everything (pp. 101–116). Academic Press. https://doi.org/10.1016/b978-0-12-817636-8.00006-5 First citation in articleGoogle Scholar

  • Matthews, G., & Campbell, S. E. (1998). Task-induced stress and individual differences in coping. Proceedings of the Human Factors and Ergonomics Society, 42, 821–825. https://doi.org/10.1177/154193129804201111 First citation in articleCrossrefGoogle Scholar

  • Matthews, G., Warm, J. S., Shaw, T. H., & Finomore, V. S. (2014). Predicting battlefield vigilance: A multivariate approach to assessment of attentional resources. Ergonomics, 57, 856–875. https://doi.org/10.1080/00140139.2014 First citation in articleCrossrefGoogle Scholar

  • Merritt, S. M., Unnerstall, J. L., Lee, D., & Huber, K. (2015). Measuring individual differences in the perfect automation schema. Human Factors, 57, 740–753. https://doi.org/10.1177/0018720815581247 First citation in articleCrossrefGoogle Scholar

  • Molloy, R., & Parasuraman, R. (1996). Monitoring an automated system for a single failure: Vigilance and task complexity effects. Human Factors, 38(2), 311–322. https://doi.org/10.1177/001872089606380211 First citation in articleCrossrefGoogle Scholar

  • Panganiban, A. R., & Matthews, G. (2014). Executive functioning protects against stress in UAV Simulation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 994–998. https://doi.org/10.1177/1541931214581208 First citation in articleCrossrefGoogle Scholar

  • Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–252. https://doi.org/10.1518/001872097778543886 First citation in articleCrossrefGoogle Scholar

  • Poposki, E. M., Oswald, F. L., & Chen, H. T. (2009). Neuroticism negatively affects multitasking performance through state anxiety (NPRST-TN-09-3). Technical Report. Navy Personnel Research, Studies, and Technology. https://doi.org/10.1037/e640702009-001 First citation in articleGoogle Scholar

  • Reinerman-Jones, L. E., Matthews, G., Langheim, L. K., & Warm, J. S. (2011). Selection for vigilance assignments: A review and proposed new direction. Theoretical Issues in Ergonomics Science, 12, 273–296. https://doi.org/10.1080/14639221003622620 First citation in articleCrossrefGoogle Scholar

  • Richards, D., Izzetoglu, K., & Shelton-Rayner, G. (2017). UAV Operator mental workload – a neurophysiological comparison of mental workload and vigilance. AIAA Modeling and Simulation Technologies Conference. https://doi.org/10.2514/6.2017-3670 First citation in articleGoogle Scholar

  • Salgado, J., Anderson, N. R., Moscoso, S., Bertua, C., de Fruyt, F., & Rolland, J. P. (2004). A meta-analytic study of GMA validity for different occupations in the European community. Journal of Applied Psychology, 88, 1068–1081. https://doi.org/10.1037/0021-9010.88.6.1068 First citation in articleCrossrefGoogle Scholar

  • Sato, T., Yamani, Y., Liechty, M., & Chancey, M. (2019). Automation trust increases under high-workload multitasking scenarios involving risk. Cognition, Technology & Work, 22, 399–407. https://doi.org/10.1007/s10111-019-00580-5 First citation in articleCrossrefGoogle Scholar

  • Schaefer, K. E., Chen, J. Y. C., Szalma, J. L., & Hancock, P. A. (2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400. https://doi.org/10.1177/0018720816634228 First citation in articleCrossrefGoogle Scholar

  • Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 298–307. https://doi.org/10.1037/0033-2909.124.2.262 First citation in articleCrossrefGoogle Scholar

  • Schmitt, N. (2014). Personality and cognitive ability as predictors of effective performance at work. Annual Review of Organizational Psychology and Organizational Behavior, 1, 45–65. https://doi.org/10.1146/annurev-orgpsych-031413-091255 First citation in articleCrossrefGoogle Scholar

  • Sellner, B., Heger, F. W., Hiatt, L. M., Simmons, R., & Singh, S. (2006). Coordinated multi-agent teams and sliding autonomy for large-scale assembly. Proceedings of the IEEE, 94(7), 1425–1444. https://doi.org/10.1109/JPROC.2006.876966 First citation in articleCrossrefGoogle Scholar

  • Sibley, C., Coyne, J. T., & Sherwood, S. (2016). Research considerations and tools for evaluating human automation interaction with future unmanned systems. In W. F. LawlessR. MittuD. SofgeS. RussellEds., Autonomy and artificial intelligence: A threat or savior? (pp. 157–178). Springer. https://doi.org/10.1007/978-3-319-59719-5_7 First citation in articleGoogle Scholar

  • US Department of Labor. (1999). Testing and assessment: An employer’s guide to good practices. Employment and Testing Administration. First citation in articleGoogle Scholar

  • Wilmot, M. P., & Ones, D. S. (2019). A century of research on conscientiousness at work. Proceedings of the National Academy of Sciences, 116(46), 23004–23010. https://doi.org/10.1073/pnas.1908430116 First citation in articleCrossrefGoogle Scholar

  • Wohleber, R. W., Matthews, G., Lin, J., Szalma, J. L., Calhoun, G. L., Funke, G. J., Chiu, P., & Ruff, H. A. (2019). Vigilance and automation dependence in operation of multiple unmanned aerial systems (UAS): A simulation study. Human Factors, 61(3), 488–505. First citation in articleCrossrefGoogle Scholar

  • Wynne, K. T., & Lyons, J. B. (2018). An integrative model of autonomous agent teammate-likeness. Theoretical Issues in Ergonomics Science, 19(3), 353–374. https://doi.org/10.1080/1463922X.2016.1260181 First citation in articleCrossrefGoogle Scholar