Aptitude Requirements for Human Operators in Human–Automation Interaction
A Meta-Analytic Review
Abstract
Abstract. The purpose of this work was to identify individual differences that affect aptitude requirements for jobs involving autonomous systems and human–automation interaction (HAI). This was addressed in two stages. First, we conducted a literature review of task demands and operator states relevant to HAI. On the basis of this review, we formed a model for understanding performance as a composite of operator states, operator behaviors, and distal outcomes. Second, we conducted a meta-analysis of correlations between individual differences and criteria reflecting job demands of an HAI environment. Results suggest cognitive skills such as working memory are important to performance in an HAI context. Inconsistent findings for personality across studies underscore the need for more research. Measurement challenges and research gaps are identified.
References
2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection & Assessment, 9(1), 9–30. https://doi.org/10.1111/1468-2389.00160
(2008). Predictive validity of the Air Force Officer Qualifying Test for USAF air battle manager training performance, AFRL-RH-WP-TR-2009-0007, Air Force Research Laboratory, Human Effectiveness Directorate, Crew Systems Interface Division, Supervisory Control Interfaces Branch.
(2003).
(Pilot selection methods . In P. S. TsangM. A. VidulichEds., Human factors in transportation. Principles and practice of aviation psychology (pp. 357–396). Erlbaum.2016). Training affects variability in training performance both within and across jobs. International Journal of Selection and Assessment, 24(1), 71–76. https://doi.org/10.1111/ijsa.12131
(2017).
(Selection methods for operators of remotely piloted aircraft systems . In N. J. CookeL. RoweW. BennettD. Q. JoralmonEds., Remotely piloted aircraft: A human systems integration approach (pp. 137–162). Wiley. https://doi.org/10.1002/9781118965900.ch62015). Vigilance impossible: Diligence, distraction, and daydreaming all lead to failures in a practical monitoring task. Consciousness and Cognition, 35, 33–41. https://doi.org/10.1016/j.concog.2015.04.019
(2000). Examination of relationships among trait-like individual differences, state-like individual differences, and learning performance. Journal of Applied Psychology, 85(6), 835–847. https://doi.org/10.1037/0021-9010.85.6.835
(2012). Supervisory control of multiple robots: Effects of imperfect automation and individual differences. Human Factors, 54, 157–174. https://doi.org/10.1177/0018720811435843
(2010). Intelligence, working memory, and multitasking performance. Intelligence, 38, 543–551. https://doi.org/10.1016/j.intell.2010.08.002
(1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI) manual. Psychological Assessment Resources. https://doi.org/10.1037/t07553-000
(2012). An evaluation of the consequences of using short measures of the Big Five personality traits. Journal of Personality and Social Psychology, 102(4), 874–888. https://doi.org/10.1037/a0027403
(2013). Boredom and distraction in multiple unmanned vehicle supervisory control. Interacting with Computers, 25(1), 34–47. https://doi.org/10.1093/iwc/iws011
(2019). Vigilance tasks: Unpleasant, mentally demanding, and stressful even when time flies. Human Factors, 61, 225–242. https://doi.org/10.1177/0018720818796015
(1990). Multidimensional assessment of coping: A critical review. Journal of Personality and Social Psychology, 58, 844–854. https://doi.org/10.1037/0022-3514.58.5.844
(2015). Autonomous horizons: System autonomy in the Air Force – a path to the future Technical report. US Air Force Chief Scientist.
(1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. 10.1518/001872095779064555
(2017). Unmanned systems integrated roadmap, 2017–2042 (Technical report). Department of Defense. https://doi.org/10.21236/ada522247
(1986). Boredom proneness: The development and correlates of a new scale. Journal of Personality Assessment, 50, 4–17. https://doi.org/10.1207/s15327752jpa5001_2
(2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37, 504–528. https://doi.org/10.1016/S0092-6566(03)00046-1
(2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537. https://doi.org/10.1038/nature01647
(2011). Use of RoboFlag synthetic task environment to investigate workload and stress responses in UAV operation. Behavior Research Methods, 43, 771–780.
(2011). A meta-analysis of factors affecting trust in human–robot interaction. Human Factors, 53, 517–527. https://doi.org/10.1177/0018720811417254
(2013). Evaluation of tests of processing speed, spatial ability, and working memory for use in military occupational classification. NPRST in Work Report. Navy Personnel, Research, Studies and Technology. https://apps.dtic.mil/sti/citations/ADA589951
(2018). Does state boredom cause failures of attention? Examining the relations between trait boredom, state boredom, and sustained attention. Experimental Brain Research, 236, 2483–2492. https://doi.org/10.1007/s00221-016-4749-7
(2004). Methods of meta-analysis. Sage.
(2000). Personality and job performance: The Big Five revisited. Journal of Applied Psychology, 85, 869–879. https://doi.org/10.1037/0021-9010.85.6.869
(1997).
(The combined effect of level of automation and adaptive automation on human performance with complex, dynamic control systems . In HFESEd., Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (pp. 205–209). Human Factors and Ergonomics Society. https://doi.org/10.1177/1071181397041001471990).
(Motivation theory and industrial and organizational psychology . In M. D. DunnetteL. HoughEds., Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 75–170). Consulting Psychologists Press.2018). Automation trust and attention allocation in multitasking workspace. Applied Ergonomics, 70, 194–201. https://doi.org/10.1016/j.apergo.2018.03.008
(2005). Working memory, fluid intelligence, and attention are predictors of multitasking performance, but polychronicity and extraversion are not. Human Performance, 18(3), 243–266. https://doi.org/10.1207/s15327043hup1803_3
(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50.30392
(2017). The Impact of automation and stress on human performance in UAV operation (Unpublished doctoral dissertation). University of Central Florida. https://stars.library.ucf.edu/etd/5710
(2016). A trust-based analysis of an Air Force collision avoidance system: Test pilots. Ergonomics in Design, 24, 9–12. https://doi.org/10.1177/1064804615611274
(2019).
(Trust and human-machine teaming: A qualitative study . In W. LawlessR. MittuD. SofgeL. MoskowitzS. RussellEds., Artificial intelligence for the Internet of everything (pp. 101–116). Academic Press. https://doi.org/10.1016/b978-0-12-817636-8.00006-51998). Task-induced stress and individual differences in coping. Proceedings of the Human Factors and Ergonomics Society, 42, 821–825. https://doi.org/10.1177/154193129804201111
(2014). Predicting battlefield vigilance: A multivariate approach to assessment of attentional resources. Ergonomics, 57, 856–875. https://doi.org/10.1080/00140139.2014
(2015). Measuring individual differences in the perfect automation schema. Human Factors, 57, 740–753. https://doi.org/10.1177/0018720815581247
(1996). Monitoring an automated system for a single failure: Vigilance and task complexity effects. Human Factors, 38(2), 311–322. https://doi.org/10.1177/001872089606380211
(2014). Executive functioning protects against stress in UAV Simulation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 994–998. https://doi.org/10.1177/1541931214581208
(1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–252. https://doi.org/10.1518/001872097778543886
(2009). Neuroticism negatively affects multitasking performance through state anxiety (NPRST-TN-09-3). Technical Report. Navy Personnel Research, Studies, and Technology. https://doi.org/10.1037/e640702009-001
(2011). Selection for vigilance assignments: A review and proposed new direction. Theoretical Issues in Ergonomics Science, 12, 273–296. https://doi.org/10.1080/14639221003622620
(2017). UAV Operator mental workload – a neurophysiological comparison of mental workload and vigilance. AIAA Modeling and Simulation Technologies Conference. https://doi.org/10.2514/6.2017-3670
(2004). A meta-analytic study of GMA validity for different occupations in the European community. Journal of Applied Psychology, 88, 1068–1081. https://doi.org/10.1037/0021-9010.88.6.1068
(2019). Automation trust increases under high-workload multitasking scenarios involving risk. Cognition, Technology & Work, 22, 399–407. https://doi.org/10.1007/s10111-019-00580-5
(2016). A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems. Human Factors, 58(3), 377–400. https://doi.org/10.1177/0018720816634228
(1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 298–307. https://doi.org/10.1037/0033-2909.124.2.262
(2014). Personality and cognitive ability as predictors of effective performance at work. Annual Review of Organizational Psychology and Organizational Behavior, 1, 45–65. https://doi.org/10.1146/annurev-orgpsych-031413-091255
(2006). Coordinated multi-agent teams and sliding autonomy for large-scale assembly. Proceedings of the IEEE, 94(7), 1425–1444. https://doi.org/10.1109/JPROC.2006.876966
(2016).
(Research considerations and tools for evaluating human automation interaction with future unmanned systems . In W. F. LawlessR. MittuD. SofgeS. RussellEds., Autonomy and artificial intelligence: A threat or savior? (pp. 157–178). Springer. https://doi.org/10.1007/978-3-319-59719-5_71999). Testing and assessment: An employer’s guide to good practices. Employment and Testing Administration.
. (2019). A century of research on conscientiousness at work. Proceedings of the National Academy of Sciences, 116(46), 23004–23010. https://doi.org/10.1073/pnas.1908430116
(2019). Vigilance and automation dependence in operation of multiple unmanned aerial systems (UAS): A simulation study. Human Factors, 61(3), 488–505.
(2018). An integrative model of autonomous agent teammate-likeness. Theoretical Issues in Ergonomics Science, 19(3), 353–374. https://doi.org/10.1080/1463922X.2016.1260181
(