The Impact of Technology on Psychological Testing in Practice and Policy

Without a Closer Integration of Technology, Testing As We Know It May Disappear – But It May Do so Equally Because of the Integration With Technology

We will start by making a clear and probably provocative statement: (traditional) psychological testing and technology have always had an uneasy alliance. At the 9th International Test Commission (ITC) conference in 2014, Dave Foster, a then Council member of the ITC, gave a keynote address stating that “technology was never more than a page-turner for testing” (Foster, 2014). He supports this claim by showing that the advantages of technology were never fully leveraged by test developers (authors) or companies (providers). In fact, these stakeholders rather seized technology as a convenient way to get classical tests towards new and extended audiences. “Technology” therefore replaced paper-and-pencil for the administration and scoring of tests, but for little else. However, during the past decade, the world has certainly changed and continues to change at an accelerated pace – and technology is the great game-changer in virtually every area of psychology in profound ways (World Economic Forum, 2019) and certainly more profound than just being a mere page-turner.

In developing predictions about the future of the testing industry and how it might be affected through technology, we draw inspiration from what has already happened in other domains – for example, the music industry. The music industry has changed over the past 30 years in substantial ways: it was not so much digitalization of music (which of course was itself driven by technological advancement), as it was the change in how music is accessed and played (which also happened through the advancement of technology). In the early 1990s, most music was still copied on a carrier media (e.g., MCs, CDs), but at about the end of the 1990s, downloading had begun to dominate the industry – taking on in excess of 90% of all music usages by 2005, when it began to be slowly replaced by streaming (see Figure 1). The music industry was and continues to be disrupted by technology, and the current shift is clearly from paying for music to paying for the experience. That is, from owning an actual hard copy of the music to having access to a subscription-based model.

One can only wonder how much of this shift may also follow in the testing industry. The traditional model of psychological testing was based on a competent test user buying (and owning) the test (i.e., the test materials: item booklet, scoring sheets, manual, and so forth) from a test publisher or author and being responsible for the administration, scoring, and interpretation of the test. This traditional model is fading, while an alternative model creeps in: not “owning” the test, but owning access to the test. That is, test users have access to the test in form of a service: they can log into the test platform, invite clients to take the test, and the test platform (the “publisher”) administers and scores the test and finally generates one or several reports. In such cases, the service fee for the test user is per usage, and in some cases rightly so, because “the test” could not be directly administered by the test user – imagine tests based on item banks, automated item generation, serious games, and other emerging technologies.

Increasing integration with technology will, in fact, encourage this shift toward “testing as service.” We believe that one of the major barriers for psychologists embarking on this technological bandwagon is their limited understanding and command of technology. Granted, using technology for research is fairly easy (Langer, Schmid Mast, Meyer, Maass, & König, 2019), but developing technology requires a completely different set of skills and interests. Modern computer technology has a great advantage in this respect: it is to some extent reusable once it has been developed. Many lucrative algorithms that have been developed by others and that are usable in psychological testing are open through an “Application Programming Interface,” or API. APIs are server-side applications that can be accessed by third parties, and offer their algorithms as a service. For example, facemood recognition APIs are open and offer emotion recognition as a service. APIs for DNA genetic testing and analysis are now open as well. Impressive sensors are continuously at our fingertips in our phones, bracelets, and even in our shoes – and the APIs to access that data are reasonably easy to integrate in applications. And when all this wealth of data that is passively collected is not enough, active collection of data gets increasingly easier with the use of chatbot APIs that work incredibly well.

All these already existing technologies can be fairly easily built into testing applications – but will definitely shift usage towards “testing as a service,” the equivalent of “streaming” content in the music industry, which already takes over more and more of the market from the traditional “ownership” form of testing.

Psychological Testing Will Continue to Generate Data, but It Will Also Begin to Preferentially Use Existing Data, That Was Routinely Collected for Other Objectives

Traditional testing generates the data it needs – in fact, an argument could be built for the fact that testing is just about this: generating data. When test users administer a test, they generate much needed data about psychological and/or behavioral constructs in a target person. They then integrate that data with data from other sources, in order to describe, explain, or predict a phenomenon. But description, explanation, or (more than anything) prediction can also be based on already existing data, data that is not obtained at the moment when it is needed, but is continuously harvested from past behaviors of the target person, and only analyzed when needed. Would this still be labeled as “psychological testing”?

Going back to the above example, there’s another important and less visible shift in the music industry brought along by the usage of real-time streaming: the new king of the game is not “content,” it is “data.” Impressive business investments are nowadays made on nothing more than data on which predictive algorithms may be trained. The potential utility and hence commercial value of the data is seen as larger than the value of the actual algorithm developed from that data (Biran & Cotton, 2017). For example, a well-known company that provides shared workspaces was considered in 2018 the 6th most valued start-up in the world, pointed at a $20 billion valuation (see https://www.cnbc.com/2017/07/11/wework-raises-series-g-round-at-20-billion-valuation-report-says.html), and their wealth is not so much the co-working spaces they offer, but the wealth of data they produce for those who use those spaces: personal details, as well as professional details, preferred work hours, productivity, size (and content) of their personal networks, number of conference calls, number of coffee cups, preferences regarding heat and sound, and many more. It’s easily conceivable that such a company will branch out into psychological testing, for instance through providing HR tests, measuring performance, and finding future jobs.

While most interest for such data-driven emergent technologies comes from outside of psychology, some attention has also been given to this evolution from the inside. Research in this area is going strong (e.g., Landers, 2019), and a number of lucrative initiatives focus on gamified assessment, automatic scoring of CVs, or social-media-based profiling. We wonder how much of testing will, over the next decade, continue to rely on the traditional “generate-data-when-needed” model, and how much “testing” will be based (maybe exclusively) on already existing data about the focal person.

We Have No Clear Criteria on Which to Decide Whether Psychological Testing Is “Better” Than Technology-Driven Profiling and Prediction

From all these recent developments, we would advance this tentative conclusion: technology is, as always, the great game-changer. And it will not treat the testing industry gently, because people, companies, or communities do not care about “tests”, they care about having a problem solved. Whichever technology solves the problem better, wins the bid, and, ultimately, stays alive. Put differently, in order to survive, psychological tests need to be able to prove their utility for society. If society would ask, in the vein of Monty Python’s Life of Brian: “What did the tests (“the Romans”) ever do for us?”, would psychological testing have a positive answer to give? We are sure that psychologists ask themselves this very question, comparing psychological tests with competing technology-driven computer algorithms. Inevitably, with a zeal that also shows frustration, a number of critiques are advanced regarding the latter – among others, the fact that computerized prediction algorithms habitually lack interpretability (Liem et al., 2018), are shallow, and have a limited capacity for transfer and generalization, struggle with integration of prior knowledge, and so forth (Marcus, 2019). And while these points can sometimes also be made against classical testing, we have to acknowledge the habitual struggle of modern psychology to leave empiricism behind and opt for theoretically sound (or at least defendable) predictive models.

When psychologists do the comparing, they never forget to underscore that “orthodox” psychological testing is better also because of its adherence to strong ethical principles. Indeed, psychological testing has a tradition of ethical thought and practice, possibly by reason of its embeddedness in psychology and educational sciences, two domains of study and professional practice immersed in ethical thought and moral judgment. It is quite possible that, had psychometricians and psychological test developers been pushing the buttons, events such as the Cambridge Analytica scandal would never had happened. On the other hand, the fact that due to ethical (or whatever other) reasons, a whole professional category stays its hand and does not let “unclean thoughts” drift up, does not mean that everybody else has the same compulsion. We will, therefore, continue to read about disturbing news coming from, for example, life sciences (e.g., “DNA of every baby born in California since 1983 is stored. Who has access to it?”; https://cbsnews.com/news/california-biobank-dna-babies-who-has-access/), or IT & computer science (“We have created a model with an accuracy of 91.02% for the prediction of sexual orientation”; https://cs229.stanford.edu/proj2015/019_report.pdf).

We are unsure whether this competition is going to be played out in the territory of ethics. If so, psychological tests as we know them are more or less protected. And yes, we strongly believe that it is important to keep ethical standards high, and that it is possible that those who have the (dangerous) knowledge to push the boundaries of moral judgment in this technological revolution, will adhere to high ethical standards. However, we also know that no other technological revolution so far has ever placed much store in what was considered conventional ethics at its start (Bostrom, 2007). Independent of how ethics will be woven into this fabric, we are sure that this bid is going to be played out by how we respond to the great world problems. Put differently: Do psychological tests respond to the great world problems? To name but a few from the UN “Global Issues Overview”: ageing, atomic energy, climate change, democracy, poverty, human rights, migration, population growth, and so forth. Unfortunately, the answer oftentimes (usually?) is “no”. This makes psychological testing less visible and less relevant than it could and should be, it leaves other professions to speak about psychological testing, and leaves psychological testing wide open to infringement.

Active Advocacy for Psychological Testing Is Critical in the Future, If We Want to Become More Visible for Society

Maybe it is too harsh to affirm that we as specialists in psychological testing are not very relevant for society, but we think it’s no stretch to at least say that we are not very visible for society. One of the reasons may have been outlined by the editors of “Scientific American” (The Peculiar Institution, 2002, p. 8), when they said that “whenever we run articles on social topics, some readers protest that we should stick to ‘real’ science”. We as psychologists certainly publish an immense volume of science about psychological and educational assessment, but we publish for each other and tend to not be represented in society, in the public discourse, among decision-makers, in talk-shows, and so forth. Advocacy may be useful in this case as an activity by which an individual or a group tends to influence decisions in political, economical, or social systems and institutions. It may include multiple activities, such as media campaigns, public speaking, commissioning research, publishing, and others. Advocacy is certainly one of the ways through which psychological testing may become perceived as more useful and more present in society.

A number of excellent examples of advocacy in psychology may be given here, at the very least in order to show that it is possible for relatively small but enthusiastic groups to advance our common public image: e.g., the Center for Evidence-Based Management (CEBMA; http://www.cebma.org) and the Campbell Collaboration (CC; https://campbellcollaboration.org) are both good examples. These are all not-for-profit initiatives that digest relevant psychological science and “translate” it in the language of policy-makers, using scientific findings to tackle important global issues. Based on these and other experiences, psychometricians and others who are interested in psychological testing may want to develop a reference guide for how to advocate for psychological testing. Until such a guide is developed, we would urge towards more awareness from us all concerning our impact on society, and we would cherish changes in testing that will embrace and not reject new technologies. Maybe we will soon have a formal advocacy group for testing, advancing the systematic development of “digested,” policy-actionable information and encouraging more public speaking on behalf of testing.

Until then, we urge all those stakeholders who are active in testing to take a lucid look into their own practices, and ask themselves how they stand on these crucial issues. Do they really integrate the latest technology into their tests? If not, can they achieve in the foreseeable future such an integration? Can they contribute (through active advocacy or in other ways) even a little bit to the increased visibility of psychological testing as a force for good in modern society? If such a contribution to our domain is possible, it should not be avoided – we consider it a responsible way in which the practical domain of psychological assessment could be advanced, on par with any other advancements in our larger field of science.