Developing an Instrument to Evaluate Undergraduate Healthcare Students’ Professionalism
Abstract
Abstract. Professionalism is a multidimensional quality acquired over time. Undergraduate years lay a foundation for the development of professionalism. Tools monitoring the students’ professional development are needed. Our tool development followed three phases: 1) identifying meaningful criteria for professionalism adapted to the education level, 2) developing an evaluation instrument in a process maximising construct validity, 3) testing the evaluation instrument in an interprofessional study. The evaluation instrument proved to be applicable in the field and it meets validity standards. Some differences between professions were found and discussed. Professionality starts to develop during the education, and early monitoring is important to support students’ optimal development. The evaluation instrument supports both self- and expert evaluation of healthcare students’ professional development.
Zusammenfassung. Professionalität ist eine multidimensionale Eigenschaft, die mit der Zeit erworben wird. Das Studium legt den Grundstein für die Entwicklung der Professionalität. Es braucht demnach auch Instrumente zur Beobachtung der professionellen Entwicklung der Studierenden. Die Entwicklung unseres Instruments erfolgte in drei Phasen: 1) Identifizierung aussagekräftiger Kriterien für Professionalität auf Ausbildungsniveau, 2) Entwicklung eines Evaluationsinstruments mit Maximierung der Konstruktvalidität, 3) Testen des Evaluationsinstruments in einer interprofessionellen Studie. Das Evaluationsinstrument erwies sich in der Praxis als anwendbar und erfüllt die Validitätsstandards. Es wurden Unterschiede zwischen den verschiedenen Berufszweigen festgestellt und diskutiert. Professionalität entwickelt sich während der Ausbildung, und eine frühzeitige Begleitung der Studierenden ist wichtig. Das Evaluationsinstrument unterstützt sowohl die Selbst- als auch die Expertenevaluation der professionellen Entwicklung von Studierenden.
Résumé. Le professionnalisme est une qualité multidimensionnelle acquise au fil du temps. La formation des étudiants pose les bases pour le développement de leur professionnalisme. Des outils d’évaluation du développement professionnel des étudiants sont donc nécessaires. Le développement de notre outil a suivi trois phases : 1) identifier des critères significatifs de professionnalisme au niveau de formation, 2) développer un instrument d’évaluation dans un processus maximisant la validité de conception, 3) tester l’instrument d’évaluation dans une étude interprofessionnelle. L’instrument d’évaluation s’ est avéré être applicable sur le terrain et il répond aux normes de validité. Certaines différences entre les professions ont été constatées et discutées. Il est important de soutenir le développement du professionnalisme des futurs professionnels de la santé déjà pendant leur formation. L’instrument d’évaluation permet aux experts d’évaluer et aux étudiants de la santé d’auto-évaluer ce développement.
Introduction
Professionalism is a multi-dimensional construct [1]. In the healthcare system, professionalism is defined as "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served." [2]. It is widely recognised that professionalism is central to medical practice as a core enabler for the provision of high-quality patient care, patient satisfaction, professional career satisfaction, and even healthcare outcomes [3]. Professionalism is often attributed to values such as altruism, honour, integrity, excellence, accountability, respect for others, compassion, continuous improvement, and partnership [4, 5]. It is also related to personal attitudes, social norms, and cognition [6, 7, 8, 9, 10]. In addition, professional healthcare providers also must possess general skills and competencies [11, 12], technical skills and significant clinical knowledge [13]. Professionalism has clear interprofessional components, as core values are shared [14, 15].
In the CandMeds Framework [16], professionalism is one of seven aspects that make up a medical expert. The learning objectives framework, PROFILES, for Swiss medical schools [17], states professionalism as one of four characteristics of a good doctor. PROFILES adopts the concept of "entrustable professional activities", which imply that professionalism is an evolving quality that needs to be validated. Wilkinson [1 presents five dimensions of professionalism, which unifies the attributes mentioned above. These are based on a systematic review of 82 studies describing a wide range of instruments assessing professionalism: 1) adherence to ethical practice principals, 2) effective interactions with patients and their relatives, 3) effective interactions with people working within the health system, 4) reliability, 5) commitment to improvement of competence in oneself, others and systems. These dimensions offer a comprehensive view on professionalism in our context.
Professional identity formation starts during education and continues to develop in clinical practice settings [18]. Clinical teachers represent important role models for students both in terms of helping them develop professional skills and through the provision of appropriate feedback [19]; they therefore need to know how to address professionalism when validating student interactions with patients [20]. As professionalism is generally regarded as difficult to assess 1 and is associated with post graduate healthcare providers, and a consistent definition in the context of students is lacking [21], we identify effective operationalisation with students as a research gap [21, 22, 23].
Monitoring the early professional development of healthcare students often focus, on the way they interact with patients [24]. Feedback is an important aspect in evaluating student professional development. An instrument that can be applied for self- and expert evaluation of professionality is needed, enabling students and teachers to compare their judgements and suggest areas for development [25, 26, 27]. In a recent systematic review, ten observer-based instruments for the evaluation of medical professionalism [28] were analysed, concluding that the evidence of reliable behavioural anchors was not established with these instruments and that future studies are needed to identify the relevant domains of medical professionalism.
Based on the line of arguments, this paper presents the development of an evaluation instrument to assess professionalism of healthcare students, following three steps: 1) identifying meaningful criteria for professionalism adapted to the education level, 2) developing an evaluation instrument in a process maximising construct validity, 3) testing the evaluation instrument in an interprofessional study. The following research questions (RQs) result:
RQI What are the meaningful criteria of professionalism on the education level?
RQII Do the results of the professionality evaluation vary between different health professions?
RQIII Are the self-evaluation and the related expert evaluations aligned?
Materials and Methods
Identifying criteria for professionalism
To achieve a comprehensive sampling of relevant criteria and existing instruments used to measure professionalism, we performed a ’State-of-the art-review’ according to the definition of [29], aiming at updating new literature following Wilkinson 1. We searched PubMed and Google Scholar databases using the search terms professionalism, professional attitude, competence, behaviour and professional performance. We also used the following MeSH-Terms to address healthcare professionals: physicians, medical students, residents, health personnel, clinical clerkship, general practice, medical undergraduate, medical schools and medical students. The search generated a total of 7,069 records from which we included 4,976 results from the last eleven years for initial screening (2009–spring 2021). Papers including tool descriptions were included. Papers without empirical data were excluded. From the 34 papers which met our criteria, based on consensus in the research team, we derived 17 criteria for professionalism that were estimated to be applicable also to healthcare students (Table 1).
Developing the evaluation instrument
We followed a scientific rationale of maximising content validity of the items [30, 31], and the seven steps of item development suggested by [32]. For each of the criteria, possible observable events on the educational level were discussed [19, 33]. Four members of the research team individually wrote at least two observable events to each criterion. The first set of 68 items was reviewed by the research team, which then selected two items for each of the 17 criteria. In order to avoid a potential bias within the researcher team, we conducted review in which the 34 raw items were revised by four groups: 1) 13 independent clinicians, 2) five medical students, 3) five medical-education experts and 4) four nurse educators. Clinicians were also asked to comment on the degree to which the items represented observable incidences. The two principal investigators met the members of the four groups and discussed their feedback in detail and included 33 revised items in the instrument.
Finally, one instrument for self-evaluation (Table 2) and one for expert evaluation were prepared. In the latter, statements like "I accept feedback" were reformulated to "The student accepts feedback". We implemented a 5-point Likert Scale ranging from "I totally agree" (5) to "I strongly disagree" (0), including also ’not applicable’.
Finally, we applied the five Wilkinson dimensions to group the criteria and related items. Two team members manually mapped the items to the five dimensions independently, any differences of allocation were discussed and items reallocated by consensus. The items and the groups are shown in Table 2. The evaluation instrument was professionally translated from German to English for this publication.
Testing the professionalism evaluation instrument between two health professions
The testing of the psychometric quality of the items with students represented the final step of the process. 238 students participated in total; 172 were 4th year medical students (55% females) from the University of Bern and 66 were 3rd year nursing BSc students, from Bern (86% females); all participated in a general practitioner-led internship. In both groups the supervisors of the students participated as expert evaluators.
The self- and expert evaluation instruments were distributed per e-mail to each student participant as a pdf document, both versions of the document were printed and a copy handed personally to their supervisor for the expert evaluation. All students completed the self-evaluation before they met with their supervisors. To standardise instructions, a cover letter explaining the aim of the study and with instructions for use was attached to all instruments. The pdf documents were returned to the research team by post.
Descriptive analysis was performed to compute the means, standard deviations, minimum and maximum scores for each item as well as item discrimination and item difficulty. As an indicator for reliability, we calculated Cronbach’s alpha. To analyse differences between the two health professions and to compare self- and expert evaluation of medical and nursing students, we used ANOVA. All analyses were conducted using R version 3.2.0 [34, 35].
Results
Meaningful criteria for professionalism on an educational level (RQI)
Based on the theoretical sampling and the multistep development, the validity of the 17 selected criteria from the literature was considered good. Table 3 shows the descriptive values for the items. We found that for most items the two lowest categories of the scale (0 and 1) were not used either by experts or by the students. Correspondingly, values for the difficulty index (P values) were high, ranging from 68.6 to 93.1 for self-evaluation and 79.5 to 95.7 for expert evaluations, indicating that most participants agreed on the items. Despite the high difficulty indices, discrimination indices (r values) showed good results, ranging between 0.30 and 0.66 for self-evaluation and 0.56 and 0.75 for expert evaluations. Cronbach’s alpha is 0.94 for self-evaluation and 0.96 for expert evaluation, indicating high reliability.
Variations between health professions and between students and observers (RQ II, RQ III)
RQ-II addresses differences between the professions, and RQ-III differences between self- and expert evaluations. We analysed these differences on the mean responses of the items grouped according to Wilkinson and group differences on the level of self- versus expert evaluation. In both cases we found differences, see Table 4.
For all variables significant interaction effects between self- and expert evaluation and the respective professions resulted. The medical students’ self-evaluation was lower than the nursing students’ self evaluation. Ratings of nursing students and their supervisors were similar. Expert evaluation for medical students was higher than their self-evaluation and higher than nursing students’ expert evaluation (ethical practice: p <0.05, F(1,236) = 6.19; interaction with patients: p <0.05, F(1,236) = 5.85; interaction with team: p <0.05, F(1,236) = 10.10; reliability: p <0.05, F(1,236) = 4.11; continued education: p <0.05, F(1,236) = 11.51; professionalism: p <0.05, F(1,236) = 9.46.
Discussion
Meaningful criteria for professionalism on an educational level (Step I and RQI)
We found that the literature describes professionalism using value-related constructs rather than clinical skills and knowledge. The latter is assessed well over the education course [36], whereas the evaluation of professionalism has been given comparatively little attention [37]. We were able to substantiate the granular nature of professionalism via a set of 17 criteria that interprofessional clinical teachers and educators deemed relevant during student education. Based on consensus in the research team, five dimensions of professionalism from 1 were found suitable to group the criteria and the related items. These groups were applied for further statistical analysis.
Development of valid items for an evaluation instrument (Step II)
We applied a systematic approach to item development, combining literature and qualitative measures, and invested due effort to ensure that the items are valid for the target groups. The validation process established that all the items were regarded as clearly understandable and applicable for students. The operationalisations of the items reflected what experienced clinicians actually observe when they work with students. Due to the fact that students do not have the same competencies as postgraduate clinicians, the items had to reflect what they are expected to do. Therefore, criteria described in the literature regarded as unsuited to a junior educational level (e.g. "takes decisions independently") were not included. We established good item validity through the method of item generation.
Item validation in an interprofessional study (Step III)
After application of the instrument, the statistical analysis of the items confirms high item quality, by good discrimination indices and high reliability. The descriptive data showed that the lower end of the scale was rarely used. This may be due to the fact that experts did not want to discourage students or put obstacles in their way, alternatively it may indicate that the students’ level of professionalism was already high and that low performing students may be an exception at this stage. A solution to this could be to shorten the scale by combining the lowest two categories (strongly disagree/rather disagree) into one for poorer candidates. This would resemble an assessment principle also applied at the practical Swiss federal licensing exam [38]. Altogether, our approach to item development and validation resembles the process described for the development of an instrument for interprofessional professionalism [39].
Variations between health professions and between students and observers (RQ II, RQ III)
Medical students’ self-evaluation scores were lower than those of the nursing students, and this was significant in two cases: "interaction with patients" and "reliability". This may be related to the fact that the medical students were only half-way through their education, while the nursing students were in their final year. Thus, the medical students may have a realistic perception of their level of professional development. This result is comparable to another study with 4th year students from the university of Bern [40], comparing self- and expert evaluations of scores from mini-CEX assessments. A comprehensive review and meta-analysis [25] has reported similar results. The high self-evaluations of nursing students, on the other hand, may reflect the fact that they are close to graduating and have gathered clinical experience through approximately 40 weeks of internships.
Whereas the nursing students’ self-evaluations were similar to the respective expert evaluations, there was a significant difference between students’ and experts’ evaluations for the medical students. The discrepancy between the experts’ evaluations of medical students may indicate that experts wish to encourage and motivate students in their progress. The experts’ tendency to overestimate medical student performances has been recognised as "the failure to fail effect" [41], stating that medical teachers find it problematic to communicate to students that they are not performing well enough.
Limitations
A critical question is whether our results are relevant across cultures and other professions. Since the literature shows that cultural aspects are relevant for the evaluation of professionalism [23], we would welcome further research addressing the applicability of our instrument across various countries and cultures, as well as other health professions (e.g. physiotherapists, pharmacists, dentists).
Conclusions
We have developed and tested a new instrument for the evaluation of professionalism in students. The instrument was developed in response to research which indicates that professionalism starts to develop during education and should be guided by feedback and interaction with medical teachers. We have shown that the instrument has high construct validity and is applicable at the educational level in an interprofessional context. Our instrument offers support in the early development of professionalism by purposeful monitoring during healthcare education. A potential need for action was identified in so far as medical teachers were observed to potentially over-estimate medical students’ performance.
- •We emphasise that professionalism is a quality that starts to form during the educational years. Therefore, clinical teachers have an important role in monitoring the healthcare students’ professionalism development.
- •We have identified relevant criteria appropriate to the educational level. The criteria and the related evaluation items help educators and students to address the relevant aspects during their educational development.
- •Based on current literature, a formative evaluation instrument was developed, thoroughly validated, and found suitable for healthcare students across professions.
- •The instrument is applicable to both self- and external evaluations and supports feedback and communication between student and clinical teacher on aspects related to professional development.
We are grateful to the department of Nursing at the Bern university of applied sciences and the team, particularly Theresa Scherer; and also to the team at "Berner Bildungszentrum Pflege", particularly Claudia Schlegel, for supporting us by distributing the questionnaire to their students. Our thanks also go to the Bern institute of Family Medicine and the mentors and mentees in the mentoring program in general practitioners’ offices for supporting the study.
Bibliography
. A blueprint to assess professionalism: results of a systematic review. Acad Med . 2009; 84 (5):551–558.
. Defining and Assessing Professional Competence. JAMA . 2002; 287 (2):226–235.
. Developing ideas of professionalism. Clin Teach . 2013; 10 (3):165–169.
. Peer estimation of lack of professionalism correlates with low Conscientiousness Index scores. Med Educ . 2009; 43 (10):960–967.
. What are the most important non-academic attributes of good doctors? A Delphi survey of clinicians. Med Teach . 2010; 32 (8):e347–e54.
. The utility of vignettes to stimulate reflection on professionalism: theory and practice. Adv Health Sci Educ Theory Pract . 2013; 18 (3):463–484.
, Assessment of professionalism: recommendations from the Ottawa 2010 Conference. Med Teach . 2011; 33 (5):354–363.
. Moving Beyond Nostalgia and Motives Towards a Complexity Science View of Medical Professionalism. Perspect Biol Med . 2008; 51 (4):599–615.
. Professionalism education should reflect reality: findings from three health professions. Med Educ . 2014; 48 (4):361–374.
, 3rd, Linas SL, Levinson W. A behavioral and systems view of professionalism. JAMA . 2010; 304 (24):2732–2737.
. Incorporating professionalism into Medical Education: The Mayo Clinic experience. Keio J Med . 2009; 58 :133–143.
. Professionalism in medicine: definitions and considerations for teaching. Proc (Bayl Univ Med Cent). 2007; 20 :13–16.
. A Framework for Understanding Lapses in Professionalism Among Medical Students: Applying the Theory of Planned Behavior to Fitness to Practice Cases. Acad Med . 2016; 91 (12):1622–1627.
, Defining and measuring construct of interprofessional professionalism. J Allied Health . 2012; 41 (2):e49–e53.
. Interprofessional professionalism: linking professionalism and interprofessional care. J Interprof Care . 2011; 25 (5):383–385.
(eds). CanMEDS 2015 Physician Competency Framework. Ottawa: Royal College of Physicians and Surgeons of Canada . 2015.
. The Profiles Working G. The "Profiles" document: a modern revision of the objectives of undergraduate medical studies in Switzerland. Swiss Med Wkly . 2016; 146 :w14270.
. In the eye of the beholder: student perspectives on professional roles in practice. Med Educ . 2013; 47 :397–407.
. The Professionalism Assessment of Clinical Teachers (PACT): the reliability and validity of a novel tool to evaluate professional and clinical teaching behaviors. Adv Health Sci Educ Theory Pract . 2014; 19 (1):99–113.
. More than a list of values and desired behaviors: a foundational understanding of medical professionalism. Acad Med . 2014; 89 (5):712–714.
. Defining professionalism in medical education: a systematic review. Med Teach . 2014; 36 (1):47–61.
. The conscientiousness index: a novel tool to explore students’ professionalism. Acad Med . 2009; 84 (5):559–565.
., McAleer S, Gibson J. Cultural similarities and differences in medical professionalism: a multi-region study. Med Educ . 2012; 46 (3):257–266.
. Defining medical professionalism: a qualitative study. Med Educ . 2007; 41 (3): 288–294.
. Medical students’ self-assessment of performance: Results from three meta-analyses. Patient Educ Couns . 2011; 84 (1):3–9.
. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA . 2006; 296 (9):1094–1102.
. Assessment of professionalism in medical students during a longitudinal primary care clerkship, MedEdWorld . 2012.
, A Systematic Review of the Quality and Utility of Observer-Based Instruments for Assessing Medical Professionalism. J Grad Med Educ . 2018; 10 (6):629–638.
. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J . 2009; 26 (2):91–108.
.
Reliability and Validity . In: Gellman MD Turner JR (eds). Encyclopedia of Behavioral Medicine. New York, NY: Springer New York: 2013. 1643–1644..
Questionnaire Design . In: Vannette D.Krosnick J. (eds) The Palgrave Handbook of Survey Research. Palgrave Macmillan, Cham . 2018. https://doi.org/10.1007/978-3-319-54395-6_53. 2018.. The Art of Question Phrasing. In Handbook of Survey Methodology for the Social Sciences. New York; Springer: 2012. 91–107.
. Defining professionalismfrom the perspective of patients, physicians, and nurses. Acad Med . 2009; 84 :566–573.
Team RC . R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2019. https://www.R-project.org; last access 25.07.2022.. ez: Easy Analysis and Visualization of Factorial Experiments. R package version 4.4–0. 2016. https://CRAN.R-project.org/package=ez.; last access: 25.07.2022
. An overview and approach to selecting appropriate patientrepresentations in teaching and summative assessment in medical education. Swiss Med Wkly . 2020; 150 : w20382. DOI: https://doi.org/10.4414/smw.2020.20382.
, Assessing Professionalism in Medicine – A Scoping Review of Assessment Tools from 1990 to 2018. J Med Educ Curric Dev . 2020; 7 :2382120520955159.
, The new final Clinical Skills examination in human medicine in Switzerland: Essential steps of exam development, implementation and evaluation, and central insights from the perspective of the national Working Group. GMS Z Med Ausbild . 2015; 32 (4):Doc40. DOI: 10.3205/zma000982, URN: urn:nbn:de:0183-zma0009828.
, The intersection of professionalism and interprofessional care: development and initial testing of the interprofessional professionalism assessment (IPA). J Interprof Care . 2019; 33 (1):102–115.
. Variability and dimensionality of students’ and supervisors’ mini-CEX scores in undergraduate medical clerkships – a multilevel factor analysis. BMC Med Educ . 2018; 18 .
. ’Failure to fail’: the teacher’s dilemma revisited. Med Educ . 2019; 53 (2):108–110.
. Medical professionalism: development and validation of the Arabian LAMPS. Med Teach . 2013; 35 Suppl 1:S56–S62.
, Adaptation, psychometric properties and feasibility of the Professionalism Scale Germany. Z Evid Fortbild Qual Gesundhwes . 2016; 113 :66–75.
, Understanding, teaching and assessing the elements of the CanMEDS Professional Role: canadian program directors’ views. Med Teach . 2014; 36 (5): 390–402.
. Developing ideas of professionalism. Clin Teach . 2013; 10 :165–169.
. Development and validation of a professionalism assessment scale for medical students. Int J Med Educ . 2014; 5 :205–211.
. Assessment of professionalism in medical students during a longitudinal primary care clerkship. MedEdWorld . 2012.
. Speaking the same language: Cross-sectional assessment of perceived contributors to professionalism across generations. Neurol Clin Pract . 2016; 6 (3):230–240.
. Residents’ perceptions of their own professionalism and the professionalism of their learning environment. J Grad Med Educ . 2009; 1 (2):208–215.
. Psychometric analysis of the Swedish version of the General Medical Council’s multi source feedback questionnaires. Int J Med Educ . 2017; 8 :252–261.
