Development and Validation of a Safety Climate Scale for the German Armed Forces

Sample

The questionnaire was filled out by 989 participants during safety audits conducted by the Directorate Aviation Safety of the German Armed Forces. Informed consent was given, and the participation in the study was voluntary. To keep the threshold for participation as low as possible, demographic data were excluded. Therefore, with the questionnaire not including questions about occupation and profession, no deeper analysis of the sample structure was possible. However, since the questionnaire was administered personally during safety audits, it can be said that the sample consisted of military pilots (jet, transport, helicopter, search and rescue) and personnel from maintenance/repair as well as air traffic controllers.

Materials and Methods

The adapted ASCS-GAF was generated during the following steps: First, the original scale was translated by two German native speakers: one military officer (former jet pilot) and one aviation psychologist. Later, the scale was retranslated by an exchange officer from the UK Royal Air Force. During this translation process, the items were adapted to fit the needs of the user. Essentially, the modifications included changing the term pilot to a more extensive phrasing so that further personnel (e.g., maintenance) could be included and rephrasing the items into a present form since the interest was in the current status of safety climate. A detailed overview of the adapted items can be found in Table E1 in Electronic Supplementary Material 1 (ESM 1).

Procedure

Based on the translated and adapted items, the preliminary ASCS-GAF was designed. This questionnaire followed the original scale structure, and answers were also given on a five-point Likert scale reaching from strongly disagree to strongly agree (in German: stimme gar nicht zu, stimme eher nicht zu, teils/teils, stimme eher zu, stimme voll und ganz zu). The preliminary ASCS-GAF was distributed to personnel of the flying units during safety audits. More information about the procedure/instructions can be found in Electronic Supplementary Material 2 (ESM 2). The obtained data were submitted to an EFA and CFA, and psychometric properties were determined. This approach was chosen (1) to secure comparability to the original study from Evans et al. (2007) and (2) because during the translation process items were modified and therefore both scales differed to some extent.

For determining concurrent validity, the same two questions as in the original study were chosen. The original questions were “How safe do you think flying operations were in the company you worked for in the last 12 months?” answered on a five-point Likert scale from very poor to very good and “How has the overall level of flight operations safety changed in the company you worked for in the last 12 months?” answered on a five-point Likert scale from very much deteriorated to very much improved. Those questions were translated and retranslated by the same people that translated the original items.

Item Analysis

Before the initial data analysis was carried out, the data set was checked for outliers or incorrect entries. Therefore, a descriptive analysis of the sample was conducted that yielded no implausible values outside the range given within the questionnaire. Thus, all 18 items were left for analysis. Due to missing values, only 483 of the 497 data sets formed the basis for the factor analysis.

Exploratory Factor Analysis

As in the original study, a principal axis factor (PAF) analysis with direct oblimin rotation was carried out since it can be assumed that possibly subfactors of safety climate show a certain correlation with one another. Due to the choice of the analysis method, the result is not only of a descriptive character but actually allows a conclusion to be drawn about the latent variables on which the scales are based, that is, the corresponding theoretical construct (e.g., maintenance). The prerequisites for the PAF analysis were deemed appropriate for, as indicated by the Kaiser–Meyer–Olkin measure of sampling adequacy value of .94 (Hair et al., 1998). Communalities ranged from .26 to .74, and Bartlett's Test of Sphericity was significant, χ²(153) = 3,895.37, p < .001, implying that correlations existed among response categories.

As depicted in Table 1, the factor structure of the adapted German version essentially corresponds to the structure of the original scale. Three factors were extracted which collectively explain 50% of the variance (compared to 68% in the original study). Factors were extracted by using the eigenvalue >1 criterion.

Table 1 Summary of the three-factor model (Sample A, n = 483)

Items	Item loadings
	1	2	3
Note. Items of the original scale are presented due to better readability. Factor loadings <.30 are not depicted. Total variance explained = 50.26% (Factor 1 = 40.45%; Factor 2 = 5.85%; Factor 3 = 3.97%). Summary data for Factor 1 (eigenvalue = 7.76, M = 3.50, SD = 1.00), Factor 2 (eigenvalue = 1.48, M = 3.17, SD = 1.07), and Factor 3 (eigenvalue = 1.22, M = 3.23, SD = 1.01).
1. Suggestions for improving safety were encouraged	.58
2. Management were genuinely interested in safety issues	.78
3. Pilots were consulted about safety issues	.76
4. Pilots were able to openly discuss safety problems with supervisors or managers	.81
5. Pilots were given sufficient feedback regarding safety incidents involving company aircraft	.41
6. Management had a good understanding of operational issues that impacted on flight safety	.70
7. Management regarded safety to be an important part of company operations	.76
8. Management looked for underlying factors that contributed to safety incidents rather than blame the people involved	.64
9. Management encouraged pilots to consider safety more important than keeping to the schedule	.57
10. Management allocated sufficient resources to safety			.45
11. Training was received at regular intervals to refresh and update knowledge		.62
12. Regular training was provided for a range of emergency situations		.82
13. Company training provided adequate skills and experience to carry out normal operations safely		.84
14. Training was received when new procedures or equipment were introduced		.59
15. Aircraft was maintained to safety standards	.33
16. Equipment was updated and replaced when necessary			.61
17. Adequate resources were allocated to perform maintenance			.42
18. Reported technical faults that impacted on safety were rectified			.73

Table 1 Summary of the three-factor model (Sample A, n = 483)

View as image HTML

As in the original scale, Factor 1 (Management Commitment and Safety Communication) consisted of 10 items. However, only nine were identical while Item 10 loaded on Factor 3 and instead Item 15 loaded on Factor 1. Also noticeable is the relatively low factor loading of Item 5 on Factor 1. Factor 2 (Safety Training and Equipment) consisted of four items which completely corresponded to the original scale structure. Factor 3 (Equipment and Maintenance) also consisted of four items, but instead of the originally planned Item 15, Item 10 loaded on this factor, although with a low factor loading. Furthermore, Item 17 had a comparatively low loading on Factor 3. All in all, no item loaded on more than one factor.

Internal Consistency

Subsequent to the EFA, internal consistencies were calculated to estimate reliability of the scores derived from the adapted questionnaire. Cronbach’s α ranged between .70 and .90. Table 2 gives an overview of the internal consistencies for scores from both possible versions of the adapted scale, that is, the scale structure based on the EFA for the German sample and the original scale structure. As can be seen, the swap of the two items between Scale 1 and Scale 3 had only a very limited effect on the reliability of the two scores.

Table 2 Internal consistencies for the adapted ASCS-GAF

Scale	Items	Cronbach’s α (full sample) Nrange = 956–986	Cronbach’s α (Sample 1, EFA) nrange = 481–487	Cronbach’s α (Sample 2, CFA) nrange = 472–481
Note. ASCS-GAF = Aviation Safety Climate Scale for the German Armed Forces.
Original structure
1 Management Commitment and Communication	10	.90	.90	.89
2 Safety Training and Equipment	4	.84	.84	.84
3 Maintenance	4	.71	.70	.72
Adapted structure
1 Management Commitment and Communication	10	.90	.90	.89
2 Safety Training and Equipment	4	.84	.84	.84
3 Maintenance	4	.73	.74	.73

Table 2 Internal consistencies for the adapted ASCS-GAF

View as image HTML

Concerning the selectivity of the items, the omission of Item 5 on Factor 1 would lead to a slightly higher internal consistency only for the score form based on the original (α = .90) structure.

Confirmatory Factor Analysis

A CFA was performed on the split Sample B (n = 492) using AMOS 26.0 to test whether the data fit an a priori structure. As in the original study, the goal was to confirm the factor structure identified in the EFA. The CFA was based on the original structure from Evans et al. (2007) for several reasons that are presented during the discussion section of this study. Further information about the method used and the fit indices can be found in ESM 2. The model fit for the conducted CFA was primarily interpreted based on the SRMR = .05, secondary by CFI = .92 and RMSEA = .07, and tertiary on the χ² test = 427.94 (132), p < .001. Considering all relevant indices, the model fit can be interpreted as acceptable. For an overview of the results of the CFA regarding the ASCS-GAF, see Figure 1.

Model Comparison

The three-factor solution was compared against two alternative models. First, a parallel analysis according to O’Conner (2000) was performed on the split Sample A (n = 483) that yielded a four-factor solution. Second, a two-factor model was examined. Detailed results of the four-factor and two-factor solutions can be found in Table E4 and Table E5 in ESM 1, respectively.

Based on the parallel analysis, a CFA was again performed on the split Sample B (n = 492) using AMOS 26.0. The procedure and interpretation of the fit indices were conducted analogously to the CFA of the three-factor model. For the four-factor CFA, the results were SRMR = .05, CFI = .94, RMSEA = .07, and χ² = 271.90 (84), p < .001. An overview of the results of the CFA of the four-factor model is depicted in Figure E1 in ESM 1. Furthermore, the same procedure was conducted for the two-factor model. The results were SRMR = .06, CFI = .89, RMSEA = .08, and χ2 = 585.20 (135), p < .001. An overview of the results of the CFA of the two-factor model is depicted in Figure E2 in ESM 1.

General Discussion

The current study attempted to adapt a psychometrically solid questionnaire for assessing scientifically based data about safety climate in the flying units of the German Armed Forces. Therefore, the original Aviation Safety Climate Scale from Evans et al. (2007) was taken as a basis. The results showed that the adapted scale structure differs only slightly from the original when sticking to the three-factor structure Management Commitment and Safety Communication, Safety Training, and Equipment Maintenance while explained variance was 50% for this solution.

The comparison of the original three-factor model versus a four-factor solution shows a comparable fit. SRMR and RMSEA values are the same, CFI differs slightly (.94 vs. .92) but does not reach the threshold Bühner (2006) recommends, and χ² remains significant in both cases, although this value is particularly problematic for larger samples and should therefore be interpreted with caution. Concerning the parallel analysis on which the CFA of the four-factor model was based on, Bühner (2006) also points out that within a principal component analysis, the accuracy of a parallel analysis decreases whenever components are correlated and that within a PAF analysis it is not uncommon that the number of factors extracted is even more overestimated and that the additional factors mostly are of low statistical value. Since the four-factor solution does not show a noticeably better fit and the three-factor solution has already been established by Evans et al. (2007), furthermore under the mentioned restrictions of Bühner (2006) concerning the parallel analysis, and considering the scientific principle of parsimony, the four-factor solution does not seem to be a reasonable alternative.

The two-factor solution shows an even worse fit than the four-factor model. That is, here, too, no improvement compared to the three-factor model can be achieved. In this model, the original factors remain relatively stable (Factor 3 of the original model is forced to be broken up, and the core of the original Factor 1 and Factor 2 remains almost the same). Across both alternative models, Factor 2 in particular proves to be robust, as it always returns in its original form. The fact that Item 10 in this solution no longer loads on Factor 1 also fits the result that the original EFA showed. There is certainly proximity to Factor 3 of the three-factor model in terms of content, as this item checks for the allocated resources provided for safety. Although the content of this item is geared more toward resources in general (e.g., personnel resources, time aspects) and the items of Factor 3 of the original scale structure refer specifically to equipment and maintenance, there is a certain similarity between Item 10 (management allocated sufficient resources to safety) and Item 17 (adequate resources were allocated to perform maintenance). This circumstance should be further investigated in follow-up studies. In particular, the question arises whether this difference was actually perceived by the participants. It would make sense to ask the respondents directly and, depending on the results, to consider changing the items at this point (e.g., merge into one item, omit one of the two) and then see if this causes any changes in the model fit.

Building on the above, the three-factor model appears to be the best option for the moment. However, since the three-factor structure determined during the EFA for the ASCS-GAF differed at least to some extent to the original, it was necessary to decide whether the CFA should be based on the original structure from Evans et al. (2007) or the adapted structure of the ASCS-GAF. Due to the following reasons, the original structure was chosen: First, only two items loaded on different factors (i.e., Item 10 and Item 15 switched factors). Second, only two items (i.e., Item 5 and Item 17) had a relatively low factor loading but still loaded on only one factor. Third, the internal consistencies for both versions did not differ considerably. Fourth, statistical results are basically dependent on the sample (as mentioned above), and therefore, certain differences can be expected when adapting a questionnaire. Changing the scale structure after every analysis therefore does not appear expedient. Fifth, correlation values between the three factors and both questions for determining concurrent validity did not differ notably, meaning both possible scale structures (original and translated/adapted) show a similar association with perceived safety performance. Hence, it was decided to assume the original structure from Evans et al. (2007). CFA confirmed this assumption revealing an acceptable model fit. Furthermore, the adapted scale showed sufficient results of reliability and some indications of validity (although it should be noted that the approach in this study to capture convergent validity is a makeshift solution, and thus, the results here should be interpreted with caution). Thus, it is recommended for the time being to use the ASCS-GAF in the original scale composition for practical use and any future evaluations.

In the three-factor model and the alternative models, the explained variance was lower than in the original study (68%). This is surprising at first since the sample in the current study could be described as more heterogeneous than in the original because it not only included pilots but also technicians and air traffic controllers. Thus, one would expect a weaker effect of variance restriction and, hence, more explained variance. However, various circumstances could have come into play here: First, the survey was administered during flight safety audits, which have a supervisory character and can thus be perceived as a stressful experience by the units that are visited and checked. Therefore, it cannot be ruled out that a certain proportion of the personnel might have shown a lack of concentration and/or motivation when they filled out the questionnaire, thus possibly answering with a tendency toward the middle and so creating a loss of variance. Second, some differences may have occurred because the original scale was formulated in the past tense, but the German version was formulated in the present tense. Third, a clear temporal distance exists between the questioning of the subjects in the original study (2007) and the questioning of the subjects in the present study. To what extent this had an influence remains unclear. Fourth, the participants come from different cultural backgrounds (Australia/Germany). An interaction of these factors may have led to the decreased explained variance. It must be noted, however, that the explanatory attempts are rather speculative at this stage and should be further illuminated in evaluation studies for the adapted scale.

In summary, the approach to adapt an already existing scale to fit the specific needs of the user (i.e., the German Armed Forces) proved to be expedient for the moment. This is an important finding because of the following reasons: First, as already mentioned, at the beginning of the study, it seemed neither scientifically nor economically justifiable to produce an entirely new tool to assess safety climate. Second, the scientifically based further development in military aviation is often disadvantaged by small numbers, and therefore, specific solutions for its own needs must be developed. Third, the adaptation of an already existing tool might open up the possibility for cooperation with military partners. Thereby, new findings could be brought to effect much faster. In case of safety climate, this would be an advantage for all.

Still, there are a number of limitations (see the next section) that need to be taken into account. This implies that at the current stage it is not yet possible to say definitively whether the scale is fit for purpose in this form or whether changes still need to be made. However, the present study represents an important first step on the way to a solid scale for measuring safety climate in flying units of the German Armed Forces.

The adapted scale can for now be used by the Directorate Aviation Safety of the German Armed Forces to gain information about safety climate in the flying units and to derive measures to maintain an already high level or to further optimize it. Since a good safety climate goes along with a higher safety compliance and less accidents (Griffin & Neal, 2000; Neal et al., 2000; Varonen & Mattila, 2000; Zohar, 2000), the ASCS-GAF can make a significant contribution to the prevention of aviation accidents and incidents and thus to overall flight safety.

Limitations and Recommendations

While the current study makes an important first step toward adapting a scale for the measurement of flight safety, it is also important to name its limitations and recommendations for the future.

First, more information about the status of safety climate could be gained by including basic demographic variables such as age, grade group, area of activity, etc. These data were only omitted due to the sensibility of the issue. Of course, it is important to outweigh the guaranteed anonymity of the respondents and the gain in information.

Second, handing out the questionnaires during the safety audits had the advantage that the subjects could ask questions and be given detailed background information in person, which was very important in this study. At the same time, however, it must be considered that this could have influenced the participants' answers to a certain degree. In follow-up studies, this circumstance should be included.

Third, the relatively low reliability of the maintenance scale is less than desirable. A possible reason for that might be found in the fact that it is quite difficult for the average aircrew and air traffic controller to fully evaluate the quality of aircraft maintenance. The heterogeneity of the sample – as valuable as it was in other aspects of this study – might have shown a detrimental effect here.

Fourth, the approach to determine validity through correlations with questions about general operational safety as chosen in the original study can only be seen as methodical emergency solution for the fact that accidents/incidents happen (fortunately) too infrequently to be chosen as validity criteria. This approach was also chosen in the current study to increase comparability and due to a lack of other options at the time (e.g., safety data from error reporting systems are exclusively anonymous and can therefore also not be related to the flying units). However, it is strongly recommended to examine the validity of the ASCS-GAF further by means of content validity based on expert judgment. For a substantiated basis, a commission of technical experts should be formed, who preferably (1) have not previously been involved in this project and are therefore not biased, (2) come from different safety relevant areas, but all of which are related to flight operations, and (3) should describe the meaning of the items and scales (according to their assessment) independently of one another. The practical implementation of this content validation would have to be specified more precisely during a follow-up study.