Effects of Action Planning After Employee Surveys
Abstract
Abstract. Employee surveys are commonly used tools in organizations for the purpose of organizational development. We investigated the post-survey action planning process in 3,091 organizational units (OUs) of one large company in Germany. We expected action planning to lead to improvements on subsequent employee survey scores, with OUs that continuously and repeatedly planned actions showing the greatest improvements. Results suggest that the development of action plans can lead to improvements on subsequent survey scores, but effect sizes were generally small. Furthermore, managers who initiated action planning in the previous year were more likely to do so again the following year. Overall, these findings contribute to the literature on employee surveys by investigating effects of post-survey action planning.
Employee surveys are commonly used tools in organizations, with estimates of around 70% of organizations that survey their employees (Knapp & Mujtaba, 2010). Their purposes include increased communication, diagnostics, increasing job satisfaction, and/or intervention (Kraut, 2006). Traditionally, surveys asked questions about administrative topics, and acting on the results was the responsibly of human resource (HR) professionals. Their meaning has evolved though since then from a passive measurement device to an employee and leadership tool that can drive organizational development (OD; Hinrichs, 1991).
The existing literature on employee surveys is permeated by research on the surveying process itself as, for example, survey (non)response behavior (see e.g., Rogelberg et al., 2000), the issue of social desirability in survey responses (see e.g., Keiser & Payne, 2019), or modes of surveying (see e.g., Mueller et al., 2014). In contrast, there is little research on the steps that follow the data collection and that constitute the employee survey follow-up process. If the purpose of the survey is OD, it is desirable that during the survey follow-up, actions are taken to alleviate possibly found issues within the organization. Nevertheless, these steps are often neglected in practice, which can lead to frustration among staff when they feel that actions should follow after they have been asked for their opinion (Kraut, 2006).
The goals of this article are twofold. First, we report the results of a large-scale study in one company in Germany that investigated under which circumstances employee surveys can facilitate the OD process. This adds to the scarce research regarding this topic, as few studies have empirically examined the effectiveness of post-survey action planning. By doing so, we also want to set the stage for further research in this underdeveloped area, as this neglect limits the theoretical and practical progress of this common HR practice.
Second, we aim to contribute to evidence-based practice by demonstrating some of the challenges surrounding post-survey action planning, as many organizations expend resources on employee surveys that might not put forth the desired outcomes, as for example sustainable OD. The results show that the employee survey follow-up process can contribute to positive change but might not be easily mastered by managers and their teams, and we provide a set of recommendations on how to improve the process at an organization. We also discuss the potential challenges that the scientist–practitioner gap entails when surveys used in practice deviate from scientific best practices (i.e., theoretical development of surveys and factor analyses). Overall, this study provides insight to an example of an applied employee survey and sheds light on potential challenges that organizations might encounter when trying to utilize such as an effective change mechanism.
Employee Surveys
Generally, the employee survey process consists of several steps: survey development, preparation, survey period, data analysis, discussion of results, action planning, and review (Bungard et al., 2007). The focus of this study lies on the last three phases, which constitute the follow-up process of employee surveys, as the success of employee surveys that serve the goal of bottom-up change is especially dependent on these phases, more specifically on action planning and action implementation (Welbourne, 2016). Oftentimes though, the employee survey is considered completed once the data are collected, and conducting the data collection is equated with an intervention (Church & Oliver, 2006). On the contrary, employee surveys require dedication and commitment from the top management down to direct supervisors (Knapp & Mujtaba, 2010), and their purpose as change tools can only be met when survey results are acted upon (Church et al., 2012).
Evaluations of general feedback interventions have suggested that feedback itself might be a rather weak intervention (Kluger & DeNisi, 1996). There has, however, been some research examining the effectiveness of survey feedback and post-survey action planning that provides support for its utility in leading to improvements. Church et al. (2012) compared different groups of employees who were categorized by their impressions of the survey follow-up process. Employees whose organization shared employee survey results and followed up on them with action planning showed more favorable results than employees who did not experience this. Improvements occurred on the overall survey score, which included items about management, the company in general, and job training. Similarly, Björklund et al. (2007) found that groups of employees that reported having received survey feedback following an employee survey and that also developed action plans reported greater improvements on measures of leadership and commitment to the organization, than groups that did not receive feedback or that did receive feedback, but did not act on it.
When organizations or organizational units (OUs) experience issues on particular work-related topics, employee surveys can aid in discovering such, and engaging with the data and developing and implementing action plans can be useful in targeting such issues with change initiatives (Church et al., 2012). Hence, if issues can be ameliorated with concrete activities to enhance the work environment, this should manifest in subsequent survey scores. Even though it is usually up to managers to act on the data by initiating a collaborative follow-up and dialog with employees (Welbourne, 2016), we believe the OU level to be the appropriate level of analysis to measure improvements because it represents the weighted average perceptions of members of the OU regarding work-related conditions. The employee survey of this organization assesses such conditions by measuring a variety of work-related topics, making the overall aggregate survey score a formative measure (Howell et al., 2007) that the company uses as an indication of overall employee attitudes regarding their work (from here on referred to as the work attitude index; for a summary of topics measured in the survey, see Table 1). Furthermore, just as any effect requires a meaningful span of time to unfold and show (Wang et al., 2017), such positive effects on the work attitude index should be measurable after enough time has passed, as is given in annual employee surveys.
Organizational units that engage in action planning show improvements on subsequent employee survey results (i.e., the work attitude index) the following year.
Furthermore, as change research has shown, change processes typically occur in multiple steps that take a considerable amount of time (Armenakis & Bedeian, 1999), and organizational change processes require repeated following up on and revisiting (Stouten et al., 2018). Consequently, it is most likely that continuous action planning and long-term engagement with the data are called for to produce long-term positive changes. We therefore expect OUs that engage in continuous action planning over the course of both years to experience the most improvements on the work attitude index, as such OUs demonstrate sustainable engagement with their employee survey data.
Organizational units that continuously engage in action planning over 2 years show greater improvements on subsequent survey results (i.e., the work attitude index) than units that did not engage in action planning or only engaged in action planning following one survey year.
As any other work-related process, working with employee survey results within a team is a learning process, and both managers and employees must learn to master the tasks and challenges it entails (Welbourne, 2016). Consequently, the action planning process becomes easier to tackle for teams, and specifically for managers to initiate and guide. Generally, more practice in an action leads people to initiate goal-directed behavior without much consideration and deliberate intention, forming a habit (Danner et al., 2008). Conducting the employee survey result discussion and action planning is arguably not a habit, as intentional processes guide occasional (e.g., annual) behaviors (Danner et al., 2008). On the other hand, action planning is most likely not successfully done in one annual session, but rather requires several meetings and actions, which could become habitual of nature. Most likely, different components of the act as a whole range from automatic/semiautomatic to deliberately intentional, but nevertheless, we expect past behavior to be a contributing factor (perhaps indirectly through attitudes, perceived normative pressure, and perceived control toward the behavior) to whether an action will be performed again (Ouelette & Wood, 1998). Consequently, we expect that managers who initiated a follow-up process and conducted action planning with their OUs in 1 year are more likely to do so again the following year, as the process is better known and the hurdle to do so is lower.
Managers who initiated action planning with their organizational units the previous year are more likely to initiate action planning again in the following year.
Method
Organizational Setting and Procedure
A large company in Northern Germany has been conducting an annual employee survey since 2008, participation in which is voluntary and anonymous. To strengthen the employee survey as a change tool, the organization collaborated in this research. For the purpose of this study, we were able to obtain survey data from 2016, 2017, and 2018 for analysis. We matched OU survey results to the data of logged action plans over the years by their unique OU names. This was only possible for OUs that did not change their names between 2016 and 2018. The data reported in this paper are based on a larger data collection effort. Another manuscript currently under review is based on the same data set, but examines different research questions regarding post-survey action planning. We provided the Editor with a data transparency table.
After every annual survey, result reports for all OUs are generated (when > 5 participants), and managers and employees gain access to their respective OU results in an online reporting tool. This online platform also provides an action plan logging tool, whereas participation in the employee survey follow-up process, as well as the use of the tool, is not mandatory, but strongly encouraged. Managers receive a short one-time training session on effectively using the employee survey data and initiating action planning with their OUs. After results are available for viewing, managers must first enter a date for when they discussed the results of the employee survey with their OU. After that, managers and/or employees of the respective OUs can enter action plans into the tool and assign them to single or multiple corresponding survey items, making them accessible to the entire team. Whereas quantitative statistics on the action plans (i.e., how many and which survey items they were assigned to) are available to upper management und our research team, the specific task descriptions of the action plans are only available for viewing and editing to the respective OU members. The goal is to foster trust in the tool and encourage its use without fearing criticism from upper management regarding any descriptions of tasks.
Measures
The annual employee survey includes a variety of items pertaining to different work-related topics. The company did not consent to publishing the exact wording of all 24 items of their internal employee survey, but Table 1 summarizes the work-related topic each item of the survey covers. Example items include “I enjoy my current job,” “I believe <COMPANY> is an attractive employer,” or “My direct supervisor/line manager has an exemplary and persuasive leadership style.” Until 2016, the survey contained a total of 22 questions. In 2017, two additional questions regarding integrity within the OU (Item 23) and cooperation with subsidiary companies (Item 24) were added to the survey, but as “special topics,” their results are not included in the calculation of the overall survey score. The overall survey score is calculated by aggregating all responses to the 22 items of an OU to one overall score, which is used as an indicator of overall employee work attitudes (referred to as the work attitude index).
All survey question answers are provided on a 5-point scale with percent point values (0 = strongly disagree, 25 = slightly disagree, 50 = neither agree, nor disagree, 75 = slightly agree, 100 = strongly agree). The company chose to use percent point values instead of point values from 1 to 5 because it considers it more intuitive for employees to respond to items when thinking about the measured construct in percentages out of 100%. Due to data privacy restrictions of this organization, we were not able to attain survey responses at the individual level, but instead data aggregated to the OU level. We recoded the data to reflect a standard 1–5 Likert-type scale by using the transformation formula x/25 + 1 to ensure an accurate interpretation of simple effect sizes (i.e., raw mean differences in scores across years). As a limitation of using aggregated data, we were not able to calculate intraclass correlation coefficients (ICCs) for the items within OUs.
Assessing Survey Structure
As described previously, the data protection policies of this company generally do not allow the analyzing of survey response data at the individual level. However, to assess this survey’s structure, we attained two samples (n = 300; n = 301) of survey response data at the individual level from four randomly selected divisions of the organization who completed the survey in 2018 with consent from the data protection department. We recoded these data also to reflect a standard 1–5 Likert-type scale and hence recoded as follows: 0 = 1, 25 = 2, 50 = 3, 75 = 4, and 100 = 5.
Whereas this company uses the overall survey score as a measure of overall work attitude, it structures the survey into five thematic blocks of items for participants during survey participation. We examined this five-factor structure (see Table 1 for division of items into thematic blocks) with a confirmatory factor analysis (CFA; n = 301; all 24 items of the survey), which rendered poor fit statistics, (χ2[242] = 716.176, p < .001; comparative fit index (CFI) = .873; Tucker--Lewis index (TLI) = .855; root mean square error of approximation (RMSEA) = .089, 95% CI [.081, .096]; standardized root mean square residual (SRMR) = .057), whereas the significant χ2 test likely resulted from the large sample size to which this type of test is sensitive (Raykov & Marcoulides, 2011). Hence, in alignment with the organization’s effort to measure overall employee work attitudes by calculating an overall survey score, we proceeded by examining the survey as a one-factor solution.
One-Factor Confirmatory Factor Analysis
We tested a one-factor model of the data with the sample of individual survey responses (n = 301; all 24 items of the survey), and the CFA yielded poor model fit (χ2[252] = 843.409, p < .001; CFI = .839; TLI = .824; RMSEA = .098, 95% CI [.090, .105]; SRMR = .063). The factor loadings were all high (see the left of Table 1), but the poor fit statistics indicate that a multiple factor structure might render a better model fit.
Alpha is heavily relied on in the field as a measure of reliability, but it is not an accurate estimate when errors are correlated; a much more direct estimate of reliability is omega (Hayes & Coutts, 2020; McDonald, 1999). The omega for this scale indicates sufficient reliability (see the bottom of Table 1).
Exploratory Factor Analysis
Due to the poor model fit of the one-factor solution, we conducted an exploratory factor analysis (EFA) on the other sample of survey responses at the individual level (n = 300) using the oblimin rotation as oblique rotations allow factors to correlate. Based on the Kaiser rule, the EFA suggested a four-factor model (see the right of Table 1 for factor loadings). The first factor accounts for a total variance of 48.02% (with an eigenvalue of 11.53), the second factor accounts for 6.11% (with an eigenvalue of 1.47), the third factor for 4.76% (with an eigenvalue of 1.14), and the fourth factor for 4.39% (with an eigenvalue of 1.05).
Even though the survey was not developed based on and in accordance with the factor structure suggested by the EFA, the grouping of the items shows somewhat of a pattern. The first factor mainly contains items that relate to work processes and tasks, for example, process optimization. The second factor mainly contains items that cover interpersonal topics and relations, for example, culture and atmosphere. The third factor contains items that relate to personal well-being and stress, for example, pressure to perform, and the fourth factor contains items that are of an organizationally overarching nature, for example, the reputation of the company. However, as this factor structure is not based on preliminary theoretical considerations, some items do not thematically fit the factor they load onto well, as for example Item 3 (access to further job training) loading onto Factor II (interpersonal relations).
Four-Factor CFA
Following the results of the EFA, we conducted another CFA on the sample (n = 301) and fit a four-factor model to the data. The CFA yielded improved model fit (χ2[246] = 578.369, p < .001; CFI = .910; TLI = .899; RMSEA = .074, 95% CI [.066, .082]; SRMR = .053). When examining the χ2 difference, we obtained χ2(6) = 265.04, p < .001, indicating that the more complex model is a significantly better fit to the data. The factor loadings and omega estimates of each factor can be viewed in Table 1.
Uni- Versus Multidimensionality of the Survey
Generally, the psychometric properties of this measure are ambiguous in regard to its data structure. Whereas a one-factor solution seems to not fit the data well, the additional factors suggested by the application of the Kaiser rule do not meaningfully explain more variance. Past research has shown that this commonly used rule tends to produce too many factors and is not consistent in giving a meaningful number of factors (Conway & Huffcutt, 2003). The examination of the scree plot also provides evidence for a one-factor solution and can be viewed in Figure E1 in Electronic Supplementary Material (ESM) 1. Due to this ambiguity surrounding the survey structure and the mismatch between the structure suggested by factor analyses and the one implemented by the organization, we proceeded with the main analyses using the work attitude index (as a one-factor solution) as the outcome.
Statistical Analyses
To test Hypotheses 1 and 2, OUs (n = 3,091; see Table 2 for descriptive statistics) were coded into four groups based on whether they logged action plans for any of the 22 items used to calculate the work attitude index: Group 1: No action plans logged in either year, Group 2: Action plans logged in 2016 only, Group 3: Action plans logged in 2017 only, and Group 4: Action plans logged in both years. We conducted a 4 × 3 split-plot ANOVA with the work attitude index as the outcome and three measurement time points (2016, 2017, and 2018). We examined the interaction between time and the different groups of OUs. To test Hypothesis 3, we conducted a logistic regression with action plans logged in 2016 (OUs with action plans for any of the 24 items of survey; n = 721), predicting action plans logged in 2017 (OUs with action plans for any of 24 items of survey; n = 537). OUs were coded as 0 (no action plan[s] logged) and 1 (action plan[s] logged).
Results
Hypothesis 1
To assess whether OUs that logged action plans showed improvements on the work attitude index the following year, we examined the results of the 4 × 3 split-plot ANOVA (see Table 3 for Ms and SDs of work attitude index).
As group sizes become more discrepant when conducting an ANOVA, the assumption of homogeneity is more important to fulfill to avoid Type I error (Tabachnick & Fidell, 2007). Therefore, we examined the Levene test to ensure that the error variance of the dependent variable is equal across all groups at all time points. The results of the Levene test were not significant for any group at any time point (see Table 4), indicating no violation of the assumption of homogeneity. We found a significant interaction effect of work attitude index over time and groups (see Table 4).
We examined the within-group differences with post hoc comparisons (see Table 5). They showed that OUs without any action plans significantly decreased on work attitude index from 2016 to 2017 and showed no significant changes from 2017 to 2018. OUs with action plans in 2016 showed a significant increase on the outcome from 2016 to 2017 and no significant change from 2017 to 2018. OUs with action plans in 2017 showed no significant changes on the outcome in any year. Finally, OUs with action plans in both years showed a significant increase on the work attitude index from 2017 to 2018 and overall from 2016 to 2018.
Overall, these findings provide partial support for Hypothesis 1. OUs with action plans in 2016 showed an increase on work attitude index the following year, and OUs with action plans in both years showed an increase on the outcome from 2017 to 2018 and overall from 2016 to 2018. On the other hand, OUs with action plans only in 2017 did not show a significant increase on the outcome despite logged action plans in 2017 (see Figure 1).
Hypothesis 2
To assess whether OUs that continuously logged action plans over 2 years showed greater improvements on the work attitude index in comparison to the other groups, we examined the between-group differences in addition to the within-group differences with post hoc comparisons of the 4 × 3 split-plot ANOVA (see Table 3 for Ms and SDs). They showed that in 2016, OUs with action plans in 2016 had a significantly lower work attitude index than OUs without any action plans. There were no other significant differences between groups at baseline. In 2017, there were no significant differences between any of the groups. In 2018, OUs with action plans in both years had a significantly higher work attitude index than OUs with action plans in 2016.
Overall, the within-group differences provide partial support for Hypothesis 2 because OUs with action plans in both years showed, as the only group, overall improvements on work attitude index from 2016 to 2018. The between-group differences also provide partial support because, in 2018, OUs that logged action plans in both years showed a higher work attitude index than OUs that only logged action plans in 2016, but not in comparison to the other groups (see Table 6).
Hypothesis 3
To determine whether OUs that logged action plans in 2016 were more likely to do so again in 2017 (when the manager had initiated the survey follow-up process), we examined the results of the logistic regression. The significant unconditional model suggested that there was significant variation in the intercepts across the groups: B = −1.56, SE = 0.05, Wald = 1,079.00, Exp[B] = 0.21, p < .001. Consequently, adding a predictor to the model could help explain the variation, and we fitted a one-predictor logistic model to the data. Action planning in 2016 predicted action planning in 2017, providing support for Hypothesis 3. Having logged action plans in 2016 made it 3.74 times more likely that action plans were logged again for an OU in 2017: B = 1.32, SE = 0.10, Wald = 172.32, Exp[B] = 3.74, 95% CI [3.07, 4.55], p < .001.
Additional Analyses
Baseline Model
In response to feedback received from the reviewers, we performed additional analyses and accounted for baseline differences. This has also been suggested as an approach to account for effects caused by regression to the mean, which is a phenomenon in which relatively high or low values are likely to be less extreme and closer to the true mean at the next measurement when observing repeated measurements in the same subject (Barnett et al., 2005). Specifically, we performed mixed ANCOVAs with the same four groups of OUs as with the ANOVA, work attitude index as the outcome, and the baseline measure (2016) as a covariate.
The interaction between the work attitude index and its baseline measurement (2016) was significant, indicating there were significant differences of the outcome at baseline, which is analog to the results of the ANOVA analyses. The three-way interaction of work attitude index, its baseline measurement, and group belonging of OUs was not significant (see ESM 2).
Revisiting the Four-Factor Model
As another additional avenue to analyze these data, we conducted ANOVAs by treating each of the four factors of the survey as suggested by the EFA (see Table 1) as separate outcomes. We examined changes on the according factors within and between four groups: Group 1: No action plans logged for according factor in either year, Group 2: Action plans logged in 2016 for according factor, Group 3: Action plans logged in 2017 for according factor, and Group 4: Action plans logged for according factor in both years.
Overall, the results generally show similar patterns as those of the main analyses. Groups with action planning for Factors I, II, and III indicated improvements, but for Factor I, the group without any action plans also showed increases. However, action planning seemed to not have an effect on Factor IV (see ESM 2).
Reverse Causal Effects
Finally, to examine reverse causal effects, we investigated the data with cross-lagged panel regressions. Results generally indicated that work attitude index 2017 did not predict action planning in 2017, while controlling for action planning in 2016. Also, action planning in 2016 did not predict work attitude index 2018, while controlling for work attitude index 2017 (see ESM 3).
Discussion
We interpret our results in light of the findings from the ANOVA and comment on the results from the ANCOVA in a later section of the discussion. First, it is important to mention that overall, all found effects were small (see Table 4 for partial η2) when matching them to Cohen’s (1988) proposed benchmarks for small (0.0099), medium (0.0588), and large (0.1379) effects, whereas recent literature has cautioned against the use of these benchmarks to interpret effect sizes, as findings in the field bear little resemblance to those suggested values (Bosco et al., 2015; Norouzian & Plonsky, 2018). In addition, we examined the raw mean differences as simple effect sizes. For example, OUs that developed action plans in both years showed a significant increase from 2016 to 2018 with a raw mean difference of 0.08 (see Table 1). The small increases in the work attitude index are illustrated in Figure 1. OUs that did not develop any action plans showed an overall raw mean difference of −0.02 from 2016 to 2018. However, these two groups did not differ from each other in 2016 (at baseline), which indicates that action planning seemed to show some effect.
We believe that there could be several reasons for why the effects were small. First, even though OUs entered action plans into the online tool, this does not mean that all action plans were rigorously implemented. Hence, it is possible that action plans were developed and their implementation was started, but they were not followed up on and completed -- an issue common in practice (Brown, 2021). Unfortunately, we were not able to examine how exactly the action planning meetings were conducted. It would have been desirable to have more follow-up information available on the team meetings.
Second, even though managers receive some introductory training on using employee survey data and conducting action planning, this process can be difficult to master without extensive training or additional support, as for example that of consultants who help managers and employees make sense of the data (Brown, 2021). The literature on 360° feedback or other feedback processes that include subordinate ratings commonly recommends proper training for managers in the process and/or additional support through consultants, and yet this is often neglected in practice (Vukotich, 2014), similarly as it is with employee surveys (Hinrichs, 1996). Employee surveys demonstrate close parallels to upward feedback programs, especially when the employee survey contains items on direct leadership (Bungard et al., 2007). The utility of 360° feedback is limited without proper supportive resources (Vukotich, 2014), and there is reason to believe that the same applies to employee surveys. Hence, this could have contributed to the small effects found in this study for OUs that conducted action planning.
Additional analyses suggest that some managers could benefit from additional training and support for the follow-up process. In 2018, we conducted interviews with five managers who repeatedly showed poor employee survey results. Example quotes can be reviewed in Table E10 in ESM 4. They were asked about the issues they encounter during result discussions and action planning with their OUs. Those interviews indicated that managers experienced typical issues surrounding employee surveys: a lack of time to conduct the process properly, a certain level of helplessness on how to handle the data, and difficulties engaging their teams in the necessary discussions. It is important to note that these managers are not representative of the entire organization, but these interviews rather give insight to a particular group that experiences issues.
Overall, the findings of this study suggest that action planning following an employee survey can be beneficial for organizations and contribute to positive development and change, even though effects could possibly be enhanced by the above-mentioned measures. OUs with action plans in 2016 showed improvements on the work attitude index in 2017, and OUs with action plans in both years did after their second year of action planning (from 2017 to 2018). However, OUs with action plans in 2017 did not show improvements on their scores in 2018, and OUs with action plans in both years also did not from 2016 to 2017 (i.e., in their first year of action planning). It is possible that the action plans of these OUs were not effective enough (in those years) to produce improvements on subsequent scores. It shows though that OUs in this organization benefitted most and noticeably when they developed action plans across 2 years, as only these OUs showed improvements overall from 2016 to 2018. These OUs also ended up with a higher work attitude index in 2018 than OUs with action plans only in 2016, even though they did not differ at baseline or in 2017. OUs seemed to benefit from a second year of action planning, again suggesting that continuous and repeated action planning is most beneficial. This is in alignment with the notion that change takes time to unfold (Armenakis & Bedeian, 1999; Wang et al., 2017) and suggests that the employee survey process should not be seen as completed after result generation and action planning in 1 year. It is an ongoing all- and multi-year process that requires continuous engagement with one’s data (Kraut, 2006). It is also noteworthy to mention that OUs without any action plans decreased on work attitude index from 2016 to 2017. This indicates that not acting on survey results can even have negative effects over time (Church et al., 2012).
Interestingly, the results of the additional analyses examining improvements on each found factor of the survey (see ESM 2) suggest that topics of an organizationally overarching nature (e.g., the reputation of the employer) might not benefit much from team-level action planning, as OUs did not improve on this factor despite action planning for the associated items. The implementation of action plans for such topics might require external stakeholders and resources, and hence, improvements might be more difficult to achieve in the context of collaborative action planning between managers and their OUs.
The results of Hypothesis 3 show that OUs are (almost four times) more likely to engage in action planning again if they did the previous year. This supports the notion that managers are more likely to initiate the follow-up process after an employee survey when they have done so before, which could have made them more familiar and comfortable with the process. It is important to note though that past behavior is most likely just a proxy variable for psychological factors that generate consistent behaviors (Ouelette & Wood, 1998). Previous behavior alone will not lead to future behavior.
We also examined changes in the work attitude index while taking baseline differences into account, but there were no differences between groups at 2017 or 2018. As this is not a randomized control trial (RCT), we do not have randomized assignment to groups, which can lead to systematic differences at baseline. In this study, OUs with action plans in 2016 had a lower work attitude index in 2016 in comparison to OUs without any action plans. However, there is usually no expectation that the baseline is equal across groups in observational studies (Van Breukelen, 2006). It may be possible that OUs will self-select into action planning if their work attitude index is low and they wish to improve on such. In contrast, OUs with a higher work attitude index might not consider action planning as necessary. However, our supplemental analyses revealed that the work attitude index did not have a significant reverse effect on action planning.
These divergent results can be attributed to what is called Lord’s paradox (Lord, 1967). When baseline differences are zero (as mostly the case in RCTs), ANOVAs and ANCOVAs are expected to deliver the same results. However, when groups differ at baseline, these two different statistical analyses are not expected to do so. This paradox has led to much discussion and disagreement in the literature regarding the appropriateness of using change-score analysis or analyses of follow-up adjusted for baseline. Whereas some researchers argue that the adjustment of baseline measures is not appropriate in observational studies (e.g., Twisk et al., 2018; Van Breukelen, 2006), others argue that this type of analysis leads to more unbiased results in comparison to change-score analysis (e.g., Senn, 2006). This study sheds light on the broader issue that the choice of statistical methods influences obtained results in research (e.g., Silberzahn et al., 2018). Ultimately though, we agree that transparency regarding the research process should be best practice because it gives the reader a chance to evaluate the drawn conclusions (Silberzahn et al., 2018).
Consequently, our results should be interpreted with caution. Using ANCOVA has been recommended to account for effects caused by regression to the mean (Barnett et al., 2005), and the fact that our results are not significant after performing the ANCOVA and that we were not able to implement randomized assignment to groups suggests that we cannot exclude regression to the mean as a possible contributing cause. As noted earlier, it may be that groups with lower employee survey scores would self-select into the action planning group in an effort to improve on them. However, our supplemental analyses on reverse effects did not provide evidence for this assumption. Hence, groups were not equal at baseline, and in addition, the effects found with the ANOVA were small. Therefore, it is not surprising that effects could not be retained after accounting for baseline.
The Scientist–Practitioner Gap
Whereas the reliance of management practices on university-based research has increased over the years, the scientist–practitioner gap persists in that oftentimes management practices are not in alignment with the conclusions drawn from research (Lawler III & Benson, in press). It is not uncommon that surveys used in practice are built using less rigorous psychometric standards in comparison to surveys used in industrial and organizational psychology research. That is not to say that surveys in practice are poorly designed, but that the development process is different in that they are not so much based on theory, but more on the conditions and interests of the organization (Falletta & Combs, 2002).
It is desirable to administer surveys that show high validity and reliability, but surveys that are too long can lead to high attrition and low participation rates, whereas surveys that are too short can fail to properly measure the intended constructs (Kraut, 2006). That is not to say that constructs cannot be reliably measured with single items (Wanous et al., 1997) and that the survey this organization uses or surveys generally used in practice are not valid or reliable (Falletta & Combs, 2002). Whereas we do not have specific information on the development process of this survey (as this was undertaken in 2008), the items seem to have high face validity and there is no reason to believe that they fail to measure the intended constructs. This is, however, a limitation of this study and will be elaborated on in more detail in the limitations.
Depending on the resources a practitioner might consult for advice, there are many different recommendations concerning action planning. One popular approach is SMART – action plans should be specific, measurable, aligned, reachable, and time-bound (Kraut, 2006). However, we still lack research on the effects that survey composition and structure could have on action planning. Perhaps effects on work attitude index would have been stronger if the survey was presented to managers and OUs in alignment with the factors found in the factor analyses. It is possible that grouping survey items in psychometrically established factors could improve the action planning process because spillover effects could be easier to achieve (i.e., improvements in one area can spill over into improvements in other areas; Kraut, 2006). This could make it easier for organizational members to develop action plans that cover several similar topics, which could reduce the number of redundant actions and help efficiently invest available, yet scarce, resources.
Implications for Practice
When conducting an employee survey for the purpose of OD, it is advisable that organizations implement a rigorous follow-up process to ensure that survey results are not only generated but also reviewed and acted upon when appropriate (i.e., when issues are revealed). This step is oftentimes neglected, sometimes leading to more detrimental than beneficial effects (Church et al., 2012). Thorough communication of the survey process can support this and increase participation in the follow-up tasks (Welbourne, 2016).
The found effects of this study were small, and we cannot say without absolute certainty that they were not partially caused by regression to the mean. Yet, we believe there is utility in reporting small effects or even null results. Generally, in research, we are conditioned to view null results as worthless, but there is utility in considering what we can learn from such results and if they can add to scientific progress (Landis et al., 2014; Landis & Rogelberg, 2013). This study shows that a total of 992 OUs at this organization conducted action planning between 2016 and 2018 and showed some positive changes, but they might not have benefited from the process in their full capacity. The process seems to require improvement at this organization, as managers receive a short training on using employee survey data, but they are then solely responsible for the process. Hence, organizations might want to consider manager training on conducting a useful follow-up process with their teams to make them more confident in and knowledgeable of the process. Managers can easily feel overpowered by the employee survey follow-up tasks and might either not take action or be overwhelmed by too many initiatives they commit to (Hinrichs, 1996; Kraut, 2006). Supportive measures could also entail involving an impartial third party to guide managers and OUs through the result discussion process.
Furthermore, organizational members should not expect rapid changes after action planning following one survey. This can be demotivating to managers and OUs as they might not see immediate rewarding improvements on their subsequent survey results. With a long delay in gaining the reward, it is important to motivate managers to initiate action planning and familiarize them with the process because this makes it more likely for them to continue the process the following year. Hence, if organizations can motivate its leaders to engage in the follow-up process and remain engaged, the employee survey process as a whole can become a positive and rewarding experience for all stakeholders. To achieve this, it can be useful to inform managers and teams of the positive effects the process can have, but also of the prospective timeline, so that they do not become discouraged in the meantime.
Whereas undoubtedly it is desirable to use valid and reliable measures, practitioners are faced with the trade-off between psychometric rigor and feasibility of implementing a survey in practice. We would like to urge practitioners to pay close attention to the tools they are using and to regularly check, whether these are fulfilling their purpose in the organization, as for example OD. If they are not, this could mean that either the process surrounding the tool requires adjustment or the tool itself could benefit from revision. Science has not yet fully explored how factors such as format, scaling, and content affect post-survey actions, but we should strive to providing practitioners with answers to these questions in an effort to further close the gap between practice and science.
Limitations and Future Research
There are several limitations to this study, and hence, results should be interpreted with caution. One limitation includes the possibility that some OUs may not have used the online action plan tracking tool, but still developed action plans. Because the organization pursues thorough communication and strongly recommends using the tracking tool, this number of OUs is expected to be low. Furthermore, a possible confound that should not be neglected is the structural changes occurring within an organization of this size; the worker compositions of the OUs may not have stayed exactly the same over the years. Smaller structural changes are daily business in an organization of this size and are an inevitable limitation of using real company data. The greatest structural changes are expected for OUs that are removed, rearranged, and merged with other OUs, which would most likely also result in new OU names. Hence, such OUs would not be included in this study because we were not able to match them across survey years, but we suggest that such should be examined in future research as such restructuring could have consequences for behavior regarding action planning.
Another limitation were the unequal sample sizes for the calculations of the ANOVA, which is a condition of using real company data. Also, given the organizational constraint of only being able to analyze survey data aggregated to the OU level, the degree of within-unit variation in survey scores could not be assessed, which might vary based on the size of the OUs (LeBrenton & Senter, 2008). In addition and as previously mentioned, this is an observational study and not an RCT. Consequently, we lack random assignment and are faced with the possible limitations of selection bias.
As described in the discussion, the lacking information regarding the survey development process is another limitation of this study. Whereas the items have high face validity, the ambiguous results regarding the factor structure show that the survey was most likely not developed based on theoretical considerations. Nevertheless, this company has been using this questionnaire for 13 years in 36 different languages, and surveys without rigorous psychometric development processes are not uncommon in practice (Kraut, 2006), which, in our opinion, should not be equated with the assumption that they are poorly designed surveys.
Future research should investigate potential moderators of the relationships shown in this study, as for example leadership or readiness for change, as it would be useful for science and practice to identify factors that enhance or inhibit the follow-up process. Also, a qualitative approach for looking into reasons for why some OUs did not engage in the follow-up process could lead to important findings. A qualitative approach would also be useful to further investigate teams’ strategies for choosing among its low survey items for action planning; some topics might be especially avoided or preferred. It would also be interesting to investigate OUs’ ICCs for specific items and how this affects action planning. Perhaps more or less agreement on an item affects OUs’ choice in action plans. It would also be desirable to examine these effects in an experimental setting to avoid some of the limitations this study faced (i.e., no random assignment to groups). Finally, future research should integrate the survey design and survey follow-up more by investigating how survey design and administration affect action planning and its effectiveness.
Conclusion
This study examined the effectiveness of action planning in leading to subsequent survey score improvement. Generally, found effects for improvement on survey scores were small, which indicates that action planning can be beneficial, but this organization might benefit from improving the survey follow-up process. A possible avenue of action could be providing managers with resources, such as training and consultation, for the feedback meetings with their OUs. When not trained properly in OD and in using survey feedback, the process can be overwhelming, and managers and OUs might not harvest the full benefits of engaging with the survey data and conducting action planning. This example of an applied employee survey shows that not all OD surveys are designed in accordance with rigorous psychometric development processes. Yet, they might still be useful for OD and action planning when managers are trained properly as change agents.
Electronic Supplementary Material
The electronic supplementary material is available with the online version of the article at https://doi.org/10.1027/1866-5888/a000285
References
(1999). Organizational change: A review of theory and research in the 1990s. Journal of Management, 25(3), 293–315. 10.1177/014920639902500303
(2005). Regression to the mean: What it is and how to deal with it. International Journal of Epidemiology, 34(1), 215–220. 10.1093/ije/dyh299
(2007). Does survey feedback enhance the psychosocial work environment and decrease sick leave? European Journal of Work and Organizational Psychology, 16(1), 76–93. 10.1080/13594320601112169
(2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431–449. 10.1037/a0038047
(2021). Does action planning create more harm than good? Common challenges in the practice of action planning after employee surveys. The Journal of Applied Behavioral Science. Advance online publication. 10.1177/00218863211007555
Bungard, W.Müller, K.Niethammer, C. (Eds.), (2007). Mitarbeiterbefragung – was dann …? MAB und Folgeprozesse erfolgreich gestalten [Employee surveys – and then what …? Successfully designing employee surveys and the follow-up process]. Springer Medizin Verlag Heidelberg. 10.1007/978-3-540-47841-6
(2012). Without effort there can be no change: Reexamining the impact of survey feedback and action planning on employee attitudes. In R. W. WoodmanW. A. PasmoreA. B. Rami Shani (Eds.), Research in organizational change and development (Vol. 20, pp. 223–264). Emerald Group Publishing. 10.1108/s0897-3016(2012)0000020010
(2006). The importance of taking action, not just sharing survey feedback. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 102–130). Jossey-Bass.
(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
(2003). A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6(2), 147–168. 10.1177/1094428103251541
(2008). Habit vs. intention in the prediction of future behaviour: The role of frequency, context stability and mental accessibility of past behaviour. The British Journal of Social Psychology, 47(pt 2), 245–265. 10.1348/014466607X230876
(2002). Surveys as a tool for organization development and change. In J. WaclawskiA. H. Church (Eds.), Organization development: A data-driven approach to organizational change (pp. 78–101). Wiley.
(2020). Use omega rather than Cronbach’s alpha for estimating reliability. But…. Communication Methods and Measures, 1(14), 1–15. 10.1080/19312458.2020.1718629
(1991). Survey data as a catalyst for employee empowerment and organizational effectiveness. In R. J. NiehausK. F. Price (Eds.), Bottom line results for strategic human resource planning (pp. 301–308). Plenum Press. 10.1007/978-1-4757-9539-4_23
(1996). Feedback, action planning, and follow-through. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 255–279). Wiley.
(2007). Reconsidering formative measurement. Psychological Methods, 12(2), 205–218. 10.1037/1082-989X.12.2.205
(2019). Are employee surveys biased? Impression management as a response bias in workplace safety constructs. Safety Science, 118, 453–465. 10.1016/j.ssci.2019.05.051
(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. 10.1037/0033-2909.119.2.254
(2010). Designing, administering, and utilizing an employee attitude survey. Journal of Behavioral Studies in Business, 2(1), 1–14.
(2006). Moving the needle: Getting action after a survey. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 1–30). Jossey-Bass.
(2014). When is nothing something? Editorial for the null results special issue of Journal of business and Psychology. Journal of Business and Psychology, 29(2), 163–167. 10.1007/s10869-014-9347-8
(2013). Our scholarly practices are derailing our progress: The importance of “nothing” in the organizational sciences. Industrial and Organizational Psychology, 6(3), 299–302. 10.1111/iops.12054
(in press). The practitioner-academic gap: A view from the middle. Human Resource Management Review. 10.1016/j.hrmr.2020.100748
(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. 10.1177/1094428106296642
(1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68(5), 304–305. 10.1037/h0025105
(1999). Test theory: A unified treatment. Lawrence Erlbaum.
(2014). Effects of personalized versus generic implementation of an intra-organizational online survey on psychological anonymity and response behavior: A field experiment. Journal of Business and Psychology, 29(2), 169–181. 10.1007/s10869-012-9262-9
(2018). Eta- and partial eta-squared in L2 research: A cautionary review and guide to more appropriate usage. Second Language Research, 34(2), 257–271. 10.1177/0267658316684904
(1998). Habit and intention in everyday life: The multiple processes by which past behavior predicts future behavior. Psychological Bulletin, 124(1), 54–74. 10.1037/0033-2909.124.1.54
(2011). Introduction to psychometric theory. Routledge.
(2000). Employee attitude surveys: Examining the attitudes of noncompliant employees. Journal of Applied Psychology, 85(2), 284–293. 10.1037/0021-9010.85.2.284
(2006). Change from baseline and analysis of covariance revisited. Statistics in Medicine, 25(24), 4334–4344. 10.1002/sim.2682
(2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. 10.1177/2515245917747646
(2018). Successful organizational change: Integrating the management practice and scholarly literatures. Academy of Management Annals, 12(2), 752–788. 10.5465/annals.2016.0095
(2007). Using multivariate statistics (5th ed.). Pearson International Edition.
(2018). Different ways to estimate treatment effects in randomised controlled trials. Contemporary Clinical Trials Communications, 10, 80–85. 10.1016/j.conctc.2018.03.008
(2006). ANCOVA versus change from baseline had more power in randomized studies and more bias in nonrandomized studies. Journal of Clinical Epidemiology, 59(9), 920–925. 10.1016/j.jclinepi.2006.02.007
(2014). 360° feedback: Ready, fire, aim-issues with improper implementation. Performance Improvement, 53(1), 30–35. 10.1002/pfi.21390
(2017). Longitudinal research: A panel discussion on conceptual issues, research design, and statistical techniques. Work, Aging and Retirement, 3(1), 1–24. 10.1093/workar/waw033
(1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82(2), 247–252. 10.1037/0021-9010.82.2.247
(2016). The potential of pulse surveys: Transforming surveys into leadership tools. Employment Relations Today, 43(1), 33–39. 10.1002/ert.21548
