Calculating ROI for Engagement Surveys: Understanding the Potential of Analytics (Part 2)

Last month I provided an overview of analytic techniques described in the 2011 book Investing in People: Financial Impact of Human Resource Initiatives by Wayne Cascio and John Boudreau. This month’s posting will show how to use a few of these techniques to examine the costs and benefits of running an employee engagement survey.


Employee engagement is a concept defined as the logical commitment to an employer, an emotional commitment to that employer, and extra effort and energy expended for that employer. In simpler language, engagement is the head, heart, and hands of the workforce (See Mastrangelo, 2009, article in OD Practitioner, also available at HR.com). Typically, engagement is measured via an employee survey, where specific items are used to assess the attitudes, intentions, and behaviors that define the concept. One reason that employee engagement surveys are so common is because of research that showed how engagement was a leading indicator of two business outcomes. The first is what Cascio and Boudreau refer to as “participation membership” because it pertains to an employee’s decision to continue to work for the employer and to show up to work on time and ready for duty. The Corporate Executive Board (CEB: www.executiveboard.com) refers to this as Intent to Stay, and these intentions are excellent predictors of subsequent turnover. The second outcome linked to engagement is what the authors call “work strategies” because it pertains to effort and job performance. CEB refers to this as Discretionary Effort, and it is highly related to an employee’s energy level and proactive behavior at work. Obviously, if you have a survey measure that predicts who shows up and who pitches in everyday, you can detect and rectify problems as well as improve on mediocrity. How much value can the survey create? Let’s answer this question first for turnover costs and then for performance.


Survey Value from Preventing Turnover

The CEB’s Corporate Leadership Council examined data from approximately 50 large companies in 2004, and published what I consider a starting point estimate for the value of the survey. Their “10:9 Rule” states that every 10% improvement in commitment (measured through an engagement survey) can decrease an employee’s probability of departure by 9%. If you have no data from your own organization, than this finding does provide some help in estimating the potential turnover savings. If you know, for example, that each employee departure costs the company 2.5 times his/her salary in replacement costs and lost productivity in the subsequent year, then you can calculate savings based on the assumption that you can raise commitment scores by 10% in one third of the operating units. Even if you use the average salary at your company, you have an estimated monetary value based on turnover alone. The problem is that your estimate will be based on many assumptions. First, the “10:9 Rule” came from one set of companies that were studied a few years ago. Your organization is quite likely different from the “average” in this study. You would have a better estimate if you used your own organization’s data to statistically regress turnover on prior survey scores. You also might find that there are different “Rules” for different locations or jobs within your organization. Furthermore, employees from different jobs can have a much different replacement cost. The departure of highly trained employees in complex jobs costs far more than will the departure of employees in simpler jobs that require minimal skills – not to mention different job markets in different geographies. Thus, the calculation of costs could become much more complex.


Before leaving this topic of preventing turnover, it deserves mentioning that most organizations in 2012 are not experiencing too much turnover. In fact many companies tell me that there is not enough turnover. Employees who would have left the company if the economy was better have been forced to remain in jobs that they don’t want or that do not match their skills. Their performance may be enough to keep their jobs, but not enough to fulfill expectations. If your company is in this situation, turnover costs may only be a concern among high potentials and key performers.


Survey Value from Improving Performance


The same CEB Corporate Leadership Council data from 2004 led to their 10:6:2 Rule, which states that every 10% improvement in commitment can increase an employee’s effort level by 6%, which can improve an employee’s performance by 2%. Once again, I would only rely on this rule in the absence of data specific to your situation. For example, you might choose to look at the individual employee as the unit of analysis in your company’s data set and correlate individual survey scores with individual performance measures. With this analysis you would be able to show that increasing the survey score by one standard deviation would result in an increase in performance equal to the standard deviation of performance multiplied times the value of the correlation r (for example, +.35). Your estimation would be more accurate if you could use an objective measure of performance (e.g., production numbers, sales figures) that was observed at least six months after your survey scores were calculated. Note that you still do not have a monetary figure that would show the benefits of acting on the survey. To take that next step, you could rely on the rule of thumb that the difference between average performance at the 50th percentile and one standard deviation above average at the 85th percentile is equal to an increase in performance that equates to 40% of the total annual salary for that group of employees. If an increase in survey scores of one standard deviation would suggest a performance improvement equal to .35 of a standard deviation, then the monetary benefit would be .35 multiplied times 40% of the total annual salary multiplied by the number of employees whose engagement improves by a standard deviation.


Many companies choose not to connect survey scores with the performance of the individual employees who provided those scores. In those situations, you can still estimate the value of improving group performance. Instead of using the individual unit of analysis, you would aggregate survey scores for groups of employees and then correlate these figures with the groups’ aggregated performance scores. In these situations, you may be able to create more accurate monetary values because you probably have access to the difference in revenue or profit for these groups; therefore, you can calculate the difference between the average monetary value and the value for the group that is one standard deviation above the average. As long as you have a large number of groups to calculate your correlation (I would say 30 is the bare bones minimum), then once again you can calculate the monetary value of raising group survey scores by one standard deviation.


Before leaving this topic, I need to state that correlations and percentile distributions assume that the data form a normal distribution (or bell curve), and often this assumption is not met in survey data or performance data (both of which tend to be skewed toward the favorable end of the scale). Some other statistics may be required. Furthermore, there are far more complex techniques for calculating the monetary value of one standard deviation above the average performance rate. The 40% rule is by far the easiest to explain, and many believe it is reasonably accurate. Still, a lot of industrial/organizational psychology papers have been written on the dollar value of SDy, as it is known in those circles. As I said, in some cases you can get a more accurate figure with your company data (e.g., the standard deviation of individual sales employees’ revenue totals, the standard deviation of revenue across retail branches).


Return on Investment


Note that even if you calculate the benefits of improving survey scores, you have not shown ROI until you demonstrate that these benefits are greater than the monetary costs of running the survey and creating the actions that will improve engagement. You may want to consider the cost of outsourcing the survey administration and reporting, the cost of employees’ time to complete the survey, the cost of HR managers’ time to prepare for administration of the survey and interpretation of results, and the cost of leaders’ time and resources to make changes. If that sounds depressing, then consider the fact that the “shelf life” of these investments may last multiple years. Imagine if you keep a third of your high performers from leaving the organization for 3 years, and that their performance increases in year 1 and is sustained for year 2 and year 3. Now, your costs are offset by multiple years’ worth of benefits. The bottom line (no pun intended) is that calculating ROI for an engagement survey can be complicated and fraught with assumptions. Let's end on a simpler thought.


Perhaps the easiest approach, if you are looking to justify a survey project, is to estimate the total cost of administering and interpreting the survey, and then divide that figure by the number of managers who will be expected to take action on the surveys. For example, if your total estimated costs for the survey are $500,000, and you expect just 100 leaders to be held accountable for making improvements based on the survey, then your break-even point occurs when each manager saves or earns $5,000 during the same fiscal year. That is not an intimidating figure for a motivated leader. In fact, managers may enjoy collaborating with their employees to discover problems and solutions that will reach or surpass that $5,000 figure. For that matter, employees may enjoy the challenge of helping managers reach or surpass this goal. This is the type of analytic measure that leads to business outcomes, and that’s what leaders want to see.




This content is protected by the 1976 Copyright Protection Act of the United States of America. The proper citation for this blog is as follows: Mastrangelo, P. M. (date posted). Title of Post. The First Domino, available at http://the-first-domino.blogspot.com. This post is not intended to represent any person or organization other than Paul M. Mastrangelo.

The Myth of Generational Differences: Forget What Seems Intuitive

As we approach winter, it seems appropriate to review some questions that we hear this time of year:

1. Why does it get cold in the winter?
2. Why do we get sleepy after a turkey dinner?
3. Why do we keep poinsettia plants out of reach of small children?

Most of us honestly believe that we know the answers to these questions, but in reality we know the popular, yet incorrect answers. In other words, we often cling to false information simply because it is prevalent. Consider the commonly heard responses to these questions versus the actual correct answers:

1. Why does it get cold in the winter?

• Common, but Wrong Answer: The Earth moves away from the Sun.
    
• Correct Answer: The back and forth tilt of the Earth puts one hemisphere closer to the Sun (i.e., summer), while the other hemisphere is further away from the Sun (i.e., winter). If you remind people that Australia has its summer while the US and Europe have its winter, they might come around to the correct answer.

2. Why do we get sleepy after a turkey dinner?

• Common, but Wrong Answer: Tryptophan in the turkey makes us sleepy.
• Correct Answer: Turkey is usually served with starchy carbohydrates (e.g., mashed potatos, stuffing, pumpkin pie) that we often overeat, causing insulin resistance. As a result, we do not get all the energy from the food we ate, and we feel tired. As fun as it is to use a big word from chemistry, there is not enough tryptophan in turkey to cause this reaction.

3. Why do we keep poinsettia plants out of reach of small children and pets?

• Common, but Wrong Answer: Poinsettias are poisonous.
• Correct Answer: The poinsettia plant was wrongfully accused of causing the death of a young child in 1919, and since that time the urban legend is that the plant is poisonous. More correctly, however, the plant is not edible, meaning that it is not digestible. So, if your child or pet were to devour a large amount of the plant, you would likely see some vomitting, but the plant would cause no harm.

Notice how the common, but wrong answers tend to be simpler to explain, easier to comprehend, and more memorable. They have the sense of logic, by which I mean that there is just enough intuitive knowledge mixed in to make the explanations seem plausible. Perhaps people even figured out the answers themselves, using incorrect, but plausible logic. It is difficult to give up what we discovered for ourselves, even when we later learn that we were wrong in our conclusions.

Okay, so what about the behavioral and attitudinal differences among employees from different generations? We see differences all the time among our coworkers. The Millennials or Generation Y (1980-2000) are different from Generation X (1965-1979), who are different from the Baby Boomers (1946-1964), who are different from the Traditionalists (born before 1964), right? Think of all the changes that have taken place across those generations! It's no wonder that each generation approaches work differently, right? Surely, all the books, articles, and hype cannot be wrong! (I can’t even type that with a straight face.)

The correct answer is that the generational differences have been overstated. Just like the enduring wrong answers to the other questions we discussed, the real facts about people across generations is more complicated than what many writers would have you believe. Let me address the concept of employee engagement and how it varies by generation. As many of you know, employee engagement is a person’s logical commitment to an employer as well as an emotional commitment and a willingness to do more than what is called for. I refer to these three components as the mind, the heart, and the hands. Engagement is usually measured through employee surveys, and I have been working with that type of data for over 10 years. Are there differences in engagement scores by generation? Yes, but consider two factors that "co-vary" with generation.

First, there is tenure. Employees who are new to an employer have the most positive survey scores, and then scores decline after 2 years with the employer. This is the “honeymoon effect” and it is pervasive in survey data. Paradoxically, turnover is highest among this same group of newcomers (much to the dismay of engagement researchers). Now, if you see this pattern in recent data, you say "Ah, the Gen X and Boomers are less engaged than the Gen Y, but those Gen Y employees are still more likely to leave the company!" Well, yes, but that would be because the Gen Ys are newer to the company. This pattern has been around for decades - long before the Gen Ys were around. What appears to be a generational difference is really a tenure difference.

Second, there is age itself. As employees grow older, they go through life stages (relationships, kids, house buying, school, empty nest...). An employee's mobility is greatest before dual-career issues, kids, and school districts anchor them down. Again, if you look at one data set, you would say "Ah, the Gen Y crowd is so much more willing to move for a job," because Gen Y happens to be at a stage that allows more mobility. In reality that pattern occurred 50 years ago, too, when the Traditionalists were much more willing to move than the previous generation (My dad in RI considered moves to Florida and Alaska after returning from WWII, but mom said no way!).

Sure, people from different generations experienced different environments (e.g., technology, wars), but the effects of these environmental differences is small compared to other variables. You will have a deeper and more accurate understanding of your employees if you analyze their behaviors and attitudes according to tenure, age, personality, education level, and job market… instead of generational cuts. Why do you not hear this more often? Simpler explanations are easier to pass along and to remember... and bad science (sadly) seems to sell better.

  

This content is protected by the 1976 Copyright Protection Act of the United States of America. The proper citation for this blog is as follows: Mastrangelo, P. M. (date posted). Title of Post. Bump on a Blog, available at http://paulmastrangelo.blogspot.com. This post is not intended to represent any person or organization other than Paul M. Mastrangelo.