Calculating ROI for Engagement Surveys: Understanding the Potential of Analytics (Part 2)

Last month I provided an overview of analytic techniques described in the 2011 book Investing in People: Financial Impact of Human Resource Initiatives by Wayne Cascio and John Boudreau. This month’s posting will show how to use a few of these techniques to examine the costs and benefits of running an employee engagement survey.


Employee engagement is a concept defined as the logical commitment to an employer, an emotional commitment to that employer, and extra effort and energy expended for that employer. In simpler language, engagement is the head, heart, and hands of the workforce (See Mastrangelo, 2009, article in OD Practitioner, also available at HR.com). Typically, engagement is measured via an employee survey, where specific items are used to assess the attitudes, intentions, and behaviors that define the concept. One reason that employee engagement surveys are so common is because of research that showed how engagement was a leading indicator of two business outcomes. The first is what Cascio and Boudreau refer to as “participation membership” because it pertains to an employee’s decision to continue to work for the employer and to show up to work on time and ready for duty. The Corporate Executive Board (CEB: www.executiveboard.com) refers to this as Intent to Stay, and these intentions are excellent predictors of subsequent turnover. The second outcome linked to engagement is what the authors call “work strategies” because it pertains to effort and job performance. CEB refers to this as Discretionary Effort, and it is highly related to an employee’s energy level and proactive behavior at work. Obviously, if you have a survey measure that predicts who shows up and who pitches in everyday, you can detect and rectify problems as well as improve on mediocrity. How much value can the survey create? Let’s answer this question first for turnover costs and then for performance.


Survey Value from Preventing Turnover

The CEB’s Corporate Leadership Council examined data from approximately 50 large companies in 2004, and published what I consider a starting point estimate for the value of the survey. Their “10:9 Rule” states that every 10% improvement in commitment (measured through an engagement survey) can decrease an employee’s probability of departure by 9%. If you have no data from your own organization, than this finding does provide some help in estimating the potential turnover savings. If you know, for example, that each employee departure costs the company 2.5 times his/her salary in replacement costs and lost productivity in the subsequent year, then you can calculate savings based on the assumption that you can raise commitment scores by 10% in one third of the operating units. Even if you use the average salary at your company, you have an estimated monetary value based on turnover alone. The problem is that your estimate will be based on many assumptions. First, the “10:9 Rule” came from one set of companies that were studied a few years ago. Your organization is quite likely different from the “average” in this study. You would have a better estimate if you used your own organization’s data to statistically regress turnover on prior survey scores. You also might find that there are different “Rules” for different locations or jobs within your organization. Furthermore, employees from different jobs can have a much different replacement cost. The departure of highly trained employees in complex jobs costs far more than will the departure of employees in simpler jobs that require minimal skills – not to mention different job markets in different geographies. Thus, the calculation of costs could become much more complex.


Before leaving this topic of preventing turnover, it deserves mentioning that most organizations in 2012 are not experiencing too much turnover. In fact many companies tell me that there is not enough turnover. Employees who would have left the company if the economy was better have been forced to remain in jobs that they don’t want or that do not match their skills. Their performance may be enough to keep their jobs, but not enough to fulfill expectations. If your company is in this situation, turnover costs may only be a concern among high potentials and key performers.


Survey Value from Improving Performance


The same CEB Corporate Leadership Council data from 2004 led to their 10:6:2 Rule, which states that every 10% improvement in commitment can increase an employee’s effort level by 6%, which can improve an employee’s performance by 2%. Once again, I would only rely on this rule in the absence of data specific to your situation. For example, you might choose to look at the individual employee as the unit of analysis in your company’s data set and correlate individual survey scores with individual performance measures. With this analysis you would be able to show that increasing the survey score by one standard deviation would result in an increase in performance equal to the standard deviation of performance multiplied times the value of the correlation r (for example, +.35). Your estimation would be more accurate if you could use an objective measure of performance (e.g., production numbers, sales figures) that was observed at least six months after your survey scores were calculated. Note that you still do not have a monetary figure that would show the benefits of acting on the survey. To take that next step, you could rely on the rule of thumb that the difference between average performance at the 50th percentile and one standard deviation above average at the 85th percentile is equal to an increase in performance that equates to 40% of the total annual salary for that group of employees. If an increase in survey scores of one standard deviation would suggest a performance improvement equal to .35 of a standard deviation, then the monetary benefit would be .35 multiplied times 40% of the total annual salary multiplied by the number of employees whose engagement improves by a standard deviation.


Many companies choose not to connect survey scores with the performance of the individual employees who provided those scores. In those situations, you can still estimate the value of improving group performance. Instead of using the individual unit of analysis, you would aggregate survey scores for groups of employees and then correlate these figures with the groups’ aggregated performance scores. In these situations, you may be able to create more accurate monetary values because you probably have access to the difference in revenue or profit for these groups; therefore, you can calculate the difference between the average monetary value and the value for the group that is one standard deviation above the average. As long as you have a large number of groups to calculate your correlation (I would say 30 is the bare bones minimum), then once again you can calculate the monetary value of raising group survey scores by one standard deviation.


Before leaving this topic, I need to state that correlations and percentile distributions assume that the data form a normal distribution (or bell curve), and often this assumption is not met in survey data or performance data (both of which tend to be skewed toward the favorable end of the scale). Some other statistics may be required. Furthermore, there are far more complex techniques for calculating the monetary value of one standard deviation above the average performance rate. The 40% rule is by far the easiest to explain, and many believe it is reasonably accurate. Still, a lot of industrial/organizational psychology papers have been written on the dollar value of SDy, as it is known in those circles. As I said, in some cases you can get a more accurate figure with your company data (e.g., the standard deviation of individual sales employees’ revenue totals, the standard deviation of revenue across retail branches).


Return on Investment


Note that even if you calculate the benefits of improving survey scores, you have not shown ROI until you demonstrate that these benefits are greater than the monetary costs of running the survey and creating the actions that will improve engagement. You may want to consider the cost of outsourcing the survey administration and reporting, the cost of employees’ time to complete the survey, the cost of HR managers’ time to prepare for administration of the survey and interpretation of results, and the cost of leaders’ time and resources to make changes. If that sounds depressing, then consider the fact that the “shelf life” of these investments may last multiple years. Imagine if you keep a third of your high performers from leaving the organization for 3 years, and that their performance increases in year 1 and is sustained for year 2 and year 3. Now, your costs are offset by multiple years’ worth of benefits. The bottom line (no pun intended) is that calculating ROI for an engagement survey can be complicated and fraught with assumptions. Let's end on a simpler thought.


Perhaps the easiest approach, if you are looking to justify a survey project, is to estimate the total cost of administering and interpreting the survey, and then divide that figure by the number of managers who will be expected to take action on the surveys. For example, if your total estimated costs for the survey are $500,000, and you expect just 100 leaders to be held accountable for making improvements based on the survey, then your break-even point occurs when each manager saves or earns $5,000 during the same fiscal year. That is not an intimidating figure for a motivated leader. In fact, managers may enjoy collaborating with their employees to discover problems and solutions that will reach or surpass that $5,000 figure. For that matter, employees may enjoy the challenge of helping managers reach or surpass this goal. This is the type of analytic measure that leads to business outcomes, and that’s what leaders want to see.




This content is protected by the 1976 Copyright Protection Act of the United States of America. The proper citation for this blog is as follows: Mastrangelo, P. M. (date posted). Title of Post. The First Domino, available at http://the-first-domino.blogspot.com. This post is not intended to represent any person or organization other than Paul M. Mastrangelo.