What's the ROI of ROI?
By Kris Ellis

Booz Allen Hamilton's Center for Performance Excellence (CPE) launched an executive coaching program a few years ago with a clear-cut objective: to connect leaders with executive coaches who could help them further their development goals.

The program allowed leaders of the McLean, Va.-based professional services firm to choose a coach qualified by CPE or find one by other means, if they preferred. In short order, participants reported that coaching was improving their leadership abilities. CPE tracked who was making use of the program, as well as general satisfaction levels, but was unsure how to measure the program's value in greater depth.

It would be four years before Booz Allen brought in a vendor to help determine appropriate measurements and calculate return on investment (ROI) for its executive coaching program. When the study was completed earlier this year, it documented a ROI of about $3 million a year.

Should the study have been done earlier? Not according to Ed Cohen, CPE's senior director. "No one was asking us for the data, so we knew that there wasn't a lot of urgency to invest the time and resources to get the information," he says. "We also had a whole list of initiatives that we were putting into place and that needed to be measured, so we knew that the [coaching ROI] data wouldn't result in either more or less investment in the program."

That measured approach to measuring is deliberate, Cohen says. He believes ROI calculation isn't something you do just because you can. "You have to decide at what point you are going to measure," he says. "Measuring constantly can be very costly; measuring strategically is definitely much better and much more effective."

Timing is just one of the questions training professionals must consider if they are going to evaluate a learning program at the ROI level. Other considerations abound: What makes gathering ROI data worth the expense? What programs should be evaluated for ROI? What programs shouldn't? Where do you draw the line?

Consider the Cost

Jack Phillips, founder and president of Chelsea, Ala.-based ROI Institute and author of Return on Investment in Training and Performance Improvement Programs (Butterworth-Heinemann, 1997), recommends that learning organizations evaluate 5 percent to 10 percent of their initiatives at the level of ROI. "It is a small number," he acknowledges, "but keep in mind that you are collecting data all along the levels of evaluation to get to ROI. So every program is evaluated at some level."

Phillips suggests reserving ROI analysis for certain types of learning programs. The best candidates: programs with a great deal of visibility, interest from management, or strong ties to the company's strategic objectives. ROI analysis is often not appropriate for task-oriented or technical training, says Phillips.

Todd Clyde, practice leader for learning solutions at Convergys, a Cincinnati provider of training and development products and services, can attest to that. When a telecommunications company hired Convergys to help determine the ROI of a training program for field technicians who install cable and phone systems and maintain the equipment, Convergys found that an ROI evaluation was possible but probably not cost-effective. "We didn't have tangible metrics to measure and correlate, and we proposed to the company that to get them would take too many people's time," says Clyde. "They agreed that the cost to calculate the return on investment wasn't worth it."

Booz Allen's Cohen agrees that only certain programs should be evaluated. "Booz Allen would never calculate the ROI of a specific course, but we do calculate the ROI of specific areas of executive development, such as coaching or its curriculum," he says.

The company also conducts ROI studies for large-scale strategic initiatives, such as its e-learning strategy. Rather than figuring the ROI of each learning project, Cohen says, the firm looked at total costs, including its investment in building the infrastructure, purchasing off-the-shelf courses and creating customized programs.

The motivation and the driving force behind an ROI evaluation vary. "Sometimes the decision to do ROI comes from the CPE's advisory team, which includes senior leaders from across the firm," Cohen says. "Sometimes it's driven by the fact that to get a reinvestment to work on an initiative, we have to demonstrate ROI. And sometimes it's driven by our own internal desire to measure the program."

Before-the-Fact Evaluation

Although 75 percent of the companies in the 2004 Training Top 100 report calculating ROI, Phillips believes that the proportion who do so in the learning and development industry overall is closer to 20 percent.

Organizations that have yet to implement ROI evaluation face certain barriers. "Today most customers talk about ROI, and at a high level they recognize the significance of measuring the impact of learning on an employee behavior or business result," says David Fabianski, president of Pearson Performance Solutions, a consulting firm based in Pawcatuck, Conn. "But most don't ask for a comprehensive ROI model or ROI analysis. I wish this wasn't the case, but tight budgets and poor pre-solution data often create an environment where ROI takes a back seat."

In fact, Fabianski points out, "if you don't have good data going into an initiative, then you can't truly measure ROI. You need to be able to clearly represent the before and after to deliver a true ROI."

Phillips estimates that the ROI process can be added to about 5 percent of an organization's learning solutions for about 3 percent to 5 percent of the overall training budget. If that sounds high, Phillips says, consider that the investment is sometimes offset by the additional results achieved when ROI study findings are implemented.

Booz Allen's executive coaching ROI study cost between $30,000 and $35,000 for outside consultant fees, as well as internal labor costs—a nominal amount compared with the roughly $3 million identified as the program's annual ROI. The firm also calculated the ROI of its e-learning initiative, focusing on cultural acceptance of e-learning as a delivery tool, as well as on cost savings when compared with classroom instruction. The company invested about $20,000 in the study. "We determined we were about $100,000 in the positive with our e-learning strategy," says Cohen.

Decide whether you are going to evaluate a program or initiative at the ROI level as it is being designed, Phillips recommends. "That's a big paradigm shift because people think about evaluation as after the fact," he says. "But the whole process works much better when you think about evaluation up front."

Some organizations still believe that they can build a sufficient case for the ROI of training by calculating the costs and benefits of more easily quantifiable factors like time and quality. But Fabianski says he sees an increasing recognition that a true ROI study should assign a monetary value to less tangible factors, such as work habits and attitudes. But that, he concedes, can be difficult. "More often than not, it comes down to the customer experience, so there can be considerable costs," he says.

Convergys often suggests clients limit ROI studies to certification programs. That way, they minimize their costs by focusing on the most significant data. "Calculating ROI on one course is very difficult," says Clyde. "You might see an initial bounce in performance but it degrades very quickly or it is very hard to show direct cause and effect. So we recommend calculation of ROI on the level of certified vs. noncertified." For example, you might compare how long it takes newly hired salespeople with certification to achieve quota vs. those without certification or the differences in retention rates for the two groups.

In addition to having more clients ask about ROI, Clyde notes, more of them ask about Convergys's track record in this area. "We have had a number of forward-thinking sales vice presidents challenge us to prove that we can deliver a measurable improvement," he says.

Investing in Learning

Armonk, N.Y.-based IBM takes pride in calculating the ROI of its training and learning efforts; an estimated 35 percent of its programs get that level of scrutiny. "Learning is treated like an investment, just as you would treat a capital investment," says Nancy Lewis, vice president of On Demand Learning for IBM. "If you approach learning with that kind of mentality, then it's a given that there will be some type of articulation of the value that it brings."

In recent years, Lewis says, IBM's definition of value at the ROI level has become more complex and now includes different levels. Of course, the value of learning is analyzed for individual employees, but the impact of learning on teams is also considered. That means an ROI analysis takes into account the impact of a learning solution on an individual's ability to work within his or her team, on the teams' ability to work together, and even on how the organization as a whole is affected. Evaluation questions must be formulated to capture the effects of the learning in each area.

Lewis thinks IBM isn't just redefining what constitutes value at the ROI level of evaluation. "As we start getting into breakthrough and innovative learning and development, it forces a new paradigm for how we think about the impact the solution has," she says. That means even the usual evaluation models, like Kirkpatrick's four levels, take on new meaning for IBM. For instance, Level One evaluations are often dismissed as nothing more than smile sheets, but in a new learning context, evaluating awareness and preference can reveal much deeper information.

Consider IBM's Basic Blue for Managers, a blended e-learning program for new managers. It consists of three tiers of technology-delivered training and one week in the classroom to develop higher-level proficiencies. Before new students started the program, they were asked if they preferred online or classroom development; all of them said they preferred classroom. After graduating from the course ten months later, they were asked the same question. This time, most reported that they wouldn't "suffer" through traditional classroom training after their experience with Basic Blue.

In calculating ROI, IBM draws on its deep in-house expertise and also works with outside experts. Booz Allen does the same, on the theory that while having the internal capabilities to do a ROI study reduces the cost of the process, going outside the company can garner an objective, fresh and even leading-edge perspective.

While changes in how measurement and ROI are viewed and implemented continue to evolve, Lewis urges organizations that are not yet conducting ROI-level evaluation to get on board. Booz Allen's Cohen agrees. "We in the learning profession have for years avoided going all the way to ROI out of fear," he says, "but if we are going to continue to get the investments we are now getting for learning, we have to show how it impacts the bottom line."

Reprinted with permission from Training Magazine published by VNU

Kristine Ellis is a freelance writer for Training. edit@trainingmag.com