ROI on a Shoestring: Evaluation Short-Cuts for Resource-Constrained Environments

By Holly Burkett

Despite the heightened emphasis on accountability of HRD programs, many organizations are deterred from comprehensive measurement and evaluation due to concerns about the cost, time, and human resources necessary to fully implement the process.

In resource-constrained, just-in-time business environments - where many professionals are increasingly charged to do more with less - there are special challenges to proving an intervention's bottom-line value. Yet the lack of results based information often causes the reduction or elimination of training resources when resource reallocation within the organization takes place.

To meet these challenges, many organizations around the globe are using proven cost saving approaches to ROI (Phillips, 1997), which significantly decrease resource requirements while still providing sound, credible data.

These include:
*  Plan for evaluation early in the process. 
*  Build evaluation into the training process. 
*  Share the responsibilities for evaluation. 
*  Require participants to conduct major steps. 
*  Use short-cut methods for major steps. 
*  Use sampling to select the most appropriate programs for ROI analysis. 
*  Use estimates in the collection and analysis of data. 
*  Develop internal capability to implement the ROI process. 
*  Streamline the reporting process. 
*  Utilize technology

These strategies can be easily adapted to training organizations of any size or scope and can be modified to meet evolving business needs.

Plan for Evaluation Early in the Process

Planning early and developing clear program objectives allows the training function, of any size or scope, to:

*  determine priorities and focus resources where they can have the most impact 
*  identify all possible solutions needed to improve performance.

For example, training may not always be the best solution, in which case, allocating resources to a training program would be a waste of time and money

*  clarify priorities and expectations of organizational sponsors who pay for training and advocate its value 
*  link program and evaluation objectives to business performance indicators

Assume that every program succeeds because of proper planning and that the best way to save time and resources is to know what you are doing and where you are going.

Build Evaluation into the Training Process

In making the transition from a traditional reactive evaluation process to a more systemic and results-based effort, it is important to dispel the myth that evaluation is an "add-on" process occurring at the end of a program. Integrating evaluation into the performance improvement process requires a framework in which evaluation strategies are linked through-out the cause analysis, design, and delivery stages of the performance improvement intervention. It also includes the development of processes, policies, and procedures concerning measurement and evaluation, which save time by providing focus and direction for those who work closely with the ROI process.

Share Responsibilities for Evaluation

A well-managed "shoestring" approach includes provisions for sharing responsibilities across all functions. This approach not only validates HRD work, but also serves to garner support and cooperation for implementing it. For example, management and stakeholder input is needed to ensure that the needs assessment includes specific business impact measures. Training facilitators and instructional designers need to work together in order to develop specific application objectives (Level 3) and business impact objectives (Level 4) and to ensure that program exercises, case studies, and skill practices relate to these desired objectives.

Use Short-Cut Methods for Major Steps

For a department with limited time and resources, using short-cut methods for major steps in the evaluation process provides a practical means by which an ROI evaluation can be developed.
One short cut method to use in lieu of a comprehensive needs assessment involves a Just-in-Time Gap Analysis process. This approach saves time by making assumptions about gaps and checking them out with reliable sources through a focus group format. Targeted questions to this group may include:

*  What are some other assumptions we can make about this issue? 
*  What other information is needed to tell a better story or give a bigger picture? 
*  What are the driving and restraining forces with the proposed solution? 
*  Can gaps be closed by redesigning a work process instead of creating a training event or a training solution?

Figure 1 depicts a tool showing how this process was applied to an operational performance gap, where poor job performance by operators was first identified as the cause and operator training was first considered the only solution. The Just-in-Time Gap Analysis process gathered input from 8 subject matter experts, in a 60-minute session, to "drill down" the issue and identify other contributing factors and potential solutions. This approach can also be used to check out assumptions or interpretations in the data analysis step of ROI calculations.

It is important, however, to integrate just-in-time solutions with existing systems and to caution against an over-reliance on a "quick fix" performance improvement mentality. While just-in-time solutions are a practical way to shave time and provide direction, they are most effective when provided in the context of established client relationships based upon trust and support. In other words, short-term techniques are no substitute for long-term business relationships, which ultimately drive the evaluation purpose, scope, and process.

Require Participants to Conduct Major Steps

Another cost savings approach to ROI includes having participants conduct major steps in the process. Participants are the primary source of understanding the degree to which learning is applied. So, when collecting data for Level 1, 2, and 3 evaluations, participants are a logical source. Participants can also be a source for Level 4 data as well as providing cost data and program isolation and data conversion input.

Use sampling to select the most appropriate programs for ROI analysis

Selecting a program for ROI analysis is critical. Typical criteria for identifying programs for analysis are to select programs that:
*  Involve large target audiences 
*  Are expected to be viable for a long time 
*  Are important to overall strategic objectives 
*  Are expensive 
*  Have high visibility, and 
*  Have a comprehensive needs assessment

The next major step is to determine how many projects to undertake initially and in which particular areas. A small number of manageable, initial projects are recommended -- perhaps two or three programs.

Use estimates in the collection and analysis of data

Accounting firms, engineering firms, and CFOs routinely use estimates in annual or quarterly reports. In the ROI process, estimates cut costs of data collection, program isolation, and data conversion.

Two key areas where estimates conserve resources are in the isolation and data conversion steps. Estimates can just as easily cut down on the credibility of the study if not carefully and methodically used. The degree to which a target audience will believe data incorporating estimates depends upon the following factors: reputation of the source of the data; reputation of the source of the data; motives of the evaluators; methodology of the study; and assumptions made in the analysis.

One way to build acceptance of the estimation process is to use a second, and even a third methodology, to back up findings.

Develop internal capability to implement the ROI process

There are many compelling reasons to enhance organizational capability in evaluation. First, evaluation makes good economic sense and should be required of any activity, which represents a significant expenditure of funds. Secondly, in a cost-competitive climate where budgets are constrained, organizations need specific measurements of a program's success to approves additional funds for the future.

From a professional standpoint, evaluation capability is essential to providing credible work-system diagnosis and effective gap and cause analysis. How can you know if you've matched the right solution to the right problem without some form of evaluation and measurement?

Routinely assess and develop your own evaluation competencies. The reality is that the message of a technically sound evaluation report is never good enough unless people believe the messenger.

Streamline the reporting process

Communicating the results of an ROI impact study is vital and can be time consuming. While a comprehensive report is important in the beginning, once management is comfortable with the evaluation process, a shoestring reporting approach may be more appropriate. This approach typically involves a one-page report presenting six PowerPoint slides summarizing results across all five evaluation targets, including a slide describing intangible benefits of the intervention.

This should only be used after several impact studies have been presented and the audience is familiar with the process. It's important to remember, however, that effective reporting, in any context, implies that communication will occur throughout the project.

Utilize Technology

The use of technology in conducting ROI impact studies cannot be emphasized enough. Use technology to:
*  develop surveys and questionnaires 
*  offer on-line needs assessment, self-assessments, or evaluation templates 
*  Create an internal listserv to exchange information about training needs, business issues, and evaluation trends 
*  Educate project sponsors, ROI Advisory groups, or site ROI champions 
*  Publish testimonials using newsletters, email, Web pages, or on-line discussion groups 
*  Create on-line discussion forums

Summary

Achieving a results-oriented evaluation strategy can be time consuming, labor intensive and sometimes perceived as intrusive. Yet with proper planning around a proven framework and shared responsibilities for major steps, the ROI process can be fully implemented in a cost-effective, systemic manner within any organization.

Practical application of these cost-saving approaches allows the HRD professional to present their work in terms of financial benefits that leaders understand and have come to expect, and ultimately leads to business partnerships that will enhance commitment for HRD initiatives going forward

References
1. Burkett, H. & Phillips, P.P., (2001). Managing evaluation shortcuts. Infoline. Alexandria, VA: ASTD.
2. Kirkpatrick, D.L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler Publishers.
3. Phillips, J.J. (1983). Handbook of training evaluation and measurement methods. Houston, TX: Gulf Publishing.
4. Phillips, J.J. (1997). Return on investment in training and performance improvement programs. Boston, MA: Butterworth-Heinemann.
5. Phillips, P.P. (2002). The bottomline on ROI. Atlanta, GA & Silver Spring, MD: Center for Effective Performance & International Society for Performance Improvement.
6. Rossett, Allison. First things fast, A handbook for performance analysis, Jossey-Bass Pfeiffer, San Francisco, Ca. 1999

Reprinted from the American Society of Training and Development, ASTD Links.

Holly Burkett is Principal of Evaluation Works, a consulting practice focused on helping organizations determine the business value of their HRD interventions. Holly authored the evaluation chapter of HPI Essentials (2002) and is the co-author of Managing Evaluation Shortcuts (2001), both published by ASTD. She may be reached at burketth@earthlink.net.