The Learning Measurement Staircase
By Jeffrey Berk

All too often organizations attempt to reach the holy grail of learning measurement without taking baby steps first. The problem with biting off more than you can chew is that you end up dragging out projects that have lofty goals and it ends up losing momentum or fails to even get off the ground. This article proposes a three-step hierarchy to organizing your learning measurement continuum. Following it will ensure that you keep improving building on the prior success to reach the next step.

Step One: Day-to-day continuous improvement

Before you can do a deep dive into ROI analysis or optimizing the learning budget based on business results you need to put in place a model that collects solid performance improvement metrics and key indicators of value. This first step is analogous to Six Sigma. Define, Measure, Analyze, Improve and Control. You need a solid foundation of metrics to help make sure that what you do on a daily basis in your learning organization is always getting better.

The key metrics in this initial step focus on customer satisfaction and quality. Satisfaction with the basics such as instructor performance, online delivery effectiveness, courseware quality, and facilities conducive to learning are basic items that should be measured before you attempt to measure anything else. These are fairly easy to measure such as through end of training evaluations and if benchmarked internally and externally can be very valuable in monitoring quality of your learning organization.

While gathering satisfaction and quality indicators on a day-to-day basis, also gather predictors and indicators of some additional measures you might get more detailed on when you advance to the next step of measurement. Indicators regarding learning effectiveness, job impact, business results, and ROI can be gleaned through a standardized, scaleable, replicable and practical approach to data collection.

Asking the right predictors and indicators for these higher levels of learning (if one uses the Kirkpatrick taxonomy) and then benchmarks the indicators across dimensions such as course, program, curricula, location, business unit, instructor and learning delivery can yield a measurement system that is a predictive model to help you not only with historic results but in trending and predicting future training investments. Even a financial ROI can be predicted if you ask the right questions and perform the right math on them.

Day to day continuous improvement should not be a significant drain on your financial, physical and human resources. The system should leverage technology and standards to alert you to problems so corrective action can be taken before things get out of control.

Step Two: Deep impact ROI analysis

Once you have mastered the day-to-day business of learning you are ready to promote yourself to the world of ROI analysis that is more in depth than the indicators achieved in your daily metrics. This type of analysis is probably undertaken for 5 to 10% of all learning programs. However, when strategic or risk sensitive it makes sense.

In this step, you are drilling deeper to define the key business results that changed from the program. You are estimating their monetary value, isolating it to training and adjusting it for conservatism and confidence. Although similar techniques should be deployed in your daily metrics, this is significantly more in depth. For example, each participant would work with the learning organization to determine the measures, the changes to the measures, the value of the change and the training impact. Methods such as observation, time motion studies, control group analysis etc. could be used as tools to drill deep into this analysis.

The key with this solution is that you are still measuring with a solid sense of breadth across your balanced scorecard of measurement indicators, but the depth of the measurement exercise has increased.

In contrast to the day-to-day measures, this is a more complex and costly solution. Where the day-to-day solutions help forecast, predict and estimate using metrics that are roughly reasonable, this approach invests more measurement resources to get that 95% confidence in the data.

As a best practice, organizations should truly know when to move to this level of detail and only do so when justified. Recent studies have shown that executives tend to make decisions on reasonable indicators as opposed to data that was gained with a high cost of accuracy, so be careful here.

Step Three: Value Optimization

The third and final step in your roadmap to measurement is what we call value optimization. This goes beyond ROI. ROI in its classical sense is still historic in nature. It looks back on investments already made to determine if they were worth it. Certainly there are ROI calculators that can predict ROI before you invest in a training initiative, but this step takes this concept to an entirely different level.

Value optimization helps executives with data modeling. The concept of value optimization asserts that for each dollar you invest in training will yield a known increase in shareholder value as measured by increased operating margins and share price but also more tactical measures such as increased revenue, decreased cost, increased productivity etc.

Optimizing value means that if you want to achieve a desired change in a business result, a value optimization model tells you how much you need to invest in training and in other tools that drive the result. It is the inverse of historic ROI analysis that looks back instead of forward.

Let's use an example of a value optimization model. A company determines that it can increase operating margins if it can increase productivity by a defined percentage. They need to make investments to yield that level of productivity change. A value optimization model will provide the organization with the recommended training investment that should be made to achieve that productivity change.

So how does a value optimization model work? It uses historical data residing in various systems such as ERP, CRM, and MRP systems for starters. It then factors in assumptions and mathematics, that when combined with data from stakeholders (learners, managers etc.) helps to predict investment patterns required to achieve expected business results.

Sound complex? That's because it is. Integration between disparate systems isn't the hurdle here as one might think. In fact, when there is a need to integrate systems organizations tend to find a way. And, in this case there may not be a need as standard input templates can be used to collect the data that is then 'processed' to yield the outputs. The brainpower comes in knowing what the right data inputs to collect should be and how to then process the data to produce the desired outputs. No technology in the world by itself can do value optimization without solid methodology backing it up.

At this point in the evolution of learning analytics we are not quite at the stage where technology can achieve complete value optimization. But, we're getting there. Models have been formulated and methodologies have been written to do it. It's just a matter of time. Like its predecessors, customer satisfaction and quality and now ROI, have all been automated in a similar way. In these cases technology has wrapped itself around methodology to make measurement practical. So to will this take place for value optimization.

If you are a learning organization today looking to achieve a state of value optimization, it can still be done. What you need is expertise in three primary areas: measurement, learning, and technology. That combination can build a custom value optimization model that should achieve your desired results.

Conclusion

In today's world of learning analytics the majority of learning organizations collect basic smile sheet data and do a limited amount of analysis on this data. It is always best to take a crawl, walk, run approach to measurement before you over promise and under deliver to your stakeholders. Start small with automation of what you currently do to shift resources from administration to analysis. Build from there. In time you too can reach the nirvana state of value optimization. Dare to dream and set goals that are challenging yet achievable and you'll get there.

Reprinted from Workindex.com, Human Resources Executive

Jeffrey Berk, vice president, Products and Strategy, for KnowledgeAdvisors. Contact Jeffrey at jberk@knowledgeadvisors.com or go to www.knowledgeadvisors.com