MET:Approaches to Corporate Training Assessment

This page originally authored by Alex Monegro in March 2014.
This page was revised by Stephanie McGinnis in January 2016.

In 2012 corporations spent $164 billion on learning and development programs, or about $1,195 per employee . With such a staggering amount being spent, it is concerning that very often these efforts are not evaluated to understand if they have delivered value to the business. While the number varies by initiative type and industry, up to 80% of sales training is not assessed for Return on Investment (ROI). There is increasingly however a strong trend towards accountability across business functions, creating pressures for each to demonstrate their value to the business . This trend has increased the importance for training professionals to demonstrate the value of their efforts to the business. The lack of assessment has been driven traditionally by what can be summarized as a lack of resources and organizational support, a serious of myths about it's lack of usefulness to the business and managers. This Wikipedia entry will attempt to provide an introduction to the challenges faced, and approaches used in corporate training and development.

Challenges to Corporate Training Assessment

Tan and Newman paint a picture of some of the structural challenges that have limited assessment of corporate training historically. Firstly, they suggest that starting at the end of the 1970s, training department budgets were consistently decreased and many of the functions of those departments were outsourced. Secondly, that the academic world widely criticized many efforts at creating practical assessment frameworks, but did not provide practical guidance. Thirdly, they point to a lack of resources. They point to research suggesting that only 11% of the effort and 8.5% of the budget of training initiatives are allocated to evaluation purposes. Lack of money and time was frequently cited as a reason for managers not to perform training. Lastly, they point to manager resistance. Five reasons commonly cited by managers for not assessing training initiatives include :

1. The idea that a successful company implies that the training is working
2. A belief that any training is good training
3. The potential threat negative assessment results might pose to the job security of those involved
4. Manager's belief that all responsibility for assessment falls on the training team, not on them
5. The belief the because assessing learning is very complex, and it might not prove anything, there is no reason to spend money or time on assessing

Lastly, the American Society for Training and Development (ASTD) argues that the large number of theories and models for assessing corporate training are a challenge in itself. They argue that since Kirkpatrick's typology (discussed below), '25-plus models and theories for evaluation offered to practitioners', plus dozens of general book, and a dozen more field specific book (e.g. social sciences focused, government focused, etc...) have potentially contributed to confusion and hesitation to assess in the corporate world.

Theories and Typologies

One of the main tensions in corporate training assessment is the dynamic that practitioners tend to eschew robust theories for easier to implement typologies. This means that while there are theories of corporate assessment that provide reliable and valid assessments, they tend to have lower adoption rates. While there has been some progress in establishing more robust typologies, and best practices that increase the reliability and validity of the assessments performed, most corporate training practitioners are still using less robust methodologies such as the pervasive four levels of assessment developed by Kirkpatrick in the 1950s. We will focus in this wiki entry on the currently most commonly used methods in assessing corporate training.

Approaches to Corporate Training Assessment

Kirkpatrick's 4 Levels

In 1976, Donald Kirkpatrick published his four level assessment typology. It has been the most popular tool for corporate training assessment for the last four decades. Kirkpatrick proposed that we could assess training by focusing on four different levels of evaluations which would give us a complete picture of how effective and efficient the training intervention or program had been. The four levels are:

1. Reaction - participants' reaction to the training
2. Learning - participants' learning as a result of the training
3. Behaviour - change in behavior as a result of the training
4. Results - subsequent impact on the organization as a result of participants' behavior change

Kirkpatrick believed there were three reasons you might want to evaluate training programs:

• To justify the existence of a training function by showing how it contributes to organizational goals and objectives
• To decide whether to continue a training program
• To improve training.

Critiques/Limitations

Kirkpatrick's four levels have been widely critiqued in the literature . One of the strongest critiques comes from Holton in that Kirkpatrick's four levels isn't truly a framework, as elements in a framework should be casually related which has led to it being considered a typology for assessments rather than a framework: categories to be assessed, not a way to assess. Other criticisms include:

• A failure to adapt the typology to changes in the business environment, mainly a change in the type of work being performed, and the breadth and complexity of training being delivered. While these have changed massively since the publication of the four levels, the definition of the four levels has remained more or less constant
• It's focus on summative evaluations, to the detriment of formative ones. However, this has been addressed by others by noting that each level could be assessed with either summative or formative evaluations if the practitioner so wished
• A lack of focus on the economic benefits of training.
• A lack of guidance as to what measures are most appropriate for each level. Alliger et al's research suggest that the measures that correlate the most with knowledge transfer are utility type measures, in contrast with affective type measures which seemed to be less predictive of knowledge transfer

Many extensions to the original typology have emerged addressing these issues, including Brinkerhoff's which incorporates two formative evaluation levels , and Phillip's which incorporates Return on Investment as a fifth level.

Phillip's Fifth Level

In 1996, Jack J. Phillips argued that Kirkpatrick's original model did not lend itself to quantifying the return yielded by training initiatives. He argued that while the fourth level would yield information about what results that training led to, for example lower absenteeism, it did not encourage practitioners to take the next level and translate those results into the monetary benefit yielded . His solution was to add a fifth level to the Kirkpatrick typology that would focus explicitly on measuring monetary value in the form of the popular business metric, Return on Investment (ROI).

Guidelines for Calculating ROI

ROI is traditionally defined as ( net benefits / net costs ) x 100.

In their book, Project Management ROI, Phillips et al outline 12 guidelines for calculating ROI. Some of these directly address some common mistakes include myths about not being able to isolate the impact of an intervention, and not using fully loaded costs.

• When conducting an evaluation at higher levels, collect data at lower levels.
• When analyzing data, select the most conservative alternatives for calculation.
• Use at least one method to isolate the effect of the intervention or program. Suggested methods include using control groups, trend line analysis, and using estimates from different sources such as the learners themselves, staff experts, and customers.
• Avoid the use of extreme data and unsupported claims when calculating ROI.
• Fully load all costs of a solution, project or program when analyzing ROI, where fully loaded costs are defined as including any and all costs, direct or indirect; When in doubt about a cost, he suggests you include it in your calculation.
• Use standard values for your impact assessment and ROI calculations, where a standard value is the official dollar value used to measure the metric you are using to calculate the benefits. Most standard values are internally determined by the relevant department in your organization. When standard values are not available different methods can be used to estimate them, including: using recorded historical values, using inputs from internal or external experts on setting a value, using values from external databases, using other measures as proxies, or using estimates from learners or the management team.

As a rule thumb, Phillips et al suggest that when attempting to isolate effects, or to estimate standard values, using multiple methods might lead to a more decreased error, and a more accurate estimate.

Assessment embedded in instructional design theories

Many instructional design frameworks have embedded assessment as part of the instructional design process. This has led to a variety of approaches being used, as each framework would advocate for a specific methodology. A popular instructional design framework using in the corporate world that embeds assessment into it's process is the ADDIE framework. ADDIE stands for Analyze, Design, Develop, Implement, and Evaluate. The Evaluate step in the ADDIE framework uses a modified Kirkpatrick approach, that includes Phillip's fifth level. It divides assessment activities into three categories: Perception, Learning, and Performance.

Reporting Assessments

In the The Business of Learning, Shlomo argues that just as important as assessing the impact of training in the corporate setting, is communicating that impact to stakeholders across the organization. He suggests that as executives' main concern is seeing impact from training efforts, if the impact data is poorly communicated to stakeholders, decisions will be made without the right information, the assessment effort will be for naught, and training programs might be negatively impacted.

Balance Scorecard Approach

One approach to tracking and reporting assessments of corporate initiatives is the Balance Scorecard developed by Kaplan and Norton in the early 1990s. While mainly a strategy performance tool, it has been adapted to be used in the corporate training environment by Pangarkar in his book the Trainer's Balance Scorecard. Scorecards are a powerful way to tie the impact of training to the organizations overall strategy. In this way, the link between the impact of the training and how it advances the organization's goals is clear There are four parts to a scorecard:

1. Objectives: "Where do we want to be?"
2. Measures: "How will we measure what we're doing?"
3. Targets: "How will we know when we arrive?"
4. Initiatives: "What are we going to do to get to where we want to be?"

Talent Development and Reporting Initiative

File:TDR Process.png

Process suggested by the Center for Talent Development Reporting to measure and communicate impact of learning and development initiatives.

The Center for Talent Development and Reporting was formed in 2012 as an industry led initiative that seeks to establish reporting principles for Learning and Development (L&D) initiatives in organizations that mimic the Generally Accepted Accounting Standards used by professional accountants. In 2012 they published a whitepaper outlining four recommendations to help learning and development practitioners moved towards a standard reporting format.

1. Organizations should adopt a set of guiding principles that will inform their assessment efforts.
2. Organizations should adopt standard definitions and calculations for key metrics.
3. Organizations should adopt three standard reporting statements to be issued at in line with the regular reporting cycle of organizations. The three reports are: an outcome statement, an effectiveness statement, and an efficiency statement. Each report is mostly mutually exclusive to the others.
4. Organizations should adopt three levels of executive reports that mimic standard reporting elsewhere in the organizations. These reports are: a monthly L&D program report that focuses on effectiveness, a monthly L&D operations report that focuses on efficiency, and a quarterly L&D summary report that draws on outcomes, effectiveness and efficiency. The first two reports would be for viewing by the L&D department, while the latter would be targeted at corporation executives.

Visual Artifacts and Stop Motion

https://youtu.be/7B0bbCe69n8 2016 by Stephanie McGinnis. Duration: 5:58. More information about Kirkpatrick's 4 Levels and ideas on how to assess each of the levels.

See Also

• Assessment
• Corporate E-Learning
• Learning Analytics
• The Learning Organization

External Resources

• Balance Scorecard Usage Cases
• Phillip's ROI Institute Brochure
• ASTD Publications
• Center for Talent Reporting