1.  Training Assessments

Most training assessments occur principally after a training session and primarily reflect the participants' satisfaction with training/learning events, and, to a much lesser extent, the level of impact on organizational objectives.

However, now many organizations are using assessment help more and more to support the achievement of organizational purposes.  They are linking training and assessment to their strategic objectives and to self-managed competency standards that employees can upgrade.

As organizations tighten the belt, budgetary restrictions and a focus on results are increasingly forcing organizations to demonstrate the effectiveness of their training investments. Forward-looking organizations are viewing training/learning as a valuable investment to be monitored and linking assessment with organizational mission, goals, and objectives. In a knowledge age, intellectual capital is viewed as a means to acquiring the leading edge. organizations are striving to find simple, affordable, yet reliable, ways of measuring the results of their investments.

Unfortunately, much assessment data is inconsequential reaction data.  It can be costly and time-consuming to collect and has little value.  Also, there are additional challenges in insuring adequate assessment of "soft" training/learning.   In many cases, there is even a climate of weak management support for evaluation.  

Organizational decision-makers need to demonstrate whether organizational learning initiatives have achieved their desired outcomes.  However, most organizations still evaluate participants' reactions to training courses but rarely measure the impact of training on the job or on organizational objectives.

2.  Levels of Evaluation

The most well-known model of evaluation of training is the 4-level Kirkpatrick Model, which consists of four levels of assessment: Reaction, Learning, Behavior, and Results.

The Reaction level is where participants give their assessment of a training course or learning event, generally based on their level of satisfaction with the training/learning, typically at the end of the course or event.  These data are usually suspect because of what is called “social desirability” bias.  Just the fact that someone paid attention to them, often gives trainees a desire to report a high value associated with the training whether they ever use the skills or not.

At the Learning level, participants demonstrate what knowledge or skills they have acquired through achievement or on performance tests or exercises. It should be noted, however, that if this knowledge is not used soon on the job, a loss of learning usually occurs, as with any knowledge that is not applied.

The Behavior level is where the trainees show how much knowledge or skills have been transferred by their behavior on the job.  In some cases, the skills are different and often more advanced than the trainees’ managers.  This can cause resentment from some supervisors who may feel threatened by their subordinates.

Finally, at the Results level the trainers can assess the impact of the on-the-job changes on project/programme purpose or organizational objectives.  This is where organizational leaders can assess the Return on Investment (ROI).  This is the examination of the monetary value of the results with respect to the cost of the training/learning. 

training indicators

3.      Measuring Return on Investment

A 1998 survey by the American Society for Training and Development found that while the importance of training has increased in most of the organizations surveyed, fewer predicted their training budget would increase as well. Therefore, the only way to meet growing training demands without dramatically increasing your budget is through more effective training.

The most precise way to measure and improve training effectiveness is through a return on investment analysis of training programs. Because there isn't much incentive to conduct an ROI study if a training program will be approved without one, it is up to the management to require employees to justify a training program before it is budgeted.

Conducting an ROI of training calls for stringent goal setting and continuous analysis. Mathematically, ROI is a comparison of benefits to cost expressed as a percentage of the original investment. In simpler terms, it is a way of finding if the training met its goals to an extent that justifies its cost. An ROI study seeks to justify and improve a training program as well as to devise an overall training strategy. In fact, doing a training ROI is just a fancy way of doing cost-benefit analysis.

Not conducting an ROI analysis of training can cost your organization valuable input that is necessary for improving results. Just because a program succeeded at one organization, it won't necessarily work at another; ROI measures too many factors that are unique to any given organization. ROI analysis requires an in-depth understanding of your organization's strengths and weaknesses, strategies, and extensive goal setting.

 In addition ROI analysis takes time. Training programs that have a very long life cycle, are very expensive and have a high visibility are good candidates for ROI study.

4.      The ROI Process

Because ROI requires measurements before, during and after training, finding the ROI of a particular training program is not a discrete event but should be part of an ongoing process. The focus on pre-training measurements also helps align a program with the overall organizational strategy.

The first step is to identify needs and goals. Then you compare the existing workforce and the skills that they have to meet these goals. A well-planned training program should be designed to close the skills gap—and an ROI study is the only way to measure that.

To be certain that the training—and not random, external factors—is responsible for closing the gap, the effects of the training need to be isolated. The easiest way to do this

is to measure a behavior in a group of employees before training and again after training. For example, if exit surveys show that many employees leave due to dissatisfaction with the level of training, the organization can implement a skills development program. The ROI should measure the (hopeful) decrease in turnover that would represent a savings greater than the cost of the training.

As an example a organization has a turnover of 23%.  First you measure the average turnover rate and then calculate the cost of training. If the average employee wage is $35 per hour, and you conduct 80 hours of training for 100 people, it would cost $280,000. Add the costs of instructors, classrooms and other costs, which brings the total to $350,000. If you reduce turnover to 8 percent, you might save $453,000 in related costs for new hires. The training has saved you $103,000 after training without calculating any increase in efficiency from the training!

5.  Five Steps to Determining ROI

Here are the five steps the financial organization followed to determine the ROI of its training program:

1.  Measure the initial reaction to the training. This is the first and most basic level, usually involving a post-training survey assessing the quality of the class. You can ask the employees if the material was easy to follow, if the instructor was familiar with the subject matter, and if the facility was conducive to learning.

2.  Analyze the learning. There are a number of ways to do this, such as testing, simulations and instructor evaluations. What matters most is finding out if the employees actually absorbed the class content. Without positive feedback and good marks on the testing, a positive ROI is unlikely.

3.  Analyze the skills gained over the long term.  The goal of this step is to assess whether on-the-job behavior changed after the training. These data should be collected over a longer period of time. During this phase, data are also gathered on how participants have used their new skills. This skills analysis can yield valuable data on what worked—and what didn't—in a training program. If the employees are not using the skills, you need to find out why.

4.  Measure the project/programme purpose impact. This step of the ROI calculation determines the monetary value of the measured change.

5.  Analyze the actual ROI. This phase is really just another way to express the project/programme purpose impact, this time taking into account the program's cost. To find the ROI, you take the benefits and subtract the total cost, and then divide the result by the total cost. Multiplying this number by 100 yields the ROI percentage.

6. Measuring the ROI of Online Learning

One way many organizations have realized significant training cost savings is through online learning. Electronic learning accounts for only a small part of training today.  Currently the online learning market is estimated at $2.2 billion, but it is expected to grow to a staggering $11.4 billion by 2003.

General Electric is one organization pursuing for online training in a big way. In 1998, GE analyzed the efficiency of an orientation course aimed at GE Aircraft Engine beneficiaries that required them to travel to GE's Cincinnati headquarters. The course would last for three days and cost the organization $47 an hour in lost productivity per beneficiary and $1,500 for travel and living expenses for those [beneficiaries] flown in. In addition, the training facility needed to be upgraded at a cost of $4.5 million. Rather than do this, GE decided to implement an online version of the training.

An ROI of the online course revealed that employees were able to absorb the same amount of learning as in the onsite training and were able to do this in only three hours versus the three-day course. The organization was able to reduce the amount of time employees spent away from their jobs as well as save money on travel costs and training facilities upgrades.

In fact, the savings were so substantial that the GE CEO at the time, Jack Welch, initiated a organizational-wide plan to move 50 percent of all GE's training online by the end of 2001. At GE Aircraft Engines, the IT department was tasked with reducing its training budget by $100,000 by replacing traditional classroom training with online learning. Using technology is a key way to take cost out of the project/programme purpose.  Online learning eliminates days on airplanes and nights in hotels.  Online training is being looked at as a key job satisfier so that employees don’t miss time with their families.

7.      Approaches to Assessment

Organizations are challenged to devise better approaches to the assessment of training so that only essential evaluation information is obtained efficiently and with the least possible cost.

One response to this need is to seek "persuasive" rather than "conclusive" evidence of the impacts of training. Another response, consistent with the trend to give more responsibility for training to employees and their managers, is to place more reliance on self-administered assessment tools that are quick and simple to complete.

In some organizations, built-in indicators of training impact already exist: absenteeism, complaints, unit costs, items sold, etc. Of course, some training may not lend itself to the easy establishment of measurement indicators.

With the spreading use of computer-based training, it is possible to deliver the assessment components by computer. Software programs that compile results and produce simple statistical reports are becoming more accessible for individuals with little evaluation experience. As always, however, care must be taken to ensure that data are not compromised and that the interpretation of results is appropriate.

One promising avenue to explore is training investment analysis. This approach has been touted in some corners as assessment of training results before training. Training investment analysis gives a forecast of monetary benefits that are likely to be gained from training. A simple worksheet can be provided for calculating revenue that training generates. While this is a return-on-investment approach, it would be interesting to use the same principles to establish non-monetary benefits to be gained and to calculate these before training.  Of course, calculating these benefits are difficult and open to question.

8.      Trends in Assessment Practices

Let’s look at how training assessments are evolving by looking at current, emerging, and future practices.

Current assessment practices are mainly an assessment of reactions of the participants and some assessment of learning.  The main beneficiaries of the assessment information are training departments.  The assessments are usually linked to training course objectives which are focused on prescribed needs.  The training is process-oriented, and the assessments are to provide feedback on the quality of the training experience.

Emergent assessment practices are more concerned with assessing performance on the job and its impact on project/programme purpose.  The main beneficiaries of the assessment information are managers and those charged with directing project/programme purpose, although all employees also have interest.  The assessments are linked to the project/programme purpose plan. They show if the training was results-oriented and their major concern is with improving overall project/programme purpose performance and the bottom line.

Future assessment practices will most likely assess the employees’ competency for future organizational requirements. The benefits of this information should be disseminated throughout the organization. The assessments will be linked to the organization’s anticipated future needs and will be focused on competencies that permit organizational adaptation and change.  These assessments will reflect the organization’s future-orientation by anticipating future competencies/requirements.

Increasingly, assessment is viewed as a tool to assist management in its decision-making and in monitoring the implementation of change.  However, many managers hesitate using assessments for fear their programs will be shown to have low ROIs.

9. Training as a Strategic Investment

Training is a tool to effect strategic change and a means to achieve a effective edge.  Once organizations become aware of the directions that they need to take to survive or to grow, the shaping of the strategy to achieve the desired directions may involve considerable retooling. With such an important investment, organizations see the value of monitoring how the investment is paying off. Increasingly, training is seen as an investment that will provide beneficial returns, rather than a cost to be borne.

High-performing organizations focus on training as a strategic objective. Training and development is the most frequently measured of all the HR functions, another indication that decision-makers are paying attention to the value that this function brings.

Assessment will focus more on "intangibles", such as the impact on organizations of developing intellectual capital. What people and organizations know is becoming ever more important.

While it is evident that the measurement will be complex and varied from one organization to another, it is important that knowledge capital start to have the same status as that of hard assets. With more and more emphasis on organizations as providers of information, knowledge and ideas aided by information technology, these "soft" intellectual products, difficult to capture though they may be, will need to be assessed and evaluated as important outcomes of programs.

It will be Training and Development’s responsibility to work with other organizational members to develop assessments that can provide relevant and accurate ROIs for all types of learning activities, whether they are considered “hard” or “soft.”

10.  Basic Suggestions for Evaluating Training – Closing Thoughts

Typically, evaluators look for validity, accuracy and reliability in their evaluations. However, these goals may require more time, people and money than the organization has. Evaluators are also looking for evaluation approaches that are practical and relevant.

Training and development activities can be evaluated before, during and after implementing the training.  Here are some things to consider.

Before implementing ask: will the selected training and development methods really result in the employee's learning the knowledge and skills needed to perform the task or carry out the role? Have other employees used the methods and been successful?  Consider applying the methods to a highly skilled employee prior to conducting a training session and get their impressions of the methods. Do the methods conform to the anticipated trainees’ preferences and learning styles? Do you think the trainees will experience any difficulties understanding the methods?

During implementation ask the employee how they're doing. Do they understand what's being said?  Periodically conduct short tests.  For example, have the employees explain the main points of what was just described to them.  Is the employee enthusiastically taking part in the activities? Is he or she coming late and leaving early. It's surprising how often learners will leave a course or workshop and immediately complain that it was a complete waste of their time.

After completion of the training give the learner a test before and after the training and compare the results. Interview the learner before and after and compare results. Watch the learner perform the task or conduct the role or assign an expert evaluator from inside or outside the organization to evaluate the learner's new knowledge and skills.

Template for evaluation of training

See also:  training diagramme

Assignments

Matching the Columns

Answers:

1.)    D

2.)    A

3.)    C

4.)    E

5.)    B

6.)    F

Multiple-Choice

1.         Assessment is viewed as a tool to assist management in its:

a.       Decision making

b.      Monitoring the implementation of change

c.       Both a and b

d.      None of the above

2.         The benefits of the information should be disseminated throughout the organization.

a.       Current assessment practices

b.       Emerging assessment practices

c.       Future assessment practices

d.      None of the above

3.         High performing organizations focus on training as a ________.

a.      Strategic objective

b.      HR function

c.       Intangible

d.      All of the above

4.         During T&D, evaluations are used to determine if the employees

            a.   Understand what’s being said.

            b.   Are enthusiastically taking part in the activities.    

            c.   Are coming late or leaving early.

            d.   All of the above