The E Word – “Evaluation”

Blogging really helps me think out and frame how to explain my strategies. Right now, I’m trying to work out the details of my training Evaluation strategy for the project I’m working on.  I will focus on the first three levels for this project.  We don’t have a lot of time to be able to perform the fourth level of evaluation.

Level 1 =  Smile Sheets.  Questions/focus areas: Was the training good? Was it appealing? Was the delivery good (for eLearning did it work, were you able to navigate through it) Did you feel like you learned from the training?  Note: you can actually do mini-smiley sheets for individual pieces of learning content (example: modular learning objects in a chunked elearning).

Level 2 = Knowledge Testing. Did you learn the knoweldge/concepts/tasks built around the learning objectives

Level 3 = Performance. Are you performing the behaviors/tasks from the training in your job successfully?

I used the table below to explain the evaluation types and methods to my stakeholders (without the unicorn of course)* 

Click on the image to view a larger image of the table

I firmly believe you cannot test people’s learning on the 2nd and 3rd level, unless you’ve written out the tasks/concepts and sometimes beliefs that were needed to make sure that new performance behaviors were being accomplished. I’ve seen a lot of training programs that were built just around the content (tool or system design) and the tasks (in our business this means the transactions you can perform in the tool).  This is okay if you just want them to learn how to do the transactions separately, but not as part of a working business process to get business goals accomplished. There’s something very haphazard about developing training materials without objectives (and yet I see it happening all the time – sometimes because people don’t have a lot of time on a project or training developers were not engaged early enough).

Explaining the importance of developing performance (learning) objectives continues to be a regular challenge when working with content experts and stakeholders to develop training.   But I always try to couch what I’m trying to accomplish as a training developer by noting that in the end we’re just trying to get the same goal accomplished: We want to the user to be able to do his or her job well and with as little or no issues as possible.   It also helps to have a well organized template for interviewing content experts about the tasks and how they fit into a working business process. After the implementation of the training, I’ve rolled out “Level 3” survey data out to my customers as part of a post-mortem presentation to show them how the training did effect successful performance of tasks.

Addendum: How to check if the learning stuck

For the Level 3 evaluation along with the manager/user survey, I’m going to try a “Check” with the Super Users for the project/tool.   Super User is the term used for trained experts in the field.  These folks are hand-picked and in positions which do not rotate as frequently and are chosen by group leaders and managers for their mentoring/coaching skills. Sometimes we train the Super User as instructors for the formal stand-up training. 

I would like to include them in addition to the managers as part of a survey/assessment of the training’s success because As part of their training, I think I will include a watch list for user adoption/performance success. This would include performance behaviors tied to a handful of the critical high-level learning objectives. A question on the checklist might look something like this:

  • Are they performing task X in a timely fashion?

I would use the Super Users as a resource for determining this information in addition to the managers, because, honestly, as the SU’s are working in the field they may have a more accurate representation of what’s happening and be able to provide more insight on what to do if it’s not happening.

* Note: I have never seen this “Fifth”  level of evaluation happen. To me this takes a great deal of time to assess, and honestly may not be worth the Return on Investment (ROI).  I’ve seen a few articles out there about the validity of performing all five levels of evaluation.

Nuggets or resources:

  • Kirkpatrick’s evaluation theory
  • How to Measure Training Results: A Practical Guide to Tracking the Six Key Indicators.  by Jack J. Philips and Ron Drew Stone, McGraw-Hill (c) 2002
  • Achieving Results from Training. Robert Brinkerhoff, Jossey-Bass (c) 1988
Advertisements

0 Responses to “The E Word – “Evaluation””



  1. Leave a Comment

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Why?

My place outside of work to explore and make connections with the ideas and things (sometimes work-related) that I'm passionate about.

My Tweets

Blog Stats

  • 277,930 hits

%d bloggers like this: