Level 6: They Might Say You Can Do It, But What the Heck Is It Really?

A friend recently sent me a link to the article, “Take Your ROI to Level 6: Get Greater Return on Your Training Dollars by Comparing Web-Based, Instructor-Led, and Blended Delivery Approaches [highlighting in the title used in the magazine, too] in the web-zine, Training Industry Quarterly, Paul Leone of American Express claims to have evaluated a training program at Level 6.

My friend added, “This article makes me wish I was a master plumber.”

For starters, what the heck is Level 6? To be honest, I’ve always had issues with calling ROI Level 5. Impact is impact—whether it be measured financially (ROI) or measured with some other performance metric. That sounds like Level 4 to me.

Besides, years ago, Roger Kaufman proposed that Level 5 referred to societal impact. Although I don’t know how one would measure societal impact, that it was looking for an impact beyond that of the immediate organization seemed to build on the progression that is inherent in Kirkpatrick’s model—that is, that each level assess something that is progressively larger and has a wider impact.

With Level 5 already claimed and claimed again, I guess it was only a matter of time before someone proposed a Level 6.

But the Level 6 proposed by Leone has three significant problems. First, rather than assess something bigger than organizational impact, his Level 6 measures the transfer climate. More specifically, it seems to measure characteristics of the environment that foster the transfer of learning.

Which raises the second problem with it: why isn’t this considered part of the Level 3 assessment, as Level 3 assesses transfer?

More significantly, the assessment doesn’t consider the training program itself, it merely looks characteristics of the work environment that promote or inhibit transfer. Although this is extremely important information, it isn’t Level 6.

In fact, it’s not even the type of question that’s best considered in a summative evaluation. In an academic context, this is a basic research question. In a corporate context, it’s a question that’s better considered for a formative evaluation, while the training program is still under development. By identifying which characteristics foster transfer during the formative evaluation, instructional designers can design them into the “production” version of the course to ensure the highest possible transfer of learning.

In fact, that the study reports on the use of a summative evaluation (which is supposed to be used on a validated training course) to conduct a formative evaluation (such as the comparison of different types of training programs, to determine which version to prefer) seems a bit problematic.

On the one hand, it's great that the author reports on the comparative effectiveness of different delivery modes and the environmental factors that promote and inhibit training. The field needs more of these field-based studies. But on the other hand, it's lousy that the author mislabeled the nature of evaluation and didn't use terminology correctly.

Because he didn’t use terminology correctly. That's where the editor is supposed to come in, but the editor of Training Industry Quarterly failed to exercise appropriate editorial authority and correct the problem to avoid confusing the reading public.

But no one did anything to correct this significant error in terminology. This failure to use terminology correctly is not some abstract problem raised by some out-of-touch academic.

It’s a real problem. At the least, the misuse of terminology causes confusion and some professor will need to un-teach an over-eager student one day, probably after that student insists on performing an ill-guided thesis and has the proposal rejected (or worse, the final thesis) because of essential theoretical flaws. That the student saw this in print won't help the case.

At the most, should someone act on this error in terminology in front of an executive who really DOES know his or her terminology, the person will have made a fool of themselves in the moment, and may have dealt their career a serious blow for years to come.