Many people need to maintain hours to retain their certification and those hours have moved to online training.  Even more, virtual workshops, lectures and social distancing measures have increased as more people are forced to work from home.

How does a training manager measure the effectiveness of their training? There are significant obstacles. Most notably competing priorities, lack of data, or simply not knowing where to start. Beyond that, what type of assessments should be used to show that the training was effective?

It isn’t always possible to quantify an immediate, or direct ROI in business training. So how can instructors, managers and trainers go about creating a plan to justify and show training effectiveness?

Show Notes:

Links and Content Cited in this Podcast:

Kirkpatrick’s Model of Training Evaluation

[Article] Survey- Measuring the Business Impact of Learning

[Article] The New World Kirkpatrick Model

[ Article] Kirkpatrick’s Four Level Evaluation Model

[Article] 3 Ways to Measure Training Effectiveness– John Eades

From Lies about Learning, Larry Israelite (ed.)

The typical corporate learning values meeting business objectives, but puts little resources into meeting business objectives or measuring them.

Transcript

Michael Stebbins: [00:00:00] And if we can go through and say, does the person who completed the training have this skill now, or have they shown that they can define viable repositories for leading customer data for a single view customer? Yes or no?

Bumper Intro-Outro: [00:00:20] Welcome to endless coffee cup, a regular discussion of marketing news, culture and media for our complex digital lifestyle.

Join Matt Bailey as he engages in conversation to find insights beyond the latest headlines and deeper understanding for those involved in marketing. Grab a cup of coffee, have a seat, and thanks for joining.

Matt Bailey: [00:00:44] To the endless coffee cup podcast, and if you’re like me, you are sheltering in place. And so this is going to be our, our our Corona virus additions. I would say because a, the podcast is going to come out much more frequently over the next few weeks, because most of the people that I have regularly, or are people that I want have been wanting to bring on the podcast, they’re all home too.

So it gives us a great opportunity to really talk to some smart people and find out what’s going on beyond the hype. And today I have a regular guest Mike Stebbins from OMCP. Mike, how are you doing?

Michael Stebbins: [00:01:22] I’m great and happy to be here, man. Thanks.

Matt Bailey: [00:01:25] Oh, fantastic. Well, Mike and I were talking a little bit offline and if you’ve listened to any of our previous podcasts, they usually surround training as a, the OMCP is a certification body for digital marketing.

Anything you want to add in there Mike?

Michael Stebbins: [00:01:42] Accrediting body, certification body. We don’t do training. We just evaluate it.

Matt Bailey: [00:01:47] Great. One thing we were talking about is especially now with the massive shift in how people work. What I’m seeing is an incredible increase in now a demand for virtual workshops, virtual training.

And then now I see more and more articles also talking about, well, we need to move more education online. We need to make more opportunities for people to train online. And while part of me is excited about that, there’s also a part of me that sees how training is done online now. And I’m not exactly thrilled because there’s still a lot of, I’m trying to think of a better way to say this old thinking that; I talked to someone the other day and they told me for their job requirement, they had to watch a certain number of hours of training and it was training videos, and they said it was obvious that the trainer didn’t know the content because they were stumbling over terms and acronyms that were common in the industry.

And it was just obvious they didn’t know what they were and they were reading a script and it was just completely boring. And unfortunately that’s kind of the, I would say par for the course when it comes to online training and in a lot of cases.

Michael Stebbins: [00:03:11] And when, when a trainer uses the phrase, “the Google,” that’s your, that’s your first warning.

Matt Bailey: [00:03:18] You mean it not in a humorous way.

Michael Stebbins: [00:03:20] Exactly. Yeah, exactly. Either they’re in the know of the joke or they’re serious. If they’re serious, you’re in trouble.

Matt Bailey: [00:03:28] Yes. Yeah. I have been known to throw out the Twitter just as a, but yeah. Yeah. You got to know if they’re they’re dead serious then, yeah, there’s a problem.

But it’s interesting, Mike, that somehow, hours watched has turned into a measurement of some goal. It just fascinates me. And this is what I wanted to talk with you about is, is how do you effectively measure the impact of learning?

Michael Stebbins: [00:03:56] Right. And you know, Matt, your audience may not know that it’s been 13 years, that you and I have been working together specifically on online training.

And I think you would come off as an unbiased in terms of online versus in-person because you do both. And just to point out the efficiency of you know, hours watched, you know, at our gym down the road, which of course is closed now, they, they hire a repair guy to keep the equipment in shape, like the treadmills and the stair steppers and the cycles and things like that.

And it’s become quite entertaining for us all to watch this guy come in because he rolls all of his parts and equipment and tools in up against the wall downstairs. And then he goes upstairs with maybe one tool and starts working on one of the machines. And then he slowly comes down the stairs and picks up one part.

Matt Bailey: [00:05:00] Oh no.

Michael Stebbins: [00:05:01] And it goes up and he puts it down and does something. And then he comes down the stairs and picks up another tool. And we’re all looking, of course, obviously the efficient thing to do would be to take his toolbox up there, you know, and then he’d be able to be much more efficient, but we all came to the…

Matt Bailey: [00:05:16] Oh, absolutely.

Michael Stebbins: [00:05:16] …that he was paid by the hour, you know, watching training by the hour, just that analogy comes to mind. Second point, Matt, and using you as an example, the average didactic requirement for a course that teaches to OMCA standards used to be 26 hours.

Matt Bailey: [00:05:38] Well, yeah.

Michael Stebbins: [00:05:39] And you ruined that.

Matt Bailey: [00:05:41] I ruined that. And it’s good to know. I broke something?

Michael Stebbins: [00:05:46] You broke, you broke the competency standards requirement for didactic hours because your course on LinkedIn, the OMCA preparatory course comes in at under four hours period in time.

And we, and the academic committee here at OMCP didn’t know what to do with that.

Matt Bailey: [00:06:08] Oh.

Michael Stebbins: [00:06:08] ‘Cause the requirement was 26 hours, however, we know that it, it was purpose built by you and LinkedIn to prepare. Okay. So now we have this, this problem. Could a course that is taught in less than four hours convey the same knowledge requirement that in the past has required 26 hours or more?

There was two issues. Number one was, did it have the academic integrity that was necessary to be approved and accredited. And the other issue was raised by our customers, by people who use OMCP as a standard within their company and require there…

Matt Bailey: [00:06:44] Yeah.

Michael Stebbins: [00:06:45] …to know this. So we took it to them. They, the committees always check in with people who are using this and the feedback came back.

Even if it is integral academically, it will hurt your reputation. And you really only need four hours to prepare for such a significant exam. And Matt, we, we struggled and I don’t know if you knew this whole conversation was gonna go…

Matt Bailey: [00:07:09] No, no, this is…

Michael Stebbins: [00:07:11] …just doing a great job teaching, but it’s like…

Matt Bailey: [00:07:14] I’m kind of in shock here. I’m like, Ooh, I didn’t mean to do that, Mike. I mean, if it were my choice, if it were my choice, this would have been much longer, you know, we had to constrain to a typical LinkedIn format and, you know, ideally. Well, I mean, just to give you a sense right now, I I’m, I’m looking to develop a, a coach OMCP course you know, full of, you know, activities, office hours, feedback assessments you know, and I’m looking at each course like just your SEO is going to be 30 hours. So…

Michael Stebbins: [00:07:55] Here’s what we did with your course. So, I mean, there was a strong argument not to approve it and, you know, we were kind of at an impasse for a little bit. And so here’s what we decided to do. We said we can get by on the technicality that in your course, you point to additional material that when all consumed exceeds 26 hours, and you give people the option to do so.

So that would be available to them. So on that technicality, we told LinkedIn, we’re going to give this a one-year trial and see how the pass rate is. And we don’t advertise what our pass rates are. We allow the vendor to know that and to choose whether or not they amplify that, but I can tell you that the pass rate on the LinkedIn OMCA course that is less than four hours is phenomenal.

Matt Bailey: [00:08:50] Wow.

Michael Stebbins: [00:08:51] It’s phenomenal. It’s one of the best that we see and folks come highly prepared. Now we think that there’s a couple of reasons for that. Number one is the quality of what you put together. Well done.

Matt Bailey: [00:09:01] Thank you.

Michael Stebbins: [00:09:03] The…

Matt Bailey: [00:09:03] Dear listener. This is breaking news and news to me as well. The, the OMCA Prep Course.

Michael Stebbins: [00:09:12] We’re talking about hours watched and I’m trying to make a case that it’s, it’s just not relevant when you have somebody who can deliver efficiently, which you clearly did. And the second issue, which may not relate to our argument here as much, is that a lot of the terminology and principles that you teach are in a context that is much more accepted and known now than it was three years ago.

Matt Bailey: [00:09:42] Yeah.

Michael Stebbins: [00:09:43] Okay. And you can use a phrase like keyword research and people will know that it applies to both SEO and paid search or even content marketing. Whereas use that term two or three years ago, you would have to take the time to explain what it is. We think that…

Matt Bailey: [00:09:56] Absolutely.

Michael Stebbins: [00:09:57] … needed to some of the efficiency is that your market is a little bit more educated as they come in.

[00:10:00] But the factors, the first one especially, just removes that whole hours watched. That’s like taking one part at a time up to fix the treadmill.

Matt Bailey: [00:10:16] Well, and that’s one thing that I, I’ve really started to see. I, I think a lot about how we delivered training 10 years ago, especially even in a workshop context, it wasn’t really a workshop.

It was a four hour lecture. Now I felt like we could get away with it because what we were teaching, SEO analytics was so stinking new, that people were just hungry for it. And you could get away with lecturing if you were a good lecturer teacher because people just wanted the knowledge and it, and it seemed like they just didn’t care how they got it.

But now that people have gotten a little more savvy, they understand a little more about it. They’ve also become a bit more demanding. And now I see this, especially in the digital marketing educations side, there has to be, you know, we’ve got to start adopting some of the educational techniques, you know, the activities that, are people doing what you’re instructing them to do?

Are they being assessed on it? And what kind of feedback are you giving them so that they can do it better? You know, and, and spending less time on lecture, more time on activity. Those are just some of the things that I, I’ve just learned over the past few years. And I would say, you know, it’s just such, if I compare it to what I was doing just, you know, six years ago, it is a, a vast difference in just the approach now to training, because I want to be sure people are, they’re getting this real world experience that they can turn around and use as soon as they, before they even get out of the room. How’s that?

It’s that’s yeah, that’s, that’s really my goal. And that’s where I see highly effective trainers, they’re focusing more on enabling people and guiding them rather than just standing there talking all day.

Michael Stebbins: [00:12:22] And only a few quality trainers take the time to implement because you know, it’s hard work to teach them and guide people through. I’m going to name two names of people who are your colleagues in the early days at market motive who are particularly skilled at that.

And we’re the only ones who kept teaching that way, even though the format of workshops in our training company back then start falling into more of a lecture for convenience because otherwise the preparation was long. So two people, one of them, Brad Gettys who we know and love, Brad was particularly skilled at taking people through in his paid search workshops at having something tangible when they were done.

And so, you know, in cooperation with us, we would say, look, you need to have a spreadsheet ready to go in that days, you know, ad-words open and we’re going to create a campaign in an ad group together based on the research that we do in this spreadsheet. So when we’re done, you’ll be ready.

Matt Bailey: [00:13:30] Exactly. Yeah.

Michael Stebbins: [00:13:31] That took a lot of prep time on Brad’s part.

The other, who we both know and love, is Avinash Kaushik. And, Avinash, I don’t know if he did the prep or not, but he was so darn inspiring. He was so excited about what he was sharing that you were just compelled to go do it while he was teaching it. And so we had great feedback that, that kind of level of inspiration and knowledge really drove people to do, even during the workshop, and learn from it. And so those, those formats definitely stood out from the others as being something that left somebody with a good memory. And I know you do that now in your life in in-person workshops.

Matt Bailey: [00:14:14] Well, and that’s a challenge is, you know, like I said, 6 years ago, 10 years ago, if you were engaging enough, you could get away with it.

And part of that now is, and, and I do see a lot of very skilled trainers that I worked with 10 years ago, still doing the same thing, that they’re defaulting to the 2 or 3 or 4 hour lecture. And that’s not engaging anymore. That, and really what it comes down to is it’s the list of what I think you should know, rather than what is essential to your job.

Michael Stebbins: [00:14:54] Well the list doesn’t change if you do that and you get to go give the same talk over and over again and all the conference circuit and your, you know, in-town presentations and it’s easier.

Matt Bailey: [00:15:07] Yeah.

Michael Stebbins: [00:15:07] It’s easier for the trainer.

Matt Bailey: [00:15:09] It is. And, and now, I mean, you know, developing courses and now, especially doing virtual workshops and coming up with virtual activities for people to work on, and then also discuss those virtual activities and being able to provide feedback.

I mean, the challenge and the prep and the, I mean, this is, it’s almost more than doing it in person. Doing a virtual. The amount of moving parts that to take place is, is incredible.

Michael Stebbins: [00:15:42] And I know Matt, you also do it in the context of the business goals, which is, that’s like, you know, going across a tight rope while on fire.

Matt Bailey: [00:15:51] Well, I know I’m glad you brought that up because this is why I wanted to talk to you. I’m glad you’re here. This is what I wanted to tell you. So this happened the other day I was doing an in-person delivery and afterwards they prompted everyone to go fill out the survey, which happens all the time.

And I received the survey. And Mike it’s, it’s a vanilla survey. How how valuable was the training? How relevant was the information? How good was the instructor? How would you rate this? Would you recommend it? You know, and I’m like, well, wait a minute. There’s nothing in this evaluation about, do I have something I can take away right now and implement?

Did I learn something that will improve my job? What was it? There’s no feedback as far as specific content or, you know, there was no tangible connection to a business objective.

Michael Stebbins: [00:16:53] Which is, which is hard to do.

Matt Bailey: [00:16:55] Yeah. But then, and then I sent this to you and that’s what prompted this. I, I found this research chart about effective training and how there is a, a high value on business linkage.

However, It is one of the lowest areas of investment in developing a training program. Yeah. Measurement of the effectiveness of the training is a medium value with again, little to no investment in measurement. And that just blew my mind, but yet I’ve seen it time and time again. What is the lowest value, but receives the highest investment is the design, the development, and the delivery of the content.

Michael Stebbins: [00:17:51] Yeah. Going back to Kirkpatrick’s from 1970.

Matt Bailey: [00:17:55] Absolutely.

Michael Stebbins: [00:17:57] You know, and, and it’s, so whoever is doing that is being dutiful and probably doing what they learned in their degree program. And so I, I don’t denigrate it. Right. I mean, it’s at least they’re trying.

Matt Bailey: [00:18:08] But I do like Kirkpatrick’s if you follow it the way it was intended and it’s Kirkpatrick’s four levels of evaluation and number four is what did it change? I love that. What did it change? Did it change anything?

Michael Stebbins: [00:18:22] But the isolation possibilities for his fourth level though, I think are too difficult to do quite honestly. And I know it’s controversial and as I’ve been working with teams to help them evaluate the training results, it’s too lost in the noise, unless you have a control group who doesn’t do it and you have specific measurement criteria. How do you know, you know?

Matt Bailey: [00:18:49] Yeah. Well, and the fourth level for the, if you listeners the fourth level I have, I’m going to, I’ll put the four levels of evaluation on the, on the notes here. But it’s, it’s really the ROI calculation. It’s, and that’s where you need follow up. You need to know, okay, six weeks later are, has this changed people’s behavior? Has this changed how they approach things? Had they seen differences in return on ad spend in efficiency in campaign value?

And unfortunately that, like you said, it’s near impossible to track that. And then how do you track that isolated from other factors?

Michael Stebbins: [00:19:36] Usually with a control group, but you’ve got to have a big organization to do that.

[00:20:00] Like for example, I’m working with the Latin American division of one of the world’s larger agencies right now. And they actually are doing this with a control group. One gets no training, doesn’t take the preliminary exams and the rest of them are, and they do plan on tracking actually just that return on advertising spend and some other factors to see if it actually hits the bottom line.

But they’re huge. They have enough people to create a statistical sampling on the base. I don’t think very many organizations have that luxury.

Matt Bailey: [00:20:14] No for most of my training when I have a small to medium-sized business especially in some of these university programs the best feedback I get is, “Hey, I went home, I implemented this, I understand it now, and I’ve seen results. I’ve had sales go up. We’re on the same page.” That’s more of the, the feedback I tend to get. I even get somewhere, “Hey, I got a new job and I’m getting paid more and I’m doing this.” And, you know, so it’s, it’s a lot of anecdotal. Once in a while, someone will show me some data and just, you know, “Look at what the difference was. Thank you again.” You know, because, you know, from a small to medium size, they’re just running back to their business and trying to implement it as fast as possible.

Michael Stebbins: [00:21:02] Yeah. Well there’s okay. So I have, I have a few thoughts on that as you might imagine and you just tell me when you want me to share them in terms of what are for measurement.

Matt Bailey: [00:21:14] Yeah, go ahead.

Michael Stebbins: [00:21:15] Okay. Number one is if you don’t start with a map, you don’t know where you’re going. And it’s kind of an obvious lead in, but the, I have a lot of I guess it’s empathy for groups who are trying to measure all the way to the bottom line of business goals, because, it can, it’s so many other factors can affect that, right?

If you can do it, great. But if you can’t, if you don’t have enough of an employee base that you can segment, the number one way, the number one way to measure this is to measure competencies. And the easiest way to do that, I give it a C minus, the easiest way to do that is to simply use your job posting.

Okay. Identify and create a testing approach for demand creation program pilots. Established specify, or specify, and then establish AB test results. You know, create landing pages. And I mean, so oftentimes with people putting on the job description, that’s your first hint to the competencies.

Matt Bailey: [00:22:28] I’m going to okay. So I got to jump in. So my last training, we had a big detour about job descriptions because as I’m describing it, you know, we, we were developing the customer journey and marketing throughout the customer journey, but also that measurement piece alongside of it, and almost, I would say a 4th to a 3rd of the room at this point, we’re saying, well, wait a minute.

My job is lead generation, and I can, I can see how many contact forms we get, but I can’t see how many of those people turn into leads and how many of them are then nurtured into qualified leads, and I can’t see how many of those turn into sales. And my response to that is, well, then you need to talk to somebody about your job description.

Because if you can’t see those other silos, if that is hidden from you, then you can’t refine your campaign to create a better lead. That all you can do is create volume. If that’s all you can measure and that’s all you should be responsible for.

Michael Stebbins: [00:23:44] That’s right.

Matt Bailey: [00:23:46] Yeah. If you’re testing anything, you’re testing volume. You’re not testing quality. You’re not testing, you know, which campaign created a better lead. We don’t know. We can’t see it. And so there’s now a challenge for marketers that yeah, you need to know your job description and see what are you responsible for? Because if you’re being called on the carpet for it, if you’re being challenged by it and you can’t see that data. If someone something’s got to change on that. So that’s a great point.

Michael Stebbins: [00:24:15] And, and folks who work with me would groan if they heard me say, “Go to the job description,” but I’m, I’m saying in a pinch, that’s a great start, but I give it a C minus. An A plus would be to build out a competency document. Okay.

So the the, the marketing staff member must define viable repositories for leading customer data for a single view of the customer. Just adding onto your point right there. Right?

Matt Bailey: [00:24:45] Right. Yeah.

Michael Stebbins: [00:24:46] You know, they must be able to explain list management, define the basic components of a landing page. Right. And so when, this is the better step, it’s actually the foundation for job descriptions.

So it’s kind of any, plus if somebody puts together a competency list, now I have a foundation for the job description. I have a foundation for employee reviews and performance reviews. I have a foundation for how I choose training foundation, for how I test for that training, was the training effective?

And if we can go through and say, does the person who completed the training, have this skill now? Or have they shown that they can define viable repositories for leading customer data for a single view of the customer? Yes or no?

Matt Bailey: [00:25:36] Well, and, and so much of that in developing the competencies comes down to the language that’s used.

Michael Stebbins: [00:25:43] Well, it’s our friend, right?

Matt Bailey: [00:25:45] Yeah. Yeah, because you know, a lot of these competencies look like also training requirements, which is they should know how to, or they should know this or know this. Well, okay. How do you demonstrate the knowledge though? What is it that they have to do? And so many organizations, as well as trainers, they rely on that knowledge.

That they should know this. They should know list management. They should know AB testing. Well, in that case, now we have to rewrite everything to like, you were very specific, used very specific verbs and how that knowledge was going to be applied. And here’s the key demonstrated.

Michael Stebbins: [00:26:30] Yes.

Matt Bailey: [00:26:31] And that comes down to your requirements your, your description of competencies. It’s not something you just sit down and well, they have to know this, know this, know this, it’s not that kind of a checklist. It’s how will they, how will these competencies be demonstrated within your organization on a day-to-day basis?

Michael Stebbins: [00:26:52] How do they increase our bottom line? How do they rate our return on investment?

What does, what do these people need to know in order to be amazing at their job and their role at the company, and really help us grow? Yeah. And so that’s where the connection happens. So instead of attaching the connection of training to ROI, I suggest this is competency becomes a foundation. And if your competencies don’t attach to your ROI, then why are you even training to them?

Matt Bailey: [00:27:21] That’s fantastic. Yeah, I like that.

Michael Stebbins: [00:27:23] That’s just, that’s a difficult step though, Matt, I want to acknowledge that. It’s kind of like you know, Brian Eisenberg and you, and a lot of the superstars out there teach AB testing, right? Do AB testing. And it’s the greatest thing in the world that it yields incredible returns and, and yay.

It’s fun. Right? And, and, and there’s a whole bunch of reasons to do AB testing, but the reality is, is that we’re also busy. We’re lucky enough to get one landing page up.

Matt Bailey: [00:27:51] That is exactly the case. And that’s why I have backed off on so many of the, I would call them interjections into the training about, you know, to us, it makes complete sense, but, you know, especially, you know, running your own business?

Michael Stebbins: [00:28:09] Yeah. You’re probably buying the display cases well, you know, and it’s like, oh, because I’ve got to defer this and the bookkeeper doesn’t know how to do it. So it’s the same thing for somebody who’s putting together training. It’s real easy for me or you to say, oh, give yourself an a plus if you write out your competencies, but the truth is just like AB testing, it takes a little bit of extra work to put that in place.

Now the good news is once you do it, everything flows out of it really well. Right. Like in construction, if you have a good foundation, it’s a lot easier to align things up later. And so when, and the reason that I’m such a fan of taking the time to put together a competency is that it really does make performance reviews a snap because you’re looking at the same map.

Matt Bailey: [00:28:58] Wow. Yeah.

Michael Stebbins: [00:28:59] You know, your training, it goes exactly to this and yeah, what’s great is that if somebody came to you, Matt, with a competency document like that, and put it down and said, can you train to this? I think you’d be thrilled.

Matt Bailey: [00:29:12] Oh, absolutely. Absolutely.

Michael Stebbins: [00:29:15] The annual readiness testing, because how do we evaluate this? Well we can, based on these competencies and Matt came in and taught it to them.

So was the training effective or not? Well they darn well better pass their annual readiness test. Right? And it even goes to play for candidate skills verification at the early start. If I’m going to hire somebody, I can ahead of time if they already have these competencies. So it, it becomes a real winner of an investment because you can use it for so many things.

And so that’s why I’m such a fan of that. But again, if you’re really struggling with that, it’s okay to fall back to the job description. It’s just not ideal.

Matt Bailey: [00:30:00] Well, and the better the job description you know, it’s, it’s one of those things I enjoy seeing online is is these I saw something the other day where they required 5 years of experience in this programming language that only has been out for 2.

Michael Stebbins: [00:30:14] Yeah.

Matt Bailey: [00:30:15] Yeah, some of these job descriptions are just completely bonkers. And by doing that, I think you’re actually lowering the quality of candidate that you’re getting. You know, someone’s going to know right away. You don’t know what you’re talking about.

Michael Stebbins: [00:30:31] And not all candidates are confident enough to just bluster through that.

Matt Bailey: [00:30:36] Yeah.

Michael Stebbins: [00:30:36] I know many that I talked to and they’re like, well, I can’t apply for this. I don’t have 5 years of experience, and I said, are you serious? Just apply. It’s okay. I mean, obviously if you miss like 80% of what they’re asking for, you’re wasting people’s time, but if you can do what they’re asking for in terms of, and you used the right word, responsibilities, then make sure they know that you’re there.

Matt Bailey: [00:30:58] Yeah. It’s a great angle to take. I actually was looking at this survey that was created looking at the past 3 years of data for the same questions and the desire to measure the business impact of learning programs three years ago was just over 50%. Now it’s over 70% of organizations that want to measure. It’s funny because 3 years ago, the belief in the possibility to demonstrate the impact was 30%. Now the belief of the possibility to measure the effectiveness of training is over 50%. So…

Michael Stebbins: [00:31:43] It’s headed in the right direction.

Matt Bailey: [00:31:45] …yeah, it’s going that way. But the obstacles that people are running up against, it’s interesting that competing priorities within the organization is the number 1 obstacle. The number 2 obstacle is they don’t know where to start.

And so I think, you know, like you said, your job description, your competencies based on the job description you know, the 3rd is no access to any data. So you almost have to create your own data and your, your conclusion there really does that. It creates its own data and which develops its own data set and its own, then goals and priorities of how it should be accomplished.

Michael Stebbins: [00:32:26] Yep. And there’s one other, I want to slide in there. Touched on what you were talking about earlier, when you said that some of the evaluation was, you know, well, and I hate this question. How do you, do you feel about the training?

Matt Bailey: [00:32:38] Yes.

Michael Stebbins: [00:32:40] First off, feeling you have feelings about training. It’s like, no. You judge the training, was it good or not?

Matt Bailey: [00:32:46] Right. Right. I’ll tell you how I feel about my, you know, 11th grade algebra teacher. I’ll tell you how I feel right now.

Michael Stebbins: [00:32:54] A little depressed, you know, feelings are neither good, nor bad. Right? It, judging is important. How did you judge it? I just love the idea that you touched on in terms of workshops and having a tangible thing.

You know, as, you know, one school of thought is that the best way to learn something is to teach it that you can argue for it against that. But I think I believe her in that. And you know, if there is actually what it was an article I’ll refer to it, give credit. This is an e-learning seven innovative ways to measure training effectiveness.

And I want to give credit to whoever wrote this, and I’m having a hard time finding that. But I’m looking at the article under academywhatfixed.com just to give it some reference. And what I’m seeing here is there’s a section that talks about visual authentication, and this shows up in a number of different reports, visual, visual, visual.

When they ask the candidates to actually do a recording, like a video recording of what, what did you learn? What have you done? What are you, what did you come out with? What did you do? And they, they take visual proof to reiterate what they took away from that. And I just, I really, I haven’t, I gotta confess, I haven’t experienced this a whole lot myself, but other than the workshops that we described.

But this, this I think is a phenomenal way to do with the early let’s replace, “how do you feel about the training?” with, “show us what you came out with?”

Matt Bailey: [00:34:34] Oh, absolutely. Absolutely.

Michael Stebbins: [00:34:36] About what you learned or reiterate a concept to the rest of the audience here that was a takeaway for you.

Matt Bailey: [00:34:45] Yeah. It’s the wrong question.

Michael Stebbins: [00:34:50] Those things I think will really establish in the earlier part of the scale, the early effectiveness of the training, because I do give ma I do give some understanding to somebody who just made an investment in training, and they’re nervous that it’s going to get a bad reception. It’s boring. You know, it’s not up to date and that person gets skewered and roasted and tossed out the door because they just subjected a thousand employees to go through hours of something that was dull, it didn’t have a pickup to it.

So it’s a valid thing to get early feedback. So there’s some credit to Kirkpatrick’s model, but I would. I would modify it slightly to, can these people regurgitate, you know, or put themselves out there because when the red lights on you got to deliver.

Matt Bailey: [00:35:43] Absolutely.

Michael Stebbins: [00:35:43] I’m thinking about having a camera on you or recording where you get your ducks in a row before you speak and whether or not you actually have learned something, you know, becomes somewhat apparent when you need to do that, you know, with a little bit of allowance for people who get nervous about that or something like that.

But, for the most part, visual proof, visual proof.

Matt Bailey: [00:36:03] I like that. I liked that. I well, and it’s interesting. So I’m looking at this survey here about how their department is evaluated.

Michael Stebbins: [00:36:13] Yeah.

Matt Bailey: [00:36:13] And it used to be learner satisfaction was number 1, now learner satisfaction is number 2 to organizational impact.

Michael Stebbins: [00:36:25] And it’s just hard to measure.

Matt Bailey: [00:36:27] Yeah. Return on investment is, is the lowest…

Michael Stebbins: [00:36:30] yeah.

Matt Bailey: [00:36:30] …of, of how people are evaluated. I thought that was very interesting.

Michael Stebbins: [00:36:34] I’m going to give credit here. I just found credit for what I was referring to it’s by John Eades, E A D E S.

Matt Bailey: [00:36:41] I’ve heard of him, yes.

Michael Stebbins: [00:36:42] Yeah, September, 2014, 3 ways to measure training effectiveness. It’s an article in e-learning industry.

And he actually has 3 ways. Visual confirmation, social ownership and skills assessments. I’m already a fan of John here. But I was borrowing from his visual confirmation there. So I just want to credit goes to him for, uh.

Matt Bailey: [00:37:05] Well send me that link and I will, I will post that on the page as well. But yeah, it’s, it’s interesting that when you look at how the department is evaluated and you have learner satisfaction, which I, hey, believe me, the learner better come out of there thinking, “I, I didn’t waste my day.”

If someone comes out of it with that attitude, then yeah. You, like you said, we’ve just successfully bored everyone and, and wasted a lot of time. There’s got to be an element and it’s interesting, I just, I got to review the training for a very significant brand and they’re trying to develop analytics training and they sent me a video.

And from the first 10 seconds, It was the announcer voice and it was obvious they’re reading a script and it’s that typical pace and inflection that within 10 seconds, I’m like, oh boy. Oh boy. And then within 2 minutes they are so deep in the weeds.

Michael Stebbins: [00:38:14] Yeah.

Matt Bailey: [00:38:15] And all it is is like a fancy definition of things rather than, how do I make this work? I mean that I, when I do my analytics courses, I’ve started now letting the room define the objectives for the day that I, what do you hate? What, what discourages you about analytics? What, what’s your problem with analytics? And we write it all down on the board and that sets the objectives then.

And part of it is I know 99% of what they’re going to say. So I can, because it’s the same almost every time.

Michael Stebbins: [00:38:53] When they’re thinking this is amazing. He’s solving all of our problems.

Matt Bailey: [00:38:59] And that’s my secret is I know what they’re going to say. And then, and so as we go through it, I walk over to the whiteboard and I cross it off and I ask everyone, is that answer your question?

Yes, it does. So in that way, it reinforces this progression through the content and through their questions. And at the end of it, you know, we’re looking at this list of things that are all crossed off and I’m like, okay,  what questions do we have left? And by that, now they’re that satisfaction levels up because they’ve seen the progression and into a very tangible format.

And I think that is all in the preparation planning and the activity of actually accomplishing something rather than sitting and listening.

Michael Stebbins: [00:39:48] Yeah, those who do, and then those who teach.

Matt Bailey: [00:39:51] Yeah, exactly, exactly.

Michael Stebbins: [00:39:55] Speaking of the audience, cause I know you do both so.

Matt Bailey: [00:39:57] Well, thank you, Mike. I really appreciate your time coming in here.

[00:40:00] And hopefully all of this turns out not to be terribly bad. But for the foreseeable future, we are sheltering in place and doing the best we can. So thanks for your time and your, your, your opinion on training. I really appreciate,

Michael Stebbins: [00:40:17] It’s been a pleasure and it’s always fun, Matt, real quick, a resource if you don’t want to write your own competencies or you want to use the same it’s used for the OMCP standard, they are available on the omcp.org website. I think it’s a hundred dollars just to cover our costs for publishing and maintaining how you can download those and use them for your own competency guides your own training, or you can use them to define what Matt comes and does for your company.

Matt Bailey: [00:40:44] Fantastic. I think they’re powerful. They really define it and yeah. Use it to write your job performance, your job performance, your job openings. I mean, they are a fantastic resource. Certainly recommend that. Mike, hey, thanks again. Hope you’re doing well. Stay safe.

Michael Stebbins: [00:41:01] My pleasure. Take care Matt.