When discussing the Flipped Classroom last year, we ended our four-part discussion with an appreciation of how to measure its effectiveness. Several ways of evaluating trainings were identified including informal feedback from students and formal assessments of content mastery among others. Our discussion was intended to offer different ways for to evaluate trainings from various perspectives.
As you probably guessed, a well-developed field exists to evaluate courses, educational techniques, and training approaches: Instructional Design Models (IDM). There is a range of IDMs, but the best-known model is the Kirkpatrick Model.
Donald L. Kirkpatrick developed the Kirkpatrick Model, which was based on his 1954 dissertation, and later serialized in the US Training and Development Journal, the organ for the American Society for Training and Development (ASTD). In 1994, he and his son, James D. Kirkpatrick, published Evaluating Training Programs, which provided a complete and formal foundation and basis for his original ideas. Kirkpatrick with the assistance of his son and daughter, Wendy K. Kirpatrick, founded the business enterprise Kirkpatrick Partners, to offer consulting, products, and various events and training based on the “one and only Kirkpatrick” model. Donald passed away in 2014, but his children continue promoting the Kirkpatrick model, and James Kirkpatrick, who also has a doctorate, created the New World Kirkpatrick Model, which adds additional facets to each of the four levels for evaluating trainings.
The following four elements constitute the basis for the Kirkpatrick Model: Reaction, Learning, Behavior, and Results. Kirkpatrick originally used a pyramid schematic to visualize his concepts, but the Kirkpatrick Partners on their website currently use the image of interconnected links of a chain. Each level progressively leads to the next and is best understood as the straightforward definition of the level’s name.
LEVEL 1: REACTION
The basic understanding for level one is simple: how did training participants react to the training? This is a simple method for appreciating customer satisfaction. Or as described on its website: “the degree to which the participants find the training favorable, engaging, and relevant to their jobs.” The New World Kirkpatrick Model (NWKM) added the latter two elements. They describe engagement as the following: “The degree to which participants are involved in and contributed to the learning experience.” Similarly, relevance is described as the following: “The degree to which training participants will have the opportunity to use or apply what they learned in the training on the job.”
A. Reaction Sheets/Smile Sheets
The basic means of determining reaction is the use of “smile sheets,” otherwise known as a survey of participants’ reactions. Such a survey, generally speaking, is handed out and completed just after the training using paper and pencil or on-line.
B. Survey Questions
Most surveys query participants about a number of the training facets. What is the facility like? Was the facility located in a convenient place? Did the training start on time? Were the training goals clearly outlined? Were the training materials helpful? Was the facilitator knowledgeable? Did you like the facilitator’s style? Were there a sufficient number of breaks during the training? Was the content relevant to your work?
Training participants are prompted to answer each question in either a binary yes/no fashion or rate the response along a scale (Likert Scale). The distinctive element of these questions is that the focus is on the training and the trainer.
C. NWKM Shift of Focus
Jim Kirkpatrick realized that there was something missing in the traditional Kirkpatrick Model. He realized the surveys were self-centered around the environment and themselves: their facility, their course, and their trainer.
The NWKM remodeled the smile sheets to be “learner-centered.” As noted in their training material, instead of the training-centered category “The program objectives were clearly defined,” the learning-centered category is “I understood the learning objectives.”
Jim Kirkpatrick believes, most importantly, that this level needs to be tied to the last two levels, behavior and performance. That is, the NWKM is structured and functions to reinforce or stimulate positive on-the-job practices, which in turn directly impact organizational goals. He notes that the since the “smile sheets” generally are training-centered that they create a perception for training participants that are anchored between the participant and her/his job. Rather, the training needs to be learner-centered in a manner that links the participant to the organizational goals through their jobs.
Next week we’ll look at Level 2: Learning.
Craig Lee Keller, Ph.D., JAG Learning Strategist