Measuring Flipped Classrooms

New ideas are great! That is…until the next new idea comes around? Hmmm?? Do we need another update?! Think for a moment, lest we forget, the Flipped Classroom is not so much about with replacing the OLD with the new. By understanding various bases for learning, one can pragmatically apply such to enhance the overall learning experience so learners can better comprehend, retain, and apply information. So how does one “grade” flipped classrooms compared to other models? Numerous metrics exist, and organizations should selectively draw upon them to best understand the impact of their trainings.

Measuring Flipped Classrooms

Impact of the Flipped Classroom

When trying to determine the impact of any training, information should be drawn from many sources. We should welcome training comments not just afterwards but at times prior to and during the classroom. Flipped classrooms have two-three distinct stages: first, delivery of training content—often on-line—prior to any classroom engagement; second, discussion of content—the homework, as it were—in the classroom setting with the “instructor” assuming the role of a mentor; and, third, post-classroom review, integration, and completion. Obtaining feedback at each stage is essential.

A. Informal Training Commentary

There are two types of informal training commentary:

  • In-Training Input. On-line and in-class commentary can be extremely valuable. Even before the classroom meeting, students should be encouraged to suggest and question the training content. I thought experts were creating the content? Yes, trainers should create eLearning content that is accurate, complete, logical, and thoughtful. Yet nobody is perfect, and even award-winning writers benefit from editors. Think crowd sourcing, the use of ostensibly disconnected individuals with the common goal of gathering information and/or completing tasks. Here, staff are not only valued but empowered, giving them a sense of ownership. The suggestions can be something as simple as pointing out typographical errors, adding useful examples, et cetera. The goal here, remember, is creating the best training. Indeed, the mentor can reinforce such suggestions during the in-class time, thereby affirming specific staff members in the context of creating an educational community. Soliciting comments prior to the classroom assists mentors to more efficiently utilize that time. In-class comments generally will take place on a one-to-one or group basis; remember, the classroom is a site of mentoring, not the site of the traditional top-down didactic.
  • Post-Training Opinions. Staff comments during trainings are frequently different from those collected afterwards. Managers can glean such attitudes at the proverbial water cooler or during individual and team meetings. The trainer, too, should definitely forward to the administrator a self-assessment of that specific meeting(s). Each should be cross-referenced with the other, and any striking difference should be addressed. Special attention should be paid to the role of technology, the structure and delivery of the training, and the overall efficacy of the teaching model.

B. Formal Learning Assessments

To assess the utility of Flipped Classrooms as an educational/learning model, focus should be placed upon two student areas: satisfaction; and, content understanding, retention, and application.

  1. Training Questionnaires: Training administrators need to assess subject matter (content), evaluate educational style (both training format and mentor performance), determine cognitive understanding, and reflect upon its professional and organizational value. Traditionally, administrators and teachers generally concerned themselves with only cognitive understanding and measured this through testing and grading. However, if organizations are truly concerned with actual learning, then they need to appreciate how learners receive trainings. Staff members and students who do not care generally do not learn. So training questionnaires are essential and should not be taken pro forma. Can’t the “learners” just get inline? An anecdote: while working as a waiter in graduate school, my colleagues and I would become frustrated with patrons regarding a given tip. We would regale others with that customer’s bizarre behavior. All of these conversations were just therapy. The customer’s reality is the reality, and the same goes in education. This is not to say that anything goes, but administrators need to refocus and appreciate that the traditional top-down pedagogy is not the sole solution. Hence understanding the perspective of learners is key for improving the learning process.
    • Types of Questions: When composing questionnaires, the type of question is just as important as the subject matter. Should the questions be Yes/No, multiple choice, or free expression? Generally it is best to have a combination of types. Binary questions (Y/N) should query subjects that have defined answers (was the instructor on time). Alternatively, Likert scale questions (multiple choice) are best used for opinion-based answers (the training emphasized ABC: not at all, not enough, a fair amount, too much). With Likert scales, consider whether or not to have an even or odd number of options. An even number forces learners to affirm one side of the spectrum, whereas an odd number leaves open the “neutral” response. Finally, open-ended questions (is there anything else you would like to add, et cetera) are essential to obtain information otherwise not captured in the preceding question types.
    • Timing of the Questionnaire: We do this at the end of the training, right? At the end of the training, yes, but not at the end of the classroom session. Frequently, learners (and instructors) are impatient to leave the classroom to attend to other matters. In fact, many participants do not even fill out the questionnaire or only do so hastily. So what is the remedy? Administrators increasing are requiring training participants to fill out questionnaires online after the training, often several days after the classroom session. Why? The intervening days give participants the proper perspective to more fully appreciate the training and separate it from emotions of unrelated matters. Moreover, administrators can make a response—a complete response—a requirement for credit or certification. While participants might try to skip out early on the class (and the questionnaire), none of them will be able to fully complete the training without their online response.
    • Answer Consistency: Don’t staff members just respond based on what the administrator wants to read? That is a tough issue. Sometimes learners are overly effusive (excellent, excellent, excellent . . .), sometimes they are overly negative, and sometimes, they just want to finish the questionnaire as quickly as possible! First, by increasing training effectiveness and value (through innovations like the Flipped Classroom), learners are more likely to take the time and offer constructive feedback. Second, administrators should design questionnaires to query smaller, constituent parts of training value. Answers about “micro” value assist administrators when comparing them to a final Likert scale about the “general” value of training.
  2. Testing Diagnostics and Efficacy:
    1. Pre-On-Line, Post-On-Line: Think of this . . . Training as diagnostics. What do you mean? OK, it is exactly what you think. That is, at differing stages of the training, users are queried about the subject and given differing storyboards based on their responses. This serves not only to gather information, but also to stylize the delivery of information for each user. So if a user’s response suggests a level of understanding demonstrating mastery, then the subsection is concluded with the progression to the next stage. If, however, the user’s response suggests he or she did not fully grasp essential parts of the stage, then that user would be redirected back through training in a fashion that even further explicates the content and its connection to the larger subject of the training. All of this serves to assist the learner in a stage-based approach by using a micro-learning algorithm and to provide the basis for improvements in the training itself. This information would be available to the mentor prior to the classroom session.
    2. In-Classroom Learning and Mentoring: Again, learners in the Flipped Classroom are not engaged using the traditional teacher-led didactic model; rather, mentors help learners to better understand material and appreciate why they might have misinterpreted the training content. If the mentor chooses to utilize group learning, then he or she should be careful that the groups do not devolve into unrelated social interactions, which, on one hand, is good for bonding amongst learners but not so good for learning. One way to handle such is to direct individual members of a group to assume different roles (for example, one person takes notes, et cetera). Diagnostics during this period, as one might suspect, tends to be more qualitative than quantitative; however, when assessed in a longitudinal fashion over numerous trainings, then such data can be useful in terms of streamlining the content and structure of the training as well as gaining a better understanding as to how to facilitate the classroom mentoring for that subject matter.
    3. Periodic e-mail/text reminders/queries: After the in-classroom learner-mentor sessions, learners should receive periodic e-mails refreshers about key content and subject overviews. On occasion, they can be prompted to follow links to an online portal testing them on the training, which, in turn, serves to condition the content and duration of any subsequent e-mails. The training administrator, of course, keeps abreast of learner responses to this process, which also serves to inform interactions in next part.
    4. Individual/Group Post-Training Review: Individual learners will discuss trainings with their supervisors to reaffirm content; supervisors should be able to appreciate whether or not staff members have been able to operationalize training content for work.
    5. Testing/Grading/Passing: Do Flipped Classrooms have traditional testing? How are learners graded? Or is there simply a Pass/Fail approach? All good questions, and there is not one easy simple answer. Part of this issue is based on the nature of the material; part of this issue is based on whether or not the training was required; and, part of this is based on organizational culture. For example, if the training is required in the context of a contract or certification, then, most likely, some type of direct testing and grading will be necessary in order to document proficiency and the like. OK, are you telling me that we didn’t really flip the classroom? No, that is not it at all J Tests in the context of determining levels of mastery is not a defining element of the flipped classroom per se. Rather, the Flipped Classroom is based on inverting educational dynamics and transforming educational roles. Whether or not learners take a “final” test is unrelated to the flipped learning model.

C. Discerning Training Efficacy

Keep in mind, discerning training efficacy is the equivalent to program analysis, which is an extremely large and nuanced field of study. The subsections below offer a structure of an approach, not a listing of specific techniques. Each one focuses on a different dimension and, moreover, how all of this is related to measuring the Flipped Classroom.

  1. Individual Performance: Determining whether or not a given training “worked,” that is, helped the learner do her or his job better can be tricky. First, obtaining any quantitative comparison regarding output or external client feedback is only possible if such information is available from prior to the training. Second, sometimes individual performance might be difficult operationalize. Third, qualitative comparisons are useful but limited. But, if prior quantitative data is available, then comparisons can be made; similarly, administrators and program directors can collaborate by assessing training and performance in a longitudinal fashion.
  2. Organizational Goals: Discerning training efficacy regarding organizational goals is, perhaps, even trickier than assessing individual performance. As we know, many factors—internal and external—impact whether or not an organization has been more or less successful in executing its mission. Again, sometimes it is possible to utilize quantitative metrics, and, ultimately, this is key, especially if an organization is investing greater capital and money in its system of training. So be sure to develop mechanisms and metrics when designing trainings in order to operationalize its value.
  3. Cybernetic Feedback: Cybernetic? Yea, now we’re talking! Finally the cool stuff!! Cybernetics sounds very futuristic if not like science fiction. (So, ehh, what is cybernetics?) If we think of organizational training as a system, that is, relationships amongst course content, educational style, and the learners, then learning is a process. Certain systems—and organizations—can be incredibly efficient; others can be terribly dysfunctional. Most, as we know, are somewhere in between. Being creative—remember, welcome to the Flipped Classroom—is a start, a big start. But simply importing learning models is not the solution per se. The solution comes from using tools to make sure the system, here, organizational training, is effective and adjusting the approach in a manner to make it even better. How do we do that: informal training commentary, formal learning assessments, and discerning training efficacy. Ultimately, organizations will modify their trainings while getting an even better appreciation of the value of those trainings vis-à-vis its organizational goals.

These past three blogs provided readers with an intellectual orientation and pragmatic tools about Flipped Classrooms. We will be discussing select topics about the Flipped Classroom from this point while taking the occasional detour to discuss emerging ideas and issues in the field of eLearning. See you next week!

Contact us to learn how we can help your company or organization utilize these tools.

Craig Lee Keller, Ph.D., Learning Strategist

Building Flipped Classrooms

Any organization can benefit from using the principles of the flipped classroom. Your employees will greatly benefit, which ultimately creates a positive impact on your mission. So here we go! This is the beginning of adapting and integrating flipped classrooms to your existing system of education and training. Probably the most important part of embracing and using a flipped classroom is appreciating how it could fit into your existing organizational culture.

Jumping into the Flipped Classroom

A. Organizational Analysis. For those choosing to include flipped classroom models in their educational plan, it is easy, very easy, to focus on exciting technologies and the like. When do I get to wear my virtual reality glasses!? Though before even considering such, thoughtful leaders and administrators, first, need to analyze the nature of their educational requirements and map them with the different approaches to the flipped classroom. Second, assess deficits about existing training and determine how flipped classrooms can address those deficits and aid in resolving other problems in general. Third, after determining the requirements and existing deficits, project organizational needs vis-à-vis technology, support contracts, and administration.

 

  • Requirements. There are different types of educational requirements for different employees. First there are basic trainings in which everyone participates; think organizational orientations. Second, educational requirements might exist due to contractual obligations; think, stipulations requiring all program employees must receive “safety” trainings for onsite work, or, understanding proprietary concepts and information. Third, some organizations may require by contract requirement or their own internal standards that certain employees obtain certain types of certification; think employee competency in emergency first aid, or professional licensure for, for example, nursing and social work. Fourth, similarto the third, many licensed professions are required to take CEU (continuing education units) in order to maintain their legal standing.
  • Existing Deficits. While looking at the application of flipped classrooms, organizations already have an existing structure of education and training. What is working? What is not working? Do employees consider training to be a waste and/or burden? Flipped classrooms can help with each question. It is fairly easy to identify trainings that “work.” (Though they can become even better when employing a flipped classroom model.) These trainings operate smoothly without any type of employee “push back.” Staff ask pertinent questions, receive proper answers, and are able to operationalize the training, for some, immediately. Here, think about a training about using the new entrance security system or using the new voicemail system on the phone. Spotting trainings that don’t “work” can be difficult. Frequently staff employ work-around strategies, including drawing upon the staff “expert” to help her or him with the problem; such is not only inefficient but also a constant drain on time. Probably the worst deficit associated with training is the impact on organizational moral. Poor trainings create beliefs about poor management, reinforce divisions amongst different parts of the organization, and create a sense of cynicism about one’s job.  Such deficits are highly corrosive! To deal, briefly, with this latter deficit, flipped classrooms are effective in increasing faith in management (mentor role model), bringing staff—literally—together in the context of an educational community, and generating a sense of meaning and value about organizational work.
  • Needs. With an understanding of the requirements and deficits, organizations then can successfully determine their needs to move forward. Do we need to purchase technology? Do we need a training specialist? All very good questions that require an answer inorder to obtain the most efficient and cost-effective results. While needs may be governed by fiscal restraints, needs also are governed by the type of training. What is the training goal? Not withstanding matters of content, two training goals are easily identified: mastering and applying training content in some analytical fashion and mastering performance and/or procedural operations. Think understanding new government regulations and applying them to organizational operations versus understanding how to deliver CPR to those in need. So the first type noted above might employ a blended learning, flipped classroom that includes micro-learning and gamification, while the second type might employ animation and gamification. Generally speaking, this part is best handled by a learning professional who understands your requirements and deficits and who can assist you in determining the right blend. Unless you have this talent in-house, most likely a consultant can easily assist you in this process.

B. Moving Ahead. This section contains a quick jumpstart and two different and extremely useful tools for a Flipped Classroom.

  • Jump Starting a Flipped Classroom: In the flipped classroom, students learn the bulk of the material outside of the classroom setting. O.K., let’s assume that employees take turn going through a guided, node-based presentation; along the presentation, an employee is peppered with related questions before continuing. Inaccurate answers will prompt her or him back to that issue area for review. By the end of the presentation, managers can derive information regarding the parts of the presentation with which the employee had difficulty. This can serve as part of the basis for in-class mentoring. Mentors can either engage employees individually, or, perhaps group together those students who had similar challenges and work with them as a team. There are a variety of approaches one can take. 
  • Micro-learning: There are numerous ways to employ the concept of micro-learning. All of them, however, deal with breaking the subject matter into smaller pieces. This provides learners with the critical information they need, just in smaller chunks. If we try to intake large amounts of information all at once, the vast majority of people confront “cognitive overload.” In short, our memory and ability for synthesis progressively declines based on attention span, fatigue, a lack of specific interest, et cetera. With micro-learning, the scope of learning is narrowly defined, permitting students to grain mastery over individual pieces, often through repetition, which is facilitated by different types of technology. Problems associated with cognitive overload decline. Let us recall the metaphor “One cannot see the forest for the trees.” Advocates of micro-learning understand that focusing on everything at once creates cognitive overload, which in turn, not only makes it difficult to appreciate particularity but also creates a blur when trying to comprehend the larger meaning.  The big picture, the forest, as it were, often gets lost when people get bogged down trying to remember all of the details, most of which they cannot remember anyway because of cognitive overload. Using micro-learning in the flipped classroom creates a patterned, sequential approach enabling learners to reinforce information mastery while gaining a better comprehension of larger ideas, which, in turn, greatly improves the learner’s ability to remember and use that information in the context of real-world work situations. Again, all of this is facilitated with different types of technology with the overall goal of effective learning. There are numerous techniques you can employ in this regard in the context of integrating learning with traditional work:
    • Short, informational e-mails or videos: these bits of “micro” information are reinforced at daily/weekly staff meetings and later assessed, perhaps, through an automated diagnostic query answered by employees at their desk. With this information in hand, managers can bring staff together to culminate the training through the process of mentorship that helps individual employees with challenges identified through their diagnostics while fostering team comprehension and cohesion.
    • Flashcards: instead of an informational e-mail/video, employees can learn through applications that simulate flashcards. Similar to language acquisition, the flashcard application is programmed to continue to the next stage or repeat based on employee comprehension.
    • Mobile Technology: Here, the delivery of information is accented throughout the workday. Instead of a “fixed” training time, employees are periodically prompted with information through texts and the like. This approach is better suited for those who do not have “desk-based” jobs.

Next week’s discussion will help you develop metrics for determining how well your flipped classroom is working. Again, please comment or email any suggestions or thoughts about this issue!

Craig Lee Keller, Ph.D., JAG Learning Strategist

Welcome to the Flipped Classroom

The what! classroom . . . ? For many, flipped classrooms represent radical change. If the old classroom was good enough for me, it’s good enough for my children! If it ain’t broke, don’t fit it!! Yet even skimming newspapers reveals growing concern about the state of education: the United States lags other nations in mathematics and science while children do not know “basic” historical facts; calls for reform range from cultural vitriol to dry bureaucratic reports. So what is and where does the Flipped Classroom fit in? For those unfamiliar, this blog—the first of a three-part series—will provide a quick historical context along with the basic concept, principles, and rationale for the flipped classroom; for those familiar, this blog will offer a review along with some amusing anecdotes for points of reflection.

A Brief Historical Context

Three closely related issues exist in educational reform: the purpose of education, the content of education, and style of education. The U.S. educational system was structured to create model citizens, drawing upon classical sources: Greek and Roman literature along with the Bible. This model used a traditional, teacher-centered didactic, top-down learning with teachers distributing knowledge and disciplining students who were viewed as passive vessels receiving information. Failures in learning frequently were interpreted as character flaws.

With the rise of industrialism, bureaucratic and production efficiencies created new educational demands, with expanded political rights and social roles, educator demographics expanded and included neglected perspectives. Tensions arose from these changes: one questioning the purpose—and value—due to market demands and one questioning the content—and social role—due to an expanded range of participating stakeholders. Each dynamic, though, frequently maintained the traditional, teacher-centered didactic. Still, evolving educational theories created innovations to improve student learning. For example, philosopher John A. Dewey questioned the passive role of the students and argued for experiential learning.

Flipped Classroom

1. Basic Concept. K. for the first time . . . what is this idea all about? The Flipped Classroom is an integral part of alternative teaching models emerging over the past couple of decades. More specifically, the framework was derived from the research of Alison King and Eric Mazure among others, while gaining impetus from Louisiana-born entrepreneur Salman Khan and his creation of the Khan Academy. By utilizing technology and transforming educational relationships, flipped classrooms promise greater student comprehension, retention, and utilization of materials in a range of disciplines and educational formats. In short, students receive the bulk of educational material remotely, generally on-line, from the classroom setting; the classroom, then, serves as a forum for the students to better question, understand, and integrate the material through student collaboration and the guidance of a teacher. Contrary to the traditional model, flipped classrooms are student-centered, break down hierarchy, and create educational partnerships amongst students, their peers, and teachers.

2. Principles. This all seems to include everything that’s already being used in the classroom . . . aren’t we just moving things around a bit? The teacher still is important . . . right? Yes and No. With the Flipped Classroom, elements of the educational process are redefined and retooled. In traditional styles, the teacher imparted information in the classroom setting, reinforced it on the chalkboard, and was the final arbiter of its interpretation; students were required to be alert and take notes. In addition to lectures, students might perform in-class assignments and readings to be judged by their teacher. In short, there was an educational vector from the teacher to individual students; homework served as reinforcement if not a test. Blended Education, alternatively, draw upon different technologies and types of materials. The significant surge of technology over the past couple of decades—wider access to the internet most importantly—created the option to alter and relocate the traditional educational vector.

  • Technological Innovations. While digital and internet technologies are the key elements of blended learning, it is important to remember that technology always has impacted the style of educational practices. I fondly remember my father recounting how he constructed a giant (and operational) slide rule to teach his students; similarly, nobody could hold a candle to his overhead projector and transparencies. Why are these examples significant? Because technology gave the teacher the ability to condition content and delivery based on real-time student needs and questions. Technology can condition the educational dynamic instead just offering a simple flow of information. Digital technologies and the internet offer students the option to learn outside of the classroom (at home or the library) and/or without direct teacher instruction (using a digital/technological interface). This flexibility can accommodate a variety of student needs to progress at their own speed and more closely review difficult concepts. For many younger learners, this technology already is second-hand and reinforces existing skill sets. Blended education has a wide array of options and models. Educational goals and existing educational environment influence one’s choices.
  • Alternative Aptitudes. The scholarly community differentiates between a range of cognitive skill sets (or, as suggested by Martin Gardner, “types” of intelligence); similarly, there are different means to develop such. For example, many are familiar with the notion that artists and designers are “visual learners.” Such is the same with other skill sets (e.g., memory, ordering, et cetera.) Information can be adapted using the internet and digital means best suited for an educational task. The classical Socratic method still can be valued (direct learning through questions and answers), but lest we forget, writing and speaking, too, are mediums of information.
  • The Educator’s Role and Homework. With traditional didactics, homework was integral to the learning process; inability to succeed was interpreted as a lack of discipline. As such, the teacher served as a performance judge; alternatively, in flipped classrooms, the teacher serves as a mentor who fosters a community different from the traditional classroom. This should not be all that foreign: think of the roles of students and teachers in science labs. Instead of being a point of judgment, homework is transplanted back into the classroom, that is, the classroom becomes the arena where concepts and information are more fully explained and integrated together. This makes sense if the ultimate goal of education is mastery.

3. Rationale. So, again . . . remind me . . . why is this important? Shouldn’t students just listen and spend enough time doing their homework and less time watching television? This, again, is a very good question. With all of the new educational approaches and theories, let us briefly reflect back to the historical context: the Flipped Classroom is NOT about the grand purpose of education, and the Flipped Classroom is NOT about the content of education. Rather, flipped classrooms can be utilized to improve educational goals, that is, helping students to understand, remember, and, actually, use the material at hand. There may be some differences across disciplines, but . . . it is a universal educational style.

  • Comprehension and Micro-Learning: In the traditional classroom setting, student comprehension could be limited by a variety of factors—distractions, length of instruction, et cetera. We all have memories or can simply point to popular culture references of the teacher “droning” on . . . Wha, wha, wha, wha-wha . . . But if education is viewed as a supportive community, not a top-down convention, the process of learning can be satisfying, even an adventure. One of the key elements of comprehension is micro-learning. What is it? An anecdote first, if I may. As an undergraduate, I had a difficult time learning new material. O.K., then, just put in more time? But the more time spent seemed fruitless, reading the same pages over and over again. I began to realize that it was easier and more efficient to break down the information in smaller bites (no pun intended). In so doing, I was able to more easily retain information by envisioning relationships and arguments. Micro-learning draws on this logic.
  • Retention and Mastery Learning: While most educational systems have year-end examinations, a problem exists with the relation of such tests for learning. Short-term learning is a very different skill than maintaining a mastery of a body of knowledge. Ultimately, one can comprehend when being taught while not being able to retain that information for future use. I recall a high school physics teacher who permitted his students, including this writer, to take any of classroom test as many times as desired (changing numbers and the like of questions). What? Isn’t this cheating? Well it depends upon your definition. If cheating is crude competition, then yes; however, if education is viewed in the context of educational mastery, then no. Part of the problem associated with the mastery of knowledge is based on alternative approaches to the education. Recall apprenticeships undertaken by sushi cooks: for one year, the apprentice only works on cooking rice. Again, what? If I am paying for this, I want my money’s worth! Yet the approach of a master is not to offer a certificate, but rather to ensure individuals will be competent, independent professionals. Without student mastery, the teacher is regarded as remiss. Regarding academic education, we should, with some irony, remember that universities originally were based on this guild model, which accounts for its top-down control and narrow scope. Yet, mastery learning is still essential regardless of counterproductive traditional didactic styles. This is the role of the flipped classroom, adapting teaching styles so that everyone can be a master.
  • Utilization and Experiential Learning: As noted in the section on mastery learning, the model of education utilized by guilds and trades draws heavily upon experiential learning. The style of education is not simply mastery of information, but a means for understanding when and how to use that information in “real-life” settings. Recalling one of science classes, the final exam presented students with a geological map charting the outcrops of various rock types; based on the year’s learning, students were expected to generate a geological cross-section depicting the various inclines and synclines. While I found the test an extremely satisfying test and use of the year learning, many stared at the test and did not know what to do. Unfortunately, the class did not employ the type of experiential learning that would have empowered the students to more easily connect “textbook” information with the “real world.”

The efficacy of experiential learning draws upon many of the previously discussed issues—technology, alternative aptitudes, micro- and mastery learning, et cetera—all directed toward linking theory with practice. The key point with experiential learning is the use of these different components in a manner that enhances learning, not just granting the teacher free time.

Next week’s discussion will focus on how to utilize the flipped classroom in your educational and/or organizational setting. Again, please comment or email any suggestions or thoughts about this issue!

To get added to our mailing list to get regular updates on JAG learning best practices and white papers, please sign-up!

Craig Lee Keller, Ph.D., Learning Strategist