Seat Time or Outcomes: What’s More Important?

The students want to learn the knowledge and skills necessary to become a firefighter or EMT. The exact same scenario unfolds for any training session, be it a three-hour continuing education (CE) session or a 12-hour weekend seminar.

Merriam-Webster defines the verb, “training” as: a process by which someone is taught the skills … for an art, profession, or job and the noun, “training” as the skill, knowledge, or experience acquired by one that trains. Similarly, Merriam-Webster defines the verb “learn” as: to gain [acquire] knowledge or skill … and the noun “learning” as: the activity or process of gaining knowledge or skill. These definitions, when simplified and combined could say that training is “the process by which someone acquires a skill or knowledge” and learning is the actual “knowledge or skill” acquired by the students. Therein lies the core and essence of education: the acquiring of knowledge or skills.

A well designed educational session has clearly defined a goal, objectives to meet that goal, and can show learner mastery of the objectives in outcomes. The goal is an abstract statement about what should happen as a result of the training session. Objectives are statements that describe the expected knowledge, skills, competencies or attitudes the learners will acquire in measurable terms. Outcomes are the actual knowledge, skills, competencies, or attitudes acquired through the training session.

Therefore, if the purpose of education is to acquire knowledge or skills — goals and objectives — then an educator/trainer is responsible for certifying that the student/learner has actually acquired (outcomes) the knowledge or skills. They accomplish this by measuring learner outcomes with assessments that “evaluate, measure, and document learning [and] skill acquisition.” (Educational Reform) Thereby proving that the individual has achieved the goal by meeting the objectives. Notice, not once have we mentioned anything regarding time; in class, on task, in review or otherwise.

The amount of time necessary to achieve the defined outcomes is truly an individual variable. “Recommended” hours are assigned to EMT and fire courses based on an average time for the instructor to effectively deliver a multimodal educational session addressing the different types of learners — visual, auditory, kinesthetic. But what if the learners were able to identify their preferred learning style and remove the time used teaching to other styles? Some learners are already doing this and have been for years. Remember that guy who regularly slept, daydreamed, or was always texting through class but somehow managed to score well on all the assessments? He figured out how he best acquired the knowledge and skills and would “turn off” when a teaching modality was not conducive to his style.

How long did it take you to complete your certification? Yes the class met for a certain number of hours, but how long did you actually spend in class? Did you miss any sessions? Were you engaged and paying attention the entire time you were in class? What about homework, reading and study time? That is above and beyond the number of hours you were scheduled in class. Did you have to review anything a second or maybe even a third or fourth time before you finally understood it? I personally took three different 12 lead interpretation classes, including classes from Bob Page, Tim Phalen, and local cardiology experts before I was finally able to understand it while colleagues took right to it after one class. Does that make me three times the EKG paramedic? Is someone twice the EMT because he or she took an EMT class a second time? Of course not, as a matter of fact, the question itself seems kind of silly right? The amount of time spent in the educational process does not directly or accurately reflect the amount of knowledge or skills acquired yet we are still mandating hours not outcomes.

Most certifications/licenses still have continuing education requirements based on hours, not outcomes. Most educational institutions still require EMTs and paramedics to complete 96 hours of CE every four years, Refresher courses are set at 48 hours, firefighters have to keep 36 hours of CE every year to maintain death benefits. Each firefighter certification class has a defined minimum and maximum number of hours. Some EMS programs require paramedic students to obtain (NCOEMS recommended minimum) 200 hours of clinical time and then say they are ready to be paramedics. Apparently 200 hours of ride time is sufficient for everyone to achieve entry-level competency. Dave Page wrote an article, “The Quest for Competence,” on this very thing. Take some time to read it if you haven’t.

Think about it in terms of continuing education and the needs of each individual. For example, Janie is a paramedic that works for a cardiac transport team and obtains countless cardiac and airway educational sessions annually. However, it has been three years since she completed any CE on or even responded to any type of trauma. Zack is a paramedic in a neighboring county that teaches ITLS several times a year. However, he seldom responds to cardiac calls and is self-described as “rusty at best” with reading 12 Lead ECGs.

Under most CE models, Zack and Janie are expected to complete the same number of hours in each topic, even if they are experts in one particular area, they still must show hours for the area. Under a competency-based program, Zack and Janie would each have CE customized to their specific needs. Training officers or even adaptive testing could identify objective competencies and weaknesses followed by an educational plan to remediate the individual. Zack may need to review four specific objectives in cardiology while Janie needs to review all of the objectives in trauma.

How do we know that Zack and Janie have mastered the outcomes? This is determined through valid and reliable assessment. Whether it is a psychomotor assessment — practical skill station — or cognitive assessment, John Q. EMT or Suzie firefighter’s outcomes are proven by their performance on the assessments.

Assessments come in many forms, most commonly multiple choice tests and practical skill station tests. Other methods of assessment include the use of matching questions, multiple answer, and short answer, essay, written papers, projects, group work and even oral interviews/boards. The assessment, if properly designed, will assess and document the learner’s mastery of the objectives and achievement of the course goal.

A properly designed assessment will be valid and reliable. Where reliability simply means that the assessment will yield consistent results and validity means the assessment actually measures the objectives for which it was written. The assessment can then be defended when saying the learner has mastered the necessary outcomes.

In order to better understand outcomes, objectives, and assessments, let’s take a quick look at how an educational topic (course) is built. Let’s assume we wanted to build a certification course for wood splitting. (Disclaimer/Note: For the purpose of this article, we are going to skip through the learner analysis phase and move into the design and development of the material.) We have formed a working group of the state’s experts in wood splitting. They meet and discuss their craft and identify a list of essential skills and knowledge for the Wood Splitter Apprentice (WSA) certification. This list is then converted into measurable objectives written by an instructional designer. Once the objectives exist, a subject matter expert and instructional designer work together to create an educational outline. They finish up by writing multiple assessments including a final exam and skill station test. Let’s take a quick look at the objectives and then the assessment of those objectives. Notice how each objective can easily be measured to document the outcomes.

Cognitive Objectives

At the end of this course the learner will be able to:

1.1 Differentiate hard wood and soft wood.

2.1 Define seasoned wood.

3.1 Identify the appropriate length for pre-split logs.

Psychomotor Objectives

4.1 Given an axe, the learner will be able to split seasoned hardwood.

5.1 Given a wedge and sledgehammer, the learner will be able to free a stuck axe from an unseasoned log.

The Assessment

(Objective 1.1)

1. You are working at the site when a customer approaches you and says, “…can you tell me the difference in hard wood and other types of wood?” what would you say? (Write your response in the area provided below)

(Objective 2.1)

2. In your own words, how would you define “seasoned wood?” (Write your response in the area provided below)

(Objective 3.1)

3. Which of the following lengths of logs are appropriate for seasoning before being split?

1. 10”
2. 24”
3. 18”
4. 36”

The practical skill component would have a detailed skill evaluation sheet for each skill station that evaluates the learner’s ability to perform the skills listed in the outcome. The learner would be brought into the practical station and evaluated by a trainer while completing the station.

Take a quick look at the hypothetical course we just designed. Do you notice how everything is built around the objectives? Look at how the assessments are very similar to the objectives and are easily tied to them to show outcomes. This is the exact same way any firefighter or EMT course is designed. A goal is set, objectives are identified, the material is created, and assessments are built. Sometimes everything comes in one package while others allow the educator flexibility in building the material and assessments.

Notice I have not mentioned anything about the delivery method — traditional, online, hybrid, flipped etc. Why? Because we are talking about a course, irrespective of the delivery method, the design of the course is always the same: Write the objectives, design the course, build the assessments, launch the course and evaluate it.

Where We Are Now

NCOEMS lists all of their hours for initial programs as “Recommended” and not “required” allowing for institutional and individual variance. Additionally, years ago (circa 2007) NCOEMS removed the mandatory 96 hours of CE in favor of a more competency/need base approach leaving the credentialing requirements up to each individual institution — most of which continue requiring 96 hours. System based CE has fallen away from that strictly defined 96-hour model and replaced it with need based CE as identified by the local QA/QI process.

The National Registry of EMTs (NREMT) adopted a competency-based model several years back by allowing credentialing through testing. Rather than keeping track of and filling out their convoluted recertification packet, a Nationally Registered Paramedic can take the written test and be done with recertification. Why is this an option? Because it is reliable and effectively mapped to the paramedic objectives, the written test is valid. The NREMT is in effect saying that the person taking the test has a level of competency (entry level) worthy of credential.

Accredited colleges and universities all over the world allow “credit through testing.” College Level Examination Program (CLEP) programs are in place allowing students to receive college level credit by taking an assessment. The assessment is a comprehensive assessment of the course objectives similar to a final exam. This is basically a method of acknowledging the learner’s current knowledge and skills regardless of how they were obtained. The learner that successfully passes the assessment then receives college credit for the course, just as if he/she took an entire semester of it.

Where We Can Go

Outcome/Competency-based programs and assessments could dramatically shorten the time it takes for some learners while lengthening time for others. One particular educational model would use outcome-based assessments to create a customized educational program while giving the learner credit for current knowledge.

Initial courses would have the learner begin the program by taking the comprehensive assessment for each topic until he can’t pass one. He would then be directed to the training module for that section. After completing and passing the assessment, he picks back up with the assessments until another weakness is identified. (See Figure “Outcome Based Flow Chart.”) This method would effectively allow the learner to “test out” of certain topics while being directed to complete others. This could work well for those firefighters with years of experience but lacking time to complete official courses or nurses wanting to become paramedics and vice-versa.

Continuing Education could identify individual weaknesses and direct learners to spend their energy improving those weaknesses instead of allowing them to continue weakening. Thereby improving the overall readiness of the responders and safety of their districts.

At the end of the day, does someone stuck inside a burning building or pinned in a mangled car really care how long the firefighter or EMT spent in class? Or is it more important that they have the knowledge necessary and are able to safely perform the tasks to mitigate the situation? They care about the outcome, not how long it takes to achieve it. Why then are we so entranced with the seat-time and not focusing on the outcomes?

References for this article are available upon request at editing@carolinafirejournal.com.