Failure-triggered training, or, secret shoppers go phishing

At the Innovations in e-Learning Symposium this week, Dan Bliton and Charles Gluck from Booz Allen Hamilton presented a session on “failure-triggered training.” I was really impressed by their description of a study that explored different approaches to reducing the risk of phishing attacks in a corporate setting. For one thing, as I told Charles immediately after the session, they invented the flip side of a job aid.  But I’m getting ahead of myself.

In this post:

  • Their session description (from the symposium brochure)
  • My summary of the session, with a few excerpts from their presentation
    (I’ll repeat this link a few times in this post; all those links are for the same set of materials. You don’t need to click more than once.)
  • (At least) three implications for improving performance

The session description

Study Results: Failure-Triggered Training Trumps Traditional Training

We didn’t expect our highly interactive eLearning (that generated great post-test scores) to be completely ineffective in changing behaviors in the work environment! Could the same eLearning be made effective if delivered as failure-triggered training? Come learn the outcomes of a blind study of nearly 500 employees over nine months which analyzed multiple training approaches. The study shows that the same eLearning was significantly more effective when delivered as spaced events that employed learning at the point of realization. This combination of unannounced exercises and failure-triggered training (a See-Feel-Change approach) significantly reduced improper responses to phishing attacks by 36%.

I didn’t ask Bliton or Gluck about this, but “see-feel-change” seems related to what John Kotter talks about here: making a seemingly dry or abstract concept more immediate and concrete.

What I heard: BAH’s study

(Note: this is my own summary. I’m not trying to put words in their mouths, and may have misunderstood part of the session. If so, that’s my fault and not theirs.  In no way am I trying to take credit either for the work or for the presentation by Dan Bliton or Charles Gluck.)

The Booz Allen Hamilton (BAH) study, involving 500 employees over 9 months, analyzed different training approaches to “phishing awareness.”  The training aimed at making employees aware of the risks of phishing attacks at work, with the goal of reducing the number of such attacks that succeed.

The study wanted to see whether interactive awareness training produced better results than static, page-turner training. In addition, the study used fault-triggered training, which Bliton and Gluck explain this way:

Unannounced, blind exercises [simulated phishing attacks] delivered in spaced intervals, combined with immediate, tailored remedial training provided only to the users that “fail” the exercises.

In other words, if you click on one of the fake phishing attempts, you immediately see something like this:

BAH phishing failure

BAH divided the study participants into three groups:

  • The control group received generic “training” about phishing that did not tell them how to respond to attacks.
  • The wiki group’s training consisted of a non-interactive pageturner, copied from a wiki.
  • The interactive group’s training included practice activities (how to identify likely phishing, how to respond).

In post-training comments, the Interactive group gave their training an overall score of 3.8 out of 5.  As the presenters noted somewhat ruefully, the Wiki group gave theirs 3.7  — and the control group gave theirs 3.4.  (See slide 11 in the presentation materials.)  The page-turning Wiki group actually felt better prepared to recognize phishing than the Interactive group.

Posttest questions indicated that 87.8% of the Wiki group and 95.6% of the Interactive group knew whom to notify if they spotted suspicious email.

From the response to the first simulated attack, however, Dan and Charles learned there was no significant difference between the three groups (Control, Wiki, Interactive) — nearly half the participants in each group clicked the link or replied to the email.

What happened next at BAH

Over six months, participants received three “exercises” (mock phishing attempts). “Failure” on these exercises consisted of either clicking an inappropriate link (producing an alert like the example above) or replying to the email — hence, “failure-triggered training.”

The study provide good data about actual performance, since it captured information like who clicked a link or replied to the simulated phishing.

Incorrect responses fell dramatically between the first and second exercises, and further still between second and third:

Booz Allen results from failure-triggered training

 Bliton and Gluck attribute this decrease to two main factors: the spaced-learning effect produced by the periodic exercises, and “learning at the point of realization,” since what you could think of as failure-feedback occurred just after someone responded inappropriately to what could have been an actual phishing attack.

If you’re familiar with ideas like Gottfredson and Mosher’s Five Moments of Need, which Connie Malamed summarizes nicely, this is #5 (“when something goes wrong”).

I’ve left out plenty; if you’ve found this description intriguing, take a look at their presentation materials. I can tell you that although Bliton and Gluck’s presentation at IEL12 had a relatively small audience, that audience really got involved: question, opinions, side conversations–especially striking at 4 o’clock on the last afternoon of the symposium.

What I thought, what I think

This approach is much more than training, in the sense of a structured event addressing some skill or knowledge need. I told Charles Gluck that it’s almost the flip side of a job aid.  A job aid tells you what to do and when to do it (and, of course, reduces the need to memorize that what-to-do, since the knowledge is embedded in the job aid).

At first I thought this approach was telling you what not to do, but that’s not quite right, because you just did what you shouldn’t have.  You can think of it  as being like a ground fault circuit interrupter (GFCI), a special type of safety device for an electrical circuit.

GFCIs can respond to a problem too small for a circuit breaker to detect. So you’re blow-drying your hair, when click! the wall outlet’s GFCI trips, safely breaking the circuit and interrupting your routine.  Not only do you avoid a shock; you also have feedback (if you know about how GFCIs work) that you’d been at risk from electrical hazard.

In the same way, BAH’s mock-phishing exercise interrupts the flow of work. By following the interruption with immediate, relevant, concrete feedback, as well as an offer for further details via a brief training program, this short circuit is turned into a smart circuit.

Which to me opens the door to — let’s use a different term instead of “failure-triggered” — task-triggered performance support. Like a virtual coach, the BAH exercises detect whether I responded inappropriately and then help me not only to recognize and even practice what to do instead.

What I’m leaving out

This was a study and had limits.  For one thing, because of the failure-trigger, we don’t know much about the people who didn’t click on the phishing attempts: have they really mastered this skill, or did they just not happen to click on these trials?

There’s also some data about the best response (don’t click the link, do report the attempt), though the numbers seem very small to me.  (I don’t recall anyone asking about the details on this topic, so I could well be misunderstanding what the numbers represent).

BAH study: attempt reported

On the corporate-culture side, what happens within the organization?  Does this seem Orwellian?  Can the organization apply it as formative feedback intended to help me improve, or do I just end up feeling that Somebody’s Watching? I’d like to look for some data about the effects of retail mystery-shopper or secret-shopper programs, a similar activity that can seem either like trickery or like process improvement.

What about habituation? Will the effectiveness of this approach fade over time?

Most intriguing: can you harness this as a form of ongoing training?  For example, along with informing people about some new security threat, create and send out further exercises exemplifying such a threat. Their purpose would be to provide a kind of on-the-job batting practice, with “failure” producing two-part feedback (“You missed this security threat, which is…” “To find out more, do this…”).

Dan Bliton, Charles Gluck, and their colleagues have done more than make BAH more secure from phishing.  They’ve also shared a creative, practical experiment.


When to build a job aid: watch out for obstacles

This entry is part 3 of 3 in the series When to Build a Job Aid.

The first two parts of this series, in one line each:

  • Is a job aid mandatory? If not, does speed or rate on the job prohibit the use of a job aid?
  • Do the characteristics of the task tell you that a job aid makes sense?

If they do, you might feel ready to leap right into design.  But in the real world, people don’t just perform a task; they work within a complex environment.  So the third part of your decision is to ask if any obstacles in that environment will hamper the use of a job aid.

Part 3: what obstacles does your job aid need to overcome?

You could ask these question in either order, but physical barriers are sometimes easier to address than social ones.

Often people have to work in settings where a job aid might be a hindrance or even a danger.  Someone repairing high-tension electrical lines, for example.  Or someone assembling or disassembling freight trains at a classification yard:

You don’t need to watch this video about humping railroad cars, but as the narrator points out around the 4:00 mark, in the distant past a worker would have to ride each car as gravity moved it down a manmade hill (the hump), applying the brake by hand if the car was moving faster than about 4 mph. It would have been impossible to give the brakeman a job aid for slowing the car, so his training (formal or otherwise) would have required lots of practice and feedback about judging speed.  And possible trial and error.

Amarillo by morning?
Texas highway map, 1936

Rather than develop impractical job aids for aspects of this set of tasks, modern railroads rely on computers to perform many of them.  For example, radar monitors the speed of cars more accurately than a person could, and trackside retarders act to moderate that speed.

Remember, the goal is not to use job aids; the goal is to produce better on-the-job results.  Sometimes you can do that by assigning difficult or repetitive tasks to machinery and automation.

In many cases, though, you can overcome physical obstacles to the use of a job aid  by changing its form.  No law requires a job aid to be on an 8 1/2 by 11 inch laminated piece of paper. Nor on the formerly ubiquitous, multifolded paper of a highway map.

A road map can support different kinds of tasks.  You can use it at a table to plan where you’re going to go, to learn about the routes.  No barriers to such use.  But for a person who’s driving alone, a paper road map is at best a sub-optimal support.  It’s hard to use the map while trying to drive through an unfamiliar area.

In a quarter mile, turn left
Deep in the heart of Oslo

Real-time support for the driver now includes geosynchronous satellites, wireless technology, a constantly updated computer display–and a voice.

That voice is transformative: it’s a job aid you don’t have to read. Because the GPS gives timely, audible directions, there’s no need to take your eyes off the road and decipher the screen.

Other examples of overcoming physical barriers: attach the job aid to equipment. Use visual cues, like a change of color as movement or adjustment gets closer to specification.  Combine audio with voice-response technology (“If the relay is intact, say ‘okay.’ If the relay is damaged, say ‘damaged.'”)

But he had to look it up!

Overcoming physical barriers is one thing.  Overcoming social barriers is…a whole bunch of things. Your job aid will fail if the intended performer won’t use it.

Popular culture places a great value on appearing to know things.  When someone turns to an external reference, we sometimes have an irrational feeling that she doesn’t know what she’s doing–and that she should.  In part, I think we’re mistaking retention of isolated facts with deep knowledge, and we think (reasonably enough) that deep knowledge is good.

Don't go off on the wrong track.At its worst, though, this becomes the workplace equivalent of Trivial Pursuit. A railroading example might be someone who can tell you not only the train numbers but the locomotive numbers that ran on a certain line decades ago–but who can’t issue you a ticket in a prompt, accurate, courteous manner.

The performer herself may be the person believing that performance guided by a job aid is somehow inferior.  Coworkers may hold it, putting pressure on the individual.  Even clients or other stakeholders may prefer not to see the performer using a job aid.

Maybe there’s a way around this bias.  The job aid could be embedded in a tool or application, such that the performer is merely applying one feature.  That’s essentially what a software wizard does.  Watch me turn this data into a chart–I just choose what I want as I go along.

(And doesn’t “choose what I want” sound much more on top of things than “look stuff up?”)

For a injection gun used for immunizations in third-world settings, healthcare workers occasionally had to make adjustments to clear jams and similar equipment glitches.  Some senior workers did not want to seem to need outside help to maintain their equipment, but couldn’t retain all the steps.  (Remember in Part 2?  Number of steps in task, complexity of steps?)  So the clearing instructions were attached to the equipment in such a way that the worker could follow the job aid while clearing the gun.

♦ ♦ ♦

The considerations here aren’t meant as either exhaustive or exclusive.  They are, however, important stops to make, a kind of reality check before you hit the on-ramp to job aid design.  The reason for building a job aid is to guide performance on the job while reducing the need for memorization, in order to achieve a worthwhile result.  If the performer can’t use it because of physical obstacles, or won’t use it because of social ones, the result will be… no result.


CC-licensed photos:
1936 Texas highway map by Justin Cozart.
Norwegian GPS by Stig Andersen.
1879 Michigan Central RR timetable from the David Rumsey Map Collection.

Job aid: the Scrooge-O-Meter

The Scrooge-O-Meter from LSS Financial Counseling Service is an example of a calculator job aid.  Calculators guide someone through a task by prompting for numerical values and performing calculations. The idea is to help a person reach some conclusion without having to master the factors or the math involved.

The Scrooge-O-Meter

(LSS Financial Counseling Service is part of the work of Lutheran Social Service of Minnesota.)

Who uses this job aid?

Most likely someone trying to learn the added financial burden of buying on credit.  (See additional thoughts from the group that created it, later in this post.)

What is the task supported?

I would say “awareness” or even “empowerment.”  The goal is to help someone understand the additional cost of purchasing on credit.  I filled in the numbers you see in this example.  The result says to me that “spreading out” credit payments for my holiday buying makes those purchases nearly 10% more expensive than I’d thought.

Notice that it doesn’t render judgment (“$68.72 extra?  Are you nuts?!?”).  The job aid simplifies the process so I can more readily see and understand the impact of buying on credit.  I’m free to make my own decisions about what to do next.

More about the Scrooge-O-Meter

LSS Financial Counseling Service wants consumers to know that they can turn to a national network of nonprofit financial counseling and debt management (FCS is a member of that network).  The page with the Scrooge-O-Meter offers a toll-free number, online counseling, a newsletter, and other resources.

Darryl Dahlheimer, program director of LSS Financial Counseling Service, was kind enough agree to its appearing here and also to provide these details:

There are many tools to help consumers calculate credit card repayment, but here are three reasons we like this one:

  1. It sets a playful tone, to overcome the shame/intimidation of finances for so many who feel “dumb about money” but want to learn.
  2. It helps make the true cost of using credit visible.  Plug in an example of buying that $500 iPad at a major store on their 21% interest credit card and then paying only the $15 minimum each month. You will pay a whopping $757 and take over four years to pay off.
  3. Conversely, it allows you to see the tangible benefits of paying more than minimums.


When to build a job aid: ask the task

This entry is part 2 of 3 in the series When to Build a Job Aid.

The previous post in this series covered the initial go/no-go decisions: are you required to build a job aid?  Does a need for rate or speed make a job aid impractical?

If the answer in both cases is no, then you don’t have to build a job aid, yet there’s no reason not to (so far).  A good way forward at this point is to consider the characteristics of the real-world performance you have in mind.  This is related to though not the same as task analysis.  I have my own name for it:

What that means is: use what you know about the task to help determine whether building a job aid makes sense.  You can go about this in many ways, but the following questions quickly cover a lot of the territory.

♦ ♦ ♦

How often does someone perform the task?

“Often” is a relative term–in fact, most of the questions in Ask the Task are relative.  That doesn’t mean they’re not pertinent.  Asking “how frequent is frequent?” turns your attention to the context of the task and the people who typically carry it out.

Frequency isn’t the same thing as regularity.  Some tasks are frequent and predictable, like a weekly status update.  Some are more random, like handling a payment by money order.  And some are much more rare, like a bank teller in Vermont handling a money transfer from Indonesia.

Whether you end up building a job aid, designing training, or just tossing people into the deep end of the performance pool, you need some idea of how frequent “frequent” is, and where the specific task might fall along a job-relevant frequency scale.

Think about what frequency might tell you about whether to build a job aid.  Yes, now.  I’ll tell you more at the end of the post, but we both know you ought to do some thinking on your own, even if we both suspect few other people will actually do that thinking while they read this.

♦ ♦ ♦

How many steps does the task have?

It’s true, some tasks don’t really seem to have steps.  Or they have very few: look up the arguments for the HTML <br> tag.  And some tasks have so many that it might make sense to break them up into logical subgroups: setting up the thermoformer.  Testing the thermoformer.  Troubleshooting problems after the test.

Think of “step” as the lowest level of activity that produces a result that makes sense to the performer on the job.  If I’m familiar with creating websites, then “create a new domain and assign it to a new folder in the \public_html directory” might be two steps (or maybe even one).  If I’m not familiar with creating websites, I’m going to need a lot more steps.

That makes sense, because a job aid is meant to guide a particular group of performers, and the presumption is that they share some background.  If you have widely differing backgrounds, you might end up with two versions of a job aid–see the Famous 5-Minute Install for WordPress and the more detailed instructions.  Essentially, that’s two job aids: one for newcomers (typically with more support) and one for more experienced people.

As with frequency, you need to think about how many steps the task involves, and whether you think of those as relative few steps, or relatively many.

♦ ♦ ♦

How difficult are the steps?

You can probably imagine tasks that have a lot of steps but not much complexity.  For someone who’s used to writing and who has solid, basic word processing skills, writing a 25-page report has plenty of steps, but few of them are difficult (other than getting reviewers to finish their work on time).

In the same way, a task can have relatively few steps, but many of them can be quite difficult.

That’s the reason for two step-related considerations when you Ask the Task whether a job aid makes sense: how many? How hard?

Pause for a moment and think which way you’d lean: if the steps in a task are difficult, does that mean “job aid might work,” or does that mean “people need to learn this?”

♦ ♦ ♦

What happens if they do it wrong?

This question focuses on the consequences of performing the task incorrectly.  Whether a person has a job aid or not is immaterial–if you don’t perform correctly, what happens?  Personal injury? Costly waste or rework? Half an hour spent re-entering the set-up tolerances? Or simply “re-enter the password?”

As with the other questions, you need to think about the impart of error in terms of the specific job.  And, if you haven’t guessed already, about the relationship between that impact and the value of building a job aid.

♦ ♦ ♦

Is the task likely to change?

We’re not talking about whether the job aid will change, because we still haven’t figured out if we’re going to build one.  We’re talking about the task that a job aid might guide.  What are the odds the task will change?  “Change” here could include new steps, new standards, new equipment, a new product, and so on.

♦ ♦ ♦

Ask the task, and the job aid comes out?  Right!

You’ve probably detected a pattern to the questions.  So the big secret is this:

The more your answers tend to the right, the stronger the case for a job aid.

What follows is the 90-second version of why.  (As your read the lines, just add “all other things being equal” to each of them.)

  • The less frequently someone performs a task, the likelier it is that he’ll forget how to do it.  If you’re an independent insurance agent whose practice mostly involves homeowner’s and driver’s insurance, and you write maybe six flood insurance policies a year, odds are that’s not a task you can perform without support. Job aids don’t forget.
  • The more steps involved in the task, the more challenging it will be for someone to retain all those steps correctly in memory and apply them at the right time. Job aids: good at retention.
  • The more difficult the steps are, the harder the performer will find it to complete each step appropriately. A job aid can remind the performer of criteria and considerations, and even present examples.
  • The higher the impact of error, the more important it is for the performer to do the task correctly.  You certainly can train people to respond in such circumstances (air traffic control, emergency medical response, power-line maintenance) , but often that’s when the performance situation or the time requirement presses for such learning.  Otherwise, a well-designed job aid is a good way to help the performer avoid high-cost error.
  • The more changeable the task, the less sense it makes to train to memory.  Mostly that’s because when the change occurs, you’ll have to redo or otherwise work at altering how people perform.  If instead you support the likely-to-change task with job aids, you’re avoiding the additional cost of full training, and you mainly need to replace the outdated job aid with the new one.

Here are the ask-the-task questions, together once more:

Part 2: ask the task

When to build a job aid: go / no-go

This entry is part 1 of 3 in the series When to Build a Job Aid.

If you’re wondering whether you should build a job aid to support some task, this is the first of a three-part guide to help you figure things out.

Should you, or shouldn't you (part 1)?

That first consideration (“Is a job aid required?”) isn’t as daft as it might seem.  If your organization mandates a job aid for some task, then you’re stuck.  You want to do the best job you can with it (or maybe you don’t), but unless you convince the right people to reverse the policy, somebody’s going to be building a job aid.

Which means you can skip the rest of the “should I build?” stuff that will appear in Parts 2 and 3.

Assuming that a job aid isn’t mandatory, the next question is whether speed or rate is a critical factor in performing whatever the task is.  The short answer is that if speed matters, a job aid isn’t going to work.

Wing tips up, feet down, watch for that wave...First, when it comes to routinely high-volume work like factory production or air-traffic control, that normal high-volume state doesn’t allow the performer time to consult a job aid.  Successful results depend on learning–on committing skill and knowledge to memory, and on retrieving and applying those things appropriately.

I’m a pretty fast typist (65 – 80 words per minute if I’ve been writing a lot), but the moment I glance down at the keyboard my rate drops, because the visual signal interferes with the virtually automatic, high-rate process I normally use at a keyboard.

That’s rate.  As for speed, many jobs call for you to apply knowledge and skill  in an unscheduled fashion, but quickly.  Think about safely driving a car through a tricky situation, much less an emergency.  You don’t have the opportunity to consult a job aid.  If a kid on a bike suddenly pulls out in front of you, you can’t look up what to do.

Anyone who’s helped train a new driver knows what it’s like when the novice is trying to decide if it’s safe to turn into traffic.  We experienced drivers have internalized all sorts of data to help us decide without thinking, “Yes, there’s plenty of time before that bus gets here; I can make the left turn.” In the moment, the newcomer doesn’t have that fluency but has to be guided toward it–just not via a job aid.

What’s next?

Once you’ve determined that you’re not required to build a job aid, and that there’s no obstacle posed by a need for high speed or high rate, you’ll look at the nature of the performance for clues that suggest job aids.  That’ll be the next post: Ask the Task.

CC-licensed image of seabirds by Paul Scott.