A discussion on lrnchat included lots of comments and questions about data collected about people’s performance, particularly in training, testing, or learning situations.

I’m always inclined to say you can’t do evaluation if you don’t measure, which means I quickly exasperate people who think evaluating is measuring.  For them, perhaps it is.  For me, measurement is a kind of quantification (Conor weighs 187 pounds, Raylene booked $4.7 million in  sales last year), while evaluation is your comparison of the measurement with some standard (Conor is overweight, Raylene made 125% of quota).

That seems straightforward, except for a depressing tendency to assume we’re all using the same standard–and that tendency’s sidekick, the assumption that our measures take in the right requirements.  In that lrnchat discussion, Jane Bozarth mentioned an online course where the instructor based his evaluation in part on the number of comments a student posted.  Naturally, someone set out to put up 100 meaningless posts.

What to do?  Well, you could turn to Tom Gilbert, who mused about what he called the dimensions of performance measuring back in 1978 (and probably long before that).  He saw three classes (or requirements) for measurement: quality, quantity, and cost.

“When we measure an accomplishment, any one or more of these requirements may be relevant, and one of our principle tasks is to identify them.”  In other words: figure out what’s important about the desired performance, which will help you determine what to measure and the standard to use.

Gilbert saw three possible aspects to each of these dimensions.

Quality, for instance, can involve:

• Accuracy–how well does the accomplishment match a model without errors?
• Class–is the accomplishment superior to most in some way beyond accuracy?
• Novelty–does the accomplishment demonstrate originality?  Does it embody features or aspects that distinguish it favorably in particular dimensions?

Quantity or productivity can involve:

• Rate –accomplishments per unit of time.
• Timeliness–accomplishment by some end point.
• Volume–accomplishment when time is not a significant factor (e.g., sales per month).

Cost:

• Labor–the amount spent for the labor and associated items directly related to the accomplishment.
• Material–supplies, tools, equipment, and so on.
• Management–the cost of supervision, administration, and support related to the accomplishment.

As Gilbert points out, the requirements are relevant only when people’s accomplishments vary based on the requirements.  So running a 10K doesn’t normally involve accuracy.

Framing a custom home involves timeliness, and could possibly involve rate, but most often novelty wouldn’t be a requirement. However, class as a quality measure might apply, if the craftsman needs to adapt quickly and successfully to changing conditions: “Leo, this suite needs to be wheelchair-accessible.  Can we move the doorway?”

I think it’s useful to have these categories in mind regardless of the type of work you’re considering.  But don’t take my work for it.  Here’s Gilbert:

…Jobs which seem unmeasurable are actually mesurable once we identify their accomplishments and relevant requirements.  Many jobs that people say cannot be measures (“you can’t measure show-horse breeding–it’s an art”) seem that way only because we are thinking of behavior rather than accomplishment.

How would we measure the behavior required in selecting good breeding stock?  I haven’t the faintest idea.  But we can measure results…

There’s more to say on this topic, but this will do for a start.

CC-licensed photos:
Measurement (of coffee) by flyzipper / Steve Michos.
Evaluation (of accomplishment) by afternoon / Ben Godfrey.