Skip all navigation and go to page content


A Weblog from the NN/LM Outreach Evaluation Resource Center

Archive for the ‘Book Review’ Category

Telling Training’s Story: The Success Case Method

Wednesday, July 2nd, 2014

“On the average, it is true that most training does not work very well. But some programs work very well with some of the people, and this represents their great potential for being leveraged for even greater results.” Robert O. Brinkerhoff, “Telling Training’s Story.”

Most evaluation methods for program training reduce data to averages: the average number of things learned by participants; the average number of techniques applied on the job; the average number of times a skill was used post-training. Unfortunately, this approach can underestimate the true value of training for the organizations investing in the programs.

In Telling Training’s Story, Brinkerhoff writes that, in reality, the majority of participants gain little from training programs.  They either use some information but get no results, or they simply give up after a few attempts. Sometimes poor instructional design is to blame. More often, low success is caused by contextual variables, such as lack of supervisory support, no opportunity to try out the learning, or program timing. In fact, good instructional design often cannot compensate for these environmental crosscurrents.

Yet, Brinkerhoff argues that most training programs can boast a few success cases. There are usually a handful of participants (sometimes more) who apply their new knowledge or skill to produce valuable results for their organizations.  Sometimes the value of their contributions justifies program cost. Or, if the percentage of success cases was boosted by just 10%, the investment would be worthwhile to the organization.

To truly evaluate a training program, you need to identify any positive outcomes that occur, even if they are traced to a small number of participants, and assess the value of those results. You also need to determine what instructional and contextual factors influence successful use of training information. Then, organizations can make informed decisions about continuing to invest in training.

Brinkerhoff’s Success Case Method (SCM) was designed for in-depth analysis of training programs and their outcomes. The method focuses on high and low success cases. High success cases refer to incidents where participants applied training program information and attained positive results for their organizations. Low success cases are situations in which participants demonstrated no application of the training information.

Investigation of the high-success cases identifies the best possible outcomes that occur when employees apply information gained from the training program. By adding low-success cases into the mix, the method also leads to a thorough understanding of key factors, both in training design and in the organizational context, that influence participants use of their new capabilities.

Brinkerhoff’s book Telling Training’s Story provides step-by-step guidance for conducting SCM studies. Steps include working with stakeholders to define success and value; developing a program impact model, using rigorous sampling methods; and testing rival hypotheses for your findings. By following these steps, you can present a case study with compelling evidence of your program’s value. If you find your program is ineffective, the process will illustrate the factors working against its success.

So, the next time you want to evaluate a training program, consider going beyond “average.”  Check out the Success Case Method.

Source: Brinkerhoff RO. Telling training’s story. San Francisco, CA: Berrett-Koehler Publishers; 2006.


Book review: Focus Groups: A Practical Guide for Applied Research. (4th edition)

Tuesday, August 30th, 2011

I recently purchased a copy of “Focus Groups: A Practical Guide for Applied Research” by Richard Krueger and Mary Anne Casey. Krueger, professor emeritus at University of Minnesota, has written some of the classic books on focus group research and his co-author has conducted focus groups for government agencies and nonprofits. The experience of these two authors shines through in the pages of this well-organized, thorough text, which has a lot to recommend it:

  • The operative term in the title is “applied research.” The authors talk about the purpose of the study being the “guiding star” for selecting participants, writing the question guide, deciding on moderators, and analyzing and reporting findings.
  • The content is full of nuts-and-bolts suggestions, including a very practical chapter about Internet and telephone interviews
  • There is an interesting chapter presenting four different approaches to focus group research: marketing research; academic research; public/nonprofit; and participatory. The chapter summarizes the evolution of the approaches and compares them in a table that will allow the readers to choose the approach that best fits the circumstances of their studies.  This chapter explains why evaluators have different takes on how to conduct focus groups.
  • There is a nice chapter on analyzing focus group data. It can be difficult to find step-by-step descriptions of how to analyze qualitative data, so this chapter alone is a reason to read this book. (You could generalize the process to analyzing other forms of qualitative evaluation data.)
  • The final chapter provides you with responses to challenging questions about the quality of your focus group research. For example, what do you say if someone asks “Is this scientific research?” and “how do you know your findings aren’t just your subjective opinions?” Along with suggesting responses, the authors provide their own analysis of why such questions are often posed and the assumptions lurking behind them. This section will help you defend your project and your conclusions. (It would be most helpful to read this chapter before you design your project because it helps you understand the standards for a defensible project.)

I recommend this book to anyone planning to run focus groups. I have conducted my fair share of discussions, but I learned new tips to use in my next project.

Reference: Krueger RA. Casey MA. Focus groups. A practical guide for applied research. 4th ed.Thousand Oaks, CA: Sage, 2009.

Last updated on Saturday, 23 November, 2013

Funded by the National Library of Medicine under contract # HHS-N-276-2011-00008-C.