Archive for the ‘Practical Evaluation’ Category
When planning an evaluation that features interviews, it is difficult to know in advance how many interviews should be conducted. The usual approach is to continue to interview until you no longer hear new information. (Olney & Barnes, Data Collection and Analysis, p. 23) However, there are times when a numeric guideline would be very useful. For example, such a guideline would help in creating the budget for the evaluation section of a proposal when that evaluation features qualitative data collection. Guest, et al., conducted a research project in which they analyzed transcripts of sixty interviews and found that 94% of the coded topics that appeared were identified within six interviews. Analysis of six additional interviews revealed only another 3% of all the coded topics that were eventually found in the sixty interviews. They concluded that data saturation (the point at which no additional data are being found) occurred by the time they had analyzed twelve interviews. They point out that similar evidence has been presented by Nielsen and Landauer, who found that technology usability studies uncover 80% of problems after six evaluations, and 90% after twelve. In a later publication, Nielsen showed that usability testing with fifteen participants will uncover 100% of problems, but he recommends using a smaller number (“Why You Only Need to Test with Five Users” from Jakob Nielsen’s Alertbox, March 19, 2000).
Is twelve the magic number for interviews, then? Not necessarily. Guest, et al. caution that their study involved highly structured interviews with members of a relatively homogeneous group. In addition, the interview topics were familiar to the participants. With a heterogeneous group or a diffuse (vague) topic, more interviews will probably be needed. The bottom line is to select a purposeful group of interview candidates carefully. Olney and Barnes provide more details about purposeful sampling (Olney & Barnes, Data Collection and Analysis, p. 23).
The article by Guest, et al. is not open-access, but this is a link to the authors’ abstract: Guest, G., et al. “How many interviews are enough? : An experiment with data saturation and variability.” Field Methods, February 2006. 18(1):59-82.
You can find a useful list of Seven Practical Steps to Create an Effective SurveyMonkey Survey at the SurveyMonkey Blog. More detail is provided for each of these steps, which, in general, apply to creating any survey (regardless of mode of creation and distribution). Briefly, here are the steps:
I recently purchased a copy of “Focus Groups: A Practical Guide for Applied Research” by Richard Krueger and Mary Anne Casey. Krueger, professor emeritus at University of Minnesota, has written some of the classic books on focus group research and his co-author has conducted focus groups for government agencies and nonprofits. The experience of these two authors shines through in the pages of this well-organized, thorough text, which has a lot to recommend it:
- The operative term in the title is “applied research.” The authors talk about the purpose of the study being the “guiding star” for selecting participants, writing the question guide, deciding on moderators, and analyzing and reporting findings.
- The content is full of nuts-and-bolts suggestions, including a very practical chapter about Internet and telephone interviews
- There is an interesting chapter presenting four different approaches to focus group research: marketing research; academic research; public/nonprofit; and participatory. The chapter summarizes the evolution of the approaches and compares them in a table that will allow the readers to choose the approach that best fits the circumstances of their studies. This chapter explains why evaluators have different takes on how to conduct focus groups.
- There is a nice chapter on analyzing focus group data. It can be difficult to find step-by-step descriptions of how to analyze qualitative data, so this chapter alone is a reason to read this book. (You could generalize the process to analyzing other forms of qualitative evaluation data.)
- The final chapter provides you with responses to challenging questions about the quality of your focus group research. For example, what do you say if someone asks “Is this scientific research?” and “how do you know your findings aren’t just your subjective opinions?” Along with suggesting responses, the authors provide their own analysis of why such questions are often posed and the assumptions lurking behind them. This section will help you defend your project and your conclusions. (It would be most helpful to read this chapter before you design your project because it helps you understand the standards for a defensible project.)
I recommend this book to anyone planning to run focus groups. I have conducted my fair share of discussions, but I learned new tips to use in my next project.
Reference: Krueger RA. Casey MA. Focus groups. A practical guide for applied research. 4th ed.Thousand Oaks, CA: Sage, 2009.
The American Evaluation Association “Coffee Break” webinar on June 10 featured a comparison of Survey Monkey and Zoomerang, two well-known and respected web survey tools. Both feature free accounts–you can sign up and test drive them as part of deciding whether to move to the paid options. In both cases, the free accounts feature most of the system functionality but with limits on the numbers of questions and responses. The prices for paid accounts are similar for both. The presenters, Lois Ritter and Tessa Robinette, highlighted some differences between the two systems.
Survey Monkey was, as of June 10, the only online survey application that is 508 compliant and, because one subscription can share multiple surveys, it is good for work being conducted by different groups in multiple locations. Survey Monkey is available in a variety of languages and can be used with the iPhone.
Zoomerang can be used with multiple mobile devices and offers a fee-based survey translation service. Zoomerang is designed for a one account per user and project environment, and can provided rented lists for sampling frames on 500 attributes that purport to be representative of the “general population.” Zoomerang also has a nice analytic feature: “tag clouds” for thematic grouping.
For a thorough overview of how to conduct online surveys, consult the Autumn, 2007 issue of New Directions for Evaluation, number 115.
The American Evaluation Association “Coffee Break” webinar series is a benefit of membership in the association. Recordings of these 20 minute sessions are archived in the AEA’s Webinar Archive E-Library (a members-only site).
Some news from Survey Monkey’s Newsletter:
Professional (paid) subscribers can now make customized links in lieu of those long automatically-generated ones. And you can analyze data based on respondents’ answers by using the Filter by Response tool within the Analyze section. Professional subscribers can also create Custom Reports within the Analyze section.
The OERC promotes collaborative evaluation approaches in our training and consultations, so it is always nice to have published examples of collaborative evaluation projects. A recent issue of Academic Medicine features an article showing how a collaborative evaluation approach called Empowerment Evaluation was applied to improve the medical curriculum at Stanford University School of Medicine. David Fetterman, originaor of Empowerment Evaluation, is the primary author of the article. A key contribution of this evaluation approach — which is exemplified in the article — is the ongoing involvment of stakeholders in the analysis and use of program data. This article provides strong evidence that the approach can lead to positive outcomes for programs.
Citation: Fetterman DM, Deitz J, Gesundheit N. Empowerment Evaluation: A Collaborative Approach to Evaluating and Transforming a Medical School Curriculum. Academic Medicine 2010 May; 85(5):813-820. A link to the abstract is available at http://journals.lww.com/academicmedicine/Abstract/2010/05000/Empowerment_Evaluation__A_Collaborative_Approach.25.aspx
How important is it to transcribe focus group discussions? Dr. Rita O’Sullivan from the UNC-Chapel Hill School of Education sought an objective answer to that question. She and colleagues ran an experiment in which two co-facilitators ran seven focus groups and created summary reports of the discussions. Each co-facilitator produced a report for each focus group: one wrote a summary based on memory, handwritten notes and a transcript of the audio tape; the other wrote a summary using memory, notes and the audiotape. (Each facilitator prepared seven summaries, some using the first method and some using the second.) Then, 18 educational professionals who were enrolled in a graduate-level educational research class compared the pairs of summaries. Sixteen of the 18 reviewers found no substantive differences between the two versions of the summaries.
What does this mean for evaluators? The authors concluded that their findings, although preliminary, suggest that, for the typical program evaluation setting, transcripts are not necessary to produce useful focus group discussion summaries. The findings also make it hard to justify the transcription costs for focus groups in evaluation settings – because every dollar spent on evaluation is one not spent on the program.
Source: O’Sullivan et al. Transcribing focus group articles: Is there a viable alternative? 2004 November. Paper presented at the joint international meeting of the American Evaluation Association and the Canadian Evaluation Society, Toronto, Canada.
Someone recently asked me if SurveyMonkey forms are accessible to those with functional limitations and disabilities. In fact, SurveyMonkey received Section 508 certification in June 2008. According to the company’s Web site, they are the only commercial online survey application that has this certification.
While SurveyMonkey software automatically formats surveys to be accessible, there are a few practices that we need to follow to make sure SurveyMonkey questionnaires are user-friendly with screen-readers and other visual technologies. For instance, don’t add extra html coding to your questionnaire (e.g., to bold-face or italicize words) because screen-readers may read parts of html coding as text. Also, SurveyMonkey’s default color schemes are configured for maximum contrast to help low-vision users. Creating your own color schemes may make your forms less readable for this population. You can find more tips from SurveyMonkey for creating screen-reader friendly forms at this link.
AEA/CDC Training session: Utilization-focused evaluation
The first training session I took at the AEA/CDC Institute was Michael Patton’s Utilization-Focused Evaluation. This workshop was pitched primarily for evaluators who are sick of producing time-consuming evaluation report tombs that sit on shelves. (You’re thinking I should have written “evaluation report tomes,” but actually, those reports are where evaluation results go to die.) Patton commented that you could probably attach an executive report to 500 sheets of blank paper – or 500 pages from a phone book pulled from your recycling bin – and no one would ever notice because they never read past the executive summary.
Here’s some interesting food for thought: Patton said that the order of the evaluation standards (Utility, Feasibility, Propriety, and Accuracy) is deliberate: Utility, or usefulness to intended users, is listed first because it’s deemed the most important. So, in evaluation design, the evaluation’s usefulness should be considered ahead of its feasibility (practicality and cost effectiveness), propriety (legality, ethics, and concern for the welfare of others), and accuracy (technically adequate information about features that determine merit or worth of a program). All are important standards, but utility gets top ranking. (Definitions for the four evaluation standards are listed here at the American Evaluation Association web site.)
To enhance the utility of evaluation findings, Patton said it is important to identify the intended users and uses of the evaluation information at the beginning of the evaluation and create an action plan for use of evaluation results that takes the following into account:
· The decisions the evaluation findings are meant to inform
· Timing of those decisions
· The stakeholders who will see and respond to the data
The responsibility for facilitating use of the findings falls on the evaluation consultant (or whoever is in charge of conducting the evaluation.)
If you are interested in learning how to conduct more useful evaluations, I recommend Patton’s Utilization-Focused Evaluation (2008, Sage), which is now in its 4th edition.
SurveyMonkey’s newsletter reports that SurveyMonkey surveys are now optimized for use on iPhones. The June 2009 newsletter states:
“Because it is a device with a modern, standards-compliant browser, any respondent can receive a link to your survey and access it directly on their iPhone.”
Furthermore, SurveyMonkey is currently working to make their surveys optimized on other media or hand-held devices.
In addition, you now have the ability to do the following:
- Create and download custom charts to enhance the presentation of your survey data.
- Import these graphics into your own presentation software such as PowerPoint, Word, etc.
To learn more about the updates, you can visit the following topic in the help center: Creating Custom Charts