The Oregon Program Evaluators Network (OPEN) held its Annual Conference last week in Portland. OPEN is a regional affiliate of the American Evaluation Association, with members from government agencies, nonprofits, universities, and private consulting firms. Their annual meeting primarily attracts evaluators from western Oregon and southwestern Washington (Vancouver, WA down through Eugene, OR) but there was at least one international participant and several of us from Seattle. This was a very interesting meeting and I’ll provide my subjective take-aways from it in this post and the next two posts.

The opening keynote speaker was Dr. Debra Rog, 2009 president of the American Evaluation Association, and her talk was titled, “When Background Becomes Foreground: The Importance of Context in Evaluation.” She mentioned ongoing discussions in the evaluation field about whether randomized studies can actually be considered a “gold standard”–they’re great when you can do them, but they’re not always appropriate (they may not be practical or ethical). She spoke of realist evaluation, pointing out that programs are embedded in layers of social reality. For example, power and privilege can influence programs (and the evaluation of those programs) in fundamental ways. Often there are only two “degrees of separation” between the people in power and the people who are providing data–this can lead to a lack of openness and honesty. Context can be difficult to identify and this speaks to the importance of including multiple voices and views. She provided a great insight from her own experience about sharing evaluation results with stakeholders: decision makers are busy and don’t want nuances, they want the bottom line.

The afternoon featured six workshops running in two sets of three (here’s a copy of the agenda). I attended these sessions:

• “What Program Evaluators Can Learn from How We Performance Auditors Do Our Work” and
• “Groovy Graphics: Visual Displays of Quantitative Information”

Handouts and powerpoints from the afternoon workshops will be made available at the OPEN web site.

You can compile all the statistics in the world to demonstrate the effectiveness of your program – but it’s the stories behind the statistics that will be remembered. The workbook Impact and Value: Telling Your Program’s Story provides valuable tips and examples for developing success stories to demonstrate program achievements. This workbook goes beyond describing how to collect anecdotes of individuals who have benefited from your program (although anecdotes are used effectively in the workbook examples). Rather, the workbook shows how to frame your project’s successes in a story format that is easily communicated and remembered. The authors give outlines for a variety of presentation formats, from short elevator speeches delivered to high-powered stakeholders to 2-page write-ups for various audiences. The workbook includes a useful template for developing success stories. Impact and Value: Telling Your Program’s Story is a good resource for those who want to use their evaluation results for effective program advocacy.

This workbook, published by the CDC, is available for download here.

Citation: Lavinghouze SR, Price AW. Impact and Value: Telling your Program’s Story. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Oral Health, 2007.

How important is it to transcribe focus group discussions? Dr. Rita O’Sullivan from the UNC-Chapel Hill School of Education sought an objective answer to that question. She and colleagues ran an experiment in which two co-facilitators ran seven focus groups and created summary reports of the discussions. Each co-facilitator produced a report for each focus group: one wrote a summary based on memory, handwritten notes and a transcript of the audio tape; the other wrote a summary using memory, notes and the audiotape. (Each facilitator prepared seven summaries, some using the first method and some using the second.)  Then, 18 educational professionals who were enrolled in a graduate-level educational research class compared the pairs of summaries.  Sixteen of the 18 reviewers found no substantive differences between the two versions of the summaries.

What does this mean for evaluators?  The authors concluded that their findings, although preliminary, suggest that, for the typical program evaluation setting, transcripts are not necessary to produce useful focus group discussion summaries. The findings also make it hard to justify the transcription costs for focus groups in evaluation settings – because every dollar spent on evaluation is one not spent on the program.  

Source: O’Sullivan et al. Transcribing focus group articles: Is there a viable alternative? 2004 November.  Paper presented at the joint international meeting of the American Evaluation Association and the Canadian Evaluation Society, Toronto, Canada.

Someone recently asked me if SurveyMonkey forms are accessible to those with functional limitations and disabilities. In fact, SurveyMonkey received Section 508 certification in June 2008. According to the company’s Web site, they are the only commercial online survey application that has this certification.

While SurveyMonkey software automatically formats surveys to be accessible, there are a few practices that we need to follow to make sure SurveyMonkey questionnaires are user-friendly with screen-readers and other visual technologies. For instance, don’t add extra html coding to your questionnaire (e.g., to bold-face or italicize words) because screen-readers may read parts of html coding as text. Also, SurveyMonkey’s default color schemes are configured for maximum contrast to help low-vision users. Creating your own color schemes may make your forms less readable for this population. You can find more tips from SurveyMonkey for creating screen-reader friendly forms at this link.

AEA/CDC Training session: Utilization-focused evaluation

The first training session I took at the AEA/CDC Institute was Michael Patton’s Utilization-Focused Evaluation.  This workshop was pitched primarily for evaluators who are sick of producing time-consuming evaluation report tombs that sit on shelves. (You’re thinking I should have written “evaluation report tomes,” but actually, those reports are where evaluation results go to die.)  Patton commented that you could probably attach an executive report to 500 sheets of blank paper – or 500 pages from a phone book pulled from your recycling bin – and no one would ever notice because they never read past the executive summary.

Here’s some interesting food for thought: Patton said that the order of the evaluation standards (Utility, Feasibility, Propriety, and Accuracy) is deliberate: Utility, or usefulness to intended users, is listed first because it’s deemed the most important. So, in evaluation design, the evaluation’s usefulness should be considered ahead of its feasibility (practicality and cost effectiveness), propriety (legality, ethics, and concern for the welfare of others), and accuracy (technically adequate information about features that determine merit or worth of a program). All are important standards, but utility gets top ranking. (Definitions for the four evaluation standards are listed here at the American Evaluation Association web site.)

To enhance the utility of evaluation findings, Patton said it is important to identify the intended users and uses of the evaluation information at the beginning of the evaluation and create an action plan for use of evaluation results that takes the following into account:

·         The decisions the evaluation findings are meant to inform

·         Timing of those decisions

·         The stakeholders who will see and respond to the data

The responsibility for facilitating use of the findings falls on the evaluation consultant (or whoever is in charge of conducting the evaluation.)

If you are interested in learning how to conduct more useful evaluations, I recommend Patton’s Utilization-Focused Evaluation (2008, Sage), which is now in its 4^th edition.

SurveyMonkey’s newsletter reports that SurveyMonkey surveys are now optimized for use on iPhones. The June 2009 newsletter states:

“Because it is a device with a modern, standards-compliant browser, any respondent can receive a link to your survey and access it directly on their iPhone.”

Furthermore, SurveyMonkey is currently working to make their surveys optimized on other media or hand-held devices.
In addition, you now have the ability to do the following:

• Create and download custom charts to enhance the presentation of your survey data.
• Import these graphics into your own presentation software such as PowerPoint, Word, etc.

To learn more about the updates, you can visit the following topic in the help center: Creating Custom Charts

I spent the earlier part of the week (June 15-17) in Atlanta attending the AEA/CDC Summer Evaluation Institute and, as usual, came away with some great information.  I’ll be adding some separate blog entries about the sessions I attended, but I thought I would give a rundown on this particular event.  The Summer Evaluation Institute is conducted jointly by the American Evaluation Association and the Centers for Disease Control and Prevention, so many presenters and attendees were from the CDC – but those of us who attend the AEA conference or other evaluation training events found familiar names on the roll of presenters. The Summer Evaluation Institute differs from the AEA conference in that it is totally training-oriented – offering a limited number of educational sessions between 8:30 and 4:00 pm over 2.5 days.  So you don’t feel conflicted over all the options of a conference and you have plenty of downtime to meet and network with colleagues. As you might expect, there is an emphasis on health-related evaluation in many of the sessions, but that emphasis appears more in the examples used by instructors – the evaluation techniques themselves are applicable across disciplines.  The cost is reasonable.  This year, the cost was $395 for AEA members (and CDC employees) and a little more for non-members. (Sorry I can’t be more specific: the fee is no longer listed at the AEA Web site now that the event is over).  That fee includes three keynote speeches, a choice of training sessions each morning and “breakout” sessions in the afternoon. (I’m not sure how “training sessions” differed from “breakout sessions,” other than length of time – the training sessions were about an hour longer than the breakout sessions).  It also includes breakfast and lunch on most days.  Beginner workshops were offered on June 14 for an additional cost:  “Quantitative Methods for Evaluation;” and “Introduction to Evaluation.”  The Summer Evaluation Institute is held annually, so if you think you might be interested in the 2010 event, check out the AEA web site (eval.org) starting in March.

A new government Web site, Data.gov, may prove to be a good tool for locating existing data from federal agencies, particularly for those of us doing needs or community assessment.  The Web site is the public’s “one-stop shop” for raw data from economic, healthcare, environmental, and other government agencies.  Along with raw data, the site provides tools for compiling raw data into more analyzable formats (e.g. tables, maps) and widgets (interactive tools with single-service purposes, like showing users the latest news). My quick browsing of the Web site gives me the impression that it is a work in progress.   However, the “about” page says that the catalog of datasets will continue to grow and that the site will be improved based on public feedback. 

Here is a link to the blog entry about Data.gov from the Office of Management and Budget: www.whitehouse.gov/omb/blog/09/05/21/DemocratizingData/ 

House, ER Blowback: Consequences of Evaluation for Evaluation. American Journal of Evaluation. 2008 December 29 (4), 416-426.

For some, the Randomized Controlled Trial (RCT) has the mystique of separating the researcher from the method and therefore guaranteeing research objectivity. However, in a 2008 article in the American Journal of Evaluation, Ernest House disputes this myth. He describes many examples of how bias has been introduced into the RCTs for new drugs, which are funded primarily by drug companies  (over 70%, according to the article). He talks about suppression of negative results as one problem with drug-company sponsored research; but he also describes ways that drug trials can actually be manipulated to influence results favoring the drug company’s products. For example, the drugs under investigation may be compared to a lower dosage of the competitor’s drug in the control group or the competitor’s drug may be administered in a less effective manner. Studies also may be conducted on younger subjects who generally tend to show fewer side effects or conducted for short periods of time even though the drug was developed for long-term use.

House also notes that, as an evaluation tool, RCTs are very limited in providing all information needed to judge a new drug’s value. Usually the drug group is compared to a control group that gets no treatment instead of the typical dosage of the closest competitor drug. So, while consumers may know the tested drug is superior to no treatment, we know nothing about the cost-benefit or “clinical effectiveness” of a new (often more expensive) drug.

This article provides some important take-home messages for evaluators as well as for consumers of drugs. First of all, no method guarantees objectivity: even the highly acclaimed RCT can be manipulated (deliberately or unconsciously) to influence desired results. Second, evaluation – finding the value of products, services, or programs – usually involves multiple issues that must be investigated through mixed methods. Finally, evaluators need to be aware that they are not completely objective and methods cannot protect them from their own subjectivity. We need to be transparent about our data collection and analysis process and be open to feedback from peers and stakeholders.

Read even more about the American Evaluation Association meeting at the Eagle Dawg Blog, where Nikki Detmar has summarized her Ten thousand four hundred and thirty one words of notes.  Nikki attended many different sessions from the ones I went to, and where we were both in the audience for a session, Nikki took more detailed notes than I did.  She’s a fast typist who uses her laptop for notes; I’m a codger who writes with a pen in cursive scrawls on pieces of lined notebook paper.  Also, the Eagle Dawg Blog is just an all-around good read for Nikki’s perspectives on life, the universe, health informatics, and medical librarianship.