Some news from Survey Monkey’s Newsletter:

Professional (paid) subscribers can now make customized links in lieu of those long automatically-generated ones. And you can analyze data based on respondents’ answers by using the Filter by Response tool within the Analyze section. Professional subscribers can also create Custom Reports within the Analyze section.

A belated note about an interesting item at the Medical Library Association meeting in DC this past May. Christine Chastain-Warheit gave a fascinating 5-minute “Lightning Round” presentation, “Can Hospital Librarians Demonstrate Internal Revenue Service-mandated Community Benefit for Their Nonprofit Organizations? Reflecting on Value Provided and Connecting the Hospital Library to Community Benefit.” She pointed out that the IRS Community Benefit standard for not-for-profit hospitals includes activities that promote health in response to community needs. Community Benefit is the basis of the tax-exemption of not-for-profit hospitals. Her institution has agreed that the library’s outreach activities can be included in calculating hospital community benefit efforts for IRS reporting (Poster presented at Medical Library Association Annual Meeting, May 23, 2010). This approach could have good potential for libraries in not-for-profit hospitals demonstrating their value to their institutions and their institutions’ communities, so it’s definitely something to watch. Ms. Chastain-Warheit is Director of Medical Libraries at Christiana Hospital in Newark, DE.

The OERC promotes collaborative evaluation approaches in our training and consultations, so it is always nice to have published examples of collaborative evaluation projects.  A recent issue of Academic Medicine features an article showing how a collaborative evaluation approach called Empowerment Evaluation was applied to improve the medical curriculum at Stanford University School of Medicine.  David Fetterman, originaor of  Empowerment Evaluation, is the primary author of the article.  A key contribution of this evaluation approach — which is exemplified in the article — is the ongoing involvment of stakeholders in the analysis and use of program data. This article provides strong evidence that the approach can lead to positive outcomes for programs.

Citation: Fetterman DM, Deitz J, Gesundheit N. Empowerment Evaluation: A Collaborative Approach to Evaluating and Transforming a Medical School Curriculum. Academic Medicine 2010 May; 85(5):813-820.  A link to the abstract is available at http://journals.lww.com/academicmedicine/Abstract/2010/05000/Empowerment_Evaluation__A_Collaborative_Approach.25.aspx

AEA kicked off 2010 with AEA365, a new blog featuring ”tips, cool tricks, and rad resources for evaluators.”  Entry topics include computer-assisted logic modeling, time management tools, ideas for evaluation reports, and tools for visualizing and displaying data. The blog is available at http://aea365.org/blog/.

Slides and handouts from October’s OPEN (Oregon Program Evaluators Network) meeting are available at their web site in the “OPEN Past Events & Resources” section.

One of the OPEN Annual Conference afternoon workshops was “Groovy Graphics: Visual displays of Quantitative Information” from Elizabeth O’Neill and Ralph Holcomb of Multnomah County Aging and Disability Services. (Silly presentation title, not clear why they used it, they didn’t seem to like it, either)

The presentation began with a “Graphics 101″ review of some of Edward Tufte’s concepts about using graphics that “concentrate on the eloquent statement”:

• Show comparisons;
• Provide data at different levels of detail;
• Integrate information so viewers can understand it quickly;
• Provide documentation;
• Reduce clutter.

Every visual choice should convey a meaning. MS Excel provides a toolbox of fundamentals for us:

• Bar Graphs–avoid using the 3D option because it serves no purpose and simply adds visual clutter; typical axis choices are to make the x-axis nominal and the y-axis ordinal, interval, or ratio; horizontal bar graphs are especially useful in facilitating comparison of data to a benchmark;
• Line Graphs–typically the x axis is ordinal or higher and the y axis is interval or ratio;
• Scatterplots–for continuous data; both x and y are interval or ratio;
• Stem and Leaf charts–show mean, median, and mode in one glance (but I find them a bit hard to read);
• Pie Charts–for displaying many values of one variable.

The presenter stated that he is a “strong advocate of DDT”–meaning Dashboards, Drill-downs, and Templates:

• Dashboards provide quick groups of data presented simply (often featuring traffic signals in green, yellow, and red);
• Drilldowns (web pages where viewers can click on a word or a tab) are ways to provide more data with less clutter;
• Trendlines show patterns of data over time.

Sparklines are tiny graphs, charts, traffic signals embedded within text, showing lots of data in a small space. They provide very quick views of data within the context of a narrative, and can be produced with an add-on that can be purchased for Excel (they will be a standard feature in the next version).

Word clouds, such as those produced from Wordle, can be used in textual data analysis for counting the occurrences of words. In word clouds, word size matters–bigger means more–but word position does not–it’s random. So, conclusions cannot be drawn from word proximity. But, the presenter suggested that word clouds could be given to decision makers to help them generate hypotheses and could be used in generating “frequently asked questions” lists. Word clouds also have good potential for use as graphics for the fronts of reports.

Handouts and powerpoints from this workshop will be made available at the OPEN web site.

One of the OPEN Annual Conference afternoon workshops was “What Program Evaluators Can Learn from How We Performancy Auditors Do Our Work” from Gary Blackmer, Oregon Audits Director from the office of the Oregon Secretary of State. He referred to the 2007 Government Auditing Standards definition of performance auditing, pointing out that information resulting from audits is intended by use of those charged with governance and oversight to improve program performance and operations, reduce costs, and facilitate decision making. How does this differ from program evaluation? Auditors are required to follow the Government Auditing Standards from the Government Accountability Office (also known as “The Yellow Book”). Auditors are organizationally independent from the entities being audited; they do not negotiate scope, objectives, or access to data; and they always produce public reports. He pointed out that auditors make it a practice to spend one-third of their time conducting assessment and developing their audit plan, one-third gathering evidence, and one-third producing reports of findings. (Audience members generally agreed that evaluators tend to spend less time than that on planning and reporting.)

Published audit reports are typically intended to provide a window into an organization–a portrait for the public–with referrals to separately published working papers that provide details about methodology, data, and analysis. The speaker observed that bad news doesn’t travel up very well and we don’t get rewarded for doing things wrong. Organizations often only want to be assessed on things that can be controlled, leading to an emphasis on process rather than outcomes. Auditors sometimes have a more accurate view of “reality” than management and, emphasizing that there is always room for improvement in every organization, provide “bad news” (ie, suggestions regarding changes) in doses that can be tolerated.

Handouts and powerpoints from this workshop will be made available at the OPEN web site.

The Oregon Program Evaluators Network (OPEN) held its Annual Conference last week in Portland. OPEN is a regional affiliate of the American Evaluation Association, with members from government agencies, nonprofits, universities, and private consulting firms. Their annual meeting primarily attracts evaluators from western Oregon and southwestern Washington (Vancouver, WA down through Eugene, OR) but there was at least one international participant and several of us from Seattle. This was a very interesting meeting and I’ll provide my subjective take-aways from it in this post and the next two posts.

The opening keynote speaker was Dr. Debra Rog, 2009 president of the American Evaluation Association, and her talk was titled, “When Background Becomes Foreground: The Importance of Context in Evaluation.” She mentioned ongoing discussions in the evaluation field about whether randomized studies can actually be considered a “gold standard”–they’re great when you can do them, but they’re not always appropriate (they may not be practical or ethical). She spoke of realist evaluation, pointing out that programs are embedded in layers of social reality. For example, power and privilege can influence programs (and the evaluation of those programs) in fundamental ways. Often there are only two “degrees of separation” between the people in power and the people who are providing data–this can lead to a lack of openness and honesty. Context can be difficult to identify and this speaks to the importance of including multiple voices and views. She provided a great insight from her own experience about sharing evaluation results with stakeholders: decision makers are busy and don’t want nuances, they want the bottom line.

The afternoon featured six workshops running in two sets of three (here’s a copy of the agenda). I attended these sessions:

• “What Program Evaluators Can Learn from How We Performance Auditors Do Our Work” and
• “Groovy Graphics: Visual Displays of Quantitative Information”

Handouts and powerpoints from the afternoon workshops will be made available at the OPEN web site.

You can compile all the statistics in the world to demonstrate the effectiveness of your program – but it’s the stories behind the statistics that will be remembered. The workbook Impact and Value: Telling Your Program’s Story provides valuable tips and examples for developing success stories to demonstrate program achievements. This workbook goes beyond describing how to collect anecdotes of individuals who have benefited from your program (although anecdotes are used effectively in the workbook examples). Rather, the workbook shows how to frame your project’s successes in a story format that is easily communicated and remembered. The authors give outlines for a variety of presentation formats, from short elevator speeches delivered to high-powered stakeholders to 2-page write-ups for various audiences. The workbook includes a useful template for developing success stories. Impact and Value: Telling Your Program’s Story is a good resource for those who want to use their evaluation results for effective program advocacy.

This workbook, published by the CDC, is available for download here.

Citation: Lavinghouze SR, Price AW. Impact and Value: Telling your Program’s Story. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Oral Health, 2007.

How important is it to transcribe focus group discussions? Dr. Rita O’Sullivan from the UNC-Chapel Hill School of Education sought an objective answer to that question. She and colleagues ran an experiment in which two co-facilitators ran seven focus groups and created summary reports of the discussions. Each co-facilitator produced a report for each focus group: one wrote a summary based on memory, handwritten notes and a transcript of the audio tape; the other wrote a summary using memory, notes and the audiotape. (Each facilitator prepared seven summaries, some using the first method and some using the second.)  Then, 18 educational professionals who were enrolled in a graduate-level educational research class compared the pairs of summaries.  Sixteen of the 18 reviewers found no substantive differences between the two versions of the summaries.

What does this mean for evaluators?  The authors concluded that their findings, although preliminary, suggest that, for the typical program evaluation setting, transcripts are not necessary to produce useful focus group discussion summaries. The findings also make it hard to justify the transcription costs for focus groups in evaluation settings – because every dollar spent on evaluation is one not spent on the program.  

Source: O’Sullivan et al. Transcribing focus group articles: Is there a viable alternative? 2004 November.  Paper presented at the joint international meeting of the American Evaluation Association and the Canadian Evaluation Society, Toronto, Canada.