The Middle Atlantic Region’s focus group project, led last winter by Sue Hunter, recently received attention through two venues of the American Evaluation Association: at the 2010 AEA annual conference in San Antonio and on the aea365 blog. On November 12, Sue  presented Using Appreciative Inquiry Focus Groups to Engage Members in Planning for the National Network of Libraries of Medicine Middle Atlantic Region.  (Cindy Olney was a co-contributor to this presentation.) MAR used the focus group project, designed using appreciative inquiry methods, to collect network member feedback in preparation for its 2011-2016 proposal. The presentation highlighed the evaluation design and lessons learned about using the appreciative inquiry approach. The presentation abstract and PowerPoint slides are available here at the AEA Public Library.

In the run-up to the conference, AEA staff asked some presenters to submit blog entries about their conference presentations for AEA365, a tip-of-the-day blog by and for evaluators.  On AEA’s invitation, Sue and Cindy wrote a blog entry that was posted here on November 5.

The Institute of Museum and Library Services has awarded a grant that will test and implement methodologies measuring the return on investment (ROI) in academic libraries. The goals are to provide evidence and a set of tested methodologies that academic libraries will be able to use in demonstrating their value. The University of Tenessee, Knoxville will be conducting this study in collaboration with the University of Illinois at Urbana-Champaign and the Association of Research Libraries. Dr. Carol Tenopir, professor in the School of Information Sciences, is the project’s lead investigator. This news item from the UT Knoxville’s Tennessee Today provides more details: “UT Shares in Grant to Study Value of Academic Libraries.”

In a complementary project, the Association of College and Research Libraries has selected Dr. Megan Oakleaf to conduct a review of the “quantitative and qualitative literature, methodologies and best practices currently in place for demonstrating the value of academic libraries.” The Association plans to issue a completed report later this year.

The American Evaluation Association “Coffee Break” webinar on June 10 featured a comparison of Survey Monkey and Zoomerang, two well-known and respected web survey tools. Both feature free accounts–you can sign up and test drive them as part of deciding whether to move to the paid options. In both cases, the free accounts feature most of the system functionality but with limits on the numbers of questions and responses. The prices for paid accounts are similar for both. The presenters, Lois Ritter and Tessa Robinette, highlighted some differences between the two systems.

Survey Monkey was, as of June 10, the only online survey application that is 508 compliant and, because one subscription can share multiple surveys, it is good for work being conducted by different groups in multiple locations. Survey Monkey is available in a variety of languages and can be used with the iPhone.

Zoomerang can be used with multiple mobile devices and offers a fee-based survey translation service. Zoomerang is designed for a one account per user and project environment, and can provided rented lists for sampling frames on 500 attributes that purport to be representative of the “general population.” Zoomerang also has a nice analytic feature: “tag clouds” for thematic grouping.

For a thorough overview of how to conduct online surveys, consult the Autumn, 2007 issue of New Directions for Evaluation, number 115.

The American Evaluation Association “Coffee Break” webinar series is a benefit of membership in the association. Recordings of these 20 minute sessions are archived in the AEA’s Webinar Archive E-Library (a members-only site).

Some news from Survey Monkey’s Newsletter:

Professional (paid) subscribers can now make customized links in lieu of those long automatically-generated ones. And you can analyze data based on respondents’ answers by using the Filter by Response tool within the Analyze section. Professional subscribers can also create Custom Reports within the Analyze section.

A belated note about an interesting item at the Medical Library Association meeting in DC this past May. Christine Chastain-Warheit gave a fascinating 5-minute “Lightning Round” presentation, “Can Hospital Librarians Demonstrate Internal Revenue Service-mandated Community Benefit for Their Nonprofit Organizations? Reflecting on Value Provided and Connecting the Hospital Library to Community Benefit.” She pointed out that the IRS Community Benefit standard for not-for-profit hospitals includes activities that promote health in response to community needs. Community Benefit is the basis of the tax-exemption of not-for-profit hospitals. Her institution has agreed that the library’s outreach activities can be included in calculating hospital community benefit efforts for IRS reporting (Poster presented at Medical Library Association Annual Meeting, May 23, 2010). This approach could have good potential for libraries in not-for-profit hospitals demonstrating their value to their institutions and their institutions’ communities, so it’s definitely something to watch. Ms. Chastain-Warheit is Director of Medical Libraries at Christiana Hospital in Newark, DE.

The OERC promotes collaborative evaluation approaches in our training and consultations, so it is always nice to have published examples of collaborative evaluation projects.  A recent issue of Academic Medicine features an article showing how a collaborative evaluation approach called Empowerment Evaluation was applied to improve the medical curriculum at Stanford University School of Medicine.  David Fetterman, originaor of  Empowerment Evaluation, is the primary author of the article.  A key contribution of this evaluation approach — which is exemplified in the article — is the ongoing involvment of stakeholders in the analysis and use of program data. This article provides strong evidence that the approach can lead to positive outcomes for programs.

Citation: Fetterman DM, Deitz J, Gesundheit N. Empowerment Evaluation: A Collaborative Approach to Evaluating and Transforming a Medical School Curriculum. Academic Medicine 2010 May; 85(5):813-820.  A link to the abstract is available at http://journals.lww.com/academicmedicine/Abstract/2010/05000/Empowerment_Evaluation__A_Collaborative_Approach.25.aspx

AEA kicked off 2010 with AEA365, a new blog featuring ”tips, cool tricks, and rad resources for evaluators.”  Entry topics include computer-assisted logic modeling, time management tools, ideas for evaluation reports, and tools for visualizing and displaying data. The blog is available at http://aea365.org/blog/.

Slides and handouts from October’s OPEN (Oregon Program Evaluators Network) meeting are available at their web site in the “OPEN Past Events & Resources” section.

One of the OPEN Annual Conference afternoon workshops was “Groovy Graphics: Visual displays of Quantitative Information” from Elizabeth O’Neill and Ralph Holcomb of Multnomah County Aging and Disability Services. (Silly presentation title, not clear why they used it, they didn’t seem to like it, either)

The presentation began with a “Graphics 101″ review of some of Edward Tufte’s concepts about using graphics that “concentrate on the eloquent statement”:

• Show comparisons;
• Provide data at different levels of detail;
• Integrate information so viewers can understand it quickly;
• Provide documentation;
• Reduce clutter.

Every visual choice should convey a meaning. MS Excel provides a toolbox of fundamentals for us:

• Bar Graphs–avoid using the 3D option because it serves no purpose and simply adds visual clutter; typical axis choices are to make the x-axis nominal and the y-axis ordinal, interval, or ratio; horizontal bar graphs are especially useful in facilitating comparison of data to a benchmark;
• Line Graphs–typically the x axis is ordinal or higher and the y axis is interval or ratio;
• Scatterplots–for continuous data; both x and y are interval or ratio;
• Stem and Leaf charts–show mean, median, and mode in one glance (but I find them a bit hard to read);
• Pie Charts–for displaying many values of one variable.

The presenter stated that he is a “strong advocate of DDT”–meaning Dashboards, Drill-downs, and Templates:

• Dashboards provide quick groups of data presented simply (often featuring traffic signals in green, yellow, and red);
• Drilldowns (web pages where viewers can click on a word or a tab) are ways to provide more data with less clutter;
• Trendlines show patterns of data over time.

Sparklines are tiny graphs, charts, traffic signals embedded within text, showing lots of data in a small space. They provide very quick views of data within the context of a narrative, and can be produced with an add-on that can be purchased for Excel (they will be a standard feature in the next version).

Word clouds, such as those produced from Wordle, can be used in textual data analysis for counting the occurrences of words. In word clouds, word size matters–bigger means more–but word position does not–it’s random. So, conclusions cannot be drawn from word proximity. But, the presenter suggested that word clouds could be given to decision makers to help them generate hypotheses and could be used in generating “frequently asked questions” lists. Word clouds also have good potential for use as graphics for the fronts of reports.

Handouts and powerpoints from this workshop will be made available at the OPEN web site.

One of the OPEN Annual Conference afternoon workshops was “What Program Evaluators Can Learn from How We Performancy Auditors Do Our Work” from Gary Blackmer, Oregon Audits Director from the office of the Oregon Secretary of State. He referred to the 2007 Government Auditing Standards definition of performance auditing, pointing out that information resulting from audits is intended by use of those charged with governance and oversight to improve program performance and operations, reduce costs, and facilitate decision making. How does this differ from program evaluation? Auditors are required to follow the Government Auditing Standards from the Government Accountability Office (also known as “The Yellow Book”). Auditors are organizationally independent from the entities being audited; they do not negotiate scope, objectives, or access to data; and they always produce public reports. He pointed out that auditors make it a practice to spend one-third of their time conducting assessment and developing their audit plan, one-third gathering evidence, and one-third producing reports of findings. (Audience members generally agreed that evaluators tend to spend less time than that on planning and reporting.)

Published audit reports are typically intended to provide a window into an organization–a portrait for the public–with referrals to separately published working papers that provide details about methodology, data, and analysis. The speaker observed that bad news doesn’t travel up very well and we don’t get rewarded for doing things wrong. Organizations often only want to be assessed on things that can be controlled, leading to an emphasis on process rather than outcomes. Auditors sometimes have a more accurate view of “reality” than management and, emphasizing that there is always room for improvement in every organization, provide “bad news” (ie, suggestions regarding changes) in doses that can be tolerated.

Handouts and powerpoints from this workshop will be made available at the OPEN web site.