Skip all navigation and go to page content


NN/LM Outreach Evaluation Resource Center

Archive for the ‘Practical Evaluation’ Category

Data Party Like it’s 2099! How to Throw a Data Party

Friday, November 20th, 2015

two funny birthday dogs celebrating close together as a coupleWhat’s a “data party?” We attended a program by evaluator Kylie Hutchinson entitled “It’s a Data Party!” at the AEA 2015 conference last week in Chicago.  A data party is another name for a kind of participatory data analysis, where you gather stakeholders together, show them some of the data that you have gathered and ask them to help analyze it.

Isn’t analyzing the data part of your job?  Here are some reasons you might want to include stakeholders in the data analysis stage:

  • It allows stakeholders to get to know and engage with the data
  • Stakeholders may bring context to the data that will help explain some of the results
  • When stakeholders participate in analyzing the data, they are more likely to understand it and use it
  • Watching their interactions, you can often find out who is the person with the power to act on your recommendations

So how do you throw a data party? First of all you need to know what you hope to get from the attendees, since you may only be able to hold an event like this one time. There are a number of different ways to organize the event.  You might want to consider using a World Cafe format, where everyone works together to explore a set of questions, or you could use an Open Space system in which attendees create their own agenda about what questions they want to discuss.  Recently the AEA held a very successful online unconference using MIT’s Unhangout that could be used for an online Data Party with people from multiple locations.

The kinds of questions Kylie Hutchinson suggested asking at a data party include:

  • What does this data tell you?
  • How does this align with your expectations?
  • What do you think is occurring here and why?
  • What other information do you need to make this actionable?

At the end of the party it might be time to present some findings and recommendations that you have.  Considering the work that they have done, they may be more willing to listen.  As Kylie Hutchinson said “People support what they helped create.”


Group Learning about Evaluation

Friday, October 23rd, 2015

Group of young business people talking on business meeting at office.

I recently got to participate in a very successful roundtable at a library conference.  I co-moderated an evaluation roundtable entitled “Library assessment: You’ve measured your success – now how do you get people to listen?” with OU-Tulsa Schusterman Library Associate Director Katie Prentice at the South Central Chapter of the Medical Library Association Annual Meeting in Little Rock, AR.

What makes roundtables unique among educational opportunities at library conferences is that unlike presentations or papers where attendees sit and listen, in a roundtable everyone can participate. It is a moderated discussion on a given topic among the people who attend, and since anyone can chime in, learning is active instead of passive.

About 25 people attended this roundtable and enthusiastically participated in a discussion about library assessment and evaluation data. Katie and I led the discussion with questions starting from what kind of data you collect at your library and leading to what libraries do with the data and how to make it work better for them. Our goal was to use our questions to all issues and solutions to come from the attendees themselves.

As an example, when we asked the question “what would you really like to know about your library and what do you dream of finding out about your users?” one hospital librarian said that she wanted to know how doctors were using the information and how it impacted the patients. Katie Prentice asked “can anyone help her with this?” and another hospital librarian responded that she sends emails to some of her doctors to ask for a sentence or two describing how the information was used.  These sentences, when collected and analyzed, could be a powerful tool to show hospital administration the importance of the library to patient outcomes.

Other kinds of evaluation ideas that were generated from attendees at this roundtable were:

  • using heat map software to determine where people go most often on your website
  • having student workers note what pieces of furniture are being used to improve furniture types and placement in the library
  • using a product like Constant Contact or Mail Chimp to send library newsletters to the doctors and employees at hospitals with assessment data.

While not all roundtables at conferences are this successful, this roundtable demonstrated the ability of librarians brought together in a group to learn from each other and solve problems.

Creative Annual Reports

Friday, October 9th, 2015

Ah, the annual report – at its best we expect to see a glossy booklet with pie charts, short paragraphs and some quotes. At its worst it can be pages of dry text. Our main hope with annual reports is that our stakeholders and others will read them and be impressed with the successes of our organizations.

Last month I ran across the annual report from the Nowra Public Library in New South Wales, Australia, which was so compelling and understandable that over 100,000 people have viewed it on YouTube:

Photo of librarian links to Nowra library video

Since most organizations don’t have the resources to do their own music video (e.g. singers, writers, silly costumes), I thought I would look at a few other examples to consider when it’s time to do your annual report.

One of my all-time favorites is the annual report from the Schusterman Library of The University of Oklahoma-Tulsa. Their annual report is an infographic that shows the data that is collected, but also describes the data in such a way that 1) you have a better feel for what is going on in the library; and also 2) you might think “I didn’t know they would help me with that!”  For example: “7,274 Reference questions answered in person, by phone, by email, and instant message or text on everything from ADHD and child welfare to decision trees, LEED homes, and census reporting.” It is available on their website, and the librarians at the Schusterman Library say they frequently find students looking at it.

The Michigan State University College of Education won a gold ADDY and a Best in Show Award for their 2012 Annual Report (an ADDY is the advertising industry’s largest competition).  Their report featured a tri-fold, die-cut skyline that presented the college’s missions and strengths with an emphasis on “college as community.” The annual report also included a video and a website that gives detailed narratives that show institutional successes in terms of personal stories.

Of course, not all institutions want an unusual annual report.  But it is important to consider the target audience.  Annual reports reach the upper administration, potential funders, and patrons of the library. The success of this years annual report might shape the library users view of the library for years to come.

The OERC is on the Road Again

Friday, September 25th, 2015

The OERC is on the road again.  Today, Cindy and Beth Layton, Associate Director of the NN/LM Greater Midwest Region, are team-teaching Measuring What Matters to Stakeholders at the Michigan Health Sciences Library Association’s annual conference in Flint, MI.

Logo for Michigan Health Sciences Library Association

This workshop covers strategies for using evaluation to enhance and communicate a library’s value to organizational decision-makers and stakeholders who influence decision makers. The workshop combines updated information with material from the NN/LM MidContinental Region’s Measuring Your Impact and the OERC’s Valuing Your Library workshops that have been taught by a number of regional medical library staff members over the past decade.

On Saturday, Karen is presenting a brand-new workshop for the Texas Library Association’s District 8 Conference called Adding Meaning to Planning: A Step-by-Step Method for Involving Your Community in Meaningful Library Planning.

TLA District 8 Logo

The workshop is a method of involving community members in creating pain-free logic models to ensure that the long term vision is always in sight when planning.  Karen wrote a blog entry about the creating “tearless” logic models here.  This is Karen’s first experience creating and delivering a workshop that is purely about library evaluation.

The NN/LM travel season is about to go into full swing.  We know we aren’t the only ones out and about with presentations, trainings, and exhibits.  So safe travels. And we will see you in a week with another OERC blog post.

Which Online Survey Tool Should I Use? A Review of Reviews

Friday, September 4th, 2015

Quality survey close up with a thumbtack pointing on the word excellentRecently we faced the realization that we would have to reevaluate the online survey tool that we have been using. We thought that we would share some of the things that we learn along the way.

First of all, finding a place that evaluates survey products (like Survey Monkey or Survey Gizmo), is not as easy as going to Consumer Reports or Amazon (or CNET, Epinions, or Buzzillions).  A number of places can be found on the internet that provide reviews of surveys, but their quality is highly varied.   So for this week our project has been to compare review websites to see what we can learn from and about them.

Here are the best ones I could find that compare online survey tools:’s Ultimate Guide to Forms and Surveys, Chapter 7 “The 20 Best Online Survey Builder Tools”

This resource compares 20 different online survey tools. There is a chart with a brief statement of what each survey tool is best for, what you get for free, and the lowest plan cost. Additionally, there is a paragraph description of each tool and what it does best.  Note: this is part of an eBook published in 2015 which includes chapters like “The Best Online Form Builders for Every Task.”’s “18 Awesome Survey & Poll Apps”

This review was posted on May 27, 2015 which reassures us that the information is most likely up to date.  While there are very brief descriptions, it is good for a quick comparison of the survey products. Each review includes whether or not there is a free account, if the surveys can be customized, and whether or not there are ready-made templates.’s “Top Survey Software Products”

Check boxes showing the features of the different productsThis resource appears to be almost too good to be true. Alas, no date shown means that the specificity in the comparisons might not be accurate.  Nevertheless, this website lists over 200 survey software products, has separate profile pages on each product (with varying amounts of detail), and lists features that each product offers.  You can even narrow down the surveys you are looking for by filtering by feature.  Hopefully the features in Capterra’s database are kept updated for each product.  One thing to point out is that at least two fairly well-known survey products (that I know of) are not in their list.’s “Top 31 Free Survey Apps”

Another review site with no date listed. This one compares 31 apps by popularity, presumably in the year the article was written. One thing that is unique about this review site is that the in-depth review includes the history and popularity of the app, the differences of each app to other apps, and who they would recommend the app to.  Many of the reviews include videos showing how to use the app.  Pretty cool.’s 2015 Best Survey Software Reviews and Comparisons

This website has the feel of Consumer Reports. It has a long article explaining why you would use survey software, how and what the reviewers tested, and the kinds of things that are important when selecting survey software. Also like Consumer Reports, it has ratings of each product (including the experiences of the business, the respondents, and the quality of the support), and individual reviews of each product showing pros and cons. Because the date is included in the review name, the information is fairly current.

This is a starting point. There are individual reviews of online survey products on a variety of websites and blogs, which are not included here.  Stay tuned for more information on online survey tools as we move forward.


Improving Your Data Storytelling in 30 Days

Friday, August 21st, 2015

Here are some more great techniques to help with telling a story to report your evaluation data so it will get the attention it deserves.

Friends at campfire telling storiesJuice Analytics has this truly wonderful collection of resources in a guide called “30 Days to Data Storytelling.” With assignments of less than 30 minutes a day, this guide links to data visualization and storytelling resources from sources as varied as Pixar, Harvard Business Review, Ira Glass, the New York Times, and Bono (yes, that Bono).

The document is a checklist of daily activities lasting no longer than 30 minutes per day. Each activity is either an article to read, a video to watch, or a small project to do.

The resources answer valuable questions like:

  • What do you do when you’re stuck?
  • How do I decide between visual narrative techniques?
  • Where can I find some examples of using data visualization to tell a story?

A Rainbow Connection? The Evaluation Rainbow Framework

Friday, August 7th, 2015

Once you start looking online for help with your evaluation project, you will find a veritable storm of evaluation resources out there. So many that it can be confusing how choose the best ones for your needs.  But don’t worry, once you’ve looked at this online tool you will find the rainbow of hope that follows the storm (okay that was pretty cheesy – stay with me, it gets better).

A group of evaluators from all over the world created a website called for the purpose of organizing and sharing useful online evaluation resources. The framework they created to organize resources is called the Rainbow Framework because it divides the world of evaluation into seven different “clusters” which are delineated by rainbow colors.  Each cluster is then broken down into a number of tasks, and each task broken down into options and resources.

Here is an example of the Rainbow Framework in action.  By clicking on the yellow “Describe” category, the image opens a window on the right that lists seven tasks: 1) Sample; 2) Use measures, indicators, or metrics; 3) Collect and/or retrieve data; 4) Manage data; 5) Combine qualitative and quantitative data; 6) Analyze data; and 7) Visualize data.

When you click on a specific task, a page listing a variety of Options and Resources will open, like this:Rainbow Framework Options


BetterEvaluation made eight 20 minute “coffee break” webinars in conjunction with AEA that you can watch for free on the BetterEvaluation website. Each webinar describes a cluster, and there is one overview webinar.  The webinars are two years old, so the actual image of the rainbow looks a little different from the webinar video, but the content is still relevant.  Here is a link to the webinar series:

The Rainbow Framework does more than just organize resources. Here are some reasons you might want to use this Framework.

1) Help designing and planning an evaluation

2) Check the quality of an ongoing evaluation

3) Commission an evaluation – will help formulate what’s important to include when you commission an evaluator and then when you assess the quality of the proposals

4) Embed stakeholder participation thoughtfully throughout the evaluation

5) Develop your evaluation capacity – lifelong learning – to fill in gaps of knowledge.

So, somewhere over the rainbow, your evaluation skies may be blue…

Getting Started in Evaluation – Evaluation Guides from the OERC

Thursday, July 23rd, 2015

New to the world of evaluation? What is your boss talking about when she says she wants you to measure outcomes, not outputs?  What is an indicator? How many responses should you get from your surveys?

Sometimes people think evaluation is just the form that you fill out at the end of a class or event. But in fact evaluation can start at the beginning of the project when you do a community assessment and evaluation includes building support for your project from your stakeholders. And it continues through making an evaluation plan as part of your project, gathering data, analyzing data, and reporting the data back to the stakeholders in a way that it is useful.  Here is a model that the CDC uses to describe the evaluation framework:

CDC Framework for Program Evaluation

The Outreach Evaluation Resource Center (OERC) has a series of three booklets entitled Planning and Evaluating Health Information Outreach Projects that guide people through the evaluation process, from needs assessment to analyzing data.  While focusing on “health information outreach” this series of books can be used to learn how to do evaluation for any type of project.

Booklet 1: Getting Started with Community-Based Outreach

  • Getting organized: literature review; assembling team of advisors; taking an inventory; developing evaluation questions
  • Gathering information: primary data; secondary data, and publicly accessible databases
  • Assembling, Interpreting and Acting: summarizing data and keeping stakeholders involved

Booklet 2: Planning Outcomes-Based Outreach Projects

  • Planning your program with a logic model to connect activities to outcomes
  • Planning your process assessment
  • Developing an outcomes assessment plan, using indicators, objectives and an action plan

Booklet 3: Collecting and Analyzing Evaluation Data

  • Designing data collection methods; collecting data; summarizing and analyzing data for:
    • Quantitative methods
    • Qualitative methods

The books can be read in HTML, downloaded as a PDF or physical booklets can be ordered for free from the OERC by sending an email request to:

Learn more about the CDC’s Evaluation Framework:



DIY Tool for Program Success Stories (Program Success Stories Part 2)

Thursday, July 2nd, 2015

Last week, I wrote about program success stories. As a follow-up, I want to introduce you to a story builder tool available at the CDC Injury Prevention and Control web site. The story builder takes you through three steps to produce an attractive, well-written program success story. Each step offers downloadable Microsoft Word documents to walk you through the process.

Step 1: The worksheets are designed to gather and organize project information for your story. I think it would be interesting to use this step as a participatory activity. You could pull together your project team or a group of stakeholders to talk through questions in this worksheet. The discussion would help group members articulate the program’s value from their perspective.

Step 2: This step provides a story builder template to write your story, section by section. Each section has a field to develop a paragraph of your story, with some tips for writing in a compelling, user-friendly way. Each completed field prepares you for the final step.

Step 3: Here, you can download a layout template, where you transfer the paragraphs from your story builder template into the layout. Because this is a Word document, you can change background design, font, or even the size and placement of pictures and call-out quote boxes.

If you are thinking of trying your hand at program success stories, this story building web page provides some useful DYI tools to help you get started.

"What is Your Story" typed on paper in an old typewriter

Components of Process Evaluation

Friday, June 12th, 2015

At the American Evaluation Association Summer Institute, Laura Linnan, Director of the Carolina Collaborative for Research on Work & Health at UNC Gillings School of Public Health, did a workshop entitled Process Evaluation: What You Need to Know and How to Get Started. According to the CDC, process evaluation is the systematic collection of information on a program’s inputs, activities, and outputs, as well as the program’s context and other key characteristics.

Logic Model Image from CDC

Process evaluation looks at the specific activities that take place during an outreach project to ensure that planned interventions are carried out equally at all sites and with all participants, to explain why successes happen or do not happen, and to understand the relationships between the project components. Process evaluation can be extremely important in making adjustments to ensure the project’s success, and determining how or whether to do a project again.

In the workshop I attended, Linnan walked through the details covered in Chapter 1 of the book Process Evaluation for Public Health Interventions and Research by Laura Linnan and Allan Steckler. This chapter presents an overview of process evaluation methods. In it, they define a set of terms that describe the components of process evaluation (Table 1.1). These components are valuable to understand, because evaluators can look in detail at each component to determine which ones should be evaluated.

  1. Context
  2. Reach
  3. Dose delivered
  4. Dose received
  5. Fidelity
  6. Implementation
  7. Recruitment

In addition, the authors describe a step-by-step process for designing and implementing process evaluation in a flow chart shown in Figure 1.1, including: creating an inventory of process objectives; reaching consensus on process evaluation questions to be answered; creating measurement tools to assess process objectives; analyzing data; and creating user-friendly reports. And as a final note, Linnan and Steckler recommend that stakeholders be involved in every aspect of this process.

Last updated on Saturday, 23 November, 2013

Funded by the National Library of Medicine under contract # HHS-N-276-2011-00008-C.