Skip all navigation and go to page content


NN/LM Outreach Evaluation Resource Center

Archive for the ‘Practical Evaluation’ Category

Lesson Learned: Outputs are Cool!

Friday, June 5th, 2015

AEA Summer Institute Logo

Cindy Olney and I just returned from the American Evaluation Association Summer Institute in Atlanta, GA. The blog posts for the next couple of months will be filled with lessons learned from the Institute. I am going to start with Outputs, because they were the greatest surprise to me.

In his “Introduction to Program Evaluation,” Thomas Chapel, Chief Evaluation Officer for the Centers for Disease Control and Prevention, said that he thought outputs were just as important as outcomes. This was quite shocking to me, since it always seemed like outputs were just the way of counting what had been done, and not nearly as interesting as finding out if the desired outcome had happened.

Outputs are the tangible products of the activities that take place in a project. For example, let’s say the project’s goal is to reduce the number of children with Elevated Blood Lead Levels (EBLL) by screening children to identify the ones with EBLL and then referring them to health professionals for medical management. In this brief project description, the activities would be to:

1) Screen children to identify the ones with EBLL
2) Refer them to health professionals for medical management

If outputs are the tangible products of the activities, they are sometimes thought to be something countable, like “the number of children screened for EBLL” and “the number of referrals.” This is how the project manager can ensure that the activities took place that were planned.

However, if you think about the way an activity can take place, you can see that some methods of completing the activities might lead to a successful outcomes, and some might not. A better way of thinking of the outputs might be “what would an output look like that would lead to the outcome that we are looking for?” To use “referrals” as an example, let’s say that during the program 100% of the children identified with EBLL were referred to health professionals, but only 30% of them actually followed up and went to a health professional. If the only information you gathered was the number of referrals, you cannot tell why the success rate was so low. Some of the things that could go wrong in a referral is that people are referred to physicians who are not taking more patients, or to physicians who don’t speak the same language as the parents of the child. So you might want to define the referral output as including those factors. The new output measure could be “the number of referrals to ‘qualified’ physicians,” in which ‘qualified’ is defined by the attributes you need to see in the physicians, such as physicians who are taking new patients, or physicians who speak the same language as the family.

The lesson for me is that outputs are as important as outcomes because by thinking carefully about outputs at the beginning of the planning process, you can ensure that the project has the greatest chance of successful outcomes, and by using outputs during process evaluation, you can make any needed corrections in the process as it is happening to ensure the greatest success of the project.

Use Evaluation Samples as Shortcuts

Friday, May 29th, 2015

VIVA project logo

We are often asked for samples of questionnaires and evaluations for information outreach projects. This peer tutor project from the Texas Rio Grande Valley has posted a number of sample evaluation documents that can be modified for other information outreach projects

The ¡VIVA! (Vital Information for a Virtual Age) Project is a high school-based health information outreach initiative in which high school students (peer tutors) from the South Texas Independent School District are trained to use and promote MedlinePlus, a consumer-health database of the NIH National Library of Medicine. Since 2001, these teen peer tutors have taught others about MedlinePlus through class demonstrations, student orientations, school open houses, and community events.

Evaluation has been a strong component of this program since its inception. As part of an online Implementation Guide , the ¡VIVA! Peer Tutor Project team posted many sample evaluation forms and documents. These can be modified to fit your own outreach evaluation needs. Here are some examples:

Logic Model and Evaluation Plan: Here is a sample of how to tie a logic model to the project’s evaluation plan based on outcomes.

Interview guides: It can take a long time to form the perfect questions for interviewing individuals or focus groups. See if these questions will work, and if not, see if you can adjust them.

Training Assessments: Here are some basic questionnaires designed to find out what students have learned and how they would rate their training session.

Feel free to use these shortcuts to make evaluation fit more easily into your workflow! If you have any questions about the program or the evaluation forms, feel free to contact Cynthia Olney.

Improve Your Presentations

Friday, May 22nd, 2015
Death by Presentation by Frits Ahlefeldt-Laurvig (CC BY-ND 2.0)  No changes.

Death by Presentation by Frits Ahlefeldt-Laurvig (CC BY-ND 2.0) No changes.

In a recent blog post, evaluator Stephanie Evergreen suggested that people no longer ask for power point slides at the end of a presentation (Stop Asking if the Slides are Available).  Her point is that the slides should support the speaker and be fairly useless on their own.  If the audience needs a reminder of what was said, the speaker should provide handouts, with main points and resources listed, as well as links to engaging dashboards and infographics.

In her blog post, Stephanie Evergreen has 10 points for improving your presentations.  You don’t like it when people read their slides? Her first point is to remove text from slides so the focus of the audience goes back to what the speaker is saying. A complementary point she makes is that the graphics on the slide be emotional to help the audience remember what the speaker is saying.

What makes Evergreen’s 10 points unique in the world of presentation advice is that many of them are about charts and graphs.  For example, her point, “Choose the right chart so that your results tell the best story,”  ties what some might see as dry charts into the story that your presentation is telling. Another one, “Keep it easy to interpret your graphs with close data labels and a descriptive subtitle,” is a suggestion that re-occurs in her blog and book, Presenting Data Effectively: Communicating Your Findings for Maximum Impact.  For a detailed checklist of how to make a better graph, take a look at her Data Visualization Checklist.

Low Cost Mapping Tools on NLM’s Community Health Maps Blog

Friday, May 8th, 2015

Map of Childhood Lead Poisoining Risk in Philadelphia, PA from CDC Map Gallery

Have you ever wanted to be able to use mapping for your outreach needs, but thought that making maps would be too expensive, time-consuming, or just too difficult?   The National Library of Medicine has a blog called Community Health Maps: Information on Low Cost Mapping Tools for Community-based Organizations, with the goal of facilitating the use of geographic information system (GIS) mapping by providing information about low cost mapping tools, software reviews, best practices, and the experiences of those who have successfully implemented a mapping workflow as part of their work.  The blog is moderated by Kurt Menke, a certified GIS professional.

Here are some examples of the kinds of things you can find on the Community Health Maps blog:

  • A short guide for using iForm for field data collection. iForm is an app that can be used on iPads, iPhones and Android devices, and has a free version.  Using this app, you could go to different locations, gather data (for example, demographic information about attendance at your program), and view it in tabular or map format.
  • A description of a project using youth in the Philippines to collect data on the needs of their communities.  Technology + Youth = Change showed how a dozen donated phones helped 30 young adults survey and map information on access to water, electricity, jobs, and more.
  • A review of a pilot project done by the Seattle Indian Health Board’s Urban Indian Health Institute on noise pollution and health in the urban environment. One of the goals of the pilot project was to determine whether this kind of data collection and analysis would be feasible with other urban Indian health organizations, so they selected participants who had limited experience with data collection and GIS. The feedback suggested that the GIS software tools were very user-friendly and effective.

Photo credit: Childhood Lead Poisoning Risk Analysis, Philadelphia, Pennsylvania, from the CDC Map Gallery

Infographics Basics: A Picture is Worth 1000 Data Points

Friday, April 24th, 2015

Open Access Week at University of Cape Town infographicYou’ve been collecting great data for your library, and now you have to figure out how to use it to convince someone of something, for example how great your library is. Part of the trick is turning that data into a presentation that your stakeholders understand – especially if you are not there to explain it.  Infographics are images that make data easy to understand in a way that gets your message across.

It turns out it doesn’t have to be difficult or expensive to create your own infographics.  Last week I went to a hands-on workshop at the Texas Library Association called “Infographics: One Picture is  Worth 1,000 Data Points,” taught by Leslie Barrett, Education Specialist from the Education Service Center Region 13 in Austin, TX. Using this website as her interactive “handout”, Leslie walked us through the process of creating an infographic (and as a byproduct of this great class, she also demonstrated a number of free instructional resources, such as Weebly, Padlet, and Thinglink).

Starting at the top of the page, click on anything with a hyperlink.  You will find a video as well as other “infographics of infographics”  which demonstrate how and why infographics can be used.  There are also a variety of examples to evaluate as part of the learning process.

Finally, there is information on the design process and resources that make infographics fairly easy to create.  These resources, such as Piktochart and Easelly, have free subscriptions for simple graphics and experimenting.

Leslie Barrett allowed us to share this website with you, so feel free to get started making your own infographics!

Image credit: Open Access Week at University of Cape Town by Shihaam Donnelly / CC BY SA 3.0

An Easier Way to Plan: Tearless Logic Models

Friday, April 10th, 2015

Sample Logic Model

Are you apprehensive when someone says it’s time to do “outcome-based planning using a logic model?” The Wichita State University Community Psychology Practice and Research Collaborative, and the Community Psychology Doctoral Program in Wichita, KS, have come up with an easy way to do logic models. This is described in an article, “Tearless Logic Model,” in the Global Journal of Community Psychology Practice.

Their goal was to create a facilitated, non-intimidating logic model process that would be more likely to be used in planning.  This approach is designed to give community-based groups, faith-based organizations, smaller nonprofits and people with little experience in strategic planning greater impact when planning community projects.

Tearless Logic Model planning requires only flip charts, magic markers, blue painters’ tape and a safe space to work with a group.  Jargon is eliminated, and is replaced with simple terms that anyone can understand.  For example, instead of asking “what are the anticipated impacts,” a facilitator would ask, “if you really got it right, what would it look like in 10 or 20 years?”

The step by step process that anyone can use is found in their article as well as links to a Prezi visual guide and the full PDF that includes a helpful template.

Ashlee D. Lien, Justin P. Greenleaf, Michael K. Lemke, Sharon M. Hakim, Nathan P. Swink, Rosemary Wright, Greg Meissen. Tearless Logic Model. Global Journal of Community Psychology Practice [Internet].  2011 Dec [cited 2015 Apr 10];2(2). Available from


Updated Community Health Status Indicators (CHSI)

Friday, March 27th, 2015

Peer county mapsThe OERC is excited to get the word out about the Center for Disease Control and Prevention (CDC)’s newly updated and redesigned Community Health Status Indicators (CHSI).  The new CHSI 2015 represents the collaboration of public health partners in the public, non-profit and research communities, including the National Library of Medicine.

The OERC recommends the CHSI as a possible resource in the data gathering portion of planning outreach projects or needs assessments. CHSI 2015 is an interactive online tool that produces health profiles for all 3,143 counties in the United States. Each profile includes key indicators of health outcomes that describe the population health status of a county. What makes CHSI 2015 an important tool is that it includes comparisons to “peer counties” – groups of counties that are similar to each other based on 19 variables, including population size, percent high school graduates and household income.

CHSI Summary Comparison ReportFor each county, CHSI 2015 provides a Summary Comparison Report.  Using Karen Vargas’ childhood home of Union County, PA as an example, this report (right) shows that in the case of the overall cancer death rate, Union County does better than most of its peer counties.  But in the case of stroke death rate, they do worse.

Distribution Display – bar chartsBy selecting a specific indicator, such as coronary heart disease death rate, the interactive CHSI 2015 will produce a bar chart showing Union County in comparison to its peer counties, as well as the US median and the Healthy People 2020 target (left).

More detailed, downloadable data for each peer county can also be found (below), as well as a web page detailing the sources of the data for each indicator. CHSI 2015 provides a helpful How to use CHSI web page that explains each feature and provides helpful hints.

Distribution data in a downloadable format

CHSI 2015 is designed to complement other available sources of community health indicators including the Robert Wood Johnson Foundation’s County Health Rankings and Roadmaps.

What Shapes Health? A Story about Data Visualization

Friday, March 13th, 2015

Medicaid birth map 2006-08

It was an amazing a-ha moment. We kind of blinked at each other, and then simultaneously said ‘We got to do something.’ – Dr. Nancy Hardt, University of Florida

This week on National Public Radio’s (NPR) All Things Considered was a story of what happened when Dr. Nancy Hardt, an OB-GYN, used data from Medicaid birth records to see where children were born into poverty in Gainesville, FL to try and identify ways to intervene and prevent poor childhood health outcomes. She was surprised to see a 1 square mile high-density ‘hot spot’ of births in dark blue appear in her map above. Dr. Hardt was encouraged to share her map with Sheriff Sadie Darnell, who pulled out a map of her own of Gainesville.

Sheriff Darnell’s map showed an exact overlay with the ‘hot spot’ on Dr. Hardt’s map of the highest crime rates in the city. By visiting the area they identified many things in the community that were barriers to good health including hunger, substandard housing, and a lack of medical care facilities – the closest location for uninsured patients was a 2 hour bus ride each way to the county health department. You’ll want to check out the rest of A Sheriff and A Doctor Team Up to Map Childhood Trauma to learn more about a mobile health clinic, what data from additional maps showed, and other steps they have taken since to help improve health outcomes for the community.

This story is the latest from the NPR series What Shapes Health, which was inspired in response to a recent Robert Wood Johnson Foundation poll about what beliefs and concerns Americans have regarding health. You can read an overview and download the full report of their results at

OERC’s Tools and Resources for Evaluation Guide

Friday, March 6th, 2015

Data Dashboard Example

Have you ever found yourself trying to do an evaluation activity, but needing that one helpful tool? Or perhaps you need a step-by-step guide on how to do a community assessment, or are looking for ways to build evaluation into a project that you are planning?

The OERC has an online guide called Tools and Resources for Evaluation that you and your library can use to evaluate your programs. Here are some of the types of tools and resources described in the Guide.

Community Oriented Outreach

  • Tips on successful collaborations and tools for improving collaboration with community networks
  • Toolkits for practical participatory evaluation and processes for conducting outcome-based evaluations

Evaluation Planning

  • Step-by-step guides on incorporating evaluation planning into your outreach projects
  • Instructions on using logic models for program planning

Data Collection and Analysis

  • Tips for questionnaire development
  • Resources for statistical methods of data analysis
  • Guides for analyzing qualitative and quantitative data

Reporting and Visualizing

  • Help with creating popular data dashboards
  • Descriptions of data visualization methods
  • Tools and TEDtalks about how you will present your data

Another Coffee Break: Word and Excel Templates

Friday, February 6th, 2015

AEA Coffee

Here at the Outreach Evaluation Resource Center (OERC) we began 2015 blogging about the CDC Coffee Breaks. For February we’re offering a refill by featuring some notes from a recent American Evaluation Association (AEA) coffee break webcast. Unlike the CDC, the 20 minute AEA coffee break webcasts are not freely available to the public but are an included benefit of AEA membership. The webcast briefly covered best practices in data visualization using two commonly available resources (Microsoft Word and Excel) and how to automate use of them by creating templates for report format consistency and easier workflow.

Some great resources to learn more how to do this and bookmark for future reference include

Specific for Word

Specific for Excel


Last updated on Saturday, 23 November, 2013

Funded by the National Library of Medicine under contract # HHS-N-276-2011-00008-C.