Skip all navigation and go to page content

OERC Blog

NN/LM Outreach Evaluation Resource Center

Getting Started in Evaluation – Evaluation Guides from the OERC

New to the world of evaluation? What is your boss talking about when she says she wants you to measure outcomes, not outputs?  What is an indicator? How many responses should you get from your surveys?

Sometimes people think evaluation is just the form that you fill out at the end of a class or event. But in fact evaluation can start at the beginning of the project when you do a community assessment and evaluation includes building support for your project from your stakeholders. And it continues through making an evaluation plan as part of your project, gathering data, analyzing data, and reporting the data back to the stakeholders in a way that it is useful.  Here is a model that the CDC uses to describe the evaluation framework:

CDC Framework for Program Evaluation

The Outreach Evaluation Resource Center (OERC) has a series of three booklets entitled Planning and Evaluating Health Information Outreach Projects that guide people through the evaluation process, from needs assessment to analyzing data.  While focusing on “health information outreach” this series of books can be used to learn how to do evaluation for any type of project.

Booklet 1: Getting Started with Community-Based Outreach

  • Getting organized: literature review; assembling team of advisors; taking an inventory; developing evaluation questions
  • Gathering information: primary data; secondary data, and publicly accessible databases
  • Assembling, Interpreting and Acting: summarizing data and keeping stakeholders involved

Booklet 2: Planning Outcomes-Based Outreach Projects

  • Planning your program with a logic model to connect activities to outcomes
  • Planning your process assessment
  • Developing an outcomes assessment plan, using indicators, objectives and an action plan

Booklet 3: Collecting and Analyzing Evaluation Data

  • Designing data collection methods; collecting data; summarizing and analyzing data for:
    • Quantitative methods
    • Qualitative methods

The books can be read in HTML, downloaded as a PDF or physical booklets can be ordered for free from the OERC by sending an email request to: nnlm@u.washington.edu

Learn more about the CDC’s Evaluation Framework: http://www.cdc.gov/eval/framework/

 

 

Fast Track Interview Analysis: The RITA Method

If you want a systematic way to analyze interview data, check out the Rapid Identification of Themes from Audio Recordings (RITA) method described in Neal et al. (2014). This method skips the time-consuming transcription process, because you conduct your analysis while listening to the recordings.  Also, the process maintains nonverbal elements of your data (i.e., intonation), which are lost when interviews are transcribed. The authors presented a case in their article to demonstrate how to use the RITA method.

The five-step RITA process, briefly described below, is meant to be used with multiple raters:

  1. Develop focused evaluation questions. Don’t try to extract every detail from the recordings. Instead, write some focused evaluation questions to guide your analysis. For instance, you might want to know how participants applied lessons learn from a class on consumer health information or what services are desired by a specific type of library user.
  2. Create a codebook. Develop a list of themes by talking with the project team, reviewing interviewer notes, or checking theories or literature related to your project. For their sample case, the authors used eight themes. That’s probably is the upper limit for the number of themes that can be effectively used for this process. Once you have the list, create a codebook with detailed theme definitions.
  3. Develop a coding form. (See the figure below.) This will be used by all coders to record absence or presence of a theme in time-specified (e.g., 3 minute) segments of the interview. They listen to a time segment, mark if any themes were present, and then repeat the process with the next segment. (The article describes the process for figuring out the most appropriate time segment length for your project.) If you want, you can also incorporate codes for “valance,” indicating if comments were expressed positively, negatively, or in neutral tones.
  4. Have the coding team pilot-test the codebook and coding form on a small subset of interviews. The team then should refine both documents before coding all recordings.
  5. Code the recordings. At this stage, one coder per interview is acceptable, although the authors recommend that a subset of the interviews be coded by multiple coders and results tested for rater agreement.

RITA sample coding sheet (spreadsheet with themes in first column and time segments of 3-minute length in top row for recording presence of themes.

While the RITA process may seem time consuming, it is much more efficient than producing verbatim transcripts. Once the authors finalized their coding form, it took a team member about 68 minutes to code a one-hour interview. Because coded data was expressed in numbers, it allowed the authors to assess inter-rater reliability (agreement), which demonstrated an acceptable level of agreement among coders. Rater agreement adds credibility to your findings and can be helpful if you seek to publish your results.

While the RITA method is used with qualitative data, it is essentially a quantitative analytic method, producing numbers from text.  That leads me to my main concern. By reducing the data to counts, you lose some of the rich detail and subtle nuances that are the hallmarks of qualitative data. However, most evaluation studies use mixed methods to provide a complete picture of their programs.  In that spirit, you can  simply  keep track of time segments that contain particularly great quotes and stories, then transcribe and include them in your project report. They will complement nicely the findings from your RITA analysis.

Here is the full citation for the Neal et al  article, which provides excellent instructions for conducting the RITA process.

Neal JW, Neal ZP, VanDyke E, Kornbluh M. Expediting the analysis of qualitative data in evaluation: a procedure for the rapid identification of themes from audio recordings (RITA). American Journal of Evaluation. 2015; 36(1): 118-132.

 

 

 

 

 

 

Designing Questionnaires for the Mobile Age

How does your web survey look on a handheld device?  Did you check?

The Pew Research Center reported that 27% of respondents to one of its recent surveys answered using a smartphone. Another 8% used a tablet. That means over one-third of participants used handheld devices to answer the questionnaire. Lesson learned: Unless you are absolutely sure your respondents will be using a computer, you need to design with mobile devices in mind.

As a public opinion polling organization, the Pew Center knows effective practices in survey research. It offers advice on developing questionnaires for handhelds in its article Tips for Creating Web Surveys for Completion on a Mobile Device. The top suggestion is to be sure your survey software is optimized for smartphones and tablets. The OERC uses SurveyMonkey, which fits this criteria. Many other popular Web survey applications do as well. Just be sure to check.

However, software alone will not automatically create surveys that are usable on handhelds devices. You also need to follow effective design principles. As a rule of thumb, keep it simple. Use short question formats. Avoid matrix-style questions. Keep the length of your survey short. And don’t get fancy: Questionnaires with logos and icons take longer to load on smartphones.

This article provides a great summary of tips to help you design mobile-device friendly questionnaires. My final word of advice? Pilot test questionnaires on computers, smartphones, and tablets. That way, you can make sure you are offering a smooth user experience to all of your respondents.

Many smart phones with application tiles on their touchscreens
Many smart phones with application tiles on their touchscreens

 

DYI Tool for Program Success Stories (Program Success Stories Part 2)

Last week, I wrote about program success stories. As a follow-up, I want to introduce you to a story builder tool available at the CDC Injury Prevention and Control web site. The story builder takes you through three steps to produce an attractive, well-written program success story. Each step offers downloadable Microsoft Word documents to walk you through the process.

Step 1: The worksheets are designed to gather and organize project information for your story. I think it would be interesting to use this step as a participatory activity. You could pull together your project team or a group of stakeholders to talk through questions in this worksheet. The discussion would help group members articulate the program’s value from their perspective.

Step 2: This step provides a story builder template to write your story, section by section. Each section has a field to develop a paragraph of your story, with some tips for writing in a compelling, user-friendly way. Each completed field prepares you for the final step.

Step 3: Here, you can download a layout template, where you transfer the paragraphs from your story builder template into the layout. Because this is a Word document, you can change background design, font, or even the size and placement of pictures and call-out quote boxes.

If you are thinking of trying your hand at program success stories, this story building web page provides some useful DYI tools to help you get started.

"What is Your Story" typed on paper in an old typewriter

Telling Good Stories about Good Programs

Sometimes our program successes are a well-kept secret, hidden deep in our final reports under pages of statistics, tables, and descriptive details. There is a way to shine a stronger light on positive program impacts: program success stories. These are short (1-2 page) narratives that are designed to educate policy makers, attract partners, and share effective practices among colleagues.

The Centers for Disease Control and Prevention deserves credit in leading a program success story movement within the public health sector. You can find lots of resources at the CDC’s website for developing program success stories. A quick Google search will turn up many success story web pages from public health departments, such as the three listed below:

If you want to create success stories for your program or organization, you need to start with a plan. You want to establish a routine to collect information in a timely manner. To get started, check out the CDC Division of Oral Health’s Tips for Writing an Effective Success Story. For more details, the CDC offers the workbook Impact and Value: Telling Your Program’s Story. The CDC Division of Adolescent and School Health also has a how-to guide for writing success stories: How to Develop a Success Story. Finally, you might find this Success Story Data Collection Tool helpful for organizing and writing your program story.  A data collection sheet could be particularly useful if multiple team members are involved in collecting success story data. The data collection tool is available in PDF or Word formats.

stockfresh_687180_magic-book-with-pages-transforming-into-birds_sizeS (2)

Easy Email Newsletters for Keeping Stakeholders Informed

Do you find it difficult to ensure that you are keeping your stakeholders up to date throughout your program? In her AEA Summer Institute class entitled “An Executive Summary is Not Enough: Effective Evaluation Reporting Techniques,” Kylie Hutchinson from Community Solutions suggests a very interesting product for staying in touch with stakeholders. The product is Constant Contact, a tool that allows you to put the latest stats, activities, or planning updates into a newsletter format that goes into the body of an email, allowing stakeholders to easily stay updated on the progress of your program. In addition, you, the evaluator, get feedback on who opens the email and what they click on.

Here is a sample email created for free in just a few moments, showing NN/LM stakeholders what activities took place in May in an imaginary community college outreach project.

Constant Contact

Components of Process Evaluation

At the American Evaluation Association Summer Institute, Laura Linnan, Director of the Carolina Collaborative for Research on Work & Health at UNC Gillings School of Public Health, did a workshop entitled Process Evaluation: What You Need to Know and How to Get Started. According to the CDC, process evaluation is the systematic collection of information on a program’s inputs, activities, and outputs, as well as the program’s context and other key characteristics.

Logic Model Image from CDC

Process evaluation looks at the specific activities that take place during an outreach project to ensure that planned interventions are carried out equally at all sites and with all participants, to explain why successes happen or do not happen, and to understand the relationships between the project components. Process evaluation can be extremely important in making adjustments to ensure the project’s success, and determining how or whether to do a project again.

In the workshop I attended, Linnan walked through the details covered in Chapter 1 of the book Process Evaluation for Public Health Interventions and Research by Laura Linnan and Allan Steckler. This chapter presents an overview of process evaluation methods. In it, they define a set of terms that describe the components of process evaluation (Table 1.1). These components are valuable to understand, because evaluators can look in detail at each component to determine which ones should be evaluated.

  1. Context
  2. Reach
  3. Dose delivered
  4. Dose received
  5. Fidelity
  6. Implementation
  7. Recruitment

In addition, the authors describe a step-by-step process for designing and implementing process evaluation in a flow chart shown in Figure 1.1, including: creating an inventory of process objectives; reaching consensus on process evaluation questions to be answered; creating measurement tools to assess process objectives; analyzing data; and creating user-friendly reports. And as a final note, Linnan and Steckler recommend that stakeholders be involved in every aspect of this process.

Lesson Learned: Outputs are Cool!

AEA Summer Institute Logo

Cindy Olney and I just returned from the American Evaluation Association Summer Institute in Atlanta, GA. The blog posts for the next couple of months will be filled with lessons learned from the Institute. I am going to start with Outputs, because they were the greatest surprise to me.

In his “Introduction to Program Evaluation,” Thomas Chapel, Chief Evaluation Officer for the Centers for Disease Control and Prevention, said that he thought outputs were just as important as outcomes. This was quite shocking to me, since it always seemed like outputs were just the way of counting what had been done, and not nearly as interesting as finding out if the desired outcome had happened.

Outputs are the tangible products of the activities that take place in a project. For example, let’s say the project’s goal is to reduce the number of children with Elevated Blood Lead Levels (EBLL) by screening children to identify the ones with EBLL and then referring them to health professionals for medical management. In this brief project description, the activities would be to:

1) Screen children to identify the ones with EBLL
2) Refer them to health professionals for medical management

If outputs are the tangible products of the activities, they are sometimes thought to be something countable, like “the number of children screened for EBLL” and “the number of referrals.” This is how the project manager can ensure that the activities took place that were planned.

However, if you think about the way an activity can take place, you can see that some methods of completing the activities might lead to a successful outcomes, and some might not. A better way of thinking of the outputs might be “what would an output look like that would lead to the outcome that we are looking for?” To use “referrals” as an example, let’s say that during the program 100% of the children identified with EBLL were referred to health professionals, but only 30% of them actually followed up and went to a health professional. If the only information you gathered was the number of referrals, you cannot tell why the success rate was so low. Some of the things that could go wrong in a referral is that people are referred to physicians who are not taking more patients, or to physicians who don’t speak the same language as the parents of the child. So you might want to define the referral output as including those factors. The new output measure could be “the number of referrals to ‘qualified’ physicians,” in which ‘qualified’ is defined by the attributes you need to see in the physicians, such as physicians who are taking new patients, or physicians who speak the same language as the family.

The lesson for me is that outputs are as important as outcomes because by thinking carefully about outputs at the beginning of the planning process, you can ensure that the project has the greatest chance of successful outcomes, and by using outputs during process evaluation, you can make any needed corrections in the process as it is happening to ensure the greatest success of the project.

Use Evaluation Samples as Shortcuts

VIVA project logo

We are often asked for samples of questionnaires and evaluations for information outreach projects. This peer tutor project from the Texas Rio Grande Valley has posted a number of sample evaluation documents that can be modified for other information outreach projects

The ¡VIVA! (Vital Information for a Virtual Age) Project is a high school-based health information outreach initiative in which high school students (peer tutors) from the South Texas Independent School District are trained to use and promote MedlinePlus, a consumer-health database of the NIH National Library of Medicine. Since 2001, these teen peer tutors have taught others about MedlinePlus through class demonstrations, student orientations, school open houses, and community events.

Evaluation has been a strong component of this program since its inception. As part of an online Implementation Guide , the ¡VIVA! Peer Tutor Project team posted many sample evaluation forms and documents. These can be modified to fit your own outreach evaluation needs. Here are some examples:

Logic Model and Evaluation Plan: Here is a sample of how to tie a logic model to the project’s evaluation plan based on outcomes.

Interview guides: It can take a long time to form the perfect questions for interviewing individuals or focus groups. See if these questions will work, and if not, see if you can adjust them.

Training Assessments: Here are some basic questionnaires designed to find out what students have learned and how they would rate their training session.

Feel free to use these shortcuts to make evaluation fit more easily into your workflow! If you have any questions about the program or the evaluation forms, feel free to contact Cynthia Olney.

Improve Your Presentations

Death by Presentation by Frits Ahlefeldt-Laurvig (CC BY-ND 2.0)  No changes.

Death by Presentation by Frits Ahlefeldt-Laurvig (CC BY-ND 2.0) No changes.

In a recent blog post, evaluator Stephanie Evergreen suggested that people no longer ask for power point slides at the end of a presentation (Stop Asking if the Slides are Available).  Her point is that the slides should support the speaker and be fairly useless on their own.  If the audience needs a reminder of what was said, the speaker should provide handouts, with main points and resources listed, as well as links to engaging dashboards and infographics.

In her blog post, Stephanie Evergreen has 10 points for improving your presentations.  You don’t like it when people read their slides? Her first point is to remove text from slides so the focus of the audience goes back to what the speaker is saying. A complementary point she makes is that the graphics on the slide be emotional to help the audience remember what the speaker is saying.

What makes Evergreen’s 10 points unique in the world of presentation advice is that many of them are about charts and graphs.  For example, her point, “Choose the right chart so that your results tell the best story,”  ties what some might see as dry charts into the story that your presentation is telling. Another one, “Keep it easy to interpret your graphs with close data labels and a descriptive subtitle,” is a suggestion that re-occurs in her blog and book, Presenting Data Effectively: Communicating Your Findings for Maximum Impact.  For a detailed checklist of how to make a better graph, take a look at her Data Visualization Checklist.

Last updated on Saturday, 23 November, 2013

Funded by the National Library of Medicine under contract # HHS-N-276-2011-00008-C.