Skip all navigation and go to page content
NN/LM Home About NER | Contact NER | Feedback |Site Map | Help | Bookmark and Share

Archive for the ‘OERC’ Category

Components of Process Evaluation

Monday, June 15th, 2015

[Guest post by Karen Vargs, OERC]

At the American Evaluation Association Summer Institute, Laura Linnan, Director of the Carolina Collaborative for Research on Work & Health at UNC Gillings School of Public Health, did a workshop entitled Process Evaluation: What You Need to Know and How to Get Started. According to the CDC, process evaluation is the systematic collection of information on a program’s inputs, activities, and outputs, as well as the program’s context and other key characteristics.

Process evaluation looks at the specific activities that take place during an outreach project to ensure that planned interventions are carried out equally at all sites and with all participants, to explain why successes happen or do not happen, and to understand the relationships between the project components. Process evaluation can be extremely important in making adjustments to ensure the project’s success, and determining how or whether to do a project again.

In the workshop I attended, Linnan walked through the details covered in Chapter 1 of the book Process Evaluation for Public Health Interventions and Research by Laura Linnan and Allan Steckler. This chapter presents an overview of process evaluation methods. In it, they define a set of terms that describe the components of process evaluation (Table 1.1). These components are valuable to understand, because evaluators can look in detail at each component to determine which ones should be evaluated.

  1. Context
  2. Reach
  3. Dose delivered
  4. Dose received
  5. Fidelity
  6. Implementation
  7. Recruitment

In addition, the authors describe a step-by-step process for designing and implementing process evaluation in a flow chart shown in Figure 1.1, including: creating an inventory of process objectives; reaching consensus on process evaluation questions to be answered; creating measurement tools to assess process objectives; analyzing data; and creating user-friendly reports. And as a final note, Linnan and Steckler recommend that stakeholders be involved in every aspect of this process.

Lessons Learned: Outputs are Cool!

Tuesday, June 9th, 2015

[Guest post by Karen Vargas, OERC]

Cindy Olney and I just returned from the American Evaluation Association Summer Institute in Atlanta, GA. My mind exploded from how much I learned! The blog posts for the next couple of months will be filled with lessons learned from the Institute. I am going to start with Outputs, because they were the greatest surprise to me.

In his “Introduction to Program Evaluation,” Thomas Chapel, Chief Evaluation Officer for the Centers for Disease Control and Prevention, said that he thought outputs were just as important as outcomes. This was quite shocking to me, since it always seemed like outputs were just the way of counting what had been done, and not nearly as interesting as finding out if the desired outcome had happened.

Outputs are the tangible products of the activities that take place in a project. For example, let’s say the project’s goal is to reduce the number of children with Elevated Blood Lead Levels (EBLL) by screening children to identify the ones with EBLL and then referring them to health professionals for medical management. In this brief project description, the activities would be to:

1) Screen children to identify the ones with EBLL
2) Refer them to health professionals for medical management

If outputs are the tangible products of the activities, they are sometimes thought to be something countable, like “the number of children screened for EBLL” and “the number of referrals.” This is how the project manager can ensure that the activities took place that were planned.

However, if you think about the way an activity can take place, you can see that some methods of completing the activities might lead to a successful outcomes, and some might not. A better way of thinking of the outputs might be “what would an output look like that would lead to the outcome that we are looking for?” To use “referrals” as an example, let’s say that during the program 100% of the children identified with EBLL were referred to health professionals, but only 30% of them actually followed up and went to a health professional. If the only information you gathered was the number of referrals, you cannot tell why the success rate was so low. Some of the things that could go wrong in a referral is that people are referred to physicians who are not taking more patients, or to physicians who don’t speak the same language as the parents of the child. So you might want to define the referral output as including those factors. The new output measure could be “the number of referrals to ‘qualified’ physicians,” in which ‘qualified’ is defined by the attributes you need to see in the physicians, such as physicians who are taking new patients, or physicians who speak the same language as the family.

The lesson for me is that outputs are as important as outcomes because by thinking carefully about outputs at the beginning of the planning process, you can ensure that the project has the greatest chance of successful outcomes, and by using outputs during process evaluation, you can make any needed corrections in the process as it is happening to ensure the greatest success of the project.

Use Evaluation Samples as Shortcuts

Monday, June 1st, 2015

[Contributed by the NN/LM OERC]

We are often asked for samples of questionnaires and evaluations for information outreach projects. This peer tutor project from the Texas Rio Grande Valley has posted a number of sample evaluation documents that can be modified for other information outreach projects

The ¡VIVA! (Vital Information for a Virtual Age) Project is a high school-based health information outreach initiative in which high school students (peer tutors) from the South Texas Independent School District are trained to use and promote MedlinePlus, a consumer-health database of the NIH National Library of Medicine. Since 2001, these teen peer tutors have taught others about MedlinePlus through class demonstrations, student orientations, school open houses, and community events.

Evaluation has been a strong component of this program since its inception. As part of an online Implementation Guide , the ¡VIVA! Peer Tutor Project team posted many sample evaluation forms and documents. These can be modified to fit your own outreach evaluation needs. Here are some examples:

Logic Model and Evaluation Plan Here is a sample of how to tie a logic model to the project’s evaluation plan based on outcomes.

Interview guides: It can take a long time to form the perfect questions for interviewing individuals or focus groups. See if these questions will work, and if not, see if you can adjust them.

Training Assessments: Here are some basic questionnaires designed to find out what students have learned and how they would rate their training session.

Feel free to use these shortcuts to make evaluation fit more easily into your workflow!

Improve Your Presentations

Tuesday, May 26th, 2015

In a recent blog post, evaluator Stephanie Evergreen suggested that people no longer ask for power point slides at the end of a presentation (Stop Asking if the Slides are Available).  Her point is that the slides should support the speaker and be fairly useless on their own.  If the audience needs a reminder of what was said, the speaker should provide handouts, with main points and resources listed, as well as links to engaging dashboards and infographics.

In her blog post, Stephanie Evergreen has 10 points for improving your presentations.  You don’t like it when people read their slides? Her first point is to remove text from slides so the focus of the audience goes back to what the speaker is saying. A complementary point she makes is that the graphics on the slide be emotional to help the audience remember what the speaker is saying.

What makes Evergreen’s 10 points unique in the world of presentation advice is that many of them are about charts and graphs.  For example, her point, “Choose the right chart so that your results tell the best story,”  ties what some might see as dry charts into the story that your presentation is telling. Another one, “Keep it easy to interpret your graphs with close data labels and a descriptive subtitle,” is a suggestion that re-occurs in her blog and book, Presenting Data Effectively: Communicating Your Findings for Maximum Impact.  For a detailed checklist of how to make a better graph, take a look at her Data Visualization Checklist.

Please visit WP-Admin > Options > Snap Shots and enter the Snap Shots key. How to find your key