Skip all navigation and go to page content

OERC Blog

A Weblog from the NN/LM Outreach Evaluation Resource Center

Archive for the ‘Practical Evaluation’ Category

Another Coffee Break: Word and Excel Templates

Friday, February 6th, 2015

AEA Coffee

Here at the Outreach Evaluation Resource Center (OERC) we began 2015 blogging about the CDC Coffee Breaks. For February we’re offering a refill by featuring some notes from a recent American Evaluation Association (AEA) coffee break webcast. Unlike the CDC, the 20 minute AEA coffee break webcasts are not freely available to the public but are an included benefit of AEA membership. The webcast briefly covered best practices in data visualization using two commonly available resources (Microsoft Word and Excel) and how to automate use of them by creating templates for report format consistency and easier workflow.

Some great resources to learn more how to do this and bookmark for future reference include

Specific for Word

Specific for Excel

 

52 Weeks of Better Evaluation

Friday, January 30th, 2015

BetterEvaluation.org is an international collaboration that encourages sharing of evaluation methods, approaches and processes for improvement. BetterEvaluation offers yearly blog themes for their staff and guest writers to focus on, and have wrapped up the highlights of their ’52 Weeks of BetterEvaluation’ 2014 theme in a post at http://betterevaluation.org/node/4682 For 2015 they are featuring ’12 Months of BetterEvaluation’ with multiple posts during a month, starting with impact evaluation in January.

A ‘top 5′ selection from the ‘52 Weeks of BetterEvaluation‘ post that is likely to be of interest to National Network of Libraries of Medicine (NN/LM) members includes

  1. Top ten developments in qualitative evaluation over the past decade (link to part 1, part 2)
  2. Fitting reporting methods to evaluation findings and audiences (link)
  3. Infographics, including step by step instructions in piktochart (link)
  4. Innovation in evaluation (link)
  5. Presenting data effectively (link)

Freebie Friday: Measuring Success Toolkit

Friday, January 23rd, 2015

Measurement and Evaluation Staircase

Monitoring and evaluation (M&E) is a form of assessment used to help improve the performance and achievement of program results and often used by both non-government organizations (NGOs) and government agencies. The staircase diagram above describes six questions that M&E can help answer through program planning, monitoring, and evaluation. More information clarifying the difference between monitoring and evaluation as well as guidance for each of the six questions is available at this link.

While not specific to health information outreach programs, the Measuring Success Toolkit at  https://www.urbanreproductivehealth.org/toolkits/measuring-success from the Urban Reproductive Health Initiative is about health program planning, monitoring and evaluation. The toolkit provides helpful resources from the initiative’s multi-country perspective of working with the urban poor and the significant health disparities they face that may be helpful to consult with your health information outreach partners to underserved communities.  It includes subject-specific M&E resources such as maternal & child health and HIV/AIDs, and the resources within the toolkit are selected by M&E experts and reviewed quarterly following established criteria to identify important resources from diverse perspectives that include accurate, up to date information.

Say No to Spaghetti: Effective Data Graphs

Tuesday, December 23rd, 2014

Line graph with 5 multicolor lines jumbled together

The illustration above is from Stephanie Evergreen‘s excellent blog post that caution is needed with the use of line graphs in a chart to show change over time for multiple organizations so you don’t end up with a brightly colored bowl of spaghetti. The solution to passing on this pasta effect? Creating small multiple graphs, such as by each region in this example, which are done one at a time using the same scale as the original graph then stitched together with alignment tools including a ruler and Align > Align Top commands in your graphing software. Be sure to see the end result and step by step guidance on how to create these at http://stephanieevergreen.com/declutter-dataviz-with-small-multiples/

Showing change over time as a line graph instead of a bar graph is one of the quantitative data focus areas in our Outreach Evaluation Resource Center (OERC) webinar Data Burger: A ‘Good’ Questionnaire Response Rate Plus Basic Quantitative Analysis. You can listen to a recording of the Data Burger presentation for the Mid Atlantic Region at https://webmeeting.nih.gov/p2mn6k7tkv6/, and please contact us if you’d like to hear more about this or one of our other webinars.

More Qualitative Data Visualization Ideas

Thursday, December 18th, 2014

In September, we blogged about a way to create qualitative data visualizations by chunking a long narrative into paragraphs with descriptive illustrations.

Ann Emery has shown six additional ways to create qualitative data visualization: 1) Strategic world cloud use (one word or before/after comparisons), 2) Quantitative + Qualitative combined (a graph of percentages and a quote from an open-ended text comment) 3) Photos alongside participant responses (only appropriate for non-anonymized data) 4) Icon images beside text narratives 5) Diagrams explaining processes or concepts (the illustration of a health worker’s protective gear from Ebola in the Washington Post is a great example) and 6) Graphic timelines. See these examples and overviews on how to make your own at  http://annkemery.com/qual-dataviz/

Do you need more information about reporting and visualizing your data? We at the Outreach Evaluation Resource Center (OERC) have more resources available for you from the Reporting and Visualizing tab of our Tools and Resources for Evaluation Guide at http://guides.nnlm.gov/oerc/tools and welcome your suggestions for additional resources to include and your comments.

Literature Search Strategy Week at AEA

Friday, November 7th, 2014

We at the Outreach Evaluation Resource Center (OERC) have previously covered the American Evaluation Association’s (AEA) tip-a-day blog at http://aea365.org/blog as a helpful resource. This week posts about literature search strategies were shared on the AEA blog by Network member librarians from the Lamar Soutter Library at the University of Massachusetts Medical School. Have you been involved in a similar collaboration? Please let us know, we’d love to feature your work in a future OERC blog post!

Literature Search Strategy Week

  1. Best Databases – learn the most effective starting points for biomedical, interdisciplinary, specialized, and a handy Top Ten list of literature databases.
  2. Constructing a Literature Search – learn the value of a vocabulary roadmap, and the difference between keyword and controlled vocabulary searching.
  3. Grey Literature – strategies for understanding these non-traditional but highly valuable information resources and starting points on where to find them.
  4. Using MyNCBI – learn how to sign up for your free account, save your PubMed search strategies, receive email updates, customize your display and more.
  5. Citation Management – featuring both freely available and other options you may have access to through your academic organizations.

Freebie Friday: CDC Program Evaluation Resources

Friday, October 17th, 2014

CDC Program Evaluation Guide Cover

Are you new to evaluation or need assistance with planning and implementing program evaluation resources from a public or community health perspective for your projects?

The Centers for Disease Control and Prevention (CDC) has a freely available ‘how to’ resource for you with an Introduction to Program Evaluation for Public Health Programs: A Self Study Guide. Examples of public and community health programs that can be considered for program evaluation include direct service interventions, community-based mobilization efforts, research initiatives into issues such as health disparities, advocacy work, and training programs. The guide is available online or as a PDF download that consists of a six step process (from Engaging Stakeholders to Ensure Use of Evaluation Findings), a helpful Glossary of program evaluation terminology, and Resources for additional publications, toolkits and more to support public and community health program evaluation work.

A related CDC guide (A Framework for Program Evaluation) is one of several resources we at the Outreach Evaluation Resource Center (OERC) feature in the Evaluation Planning section of our Tools and Resources for Evaluation Page at http://guides.nnlm.gov/oerc/tools

 

Elegantly Simple Evaluation: Talking to Health Care Providers about Patient Health Literacy

Thursday, October 9th, 2014

logo for the VIVA project

By Yawar Ali and Cindy Olney

As the child of a physician living in South Texas, I’ve witnessed a deficiency of health literacy in patients. I volunteered in my dad’s clinic over spring break. I also participated on a medical relief trip with my father to a nonprofit charitable hospital in Pakistan. At both places, I witnessed difficulty in patient health literacy. – Yawar Ali

 In June 2014, Yawar Ali, a rising junior from the South Texas High School for Health Professions, taught physicians and physician assistants in his father’s medical clinics about patient health literacy. He also introduced them to MedlinePlus as an important tool for their patients. Yawar evaluated his project and discovered valuable insight that helped him improve the impact of his project.

Yawar conducted this health information outreach project as an internship offered through the  ¡VIVA! (Vital Information for a Virtual Age) project.  ¡VIVA! is a high school-based initiative in which students are trained to promote MedlinePlus to their classmates, teachers, families, and community members.  It is a student organization led by librarians of the South Texas Independent School District, located in the Lower Rio Grande Valley. The National Library of Medicine (NLM) funds the project.

He developed his presentation using health literacy materials available through the Medical Library Association and presented to three doctors and three PAs.  He taught them seven steps for addressing low patient health literacy and introduced them to MedlinePlus.

Yawar incorporated elegantly simple evaluation techniques into his project. Right after the presentation, he asked participants to complete a short evaluation form, asking them how likely they were to use the steps and promote MedlinePlus to patients.  They all responded positively, indicating good intentions.

Two weeks after the training, Yawar visited all of the health care providers to conduct brief semi-structured interviews. He asked if they had tried the steps and collected their feedback on the techniques. He also checked to see if they had promoted MedlinePlus to their patients. With some persistence, he was able to conduct a complete interview with each participant.

The feedback he received is of interest to anyone hoping to initiate health information outreach in partnership with primary care clinics, particularly in medically underserved areas:

  • The majority of Yawar’s participants tried teach-back, open-ended questions, and other techniques with their patients; but they were conflicted because such techniques added time to patient appointments. This interfered with their ability to stick to their busy schedules.
  • The health care providers were impressed with MedlinePlus, but they had convenient access to print materials from a database (Healthwise) that was integrated with the clinic’s Electronic Health Records (EHR) system. Furthermore, it was easier to document that they were adhering to the meaningful use requirements of the Medicare and Medicaid EHR Incentive Programs when they got patient information from Healthwise.
  • While the Healthwise database was more convenient for the providers, they recognized that the print information they were providing was limited. They believed their patients could get more comprehensive information from MedlinePlus, but the clinicians did not have a convenient way to promote the resource.

Their feedback prompted a speedy response. The project team secured MedlinePlus brochures from NLM that Yawar delivered to the clinics. The fix was relatively simple, but critical. The team may have never known about this necessary adjustment without Yawar’s elegantly simple evaluation.

Credit:  Yawar and Cindy would like to thank ¡VIVA! project team members Lucy Hansen, Sara Reibman, and Ann Vickman, for their help on this project.

The ¡VIVA! project has been funded in whole or in part with federal funds from the National Library of Medicine, National Institute of Health, under Contract No. HHSN-276-2011-0007-C with the Houston Academy of Medicine-Texas Medical Center Library.

 

 

 

Freebie Friday: Audit an Evaluation MOOC

Friday, October 3rd, 2014

Have you always been curious about Massive Open Online Courses (MOOCs) and haven’t yet checked them out, or have started a MOOC and not completed it because you found the format confusing or didn’t have the time to complete all the assignments in it?

Evaluating Social Programs is currently being offered by the Massachusetts Institute of Technology (MIT) within one of the easier-to-navigate open access MOOC formats, EdX. The course is now closed for registration for credit, but can still be accessed for free to audit with complete access to the instructional materials, activities, tests and discussions within the MOOC. It is estimated that full participation in each week of the course will take approximately 4 hours, but of course you are welcome to skip around and access the information that is of interest to you. I am currently auditing the MOOC while taking note of new resources and information that is likely to be of interest to you from a health program evaluation perspective this month, and will write a summary blog post in November.

The class description is:

This four and a half-week course on evaluating social programs will provide a thorough understanding of randomized evaluations and pragmatic step-by-step training for conducting one’s own evaluation. Through a combination of lectures and case studies from real randomized evaluations, the course will focus on the benefits and methods of randomization, choosing an appropriate sample size, and common threats and pitfalls to the validity of the experiment. While the course is centered around the why, how and when of Randomized Evaluations, it will also impart insights on the importance of a needs assessment, measuring outcomes effectively, quality control, and monitoring methods that are useful for all kinds of evaluations.

If you are interested in participating in Evaluating Social Programs and have not previously taken an EdX MOOC, I highly recommend checking out the 15 minute self-navigated DemoX which provides a very user-friendly overview of how to make the best use of the features within EdX.

 

Qualitative Data Visualization

Friday, September 26th, 2014

Flowchart of text, text with illustrations, then illustrations leading to additional text

Have you thought that only quantitative information can be used for data visualizations, and qualitative data wasn’t an option without first coding or otherwise turning this valuable content into quantitative formats?

I learned about an innovative and compelling approach to creating qualitative data visualizations with illustrations from Fresh Spectrum . They begin the process (as shown in the illustration above) of taking a long narrative such as a focus group transcription, and chunking it into a few paragraphs per concept with a unique illustration for each one. In this case custom illustrations of people were used, but you could use your organization’s existing images or Creative Commons-licensed images for illustrating concepts. The next step for the visualization uses the images with brief captions as an online data dashboard, where visitors can click on the captioned image of interest to them to then access the more detailed narrative. The author describes how to do this within a WordPress portfolio blog template, or a simpler strategy of creating HTML anchor links to each individual section within a longer text. You can see how this works by clicking on an anchor link from the original post (http://freshspectrum.com/blogging-advice/#davidson for example) that leads to the longer narrative at http://freshspectrum.com/blogging-advice/ (a great source of advice for blogging by the way!)

Need more information about reporting and visualizing your data? We at the Outreach Evaluation Resource Center (OERC) have more resources available for you from the Reporting and Visualizing tab of our Tools and Resources for Evaluation Guide at http://guides.nnlm.gov/oerc/tools and welcome your suggestions and comments about the guide.

Last updated on Saturday, 23 November, 2013

Funded by the National Library of Medicine under contract # HHS-N-276-2011-00008-C.