Archive for the ‘Practical Evaluation’ Category
Are you new to evaluation or need assistance with planning and implementing program evaluation resources from a public or community health perspective for your projects?
The Centers for Disease Control and Prevention (CDC) has a freely available ‘how to’ resource for you with an Introduction to Program Evaluation for Public Health Programs: A Self Study Guide. Examples of public and community health programs that can be considered for program evaluation include direct service interventions, community-based mobilization efforts, research initiatives into issues such as health disparities, advocacy work, and training programs. The guide is available online or as a PDF download that consists of a six step process (from Engaging Stakeholders to Ensure Use of Evaluation Findings), a helpful Glossary of program evaluation terminology, and Resources for additional publications, toolkits and more to support public and community health program evaluation work.
A related CDC guide (A Framework for Program Evaluation) is one of several resources we at the Outreach Evaluation Resource Center (OERC) feature in the Evaluation Planning section of our Tools and Resources for Evaluation Page at http://guides.nnlm.gov/oerc/tools
By Yawar Ali and Cindy Olney
As the child of a physician living in South Texas, I’ve witnessed a deficiency of health literacy in patients. I volunteered in my dad’s clinic over spring break. I also participated on a medical relief trip with my father to a nonprofit charitable hospital in Pakistan. At both places, I witnessed difficulty in patient health literacy. – Yawar Ali
In June 2014, Yawar Ali, a rising junior from the South Texas High School for Health Professions, taught physicians and physician assistants in his father’s medical clinics about patient health literacy. He also introduced them to MedlinePlus as an important tool for their patients. Yawar evaluated his project and discovered valuable insight that helped him improve the impact of his project.
Yawar conducted this health information outreach project as an internship offered through the ¡VIVA! (Vital Information for a Virtual Age) project. ¡VIVA! is a high school-based initiative in which students are trained to promote MedlinePlus to their classmates, teachers, families, and community members. It is a student organization led by librarians of the South Texas Independent School District, located in the Lower Rio Grande Valley. The National Library of Medicine (NLM) funds the project.
He developed his presentation using health literacy materials available through the Medical Library Association and presented to three doctors and three PAs. He taught them seven steps for addressing low patient health literacy and introduced them to MedlinePlus.
Yawar incorporated elegantly simple evaluation techniques into his project. Right after the presentation, he asked participants to complete a short evaluation form, asking them how likely they were to use the steps and promote MedlinePlus to patients. They all responded positively, indicating good intentions.
Two weeks after the training, Yawar visited all of the health care providers to conduct brief semi-structured interviews. He asked if they had tried the steps and collected their feedback on the techniques. He also checked to see if they had promoted MedlinePlus to their patients. With some persistence, he was able to conduct a complete interview with each participant.
The feedback he received is of interest to anyone hoping to initiate health information outreach in partnership with primary care clinics, particularly in medically underserved areas:
- The majority of Yawar’s participants tried teach-back, open-ended questions, and other techniques with their patients; but they were conflicted because such techniques added time to patient appointments. This interfered with their ability to stick to their busy schedules.
- The health care providers were impressed with MedlinePlus, but they had convenient access to print materials from a database (Healthwise) that was integrated with the clinic’s Electronic Health Records (EHR) system. Furthermore, it was easier to document that they were adhering to the meaningful use requirements of the Medicare and Medicaid EHR Incentive Programs when they got patient information from Healthwise.
- While the Healthwise database was more convenient for the providers, they recognized that the print information they were providing was limited. They believed their patients could get more comprehensive information from MedlinePlus, but the clinicians did not have a convenient way to promote the resource.
Their feedback prompted a speedy response. The project team secured MedlinePlus brochures from NLM that Yawar delivered to the clinics. The fix was relatively simple, but critical. The team may have never known about this necessary adjustment without Yawar’s elegantly simple evaluation.
Credit: Yawar and Cindy would like to thank ¡VIVA! project team members Lucy Hansen, Sara Reibman, and Ann Vickman, for their help on this project.
The ¡VIVA! project has been funded in whole or in part with federal funds from the National Library of Medicine, National Institute of Health, under Contract No. HHSN-276-2011-0007-C with the Houston Academy of Medicine-Texas Medical Center Library.
Have you always been curious about Massive Open Online Courses (MOOCs) and haven’t yet checked them out, or have started a MOOC and not completed it because you found the format confusing or didn’t have the time to complete all the assignments in it?
Evaluating Social Programs is currently being offered by the Massachusetts Institute of Technology (MIT) within one of the easier-to-navigate open access MOOC formats, EdX. The course is now closed for registration for credit, but can still be accessed for free to audit with complete access to the instructional materials, activities, tests and discussions within the MOOC. It is estimated that full participation in each week of the course will take approximately 4 hours, but of course you are welcome to skip around and access the information that is of interest to you. I am currently auditing the MOOC while taking note of new resources and information that is likely to be of interest to you from a health program evaluation perspective this month, and will write a summary blog post in November.
The class description is:
This four and a half-week course on evaluating social programs will provide a thorough understanding of randomized evaluations and pragmatic step-by-step training for conducting one’s own evaluation. Through a combination of lectures and case studies from real randomized evaluations, the course will focus on the benefits and methods of randomization, choosing an appropriate sample size, and common threats and pitfalls to the validity of the experiment. While the course is centered around the why, how and when of Randomized Evaluations, it will also impart insights on the importance of a needs assessment, measuring outcomes effectively, quality control, and monitoring methods that are useful for all kinds of evaluations.
If you are interested in participating in Evaluating Social Programs and have not previously taken an EdX MOOC, I highly recommend checking out the 15 minute self-navigated DemoX which provides a very user-friendly overview of how to make the best use of the features within EdX.
Have you thought that only quantitative information can be used for data visualizations, and qualitative data wasn’t an option without first coding or otherwise turning this valuable content into quantitative formats?
I learned about an innovative and compelling approach to creating qualitative data visualizations with illustrations from Fresh Spectrum . They begin the process (as shown in the illustration above) of taking a long narrative such as a focus group transcription, and chunking it into a few paragraphs per concept with a unique illustration for each one. In this case custom illustrations of people were used, but you could use your organization’s existing images or Creative Commons-licensed images for illustrating concepts. The next step for the visualization uses the images with brief captions as an online data dashboard, where visitors can click on the captioned image of interest to them to then access the more detailed narrative. The author describes how to do this within a WordPress portfolio blog template, or a simpler strategy of creating HTML anchor links to each individual section within a longer text. You can see how this works by clicking on an anchor link from the original post (http://freshspectrum.com/blogging-advice/#davidson for example) that leads to the longer narrative at http://freshspectrum.com/blogging-advice/ (a great source of advice for blogging by the way!)
Need more information about reporting and visualizing your data? We at the Outreach Evaluation Resource Center (OERC) have more resources available for you from the Reporting and Visualizing tab of our Tools and Resources for Evaluation Guide at http://guides.nnlm.gov/oerc/tools and welcome your suggestions and comments about the guide.
The American Evaluation Association (eval.org) sponsors a Potent Presentations Initiative (p2i) that has a stated purpose of helping evaluators improve their presentation skills, both within a conference setting and as part of individual practice. P2i challenges evaluators to hone in on three concepts: Their message, their design, and their delivery.
There are a wealth of handouts available as PDF files,Word documents and Powerpoint presentations available from the p2i tools website (http://p2i.eval.org/index.php/p2i-tools/) that sometimes include AEA conference specifications in addition to many great messaging, designing and delivery principles. For an example of each principle be sure to check out the Presentation Preparation Checklist (from 3 months ahead of time to afterwards to include modifications while the information is freshly in mind), How to Design a Research Poster (great infographic visualization and instructions on how to make your data ‘pop’), and the Delivery Glue Handout (did you know as a general rule it takes 16 times the length of your talk to make presentation slides and a script?).
The Engaged Librarian: Crafting an Effective Assessment Plan to Determine the Impact of a Key Strategic Library Initiative by Sarah Murphy at The Ohio State University (OSU) was presented during the Library Assessment Conference and provided an overview to the use of a logic model as part of library strategic planning. Ms. Murphy’s presentation slides are available by clicking here.
Their project incorporated the theory of change methodology with logic models and used the Kellogg Foundation Logic Model as a template. They storyboarded data within a data dashboard that was both aligned with and broken down by the applicable OSU strategic vision goals. Ms. Murphy reported that the benefits of using a logic model approach included having a flexible but structured way to do library assessment planning, having a collaborative and inclusive approach, creating a project focus, being able to assess linear and iterative programs and services, and the ability to communicate program accomplishments in interesting ways. During the question and answer session they noted they are also Tableau fans (we will write about Tableau for our next post) and like to create data structures in their dashboard to avoid information silos.
If you’d like to learn more about logic models and data dashboards, be sure to check out our freely available Outreach Evaluation Resource Center (OERC) Evaluation Guides, especially Planning Outcomes-Based Outreach Projects. We also offer Data Dashboards: Monitoring Progress toward Program Outcomes as one of our webinars and a recording of Data Dashboards is available by clicking here.
Do you want to learn more about outcomes-based planning and evaluation (OBPE) for your outreach project but there’s no money in the training budget to do so?
Shaping Outcomes: Making a Difference in Libraries and Museums (shapingoutcomes.org) is available as a free online course that learners can start anytime and work on at their own self-navigated pace. While there are library and museum-specific examples provided in the course the concepts of learning more about target audience needs, how to clarify desired results, developing logic models and evaluating outcomes are applicable for most other organizations’ outreach projects as well.
Modules of the class are broken into five sections (Overview, Plan, Build, Evaluate, Report) with a helpful Glossary to learn OBPE terminology and a Logic Model template. Shaping Outcomes was developed by the Institute of Museum and Library Services (IMLS) and Indiana University/Purdue University Indianapolis (IUPUI) and previously was available as an instructor-led class.
More information specific to developing logic models in health information outreach programs is available from Booklet Two: Planning Outcomes-Based Outreach Projects, part of our resources on our Outreach Evaluation Resource Center (OERC) Evaluation Guides page at http://nnlm.gov/evaluation/guides.html.
Veggie burger by Dan McKay, Flickr Creative Commons license
The Outreach Evaluation Resource Center (OERC) was pleased to present our Data Burger: A “Good” Questionnaire Response Rate plus Basic Quantitative Data Analysis webinar for the National Network of Libraries of Medicine, New England Region (NN/LM NER) Health Care Workforce Community of Interest (COI) this week. Thanks to NER for freely sharing the 1 hour webcast recording at https://webmeeting.nih.gov/p87ze5jc0do/.
Are you interested in this and other webinar training sessions the OERC has to offer? You can learn more about Data Burger and others from our presentation listing at http://nnlm.gov/evaluation/workshops/ and let us or your Regional office know that you’d like us to present a webinar. We welcome the opportunity to work with you!
As we at the Outreach Evaluation Resource Center (OERC) learn more about great evaluation resources available at an even better price (free!) in addition to our own freely available resources (http://guides.nnlm.gov/oerc/tools), we will feature some of them here for you to explore on ‘Freebie Friday’ as an occasional series.
To begin our coverage, did you know the American Evaluation Association (AEA) has an online public library of AEA conference presentations, assessment instruments (such as rubrics and logic models), and more available at http://comm.eval.org/communities/resources/libraryview/? The default library setting displays those files most recently updated in the online library and offers keyword searching, but I recommend going straight to the more advanced search functionality at http://comm.eval.org/communities/resources/searchlibrary/.
Thanks to evolving Internet and mobile technology, most of us now prefer our information presented in tapas-sized quantities, preferably in a visually appealing way. This preference is having an impact on our business communication. Before our eyes, bullet points on PowerPoint slides are gradually giving away to engaging visuals. Infographics are replacing text-dense brochures and executive summaries.
Nancy Duarte, who is motivating the world to clean up its PowerPoint presentations, is venturing into a new frontier: Written reports. She suggests creating reports using presentation software like PowerPoint, which is more amenable to manipulating layouts. The idea is to present one idea per slide, incorporating visuals, text and much-appreciated white space.
Duarte calls this type of report a SlideDoc and offers a free tutorial on how to create one (presented in, you guessed it, a PowerPoint document). The site includes downloadable templates to help you create your own SlideDoc report. You also can find Duarte’s Diagrammer here, which I wrote about in a previous post. I believe, at this point, that I should state explicitly that I do not own stock in her company. I just love her approach to presenting ideas and I think evaluation reports would be read and heard by more people if we all started using her guidelines.
As someone who wants to stop producing coma-inducing reports, I have tried my hand at creating more engaging layouts using Microsoft Word. What I learned is this: Take your anticipated timeline for report-writing and double it. Word does not lend itself to interesting layouts, so inserting pictures, graphs, and call-out quotes is like putting shoes on a toddler. You can do it, but it’s probably going to make you late.
Consequently, my results have been underwhelming.
So I‘m excited to try SlideDocs. I already can see how much more flexible PowerPoint will be for arranging text, images, and graphs. I won’t be wasting time adjusting margins and re-positioning graphs that always seem to wander around in Word documents. I also like that I don’t have to learn a new software application, nor do I have to get my audience to download a special reader to view my reports.
Just remember: Shorter isn’t easier when it comes to reports. These “to-the-point” SlideDocs require that you know both your findings and your audience extremely well so you can communicate the most important information in the most succinct way. SlideDocs will help with the layout, but you still have to do the thinking.