Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘News’ Category

Brush Up Your Evaluation Skills: Two Great Conferences

Friday, September 23rd, 2016

It’s the fall, or the beginning of conference season. It’s a very exciting time if you like evaluation/assessment.  If you want to improve your evaluation skills, two great conferences are coming up, back to back.  Take a look at some of these highlights and pick one to go to! This is a great opportunity.

Oct. 24-29, 2016 Evaluation 2016, Atlanta GA

Eavlaution 2016 October 24-29, Atlanta, GA

This is the annual conference of the American Evaluation Association, an international organization with over 7000 members, and interest groups that cover topics like Assessment in Higher Education; Collaborative, Participatory & Empowerment Evaluation; and Data Visualization and Reporting.  The theme of this year’s conference is Evaluation + Design.

The conference has 40 workshops and 850 sessions.  Here are some example programs:

  • From crap to oh snap: Using DIY templates to (easily) improve information design across an organization
  • Developing Evaluation Tools to Measure MOOC Learner Outcomes in Higher Education
  • Evaluation Design for Innovation/Pilot Projects

There’s still time for Early Bird Registration (ends October 3)!

Oct. 31-November 2, 2016 Library Assessment Conference, Arlington VA

Library Assessment Conference 2016

This conference only happens every other year and is co-sponsored by the Association of Research Libraries (ARL) and the University of Washington (UW) Libraries (disclosure – the NEO is part of the UW Libraries–something we’re quite proud of).   The theme for this conference is Building Effective, Sustainable, Practical Assessment.

This conference is bookended by workshops like Getting the Message Out: Creating a Multi-Directional Approach to Communicating Assessment and Learning Analytics, Academic Libraries, and Institutional Context: Getting Started, Gaining Traction, Going Forward.

Scholarly papers and posters with titles like “How Well Do We Collaborate? Using Social Network Analysis (SNA) to Evaluate Engagement in Assessment Program” and “Consulting Detectives: How One Library Deduced the Effectiveness of Its Consultation Area & Services” are organized on a variety of topics, such as Organizational Issues; Ithaka S+R; and Analytics/Value.

 

This is an exciting time to be in the assessment and evaluation business.  Take this amazing opportunity to go to one of these conferences.

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

From Logic Model to Proposal Evaluation – Part 2: The Evaluation Plan

Friday, September 2nd, 2016

Photo of black and white cat with fangsLast week we wrote some basic goals and objectives for a proposal about teaching health literacy skills to vampires in Sunnydale.  Here’s what the goals and objectives look like, taken from the Executive Summary statement in last week’s post:

Goal: The goal of our From Dusk to Dawn project is to improve the health and well-being of vampires in the Sunnydale community.

Objective 1: We will teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires’ ability to find consumer health information and up to date research about health conditions.

Objective 2: We will open a 12-hour “Dusk-to-Dawn” health reference hotline to help the vampires with their reference questions.

There are also three outcomes that we have identified:

  1. Short-term: Increased ability of the Internet-using Sunnydale vampires to research needed health information.
  2. Intermediate: These vampires will use their increased skills to research health information for their brood.
  3. Long-term: Overall, the Sunnydale vampires will have improved health and as a result form better relationships with the human community of Sunnydale.

To get to an evaluation plan from here you have to know that there are basically two kinds of things you’ll want to measure: process and outcomes.

Process assessment measures that you did what you said you would do and the way you said you would do it. For example, you can count the number of classes you taught, how many people attended, and whether their survey responses showed that they thought you did a good job teaching.

Also you might want to show that you were willing to make changes in the plan if review of your process assessment showed that you weren’t getting the results you wanted.  For example, if you planned all your classes in early evening, but few vampires attended, you might interview some vampires and find out that early evening is mealtime for most vampires, and move your classes to a different time to increase attendance.  Your evaluation plan could show that you are collecting that information and that you will be responsive to what you see happening.

Outcome assessment measures the extent to which your project had the impact that you hoped it would on the recipients of the project, or even greater on their overall organizations or communities. We showed the first step of outcome assessment in last week’s assignment, but I’m going to break it down a little more here.  Put in basic terms, to do an outcome assessment, you state your outcome, you add in an indicator, a target, and a time frame to come up with a measurable objective, and then you write out the source of your data, your data collection method, and your data collection timing to complete the picture.  Let’s talk about each item here:

Indicator: This is the evidence you can gather that shows whether or not you met your outcomes.  If one of your outcomes is that the vampires have increased ability to research health information, how would you know if that had happened? The indicator could be their increased confidence level in finding health information, or it could be improvement in skills test scores given before and after a training session.

Target: The target is the goal that makes this project look like a success to you.  For example, if the vampires improve their test scores by 50% over a baseline test, is that enough to say you have successfully reached that outcome?  And how many of the vampires need to reach that 50% goal?  All of them? One of them?  Targets can be hard to identify, because you don’t want them to be too hard to reach but if they’re too easy your funder may not be impressed with your ambition.  Sometimes you can work with the funder or other stakeholders on setting targets that are credible.

Time frame: This is the point in time that when the threshold for success will be achieved.  So if you want to make sure the vampires increased their ability by the end of your training, then the time frame would be by the end of your training.

Data Source: This is the location where your information is found. Often, data sources are people (such as participants or observers) but they also may be records, pictures, or meeting notes. Here are some examples of data sources.

Data Collection Methods: Evaluation methods are the tools you use to collect data, such as a survey, observation, or quiz.  Here is more examples of data collection methods.

Data Collection Timing: The data collection timing is describing exactly when you will be collecting the data.

What does your final evaluation plan look like? 

Here is a sample piece of an evaluation plan for the Dusk to Dawn proposal.

Objective 1: teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires’ ability to research consumer health information and up to date research about health conditions.

Process Assessment: The PI will collect the following information to ensure that classes are being taught; expected attendance figures are being reached; teachers are doing a good job teaching classes (including surviving the classes).  Data will be reviewed after each class and changes will be made to the program as needed to reach target goals:

◊ Participant roster to measure attendance figures
◊ Class evaluations to measure teacher performance
◊ Count of number of teachers at the beginning and ending of each class to measure survival of instructors
◊ Project team will meet after the second class to review success and lessons learned and to consider course corrections to ensure objectives are met

Outcome Assessment:
Measureable Objective: In a post-test given immediately after each class, a minimum of 75% of Sunnydale vampire attendees demonstrate that they learned how to find needed resources in PubMed and MedlinePlus by showing at least a 50% improvement over the pre-test.

Based on Level 2 (Learning) in the Kirkpatrick Model, a test will be created with some basic health questions to be researched. Class participants will be given these questions as a pre-test before the class, and then will be given the same questions after the class as a post-test.  This learning outcome will be considered successful if a minimum of 75% of Sunnydale vampire participants demonstrate that their scores improved by at least 50%.

Last wishes, I mean, thoughts

This is not a complete evaluation plan, but the purpose of these two posts has been to show how you can go from a logic model to the evaluation plan of a proposal.  Don’t worry if all your outcomes cannot be measured in the scope of your project.  For example, in this Dusk to Dawn project, it might have been dangerous to find out if the vampires had passed on needed health information to their brood, even harder to find out whether the vampires had become more healthy as a result of the information.  This doesn’t mean to leave these outcomes out, but you may want to acknowledge that measuring some outcomes is out of the scope of the project’s resources.

As Grandpa Munster once said “Don’t let time or space detain ya, here you go, to Transylvania.”

Photo credit: Photo of 365::79 – Vampire Cat by Sarah Reid on Flickr under Creative Commons license CC BY 2.0.  No changes were made.

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

From Logic Model to Proposal Evaluation – Part 1: Goals and Objectives

Friday, August 26th, 2016

Vocabulary. Jargon. Semantics.  Sometimes I think it’s the death of us all.  Seriously, it’s really hard to have a conversation about anything when you use the same words in the same context to mean completely different things.

Take Goals and Objectives.  I can’t tell you how many different ways this has been taught to me.  But in general all the explanations agree that a goal is a big concept, and an objective is more specific.

Things get complicated when we use words like Activities, Outcomes, and Measurable Objectives when teaching you about logic models as a way of planning a project.  Which of those words correlate with Goals and Objectives when writing a proposal for the project you just planned?

Bela Lugosi as Dracula

I’m going to walk through an example of how we can connect the dots between the logic model that we use to plan projects, and the terminology used in proposal writing.  There isn’t necessarily going to be a one to one relationship, and it might depend on the number of goals you have.

As has been stated in previous posts, we’ve never actually done any work with the fictional community of Sunnydale, a place where there was, in the past, a large number of vampires and other assorted demons.  But in order to work through this problem, let’s go back to this hypothetical post where we used the Kirkpatrick Model to determine outcomes that we would like to see with any remaining vampires who want to live healthy long lives, and get along with their human neighbors.  For this post, I’m going to pretend I’m writing a proposal to do a training project for them based on those outcomes, and then show how they lead to an evaluation plan.

Goals

The goal can be your long-term outcome or it can be somewhat separate from the outcomes. But either way, your goal needs to be logically connected to the work you’re planning to do.  For example, if you’re going to train vampires to use MedlinePlus, goals like “making the world a better place,” or “achieving world peace,” are not as connected to your project as something like “improving health and well being of vampires” or “improving the health-literacy of vampires so they can make good decisions about their health.”

Here is a logic model showing how this could be laid out, using the outcomes established in the earlier post:

Dusk to Dawn Logic Model

Keep in mind that the purpose of a proposal is to persuade someone to fund your project.  So for the sake of my proposal, I’m going to combine the long-term outcomes into one goal statement.

The goal of this project is to improve the health and well being of vampires in the Sunnydale community.

Objectives

The objectives can be taken from the logic model Activities column. But keep something in mind.  Logic models are small – one page at most.  So you can’t use a lot of words to describe activities.  Objectives on the other hand are activities with some detail filled in. So in the logic model the activity might be “Evening hands-on training on MedlinePlus and PubMed,” while the objective I put in my proposal might be “Objective 1: We will teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires’ ability to find consumer health information and up to date research.”

Objectives in Context

Here’s a sample of my Executive Summary of the project, showing goals, objectives, and outcomes in a narrative format:

Executive Summary: The goal of our From Dusk to Dawn project is to improve the health and well being of vampires in the Sunnydale community. In order to reach this goal, we will 1) teach 4 hands-on evening classes on the use of MedlinePlus and PubMed to improve Sunnydale vampires’ ability to find consumer health information and up to date research about health conditions; and 2) open a 12-hour “Dusk-to-Dawn” health reference hotline to help the vampires with their reference questions.  With these activities, we hope to see a) increased ability of the Internet-using Sunnydale vampires to research needed health information; b) that those vampires will use their increased skills to research health information for their brood; and c) these vampires will use this information to make good health decisions leading to improved health, and as a result form better relationships with the human community of Sunnydale.

Please note that in this executive summary, I do not use the word “objectives” to identify the phrases numbered 1 and 2, and I also do not use the word “outcomes” to identify the phrases lettered a, b, and c (because I like the way it reads better without them). However, in detailed narrative of my proposal I would use those terms to go with those exact phrases.

So then, what are Measurable Objectives?

The key to the evaluation plan is creating another kind of objective: what we call a measurable outcome objective. When you create your evaluation plan, along with showing how you plan to measure that you did what you said you would do (process assessment), you will also want to plan how to collect data showing the degree to which you have reached your outcomes (outcome assessment).  These statements are what we call measurable outcome objectives.

Using the “Book 2 Worksheet: Outcome Objectives” found on our Evaluation Resources web page, you start with your outcomes, add an indicator, target and time frame to get measurable objectives  and write it in a single sentence.  Here’s an example of what that would look like using the first outcome listed in the Executive Summary:

Dusk to Dawn Measurable Objective

We’ve gotten through some terminology and some steps for going from your logic model to measuring your outcomes.

Stay tuned for next week when we turn all of this into an Evaluation Plan!

Dare I say it? Same bat time, same bat channel…

 

 

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Evaluation Planning for Proposals: a New NEO Resource

Friday, August 12th, 2016

Angry crazy Business woman with a laptop

Have you ever found yourself in this situation?  You’re well along in your proposal writing when you get to the section that says “how will you evaluate your project?”  Do you think:

  1. “Oh #%$*! It’s that section again.”
  2. “Why do they make us do this?”
  3. “Yay! Here’s where I get to describe how I will collect evidence that my project is really working!”

We at the NEO suggest thinking about evaluation from the get-go, so you’ll be prepared when you get to that section.  And we have some great booklets that show how to do that.  But sometimes people aren’t happy when we say “here are some booklets to read to get started,” even though they are awesome booklets.

So the NEO has made a new web page to make it easier to incorporate evaluation into the project planning process and end up with an evaluation plan that develops naturally.

1. Do a Community Assessment; 2. Make a Logic Model; 3. Develop Measurable Objectives; 4. Create an Evaluation Plan

We group the process into 4 steps: 1) Do a Community Assessment; 2) Make a Logic Model; 3) Develop Measurable Objectives for Your Outcomes; and 4) Create an Evaluation Plan.   Rather than explain what everything is and how to use it (for that you can read the booklets), this page links to the worksheets and samples (and some how-to sections) from the booklets so that you can jump right into planning.  And you can skip the things you don’t need or that you’ve already done.

In addition, we have included links to posts in this blog that show examples of the areas covered so people can put them in context.

We hope this helps with your entire project planning and proposal writing experience, as well as provides support for that pesky evaluation section of the proposal.

Please let Cindy (olneyc@uw.edu) or me (kjvargas@uw.edu) know how it works for you, and feel free to make suggestions.  Cheers!

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

The Kirkpatrick Model (Part 2) — With Humans

Friday, August 5th, 2016

Disclaimer: Karen’s blog post last week on the Kirkpatrick Model used an example that was hypothetical.  We want to be clear that the NEO has never evaluated any programs directed toward improving health outcomes for vampires.

However, we can claim success in applying the Kirkpatrick Model for National Network of Libraries of Medicine (NN/LM) training programs.

The NN/LM’s mission is to promote the biomedical and consumer health resources of the National Library of Medicine.  One strategy that is popular with NN/LM’s Regional Medical Libraries, which lead and manage the network, is the “train-the-trainer” program. These programs teach librarians and others about NLM resources so that they, in turn, will teach their peers, patients, or clients. When the NEO provides evaluation consulting for train-the-trainer programs, we rely heavily on the Kirkpatrick Model.

Kirkpatrick Outcomes Levels and Logic Models

For example, the NN/LM’s initiative to reach out to community college librarians incorporated “train-the-trainer” as one of several strategies to promote use of NLM resources in community college health professions programs. While the initiative was multi-pronged, train-the-trainer programs for community college librarians was a major strategy of the project. The Kirkpatrick Model helped our task force define outcomes and develop measures for this activity.

Our logic model led us to the following program theory:

If we train community college librarians to use National Library of Medicine Resources

  • They will respond favorably to our message (Reaction)
  • And discover new, useful health information resources that (Learning)
  • They will use when working with faculty and students (Behavior)
  • Which will lead to increased use of NLM resources among community college faculty and staff (Results)

Slide1

Measuring Outcomes

We developed two simple measurement tools to assess the four outcome levels.  To measure Reaction, RML instructors administered a standard one-page session evaluation form that has been used for years by instructors who provide NN/LM training sessions. The form collects participants’ feedback, including the grade (A through F) they would assign to the class. This form was our measure of participant reaction.

The other three levels were assessed using a follow-up questionnaire sent to the training participants several months after their training. On this questionnaire, we asked them a series of yes/no questions:

Learning: At this training session, did you learn about health information resources that were NEW to you?

Behavior: Regarding the NEW resources you learned at the training, have you done any of the following?

  • Shown these resources to students?
  • Used the resources when preparing lesson plans?
  • Shown the resources to community college faculty or staff?
  • Used the resources to answer reference questions?

Results: Do you know if the resources are being used by

  • Students?
  • Faculty, administration, or staff at your organization?
  • The librarians at your institution?

We knew our Results questions were weak. They obviously were very subjective. Most of the respondents said they didn’t know about use beyond their library staff members. Unfortunately, we did not have resources for a more objective measure of our anticipated results (e.g., surveying faculty and students at participating schools). Our dilemma was not unusual. Many practitioners of the Kirkpatrick Model agree that assessing Results-level outcomes can be costly and challenging.

However, in anticipation that this Results-level measure might not work, we had a back-up plan inspired by Robert Brinkerhoff’s Success Case Method (which we posted about here). In this approach, evaluators ask training participants to describe how their training benefited the organization.  We ended the questionnaire with the following open-ended question: Please describe how the training you received on National Library of Medicine resources has made a difference for you or your organization.

This question worked well, with 57% of respondents providing examples of how the training improved their customer services. They reported using the NLM resources to provide reference services and incorporating NLM resources into their information literacy classes for health professions students.  Some also were talking to faculty about the importance of teaching health professions students about RML resources that students could use after graduation.

In the end, the Kirkpatrick Model helped us get metrics and qualitative information that helped to assess the effectiveness of our train-the-trainer activities.  Most of the training participants who responded to our follow-up questionnaire learned new resources and were promoting them to student and faculty. Their stories showed that the NN/LM training improved the services they were delivering to their users.

The NEO has drawn on the Kirkpatrick Model to design evaluation methods for similar projects, including our own evaluation training programs.  It is a great tool for helping program planners to define concrete objectives and create measures that are closely linked to desired outcomes.

 

Happy Fourth of July in Numbers!

Friday, July 1st, 2016

4th of July graphic image

Before holidays we sometimes do a post on the value of putting your data in a visually understandable format, perhaps some kind of infographic.

As I write this, some of you may be sitting at your desks pondering how you will celebrate U.S. Independence Day. To help turn your ponderings into a work-related activity, here are some examples of Fourth of July Infographics.  Since some of them have numbers but no dates (for example the number of fireworks purchased in the US “this year”) you might use them as templates for the next holiday-based infographic you create yourself.

If you like History, the History Channel has a fun infographic called 4th of July by the Numbers.  It includes useful information such as:

  • the oldest signer of the Declaration of Independence was Benjamin Franklin at 70,
  • the record for the hot dog eating contest on Coney Island is 68 hotdogs in 10 minutes, and
  • 80% of Americans attend a barbecue, picnic or cookout on the 4th of July

Thinking about the food for your picnic (if you’re one of the 80% having one)?

From the perspective of work (remember work?) here is an infographic from Unmetric on how and why you should create your own infographics for the 4th of July: How to Create Engaging 4th of July Content.

Have a great 4th of July weekend!

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Design Principles in Evaluation Design

Friday, June 17th, 2016

Robot and human hands almost touching

“Sometimes… it seems to me that… all the works of the human brain and hand are either design itself or a branch of that art.” Michelangelo

Michelangelo is not the only one who thinks design is important in all human activities.  In his book A Whole New Mind, Dan Pink considers design to be one of the 6 senses that we need to develop to thrive in this world. As Mauro Porcini, PepsiCo’s Chief Design Officer points out “There is brand design. There is industrial design. There is interior design. There is UX and experience design. And there is innovation in strategy.” ¹

There is also evaluation design. Whether we’re talking about designing evaluation for an entire project or just one section, like the needs assessment or presenting evaluation results, evaluators are still actively involved in design.

Most of us don’t think of ourselves as designers, however.  Juice Analytics has a clever tool called “Design Tips for Non-Designers” to teach basic design skills and concepts.  Some of these are very specific design tips for charts and power points (which by the way are very important and useful, like “avoiding chart junk” and “whitespace matters”).  But some of the other tips can be jumping off points for thinking about bigger picture design skills, such as:

  • Using Hick’s Law and Occam’s Razor to explain the importance of simplicity
  • Learning how to keep your audience in mind by thinking of how to persuade them, balancing Aristotle’s suggested methods of ethical appeal (ethos), emotional appeal (pathos), and logical appeal (logos)
  • Learning how Gestalt theory applies to the mind’s ability to acquire and maintain meaningful perceptions in an apparently chaotic world
  • Considering the psychology of what motivates users to take action

The September 2015 issue of Harvard Business Review highlighted design thinking as corporate strategy in their spotlighted articles (which are freely available online, as long as you don’t open more than 4 a month).  Here are some cool things you can read about in these articles:

  • Using design thinking changed the way PepsiCo designed products to fit their users’ needs (my favorite line is how they used to design products for women by taking currently existing products and then applying the strategy of “shrink it or pink it.”)
  • Design is about deeply understanding people.
  • Principles of design can be applied to the way people work: empathy with users, a discipline of prototyping and tolerance for failure.
  • Create models to explain complex problems, and then use prototypes to explore potential solutions.
  • If it is likely that a new program or strategy may not be readily accepted, use design principles to plan the program implementation.

Some people are seen as being born with design skills.  But it’s clear that a lot can be learned with study and practice.  Even Michelangelo said, “If people knew how hard I worked to get my mastery, it wouldn’t seem so wonderful after all.”


¹ James De Vries. “PepsiCo’s Chief Design Officer on Creating an Organization Where Design Can Thrive.Harvard Business Review. 11 Aug 2015.  Web. 17 June 2016.

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Summer Evaluation Institute Registration is Open!

Friday, May 13th, 2016

AEA Summer Institute Logo

Whether you call what you do evaluation or assessment, the American Evaluation Association’s Summer Evaluation Institute is an amazing event, sure to teach you something and give you a different perspective on the job you do.

The institute, held June 26-29, 2016, is made up of ½ day hands-on training sessions, taught by the best professionals in the field of evaluation.  It’s attended by people from all over the world who want to improve their skills in different aspects of the evaluation process.

Why would you as a librarian want to attend the AEA Summer Evaluation Institute?   Here are some ideas:

Let’s say you were in charge of eliminating much of your print journal collection and increasing your online journals, and you want to figure out what data you should collect that will show that your users are still getting what they need. There’s a great program called “Development and Use of Indicators for Program Evaluation” by Goldie MacDonald that covers criteria for selection of indicators. Goldie MacDonald is a Health Scientist in the Center for Global Health at the U.S. Centers for Disease Control and Prevention (CDC) and a dynamic speaker and trainer.

Are you taking on the planning of an important project, like finding ways to ensure that your hospital administration values the contributions of your library?  Logic models are great planning tools, but additionally are useful for integrating evaluation plans and strategic plans.  How would you like to take a four hour class on logic models taught by the Chief Evaluation Officer at the CDC, Tom Chapel?

What if you’re a liaison librarian to your university’s biology department and you’re looking for ways to improve collaboration with the faculty? There’s a program called Evaluating and Improving Organizational Collaboration that gives participants the opportunity to increase their capacity to quantitatively and qualitatively examine the development of inter- and intra-organizational partnerships. It’s taught by Rebecca Woodland, Chair of the Department of Educational Policy and Administration at University of Massachusetts Amherst (and recognized as one of the AEA’s most effective presenters).

Maybe you’ve been responsible for a program at your public library training the community in using MedlinePlus for their health information needs. You’ve collecting a lot of data showing the success of your programs, and want to make sure your stakeholders take notice of it. How about giving them an opportunity to work with the data themselves? There’s a program called: A Participatory Method for Engaging Stakeholders with Evaluation Findings, taught by Adrienne Adams at Michigan State University.

This is only a small sampling of the great workshops at the Summer Evaluation Institute.

For those of you who don’t know much about the American Evaluation Association: The AEA is an international professional association devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. AEA has approximately 7000 members representing all 50 states in the United States as well as over 60 foreign countries.

Cindy and I will be there – we hope to see you there too!

 

Data Party for Public Librarians

Friday, May 6th, 2016

The Engage for Health project team from left to right: Lydia Collins, Kathy Silks, Susan Jeffery, Cindy Olney

Last week, I threw my first data party. I served descriptive statistics and graphs; my co-hosts brought chocolate.

I first learned about data parties from evaluation consultant Kylie Hutchinson’s presentation It’s A Data Party that she gave at the 2016 American Evaluation Association Conference. Also known as data briefings or sense-making sessions, data parties actively engage stakeholders with evaluation findings.

Guest List

My guests were librarians from a cohort of public libraries that participated in the Engage for Health project, a statewide collaboration led by the NN/LM Middle Atlantic Region (MAR) and the Pennsylvania Library Association (PaLA). The NN/LM MAR is one of PaLA’s partners in a statewide literacy initiative called PA Forward, an initiative to engage libraries in activities that address five types of literacy.  The project team was composed of Lydia Collins of NN/LM MAR (which also funded the project), Kathy Silks of the PaLA, and Susan Jeffery of the North Pocono Public Library. I joined the team to help them evaluate the project and develop reports to bring visibility to the initiative.  Specifically, my charge was to use this project to provide experiential evaluation training to the participating librarians.

Librarians from our 18 cohort libraries participated in all phases of the planning and evaluation process.  Kathy and Susan managed our participant recruitment and communication. Lydia provided training on how to promote and deliver the program, as well as assistance with finding health care partners to team-teach with the librarians. I involved the librarians in every phase of the program planning and evaluation process. We met to create the project logic model, develop the evaluation forms, and establish a standard process for printing, distributing, and returning the forms to the project team. In the end, librarians delivered completed evaluation forms from 77% of their adult participants from Engage for Health training sessions.

What We Evaluated

The objective of PA Forward includes improving health literacy, so the group’s outcomes for Engage for Health was to empower people to better manage their health. Specifically, we wanted them to learn strategies that would lead to more effective conversations with their health care providers. Librarians and their health care partners emphasized strategies such as researching health issues using quality online health resources, making a list of medications, and writing down questions to discuss at their appointments.  We also wanted them to know how to use two trustworthy online health information sources from the National Library of Medicine: MedlinePlus and NIHSeniorHealth.

 Party Activities

Sharing with Appreciative Inquiry. The data party kicked off with Appreciative Inquiry interviews. Participants interviewed each other, sharing their peak experiences and what they valued about those experiences. Everyone then shared their peak experiences in a large group. (See our blog entries here and here for detailed examples of using Appreciative Inquiry.)

Data sense-making: Participants then worked with a fact sheet with graphs and summary statistics compiled from the session evaluation data.  As a group, we reviewed our logic model and discussed whether our data showed that we achieved our anticipated outcomes.  The group also drew on both the fact sheet and the stories from the Appreciative Inquiry interviews to identify unanticipated outcomes.  Finally they identified metrics they wish we had collected. What was missing?

Consulting Circles: After a morning of sharing successes, the group got together to help each other with challenges.  We had three challenge areas that the group wanted to address: integration of technology into the classes; finding partners from local health organizations; and promotional strategies.  No area was a problem for all librarians: some were quite successful in a given areas, while others struggled. The consulting groups were a chance to brainstorm effective practices in each area.

Next steps:  As with most funded projects, both host organizations hoped that the libraries would continue providing health literacy activities beyond the funding period.  To get the group thinking about program continuity, we ran a 1-2-4-All discussion about next steps.  They first identified the next steps they will take at their libraries, then provided suggestions to NN/LM MAR and PALA on how to support their continued efforts.

Post Party Activities

For each of the four party activities, a recorder from each group took discussion notes on a worksheet developed for the activity, then turned it into the project team. We will incorporate their group feedback into written reports that are currently in process.

If you are curious about our findings, I will say generally that our data supports the success of this project.  We have plans to publish our findings in a number of venues, once we have a chance to synthesize everything.  So watch this blog space and I’ll let you know when a report of our findings becomes available.

Meanwhile, if you are interested in reading more about data parties, check out this article in the Journal of Extension.

 

Diversity, Texas Libraries and Participatory Data Collection

Monday, May 2nd, 2016

On April 20, Cindy Olney and I facilitated a program for the Texas Library Association Annual Conference called Open Libraries! Making Your Library Welcome to All.  The program was sponsored by TLA’s Diversity and Inclusion Committee and the plan for the program was for attendees to work cooperatively to discover ways to ensure that people of diverse cultures, languages, ages, religions, sexual orientations, physical abilities, and others feel welcome at the library.  The committee wanted to get ideas from the wealth of TLA librarians’ experiences, so Cindy was invited to gather as much information from the attendees as possible. As co-chair of the TLA Diversity and Inclusion Committee, I co-facilitated the event.

The process used was a modified 1-2-4-All process, that you can find on the Liberating Structures website.  Our primary question was “What can my library do to become more welcoming to all people?”  We asked everyone in the room to brainstorm together all the different parts of a library that could be modified to make it more welcoming (e.g., reference services, facility, etc.).  We wanted to be sure that everyone thought as broadly and creatively as possible.

TLA Diversity Data Collection Program 2016

The discussion process actually had two parts.  For part one, we gave everyone two minutes to write as many ideas as they could on index cards (one idea per card).  Then we asked people to take two minutes to share their ideas with a partner.  They then shared their ideas with the entire table (up to 10 participants). The group then chose and wrote down the three best ideas and turned them in to the moderators.  Participants were instructed to leave their index cards with their ideas piled in the middle of their tables.

Here were some of the ideas that were generated through this discussion.

  • Welcome signs in different languages
  • Signage
  • Physical spaces – access to mobility

As you can see, the responses were fairly non-specific. We wanted richer descriptions of modifications of programs or services.  So part two of the process involved asking participants to develop more detailed plans for making their libraries more welcoming. Using a method involving dog, cat, and sea creature stickers, we moved participants randomly to new tables so that they ended up with a new group of colleagues.  They then chose a partner from their new table members and, as a pair, randomly chose one idea card the piles generated in part one of the process. They worked on a plan for one idea for eight minutes.  When the moderator called time, they pulled another card and worked on plans for a second idea. In the final eight minutes of the session, we asked for idea sharing by table to the entire group.

The plans in part 2 were better articulated and detailed than those we got in part one. Here are some examples of the kind of result we got from that exercise:

  • Signage: Making clearer, more colorful. Different languages signage or use digital signage.
  • Language material specific to the community and programming in various language spoken in the community. ESL classes partnered with community colleges.
  • Invite representatives from ADA/disability advocates to give suggestions on making library desks/areas more accessible.

The whole process was completed in a 50-minute conference program session.  Both myself and the other Diversity and Inclusion co-chair, Sharon Amastae from El Paso, TX, were impressed with the energy and enthusiasm that was present among attendees in the room.

The results of this data gathering event will be communicated to the TLA membership.  When that project is completed, we’ll let you know here on the NEO Shop Talk blog!

Photo credit: Esther Garcia

 

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.