Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for February, 2016

Setting a Meaningful Participation Target

Friday, February 26th, 2016

Picture of people stand on each other to reach a star

I enjoyed reading an article in Public Libraries titled “The Grass Is Always Greener” by Melanie A. Lyttle and Shawn D. Walsh.  They discuss the complexities of deciding whether a program was “well attended” or “nobody came.”  Sometimes a program that seems well attended in one situation is the same as a poorly attended program in another.

I can think of a lot of times I’ve experienced this exact situation. When I was a branch manager at a public library, the program manager at the main library would ask if she could send authors to speak at our branch library.  When I said, “maybe you should send them somewhere else – we only had ten people come to the last one,” she replied “ten is a lot – ten is more than we get anywhere else.”

When I worked at the NN/LM South Central Region, in some parts of the region 30 people could be expected to attend training sessions.  In other parts of the region, we considered 6 people a successfully attended program.  These differences often corresponded to urban vs. rural, or the travel distance needed to get to the training, or whether the librarians were largely solo librarians or worked in multi-librarian organizations, or whether their institutions supported taking time off for training.  Other considerations include whether the trainers had already built an audience over time that would regularly attend the programs. Or on the other hand, whether the trainers had saturated their market and there were very few new people to learn about the topic.

So how can you decide what a good target participation level should be, or maybe more importantly, how can you explain your participation targets to your funder or parent organization?

Tying your participation level to your intermediate and long-term intended outcomes is one way to do that.  Let me give you an example of a program in Houston that was funded by the NN/LM South Central Region. The Greater Houston AHEC received funding many years ago to do an in-depth training project with a small number of seniors in the most underserved areas of Houston.  The goals were to teach these seniors how to use computers, how to get on the Internet, how to use email, and then how to use MedlinePlus and NIHSeniorHealth to look up health information.  They planned for the seniors to take 2-3 classes a week, and each class lasted several hours. It was a big commitment, but they intended for these seniors to really know how to use the Internet at the end of the series.  There were so few seniors who saw the need to learn to use computers that they had to persuade about 10 people from each location to sign up.  However, the classes were so good and the seniors so enthusiastic, that after a couple of weeks, the other seniors wanted to take classes too.  This led to a phase 2 project which included funding for a permanent computer and coffee area in a senior center where students could practice their Internet skills. There is now a third phase of the program called M-SEARCH which teaches seniors to use mobile devices to look up their health information.

At the beginning, Greater Houston AHEC may not have envisioned these specific outcomes.  However, if they were trying to convince a funder that 10 person classes were a reasonable use of the funder’s money, it might be good to show that small in-depth classes could lead to a long-term outcome like “seniors in even the poorest neighborhoods in Houston will be able to research their health conditions on NIHSeniorHealth.”  In addition, it would be important to bring in other factors, such as your intended goals for the project, for example whether you hope to have a small group of these seniors that you can train to really use the Internet for health research or whether you want to reach a lot of seniors in underserved areas to let them know that it’s possible to find great health information using NLM resources (see the Kirkpatrick Model of training evaluation for more information on evaluating your training goals).

For more on creating long-term outcomes, see Booklet 2 of the OERC’s booklets: Planning Outcomes-based Outreach Projects

Appreciative Inquiry of Oz: Building on the Best in the Emerald City

Friday, February 19th, 2016

Cartoon image of an Emerald City

“One day not very long ago, librarians came to the Emerald City from their libraries in all of the countries of Oz. They came to visit the Great Library of the Emerald City, and to petition the Wizard allow them to borrow books and other items at the Great Library. Their hope was to transport items from one library to another using the Winged Monkeys, who offered their skills for this task after they were set free and got bored.”

Thus begins the latest OERC project – an online class in Appreciative Inquiry (AI), offered through the MidContinental Region’s Librarians in the Wonderful Land of Oz Moodle ‘game’ (i.e. series of online classes worth game points and CE credits from the Medical Library Association).  The game is made up of several ‘challenges’ (online classes) for librarians offered by NN/LM instructors.

In OERC’s challenge, Building on the Best at the Great Library of the Emerald City: Using Appreciative Inquiry to Enhance Services and Programs, the Wizard of Oz makes a deal with the librarians.  He will allow interlibrary loan of the Great Library’s resources if the librarians will assess customer satisfaction of the Great Library’s services and find things to improve.  And students in the class will learn to use a qualitative data collection technique called Appreciative Inquiry to do this assessment.

Sometimes people avoid customer service assessment because they find the methods to be complicated and time-consuming. Negative feedback can be uncomfortable on the part of the listener and the speaker. Appreciative Inquiry, with a focus on identifying and building on organizational strengths, removes that discomfort. A number of OERC workshops touch on Appreciative Inquiry but this Librarians of Oz challenge allows you to practice the technique, something that the OERC has not been able to provide in the traditional webinar or workshop context.  Completing the class is worth 14 MLA CE credits.

The class is free, but in order to take it you will need to register for the game Librarians in the Wonderful Land of Oz .  If you don’t want to take the class, but would still like to learn more about Appreciative Inquiry, I recommend these earlier blog posts:

From Cindy and Karen’s perspective, one of the best parts of this experience is that we finally get the official title of Wizard.  Special thanks to John Game Wizard Bramble of the NN/LM MCR who made all this happen.

 

W.A.I.T for Qualitative Interviews

Friday, February 12th, 2016

WAIT

Why 

Am 

Talking?

 

My sister-in-law recently told me about the W.A.I.T. acronym that she learned from a communication consultant who spoke to her staff. It’s a catchy phrase for an important communication concept: Be purposeful when you talk. This self-reflective question can be applied to any conversational setting, but I want to discuss it in the context of qualitative interviews for evaluation data collection.

Surveys and tests are examples of quantitative data collection instruments. They require careful crafting and pilot-testing to be sure they collect valid information from respondents. By contrast, in qualitative interviews, the data collection instrument is the interviewer.  The interview guide itself is important, but the interpersonal manner of the interviewer has far greater impact on the trustworthiness of the information gathered. The key responsibility of the interviewer is described succinctly by Michael Q. Patton in Qualitative Research and Evaluation Methods:

“It is the responsibility of the interviewer to provide a framework within which people can respond comfortably, accurately, and honestly to open-ended questions.”

Listening skills, of course, are key to good interviewing.  As program evaluator Kylie Hutchinson said recently in a 2016 American Evaluation Association conference presentation, evaluators need ask their questions, then shut up.  If you can learn to do this, you are more than halfway there.  Julian Treasure has a TEDtalk with excellent tips on developing your listening skills.

However, how you talk is important as well.  Here are a few ways I would answer the question “Why Am I Talking?” during an interview:

  • I want to show that I share something in common with my interviewee: People are more comfortable talking to others who are like them. I say things like “I feel that way, too, sometimes” or “I know what you mean. Something like that happened to me a few years ago.”  These statements can help me build rapport.
  • I want the interviewee to know that no answer he or she gives can surprise me. Social desirability is something that survey researchers always consider in instrument design. Even in the anonymous survey context, people may give answers to make themselves “look good.” So you can imagine that the dynamic is even greater in the face-to-face interview setting. When broaching a sensitive topic, I let my interviewee know I’ve heard it all before. I might say, for instance, “Some people have told me they spent hours researching a serious health condition. Others say they were so frightened by the diagnosis, they didn’t want to read anything about it. How did you respond when you were diagnosed?”
  • I want to allow the interviewee an opportunity to answer a question hypothetically. Sometimes you may ask an interviewee about choices or behaviors that are potentially embarrassing. Let’s say I want to know what barriers prevented them from following their doctors’ orders. This question could feel awkward to interviewees if, for example, they lacked understanding or willpower to follow a physician’s recommendations. So I frame questions that allow them to distance themselves personally from their answers. Rather than asking them to describe a time they didn’t follow a doctor’s orders, I might say “Sometimes people don’t do what their doctors tell them to. In your experience, what are some of the reasons people might not follow their doctor’s orders?
  • I want to show I’m listening and to check my understanding: Paraphrasing your interviewee’s comments is an active listening technique that demonstrates your interest in the ongoing discussion. It also is a validity check on your own interpretations of their answers. I say things like “Okay, so let me make sure I understand.  Essentially, you are saying…?”
  • I’m managing the emotional climate and turn-taking in a focus group. I choose language to maintain a neutral, non-judgmental atmosphere and to model respectful interaction. I also talk when I need to reign in someone who is dominating the discussion. I might say “So  Truman gave us quite a few great examples of how she uses MedlinePlus. What examples can someone else add to Truman’s examples?”

All of these tips, by the way, are from Patton’s book on qualitative methods. Here is the full citation:

Patton, MQ. Qualitative Research and Evaluation Methods: Integrating Theory and Practice (4th ed.). Thousand Oaks, CA: Sage, 2015.

If you would like to read more about W.A.I.T, here’s an excellent article from the National Speakers Association.  I also want to thank Lauren Yee and Donna Speller Turner from the NASA Langley Research Center for alerting me to W.A.I.T.

 

Logic Model for a Birthday Party

Thursday, February 4th, 2016

Cindy and I feel that logic models are wonderful planning tools that can be used in many life events to stay focused on what’s meaningful. This blog post is an example of such a logic model.

My daughter’s birthday is coming up this week and we are having a party for her. My husband and I have quite a few friends with children about the same age as our daughter (who is turning 3).  This means that we go to birthday parties and we have birthday parties, and we are looking forward to another 15 years or so of birthday parties.  Even though we live in the 4th largest city in the country, it’s a bit of an project to come up with a place for the party.  I could see this problem stretching out into future years of Chuck E. Cheese’s and trampoline parks. Not that there’s anything wrong with those places, but we realized that for us it was time to stop the train before we went off the rails.  Looking at my own childhood, my birthday parties growing up were all at my own house. So we decided to see if we could have a party at our house and just have fun.

To make sure we had a great event and kept our heads on straight (and had something to blog about this week), I created a logic model for my daughter’s birthday party. We needed an evaluation question, which is “is it possible to have a party of preschoolers at our tiny, not-that-childproofed-house without going crazy?”

So here is the event we have planned.

Birthday Party Logic Model

If you’re new to logic models, they are planning tools that you use from right to left, starting with long-term outcomes (what you hope to see out in the future), intermediate outcomes, and short term outcomes. Then you think of the activities that would lead to those outcomes, and then inputs, the things you need in order to do the activities. (For more information on logic models, take a look at the OERC Blog category “Logic Models“).

What I’ve learned from this process is that every time I would come up with an idea about what we could do at the party, it would need to pass the test of whether or not it leads to the long-term outcome of being willing to throw parties in the house in the future – in other words if the party takes too much work or money (or it isn’t fun), we won’t remember it as an event we are likely to do again. For example, while we are inviting a person to our house to entertain the kids, we’re bringing our daughter’s music teacher from her preschool, so it should be fun for the kids that she knows from pre-school and everyone will know the music and can sing along.  Another activity that has high enjoyment and low effort is the dance party with bubbles. All toddlers love to dance, and we can make a playlist of all of our daughter’s favorite dance songs.  Adding bubbles to the mix is frosting.

The short term goals are our immediate party goals.  We would like the party to be fun for our daughter and for most of her friends (can we really hope for 100%?  Probably not, so we put 90%).  My husband and I may be a little stressed but we’re setting our goal fairly low at being relaxed 60% of the time (you’ll have to imagine maniacal laughter here).  Our intermediate goals are simply that we all can feel comfortable having our daughter’s friends over to our house in the near future. And the long term goal is to think this is a good idea to do again and again.

Wish us luck!

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.