Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for October, 2016

How I Learned to Stop Worrying and Love Logic Models (The Chili Lesson)

Friday, October 28th, 2016

michelle

By Michelle Malizia, Director of Library Services for the Health Sciences, University of Houston

I’ll start with a full disclosure: I am a late convert to logic models. Many years ago, I worked in a department that, for a period of time, became governed by logic models. This experience made me fear… no, hate… logic models.  Several years later, through external workshops and assistance from the NN/LM Evaluation Office, I was introduced to the tremendous value of logic models.

My closest personal analogy relates to my feelings about chili. I grew up eating my mother’s chili, which basically consisted of cans of many different types of beans floating in a type of broth. I hated it. When I was 22 years old, I had no choice but to eat someone else’s chili. This chili had lots of ground beef and spices. It was delicious. Then it occurred to me, my mother’s chili was my only frame of reference for chili. I didn’t dislike chili – I disliked my mother’s chili.

And so it goes with logic models. Once I learned a different way to make and apply them, I became a dedicated user. I now design logic models whenever I plan a new service, activity or initiative.

In 2014, I was hired as the Director of Library Services for the Health Sciences at the University of Houston (UH). In 2017, UH will have its first medical library and my task is to plan the services for the new facility. Of course, I turned to logic models because they provided the framework of not only what I am planning but why I am planning each service and ultimately, how I will evaluate if I achieved the goal.

When I started with my logic models, I was tempted to begin with the activity. I had to remind myself that it is more important to document what I hope to accomplish by that activity (i.e. outcome).  Think about it: Why do librarians teach PubMed classes? Why do librarians want to be embedded in a nursing class? Why do so many libraries provide liaison services? Many of you are probably thinking: “That’s easy, Michelle. We do those things to better serve our customers.” My response is:  How do you know those activities better serve your customers?  How can you prove it to your stakeholders?  That is the reason you should start with your outcomes rather than activities.

For example, my new library will be providing assistance with NIH Public Access Policy compliance. When I developed my logic model, I called upon my inner 3-year old to ask the question best asked by toddlers:  Why.  Because I have a creative side, I use the software Visio (a Microsoft Office software) to create my logic models. It allows me to visually see connections between activities. The chart below shows  a portion of my logic model.

NIH Public Access Policy Assistance Services logic Model One. Activities are conduct workshops and assistance with compliance. Outputs are number of workshops and number of consultations. Short-term outcomes are increase awareness of NIH Public Access Policy and increased knowledge of compliance specifics. Intermediate outcome is compliance and long-term outcome is UH retains and receives NIH grants.

 

As you can see, my long-term outcome for this activity was to ensure that UH retains and receives NIH Grants. If UH researchers don’t comply with the NIH Public Access Policy mandate, their current and future funding is in jeopardy. The intermediate outcome leading to the long-term outcome is increased compliance with the policy. In order to increase compliance, I need to make researchers aware of the policy and how to comply.  That’s when I was able to determine the best methods for me to accomplish those outcomes. For my university environment, the best way to achieve these outcomes is through workshops and consultations.

Now that I knew the “what and the why,” I needed to determine the how.  How would I know if I accomplished my goals? Again, I turned to Visio to visualize how I could assess if I achieved my outcomes.

My final step was to determine my measurable indicators. For example, in the case of workshops, my indicator was “% of workshop attendees who reported being more knowledgeable about how to comply with the policy.”  My target was 85% of attendees. To evaluate this outcome, I would use a pre- and post-test.

 

Evaluation plan for NIH Public Access Policy Assistance Services. Outputs are workshops and consultations, leading to short-term outcomes of increased awareness of NIH Public Access policy and increased knowledge of compliance specifics. The outcomes will be assessed with a pre-post test and follow-up questionnaire. The intermediate outcome is increased compliance, which will be assessed with a survey and/or other follow-up

My overall work with logic models led to a pleasant surprise. Mid-way through my process, UH Libraries adopted a new strategic plan. Strategic plans are usually written in terms of goals. Some of my colleagues tried to feverishly determine where their activities fit into the library’s overall goals. Because I had already determined my outcomes, it was easy to slot my activities into the library’s overall plan.

If you have had a previous bad experience creating logic models, try it again. Ask the NEO for assistance and look at their extremely helpful guides. Like me, you may finally realize that logic models are worth the time and energy. Remember, there are many different types of chili.  Find the one that you like best.

NEO note: The evaluation field has come a long way in discovering new, less painful approaches to creating and using logic models.  If, like Michelle, you had bad experiences years ago with logic models, you might want to give them another chance.  You can learn one approach through the NEO booklet Michelle mentioned, which is  Planning Outcomes-Based Projects (Booklet 3 in our Planning and Evaluating Health Information Outreach Programs series). For alternative approaches, check out our NEO Shop Talk blog entries  Logic Model for a Birthday Party and An Easier Way to Plan: Tearless Logic Models.

 

How Many Interviews Does It Take to Assess A Project?

Friday, October 21st, 2016

A green piggy bank standing with a group of pink piggy banks to represent the cost effectiveness of individual interviews

FAQ from NEO users: How many interviews or focus groups do we need for our qualitative assessment project?

Our typical response: Um, how much money and time do you have?

At which point, our users probably want to throw a stapler at us. (Karen and I work remotely out of an abundance of caution.)

Although all NEO users are, in fact, quite well-mannered, I was happy to discover a study that allows us to provide a better response to that question.  A study published by Namey et al, which appears in the American Journal of Evaluation’s September issue, provides empirically based estimates of the number of one-to-one or group interviews needed for qualitative interviewing projects.  More specifically, their study compared the cost effectiveness of individual and focus group interviews. The researchers conducted an impressive 40 interviews and 40 focus groups (with an average of eight people per group). They then used a boot-strap sampling methodology, which essentially allowed them to do 10,000 mini-studies on their research questions.

They first looked at how many individual and focus group interviews it took to reach what qualitative researchers call thematic saturation. In lay terms, saturation means “Not really hearing anything new here.”  Operationally, it occurs when 80-90% of the information provided in an interview has already been covered in the previous interviews.

The researchers found that 80% saturation occurred after 2-4 focus groups or eight individual interviews. It took 16 interviews and five focus groups to reach 90% saturation. Note their estimates apply to studies that focus on one specific population.  If you want to explore the experiences of two groups, such as doctors and nurse practitioners, you would hold eight interviews per group to reach 80% thematic saturation.

For comparative cost assessment, the researchers used a formula that combined hourly rate of the data collector’s time, incentive costs per participant, and cost of transcription for recordings. They chose not to include cost for factors that vary widely, such as space rental or refreshments. Using more predictable costs made for cleaner and more generalizable comparisons.

Bottom line, they found individual interview methods cost 12-20% less than focus group methods.

Of course, many of us operate on shoestring budgets, so we are our own moderators and transcribers.  Even though most of us DIYers collect hourly wages, the cost for outsourcing these tasks is probably higher than for conducting them internally. Knowing this, the researchers looked at variations on moderator, transcriptionist, and incentive costs.  They also compared cost effectiveness of the two methods when lowering the standards for thematic saturation (i.e., aiming for 70% saturation instead of 80%). Across the board, individual interviews were more cost-effective than focus groups.

Cost is not always the only consideration when choosing between focus groups and individual interviews. Some assessment questions beg for group brainstorming, while others demand the privacy of one-to-one discussions. However, for many assessment studies, either method is equally viable.  In that case, cost and convenience will drive your decision. Personally, I often find individual interviews to be more convenient than focus groups, both for the participants and for me. It’s nice to know that the cost justifies using the more convenient approach.

The full article provides details on their methods, so it is a nice primer on qualitative analysis of interview transcripts. Here’s the full citation:

Namey E, Guest G, McKenna K, Chen M. Evaluating bang for the buck: a cost effectiveness comparison between individual interviews and focus groups based on thematic saturation levels. 2016 September; 37(3): 425-440.

 

Meet the NEO’s New Program Assistant Kalyna Durbak

Friday, October 14th, 2016

Kalyna Durbak

I am pleased to introduce the NEO’s new program assistant, Kalyna Durbak, MLIS, who joined our staff on October 3.  Kalyna will be our go-to person for managing the NEO website, providing technical support with webinars, and helping with the “roll-up-your-sleeves” work involved in carrying out evaluation projects.

Kalyna began working for the UW Health Sciences Library in May 2016. Prior to joining the NEO, Kalyna was the Web Content Assistant on the team that created and promotes the Response & Recovery App in Washington (RRAIN), designed to provide emergency responders with quick access to disaster-management resources. It also provides local information such as weather alerts and traffic reports. Kalyna also provided web content and social media assistance for the Health Evidence Resource for Washington State (HEALWA), a portal that provides affordable online access to clinical information and health education resources. The portal is available to health professionals who are licensed through 23 state organizations. A 2015 evaluation study conducted by HEALWA showed that many health professionals eligible to use the portal are not aware of it.  Kalyna helped promote HEALWA through social media and exhibits.

Kalyna earned her MLIS degree from University of Illinois at Urbana–Champaign and a BA in History from University of Illinois at Chicago. She was an intern at the Smithsonian Ralph Rinzler Folklife Archives and Collections and the Rochester Institute of Technology Archives. A Midwestern native, she recently moved to Seattle with her husband because they were attracted to Seattle’s mix of urban and outdoor opportunities. To introduce herself to our readers, Kalyna agreed to answer a few questions about herself.

What made you want to pursue an MLS?

I always considered myself a “jack of all trades.” At school I did not excel in one subject, but rather did fairly well in most areas of study. I also fell in love with researching, and doing “deep dives” into different subjects. I figured that with an MLS, I could end up working in vastly different environments, help others with research, and pursue my dream of being a lifelong learner.

What made you want to join the NN/LM Evaluation Office?

I recently realized that I needed to strengthen my evaluation skills. Whether I am working or volunteering, I am constantly trying to solve issues concerning outreach and training. For most of my career, I just created solutions without ever thinking “How can I measure my success in solving this issue?” and “Are these solutions working the way I intended?” These questions are key in determining whether the solution is actually solving any problems, or just wasting time and energy.

What experience have you had with evaluation?

My experiences with evaluation come from managing social media accounts. Once I realized I had a whole dashboard of statistics to my disposal, I used them to set optimization goals in terms of posting times and types of content that resonate with my audiences.

What evaluation skills do you particularly hope to develop?

I am very interested in developing my outcome assessment skills. I am usually the big idea person of a group, and enjoy setting lofty goals. In the past, I have measured the success of an initiative based on the number of tasks my group completed for the project. What I want to do going forward is measure success by the initiative’s impact on the intended audience and community.

What other interests do you have?

I am very active in a Ukrainian Scouting Organization called Plast. Through scouting I found my love for the outdoors, and made countless friends all over the United States and around the world. When I’m not working on scouting activities, I find myself crafting. My favorite crafts include quilling, card making, and traditional Ukrainian embroidery.

When I am crafting or commuting to work, I listen to various nerdy podcasts. Some of my favorites include 99% Invisible, LibUX, and Reply All

What is the bravest thing you’ve ever done?

I took a hike with my husband down the Grand Canyon. I didn’t make it that far, because I’m afraid of heights. There was one foot between my body and a drop into the canyon, and that was not where I wanted to be. I had to turn back partway down. My husband said, “I love you. Do you mind if I keep going?”  So I had to walk back up the trail alone. Looking back, I’m glad I went through it.  Once I climbed up, I felt so proud of myself.

What’s in a Name? Convey Your Chart’s Meaning with a Great Title

Friday, October 7th, 2016

Some of you may be working on conference posters and paper presentations for Fall conferences.  And some of those will probably include charts to demonstrate data representing a lot of hard work on your part.  In most cases you have minutes to use that chart to get your audience to understand the data.

Stephanie Evergreen has great advice for displaying chart data.  She literally wrote the books on it: Presenting Data Effectively and Effective Data Visualization.  Her recent blog post is about one of the simplest and most powerful changes you can make to effectively present your chart data: “Strong Titles Are The Biggest Bang for Your Buck.

What many of us do is present the data with a generic title, like “Attendance rates.” Then the viewer has to spend time working through the data and you hope that they see what you wanted them to.  What Stephanie Evergreen proposes (backed by persuasive research) is to give your charts a clear title that explains what the data shows. Your poster or paper is almost certainly making a point.  Determine how your chart supports the point of your presentation and state that in the title.  Here are some reasons why:

  • It respects your viewers’ time
  • It forces you to be clear about the point you want your data to make
  • It makes the data more memorable

Stephanie Evergreen’s post has some great examples of how a good title can really improve the impact of the chart.  In addition, here is an example from the NEO webinar Make Your Point: Good Design for Data Visualization.

Looking at this original chart, you might notice that in each activity the follow-up showed an increase over the baseline.  If you, the viewer, didn’t have a lot of time, that might be all you noticed.

Chart with title: Comparison of emergency preparedness activities from baseline to follow-up

With a simple change of title , you can see that the author of this presentation is highlighting the increased number of continuity of services plans.  This is designed to enhance the point of the presentation, and not waste the viewers’ time. Also, note that the title is left justified instead of centered.  Because the title is a full sentence, a left-justified format is easier to read.

Chart with title: The biggest improvement in emergency preparedness from baseline to follow-up was the number of network member organizations reporting that they had or were working on a service continuity plan.

So, while Shakespeare might have been correct when he wrote “What’s in a name? that which we call a rose / By any other name would smell as sweet,” what if the presenter was trying to show the fortitude of Texas antique roses to survive in harsh weather conditions, and the viewer only noticed how sweet the rose smelled?  Maybe the heading “A Rose” sometimes isn’t enough information.

Save

Save

Save

Save

Save

Save

Save

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.