Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘Data Analysis’ Category

Data Party for Public Librarians

Friday, May 6th, 2016

The Engage for Health project team from left to right: Lydia Collins, Kathy Silks, Susan Jeffery, Cindy Olney

Last week, I threw my first data party. I served descriptive statistics and graphs; my co-hosts brought chocolate.

I first learned about data parties from evaluation consultant Kylie Hutchinson’s presentation It’s A Data Party that she gave at the 2016 American Evaluation Association Conference. Also known as data briefings or sense-making sessions, data parties actively engage stakeholders with evaluation findings.

Guest List

My guests were librarians from a cohort of public libraries that participated in the Engage for Health project, a statewide collaboration led by the NN/LM Middle Atlantic Region (MAR) and the Pennsylvania Library Association (PaLA). The NN/LM MAR is one of PaLA’s partners in a statewide literacy initiative called PA Forward, an initiative to engage libraries in activities that address five types of literacy.  The project team was composed of Lydia Collins of NN/LM MAR (which also funded the project), Kathy Silks of the PaLA, and Susan Jeffery of the North Pocono Public Library. I joined the team to help them evaluate the project and develop reports to bring visibility to the initiative.  Specifically, my charge was to use this project to provide experiential evaluation training to the participating librarians.

Librarians from our 18 cohort libraries participated in all phases of the planning and evaluation process.  Kathy and Susan managed our participant recruitment and communication. Lydia provided training on how to promote and deliver the program, as well as assistance with finding health care partners to team-teach with the librarians. I involved the librarians in every phase of the program planning and evaluation process. We met to create the project logic model, develop the evaluation forms, and establish a standard process for printing, distributing, and returning the forms to the project team. In the end, librarians delivered completed evaluation forms from 77% of their adult participants from Engage for Health training sessions.

What We Evaluated

The objective of PA Forward includes improving health literacy, so the group’s outcomes for Engage for Health was to empower people to better manage their health. Specifically, we wanted them to learn strategies that would lead to more effective conversations with their health care providers. Librarians and their health care partners emphasized strategies such as researching health issues using quality online health resources, making a list of medications, and writing down questions to discuss at their appointments.  We also wanted them to know how to use two trustworthy online health information sources from the National Library of Medicine: MedlinePlus and NIHSeniorHealth.

 Party Activities

Sharing with Appreciative Inquiry. The data party kicked off with Appreciative Inquiry interviews. Participants interviewed each other, sharing their peak experiences and what they valued about those experiences. Everyone then shared their peak experiences in a large group. (See our blog entries here and here for detailed examples of using Appreciative Inquiry.)

Data sense-making: Participants then worked with a fact sheet with graphs and summary statistics compiled from the session evaluation data.  As a group, we reviewed our logic model and discussed whether our data showed that we achieved our anticipated outcomes.  The group also drew on both the fact sheet and the stories from the Appreciative Inquiry interviews to identify unanticipated outcomes.  Finally they identified metrics they wish we had collected. What was missing?

Consulting Circles: After a morning of sharing successes, the group got together to help each other with challenges.  We had three challenge areas that the group wanted to address: integration of technology into the classes; finding partners from local health organizations; and promotional strategies.  No area was a problem for all librarians: some were quite successful in a given areas, while others struggled. The consulting groups were a chance to brainstorm effective practices in each area.

Next steps:  As with most funded projects, both host organizations hoped that the libraries would continue providing health literacy activities beyond the funding period.  To get the group thinking about program continuity, we ran a 1-2-4-All discussion about next steps.  They first identified the next steps they will take at their libraries, then provided suggestions to NN/LM MAR and PALA on how to support their continued efforts.

Post Party Activities

For each of the four party activities, a recorder from each group took discussion notes on a worksheet developed for the activity, then turned it into the project team. We will incorporate their group feedback into written reports that are currently in process.

If you are curious about our findings, I will say generally that our data supports the success of this project.  We have plans to publish our findings in a number of venues, once we have a chance to synthesize everything.  So watch this blog space and I’ll let you know when a report of our findings becomes available.

Meanwhile, if you are interested in reading more about data parties, check out this article in the Journal of Extension.

 

Dashboards for Your Library? Here Are Some Examples

Friday, December 18th, 2015

illustration of different business graphs on white backgroundLast week’s blog post was about using Excel to make data dashboards. As Cindy pointed out a dashboard is “a reporting format that allows stakeholders to view and interact with program or organizational data, exploring their own questions and interests.”

What can that mean for your library? What does a library data dashboard look like?

In the OERC Tools and Resources for Evaluation, we have a Libguide for Reporting and Visualizing, which includes a section on data dashboards.  In it are some examples of libraries using data dashboards.  In their dashboards, libraries are sharing data on some of the following things:

  • How much time is spent helping students and faculty with research
  • What databases are used most often
  • How e-books are changing the library picture
  • What librarians have been learning at their professional development conferences
  • What is the use of study rooms over time
  • What month is the busiest for library instruction
  • What department does the most inter-library loan

Can you create a dashboard to tell a story? While libraries can keep (and post) statistics on all kinds of things, consider who the dashboard is for, and what story you want to tell them about your library.  Maybe it’s the story of how the library is using its resources wisely.  Or maybe it’s the story of why the library decided it needed more study rooms.  Or the story of whether or not the library should eliminate it’s book collection and increase e-books and databases.

Consider what data you want to share and what people are interesting in knowing.  Happy dashboarding!

Measuring What Matters in Your Social Media Strategy

Thursday, December 3rd, 2015

Thumbs up symbols with text "get more likes"

We’re all trying to find ways to improve evaluation of our social media efforts. It’s fun to count the number of retweets, and the number of ‘likes’ warms our hearts.  But there’s a nagging concern to evaluators – are these numbers meaningful?

Your intrepid OERC Team, Cindy and Karen, attended a program at the American Evaluation Association conference in Chicago called “Do Likes Save Lives? Measuring What Really Matters in Social Media and Digital Advocacy Efforts,” presented by Lisa Hilt and Rebecca Perlmutter of Oxfam.  The purpose of their presentation was to build knowledge and skills in planning and measuring social media strategies, setting digital objectives, selecting meaningful indicators and choosing the right tools and approaches for analyzing social media data.

What was interesting about this presentation is that the presenters did not want to rely solely on what they called “vanity metrics,” for example the number of “impressions” or “likes.”  Alone these metrics show very little actual engagement with the information.  Instead they chose to focus on specific social media objectives based on their overall digital strategy.

Develop a digital strategy

  • Connect the overall digital strategy to campaign objectives: (for example: To influence a concrete change in policy, or to change the debate on a particular issue.)

Develop social media objectives

  • You want people to be exposed to your message
  • Then you want people to engage with it somehow (for example, sharing your message) or make them work with it somehow (for example: sign an online petition after reading it).

Collect specific information based on objectives

  • Collect data about social media engagement supporting your objectives that can be measured (for example “the Oxfam Twitter campaign drove 15% of the readers to signing its petition” vs. “we got 1500 likes”)

The presenters suggested some types of more meaningful metrics:

  • On Twitter you can look at the number of profiles who take the action you want them to take, and then the number of tweets or retweets about your topic.
  • For Facebook, the number of likes, shares and comments mean that your audience was definitely exposed to your message.
  • Changes in the rate of likes or follows (for example if you normally get 5 new followers to your fan page a week, but due to a particular campaign strategy, you suddenly started getting 50 new followers a week)
  • Number of “influential” supporters (for example, being retweeted by Karen Vargas is not the same as being retweeted by Wil Wheaton).
  • Qualitative analysis: Consider analyzing comments on Facebook posts, or conversation around a hashtag in Twitter.

Overall, your goal is to have a plan for how you would like to see people interact with your messages in relation to your overall organizational and digital strategies, and find metrics to see if your plan worked.

 

Data Party Like it’s 2099! How to Throw a Data Party

Friday, November 20th, 2015

two funny birthday dogs celebrating close together as a coupleWhat’s a “data party?” We attended a program by evaluator Kylie Hutchinson entitled “It’s a Data Party!” at the AEA 2015 conference last week in Chicago.  A data party is another name for a kind of participatory data analysis, where you gather stakeholders together, show them some of the data that you have gathered and ask them to help analyze it.

Isn’t analyzing the data part of your job?  Here are some reasons you might want to include stakeholders in the data analysis stage:

  • It allows stakeholders to get to know and engage with the data
  • Stakeholders may bring context to the data that will help explain some of the results
  • When stakeholders participate in analyzing the data, they are more likely to understand it and use it
  • Watching their interactions, you can often find out who is the person with the power to act on your recommendations

So how do you throw a data party? First of all you need to know what you hope to get from the attendees, since you may only be able to hold an event like this one time. There are a number of different ways to organize the event.  You might want to consider using a World Cafe format, where everyone works together to explore a set of questions, or you could use an Open Space system in which attendees create their own agenda about what questions they want to discuss.  Recently the AEA held a very successful online unconference using MIT’s Unhangout that could be used for an online Data Party with people from multiple locations.

The kinds of questions Kylie Hutchinson suggested asking at a data party include:

  • What does this data tell you?
  • How does this align with your expectations?
  • What do you think is occurring here and why?
  • What other information do you need to make this actionable?

At the end of the party it might be time to present some findings and recommendations that you have.  Considering the work that they have done, they may be more willing to listen.  As Kylie Hutchinson said “People support what they helped create.”

 

Soup Up Your Annual Reports with Calculator Soup

Friday, August 14th, 2015

Summer is annual report time for our organization. Sometimes when I’m putting together my bulleted list of accomplishments for those reports, I feel as though our major wins get lost in the narrative. So I recently turned to an online calculator to help me create better metrics to talk about our center’s annual wins.

One of our objectives for the year was to increase participation in our evaluation training program. We developed new webinars based on our users’ feedback and also increased promotion of our training opportunities. The efforts paid off: training session attendance increased from 291 participants the previous year to 651 this year. Now that is a notable increase, but the numbers sort of disappear into the paragraph, don’t they? So I decided to add a metric to draw attention to this finding: Our participation rate increased 124% over last year’s attendance. Isn’t “percent increase” a simpler and more eye-catching way to express the same accomplishment?

Doing this extra analysis seems simple, but it takes time and gives me angst because it usually requires manual calculation. First I have to look up the formula somewhere. Then I have to calculate the statistic. Then I calculate it again, because I don’t trust myself. Then I calculate it again out of pure obsessiveness.

That’s why I love online calculators. Once I find one I like and test it for accuracy, I bookmark it for future use. From then on, I let the calculator do the computation because it is infinitely more reliable than I am when it comes to running numbers.

One of my favorite sites for online calculators is Calculator Soup, because it has so many of them. You may not ever use 90% of its calculators, but who knows when you might need to compute someone’s age from a birth date or convert days to hours. The calculators also show you the exact steps in their calculations. This allows you to check their work. You also can find formulas that you then can apply in an Excel spreadsheet.

One word of advice: test a calculator for accuracy before adopting it. I always test a new calculator to be sure the designers knew what they were doing. For Calculator Soup, I can vouch for the percent change and the mean/median/mode calculator. If I use any others at that site, I’ll test them as well. I’ll create an easy problem that I can solve manually and make sure my result matches the calculator’s.

If you want to see what Calculator Soup has to offer, check out their calculator index here.

Random array of simple arithmetic formulas

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.