Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘Simply Elegant Evaluation’ Category

Simply Elegant Evaluation: GMR’s Pilot Assessment of a Chapter Exhibit

Friday, January 15th, 2016

If you spend any time with librarians who work for the National Network of Libraries of Medicine (NN/LM), you’ll likely hear about their adventures with conference exhibits. Exhibiting is a typical outreach activity for the NN/LM Regional Medical Libraries, which are eight health sciences libraries that lead other libraries and organizations in their region in promoting the fine health information resources of the National Library of Medicine (NLM) and National Institutes of Health.  The partnering organizations are called “network members” and, together with RMLs, are the NN/LM.

Jacqueline Leskovec

Exhibiting is quite an endeavor. It requires muscles for hauling equipment and supplies. You have to be friendly and outgoing when your feet hurt and you’re fighting jet lag. You need creative problem-solving skills when you’re in one state and your materials are stuck in another.

More than one RML outreach librarian has asked the question: Is exhibiting worth it?

Jacqueline Leskovec, at the NN/LM Greater Midwest Regional Medical Library (GMR), decided to investigate this question last October. The Outreach, Planning, and Evaluation Coordinator for NN/LM GMR, Jacqueline specifically chose to assess a particular type of NN/LM exhibits: those held at Medical Library Association chapter meetings.  The question was raised at a GMR staff meeting about the value of exhibiting at a conference where most attendees were medical librarians, many of whom already knew about NLM and NIH resources.

Jacqueline decided to look at the question from a different angle. Could they consider, instead, the networking potential of their exhibit? The NN/LM runs on relationships between regional medical library staff and other librarians in their respective regions. Possibly the booth’s value was that it provided an opportunity for the GMR staff to meet with librarians from both long-standing and potential member organizations of the GMR.

Collecting Feedback

Jacqueline decided to ask two simple evaluation questions.  First, did existing GMR users stop by the exhibit booth to visit with the GMR staff at the chapter meeting booth?  Second, did the booth provide the GMR staff with opportunities to meet librarians who were unaware of the NN/LM? In a nutshell, the questions focused on the booth’s potential to promote active participation in the NN/LM. This was a valid goal for an exhibit targeting this particular audience, where the GMR could find partners to support the network’s mission of promoting NLM resources.

She worked with the OERC to develop a point-of-contact questionnaire that she administered to visitors using an iPad. Her questionnaire had five items that people responded to via touch screen.  She chose the app Quick Tap Survey because it produced an attractive questionnaire, data could be collected without an Internet connection, and she could purchase a one-month subscription for the software.  The app also has a feature that allows the administrator to randomly pull a name for a door prize. Jacqueline used this feature to give away an NLM portfolio that was prominently displayed on the exhibit table. (Participation was voluntary, and the personally identifiable information was deleted after the drawing.)

Jacqueline stood in front of the booth to attract visitors, a practice she uses at all exhibits. She did not find that the questionnaire created any barriers to holding conversations with visitors. Quite the contrary, many were intrigued with the technology. Almost no one turned down her request to complete the form. Of the 120 conference attendees (the count reported by the Midwest MLA chapter), 38 (32%) visited the GMR booth and virtually all agreed to complete the questionnaire.

What Did GMR Learn?

 Jacqueline learned that 50% of the visitors came to the booth specifically to visit with GMR staff, while 26% came to get NLM resources.  This confirmed that the visits were more related to networking than information-seeking about NLM or NIH resources. She also learned that more than half were return visitors who had visited at past conferences, while 46% had never stopped by the booth before.  It appeared that the booth served equally as a way for GMR staff to talk with existing users and to meet potential new ones. Those who were return visitors also were the more likely users of the GMR: 68% said that the GMR was the first place they would seek answers to questions they had about NLM or NIH resources. (Although one added that she would first look online, then contact them if she couldn’t find the answer on her own.)  In contrast, 56% of new booth visitors said they usually sought help from a friend or colleague. Only 26% would contact the GMR. Findings do not indicate that exhibits cause librarians to become more involved with GMR. However, when GMR offers opportunities for face-to-face interactions, their users take advantage of it.

Visitors also got an opportunity to voice their opinion about the continuation of GMR exhibits at chapter meetings. There was fairly universal agreement: 92% said they thought the GMR should continue. The other 8% said they weren’t sure, but no one said GMR should stop.

Lessons learned

Jacqueline found it was easy to get people to take her questionnaire, particularly with a smooth application like Quick Tap Survey. She also learned that, regardless of the care she took in developing her questions, she still had at least one item that could have been worded better. However, tweaks can easily be implemented for future exhibits.

Overall, she said this assessment project added depth to the typical booth assessments that GMR typically conducts. Previous assessments focused on describing booth traffic, such as number of visitors, staff hours in booth, or number of promotional materials distributed. This project described the actual visitors and what they got out of the exhibit.

 Prologue: Why The OERC Loves This Project

We love this project because Jacqueline thought carefully about the outcomes of exhibiting to this particular audience and designed her questionnaire accordingly.  She recognizes that exhibits at chapter meetings are a specific type of event. The goals of NN/LM exhibits at other types of conferences are different, so the questionnaires would have to be adapted for those goals.

We also love this project because it shows that you can assess exhibits. Back in the day, point-of-contact assessment required paper-and-pencil methods.  It was a data collection approach that seemed likely to be self-defeating. Visitors would cut a wide path to avoid requests to fill out a form. Now that we have the technology (tablets and easy-to-use apps) that makes the task less daunting, the OERC has been promoting the idea of exhibit assessment.  Jacqueline’s project is proof that it can be done!

The OERC Blog – Moving Forward

Friday, January 8th, 2016

turtle climbing up staircase

Since last week’s message the OERC has been looking at some additional data about the blog in order to update our online communications plan going forward. The earlier OERC strategy had been to use social media to increase the use of evaluation resources, the OERC’s educational services, and the OERC’s coaching services. These continue to be the goals of the OERC’s plan. However, due to the following pieces of information, a new strategy has emerged.

  • The OERC Blog is increasing in popularity. As reported last week, more people find it, share it with their regions, and engage with it by clicking on the links than ever before.
  • The blog always has new content and is time-intensive to create: it takes approximately 6 person-hours each week to write and publish new content.
  • Although the OERC does not have a Facebook page, and the OERC Twitter account @nnlmOERC has been used primarily promote the blog, still Facebook refers more people to the blog than come from Twitter (this was kind of a shocker for us!)

We feel that that the OERC Blog, based on the results described in last week’s post, has become one of the most successful products of the OERC. The blog has become a source of educational content, and is itself an evaluation resource in need of promotion. Because of this, our Online Communications Plan going forward has a special focus on promoting the blog. Here are some of the new process goals for this purpose.

  • Facebook: The OERC will create a Facebook page that will promote the blog, link to other online evaluation resources, and show photos of what the OERC team is up to.
  • Twitter: Karen and Cindy will post at least one additional post per week on Twitter to increase the Twitter presence. Included will be retweets to build social capital which may lead to more retweets of our blog tweets (here is an interesting dissertation by Thomas Plotkowiak explaining this).
  • Training: We will make a point of promoting the blog during our in-person classes and webinars. For example, we may refer people to articles in our blog that supplement the content in the training sessions.
  • Email: The blog URL will be added to Karen and Cindy’s email signatures

So, what kinds of things will we measure? Naturally we want process measurements (showing that things are working the way they should along the way) and outcome measurements (showing that we are meeting our goals).

Here are our process goals, which are the activities we are committing to this year:

  • 52 blog posts a year
  • 3 tweets a week
  • Minimum of 1 Facebook post a week
  • Blog information added to class resource information and email signatures

In the short-term, we hope to see people interacting with our social media posts. So we are hoping to see increases on the following measures of our short-term outcomes:

  • # of Twitter retweets, likes and messages
  • # of Facebook likes, comments, and shares
  • # of new followers on Twitter and Facebook

We hope that the increased interaction with our Facebook and Twitter posts will lead more readers to our blog. So we will be monitoring increases on the following long-term outcome measures:

  • # of blog visitors per month
  • # of blog average views per day
  • # of blog link “click-throughs” to measure engagement with the blog articles
  • # of people who register for weekly blog updates and add the OERC Blog to their blog feeds
  • # of times blog posts are re-posted in Regional Medical Library blogs and newsletters.

This is our strategy for increasing use of our blog content. We will keep you updated and share tools that we develop to track these outcomes.

References

Plotkowiak, Thomas. “The Influence of Social Capital on Information Diffusion in Twitter’s Interest-Based Social Networks.” Diss. University of St.Gallen, 2014. Web. 8 Jan. 2016.

An OERC Resolution Realized

Friday, January 1st, 2016

Happy new year greeting card or poster design with colorful triangle 2015 shape and vintage label illustration. EPS10 vector.

Happy New Year, readers.  We decided to start 2016 on a high note with the OERC Blog’s Annual Report.

Wait, don’t leave! 

I promise, there’s something for everyone at the end of this post. There are links to the most popular evaluation tools and resources from our blog entries this year.  We judged popularity using our “clickthrough” statistics, showing the links that were most likely to be clicked by our readers to further investigate the resources featured in our blog posts.

(If you want, you can skip to the bottom now. We’ll never know.)

But first, we are going to take a moment to describe how far our blog has come since its inception in 2006.  That’s right, August 2016 is the 10-year anniversary of the OERC blog. I’ve been a contributing blogger for almost a decade. Karen, the newcomer, started contributing in February 2015, the month she joined the OERC. Her entries are quite popular: Most of the top 10 clickthroughs below were presented in Karen’s posts.

For most of the blog’s history, OERC staff posted 12-16 times per year (about 1 per month).  However, in January 2014, the OERC staff committed to increasing our blog activity. That year, we managed a little better than three posts per month.  This year, we finally met our goal of once-per-week posts.  We had 52 entries between January 1 and December 31, 2015.

Of course, writing blog posts is one thing. Writing blog posts that people read is another. Our first indication that our blog was gaining readership was through the OERC’s appreciative inquiry interviews conducted in late 2014. (See our blog post on October 31, 2014 for a description of this project)  Many of the interviewees mentioned the blog as one of their favorite OERC services.

Now, we have quantitative evidence of a growing readership: our end-of-year site statistics. Our stats only go back to June 2014. Even with that limited timeline, you can see substantial growth. Our peak month in 2014 had 241 total views.  In 2015, our peak month had more than double the traffic, with 508 views.  In 2014, we had an average of 7 views per day.  In 2015, our average was 12 views per day.

Two graphs showing increased readership for the OERC Blog. One shows increasing monthly views (June 2014=41; December 2015=452) Line chart 2 shows increasing average daily views (June 2014=7; December 2015=12)

So thank you, readers.  You are behind those numbers. In the coming year, we resolve to continue our weekly posts on a variety of evaluation topics.

And now, as promised, here are the top 10 “clickthrough” URLS from last year.  If you missed any of the trending evaluation resources from our blog, here’s your chance to catch up.

 

 

 

 

Simply Elegant Evaluation: Appreciative Inquiry at NN/LM MAR

Friday, August 28th, 2015

KF brochure

Kate Flewelling is the Outreach Coordinator for the National Network of Libraries of Medicine’s Middle Atlantic Region (NN/LM MAR), which is located at the University of Pittsburgh Health Sciences Library.  For those unfamiliar with the NN/LM, libraries and organizations join the network to help promote health information access and use.  The program is funded by the NIH National Library of Medicine, and NN/LM MAR is one of eight regional medical libraries that coordinate a region of the network. These eight health sciences libraries partner with member organizations and sometimes fund their health information outreach activities.

After attending an NN/LM Outreach Evaluation Resource Center training session, Kate was inspired to conduct an Appreciative Inquiry (AI) evaluation project to explore how NN/LM MAR could attract health professionals into the NN/LM and provide them with useful services.  In March 2015, she conducted eight 30-minute interviews with health professionals. She recruited interviewees from among health professionals who served on a special NN/LM MAR advisory committee or represented health organizations that received funding from NN/LM MAR. She chose interviewees from three states where NN/LM MAR works (Pennsylvania, New York, and Delaware), with representation from organizations in both rural and urban areas. Her interview guide was modeled after one presented in an OERC blog post.

Kate agreed to talk with the OERC to share her experience using Appreciative Inquiry.

OERC: What motivated you to do an Appreciative Inquiry project?

Kate:I wanted to see why health professionals got involved with us and have been so committed to us. We wanted to come up with selling points to tell other potential health professional members why they should join our network.”

Kate also chose an AI approach because she wanted stories, not numbers.  She was working with a small but diverse group of representatives, so interviews seemed to be a better approach than surveys for getting unique perspectives. She also believed an AI assessment was simple enough to be completed in about a week. In fact, she completed all eight interviews in eight days.

Slide1

OERC: Did you believe you got responses that had an overly positive bias?

Kate: “I only asked people who loved us, but they also know us. So they have an idea of what we’re doing and a much broader understanding of what we do with outreach because they hear about the whole outreach program. But I got really good feedback. Not criticism, but stuff we could do to improve our services.”

In AI, it is not unusual to interview an organization’s champions, because they often can provide the most informed advice about improving a program. Kate understood that her interviewees had favorable opinions about NN/LM MAR, but she said her interviews still identified holes in their outreach efforts to health professionals. They provided good advice on how to attract other health professionals to the network.

OERC: What did you want to learn from the study?  

Kate: “The experience was great! It gave me good ideas.  I realized we weren’t using them [health professional colleagues] as much as we could. They told me ‘I will pass on whatever you need me to pass on.’  It gave me great ideas for how to use them, use their connections and develop target outreach materials and messages for special audiences.’”   

She realized that NN/LM MAR could send regular postings to the health representatives, just as they do to health sciences librarians.  The postings just needed to contain more context so that they were targeting a public health or clinical audience.

Kate: “The project also made me realize how far [NN/LM MAR’s] reach has gone in the past four years…It felt like, during our first and second year, throwing spaghetti on the wall to see if it was working with health professionals. But we were trying to make the connections. Now we know, for our most engaged people, what they value about their relationship with us.”

Most of the staff joined the NN/LM MAR in 2011, when University of Pittsburgh Health Sciences Library was awarded the contract to coordinate the Middle Atlantic Region. So the AI project was a member check with their public health and clinical health partners, to see how well the relatively new program was meeting their needs.  Before the AI project, Kate said she knew what NN/LM MAR staff was getting from their relationships with health professionals.  Afterwards, she understood what the health professionals were getting from NN/LM MAR.

OERC: How did you use the information you gathered?

Kate: “Just starting to talk to people at exhibits, I have a sense of what’s going to grab them.”

Kate developed a brochure targeted to health professionals with a summary of NN/LM selling points on the front (gleaned from her AI interviews) and resources of interest on the back. She plans to share the brochure with the other eight NN/LM regional medical libraries. She also believes the NN/LM MAR staff will tap into this information in the future when they plan programs for health professionals.

Slide2

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.