Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for January, 2017

Failure IS an Option: Measuring and Reporting It

Friday, January 27th, 2017

Back to Square One signpost

Failure.  We all know it’s good for us.  We learn from failure, right? In Batman Begins, Bruce Wayne’s dad says “Why do we fall, Bruce? So we can learn to pick ourselves up.”  But sometimes failure, like falling, isn’t much fun (although, just like falling, sometimes it is fun for the other people around you). Sometimes in our jobs we have to report our failures to someone. And sometimes the politics of our jobs makes reporting failure a definite problem.

In the NEO office we like to start our meetings by reporting a recent failure. I think it’s a fun thing to do because I think my failures are usually pretty funny.  But Cindy has us do it from a higher motivation than getting people to laugh.  Taking risks is about being willing to fail. Sara Blakely, the founder and CEO of Spanx, grew up reporting failure every day at the dinner table: https://vimeo.com/175524001  In this video she says that “failure to me became not trying, versus the outcome.”

Why failure matters in evaluation

In general we are all really good at measuring our process (the activities we do) and not so good at measuring outcomes (the things we want to see happen because of our activities).  This is because we have a lot of control over whether our activities are done correctly, and very little control over the outcomes.  We want to measure something that shows that we did a great job, and we don’t want to measure something that might make us look bad. That’s why we find it preferable to measure something we have control over. It can look like we failed if we didn’t get the results we wanted, even if the work at our end was brilliant.

sad businesswoman

But of course outcomes are what we really care about (Steering by Outcomes: Begin with the End in Mind).  They are the “what for?” of what we do.  What if you evaluated the outcomes of some training sessions that you taught and you found out that no one used the skill that you taught them.  That would be sad and it might look like you wasted time and resources.  But on the other hand, what if you don’t measure whether or not anyone ever uses what you taught them, and you just keep teaching the classes and reporting successful classes, never finding out that people aren’t using what you taught them.  Wouldn’t that be the real waste of resources?

So how do you report failure?

I think getting over our fear of failure has to do with learning how to report failure so it doesn’t look like, well, failure.  The key is to stay focused on the end goal: we all really want to know the answer to the question “are we making a difference?”  If we stay focused on that question, then we need to figure out what indicators we can measure to find the answer. If the answer is “no, we didn’t make a difference” then how can we report that in a way that shows we’ve learned how to make the answer “yes?” How can we think about failure so it’s about “learning to pick ourselves up?” or better yet, contributing to your organization’s mission?

One way is to measure outcomes early and often. If you wait until the end of your project to measure your outcomes, you can’t adjust your project to enhance the possibilities of success.  If you know early on that your short-term outcomes are not coming out the way you hope, you can change what you’re doing.  So when you do your final report, you aren’t reporting failure, you’re reporting lessons learned, flexibility and ultimately success.

Here’s an example

Let’s say you’re teaching a series of classes to physical therapists on using PubMed Health so they can identify the most effective therapy for their patients.  At the end of the class you have the students complete a course evaluation, in which they give high scores to the class and the teachers.  If you are evaluating outcomes early, you might add a question like: “Do you think you will use PubMed Health in the next month?”  This is an early outcome question.   If most of them say “no” to this question, you will know quickly that if you don’t change something about what you’re doing in future classes, it is unlikely that a follow-up survey two months later will show that they had used PubMed Health.  Maybe you aren’t giving examples that apply to these particular students. Maybe these students aren’t in the position to make decisions about effective therapies. You have an opportunity to talk to some of the students and find out what you can change so your project is successful.

Complete Failure

You’ve tried everything, but you still don’t have the results you wanted to see.  The good news is, if you’ve been collecting your process and outcomes data, you have a lot of information about why things didn’t turn out as hoped and what can be done differently. Reporting that information is kind of like that commercial about how to come in late to a meeting. If you bring the food, you’re not the person who came late, you’re the person who brought breakfast.  If you report that you did a big project that didn’t work, you’re reporting failure.  If you report that things didn’t work out the way you hoped, but you have data-based suggestions for a better use of organizational resources that meet the same goal–then you’re the person who is working for positive change that supports the organization, and have metaphorically brought the breakfast. Who doesn’t love that person?

 

Beyond the Memes: Evaluating Your Social Media Strategy – Part 2

Friday, January 20th, 2017

In my last post, I wrote about how to create social media outcomes for your organization. This week, we will take a look at writing objectives for your outcomes using the SMART method.

Though objectives and outcomes sound like the same thing, they are two different concepts in your evaluation plan – outcomes are the big ideas, while objectives relate to the specifics. Read Karen’s post to find out more about what outcomes and objectives are.

In the book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, they talk a lot about SMART objectives. We have not covered these types of objectives on the blog, so I thought this would be a good time to introduce this type of objective building. According to the book, a SMART objective is “specific, measurable, attainable, realistic, and timely” (Kanter and Paine 47). There are many variations on this definition, so we will use my favorite: Specific, Measurable, Attainable, Relevant, and Timely.

Specific: Leave the big picture for your outcomes. Use the 5 W’s (who, what, when, where, and why) to help craft this portion
Measurable: If you can’t measure it, how will you know you’ve actually achieved what you set out to do?
Attainable: Don’t make you objectives impossible. It’s not productive (or fun) to create objectives that you know you cannot reach. Understand what your community needs, and involve stakeholders.
Relevant: Is your community on Twitter? Create a Twitter account. Do they avoid Twitter? Don’t make a Twitter account. Use the tools that are relevant to the community that you serve.
Timely: Set a time frame for your objectives and outcomes, or your project might take too long for it to be relevant to your community. Time is also money, so create a deadline for your project so that you do not waste resources on a lackluster project.

As an example, let’s return to NEO’s favorite hypothetical town of Sunnydale to see how they added social media objectives into their Dusk to Dawn program. To refresh your memory, read this post from last September about Sunnydale’s Evaluation Plan.

Christopher Walken Fever Meme with the text 'Well, guess what! I’ve got a fever / and the only prescription is more hashtags'

Sunnydale librarians know that their vampire population uses Twitter on a daily basis for many reasons – meeting new vampires, suggesting favorite vampire friendly night clubs, and even engaging the library with general reference questions. Librarians came up with the idea to use the hashtag #dusk2dawn in all of their promotional materials about the health program Dusk to Dawn. Their thinking was it would help increase awareness of their objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

With that in mind, let’s make a SMART objective for this hashtag’s usage:

Specific
We will plug in what we have so far into the Specific section:

Vampires (who) in Sunnydale (where) will show an increase in awareness of health-related events hosted by the library (what) by retweeting the hashtag #dusk2dawn (why) for the duration of the Dusk to Dawn program (when).

Measurable
Measurable is probably the hardest part. What kind of metrics will Sunnydale librarians use to measure hashtag usage? How will they do it?

The social media librarian will manually monitor the hashtag’s usage by setting up an alert for its usage on TweetDeck. Each time the hashtag is used by a non-librarian in reference to the Sunnydale Library, the librarian will copy the tweet’s content to a spreadsheet, adding signifiers if it is a positive or negative tweet.

Attainable
Can our objective be reached? What is it about vampires in Sunnydale that makes this hashtag monitoring possible?

We know from polling and experience that our community likes using Twitter – specifically, they regularly engage with us on this platform. Having a dedicated hashtag for our overall program is a natural progression for us and our community.

Relevant
How does the hashtag #dusk2dawn contribute to the overall Dusk to Dawn mission?

An increase in usage of the hashtag #dusk2dawn will show that our community is actively talking about, hopefully in a positive way. This should increase awareness of our program’s objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

Timely
How long should it take for the vampires to increase their awareness of our program’s objectives?

There should be an upward trend in awareness over the course of the program. We have 7 months before we are reevaluating the whole Dusk to Dawn program, so we will set 7 months as our deadline for increased hashtag usage.

SMART!
Now, we put it all together to get:

Vampires in Sunnyvale will show an increase in awareness of health-related events hosted by the library, indicated by a 15% increase of the hashtag #dusk2dawn by Sunnydale vampires for the duration of the Dusk to Dawn program.

Success Baby

Though this objective is SMART, it is certainly will not work in every library. Perhaps the community your library serves does not use Twitter to connect with the library, or you do not have enough people on staff to monitor the hashtag’s usage. If you make a SMART objective that will be relevant to your community, it will have a better chance to succeed.

Here at NEO, we usually do not use SMART objectives method, but rather Measurable Outcome Objectives. Step 3 on the Evaluation Resources page points to many different resources on our website that talk about Measurable Objectives. Try both out, and see what works for your organization.

We will be taking a break from social media evaluation and goal setting for a few weeks. Next time we talk about social media, we will show our very own social media evaluation plan!

Find more information about SMART objectives here:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Image credits: Christopher Walken Fever Meme made by Kalyna Durbak on ImgFlip.com. Success Kid meme is from Know Your Meme.

Beyond the Memes: Evaluating Your Social Media Strategy – Part 1

Friday, January 13th, 2017

Welcome to our new NEO blogger, Kalyna Durbak.  Her first post addresses a topic that is a concern to many of us, evaluating our social media!


By Kalyna Durbak, Program Assistant, NNLM Evaluation Office (NEO)

Have you ever wondered if your library’s Facebook page was worth the time and effort? I think about social media a lot, but then again I’ve been using Facebook daily for over 10 years. The book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, can help your library or organization figure out how to measure the impact of your social media campaigns have on the world.

Not all of us work for a nonprofit, but I feel many organizations share similar constraints with nonprofits – like not being able to afford to hire a firm to develop and manage the social media accounts. It’s easy to think that social media is easy to do because we all manage our personal profiles. Once you start managing accounts that belong to an organization, it gets hard. What do you post? What can you post? How many likes can I collect?

One does not simply post memes on Facebook

Before we get into any measurement, I want to briefly write about why social media outcomes are important to have, and why they should be measured. A library should not create a Facebook page simply to collect likes, or a Twitter page to gather followers. As my husband would say, that’s simply “scoring Internet points.” Internet points make you feel good inside, but do not impact the world around you. The real magic in using social media comes from creating a community around your organization that is willing to show up and help out when you ask.

A library should create a social media page in order for something to happen in the real world – an outcome. Figuring out why you need a social media account will help your library manage its various accounts more efficiently, and in the end measure the successes and failures of your social media campaigns. If you need more convincing, read Cindy’s post “Steering by Outcomes: Begin with the End in Mind.” For help on determining your outcomes, I suggest reading Karen’s blog post “Developing Program Outcomes using the Kirkpatrick Model – with Vampires.”

What are some reasons for using social media in your library? Maybe you will have an online campaign to promote digital assets, or perhaps you will add a social media component to a program that already exists in your library. Whatever they are, any social media activity you do should support an outcome. A few outcomes I can think of are:

  • Increased awareness of library programs
  • New partnerships found for future collaborative efforts
  • Improved perceptions about the organization
  • Increase in library foundation’s donor base

None of the outcomes specifically mention Facebook, Twitter, or any other social media platforms. That’s because outcomes outline the big picture – it’s what you want to happen after completing your project. In the above examples, a library wants the donor base to be increased, or the library wants increased awareness of library programs. It’s the ideal world your library or organization wants to exist in. Facebook and Twitter can help achieve these outcomes, but the number of retweets you get is not an outcome.

To make that ideal future a reality, you need to create objectives. Objectives are the signposts that will indicate whether you are successful in reaching your outcome. Next week, we will craft social media oriented objectives for a library in our favorite hypothetical town of Sunnydale. Catch up on Sunnydale with these posts:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Make 2017 the “Yes-and” Year for Evaluation

Friday, January 6th, 2017

20117 (year) with fireworks and "Always Say Yes" written underneath

PSA: New Year’s resolutions are passé.

It’s out with dreary self-betterment goals involving celery and punishing exercise. Instead, the trendiest way to mark the New Year is to pick an inspirational word and make it your annual North Star for self-improvement.  In that spirit, I want to propose a word of the year for the NEO Shop Talk community.

That word is “Yes and.”

Now, some of you are wondering how I passed kindergarten with such poor counting skills. However, If you or anyone you know has taken training in Improv theatre, you know “Yes and” is a word, or, more specifically, a verb.

Improv is a type of theatre in which a team of actors make up scenes on the spot, usually from audience suggestions.  Because performances are unscripted, improv actors train rather than rehearse. Training is built around commonly accepted “rules,” and, arguably, the best known rule of Improv is “Always say Yes and…”   That means that you accept any scene idea your teammate presents and add something to make that idea better. Once novice improvisers experience the upbeat emotional effect of this rule, they soon find themselves “yes-anding” in other parts of their lives. Some even preach about it to others (ahem).  Notice, by the way, I added a hyphen so we can all feel better about “yes-and” as the WORD of the year.

If you can “yes-and” evaluation requirements and responsibilities, it will put you on the road to mastering this rule. Let’s face it, the thought of evaluation does not generate an abundance of enthusiasm. Usually we do evaluation because someone else expects or requires us to do it: upper administration; accreditation boards; funding agencies. We only do evaluation when forced because it’s a lot of work. I compare evaluation to physical exercise. In theory, we know it’s good for us. In practice, we don’t have time for it. “Yes-anding” evaluation may not make you do more evaluation than you’re required to do.  It might, however, make your evaluation responsibilities more enjoyable or, at least, more meaningful to you personally.

For example, if you have to write a proposal for external funding, you often have to pull together assessment information to build a case for your proposed program. Does that mean you have to locate and synthesize lots of data from lots of sources? Yes, and you get to demonstrate all of the great things your library or organization has to offer. You also get to point out areas where you could provide even more awesome services if the funding agency gave you funding to meet your resource needs. (Here’s a NEO Shop Talk blog post on how to use SWOT analysis to synthesize needs assessment data.)

If your proposal includes outreach into a new community, you probably have to collect information from that community.   Do you have to find and conduct key informant interviews with representatives of the community?   Yes, and you also get to initiate relationships with influential community opinion leaders. Listening is a powerful way to build trust and rapport. If you have the opportunity to implement your program, these key informants will be powerful allies when you want to reach out to the broader community. (If you want some tips for finding key informants, check out this blog post.)

You can “yes-and” internally mandated evaluation as well. Your library or organization may require you to track data on an ongoing basis or to submit regular reports. To do this well, do you have to document your daily work, such as keep track of details surrounding reference services, workshop attendance, or facility usage? Yes and, you also get to create a database of motivational information to inspire you and fellow co-workers working on the same objectives and goals. Compile that information monthly or quarterly, and pass it around at staff meetingd.  Celebrate what you’re accomplishing. Figure out where effort is lagging and commit to bolstering activities in that area. Then celebrate your team’s astute use of data for making good program decisions. Yes, and, at reporting time, be sure you present your data so that your upper-level stakeholders notices your hard work.

Maybe your department or office has to set and assess annual objectives or outcomes. Do you have to collect and report data to show program results? Yes, and you also get to demonstrate your value to the organization. Just be sure you don’t hide your candle deep in some organizational online reporting system.  Annual reports are seldom page-turners. Find more compelling ways to communicate your success and contributions to upper administrators and influential users. For some ideas, you might want to check out some of NEO Shop Talk posts on reporting and data visualization.

Another rule of Improv is “There are no mistakes, only opportunities.”  Let’s paraphrase that to “there are no evaluation requirements, only opportunities.”  Here’s to making 2017 the year of “yes-anding” evaluation.

As a post-script, I want to share a NEO Shop Talk success with our readers. Did we post weekly blog entries in 2016? Yes, and you showed up more than ever. NEO Shop Talk visits increased 71% in 2016. Each month, we averaged 259 more visits compared to the same month in the previous year.  Our peak month was February, with 892 visits! Thank you, readers. Please come back and bring your friends!

slide1

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.