Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘Bloggers’ Bookshelf’ Category

Hungry for More Evaluation Ideas? How About a Tip a Day?

Friday, April 28th, 2017

Sometimes people ask us where we get ideas for our posts.  I’m going to tell you one of our big secrets – when we can’t think of something we check out the American Evaluation Association’s daily blog AEA 365 | A Tip-A-Day by and for Evaluators.  Seriously, every single day some evaluator posts a tip for other evaluators.

In no way do we rewrite any of their posts, but we do in fact scrounge for ideas.  I thought you might want to scrounge for your own ideas by looking at this great blog now and then.

Just this week, there’s a post about about using very short surveys throughout a project so that you can make changes to improve your project as you go: Using pulse surveys to get rapid actionable feedback from teachers during a professional development experience by Valerie Futch Ehrlich  Along with describing how they did it, the post also recommends a survey tool called Waggl that allows people to vote on each others’ feedback, so the best ideas float to the top. Cool, right?

Here’s another post about how you can use tables to explain complicated ideas to potential funders in grant applications: Using Tables Effectively in Grant Proposals by Kate LaVelle and Judith Rhodes. 

Happy scrounging!

 

Beyond the Memes: Evaluating Your Social Media Strategy – Part 2

Friday, January 20th, 2017

In my last post, I wrote about how to create social media outcomes for your organization. This week, we will take a look at writing objectives for your outcomes using the SMART method.

Though objectives and outcomes sound like the same thing, they are two different concepts in your evaluation plan – outcomes are the big ideas, while objectives relate to the specifics. Read Karen’s post to find out more about what outcomes and objectives are.

In the book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, they talk a lot about SMART objectives. We have not covered these types of objectives on the blog, so I thought this would be a good time to introduce this type of objective building. According to the book, a SMART objective is “specific, measurable, attainable, realistic, and timely” (Kanter and Paine 47). There are many variations on this definition, so we will use my favorite: Specific, Measurable, Attainable, Relevant, and Timely.

Specific: Leave the big picture for your outcomes. Use the 5 W’s (who, what, when, where, and why) to help craft this portion
Measurable: If you can’t measure it, how will you know you’ve actually achieved what you set out to do?
Attainable: Don’t make you objectives impossible. It’s not productive (or fun) to create objectives that you know you cannot reach. Understand what your community needs, and involve stakeholders.
Relevant: Is your community on Twitter? Create a Twitter account. Do they avoid Twitter? Don’t make a Twitter account. Use the tools that are relevant to the community that you serve.
Timely: Set a time frame for your objectives and outcomes, or your project might take too long for it to be relevant to your community. Time is also money, so create a deadline for your project so that you do not waste resources on a lackluster project.

As an example, let’s return to NEO’s favorite hypothetical town of Sunnydale to see how they added social media objectives into their Dusk to Dawn program. To refresh your memory, read this post from last September about Sunnydale’s Evaluation Plan.

Christopher Walken Fever Meme with the text 'Well, guess what! I’ve got a fever / and the only prescription is more hashtags'

Sunnydale librarians know that their vampire population uses Twitter on a daily basis for many reasons – meeting new vampires, suggesting favorite vampire friendly night clubs, and even engaging the library with general reference questions. Librarians came up with the idea to use the hashtag #dusk2dawn in all of their promotional materials about the health program Dusk to Dawn. Their thinking was it would help increase awareness of their objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

With that in mind, let’s make a SMART objective for this hashtag’s usage:

Specific
We will plug in what we have so far into the Specific section:

Vampires (who) in Sunnydale (where) will show an increase in awareness of health-related events hosted by the library (what) by retweeting the hashtag #dusk2dawn (why) for the duration of the Dusk to Dawn program (when).

Measurable
Measurable is probably the hardest part. What kind of metrics will Sunnydale librarians use to measure hashtag usage? How will they do it?

The social media librarian will manually monitor the hashtag’s usage by setting up an alert for its usage on TweetDeck. Each time the hashtag is used by a non-librarian in reference to the Sunnydale Library, the librarian will copy the tweet’s content to a spreadsheet, adding signifiers if it is a positive or negative tweet.

Attainable
Can our objective be reached? What is it about vampires in Sunnydale that makes this hashtag monitoring possible?

We know from polling and experience that our community likes using Twitter – specifically, they regularly engage with us on this platform. Having a dedicated hashtag for our overall program is a natural progression for us and our community.

Relevant
How does the hashtag #dusk2dawn contribute to the overall Dusk to Dawn mission?

An increase in usage of the hashtag #dusk2dawn will show that our community is actively talking about, hopefully in a positive way. This should increase awareness of our program’s objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

Timely
How long should it take for the vampires to increase their awareness of our program’s objectives?

There should be an upward trend in awareness over the course of the program. We have 7 months before we are reevaluating the whole Dusk to Dawn program, so we will set 7 months as our deadline for increased hashtag usage.

SMART!
Now, we put it all together to get:

Vampires in Sunnyvale will show an increase in awareness of health-related events hosted by the library, indicated by a 15% increase of the hashtag #dusk2dawn by Sunnydale vampires for the duration of the Dusk to Dawn program.

Success Baby

Though this objective is SMART, it is certainly will not work in every library. Perhaps the community your library serves does not use Twitter to connect with the library, or you do not have enough people on staff to monitor the hashtag’s usage. If you make a SMART objective that will be relevant to your community, it will have a better chance to succeed.

Here at NEO, we usually do not use SMART objectives method, but rather Measurable Outcome Objectives. Step 3 on the Evaluation Resources page points to many different resources on our website that talk about Measurable Objectives. Try both out, and see what works for your organization.

We will be taking a break from social media evaluation and goal setting for a few weeks. Next time we talk about social media, we will show our very own social media evaluation plan!

Find more information about SMART objectives here:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Image credits: Christopher Walken Fever Meme made by Kalyna Durbak on ImgFlip.com. Success Kid meme is from Know Your Meme.

Beyond the Memes: Evaluating Your Social Media Strategy – Part 1

Friday, January 13th, 2017

Welcome to our new NEO blogger, Kalyna Durbak.  Her first post addresses a topic that is a concern to many of us, evaluating our social media!


By Kalyna Durbak, Program Assistant, NNLM Evaluation Office (NEO)

Have you ever wondered if your library’s Facebook page was worth the time and effort? I think about social media a lot, but then again I’ve been using Facebook daily for over 10 years. The book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, can help your library or organization figure out how to measure the impact of your social media campaigns have on the world.

Not all of us work for a nonprofit, but I feel many organizations share similar constraints with nonprofits – like not being able to afford to hire a firm to develop and manage the social media accounts. It’s easy to think that social media is easy to do because we all manage our personal profiles. Once you start managing accounts that belong to an organization, it gets hard. What do you post? What can you post? How many likes can I collect?

One does not simply post memes on Facebook

Before we get into any measurement, I want to briefly write about why social media outcomes are important to have, and why they should be measured. A library should not create a Facebook page simply to collect likes, or a Twitter page to gather followers. As my husband would say, that’s simply “scoring Internet points.” Internet points make you feel good inside, but do not impact the world around you. The real magic in using social media comes from creating a community around your organization that is willing to show up and help out when you ask.

A library should create a social media page in order for something to happen in the real world – an outcome. Figuring out why you need a social media account will help your library manage its various accounts more efficiently, and in the end measure the successes and failures of your social media campaigns. If you need more convincing, read Cindy’s post “Steering by Outcomes: Begin with the End in Mind.” For help on determining your outcomes, I suggest reading Karen’s blog post “Developing Program Outcomes using the Kirkpatrick Model – with Vampires.”

What are some reasons for using social media in your library? Maybe you will have an online campaign to promote digital assets, or perhaps you will add a social media component to a program that already exists in your library. Whatever they are, any social media activity you do should support an outcome. A few outcomes I can think of are:

  • Increased awareness of library programs
  • New partnerships found for future collaborative efforts
  • Improved perceptions about the organization
  • Increase in library foundation’s donor base

None of the outcomes specifically mention Facebook, Twitter, or any other social media platforms. That’s because outcomes outline the big picture – it’s what you want to happen after completing your project. In the above examples, a library wants the donor base to be increased, or the library wants increased awareness of library programs. It’s the ideal world your library or organization wants to exist in. Facebook and Twitter can help achieve these outcomes, but the number of retweets you get is not an outcome.

To make that ideal future a reality, you need to create objectives. Objectives are the signposts that will indicate whether you are successful in reaching your outcome. Next week, we will craft social media oriented objectives for a library in our favorite hypothetical town of Sunnydale. Catch up on Sunnydale with these posts:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Our Favorite Evaluation Blogs

Friday, April 15th, 2016

successful business woman on a laptop

We really don’t want you to stop reading our blog!  But April is a really busy month for us, so this week we’ll make sure you get your evaluation buzz by letting you know of some other great evaluation blogs.

AEA365 – This is the blog of the American Evaluation Association.  This blog shares hot tips, cool tricks, rad resources, and lessons learned by different evaluators every single day!

Better Evaluation Blog – Better Evaluation is an international collaboration to improve evaluation by sharing information about evaluation methods, processes and approaches. The blog has posts that provide new perspectives  about particular issues in evaluation.

EvaluATE – EvaluATE is the evaluation resource center for the National Science Foundation’s Advanced Technological Education program. Their blog has lessons learned, tips, or techniques on evaluation management, proposal development, evaluation design, data collection and analysis, reporting, and more.

Evergreen Data – Stephanie Evergreen writes a blog about data visualization.  For the record, she has written the book(s) on data visualization, Effective Data Visualization and Presenting Data Effectively.

Visual Brains – Sara Vaca writes about new techniques and ways of visualizing data, information, and figures to communicate evaluation findings and to improve evaluation use, but also for use in other stages such as planning and analyzing.

Book review: Focus Groups: A Practical Guide for Applied Research. (4th edition)

Tuesday, August 30th, 2011

I recently purchased a copy of “Focus Groups: A Practical Guide for Applied Research” by Richard Krueger and Mary Anne Casey. Krueger, professor emeritus at University of Minnesota, has written some of the classic books on focus group research and his co-author has conducted focus groups for government agencies and nonprofits. The experience of these two authors shines through in the pages of this well-organized, thorough text, which has a lot to recommend it:

  • The operative term in the title is “applied research.” The authors talk about the purpose of the study being the “guiding star” for selecting participants, writing the question guide, deciding on moderators, and analyzing and reporting findings.
  • The content is full of nuts-and-bolts suggestions, including a very practical chapter about Internet and telephone interviews
  • There is an interesting chapter presenting four different approaches to focus group research: marketing research; academic research; public/nonprofit; and participatory. The chapter summarizes the evolution of the approaches and compares them in a table that will allow the readers to choose the approach that best fits the circumstances of their studies.  This chapter explains why evaluators have different takes on how to conduct focus groups.
  • There is a nice chapter on analyzing focus group data. It can be difficult to find step-by-step descriptions of how to analyze qualitative data, so this chapter alone is a reason to read this book. (You could generalize the process to analyzing other forms of qualitative evaluation data.)
  • The final chapter provides you with responses to challenging questions about the quality of your focus group research. For example, what do you say if someone asks “Is this scientific research?” and “how do you know your findings aren’t just your subjective opinions?” Along with suggesting responses, the authors provide their own analysis of why such questions are often posed and the assumptions lurking behind them. This section will help you defend your project and your conclusions. (It would be most helpful to read this chapter before you design your project because it helps you understand the standards for a defensible project.)

I recommend this book to anyone planning to run focus groups. I have conducted my fair share of discussions, but I learned new tips to use in my next project.

Reference: Krueger RA. Casey MA. Focus groups. A practical guide for applied research. 4th ed.Thousand Oaks, CA: Sage, 2009.

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.