Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

ABP: Always Be Pilot-testing (some experiences with questionnaire design)

Cover of NEO's Booklet 3 on collecting and analyzing evaluation data

This week I have been working on a questionnaire for the Texas Library Association (TLA) on the cultural climate of TLA.  Having just gone through this process, I will tell you that NEO’s Booklet #3: Collecting and Analyzing Evaluation Data has really useful tips on how to write questionnaires (p. 3-7).  I thought it might be good to talk about some of the tips that had turned out particularly useful for this project, but the theme of all of these is “always be pilot-testing.”

NEO’s #1 Tip: Always pilot test!

This questionnaire is still being pilot tested. So far I have thought the questionnaire was perfect at least 10 times, and we are still finding out about important changes from people who are pilot testing it for our committee.  One important part of this tip is to include stakeholders in the pilot testing.  Stakeholders have points of view that may not be included in the points of view of the people creating the survey.  After we have what we think is a final version, our questionnaire will be tested by the TLA Executive Board.  While this process sounds exhausting, every single change that has been made (to the questionnaire that I thought was finished) has fundamentally improved it.

There is a response for everyone who completes the question

Our questionnaire asks questions about openness and inclusiveness to people of diverse races, ethnicities, nationalities, ages, gender identities, sexual identities, cognitive and physical disabilities, perceived socioeconomic status, etc.  We are hoping to get personal opinions from all kinds of librarians who live all over Texas.  By definition this means that many of the questions are possibly sensitive, and may be hot button issues for some people.

In addition to wording the questions carefully, it’s important that every question has a response for everyone who completes the question. We would hate for someone not to find the response that best works for them, and then leave the questionnaire unanswered, or even worse get their feelings hurt or feel insulted. For example, we have a question about whether our respondents feel that their populations are represented in TLA’s different groups (membership, leadership, staff, etc). At first the answers were just “yes” or “no.”  But then (from responses in the pilot testing) we realized that a person may feel that they belong to more than one population. For example, what if someone was both Asian and had a physical disability.  Perhaps they feel that one group is well represented and the other group not represented at all.  How would they answer the question?  Without creating a complicated response, we decided to change our response options to “yes” “some are” and “no.”

“Don’t Know” or “Not Applicable”

In a similar vein, sometimes people do not know the answer to the question you are asking.  They can feel pressured to make a choice among questions rather than skip the question (and if they do skip the question, the data will not show why).  For example, we are asking a question about whether people feel that TLA is inclusive, exclusionary or neither.  Originally I thought those three choices covered all the bases.  But among discussions with Cindy (who was pilot testing the questionnaire), we realized that if someone simply didn’t know, they wouldn’t feel comfortable saying that TLA was neither inclusive nor exclusionary.  So we added a “Don’t know” option.

Speaking from experience, the most important thing is keeping an open mind. Remember that the people taking your questionnaire will be seeing it from different eyes than yours, and they are the ones you are hoping to get information from.  So, while I recommend following all the tips in Booklet 3, to get the best results, make sure that you test your questionnaire with a wide variety of people who represent those who will be taking it.

Got Documents? How to Do a Document Review

Are you an introvert?  Then I have an evaluation method for you: document review. You usually you can do this method from the comfort of your own office. No scary interactions with strangers.

Truth is, my use of existing data in evaluation seldom rises to the definition of true document review.  I usually read through relevant documents to understand a program’s history or context. However, a recent blog post by Linda Cabral in the AEA365 blog reminded me that document review is a real evaluation method that is conducted systematically. Cabral provide tips and a resource for doing document review correctly.  For today’s post, I decided to plan a document review that the NEO might conduct someday, describing how I would use Cabral’s guidelines. I also checked out the CDC’s Evaluation Brief, Data Collection Methods for Evaluation: Document Review, which Cabral recommended.

Here’s some project background. The NEO leads and supports evaluation efforts of the National Network of Libraries of Medicine (NNLM), which promotes access to and use of health information resources developed by the NIH’s National Library of Medicine. Eight health sciences libraries (called Regional Medical Libraries or RMLs) manage a program in which they provide modest amounts of funding to other organizations to conduct health information outreach in their regions. The organizations receiving these funds (known as subawards) write proposals that include brief descriptions (1-2 paragraphs) about their projects. These descriptions, along with other information about the subaward projects, are entered that is into the NLM’s Outreach Projects Database (OPD).

The OPD has a wealth of information, so I need an evaluation question to help me focus my document review. I settle on this question: How do our subawardees collaborate with other organizations to promote NLM products?  Partnerships and collaborations are a cornerstone of NNLM. They are the “network” in our name.  Yet simply listing the diverse types of organizations involved in our work does not satisfactorily capture the nature of our collaborations.  Possibly the subaward program descriptions in our OPD can add depth to our understanding of this aspect of the NNLM.

Now that I’ve identified my primary evaluation question, here’s how I would apply Cabral’s guidelines in the actual study.

Catalogue the information available to you:  For my project, I would first review the fields on the OPD’s data entry pages to see what information is entered for each project.  I obviously want to use the descriptive paragraphs. However, it helps to peruse the other project details. For example, it might be interesting to see if different types of organization (such as libraries and community-based organizations) form different types of collaborations. This step may cause me to add evaluation questions to my study.

I also would employ some type of sampling, because the OPD contains over 4500 project descriptions from as far back as 2001.  It is neither feasible nor necessary to review all of them.  There are many sampling choices, both random and purposeful. (Check out this article by Palinkas et al for purposeful sampling strategies.)  I‘m most interested in current award projects, so I likely would choose projects conducted in the past 2-3 years.

Develop a data collection form: A data collection form is the tool that allows you to record abstracted or summarized information from the full documents. Fortunately, the OPD system downloads data into an Excel-readable spreadsheet, which is the basis for my form. I would first delete columns in this spreadsheet that contain information not irrelevant to my study, such as mailing address and  phone numbers of the subaward contact person.

Get a co-evaluator: I would volunteer a NEO colleague to partner with me, to increase the objectivity of the analysis. Document review almost always involves coding of qualitative data.  If you use qualitative analysis for your study, a partner improves the trustworthiness of conclusions drawn from the data. If you are converting information into quantifiable (countable) data, a co-evaluator allows you to assess and correct human error in your coding process. If you do not have a partner for your entire project, try to find someone who can work with you on a subset of the data so you can calibrate your coding against someone else’s.

Ensure consistently among teammates involved in the analysis: “Abstracting data,” for my project, means identifying themes in the project descriptions.  Here’s a step-by-step description of the process I would use:

  • My partner and I would take a portion of the documents (15-20%) and both of us would read the same set of project descriptions. We would develop a list of themes that both of us believe are important to track for our study. Tracking means we would add columns to our data collection form/worksheet and note absence or presence of the themes in each project description.
  • We would then divide up the remaining program descriptions. I would code half of them and my partner would take the other half.
  • After reading 20% of the remaining documents, we would check in with each other to see if important new themes have emerged that we want to track. If so, we would add columns on our data collection document. (We would also check that first 15-20% of project descriptions for presence of these new themes.)
  • When all program descriptions are coded, we would sort our data collection form so we could explore patterns or commonalities among programs that share common themes.

For a more explicit description of coding qualitative data, check out the NEO publication Collecting and Analyzing Evaluation Data. The qualitative analysis methods described starting on page 25 can be applied in qualitative document review.

So, got documents? Now you know how to use them to assess your programs.

NEO Announcement! Home Grown Tools and Resources

Red Toolbox with ToolsSince NEO (formerly OERC) was formed, we’ve created a lot of material – four evaluation guides, a 4-step guide to creating an evaluation plan, hosted in-person classes and webinars, and of course, written in this very blog! All of the guides, classes, and blogs come with a lot of materials, including tip sheets, example plans, and resource lists. In order to get to all of these resources though, you had to go through each section of the website and search for them, or attend one of our in person classes. That all changed today.

Starting now, NEO will be posting its own tip sheets, evaluation examples, and more of our favorite links on the Tools and Resources page. Our first addition is our brand new tip sheet, “Maximizing Response Rate to Questionnaires,” which can be found under the Data Collection tab. We also provided links to some of our blog posts in each tab, making them easier to find. Look for more additions to the Tools and Resources page in upcoming months.

Do you have a suggestion for a tip sheet? Comment below – you might see it in the future!

Failure IS an Option: Measuring and Reporting It

Back to Square One signpost

Failure.  We all know it’s good for us.  We learn from failure, right? In Batman Begins, Bruce Wayne’s dad says “Why do we fall, Bruce? So we can learn to pick ourselves up.”  But sometimes failure, like falling, isn’t much fun (although, just like falling, sometimes it is fun for the other people around you). Sometimes in our jobs we have to report our failures to someone. And sometimes the politics of our jobs makes reporting failure a definite problem.

In the NEO office we like to start our meetings by reporting a recent failure. I think it’s a fun thing to do because I think my failures are usually pretty funny.  But Cindy has us do it from a higher motivation than getting people to laugh.  Taking risks is about being willing to fail. Sara Blakely, the founder and CEO of Spanx, grew up reporting failure every day at the dinner table: https://vimeo.com/175524001  In this video she says that “failure to me became not trying, versus the outcome.”

Why failure matters in evaluation

In general we are all really good at measuring our process (the activities we do) and not so good at measuring outcomes (the things we want to see happen because of our activities).  This is because we have a lot of control over whether our activities are done correctly, and very little control over the outcomes.  We want to measure something that shows that we did a great job, and we don’t want to measure something that might make us look bad. That’s why we find it preferable to measure something we have control over. It can look like we failed if we didn’t get the results we wanted, even if the work at our end was brilliant.

sad businesswoman

But of course outcomes are what we really care about (Steering by Outcomes: Begin with the End in Mind).  They are the “what for?” of what we do.  What if you evaluated the outcomes of some training sessions that you taught and you found out that no one used the skill that you taught them.  That would be sad and it might look like you wasted time and resources.  But on the other hand, what if you don’t measure whether or not anyone ever uses what you taught them, and you just keep teaching the classes and reporting successful classes, never finding out that people aren’t using what you taught them.  Wouldn’t that be the real waste of resources?

So how do you report failure?

I think getting over our fear of failure has to do with learning how to report failure so it doesn’t look like, well, failure.  The key is to stay focused on the end goal: we all really want to know the answer to the question “are we making a difference?”  If we stay focused on that question, then we need to figure out what indicators we can measure to find the answer. If the answer is “no, we didn’t make a difference” then how can we report that in a way that shows we’ve learned how to make the answer “yes?” How can we think about failure so it’s about “learning to pick ourselves up?” or better yet, contributing to your organization’s mission?

One way is to measure outcomes early and often. If you wait until the end of your project to measure your outcomes, you can’t adjust your project to enhance the possibilities of success.  If you know early on that your short-term outcomes are not coming out the way you hope, you can change what you’re doing.  So when you do your final report, you aren’t reporting failure, you’re reporting lessons learned, flexibility and ultimately success.

Here’s an example

Let’s say you’re teaching a series of classes to physical therapists on using PubMed Health so they can identify the most effective therapy for their patients.  At the end of the class you have the students complete a course evaluation, in which they give high scores to the class and the teachers.  If you are evaluating outcomes early, you might add a question like: “Do you think you will use PubMed Health in the next month?”  This is an early outcome question.   If most of them say “no” to this question, you will know quickly that if you don’t change something about what you’re doing in future classes, it is unlikely that a follow-up survey two months later will show that they had used PubMed Health.  Maybe you aren’t giving examples that apply to these particular students. Maybe these students aren’t in the position to make decisions about effective therapies. You have an opportunity to talk to some of the students and find out what you can change so your project is successful.

Complete Failure

You’ve tried everything, but you still don’t have the results you wanted to see.  The good news is, if you’ve been collecting your process and outcomes data, you have a lot of information about why things didn’t turn out as hoped and what can be done differently. Reporting that information is kind of like that commercial about how to come in late to a meeting. If you bring the food, you’re not the person who came late, you’re the person who brought breakfast.  If you report that you did a big project that didn’t work, you’re reporting failure.  If you report that things didn’t work out the way you hoped, but you have data-based suggestions for a better use of organizational resources that meet the same goal–then you’re the person who is working for positive change that supports the organization, and have metaphorically brought the breakfast. Who doesn’t love that person?

 

Beyond the Memes: Evaluating Your Social Media Strategy – Part 2

In my last post, I wrote about how to create social media outcomes for your organization. This week, we will take a look at writing objectives for your outcomes using the SMART method.

Though objectives and outcomes sound like the same thing, they are two different concepts in your evaluation plan – outcomes are the big ideas, while objectives relate to the specifics. Read Karen’s post to find out more about what outcomes and objectives are.

In the book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, they talk a lot about SMART objectives. We have not covered these types of objectives on the blog, so I thought this would be a good time to introduce this type of objective building. According to the book, a SMART objective is “specific, measurable, attainable, realistic, and timely” (Kanter and Paine 47). There are many variations on this definition, so we will use my favorite: Specific, Measurable, Attainable, Relevant, and Timely.

Specific: Leave the big picture for your outcomes. Use the 5 W’s (who, what, when, where, and why) to help craft this portion
Measurable: If you can’t measure it, how will you know you’ve actually achieved what you set out to do?
Attainable: Don’t make you objectives impossible. It’s not productive (or fun) to create objectives that you know you cannot reach. Understand what your community needs, and involve stakeholders.
Relevant: Is your community on Twitter? Create a Twitter account. Do they avoid Twitter? Don’t make a Twitter account. Use the tools that are relevant to the community that you serve.
Timely: Set a time frame for your objectives and outcomes, or your project might take too long for it to be relevant to your community. Time is also money, so create a deadline for your project so that you do not waste resources on a lackluster project.

As an example, let’s return to NEO’s favorite hypothetical town of Sunnydale to see how they added social media objectives into their Dusk to Dawn program. To refresh your memory, read this post from last September about Sunnydale’s Evaluation Plan.

Christopher Walken Fever Meme with the text 'Well, guess what! I’ve got a fever / and the only prescription is more hashtags'

Sunnydale librarians know that their vampire population uses Twitter on a daily basis for many reasons – meeting new vampires, suggesting favorite vampire friendly night clubs, and even engaging the library with general reference questions. Librarians came up with the idea to use the hashtag #dusk2dawn in all of their promotional materials about the health program Dusk to Dawn. Their thinking was it would help increase awareness of their objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

With that in mind, let’s make a SMART objective for this hashtag’s usage:

Specific
We will plug in what we have so far into the Specific section:

Vampires (who) in Sunnydale (where) will show an increase in awareness of health-related events hosted by the library (what) by retweeting the hashtag #dusk2dawn (why) for the duration of the Dusk to Dawn program (when).

Measurable
Measurable is probably the hardest part. What kind of metrics will Sunnydale librarians use to measure hashtag usage? How will they do it?

The social media librarian will manually monitor the hashtag’s usage by setting up an alert for its usage on TweetDeck. Each time the hashtag is used by a non-librarian in reference to the Sunnydale Library, the librarian will copy the tweet’s content to a spreadsheet, adding signifiers if it is a positive or negative tweet.

Attainable
Can our objective be reached? What is it about vampires in Sunnydale that makes this hashtag monitoring possible?

We know from polling and experience that our community likes using Twitter – specifically, they regularly engage with us on this platform. Having a dedicated hashtag for our overall program is a natural progression for us and our community.

Relevant
How does the hashtag #dusk2dawn contribute to the overall Dusk to Dawn mission?

An increase in usage of the hashtag #dusk2dawn will show that our community is actively talking about, hopefully in a positive way. This should increase awareness of our program’s objectives of 4 evening classes on MedlinePlus and PubMed, which in turn would support the outcomes “Increased ability of the Internet-using Sunnydale vampires to research needed health information” and “These vampires will use their increased skills to research health information for their brood.”

Timely
How long should it take for the vampires to increase their awareness of our program’s objectives?

There should be an upward trend in awareness over the course of the program. We have 7 months before we are reevaluating the whole Dusk to Dawn program, so we will set 7 months as our deadline for increased hashtag usage.

SMART!
Now, we put it all together to get:

Vampires in Sunnyvale will show an increase in awareness of health-related events hosted by the library, indicated by a 15% increase of the hashtag #dusk2dawn by Sunnydale vampires for the duration of the Dusk to Dawn program.

Success Baby

Though this objective is SMART, it is certainly will not work in every library. Perhaps the community your library serves does not use Twitter to connect with the library, or you do not have enough people on staff to monitor the hashtag’s usage. If you make a SMART objective that will be relevant to your community, it will have a better chance to succeed.

Here at NEO, we usually do not use SMART objectives method, but rather Measurable Outcome Objectives. Step 3 on the Evaluation Resources page points to many different resources on our website that talk about Measurable Objectives. Try both out, and see what works for your organization.

We will be taking a break from social media evaluation and goal setting for a few weeks. Next time we talk about social media, we will show our very own social media evaluation plan!

Find more information about SMART objectives here:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Image credits: Christopher Walken Fever Meme made by Kalyna Durbak on ImgFlip.com. Success Kid meme is from Know Your Meme.

Beyond the Memes: Evaluating Your Social Media Strategy – Part 1

Welcome to our new NEO blogger, Kalyna Durbak.  Her first post addresses a topic that is a concern to many of us, evaluating our social media!


By Kalyna Durbak, Program Assistant, NNLM Evaluation Office (NEO)

Have you ever wondered if your library’s Facebook page was worth the time and effort? I think about social media a lot, but then again I’ve been using Facebook daily for over 10 years. The book Measuring the Networked Nonprofit, by Beth Kanter and Katie Delahaye Paine, can help your library or organization figure out how to measure the impact of your social media campaigns have on the world.

Not all of us work for a nonprofit, but I feel many organizations share similar constraints with nonprofits – like not being able to afford to hire a firm to develop and manage the social media accounts. It’s easy to think that social media is easy to do because we all manage our personal profiles. Once you start managing accounts that belong to an organization, it gets hard. What do you post? What can you post? How many likes can I collect?

One does not simply post memes on Facebook

Before we get into any measurement, I want to briefly write about why social media outcomes are important to have, and why they should be measured. A library should not create a Facebook page simply to collect likes, or a Twitter page to gather followers. As my husband would say, that’s simply “scoring Internet points.” Internet points make you feel good inside, but do not impact the world around you. The real magic in using social media comes from creating a community around your organization that is willing to show up and help out when you ask.

A library should create a social media page in order for something to happen in the real world – an outcome. Figuring out why you need a social media account will help your library manage its various accounts more efficiently, and in the end measure the successes and failures of your social media campaigns. If you need more convincing, read Cindy’s post “Steering by Outcomes: Begin with the End in Mind.” For help on determining your outcomes, I suggest reading Karen’s blog post “Developing Program Outcomes using the Kirkpatrick Model – with Vampires.”

What are some reasons for using social media in your library? Maybe you will have an online campaign to promote digital assets, or perhaps you will add a social media component to a program that already exists in your library. Whatever they are, any social media activity you do should support an outcome. A few outcomes I can think of are:

  • Increased awareness of library programs
  • New partnerships found for future collaborative efforts
  • Improved perceptions about the organization
  • Increase in library foundation’s donor base

None of the outcomes specifically mention Facebook, Twitter, or any other social media platforms. That’s because outcomes outline the big picture – it’s what you want to happen after completing your project. In the above examples, a library wants the donor base to be increased, or the library wants increased awareness of library programs. It’s the ideal world your library or organization wants to exist in. Facebook and Twitter can help achieve these outcomes, but the number of retweets you get is not an outcome.

To make that ideal future a reality, you need to create objectives. Objectives are the signposts that will indicate whether you are successful in reaching your outcome. Next week, we will craft social media oriented objectives for a library in our favorite hypothetical town of Sunnydale. Catch up on Sunnydale with these posts:

Let me know if you have any questions or comments about this post! Comment on Facebook and Twitter, or email me at kalyna@uw.edu.

Make 2017 the “Yes-and” Year for Evaluation

20117 (year) with fireworks and "Always Say Yes" written underneath

PSA: New Year’s resolutions are passé.

It’s out with dreary self-betterment goals involving celery and punishing exercise. Instead, the trendiest way to mark the New Year is to pick an inspirational word and make it your annual North Star for self-improvement.  In that spirit, I want to propose a word of the year for the NEO Shop Talk community.

That word is “Yes and.”

Now, some of you are wondering how I passed kindergarten with such poor counting skills. However, If you or anyone you know has taken training in Improv theatre, you know “Yes and” is a word, or, more specifically, a verb.

Improv is a type of theatre in which a team of actors make up scenes on the spot, usually from audience suggestions.  Because performances are unscripted, improv actors train rather than rehearse. Training is built around commonly accepted “rules,” and, arguably, the best known rule of Improv is “Always say Yes and…”   That means that you accept any scene idea your teammate presents and add something to make that idea better. Once novice improvisers experience the upbeat emotional effect of this rule, they soon find themselves “yes-anding” in other parts of their lives. Some even preach about it to others (ahem).  Notice, by the way, I added a hyphen so we can all feel better about “yes-and” as the WORD of the year.

If you can “yes-and” evaluation requirements and responsibilities, it will put you on the road to mastering this rule. Let’s face it, the thought of evaluation does not generate an abundance of enthusiasm. Usually we do evaluation because someone else expects or requires us to do it: upper administration; accreditation boards; funding agencies. We only do evaluation when forced because it’s a lot of work. I compare evaluation to physical exercise. In theory, we know it’s good for us. In practice, we don’t have time for it. “Yes-anding” evaluation may not make you do more evaluation than you’re required to do.  It might, however, make your evaluation responsibilities more enjoyable or, at least, more meaningful to you personally.

For example, if you have to write a proposal for external funding, you often have to pull together assessment information to build a case for your proposed program. Does that mean you have to locate and synthesize lots of data from lots of sources? Yes, and you get to demonstrate all of the great things your library or organization has to offer. You also get to point out areas where you could provide even more awesome services if the funding agency gave you funding to meet your resource needs. (Here’s a NEO Shop Talk blog post on how to use SWOT analysis to synthesize needs assessment data.)

If your proposal includes outreach into a new community, you probably have to collect information from that community.   Do you have to find and conduct key informant interviews with representatives of the community?   Yes, and you also get to initiate relationships with influential community opinion leaders. Listening is a powerful way to build trust and rapport. If you have the opportunity to implement your program, these key informants will be powerful allies when you want to reach out to the broader community. (If you want some tips for finding key informants, check out this blog post.)

You can “yes-and” internally mandated evaluation as well. Your library or organization may require you to track data on an ongoing basis or to submit regular reports. To do this well, do you have to document your daily work, such as keep track of details surrounding reference services, workshop attendance, or facility usage? Yes and, you also get to create a database of motivational information to inspire you and fellow co-workers working on the same objectives and goals. Compile that information monthly or quarterly, and pass it around at staff meetingd.  Celebrate what you’re accomplishing. Figure out where effort is lagging and commit to bolstering activities in that area. Then celebrate your team’s astute use of data for making good program decisions. Yes, and, at reporting time, be sure you present your data so that your upper-level stakeholders notices your hard work.

Maybe your department or office has to set and assess annual objectives or outcomes. Do you have to collect and report data to show program results? Yes, and you also get to demonstrate your value to the organization. Just be sure you don’t hide your candle deep in some organizational online reporting system.  Annual reports are seldom page-turners. Find more compelling ways to communicate your success and contributions to upper administrators and influential users. For some ideas, you might want to check out some of NEO Shop Talk posts on reporting and data visualization.

Another rule of Improv is “There are no mistakes, only opportunities.”  Let’s paraphrase that to “there are no evaluation requirements, only opportunities.”  Here’s to making 2017 the year of “yes-anding” evaluation.

As a post-script, I want to share a NEO Shop Talk success with our readers. Did we post weekly blog entries in 2016? Yes, and you showed up more than ever. NEO Shop Talk visits increased 71% in 2016. Each month, we averaged 259 more visits compared to the same month in the previous year.  Our peak month was February, with 892 visits! Thank you, readers. Please come back and bring your friends!

slide1

2016 Annual NEO Shop Talk Round-up

Top 10 List

Like everyone else, we have an end-of-the-year list.  Here’s our top ten list of the posts we wrote this year, based on number of views:

10. Developing Program Outcomes Using the Kirkpatrick Model – with Vampires

9.  Inspirational Annual Reporting with Appreciative Inquiry

8.  What is a Need?

7.  Designing Surveys: Does the Order of Response Options Matter?

6.  Simply Elegant Evaluation: GMR’s Pilot Assessment of a Chapter Exhibit

5.  A Chart Chooser for Qualitative Data!

4.  W.A.I.T. for Qualitative Interviews

3.  The Zen Trend in Data Visualization

2.  How I Learned to Stop Worrying and Love Logic Models (The Chili Lesson)

1.  Logic Model for a Birthday Party

We put a lot of links to interesting things in our blog posts.  Here are the Top Ten websites that people went to from our blog:

10. The Kirkpatrick Model

9.  Books by Stephanie Evergreen

8.  Tearless Logic Model article in Global Journal of Community Psychology Practice

7.  AEA 365 | A Tip-a-Day by and for Evaluators

6.  Public Libraries, Project Outcome – Looking Back, Looking Forward

5.  Build a Qualitative Dashboard

4.  Nat King Cole, The Christmas Song

3.  The Histomap by John Sparks

2.  Tools: Tearless Logic Model (how-to summary)

1.  Stephanie Evergreen Qualitative Chart Chooser

The NEO wishes you a happy and fulfilling New Year!!

My Favorite Things 2016 (Spoiler Alert: Includes Cats)

Little figurine of Santa standing in snow, holding gifts

During gift-giving season every year, Oprah publishes a list of her favorite things. Well, move over, Oprah, because I also have a list. This is my bag of holiday gifts for our NEO Shop Talk readers.

Art Exhibits

There are two websites with galleries of data visualizations that are really fun to visit. The first,  Information is Beautiful , has wonderful examples of data visualizations, many of which are interactive. My favorites from this site are Who Old Are You?   (put in your birth date to start it) and Common MythConceptions. The other is Tableau Public, Tableau Software Company’s “public commons” for their users to share their work.  My picks are the Endangered Species Safari  and the data visualization of the Simpsons Vizapedia.  And, in case  you’re wondering what happened to your favorite Crayola crayon colors, you can find out here.

Movies

Nancy Duarte’s The Secret Structure of Great Talks is my favorite TEDtalk. Duarte describes the simple messaging structure underlying inspirational speeches. Once you grasp this structure, you will know how to present evaluations findings to advocate for stakeholder support. I love the information in this talk, but that’s not why I listen to it over and over again.  It’s because Duarte says “you have the power to change the world” and, by the end of the talk, I believe her.

Dot plot for a fictional workshop data, titled Participant Self Assessment of their Holiday Skills before and after our holiday survival workshop. Pre/post self-report ratings for four items: Baking without a sugar overdose (pre=3; post-5); Making small talk at the office party (pre=1; post=3); Getting gifts through airport security (pre=2; post-5); Managing road rage in mall parking lots (pre=2; post-4)

I also am a fan of two videos from the Denver Museum of Natural History, which demonstrate how museum user metrics can be surprisingly entertaining. What Do Jelly Beans Have To Do With The Museum? shows demographics with colorful candy and Audience Insights On Parking at the Museum  talks amusingly about a common challenge of urban life.

Crafts

If you want to try your hand at creating snappier charts and graphs, you need to spend some time at Stephanie Evergreen’s blog. For example, she gives you step-by-step instructions on making lollipop charts, dot plots , and overlapping bar charts. Stephanie works exclusively in Excel, so there’s no need to purchase or learn new software. You also might want to learn a few new Excel graphing tricks at Ann Emery’s blog.  For instance, she describes how to label the lines in your graphs or adjust bar chart spacing.

Site Seeing

How about a virtual tour to the UK? I still marvel at the innovative Visualizing Mill Road  project. Researchers collected community data, then shared their findings in street art. This is the only project I know of featuring charts in sidewalk chalk. The web site talks about community members’ reactions to the project, which is also pretty fascinating.

Humor

I left the best for last. This is a gift for our most sophisticated readers, recommended by none other than Paul Gargani, president of the American Evaluation Association. It is a web site for the true connoisseurs of online evaluation resources.  I present to you the Twitter feed for  Eval Cat.  Even the  NEO Shop Talk cats begrudgingly admire it, although no one has invited them to post.

 

Pictures of the four NEO Eval Cats

 

 

 

 

 

 

 

 

 

Here’s wishing you an enjoyable holiday.

Holiday Outcome Planning with Nat King Cole

The holidays can be very stressful. Last minute shopping, traffic, traveling, and talking politics — it’s clearly time to focus on the outcomes we want to see for the holidays, and how we are going to reach them. For help with my planning, I looked to what I think of as the greatest Christmas song of the season, The Christmas Song, most famously sung by Nat King Cole.

Here’s the song in logic model format (I suggest listening to the song at the same time as you read the logic model https://www.youtube.com/watch?v=hwacxSnc4tI):

Logic model of The Christmas Song

 

As with so many things in the holiday season, this song doesn’t really reflect my reality in Gulf Coast Texas (although we do get mistletoe–I had it growing on my oak tree a couple of years ago).  Working from similar outcomes backwards, here’s a different logic model that only applies to me:

Logic Model Showing Karen's Holiday plans

No, I will not attempt to turn it into a song, and you’re even luckier that I’m not going to sing it for you.  But this little process has been really helpful for me in keeping things in perspective for the next couple of weeks.

Thanks Nat King Cole, Bob Wells, and Mel Tormé for a great song and a great planning tool.  What would yours look like?

For more on logic models, select the blog’s Logic Model category, and check out the NEO’s Booklet 2: Planning Outcomes-Based Outreach Projects

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.