Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for December, 2016

2016 Annual NEO Shop Talk Round-up

Friday, December 30th, 2016

Top 10 List

Like everyone else, we have an end-of-the-year list.  Here’s our top ten list of the posts we wrote this year, based on number of views:

10. Developing Program Outcomes Using the Kirkpatrick Model – with Vampires

9.  Inspirational Annual Reporting with Appreciative Inquiry

8.  What is a Need?

7.  Designing Surveys: Does the Order of Response Options Matter?

6.  Simply Elegant Evaluation: GMR’s Pilot Assessment of a Chapter Exhibit

5.  A Chart Chooser for Qualitative Data!

4.  W.A.I.T. for Qualitative Interviews

3.  The Zen Trend in Data Visualization

2.  How I Learned to Stop Worrying and Love Logic Models (The Chili Lesson)

1.  Logic Model for a Birthday Party

We put a lot of links to interesting things in our blog posts.  Here are the Top Ten websites that people went to from our blog:

10. The Kirkpatrick Model

9.  Books by Stephanie Evergreen

8.  Tearless Logic Model article in Global Journal of Community Psychology Practice

7.  AEA 365 | A Tip-a-Day by and for Evaluators

6.  Public Libraries, Project Outcome – Looking Back, Looking Forward

5.  Build a Qualitative Dashboard

4.  Nat King Cole, The Christmas Song

3.  The Histomap by John Sparks

2.  Tools: Tearless Logic Model (how-to summary)

1.  Stephanie Evergreen Qualitative Chart Chooser

The NEO wishes you a happy and fulfilling New Year!!

My Favorite Things 2016 (Spoiler Alert: Includes Cats)

Wednesday, December 21st, 2016

Little figurine of Santa standing in snow, holding gifts

During gift-giving season every year, Oprah publishes a list of her favorite things. Well, move over, Oprah, because I also have a list. This is my bag of holiday gifts for our NEO Shop Talk readers.

Art Exhibits

There are two websites with galleries of data visualizations that are really fun to visit. The first,  Information is Beautiful , has wonderful examples of data visualizations, many of which are interactive. My favorites from this site are Who Old Are You?   (put in your birth date to start it) and Common MythConceptions. The other is Tableau Public, Tableau Software Company’s “public commons” for their users to share their work.  My picks are the Endangered Species Safari  and the data visualization of the Simpsons Vizapedia.  And, in case  you’re wondering what happened to your favorite Crayola crayon colors, you can find out here.

Movies

Nancy Duarte’s The Secret Structure of Great Talks is my favorite TEDtalk. Duarte describes the simple messaging structure underlying inspirational speeches. Once you grasp this structure, you will know how to present evaluations findings to advocate for stakeholder support. I love the information in this talk, but that’s not why I listen to it over and over again.  It’s because Duarte says “you have the power to change the world” and, by the end of the talk, I believe her.

Dot plot for a fictional workshop data, titled Participant Self Assessment of their Holiday Skills before and after our holiday survival workshop. Pre/post self-report ratings for four items: Baking without a sugar overdose (pre=3; post-5); Making small talk at the office party (pre=1; post=3); Getting gifts through airport security (pre=2; post-5); Managing road rage in mall parking lots (pre=2; post-4)

I also am a fan of two videos from the Denver Museum of Natural History, which demonstrate how museum user metrics can be surprisingly entertaining. What Do Jelly Beans Have To Do With The Museum? shows demographics with colorful candy and Audience Insights On Parking at the Museum  talks amusingly about a common challenge of urban life.

Crafts

If you want to try your hand at creating snappier charts and graphs, you need to spend some time at Stephanie Evergreen’s blog. For example, she gives you step-by-step instructions on making lollipop charts, dot plots , and overlapping bar charts. Stephanie works exclusively in Excel, so there’s no need to purchase or learn new software. You also might want to learn a few new Excel graphing tricks at Ann Emery’s blog.  For instance, she describes how to label the lines in your graphs or adjust bar chart spacing.

Site Seeing

How about a virtual tour to the UK? I still marvel at the innovative Visualizing Mill Road  project. Researchers collected community data, then shared their findings in street art. This is the only project I know of featuring charts in sidewalk chalk. The web site talks about community members’ reactions to the project, which is also pretty fascinating.

Humor

I left the best for last. This is a gift for our most sophisticated readers, recommended by none other than Paul Gargani, president of the American Evaluation Association. It is a web site for the true connoisseurs of online evaluation resources.  I present to you the Twitter feed for  Eval Cat.  Even the  NEO Shop Talk cats begrudgingly admire it, although no one has invited them to post.

 

Pictures of the four NEO Eval Cats

 

 

 

 

 

 

 

 

 

Here’s wishing you an enjoyable holiday.

Holiday Outcome Planning with Nat King Cole

Friday, December 16th, 2016

The holidays can be very stressful. Last minute shopping, traffic, traveling, and talking politics — it’s clearly time to focus on the outcomes we want to see for the holidays, and how we are going to reach them. For help with my planning, I looked to what I think of as the greatest Christmas song of the season, The Christmas Song, most famously sung by Nat King Cole.

Here’s the song in logic model format (I suggest listening to the song at the same time as you read the logic model https://www.youtube.com/watch?v=hwacxSnc4tI):

Logic model of The Christmas Song

 

As with so many things in the holiday season, this song doesn’t really reflect my reality in Gulf Coast Texas (although we do get mistletoe–I had it growing on my oak tree a couple of years ago).  Working from similar outcomes backwards, here’s a different logic model that only applies to me:

Logic Model Showing Karen's Holiday plans

No, I will not attempt to turn it into a song, and you’re even luckier that I’m not going to sing it for you.  But this little process has been really helpful for me in keeping things in perspective for the next couple of weeks.

Thanks Nat King Cole, Bob Wells, and Mel Tormé for a great song and a great planning tool.  What would yours look like?

For more on logic models, select the blog’s Logic Model category, and check out the NEO’s Booklet 2: Planning Outcomes-Based Outreach Projects

‘Tis the Season to Do Some Qualitative Interviewing!

Friday, December 9th, 2016

For most of us, the end-of-year festivities are in full swing. We get to enjoy holiday treats. Lift a wine glass with colleagues, friends, and loved ones. Step back from the daily grind and enjoy some light-hearted holiday fun.

Or, we could take these golden holiday social events to work on our qualitative interviewing skills! That’s right.  I want to invite you to participate in another NEO’s holiday challenge: The Qualitative Interview challenge. (You can read about our Appreciative Inquiry challenge here.)

If you are a bit introverted and overwhelmed in holiday situations, this challenge is perfect for you. It will give you a mission: a task to take your mind off that social awkwardness you feel in large crowds. (Please tell me I’m not the only one!) If, on the other hand, you are more of a life-of-the-party guest, this challenge will help you talk less and listen more.  Other party-goers will love you and you might learn something.

Here’s your challenge.  Jot down some good conversational questions that fit typical categories of qualitative interview questions.  Commit a couple questions to memory before you hit a party. Use those questions to fuel conversations with fellow party-goers and see if you get the type of information you were seeking.

To really immerse yourself in this challenge, create a chart with the six categories of questions. (I provided an example below)  When your question is successful (i.e., you get the type of information you wanted), give yourself a star.  Sparkly star stickers are fun, but you can also simply draw stars beside the questions. Your goal is to get at least one star in each category by midnight on December 31.

Holiday challenge chart, There is a holiday border around a table-style chartt with the six categories of questions, the five extra credit techniques, and blank cells for stars

According to qualitative researcher/teacher extraordinaire Michael Q. Patton, there are six general categories of qualitative interview questions.  Here are categories:

  • Experience or behavior questions: Ask people to tell you a story about something they experienced or something they do. For unique experiences, you might say “Describe your best holiday ever.” You could ask about more routine behavior, such as “What traditions do you try to always celebrate during the holidays?”
  • Sensory questions: Sensory questions are similar to experience questions, but they focus on what people see, hear, feel, smell, or taste. Questions about holiday meals or vacation spots will likely elicit sensory answers.
  • Opinion and value questions: If you ask people what they think about something, you will hear their opinions and values. When Charlie Brown asked “What is the true meaning of Christmas?” he was posing a value/opinion question.
  • Emotions questions: Here, you ask people to express their emotional reactions. Emotions questions can be tricky. In my experience, most people are better at expressing opinions than emotions, so be prepared to follow up.  For example, if you ask me “What do you dislike about the holiday season?” I might say “I don’t like gift-shopping.”   “Like” is more of an opinion word than an emotion word. You want me to reach past my brain and into my heart. So you could follow-up with “How do you feel when you’re shopping for holiday gifts?”  I might say “The crowds frustrate and exhaust me” or “I feel stressed out trying to find perfect gifts on a deadline.“ Now I have described my emotions around gift-shopping. Give yourself a star!
  • Knowledge questions: These questions seek factual information. For example, you might ask for tried-and-true advice to make holiday travel easier. If answers include tips for getting through airport security quickly or the location of clean gas station bathrooms on the PA Turnpike, you asked a successful knowledge question.
  • Background and demographic questions: These questions explore how factors such as ethnicity, culture, socio-economic status, occupation, or religion affect one’s experiences and world view. What foods do their immigrant grandparents cook for New Year’s celebrations every year?  What is it like to be single during the holidays? How do religious beliefs or practices affect their approach to the holidays? These are examples of background/demographic questions.

To take this challenge up a notch, try to incorporate the following techniques while practicing interview skills over egg nog.

Ask open-ended questions. Closed-ended questions can be answered with a word or phrase.  “Did you like the movie?”  The answer “Yes” or “No” is a comprehensive response to that question.   An open-ended version of this question might be “Describe  a good movie you saw recently.”  If you phrased your question so that your conversation partner had to string together words or sentences to form an answer, give yourself an extra star.

Pay attention to question sequence:  The easiest questions for people to answer are those that ask them to tell a story. The act of telling a story helps people get in touch with their opinions and feelings about something.  Also, once you have respectfully listened to their story, they will feel more comfortable sharing opinions and feelings with you. So break the ice with experience questions.

Wait for answers:  Sometimes we ask questions, then don’t wait for a response.  Some people have to think through an answer completely before they talk out loud. Those seconds of silence make me want to jump in with a rephrased question. The problem is, you’ll start the clock again as they contemplate the new version of your question. To hold myself back, I try to pay attention to my own breathing while maintaining friendly eye contact.

Connect and support: You get another star if you listened carefully enough to accurately reflect their answers back to them. This is called reflective listening.  If you want a fun tutorial on how to listen, check out Julian Treasure’s TEDtalk.

Some of you are likely thinking “Thanks but no thanks for this holiday challenge.” Maybe it seems too much like work. Maybe you plan to avoid social gatherings like the plague this season.  Fair enough.  All of the tips apply to bona fide qualitative interviews. When planning and conducting qualitative interviews, remember to include questions that target different types of information. Make your questions open-ended and sequence them so they are easy to answer.  Listen carefully and connect with your interviewee by reflecting back what you heard.

Regardless of whether you take up the challenge or not, I wish you happy holidays full of fun and warm conversations.

My source for interview question types and interview techniques was  Patton MQ. Qualitative Research and Evaluation Methods.  4th ed.  Thousand Oaks, CA: Sage, 2015.

Creating Partnerships that Work

Friday, December 2nd, 2016

Multiracial Businesspeople Stacking Hands

“Five guys on the court working together can achieve more than five talented individuals who come and go as individuals.” Kareem Abdul-Jabbar

When you’re working on an outreach project, you will almost certainly have some kind of partner organizations in the project.  Funders of outreach projects love to see partnerships, and sometimes they even require it.  When everything works like it’s supposed to, a partnership between organizations working on a joint outreach project can spawn better ideas, create a richer program, and improve reach.

But have you ever felt like you’ve made some bad decisions in your choice of partners? (I’m not talking about your sordid relationship history here).  It feels like a disaster when your plans fall apart because your partner organization had a completely different understanding of their role in the project, different priorities, or there were communication problems (okay, maybe I am).

Yesterday I was reviewing our Tools and Resources Guide for a major website update coming next week.  While I was doing that, I re-discovered some great resources for choosing and maintaining partners.

The Community Tool Box from the University of Kansas has a toolkit on Creating and Maintaining Partnerships.  This toolkit is made up of steps for partnering organizations to work through together. Here are some of the main categories, but the descriptions on the website are quite detailed:

  • Describe the problems or goals that have brought partners together in common purpose
  • Outline your partnership’s vision and mission, objectives
  • Re-examine the group’s membership in light of your vision, mission, and objectives
  • Describe potential barriers to your partnership’s success and how you would overcome them
  • Describe how the partnership will function and how responsibilities will be shared among partner organizations
  • Describe how the group will maintain momentum and foster renewal
  • If the partnership is losing momentum, review current barriers to your success
  • If necessary, revisit the plan to identify and recruit new or additional members
  • When maintaining the partnership at its current level is no longer appropriate or feasible, consider other alternatives, including changing focus, adding new members, or even dissolving the partnership

The Urban Indian Health Institute (UIHI) has a Resource Guide: Establishing and Maintaining Effective Partnerships.  It is a one page document with an emphasis on building trust among partners.  Here are some of their characteristics of successful partnerships.

  • A common vision and collective commitment
  • Mutual trust and respect
  • Risks, resources and rewards are shared jointly
  • Opportunities for capacity building through learning exchanges
  • Openness to learning and teaching opportunities
  • Ground rules that create a safe space to address challenges
  • Acknowledgement of the differences between the partners
  • Flexibility

Balancing the emphasis on trust and respect, UIHI also has helpful guides on their Resources for Partnerships page for establishing Memorandums of Understanding (MOUs) so partners clearly understand what’s expected of each other.

You might take a look at these guides and think “they’re asking me to do a lot of work – I just want to do a few health information presentations at the local public library.”  While you may be correct, sometimes a small project grows into something bigger, and then suddenly you find yourselves writing a proposal for a grant from the National Library of Medicine.  When going into a partnership, large or small (or personal), it might be worthwhile to take a look at some of these these guides — even if you don’t do all the steps or complete an MOU, you will find something that can help you make sure your partnership is nourished and successful for as long as it needs to be.

 

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.