Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘News’ Category

Hungry for More Evaluation Ideas? How About a Tip a Day?

Friday, April 28th, 2017

Sometimes people ask us where we get ideas for our posts.  I’m going to tell you one of our big secrets – when we can’t think of something we check out the American Evaluation Association’s daily blog AEA 365 | A Tip-A-Day by and for Evaluators.  Seriously, every single day some evaluator posts a tip for other evaluators.

In no way do we rewrite any of their posts, but we do in fact scrounge for ideas.  I thought you might want to scrounge for your own ideas by looking at this great blog now and then.

Just this week, there’s a post about about using very short surveys throughout a project so that you can make changes to improve your project as you go: Using pulse surveys to get rapid actionable feedback from teachers during a professional development experience by Valerie Futch Ehrlich  Along with describing how they did it, the post also recommends a survey tool called Waggl that allows people to vote on each others’ feedback, so the best ideas float to the top. Cool, right?

Here’s another post about how you can use tables to explain complicated ideas to potential funders in grant applications: Using Tables Effectively in Grant Proposals by Kate LaVelle and Judith Rhodes. 

Happy scrounging!

 

Finding Evaluator Resources in Surprising Places

Friday, April 21st, 2017

The National Training Office recently posted a Free and Low Cost Tools guide to help educators create and carry out training. Many of these tools can be used by more than just trainers though — evaluators can makes use of these tools as well. Here are some of my favorite tools from NTO’s guide.

Trello
https://trello.com
I have used Trello to make personal to-do lists and to keep track of my progress with the recent NNLM website migration. Since Trello boards have a column structure, I thought to myself- could I use Trello to make an interactive Logic Model? Yes, yes I can!

I took the logic model from our Sunnydale blog series, and converted it into a Trello board. I made lists for Inputs, Activities, Reach, Short-term Outcomes, Intermediate Outcomes, and Long-term Outcomes. Each individual activity, outcome, etc is written on a Trello card under the list item. You can comment on the cards, create checklists for each card, set due dates, and mark them complete, all on an interactive and visually pleasing logic model.

Are your cards getting cluttered with comments and checklists? Create a new board and link the board to a specific card. For example, you can create a separate Kanban board for the card “Start a 12-hour “Dusk to Dawn” health reference hotline,” and attach the link to the new board to the card in the logic model. Then, once that activity is completed, you can mark the corresponding card in the logic model as complete.

Adobe Spark
https://spark.adobe.com

Here at NEO we are all about creating aesthetically pleasing and easy to read reports. I believe we all want to create pretty reports, but sometimes lack the time and energy to create one. It is much easier to type up a simple Word document and send it as an attachment.

With Adobe Spark, you can create slick web pages, social media posts, and even videos for free. I created the image on the left for a future slide deck about Sunnydale’s evaluation program in less than 5 minutes.

This image is proportioned for a PowerPoint presentation, but you can create custom sized images for physical fliers, Facebook, and a number of other social media platforms.

I also made this spoof report with Adobe Spark in about 10 minutes. No HTML/CSS knowledge was needed, and Adobe Spark hosts the web page for you. It was a fairly simple process.

SlideShark
https://www.slideshark.com
I have not used SlideShark personally, but the idea of being able to present my PowerPoint slides from my mobile device makes my day. Instead of carrying around a bulky laptop, or forgetting a small flash drive, I can use SlideShark to broadcast my PowerPoint straight from my smartphone to a projector. In addition, I can share an online version of my presentation that is viewable at any time. That means no more uploading issues, or large email attachments.

Slack
https://slack.com
Slack has to be my favorite tool on this list. At first, it might seem like a fancy forum, but it is so much more. Slack integrates with many other tools, such as Trello, Google Drive, and Skype, so you can keep all of your relevant work conversations, documents, and tools in one place, instead of hidden in various emails. Emojis and GIFs are encouraged, creating a fun and casual work environment.

Evaluators are always looking to increase stakeholder participation in their evaluation efforts. It could be especially hard to communicate if stakeholders do not live in the same area. Slack could be a useful way to keep stakeholders engaged in conversation no matter the distance. You could even throw a virtual data party! You can have one Slack channel for the entire party, or break up different activities into separate channels. Since Slack invites participants to use emojis and GIFs, the resulting conversation will certainly look like a party! 🎉

What are your favorite tools from NTO’s guide? Let us know in the comments!

Best Bargain for Evaluation Training: the AEA Summer Evaluation Institute

Friday, April 14th, 2017

Who doesn’t love a bargain?

If you love a bargain and you want to up your evaluation game, check out the American Evaluation Association’s Summer Evaluation Institute, now open for registration. This year, it will be held June 5 – 7 at the Omni Atlanta Hotel CNN Center.  For $395 (AEA members) or $480 (nonmembers), you get five 3-hour workshops.  You can choose from among 35 workshops taught by some of the most experienced evaluators in the field. The Summer Institute fee also covers lunch and snacks on most days.

The AEA Summer Institute also offers three pre-session workshops on June 4  for an additional fee. The first two on the list,  Introduction to Evaluation and A Primer to Evaluation Theories and Approaches, are ideal for those who are new to the evaluation field. The third is a special workshop offered by another renowned evaluation training organization, The Evaluators Institute. This 6-hour workshop Evaluation and Culture will teach a step-by-step approach to developing a culturally responsive evaluation.

The NEO Shop Talk bloggers attend this conference regularly. If you decide to join us, let us know on our Facebook page. Maybe we can do lunch!

Logo for the 2017 AEA Summer Evaluation Institute

Uninspired by Bars? Try Dot Plots

Friday, March 17th, 2017

Thanks to Jessi Van Der Volgen and Molly Knapp at the NNLM Training Office for allowing us to feature their assessment project and for providing the images in this post. 

Are you tired of bars?

I don’t mean the kind of bars where you celebrate and socialize. I mean the kind used in data visualization.  My evidence-free theory is that people still succumb to using the justifiably maligned pie chart simply because we cannot face one more bar graph.

Take heart, readers. Today, I’m here to tell you a story about some magic data that fell on the NEO’s doorstep and broke us free of our bar chart rut.

It all began with a project by our NNLM Training Office (NTO) colleagues, the intrepid leaders of NNLM’s instructional design and delivery. They do it all. They teach. They administratively support the regions’ training efforts. They initiate opportunities and resources to up-level instructional effectiveness throughout the network. One of their recent initiatives was a national needs assessment of NNLM training participants. That was the source of the fabulous data I write about today.

For context, I should mention that training is one of NNLM’s key strategies for reaching the furthest corners of our country to raise awareness, accessibility and use of NLM health information resources. NNLM offers classes to all types of direct users, (e.g., health professionals; community-based organization staffs) but we value the efficiency of our “train-the-trainer” programs. In these classes, librarians and others learn how to use NLM resources so they, in turn, can teach their users. The national needs assessment was geared primarily toward understanding how to best serve “train-the-trainer” participants, who often takes multiple classes to enhance their skills.

For the NTO’s needs assessment, one area of inquiry involved an inventory of learners’ need for training in 30 topic areas. The NTO wanted to assess participants’ desired level and their current level of proficiency in each topic.  That meant 60 questions. That was one heck-of-a-long survey. We wished them luck.

The NTO team was undaunted!  They did some research and found a desirable format for presenting this set of questions (see upper left). The format had a nice minimalist design. The sliders were more fun for participants than radio buttons. Also, NTO designed the online questionnaire so that only a handful of question-pairs appeared on the screen at one time.  The approach worked, because NTO received responses from 559 respondents, and 472 completed the whole questionnaire.

Dot plots for four skill topic areas. Conducting literature searches (4=Current; 5=Desired) Understanding and searching for evidence-based research ( Current-3; Desired=5) Develop/teach classes (Current-3; Desired=5; Create videos/web tutorials Current-2; Desired=4)

The NEO, in turn, consulted the writings of one of our favorite dataviz oracles, Stephanie Evergreen. And she did not disappoint.  We found the ideal solution: dot plots!  Evergreen’s easy-to-follow instructions from this blog post allowed us to create dot plots in Excel, using a few creative hacks. This approach allowed us to thematically cluster results from numerous related questions into one chart. We were able to present data for 60 questions in a total of seven charts.

I would like to point out a couple of design choices I made:

  • I used different shapes and colors to visually distinguish between “current proficiency” and “desired proficiency.” Navy blue for current proficiency was inspired from NNLM’s logo. I used a complimentary green for the desired proficiency because green means “go.”
  • Evergreen prefers to place labels (e.g., “conducting literature searches”) close to the actual dots. That works well if your labels consist of one or two words. We found that our labels had to be longer to make sense. Setting them flush-left made them more readable.
  • I suggested plotting medians rather than means because many of the data distributions were skewed. You can use means, but probably should round to whole numbers so you don’t distract from the gaps.

Dot plots are quite versatile. We used the format to highlight gaps in proficiency, but other evaluators have demonstrated that dot plots work well for visualizing change over time and cross-group comparisons.

Dot plots are not as easy to create as the default Excel bar chart, but they are interesting.  So give up bars for a while.  Try plotting!

 

 

 

 

The Dark Side of Questionnaires: How to Identify Questionnaire Bias

Monday, March 6th, 2017

Villain cartoon with survey questions

People in my social media circles have been talking lately about bias in questionnaires.  There are biased questionnaires.  Some of them are biased by accident and some are on purpose.  Some are biased in the questions and some are biased in other ways, such as the selection of the people who are asked to complete the questionnaires. Recently, a couple of my friends posted on Facebook that people should check out the NNLM Evaluation Office to learn about better questionnaires. Huzzah! This week’s post was born!

Here are a few things to look for when creating, responding to, or looking at the results of questionnaires.

Poorly worded questions

Sometimes simple problems with questions can lead to bias, whether accidental or on purpose.  Watch out for these kinds of questions:

  • Questions that have unequal number of positive and negative responses.

Example:

Overall, how would you rate NIHSeniorHealth?

Excellent | Very Good | Good | Fair | Poor 

Notice that “Good” is the middle option (which should be neutral), and some people consider “Fair” to be a slightly positive term.

  • Leading questions, which are questions that are asked in a way that is intended to produce a desired answer.

Example:

Most people find MedlinePlus very easy to navigate.  Do you find it easy to navigate?  (Yes   No)

How would you feel if you had trouble navigating MedlinePlus? It would be hard to say ‘No’ to that question.

  •  Double-barreled questions, which are two questions in one.

 Example:

 Do you want so lower the cost of health care and limit compensation to medical malpractice lawsuits?

 This question has two parts – to answer yes or no, you have to agree or disagree with both parts. And who doesn’t want to lower health care costs?

  •  Loaded questions, which are questions that have a false or questionable logic inherent in the question (a “Have you stopped beating your wife” kind of question). Political surveys are notorious for using loaded questions.

Example:

Are you in favor of slowing the increase in autism by allowing people to choose whether or not to vaccinate their child?

This question makes the assumption that vaccinations cause autism. It might be difficult to answer if you don’t agree with that assumption.

The NEO has some suggestions for writing questions in their Booklet 3: Collecting and Analyzing Evaluation Data, page 5-7.

Questionnaire respondents

People think of the questions as a way to bias questionnaires, but another form of bias can be found in the questionnaire respondents.

  • Straw polls or convenience polls are polls that are given to people the easiest way. For example polling the people who are attending an event, or putting a questionnaire on a newspaper homepage (or your Facebook page).  The reason they are problematic is that they attract response from people who are particularly interested or energized by a topic, so you are hearing from the noisy minority.
  • Who you send the questionnaire to has a lot to do with why you are sending out the questionnaire. If you want to know about the opinions of people in a small club, then that’s who you would send them to. But if you are trying to reach a large number of people, you might want to try sampling, which involves learning about randomizing.  (Consider checking out the Appendix C of NNLM PNR’s Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach). Keep in mind that the potential bias here isn’t necessarily in sending the questionnaires to a small group of people, but in how you represent the results of that questionnaire.
  • Low response rate may bias questionnaire results because it’s hard to know if your respondents truly represent the group being surveyed.  The best way to prevent response rate bias is to follow the methods described in this NEO post Boosting Response Rates with Invitation Letters to ensure you get the best response rate possible.

Lastly, the Purpose of the Questionnaire

Just like looking for bias in news or health information or anything else, you want to think about is who is putting out the questionnaire and what is its purpose?  A  questionnaire isn’t always a tool for objectively gathering data.  Here are some other things a questionnaire can be used for:

  • To energize a constituent base so that they will donate money (who hasn’t filled out a questionnaire that ends with a request for donations?)
  • To confirm what someone already thinks on a topic (those Facebook polls are really good for that)
  • To give people information while pretending to find out their opinion (a lot of marketing polls I get on my landline seem to be more about letting me know about some products than really finding out what I know).

If you want to know more about questionnaires, here are some of the NEO resources that can help:

Planning and Evaluating Health Information Outreach Projects, Booklet 3: Collecting and Evaluating Evaluation Data

Boosting Response Rates with Invitation Letters

More NEO Shop Talk blog posts about Questionnaires and Surveys

 

Picture attribution

Villano by J.J., licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license.

 

NEO Announcement! Home Grown Tools and Resources

Friday, February 3rd, 2017

Red Toolbox with ToolsSince NEO (formerly OERC) was formed, we’ve created a lot of material – four evaluation guides, a 4-step guide to creating an evaluation plan, hosted in-person classes and webinars, and of course, written in this very blog! All of the guides, classes, and blogs come with a lot of materials, including tip sheets, example plans, and resource lists. In order to get to all of these resources though, you had to go through each section of the website and search for them, or attend one of our in person classes. That all changed today.

Starting now, NEO will be posting its own tip sheets, evaluation examples, and more of our favorite links on the Tools and Resources page. Our first addition is our brand new tip sheet, “Maximizing Response Rate to Questionnaires,” which can be found under the Data Collection tab. We also provided links to some of our blog posts in each tab, making them easier to find. Look for more additions to the Tools and Resources page in upcoming months.

Do you have a suggestion for a tip sheet? Comment below – you might see it in the future!

Make 2017 the “Yes-and” Year for Evaluation

Friday, January 6th, 2017

20117 (year) with fireworks and "Always Say Yes" written underneath

PSA: New Year’s resolutions are passé.

It’s out with dreary self-betterment goals involving celery and punishing exercise. Instead, the trendiest way to mark the New Year is to pick an inspirational word and make it your annual North Star for self-improvement.  In that spirit, I want to propose a word of the year for the NEO Shop Talk community.

That word is “Yes and.”

Now, some of you are wondering how I passed kindergarten with such poor counting skills. However, If you or anyone you know has taken training in Improv theatre, you know “Yes and” is a word, or, more specifically, a verb.

Improv is a type of theatre in which a team of actors make up scenes on the spot, usually from audience suggestions.  Because performances are unscripted, improv actors train rather than rehearse. Training is built around commonly accepted “rules,” and, arguably, the best known rule of Improv is “Always say Yes and…”   That means that you accept any scene idea your teammate presents and add something to make that idea better. Once novice improvisers experience the upbeat emotional effect of this rule, they soon find themselves “yes-anding” in other parts of their lives. Some even preach about it to others (ahem).  Notice, by the way, I added a hyphen so we can all feel better about “yes-and” as the WORD of the year.

If you can “yes-and” evaluation requirements and responsibilities, it will put you on the road to mastering this rule. Let’s face it, the thought of evaluation does not generate an abundance of enthusiasm. Usually we do evaluation because someone else expects or requires us to do it: upper administration; accreditation boards; funding agencies. We only do evaluation when forced because it’s a lot of work. I compare evaluation to physical exercise. In theory, we know it’s good for us. In practice, we don’t have time for it. “Yes-anding” evaluation may not make you do more evaluation than you’re required to do.  It might, however, make your evaluation responsibilities more enjoyable or, at least, more meaningful to you personally.

For example, if you have to write a proposal for external funding, you often have to pull together assessment information to build a case for your proposed program. Does that mean you have to locate and synthesize lots of data from lots of sources? Yes, and you get to demonstrate all of the great things your library or organization has to offer. You also get to point out areas where you could provide even more awesome services if the funding agency gave you funding to meet your resource needs. (Here’s a NEO Shop Talk blog post on how to use SWOT analysis to synthesize needs assessment data.)

If your proposal includes outreach into a new community, you probably have to collect information from that community.   Do you have to find and conduct key informant interviews with representatives of the community?   Yes, and you also get to initiate relationships with influential community opinion leaders. Listening is a powerful way to build trust and rapport. If you have the opportunity to implement your program, these key informants will be powerful allies when you want to reach out to the broader community. (If you want some tips for finding key informants, check out this blog post.)

You can “yes-and” internally mandated evaluation as well. Your library or organization may require you to track data on an ongoing basis or to submit regular reports. To do this well, do you have to document your daily work, such as keep track of details surrounding reference services, workshop attendance, or facility usage? Yes and, you also get to create a database of motivational information to inspire you and fellow co-workers working on the same objectives and goals. Compile that information monthly or quarterly, and pass it around at staff meetingd.  Celebrate what you’re accomplishing. Figure out where effort is lagging and commit to bolstering activities in that area. Then celebrate your team’s astute use of data for making good program decisions. Yes, and, at reporting time, be sure you present your data so that your upper-level stakeholders notices your hard work.

Maybe your department or office has to set and assess annual objectives or outcomes. Do you have to collect and report data to show program results? Yes, and you also get to demonstrate your value to the organization. Just be sure you don’t hide your candle deep in some organizational online reporting system.  Annual reports are seldom page-turners. Find more compelling ways to communicate your success and contributions to upper administrators and influential users. For some ideas, you might want to check out some of NEO Shop Talk posts on reporting and data visualization.

Another rule of Improv is “There are no mistakes, only opportunities.”  Let’s paraphrase that to “there are no evaluation requirements, only opportunities.”  Here’s to making 2017 the year of “yes-anding” evaluation.

As a post-script, I want to share a NEO Shop Talk success with our readers. Did we post weekly blog entries in 2016? Yes, and you showed up more than ever. NEO Shop Talk visits increased 71% in 2016. Each month, we averaged 259 more visits compared to the same month in the previous year.  Our peak month was February, with 892 visits! Thank you, readers. Please come back and bring your friends!

slide1

2016 Annual NEO Shop Talk Round-up

Friday, December 30th, 2016

Top 10 List

Like everyone else, we have an end-of-the-year list.  Here’s our top ten list of the posts we wrote this year, based on number of views:

10. Developing Program Outcomes Using the Kirkpatrick Model – with Vampires

9.  Inspirational Annual Reporting with Appreciative Inquiry

8.  What is a Need?

7.  Designing Surveys: Does the Order of Response Options Matter?

6.  Simply Elegant Evaluation: GMR’s Pilot Assessment of a Chapter Exhibit

5.  A Chart Chooser for Qualitative Data!

4.  W.A.I.T. for Qualitative Interviews

3.  The Zen Trend in Data Visualization

2.  How I Learned to Stop Worrying and Love Logic Models (The Chili Lesson)

1.  Logic Model for a Birthday Party

We put a lot of links to interesting things in our blog posts.  Here are the Top Ten websites that people went to from our blog:

10. The Kirkpatrick Model

9.  Books by Stephanie Evergreen

8.  Tearless Logic Model article in Global Journal of Community Psychology Practice

7.  AEA 365 | A Tip-a-Day by and for Evaluators

6.  Public Libraries, Project Outcome – Looking Back, Looking Forward

5.  Build a Qualitative Dashboard

4.  Nat King Cole, The Christmas Song

3.  The Histomap by John Sparks

2.  Tools: Tearless Logic Model (how-to summary)

1.  Stephanie Evergreen Qualitative Chart Chooser

The NEO wishes you a happy and fulfilling New Year!!

My Favorite Things 2016 (Spoiler Alert: Includes Cats)

Wednesday, December 21st, 2016

Little figurine of Santa standing in snow, holding gifts

During gift-giving season every year, Oprah publishes a list of her favorite things. Well, move over, Oprah, because I also have a list. This is my bag of holiday gifts for our NEO Shop Talk readers.

Art Exhibits

There are two websites with galleries of data visualizations that are really fun to visit. The first,  Information is Beautiful , has wonderful examples of data visualizations, many of which are interactive. My favorites from this site are Who Old Are You?   (put in your birth date to start it) and Common MythConceptions. The other is Tableau Public, Tableau Software Company’s “public commons” for their users to share their work.  My picks are the Endangered Species Safari  and the data visualization of the Simpsons Vizapedia.  And, in case  you’re wondering what happened to your favorite Crayola crayon colors, you can find out here.

Movies

Nancy Duarte’s The Secret Structure of Great Talks is my favorite TEDtalk. Duarte describes the simple messaging structure underlying inspirational speeches. Once you grasp this structure, you will know how to present evaluations findings to advocate for stakeholder support. I love the information in this talk, but that’s not why I listen to it over and over again.  It’s because Duarte says “you have the power to change the world” and, by the end of the talk, I believe her.

Dot plot for a fictional workshop data, titled Participant Self Assessment of their Holiday Skills before and after our holiday survival workshop. Pre/post self-report ratings for four items: Baking without a sugar overdose (pre=3; post-5); Making small talk at the office party (pre=1; post=3); Getting gifts through airport security (pre=2; post-5); Managing road rage in mall parking lots (pre=2; post-4)

I also am a fan of two videos from the Denver Museum of Natural History, which demonstrate how museum user metrics can be surprisingly entertaining. What Do Jelly Beans Have To Do With The Museum? shows demographics with colorful candy and Audience Insights On Parking at the Museum  talks amusingly about a common challenge of urban life.

Crafts

If you want to try your hand at creating snappier charts and graphs, you need to spend some time at Stephanie Evergreen’s blog. For example, she gives you step-by-step instructions on making lollipop charts, dot plots , and overlapping bar charts. Stephanie works exclusively in Excel, so there’s no need to purchase or learn new software. You also might want to learn a few new Excel graphing tricks at Ann Emery’s blog.  For instance, she describes how to label the lines in your graphs or adjust bar chart spacing.

Site Seeing

How about a virtual tour to the UK? I still marvel at the innovative Visualizing Mill Road  project. Researchers collected community data, then shared their findings in street art. This is the only project I know of featuring charts in sidewalk chalk. The web site talks about community members’ reactions to the project, which is also pretty fascinating.

Humor

I left the best for last. This is a gift for our most sophisticated readers, recommended by none other than Paul Gargani, president of the American Evaluation Association. It is a web site for the true connoisseurs of online evaluation resources.  I present to you the Twitter feed for  Eval Cat.  Even the  NEO Shop Talk cats begrudgingly admire it, although no one has invited them to post.

 

Pictures of the four NEO Eval Cats

 

 

 

 

 

 

 

 

 

Here’s wishing you an enjoyable holiday.

A Chart Chooser for Qualitative Data!

Friday, November 18th, 2016

Core Values Word Cloud Concept

When people talk about data visualization, they are usually talking about quantitative data. In a previous post, we explained that data visualizations help people perform three primary functions: exploring, making sense of, and communicating data.  How can we report qualitative data in a way that performs those same functions?

We just got some exciting news from the EvergreenData blog that they have developed a Qualitative Chart Chooser. Seriously–it’s a work of art. Actually two works of art because they have two different chart chooser drafts to choose from.

The way it works is this: you think about the story you want to tell with your data, maybe about how something improved over time because of your awesome project. Then using the chart chooser, you look at the “show change over time” category, and then you could select a timeline, before-and-after “change photos,” or a histomap (what’s a histomap?  Take a look at this one).

This chart chooser is a very cool tool. But I wouldn’t wait until it was time to report findings to use it. One thing that we at the NEO suggest is that when you are first planning your project, you should think about the story or stories you want to tell at the end of your project. Maybe when you’re thinking about the story you want to tell, you could look at all these different qualitative charts in the chart chooser.  Which ones would you like to use? Do you want to tell the story of how your program aligns with the goals of your institution (you could try indicator dots)? Or maybe you want to show how the different parts of your project work together as a whole (a dendrogram might work). By looking at these options before you design your evaluation plan, you can be sure that you are gathering the right data from the beginning. Backing up even further in your planning process, if you are having trouble trying to decide what story or stories you want to tell, this Qualitative Chart Chooser can give you ways to think about that.

Here is some more information on qualitative data visualization and storytelling from NEO Shop Talk:

Qualitative Data Visualization, September 26, 2014

More Qualitative Data Visualization Ideas, December 18, 2014

Telling Good Stories About Good Programs, June 29, 2015

DIY Tool for Program Success Stories, July 2, 2015

 

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.