Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for June, 2016

Party Guides for Data Dives

Friday, June 24th, 2016

Children hands holding numbers. Black isolated multicolor numbers.

Children who grow vegetables are more likely to eat them. Likewise, stakeholders who have a hand in interpreting evaluation data are more likely to use the findings.

Traditionally, the data analysis phase of evaluation has been relegated to technical experts with research skills. However, as the field sharpens its focus on evaluation use, more evaluators are working on developing methods to engage groups of stakeholders in data analysis. While evaluation use is one objective of this approach, evaluators also are compelled to use participatory data analysis because it

  • Provides context for findings and interpretations that include multiple viewpoints
  • Generates stakeholder interest in the evaluation process
  • Determines which findings are most important to those impacted by a program

Last week, Karen and I attended Participatory Sense-making for Evaluators: Data Parties and Dabbling in Data, a webinar offered by Kylie Hutchinson of Community Solutions  and Corey Newhouse of Public Profit. They shared their strategies for getting groups of stakeholders to roll up their sleeves and dive elbow-deep in data. Such events are referred to as data parties, sense-making sessions, results-briefings, and data-driven reviews. Hutchinson also shared her data party resource page on (more…)

Design Principles in Evaluation Design

Friday, June 17th, 2016

Robot and human hands almost touching

“Sometimes… it seems to me that… all the works of the human brain and hand are either design itself or a branch of that art.” Michelangelo

Michelangelo is not the only one who thinks design is important in all human activities.  In his book A Whole New Mind, Dan Pink considers design to be one of the 6 senses that we need to develop to thrive in this world. As Mauro Porcini, PepsiCo’s Chief Design Officer points out “There is brand design. There is industrial design. There is interior design. There is UX and experience design. And there is innovation in strategy.” ¹

There is also evaluation design. Whether we’re talking about designing evaluation for an entire project or just one section, like the needs assessment or presenting evaluation results, evaluators are still actively involved in design.

Most of us don’t think of ourselves as designers, however.  Juice Analytics has a clever tool called “Design Tips for Non-Designers” to teach basic design skills and concepts.  Some of these are very specific design tips for charts and power points (which by the way are very important and useful, like “avoiding chart junk” and “whitespace matters”).  But some of the other tips can be jumping off points for thinking about bigger picture design skills, such as:

  • Using Hick’s Law and Occam’s Razor to explain the importance of simplicity
  • Learning how to keep your audience in mind by thinking of how to persuade them, balancing Aristotle’s suggested methods of ethical appeal (ethos), emotional appeal (pathos), and logical appeal (logos)
  • Learning how Gestalt theory applies to the mind’s ability to acquire and maintain meaningful perceptions in an apparently chaotic world
  • Considering the psychology of what motivates users to take action

The September 2015 issue of Harvard Business Review highlighted design thinking as corporate strategy in their spotlighted articles (which are freely available online, as long as you don’t open more than 4 a month).  Here are some cool things you can read about in these articles:

  • Using design thinking changed the way PepsiCo designed products to fit their users’ needs (my favorite line is how they used to design products for women by taking currently existing products and then applying the strategy of “shrink it or pink it.”)
  • Design is about deeply understanding people.
  • Principles of design can be applied to the way people work: empathy with users, a discipline of prototyping and tolerance for failure.
  • Create models to explain complex problems, and then use prototypes to explore potential solutions.
  • If it is likely that a new program or strategy may not be readily accepted, use design principles to plan the program implementation.

Some people are seen as being born with design skills.  But it’s clear that a lot can be learned with study and practice.  Even Michelangelo said, “If people knew how hard I worked to get my mastery, it wouldn’t seem so wonderful after all.”


¹ James De Vries. “PepsiCo’s Chief Design Officer on Creating an Organization Where Design Can Thrive.Harvard Business Review. 11 Aug 2015.  Web. 17 June 2016.

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

The Accidental Objective

Friday, June 10th, 2016

A couple of months ago, I accidentally set an objective for the NEO Shop Talk blog.

First, let me give you my definition of an objective. I think of objectives as observable outcomes. An outcome defines the effect you want to make through a project or initiative. (You can read more about outcomes here.)  Then, you add a measure (something observable), a target (what amount of change constitutes success), and a timeframe for achieving that target.  For example, say your doctor tells you to lower your blood pressure.  She likely will suggest you make some lifestyle changes, then return in six months (timeframe) so she can take a BP reading (measure) to see if it’s below 120/80, the commonly accepted target for “normal” blood pressure.

In February 2015, Karen and I had set an outcome to increase our blog readership.  We monitored our site statistics, which we considered a measure of readership. However, we never wrote an objective or set a target. Like most of the world, we only write objectives when a VIP (such as an administrator or funder) insists. Otherwise, we are as uncommitted as the next person.

But then this happened.  I was preparing slides for a webinar on data visualization design principles and wanted to show how a benchmark line improves the meaning of data displayed in a line graph. A benchmark line basically represents your target and allows readers to compare actual performance against that target.  The best “real” data I had for creating a nice line graph was our blog’s site statistics.  But I needed a target to create the benchmark line.

NEO Blog Monthly Site Visits 2014-2016 line graph showing steady increase of monthly site visits, starting near 100 in the first month represented on the graph and and steadily increasing toward the target goal of 500. There is a straight benchmark line, going straight across the graph at the 500 mark, demonstrating our target. We hit the target once in the time frame represented on the graph.

So I made one up: 500 views per month by March 1. I did check our historical site statistics to see what looked reasonable. However, I mostly choose 500 because it was a nice, simple number to use during a presentation. I didn’t even consult Karen. She learned about it when she reviewed my webinar slides.

After all, it was a “pretend” objective.

But a funny thing happened.  As luck would have it, the NEO had nine presentations scheduled for the month of February, the month after I prepared the graph. Our new target motivated us to promote our blog in every webinar. By the end of February, we exceeded our goal, with 892 site visits.

It was game on! We started monitoring our site statistics the way cats watch a gecko. Whenever we feared we might not squeak across that monthly target line, we began strategizing about how to bump up readership. At first, we focused on promotion. We worked on building our following in Twitter, where we promoted our blog posts each week. Karen created a Facebook page so we had another social media outlet to promote our blog.

Eventually, though, we shifted our focus toward strategies for creating better content. Here were some of our ideas:

  • Show our readers how to apply evaluation in their work settings. Most of our readers are librarians, so we make a point of using examples in our articles that demonstrate how evaluation is used in library programs.
  • Demonstrate how the NEO evaluates its own program. We do posts about our own evaluation activities so that we can model the techniques we write and teach about.
  • Allow our readers to learn about assessment from each other. We work for a national network and our readers like to read about and learn from their colleagues. We now seek interviews with readers who have evaluation success stories to share.
  • Supplement our trainings. We create blog posts to supplement our training sessions. We then list relevant blog posts (with links) in our workshop and webinar resource lists.
  • Improve our consulting services. We offer evaluation consultations to members of the National Network of Libraries of Medicine. We now send them URLs to blog posts that we think will help them with particular projects.
  • Introduce new evaluation trends and tools: Both Karen and I participate in the American Evaluation Association, which has many creative practitioners who are always introducing new approaches to evaluation. We use NEO Shop Talk to pass along innovations and methods to our readers.

 

In the end, this accidental objective has improved our service.  It nudged us toward thinking of our blog contributes to the mission of the NEO and the NN/LM.

So I challenge you to set some secret objectives, telling only those who are helping you achieve that target.  See if it changes how you work.

If it does, email us. We’ll write about you in our blog.

By the way, if you want to learn how to add a benchmark line to your graphs, check out this post from Evergreen Data.

What is a Need?

Friday, June 3rd, 2016

Puzzle pieces, only the Solution piece is missing

Requests for Proposals often have a “Needs” section that usually says something like “Describe the need for your project.”  For over a decade I read hundreds of proposals for funding in my last position, and it often seemed like there was a lack of understanding of what that question meant.

As you probably know from our posts (in particular Cindy’s most recent post Steering by Outcomes: Begin with the End in Mind), we think it’s very important to plan the outcomes you want to see first. Outcomes come out of changes that you want to see happen that you discover when doing a community or needs assessment.  So for example, you do a needs assessment and find out that older adults in your hospital have lower health outcomes than those in other hospitals.  You also find out that they often do not follow their doctors’ advice. So one of your outcomes that you might want to see in your project plan would be “increased compliance with doctors’ advice” and a longer-term outcome like “older adults have better health outcomes.”

This is often where the needs description ends in proposals I have read.  People clearly demonstrate with data that there is a need for the outcomes they want to see (in this case better compliance and better health outcomes).  But as a reviewer, how do I know that there is a need for their solution to the problem?  When I think of “need for the program,” I also want to know what is lacking (or needed) in the community that their program will provide that will lead to those outcomes.

The only way for you, the project planner, to uncover that missing thing is by asking more questions of your community members and stakeholders.  In Liberating Structures Nine Whys exercise, you are encouraged to ask the question “Why is that important to you?” nine times to help you determine the fundamental purpose of what you’re doing.  This can help you clarify your outcomes.  I suggest that once you have your outcomes, then ask your community members the question “Why isn’t this already happening?” nine times to find out the core reasons that the outcomes aren’t already being met. This is what will provide the ‘need’ for your project plan, and help you design the perfect activities.

Here’s an example.  Way back in 2004 I did a site visit with a group called Healthcare for the Homeless – Houston, who were recipients of NLM funding for computer and internet access.  While I was there, the director told me a story about a needs assessment they did to find out what the barriers were for homeless people to get healthcare services.  In their needs assessment they worked together with other organizations that serve homeless people to find out how their services could be used better as well.  In their interviews with homeless people, what they found out was that people wanted to use the services but could not get to them – most of the services were spread out around the city, and without transportation it was a big deal to just get to one location, not to mention all of them. This realization spawned an innovative program called Project Access, which is a free bus service for Houston’s homeless residents that travels around to 21 agencies that provide essential services such as health care, meals, shelter, and social services.

What makes this project especially impressive to me was that they did an in-depth study to find out why the need for better healthcare wasn’t already being met. If Healthcare for the Homeless had only used data to determine that there was a need for homeless Houstonians to get better healthcare, what would have driven the choice of their activities to solve that problem?  In many projects and proposals that I’ve seen, the activities chosen were potentially good ideas, but not informed by discussions with members of the community, or in some cases had already been decided before doing the needs assessment.  My definition of “need” is the thing that is missing that, once you provide it, will logically bring about the outcomes that you want to see.

When you’re writing a proposal, or planning a project, it always comes back to the story you want to tell.  You want to tell a logical story that connects all of the dots. Whether you’re talking to a funder, an administrator, a city manager, or whoever decides whether or not you get to do your project, the story you want to tell is 1) there is a serious problem you would like to address with a project you have designed; 2) there are several specific outcomes associated with that problem that your project will accomplish, and 3) you’ve learned there is this thing that is missing that is preventing those outcomes from happening, and 4) your project is going to provide that thing.  So from the program planning perspective, you need to go out and find out what those needs are before planning your activities.

Or, to paraphrase Cindy, paraphrasing Yogi Berra, “When you come to a fork in the road, check your outcomes, then figure out why they aren’t being met, and then proceed.”

To refresh your knowledge of community assessment, take a look at NEO’s booklet Getting Started with Community-Based Outreach.

 

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.