Skip all navigation and go to page content

NEO Shop Talk

The blog of the National Network of Libraries of Medicine Evaluation Office

Archive for the ‘Practical Evaluation’ Category

The Accidental Objective

Friday, June 10th, 2016

A couple of months ago, I accidentally set an objective for the NEO Shop Talk blog.

First, let me give you my definition of an objective. I think of objectives as observable outcomes. An outcome defines the effect you want to make through a project or initiative. (You can read more about outcomes here.)  Then, you add a measure (something observable), a target (what amount of change constitutes success), and a timeframe for achieving that target.  For example, say your doctor tells you to lower your blood pressure.  She likely will suggest you make some lifestyle changes, then return in six months (timeframe) so she can take a BP reading (measure) to see if it’s below 120/80, the commonly accepted target for “normal” blood pressure.

In February 2015, Karen and I had set an outcome to increase our blog readership.  We monitored our site statistics, which we considered a measure of readership. However, we never wrote an objective or set a target. Like most of the world, we only write objectives when a VIP (such as an administrator or funder) insists. Otherwise, we are as uncommitted as the next person.

But then this happened.  I was preparing slides for a webinar on data visualization design principles and wanted to show how a benchmark line improves the meaning of data displayed in a line graph. A benchmark line basically represents your target and allows readers to compare actual performance against that target.  The best “real” data I had for creating a nice line graph was our blog’s site statistics.  But I needed a target to create the benchmark line.

NEO Blog Monthly Site Visits 2014-2016 line graph showing steady increase of monthly site visits, starting near 100 in the first month represented on the graph and and steadily increasing toward the target goal of 500. There is a straight benchmark line, going straight across the graph at the 500 mark, demonstrating our target. We hit the target once in the time frame represented on the graph.

So I made one up: 500 views per month by March 1. I did check our historical site statistics to see what looked reasonable. However, I mostly choose 500 because it was a nice, simple number to use during a presentation. I didn’t even consult Karen. She learned about it when she reviewed my webinar slides.

After all, it was a “pretend” objective.

But a funny thing happened.  As luck would have it, the NEO had nine presentations scheduled for the month of February, the month after I prepared the graph. Our new target motivated us to promote our blog in every webinar. By the end of February, we exceeded our goal, with 892 site visits.

It was game on! We started monitoring our site statistics the way cats watch a gecko. Whenever we feared we might not squeak across that monthly target line, we began strategizing about how to bump up readership. At first, we focused on promotion. We worked on building our following in Twitter, where we promoted our blog posts each week. Karen created a Facebook page so we had another social media outlet to promote our blog.

Eventually, though, we shifted our focus toward strategies for creating better content. Here were some of our ideas:

  • Show our readers how to apply evaluation in their work settings. Most of our readers are librarians, so we make a point of using examples in our articles that demonstrate how evaluation is used in library programs.
  • Demonstrate how the NEO evaluates its own program. We do posts about our own evaluation activities so that we can model the techniques we write and teach about.
  • Allow our readers to learn about assessment from each other. We work for a national network and our readers like to read about and learn from their colleagues. We now seek interviews with readers who have evaluation success stories to share.
  • Supplement our trainings. We create blog posts to supplement our training sessions. We then list relevant blog posts (with links) in our workshop and webinar resource lists.
  • Improve our consulting services. We offer evaluation consultations to members of the National Network of Libraries of Medicine. We now send them URLs to blog posts that we think will help them with particular projects.
  • Introduce new evaluation trends and tools: Both Karen and I participate in the American Evaluation Association, which has many creative practitioners who are always introducing new approaches to evaluation. We use NEO Shop Talk to pass along innovations and methods to our readers.

 

In the end, this accidental objective has improved our service.  It nudged us toward thinking of our blog contributes to the mission of the NEO and the NN/LM.

So I challenge you to set some secret objectives, telling only those who are helping you achieve that target.  See if it changes how you work.

If it does, email us. We’ll write about you in our blog.

By the way, if you want to learn how to add a benchmark line to your graphs, check out this post from Evergreen Data.

What is a Need?

Friday, June 3rd, 2016

Puzzle pieces, only the Solution piece is missing

Requests for Proposals often have a “Needs” section that usually says something like “Describe the need for your project.”  For over a decade I read hundreds of proposals for funding in my last position, and it often seemed like there was a lack of understanding of what that question meant.

As you probably know from our posts (in particular Cindy’s most recent post Steering by Outcomes: Begin with the End in Mind), we think it’s very important to plan the outcomes you want to see first. Outcomes come out of changes that you want to see happen that you discover when doing a community or needs assessment.  So for example, you do a needs assessment and find out that older adults in your hospital have lower health outcomes than those in other hospitals.  You also find out that they often do not follow their doctors’ advice. So one of your outcomes that you might want to see in your project plan would be “increased compliance with doctors’ advice” and a longer-term outcome like “older adults have better health outcomes.”

This is often where the needs description ends in proposals I have read.  People clearly demonstrate with data that there is a need for the outcomes they want to see (in this case better compliance and better health outcomes).  But as a reviewer, how do I know that there is a need for their solution to the problem?  When I think of “need for the program,” I also want to know what is lacking (or needed) in the community that their program will provide that will lead to those outcomes.

The only way for you, the project planner, to uncover that missing thing is by asking more questions of your community members and stakeholders.  In Liberating Structures Nine Whys exercise, you are encouraged to ask the question “Why is that important to you?” nine times to help you determine the fundamental purpose of what you’re doing.  This can help you clarify your outcomes.  I suggest that once you have your outcomes, then ask your community members the question “Why isn’t this already happening?” nine times to find out the core reasons that the outcomes aren’t already being met. This is what will provide the ‘need’ for your project plan, and help you design the perfect activities.

Here’s an example.  Way back in 2004 I did a site visit with a group called Healthcare for the Homeless – Houston, who were recipients of NLM funding for computer and internet access.  While I was there, the director told me a story about a needs assessment they did to find out what the barriers were for homeless people to get healthcare services.  In their needs assessment they worked together with other organizations that serve homeless people to find out how their services could be used better as well.  In their interviews with homeless people, what they found out was that people wanted to use the services but could not get to them – most of the services were spread out around the city, and without transportation it was a big deal to just get to one location, not to mention all of them. This realization spawned an innovative program called Project Access, which is a free bus service for Houston’s homeless residents that travels around to 21 agencies that provide essential services such as health care, meals, shelter, and social services.

What makes this project especially impressive to me was that they did an in-depth study to find out why the need for better healthcare wasn’t already being met. If Healthcare for the Homeless had only used data to determine that there was a need for homeless Houstonians to get better healthcare, what would have driven the choice of their activities to solve that problem?  In many projects and proposals that I’ve seen, the activities chosen were potentially good ideas, but not informed by discussions with members of the community, or in some cases had already been decided before doing the needs assessment.  My definition of “need” is the thing that is missing that, once you provide it, will logically bring about the outcomes that you want to see.

When you’re writing a proposal, or planning a project, it always comes back to the story you want to tell.  You want to tell a logical story that connects all of the dots. Whether you’re talking to a funder, an administrator, a city manager, or whoever decides whether or not you get to do your project, the story you want to tell is 1) there is a serious problem you would like to address with a project you have designed; 2) there are several specific outcomes associated with that problem that your project will accomplish, and 3) you’ve learned there is this thing that is missing that is preventing those outcomes from happening, and 4) your project is going to provide that thing.  So from the program planning perspective, you need to go out and find out what those needs are before planning your activities.

Or, to paraphrase Cindy, paraphrasing Yogi Berra, “When you come to a fork in the road, check your outcomes, then figure out why they aren’t being met, and then proceed.”

To refresh your knowledge of community assessment, take a look at NEO’s booklet Getting Started with Community-Based Outreach.

 

Worksheets and Checklists to Help with Evaluation

Friday, May 27th, 2016

You may already know that the NEO offers some booklets that work through some basic evaluation processes, called Planning and Evaluating Health Information Outreach Projects. You can view them as a PDF, an HTML or you can order a print version (there are limited copies of these left, so no promises). But I think the under-marketed gem in these booklets are the checklists and worksheets at the end of each one. And the ones in the HTML version of the Booklets are Word docs that you can download, modify if you want, and use.

Covers of the three Evaluation Booklets

For example, let’s say you’d like to create a survey to find out if you’ve reached your project’s outcomes. A process for this is explained in Booklet 3: Collecting and Analyzing Evaluation Data. At the end of this booklet is a blank worksheet called “Planning a Survey” that you can download and use to work through writing your survey questions.  Along with that, there’s also an example of a filled out worksheet based on a realistic scenario that helps demonstrate how the worksheet can be used.

The importance of checklists in improving outcomes is underscored in Dr. Atul Gawande’s book The Checklist Manifesto.  While he’s mostly talking about medical scenarios, the same can be true with evaluation.  Let’s face it, even if you feel fairly confident in evaluation, there are a lot of little things to remember, especially if you don’t do it all the time.

In Booklet 1: Getting Started with Community-Based Outreach, the checklist items are sorted into the three categories “Step 1 – Get Organized,” “Step 2 – Gather Information,” and “Step 3 – Assemble, Interpret and Act.”  These are the same categories as the chapters in the book.  So for example, one of the items under “Get Organized” is “Gather primary data to complete the picture of your target community.”  If you’d like a reminder or some suggestions of how to go about this, go to the chapter heading “Step 2 – Gather Information” where you can find a list of ways to gather primary data.  These checklists can also be downloaded as a Word document and adapted to your own needs.

I hope this isn’t too meta, but while you’re using evaluation to help you reach your project’s outcomes, you also have a personal outcome of doing a good job with your evaluation plan!  So when you head into your evaluation projects, don’t forget your checklists to make sure all of your evaluation outcomes are successful.

 

Steering by Outcomes: Begin with the End in Mind

Friday, May 20th, 2016

If you don’t know where you’re going, you might not get there – Yogi Berra

Toy car sitting on a road map

Next week, Karen and I will be facilitating an online version of one of NEO’s oldest workshops, Planning Outcomes-Based Outreach Projects for the Health Science Information Section of the North Dakota Library Association.The main tool we teach in this workshop is the program logic model, but our key takeaway message is this: Figure out where you’re going before you start driving.

If you drive to a new place, your navigator app will insist on a destination, right?  Well, I’m like an evaluation consulting app: those who work with me on evaluation planning have to define what they hope to accomplish before we start designing anything.

In fact, I get positively antsy until we nail down the desired end results.  If I’m helping a colleague develop a needs assessment, I want to know how he or she plans to use the data.  To design a program evaluation process, I have to know how the project team defines success. When consulting with others on survey design, I help them determine how each question will provide them with actionable information.

My obsession with outcomes crept into my personal life years ago. Before I sign up for continuing education or personal development workshops, I consider how they will change my life.  When my husband and I plan vacations, we talk about what we hope to gain on our trip. Do we want to connect with friends? See a new landscape? Catch up on some excellent Chicago comedy? Outcomes-thinking may be an occupational hazard for evaluation professionals. Case in point: Have you seen Karen Vargas’s birthday party logic model?

Top 5 Reasons to Love Outcomes

So how did I become an outcomes geek? Here are the top five reasons:

  • Outcomes are motivating: Activities describe work and who among us needs more work? Outcomes, on the other hand, are visionary. They allow you to imagine and bask in a job well done. Group discussions about outcomes are almost always more uplifting and enthusiastic than discussions about project implementation. Plus, you will attract more key supporters by talking about the positive benefits you hope to attain.
  • Outcomes help you focus: Once you have determined what success looks like, you’ll think more carefully about how to accomplish it.
  • Outcomes provide a reality check: Once you know what you want to accomplish, you’ll think more critically about your project plans. If the logical connection doesn’t hold, you can course-correct before you even start.
  • Planned outcomes set the final scene for your project story: Ultimately, most of us want or have to report our efforts to stakeholders, who, by definition, have a vested interest in our program. Project stories, like fairy tales, unfold in three acts: (Act 1) This is where we started; (Act 2) This is what we did; (Act 3) This is what happened in the end.  Program teams notoriously focus on collecting evaluation data to tell Act 2, while stakeholders are more interested in Act 3.  However, if you articulate your outcomes clearly from the start, you are more likely to collect good data to produce a compelling final act.
  • Identifying expected outcomes helps you notice the unexpected ones. Once you start monitoring for planned outcomes, you’ll pick up on the unplanned ones as well. In my experience, most unplanned outcomes are sweet surprises: results that no one on the team ever imagined in the planning phase.  However, you also may catch the not-so-great outcomes early and address them before any real damage is done.

How to Steer by Outcomes 

When I work with individuals or small project teams, here are the questions we address when trying to identify program outcomes:

  • What will project success look like?
  • What will you observe that will convince you that this project was worth your effort?
  • What story do you want to tell at the end of this project?
  • Who needs to hear your story and what will they want to hear?

These questions help small project teams identify outcomes and figure out how to measure them. If you want a larger group to participate in your outcomes-planning discussion, consider adapting the Nine Whys exercise from Liberating Structures.

Once the outcomes are identified, you’re ready to check the logical connection between your program strategies and your planned results. The logic model is a great tool for this stage of planning. The NEO’s booklet Planning Outcomes-Based Programs provides detailed guidance for how to create project logic models.

Yogi Berra famously said “When you come to a fork in the road, take it.”  I would paraphrase that to say “When you come to a fork in the road, check your outcomes and proceed.”

Data Party for Public Librarians

Friday, May 6th, 2016

The Engage for Health project team from left to right: Lydia Collins, Kathy Silks, Susan Jeffery, Cindy Olney

Last week, I threw my first data party. I served descriptive statistics and graphs; my co-hosts brought chocolate.

I first learned about data parties from evaluation consultant Kylie Hutchinson’s presentation It’s A Data Party that she gave at the 2016 American Evaluation Association Conference. Also known as data briefings or sense-making sessions, data parties actively engage stakeholders with evaluation findings.

Guest List

My guests were librarians from a cohort of public libraries that participated in the Engage for Health project, a statewide collaboration led by the NN/LM Middle Atlantic Region (MAR) and the Pennsylvania Library Association (PaLA). The NN/LM MAR is one of PaLA’s partners in a statewide literacy initiative called PA Forward, an initiative to engage libraries in activities that address five types of literacy.  The project team was composed of Lydia Collins of NN/LM MAR (which also funded the project), Kathy Silks of the PaLA, and Susan Jeffery of the North Pocono Public Library. I joined the team to help them evaluate the project and develop reports to bring visibility to the initiative.  Specifically, my charge was to use this project to provide experiential evaluation training to the participating librarians.

Librarians from our 18 cohort libraries participated in all phases of the planning and evaluation process.  Kathy and Susan managed our participant recruitment and communication. Lydia provided training on how to promote and deliver the program, as well as assistance with finding health care partners to team-teach with the librarians. I involved the librarians in every phase of the program planning and evaluation process. We met to create the project logic model, develop the evaluation forms, and establish a standard process for printing, distributing, and returning the forms to the project team. In the end, librarians delivered completed evaluation forms from 77% of their adult participants from Engage for Health training sessions.

What We Evaluated

The objective of PA Forward includes improving health literacy, so the group’s outcomes for Engage for Health was to empower people to better manage their health. Specifically, we wanted them to learn strategies that would lead to more effective conversations with their health care providers. Librarians and their health care partners emphasized strategies such as researching health issues using quality online health resources, making a list of medications, and writing down questions to discuss at their appointments.  We also wanted them to know how to use two trustworthy online health information sources from the National Library of Medicine: MedlinePlus and NIHSeniorHealth.

 Party Activities

Sharing with Appreciative Inquiry. The data party kicked off with Appreciative Inquiry interviews. Participants interviewed each other, sharing their peak experiences and what they valued about those experiences. Everyone then shared their peak experiences in a large group. (See our blog entries here and here for detailed examples of using Appreciative Inquiry.)

Data sense-making: Participants then worked with a fact sheet with graphs and summary statistics compiled from the session evaluation data.  As a group, we reviewed our logic model and discussed whether our data showed that we achieved our anticipated outcomes.  The group also drew on both the fact sheet and the stories from the Appreciative Inquiry interviews to identify unanticipated outcomes.  Finally they identified metrics they wish we had collected. What was missing?

Consulting Circles: After a morning of sharing successes, the group got together to help each other with challenges.  We had three challenge areas that the group wanted to address: integration of technology into the classes; finding partners from local health organizations; and promotional strategies.  No area was a problem for all librarians: some were quite successful in a given areas, while others struggled. The consulting groups were a chance to brainstorm effective practices in each area.

Next steps:  As with most funded projects, both host organizations hoped that the libraries would continue providing health literacy activities beyond the funding period.  To get the group thinking about program continuity, we ran a 1-2-4-All discussion about next steps.  They first identified the next steps they will take at their libraries, then provided suggestions to NN/LM MAR and PALA on how to support their continued efforts.

Post Party Activities

For each of the four party activities, a recorder from each group took discussion notes on a worksheet developed for the activity, then turned it into the project team. We will incorporate their group feedback into written reports that are currently in process.

If you are curious about our findings, I will say generally that our data supports the success of this project.  We have plans to publish our findings in a number of venues, once we have a chance to synthesize everything.  So watch this blog space and I’ll let you know when a report of our findings becomes available.

Meanwhile, if you are interested in reading more about data parties, check out this article in the Journal of Extension.

 

Diversity, Texas Libraries and Participatory Data Collection

Monday, May 2nd, 2016

On April 20, Cindy Olney and I facilitated a program for the Texas Library Association Annual Conference called Open Libraries! Making Your Library Welcome to All.  The program was sponsored by TLA’s Diversity and Inclusion Committee and the plan for the program was for attendees to work cooperatively to discover ways to ensure that people of diverse cultures, languages, ages, religions, sexual orientations, physical abilities, and others feel welcome at the library.  The committee wanted to get ideas from the wealth of TLA librarians’ experiences, so Cindy was invited to gather as much information from the attendees as possible. As co-chair of the TLA Diversity and Inclusion Committee, I co-facilitated the event.

The process used was a modified 1-2-4-All process, that you can find on the Liberating Structures website.  Our primary question was “What can my library do to become more welcoming to all people?”  We asked everyone in the room to brainstorm together all the different parts of a library that could be modified to make it more welcoming (e.g., reference services, facility, etc.).  We wanted to be sure that everyone thought as broadly and creatively as possible.

TLA Diversity Data Collection Program 2016

The discussion process actually had two parts.  For part one, we gave everyone two minutes to write as many ideas as they could on index cards (one idea per card).  Then we asked people to take two minutes to share their ideas with a partner.  They then shared their ideas with the entire table (up to 10 participants). The group then chose and wrote down the three best ideas and turned them in to the moderators.  Participants were instructed to leave their index cards with their ideas piled in the middle of their tables.

Here were some of the ideas that were generated through this discussion.

  • Welcome signs in different languages
  • Signage
  • Physical spaces – access to mobility

As you can see, the responses were fairly non-specific. We wanted richer descriptions of modifications of programs or services.  So part two of the process involved asking participants to develop more detailed plans for making their libraries more welcoming. Using a method involving dog, cat, and sea creature stickers, we moved participants randomly to new tables so that they ended up with a new group of colleagues.  They then chose a partner from their new table members and, as a pair, randomly chose one idea card the piles generated in part one of the process. They worked on a plan for one idea for eight minutes.  When the moderator called time, they pulled another card and worked on plans for a second idea. In the final eight minutes of the session, we asked for idea sharing by table to the entire group.

The plans in part 2 were better articulated and detailed than those we got in part one. Here are some examples of the kind of result we got from that exercise:

  • Signage: Making clearer, more colorful. Different languages signage or use digital signage.
  • Language material specific to the community and programming in various language spoken in the community. ESL classes partnered with community colleges.
  • Invite representatives from ADA/disability advocates to give suggestions on making library desks/areas more accessible.

The whole process was completed in a 50-minute conference program session.  Both myself and the other Diversity and Inclusion co-chair, Sharon Amastae from El Paso, TX, were impressed with the energy and enthusiasm that was present among attendees in the room.

The results of this data gathering event will be communicated to the TLA membership.  When that project is completed, we’ll let you know here on the NEO Shop Talk blog!

Photo credit: Esther Garcia

 

What chart should I use?

Friday, April 1st, 2016

It’s time to put your carefully collected data into a chart, but which chart should you use?  And then how do you set it up from scratch in your Excel spreadsheet or Power Point presentation if you aren’t experienced with charts?

Here’s one way to start: go to the Chart Chooser at Juice Analytics.  They allow you to pick your chart and then download it into Excel or Power Point. Then you can simply put in your own data and modify the chart the way you want to.

They also have a way to narrow down the options.  As a hypothetical example, let’s say a fictional health science librarian, Susan, is in charge of the social media campaign for her library.  She wants to compare user engagement for her Twitter, Facebook and Blog posts to see if there is any patterns in their trends. Here are some fictional stats showing how difficult it is to find trends in the data.

Monthly stats of blog, Twitter and Facebook engagement

Susan goes to the Juice Analytics Chart Chooser and selects from the options given (Comparison, Distribution, Composition, Trend, Relationship, and Table).  She selects Comparison and Trend, and then also selects Excel, because she is comfortable working in Excel.  The Chart Chooser selects two options: a column chart and a line chart.  Susan thinks the line chart would work best for her, so she downloads it (by the way, you can download both and see which one you like better).  After substituting their data with hers, and making a couple of other small design changes, here is Susan’s resulting chart in Excel, showing that user engagement with both blog posts and Facebook posts shows a pattern of increasing and decreasing at the same time, but that Twitter engagement does not show the same pattern.

Line chart of Blog Twitter and Facebook engagment

By the way, the total time spent selecting the chart, downloading it, putting in the fictional data, and making chart adjustments was less than 15 minutes.  Is it a perfect chart?  Given more time, I would suggest adjusting some more of the chart features (see our January 29, 2016 post The Zen Trend in Data Visualization). But it was a very easy way to pick out a chart that allowed Susan to learn what she needed to from the data.

One thing I want to point out is that this is not a complete list of charts.  This is a good starting place, and depending on your needs, this might be enough. But if you get more involved in data, you might want to take a look at small multiples, lollipop charts, dot plots, and other ways to visualize data.  Check out Stephanie Evergreen’s EvergreenData Blog  for more chart types.

 

Inspirational Annual Reporting with Appreciative Inquiry

Friday, March 25th, 2016

Hiker enjoying the view along the Iceberg Lake trail in Glacier National Park

Do you have to file annual reports? How much do you love doing them?

Did I hear someone say “no comment?”

In January, I challenged the outreach librarians of the National Network of Libraries of Medicine Greater Midwest Region (NN/LM GMR) to experiment with a reflective exercise designed to add some inspiration to their annual reporting. The setting was a monthly webinar attended by librarians who led outreach activities at their libraries to promote health information access and use. Because their libraries received funding from the NN/LM GMR, they were required to submit annual reports for their funded activities.

My charge was to teach this group something about evaluation. In response, I presented them with this short (about 15 minute) exercise to be used when they began preparing their annual reports.

When preparing your report, answer these questions. Then write a short paragraph based on your answers and add it to your annual report:

  1. Describe one of the best experiences you had this year conducting outreach for the NN/LM.
  2. What do you value most about that experience?
  3. What do you wish could happen so that you had more experiences like this?

You may recognize these as the three signature questions of the basic Appreciative Inquiry (AI) interview. Appreciative Inquiry is a practice of influencing organizational change by identifying peak experiences and discovering ways to build on them. The book Reframing Evaluation through Appreciative Inquiry (Preskill and Catsambas, Sage, 2006) provides descriptions and examples of how to apply AI to every part of the evaluation process.

My partners for this webinar were host Jacqueline Leskovec, Outreach, Planning and Evaluation Coordinator, and presenter Carmen Howard, who is the Regional Health Sciences Librarian and Visiting Assistant Professor from UIC Library of the Health Sciences Peoria. Carmen headlined the webinar with her presentation about the Nursing Experts: Translating the Evidence (NExT) Guide, which provides resources on evidence-based practice to nurses. Good sport that she was, Carmen helped me demonstrate the exercise to our audience by participating in an AI interview about her outreach project. The outreach librarians then brainstormed ways to use the three questions to prepare their own NN/LM reports. We also talked about how to add their reflective statements to their annual reports, which are entered into an online system.

Soon after that webinar, Carmen wrote an entry about her experience using the three questions that appeared in the NN/LM GMR’s blog The Cornflower. Here is my favorite quote from her entry:

“These three simple questions which only take about 10-15 minutes to answer forced me to stop and reflect on the NExT project. Rather than just being focused on what was next on the to-do list, I was looking back on what had already been accomplished, and better yet, I was thinking about the good stuff.”

The NN/LM GMR outreach librarians have not yet filed their 2016 annual reports, so I can’t tell you how many rose to my challenge. (This exercise was a suggestion, not a requirement.) One other outreach librarian did send an email to say she was using the three questions to have a reflective group discussion with other librarians who participate in NN/LM outreach activities.

I would like to extend the challenge to our readers who may be facing annual reports. Try this exercise and see if you don’t start thinking and writing differently about your efforts over the past year.

If you want to know more about Appreciative Inquiry, we highly recommend this source:

  • Preskill H, Catsambas TT. Reframing Evaluation through Appreciative Inquiry. Thousand Oaks, CA: Sage, 2006.

You also might be interested in the OERC’s other blog posts about Appreciative Inquiry:

If you are interested in earning some continuing education credits from the Medical Library Association while trying your hand at an Appreciative Inquiry project, reach this post: Appreciative Inquiry of Oz: Building on the Best in the Emerald City 

 

Diversity in Evaluation – It’s About All of Us

Friday, March 18th, 2016

Picture of children running through different colors with text "Diversity is about all of us, and about us having to figure out how to walk through this world together.

Unless you’ve been living under a rock, you know that culture permeates everything we do, and that we live in a diverse society with lots of different cultures. Odds are good that no matter what your job is, you take into consideration issues of culture, diversity and inclusion. This applies to evaluation as it does everywhere else.

At the 2015 Summer Evaluation Institute, each attendee was given a copy of The American Evaluation Association’s Statement on Cultural Competence in Evaluation.  I was impressed that the document was frequently mentioned, because it was clear that the AEA felt that cultural competence was central to quality evaluation. As it says in the Statement, “evaluations cannot be culture free… culture shapes the way the evaluation questions are conceptualized which in turn influences what data are collected, how the data will be collected and analyzed, and how the data are interpreted.”

The Statement describes the importance of cultural competence in terms of ethics, validity of results, and theory.

  • Ethics – quality evaluation has an ethical responsibility to ensure fair, just and equitable treatment of all persons.
  • Validity – evaluation results that are considered valid require trust from the diverse perspectives of the people providing the data and trust that the data will be honestly and fairly represented.
  • Theory – theories underlie all of evaluation, but theories are not created in a cultural vacuum. Assumptions behind theories must be carefully examined to ensure that they apply in the cultural context of the evaluation.

The Statement makes some recommendations for essential practices for cultural competence. I highly recommend reading all of the essential practices, but here are a few examples:

  • Acknowledge the complexity of cultural identity. Cultural groups are not static, and people belong to multiple cultural groups. Attempts to categorize people often collapse them into cultural groupings that may not accurately represent the true diversity that exists.
  • Recognize the dynamics of power. Cultural privilege can create and perpetuate inequities in power. Work to avoid reinforcing cultural stereotypes and prejudice in evaluation. Evaluators often work with data organized by cultural categories. The choices you make in working with these data can affect prejudice and discrimination attached to such categories.
  • Recognize and eliminate bias in language: Language is often used as the code for a certain treatment of groups. Thoughtful use of language can reduce bias when conducting evaluations.

This may sound good, but how can it apply to the evaluation of your outreach project?

Recently, the EvergreenData Blog had two entries on data visualizations and how they can show cultural bias. In the first one, How Dataviz Can Unintentionally Perpetuate Inequality: The Bleeding Infestation Example, she shows how using red to represent individual participants on a map made the actual participants feel like they were perceived as a threat. The more recent blog post, How Dataviz Can Unintentionally Perpetuate Inequality Part 2, shows how the categories used in a chart on median household income contribute to stereotyping certain cultures and skew the data to show something that does not accurately represent income levels of the different groups.

Does it sometimes feel like cultural competence is too much to add to your already full plate of required competencies? This quote from the AEA Statement on Cultural Competence in Evaluation may be reassuring: “Cultural competence is not a state at which one arrives; rather, it is a process of learning, unlearning, and relearning.”

Save

Get to Know a Community though Diffusion of Innovation (Part 2)

Friday, March 11th, 2016

two granddaughter whispering some news to their grandmother

It’s really simple. You sell to the people who are listening, and just maybe, those people tell their friends.“ — Seth Godin, marketer.

Diffusion of Innovation changed my approach to community assessment. I now focus primarily on identifying the following three things: the key problem that the program or product (the innovation) offers the target community; key community members who will benefit both from the innovation and promoting it; and the best channels for capturing the attention of the majority and laggard segments of the group.

I now purposely use the term “community” rather than “needs” assessment because you have to assess much more needs. You must learn the key components of an entire social system. A community could be faculty or students in a particular department, staff in a given organization, or an online support group of people with a challenging health condition. All of these groups fit my definition of “community” by virtue of their connectedness and ability to influence each other.

Key Evaluation Questions

No matter what type of evaluation I do, I always start with guiding evaluation questions. These questions lead my investigation and help me design data collection methods. Here are my most typical community assessment questions:

  • What problems can the innovation solve for the target audience?
  • What are their beliefs, lifestyles, and values; and will the innovation fit all three? (Marketers call these characteristics “psychographics.”)
  • Who in the group is most likely to want to use the innovation and talk about it to their friends? (These are the early adopters who fit a second category: opinion leaders.)
  • Who among the early adopters will want to work with the project team and how can we work with them?
  • Where are the best places to connect with community members?
  • What are the best ways to communicate with the larger majority?

Answering Evaluation Question

 I also have a series of steps that I usually take to gather information about my key evaluation questions.  This is my typical process:

  1. Talk to “advisors” about their ideas and their contacts. Start talking with people you know who are part of or had experience with a community. I call this group my “advisors.” They don’t have to be high-level officials, but they do need to have solid social connections. It helps if they are well liked within the target community. They will know about the daily lives of target community members, as well as the influential voices in the community. They also can help you gain access.
  2. Look at publically available data: Local media provides clues to the concerns and interests of your target community. In a town or neighborhood, newspapers and websites for television stations are good sources.  Inside an organization, you should look at public and employee relation publications to see what is on the minds of leaders and employees.
  3. Interviews with key informants: Get your advisors to recommend people they think would be early adopters and opinion leaders for your innovation. But don’t stop there. Early adopters are different from those in the “later adopter” segments. You need to talk with people from the other segments to understand how to get their attention and participation. The best way to find community members in the other segments is to ask for recommendations and introductions from the early adopters. This is called “snowball sampling.”
  4. Visit the community: More specifically, visit locations where you are most likely to connect with members of your target audience. Visit the venue of a health fair where you could exhibit.  Stop by the computer lab in an academic department where you might teach students. Check out the parking and building access at the public library or community-based organization that could host a consumer health information workshop. If your community is virtual, see if you can visit and participate with group members through their favored social media channels.
  5. Put ideas together, then present them to early adopters for feedback: If at all possible, bring together a group of early adopters and potential partners to listen to and respond to your ideas. Early adopters are the people that companies use for beta testing, so you can do the same. It may be the same people you interviewed or a different crowd (or a mix).

Other Tips

For the most part, I tend to rely heavily on qualitative methods for community assessment. Diffusion of Innovation describes how ideas spread through a system of relationships and communication channels. You need details to truly understand that system.  You need to talk to people and understand how they live. Interviews, focus groups, and site visits provide the most useful planning information, in my opinion. You may have to include quantitative data, though.  Library usage statistics might indicate the best branches for providing workshops.  Short surveys might confirm broad interest in certain services. In the end, a blend of mixed-methods gives you the best picture of a community.

The downside to mostly qualitative data collection methods is that you get an overwhelming amount of information. I like to use an information sheet that allows me to summarize information as I conduct a community assessment. A version of this worksheet is available in OERC’s Planning and Evaluating Health Information Outreach Projects Booklet 1: Getting Started with Community-Based Outreach (downloadable version available here.). See Worksheet 2 on page 19. You should see how the evaluation questions I posed above are related to this worksheet.

Final Thoughts

Seth Godin said ideas that spread are remarkable, meaning they are “worth making a remark about.” Use community assessment to find out why your innovation is remarkable and how to start the conversation.

Other Resources

If you want to see an example of a community assessment that used this process, check out this article in Public Libraries.

You also might be interested in  Seth Godin’s TEDtalk How to Get Your Ideas to Spread.

Last updated on Monday, June 27, 2016

Funded by the National Library of Medicine under Contract No. UG4LM012343 with the University of Washington.