Booklet 2 - Planning Outcomes-Based Outreach Projects

Title Page & Verso information
Figures

Preface

This booklet is part of the Planning and Evaluating Health Information Outreach Projects series designed to supplement Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach [1]. This series also supports evaluation workshops offered through the Outreach Evaluation Resource Center of the Network of the National Library of Medicine. The goal of the series is to present step-by-step planning and evaluation methods.

The series is aimed at librarians, particularly those from the health sciences sphere, and representatives from community organizations who are interested in conducting health information outreach projects. We consider "health information outreach" to be promotional and educational activities designed to enhance community members' abilities to find and use health information. A goal of these activities often is to equip members of a specific group or community to better address questions about their own health or the health of family, peers, patients, or clients. Such outreach often focuses on online health information resources such as the websites produced by the National Library of Medicine. Projects may also include other sources and formats of health information.

We strongly endorse partnerships among organizations from a variety of environments, including health sciences libraries, hospital libraries, community-based organizations and public libraries. We also encourage broad participation of members of target outreach populations in the design, implementation, and evaluation of the outreach project. We try to describe planning and evaluation methods that accommodate this participatory approach to community based outreach. Still, we may sound like we are talking to project leaders. In writing these booklets we have made the assumption that one person or a small group of people will be in charge of initiating an outreach project, writing a clear project plan, and managing the evaluation process.

Booklet 1 in the series, Getting Started with Community Assessment, is designed to help you collect community information to assess need for health information outreach and the feasibility of conducting an outreach project. Community assessment also yields contextual information about a community that will help you set realistic program goals and design effective strategies. It describes three phases of community assessment:

1. Get organized,
2. Collect data about the community, and
3. Interpret findings and make project decisions.

The second booklet, Planning Outcomes-Based Outreach Projects, is intended for those who need guidance in designing a good evaluation plan. By addressing evaluation in the planning stage, you are committing to doing it and you are more likely to make it integral to the overall project. The booklet describes how to do the following:

1. Plan your program with a logic model,
2. Use your logic model for process assessment, and
3. Use your logic model to develop an outcomes assessment plan.

The third booklet Collecting and Analyzing Evaluation Data, presents steps for quantitative methods (methods for collecting and summarizing numerical data) and qualitative methods (specifically focusing on methods for summarizing text-based data.) For both types of data, we present the following steps:

1. Design your data collection methods,
2. Collect your data,
3. Summarize and analyze your data, and
4. Assess the validity or trustworthiness of your findings.

Finally, we believe evaluation is meant to be useful to those implementing a project. Our booklets adhere to the Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation [2]. Utility standards, listed first because they are considered the most important, specify that evaluation findings should serve the information needs of the intended users, primarily those implementing a project and those invested in the project's success. Feasibility standards direct evaluation to be cost-effective, credible to the different groups who will use evaluation information, and minimally disruptive to the project. Propriety standards uphold evaluation that is conducted ethically, legally, and with regard to the welfare of those involved in or affected by the evaluation. Accuracy standards indicate that evaluation should provide technically adequate information for evaluating a project. Finally, the accountability standards encourage adequate documentation of program purposes, procedures, and results.

We sincerely hope that you find these booklets useful. We welcome your comments, which you can email to one of the authors: Cindy Olney olneyc@uw.edu or Susan Barnes at sjbarnes@uw.edu

Acknowledgements

We deeply appreciate Cathy Burroughs' groundbreaking work, Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach, and thank her for her guidance in developing the Planning and Evaluating Health Information Outreach Projects series as a supplement to her publication. We also are grateful to our colleagues who provided feedback for the first edition of the series.

To update the series, we were fortunate to work with four reviewers who brought different viewpoints to their critiques of the booklets. We want to thank our reviewers for their insightful suggestions:

• Molly Engle, PhD, Professor and Extension Evaluation Specialist, Oregon State University, Corvallis, OR
• Sabrina Kurtz-Rossi, MA, Health Literacy Consultant and Principal, Kurtz-Rossi & Associates, Boston, MA
• Annabelle Nunez, MA, Assistant Librarian, Arizona Health Sciences Center, Tucson, AZ
• Ann Webb Price, PhD, President, Community Evaluation Solutions, Alpharetta, GA

This project has been funded in whole with federal funds from the Department of Health and Human Services, National Institutes of Health, National Library of Medicine, under Contract No. HHS-N-276-2011-00008-C with the University of Washington.

Introduction

"You got to be very careful if you don't know where you're going because you might not get there" - Yogi Berra¹

In the first booklet of this series, Getting Started with Community-Based Outreach [3], we discussed community assessment and provided a three-phase approach based on the process described by Altschuld in his Needs Assessment Kit [4]. The community assessment helped you determine the health information needs of the community, the community resources that would support a health information outreach project, and information to guide you in your choice and design of outreach strategies. While working on the community assessment, you and your partners probably came up with thoughts about what you could accomplish through health information outreach. New outreach projects can be exciting, especially when the project team has innovative ideas and a wave of enthusiasm for making a difference in a community.

The title of this booklet, Planning Outcomes-Based Outreach Projects, communicates the importance of first establishing your intended results, or outcomes, and then designing a project that will logically lead to those results. Why such a focus on outcomes? Think about the times you have either taken or taught a class: You know from experience that students study to the test. In project planning, if you first identify your intended results, you are more likely to plan and conduct activities that will get you there. Also, you need to articulate your outcomes so you are sure to measure them and demonstrate to others the effectiveness of your project. Foundations, businesses, and other agencies now value outcomes assessment because they know you have to gather outcomes data to distinguish a successful program from one that is failing. Outcomes measures allow you to reward program success and correct or eliminate failing projects. The mere act of articulating and tracking outcomes improves the likelihood that they will be achieved.

In this booklet, we help you plan for success by using a logic model. The logic model is a visual representation of a program that illustrates how planned activities are linked to program results. (See Figure 1 and Figure 2). It is a simple planning tool that encourages you to first articulate your outcomes for your health information outreach project. You then work on the plan by identifying the necessary project components-what we invest, what we do and who we reach-that will result in those outcomes. Finally, you identify your assumptions about the program and external factors that may impact your ability to achieve your outcomes.

There are numerous sources for designing logic models, with many available on the Internet, including materials from the Free Management Library [5], the Network of the National Library of Medicine [6], University of Wisconsin-Extension [7], the US Government Accountability Office [8], and the W.K. Kellogg Foundation [9].

A well-constructed logic model can make it easier to plan outreach programs by helping you determine the resources and activities you need to achieve the results you want. Logic models aid decision-making about new projects, assist with structuring evaluations of new and ongoing programs, and contribute to developing funding proposals and reports.

In articulating your outcomes, you are answering the "so what" question about the importance of what you are doing. Outcomes are also at the foundation of measurable objectives that guide you in collecting and reporting evidence of your accomplishments. This booklet will help you use your logic model to create outcome objectives that identify indicators and targets to guide your data collection plans and document your achievements.

Step One - Plan Your Program with a Logic Model

Logic models come in many different formats, but they all present the shared perspective of an "if...then" statement: "If we obtain the necessary resources and conduct certain activities, we will achieve our desired outcomes." In Figure 3, we present a basic logic model template that includes examples of typical health information outreach project inputs, activities, and outcomes. Inputs are the resources we need for the outreach project, including people, time, money, materials, equipment, and technology. Activities include what we do-conduct training sessions, provide services and who is reached-participants, agencies, community-based organizations. Outcomes are the results or benefits of your project, including short-term outcomes such as changes in knowledge, intermediate outcomes such as changes in behavior, and long-term outcomes such as changes in individuals' health or medical access, social conditions or population health.

A logic model is read from left to right: You use certain resources to conduct planned activities that lead to desired results. However, you complete a logic model from right to left by starting with the outcomes columns and planning backward to the resources column. In other words, logic models promote use of one of Stephen Covey's highly effective habits: "Begin with the end in mind."²

Start with Outcomes

Begin your logic model by listing the intended outcomes (or results) in the last three columns. Outcomes are stated with an emphasis on the project recipients, such as "the participants will increase their ability to find information about health topics they hear about through the media" or "the agency staff will improve their ability to find health information for their clients." 

This can be the most challenging part of developing a logic model. It might help to think of short-term and intermediate outcomes as stepping stones to the long-term outcome. First, the short-term outcomes column portrays benefits that individual participants will gain from your program, such as increased knowledge or improved satisfaction. Second, the intermediate outcomes column describes actions that participants are predicted to make as a result of their learning such as changing behavior or making decisions. Finally, the long-term outcomes column is used to describe higher-impact results. These results may be related to an individual, such as improved health outcomes. They often refer to positive changes in the community, organization or system. Since your project may only last 12-18 months, you may not be able to achieve system-level changes before the project ends. However, it is important to articulate long-term outcomes as part of your overall plan so you understand the "big picture" of what you want to achieve. Completing this column also will help you decide whether you need to collect baseline data about changes that you are not expecting until far into the future. See Figure 4 for examples of different types of outcomes. 

Trying to determine which outcomes are short-term, intermediate, or long-term can be confusing. One longstanding outcomes evaluation model that has guided trainers in many different fields is the Kirkpatrick Model, shown in Figure 5 [10]. 

In his research about effective training, Kirkpatrick visualizes outcomes of training activities in four levels that build upon each other. His model describes a chain of outcomes. At level 1, the outcome is satisfaction: People like your training. While satisfaction or "happiness" is not a very strong impact, Kirkpatrick believes you have to attract or impress people to motivate them to learn. The course evaluation forms that solicit participants' feedback after health information training workshops are a typical level 1 measure. If you conduct a MedlinePlus® [11] training and participants tell you they liked the resource and found it useful, you have achieved a level 1 outcome. 

At level 2, training participants learn something. For example, if you train people to use MedlinePlus and you end the session by having them complete an exercise that demonstrates they can find correct answers to questions by using the website, you are testing a level 2 outcome. 

The third level of outcome-behavior-is also known as "transfer." When people take the skill they learned in an educational setting and apply it in their daily lives, they have "transferred" the skill. In a health care setting, for example, health professionals would use what they learned to improve their clinical decision-making. In health information outreach, people who learn about MedlinePlus and then turn to it when they encounter a new health problem or are placed on a new medication are demonstrating a level 3 outcome. 

The highest level outcomes are those that occur as a result of changes in individuals' knowledge or behavior. Most health information outreach is motivated by a desire to contribute to high-impact results such as improved health outcomes, increased community health literacy, more favorable community health indicators, and improved societal public health. For the MedlinePlus training example, a long-term impact may be that better-informed patients will improve their management of chronic health problems and will require less expensive treatment in the long run. Not only do patients benefit health-wise, but this also can improve the cost-effectiveness of the health care system. It can be very difficult to prove a direct cause-and-effect link from a MedlinePlus class to improved health literacy, but it is logical to argue that improved health information skills learned in the class were contributing factors. 

Essentially, level 1 addresses the "did they like it?" question, and level 2 addresses the "did they learn it?" question. Both satisfaction and learning can be measured immediately after you have completed a workshop or learning activity. Because level 1 outcomes usually are not particularly compelling to program stakeholders, they often are not listed as project outcomes. Level 2 outcomes, however, would be added to the short-term outcomes column. 

Level 3 outcomes, which show changes in behavior, must be measured at some point after your initial activities with participants. Therefore, you list those in the intermediate outcomes column. Level 4 outcomes, often referred to as "impacts," may require more time and a combination of health information outreach and other health-related interventions to achieve. These are listed in the long-term outcomes column. 

Notice that as you transition from level 1 to level 4, the outcomes become more impressive. At the same time, they also get more difficult to measure. Reaction and learning can be assessed in the classroom. Longer-term effects require follow-up beyond course evaluation forms. 

Connect Activities to Outcomes

Once you have filled in the outcomes columns, it is time to plan the activities that are most likely to lead to your expected results. It might be helpful to review the process people go through when deciding to adopt an innovation-a new product, resource, or behavior (See Figure 6). In Booklet 1 of this the series, we described the Diffusion of Innovation approach [12] to describing factors that influence individuals' adoption of innovations. The five-stage process shown in Figure 6, below, begins when people become aware of a new product or resource (knowledge). They gather more information to form an attitude toward the product (persuasion). If they develop favorable attitudes, they decide whether to use the innovation (decision). They then start applying the product in their daily lives (implementation). Even after they establish a habit of using an innovation, they are constantly comparing it to alternatives to see whether another innovation may meet their needs better (confirmation). 

As you can see, they are forming and re-forming their opinion about the product as they move through the five stages. At every stage, they seek information, which ultimately affects whether they adopt the new resource. You should choose the health information outreach strategy that is best suited to the stage of innovation adoption that you want to target. (It is not unusual for health information projects to include several strategies targeting multiple stages.) To choose the activities for your project, consider the stages of the people you plan to work with and plan outreach activities accordingly: 

• Knowledge-Participants may never have considered getting consumer health information online or they may not realize that online health information resources vary in quality or that they can improve the efficiency of their search skills. Your job is to make them aware of alternatives to what they are currently using. Exhibits are typical outreach strategies aimed at increasing awareness or knowledge about health information resources.
• Persuasion-Some people are aware of the products you are promoting but are still forming an attitude toward them. Maybe they're looking over their kids' shoulders. Maybe a friend has told them about online resources. Your strategies should help them weigh the advantages of using your resources against their alternatives. Exhibits, presentations, and demonstrations are a way for you to talk about the positive aspects of health information resources.
• Decision-If they perceive that the advantages of using your resources outweigh the disadvantages, they are likely to try out, or adopt, the resources you are promoting. Training sessions provide an opportunity for individuals to not only learn about the resources but to actually have an opportunity to use them and, if it is a group session, talk with peers about how they might apply the resources in their daily lives.
• Implementation-At this stage, people will apply the resources in various ways in their lives. Your job is to help them realize the range of uses of the resources and support their at-home experimentation. Providing one-to-one support, such as through a help desk or follow-up contact, is a strategy directed toward implementation.
• Confirmation-As people start using your resources, they will start to discover the limitations of those resources. You may want to show them a broader range of online resources to meet more specific needs or suggest other ways to use the resources. For example, they may not have considered they could get information about clinical trials through online resources. Providing advanced search skills or presentations on more specialized resources are good strategies for this stage of innovation adoption.

The activities section of the logic model also has a column for "reach," where you indicate the groups you hope to engage in activities. If you remember, Diffusion of Innovation suggests targeting the innovators, early adopters, and opinion leaders. As noted in Booklet 1, activities directed toward these groups can be particularly efficient because others in the community form their attitudes toward innovations by watching and listening to the early users. Your community assessment should have helped you identify groups in the community that have the potential to fit these influential roles. This is the column to record those groups. 

Identify Inputs Needed to Conduct Activities

Once you have identified your activities, you are now ready to list the resources you need to conduct them in the inputs column. This step is relatively self-explanatory: you list what you need to implement activities effectively. The main point is to be realistic. It is easy to underestimate the investment needed to do a good outreach project. 

Finish with a Reality Check

The assumptions box in the logic model forces you to revisit your plan to see whether you are taking anything for granted. For example, you may be assuming that you will be able to recruit colleagues to teach training sessions during your project period or that training facilities will be available during hours that are convenient to your target community. (See Figure 7, above.) If you identify those assumptions, you can then verify them. 

The external factors box allows you to list barriers and challenges that you are aware of that may impact your ability to conduct your project. (You should have identified such barriers in your community assessment.) You also will list influences in this box that may have a positive impact on your project. For example, you may know that the community public library is focusing on promotion of health awareness in a given month, so you can incorporate information of your resources into its promotional materials and monthly activities. 

Get Input from Your Team of Advisors

In Booklet 1 of the series, we talked about encouraging participation of stakeholders by identifying a team of advisors. You should apply the team-of-advisors approach in project planning as well, although you might not use the same team members. 

Ideally, a logic model is created with participation of representatives of all stakeholder groups (such as librarians, representatives from participating agencies, and clients who will receive services through the project). Here is a recommended process for facilitating a group through the logic model process: 

1. Write the overall goal at the top of the logic model. The goal is the general purpose of the project, such as "To improve community members' ability to find information that helps them manage their health."
2. In logic models, outcomes determine activities, so outcomes columns should be completed first. However, most people's first ideas are triggered by activities that they intuitively know would be beneficial or even resources they need for those activities. It is okay to start with the column that you know best to initiate conversation, but you should identify very quickly the outcomes you expect to accomplish. Avoid listing more than two activities without discussing outcomes.
3. You do not have to complete one column before going on to the next. It might be easier to think across "rows," working through all the related activities and short term, intermediate and long-term outcomes. You might want to use breaks in the columns to signify rows that indicate which activities and outcomes belong together. Add resources as you think of them, but don't worry about listing all of them now because you may need the same resources for multiple activities.
4. After you have completed the activities and outcomes columns, think about what resources you will need to accomplish your project. This information belongs in the inputs column. Column breaks may not be useful here because you may use the same resources for a number of activities.
5. When the columns are completed, identify the factors you are taking for granted and list them in the assumptions box.
6. Identify the external factors that may have a positive or negative effect on your project (including your ability to obtain funding). Are there resources you can leverage? Do you have data that document the needs of the target population for your intervention? Are there any barriers you may be facing? List these in the external factors box.

In reality, developing the first draft of a logic model is complicated and time-consuming, so it often is drafted by a small working team and then revisited later by a more inclusive outreach team or advisory group. This approach is fine, as long as the logic model is viewed as a flexible plan that can be revisited by a larger group at a later time.

In fact, a logic model should be revisited and revised periodically throughout the outreach project. As you conduct your project, you are likely to adapt your strategies as you learn what works well and what doesn't. You should change the logic model to reflect changes in your activities, but keep earlier versions so that you can see what has changed. 

Logic models can be helpful at other stages of projects. Figure 8 shows how logic models can help with writing proposals and Figure 9 describes how they can contribute toward structuring final reports. In this booklet, we are focusing on using logic models with new programs. Logic models can also help you to evaluate existing services. Whether a program is new or already in place, the focus on outcomes encourages you to explore whether activities are worth the investment or whether they have made a difference to your stakeholders. Appendix 1 shows more examples of how logic models can be helpful at many stages of a project.

Step Two - Use Your Logic Model for Process Assessment

After you have completed your logic model, you move on to developing your evaluation plan. In this booklet, we discuss two different types of assessments to include in evaluation planning. Process assessment, described in Step Two, is conducted to track program progress, quality and lessons learned. Outcomes assessment, covered in Step Three, allows you to measure the results of your project strategies. 

Plan Ahead for Data Collection

Step Two describes how to plan a process assessment of your project. Your process assessment allows you to describe the implementation of your plan as described in your logic model. It serves two important functions. First, process assessment guides the fine-tuning of your project as it progresses. Then, when you have completed your project, you use your process assessment data to help analyze the strengths and weaknesses of your project plan. You will have a better understanding of the strongest features of your project and the aspects of the project that may have been ineffective or even prevented you from attaining the best possible results. In other words, process assessment will help you tell the "lessons learned" of your project story [13].

When you tell the story of your project, you probably will want to be able to report the following about your project implementation:

The extent to which the community was ready for health information outreach Your ability to access the inputs necessary for conducting the project The degree to which you were able to implement your project as planned The quality of your activities The challenges and barriers you encountered while conducting your program

Community readiness and access to inputs should have been explored to some extent during your community assessment. Review the items in your assumptions and external influences boxes of your logic model to see whether there are details you will need to investigate further. You may need to interview stakeholders, key informants, and project participants. You also may need to review records, such as attendance at events or facilities held at different times of the day (to see when activity is greatest) or enrollment numbers in different projects offered at outreach sites. An inventory of resources may require interviews and site visits as well. You will be looking at resources like staff availability during the project, meeting places, technology centers, events where you can exhibit online resources, and so on. Your assessment also may include reading the literature or talking with other outreach teams to get an idea of what you will need for your project. Your community assessment information also may need to be updated through exploration of the following:

What has changed since your community assessment information was gathered? Do you still have the support of the community or organizational leaders? (Some places have high turnover and you may need to reintroduce yourself to new executive directors or staff.) Are there things you need to know about the facilities identified in your plan? Is there parking? Are the computers working? Is the training room easy to find? Are the people you were counting on to help you still available? Who are the program participants? Are they similar to the community members you met during your community assessment?

Conduct Audience Analysis

Your community assessment probably did not provide details about the specific participants that you plan to recruit for your activities. Audience analysis should be part of each structured activity you plan (e.g., training sessions). For instance, you might be able to learn about health topics of particular interest to each group, so you can tailor your activities to include those topics. You will want to know the level of experience and comfort with computers and the Internet for the average participant in your training session. Of course, you will always have people at varying levels of experience and ability in any group, but you must determine and design for the "average" participant if you are providing some form of structured group training. Pre-training session questionnaires are an ideal way to conduct an audience analysis, but it can be difficult to locate your participants in advance of the workshop or to motivate them to answer questionnaires. Another approach is to explore participants' information needs with key informants. 

Track Progress, Make Needed Changes, and Identify Lessons Learned

Process assessment methods geared toward monitoring degree of implementation, quality of activities, and project barriers and challenges are conducted as activities are implemented. You collect information to make sure you are completing activities as expected and that strategies are working effectively. Your assessment plan allows you to check that you are meeting your deliverables in a timely manner, to identify problems pointing to a need to alter course, and to get feedback from others about their experiences with the activities.

Process assessment usually includes tracking the number of classes held and the number of participants in training sessions. These numbers help you quantify the degree of implementation of your project activities. Your process assessment plan should include a system for keeping basic records of numbers describing attendance rates, exhibit visitors, promotional materials distributed, and one-to-one services provided.

Process assessment also requires detailed, descriptive information for making project adjustments along the way. For this reason, qualitative methods often are emphasized in process assessment. Interviews, focus groups, observations, and written responses to open-ended questions on project feedback forms are popular process assessment methods. Interviewees often include users, project staff and staff from participating community-based partners, and users. These methods will help you know how well your activities are being received and what barriers are preventing them from being effective.

Figure 10 presents some typical process assessment questions and methods for addressing them.

Process assessment is meant to be used for program improvement. Therefore, be sure to analyze your data as you collect it. Share it with those who are helping you implement your project, such as project staff and staff in your community-based partner organization. (For ideas of how to interpret the data you collect, see Booklet 3 [14] of this series, Collecting and Analyzing Evaluation Data.) It is okay if your findings lead you to adjust your strategies or your expected results. Health information outreach is a learning experience. Do not be afraid to reassess your logic model and revise it midway through the project. However, keep earlier versions of the logic model; seeing how it changes over time can be part of the process assessment.

Step Three - Use Your Logic Model to Develop an Outcomes Assessment Plan

Identify Quantifiable Indicators

The outcomes assessment plan helps you determine methods for collecting objective evidence of the results of your project. Outcomes themselves, such as attitudes, knowledge, or skills, are often not directly observable. Outcomes are sort of like illnesses. Those of us who are of a "certain age" cannot see our arthritis, but we are pretty sure we have it when our knees ache during overexertion or a shift in the weather. You identify some observable signs that you can logically argue are evidence of the outcome. These are known as outcome indicators. 

For example, your short-term outcome for a training class might be "participants will increase their ability to find out about their medications." You really cannot directly observe "increased ability." So you ask yourself "What outward signs will show that people will know how to find online information about their medications?" You decide to have a quiz at the end of the training session where you give participants a list of prescription drugs and ask them to find the side effects. You could then argue that those participants who can answer all or most of the questions are skilled enough to find drug information about their own medications. You may also ask them directly in a training session evaluation whether the session has improved their ability to get information about their prescription drugs. An affirmative response is an indicator of increased ability.

Note that, of the two indicators, the quiz may be more believable as evidence of acquired skill. However, the self-report indicator may be easier to use. Your choice of indicators must strike a balance between what is credible and what is feasible. An imperfect indicator is better than no indicator at all because it still serves to keep you focused on your intended results. You and your stakeholders have to make an educated judgment about the credibility of your indicators. Also, collecting a couple of indicators for one outcome helps you see a pattern that supports your outcomes more strongly. (But you do not have to overdo it.)

It is preferable to have some countable (or quantifiable) indicators in your outcomes assessment. Stakeholders often want to know the degree of change, such as "how much knowledge was gained?" or "how many more volunteers are helping clients get health information?" (However, as will be discussed later, qualitative methods such as testimonials can make the number more compelling to stakeholders.)

Choose a Target and Time Frame

Once you have identified quantifiable indicators, you then choose a target (the threshold or level that must be attained to determine success) and time frame (the point in time that when the threshold for success will be achieved). In the previous quiz example, your target is the percentage of people who will get a passing grade on your quiz. Your time frame states the point at which you expect to achieve the objective. If you think that it is reasonable to expect that your target can be met after one workshop, your time frame is "by the end of one workshop." Each objective should be achievable given your time and resources and the priorities of those involved in the project. Figure 11 shows examples of outcomes, indicators, and objectives.

Write Objectives

Now you are ready to write the objectives, which are statements that pull together indicators, targets, and time frames for each outcome. For examples of building objectives from outcomes, see Figure 12 and Figure 13. These are two different examples of objectives. Figure 12 shows one using "success criteria" as the target and Figure 13 shows "change over time" as the target. Once you have written the objectives, review them for feasibility. Each objective should be achievable given your time and resources and the priorities of those involved in the project.

Specifying criteria for success can be a challenge. If you are fortunate, others will have conducted projects similar to yours and published their outcomes. Their work can help you set reasonable expectations for your own project. A second approach is to get feedback from stakeholders as to what they would consider an acceptable "return on investment" of time and resources.

Your expectations should vary based on how long your project has been in place. If you are engaged in a pilot project, small changes may be adequate because your primary goal is to initiate your project. If you are involved in expansion of a pilot project, stronger results may be expected to justify continued investment of resources. As difficult as it can be to specify targets for your results, it is an important exercise in setting goals for your project and should not be avoided. However, be careful not to set your targets too high because you may not be able to achieve them given the typical time and resource constraints faced by outreach teams.

Create an Action Plan

After defining your objectives, put together a more specific plan of action for evaluating each outcome. You will want to detail your data source, evaluation method, and data collection time line to assess your indicator. Data sources refer to the location of your information. Often, data sources are people (such as participants or observers) but they also may be records, pictures, or meeting notes. (see Appendix 2 for examples of data sources). Evaluation methods are the tools you use to collect data, such as a survey, observation, or quiz. (See Appendix 3 for examples of methods.) You also can find examples of sources and methods in Measuring the Difference [1] and in Booklet 3 of this series.

Data only make sense in context, so you need to think about how you will analyze your findings. There are two basic approaches. First, the "success criteria" approach involves comparing your data against the targets you listed in your outcome objectives. For instance, referring back again to our medication quiz example, if you set a goal that 80% of participants in your outreach project can find drug information by the end of your training session, then you can analyze your findings by comparing them against your target. You can claim success if you exceed your target. Figure 12 shows how to evaluate an outcome using a "success criteria" approach.

On the other hand, your objective may be a target that specifies improvement, such as the objective in Figure 13. This objective would require a baseline assessment, in which you determine the number of visits to the library's NLM resource web page from library computers prior to conducting your training sessions. This analytic method often has a drawback. Once you have started your activities, it is too late to get a good baseline assessment. For our medication quiz example, if you do not plan ahead of time, you will not know how many people could find good drug information before your training session, so you will not be able to talk about improved skill among participants.

There are exceptions, of course. For instance, web traffic reports or customer databases that are collected on an ongoing basis allow you to compare data before and after the program. Your outcome evaluation plan, however, will help you determine in advance whether you do, in fact, need to collect baseline data in a narrow window of opportunity. Therefore, before you begin, it is very important to review your objectives and create an evaluation plan to ensure that you collect the baseline data you need.

This section has emphasized collecting quantitative data to show evidence of outcomes. Quantitative methods are best for assessing objectives. However, many of us know that educational or promotional efforts often lead to outcomes that you never imagined were possible. The details of how people use your resources can be very compelling to you and your stakeholders. There also is a chance that there are negative outcomes that you will not pick up in your assessment of the planned outcomes specified in your logic model. You may discover, for example, that public library staff members who learned how to help library patrons find health information are feeling stress over the kind of health problems suffered by their patrons. For this reason, many outcomes assessment plans build in qualitative methods to gather detailed stories about their participants' experience with health resources. Booklet 3 of this series gives some examples of qualitative methods for collecting stories. You can also refer to the Centers for Disease Control and Prevention's publication Impact and Value: Telling your Program's Story [15] for additional ideas for collecting and presenting stories about your project.

Take-Home Messages

Evaluation should be incorporated into your overall health information outreach plan. Your logic model will provide the evaluation plan framework for the process and outcome assessment. If integrated into the project plan, evaluation becomes more meaningful and useful both to the outreach team and stakeholders. The steps for planning a project with an integrated evaluation plan are the following:

1. Plan your project with a logic model that links inputs and activities to outcomes. Think of the logic model as a flexible document. Revisit and revise it often as your project develops, but keep track of earlier versions so that you can see what has changed.
2. Develop a process assessment plan to monitor the degree and quality of implementation of your plan.
3. Write measurable objectives based on the short-term and long-term outcomes columns from the logic model.
4. Develop a reasonable data collection and analysis plan for outcomes evaluation based on your measurable objectives.

References

1. Burroughs C. Measuring the difference: Guide to planning and evaluating health information outreach [Internet]. Seattle, WA: Network of the National Library of Medicine, Pacific Northwest Region; 2000 [cited 28 Feb 2012].
2. Yarbrough, DB; Shulha, LM; Hopson, RK; and Caruthers, FA. The Program Evaluation Standards: A Guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage, 2011.
3. Olney CA, Barnes S. Planning and Evaluating Health Information Outreach Projects. Booklet 1: Getting Started with Community Assessment. Seattle, WA: Network of the National Library of Medicine Outreach Evaluation Resource Center; Bethesda MD; 2012.
4. Altschuld JW, Kumar DD. Needs assessment: an overview. Book 1. Altschuld, JW (ed.) Needs assessment kit. Los Angeles, CA: Jossey-Bass; 2010.
5. McNamara C. Guidelines and framework for designing basic logic model [Internet]. St. Paul, MN: Free Management Library; 2000 [cited 1 March 2012]. http://managementhelp.org/freenonprofittraining/diagramming-your-nonprofit.htm​ [new link 13 February 2017]
6. Network of the National Library of Medicine. Guide 5: define how a program will work: the logic model. In: Guides for Developing a Community Health Information Program [Internet]. Bethesda, MS: NN/LM; 2006 July 28 [cited 1 March 2012]. http://nnlm.gov/libinfo/community/logicmodel.html [link will take you to archived page]
7. Taylor-Powell E, Jones L, Henert E. Enhancing program performance with logic models [Internet]. Madison, WI: University of Wisconsin-Extension; 2003 Feb [updated 9 February 2012].https://fyi.extension.wisc.edu/programdevelopment/files/2016/03/lmcourseall.pdf [new link 10 July 2019]
8. US Government Accountability Office. Designing evaluations 2012 revision (GAO-12-208G) [Internet]. Washington, DC: GAO; 2012 Jan [cited 9 Feb 2012]. http://www.gao.gov/products/GAO-12-208G [new link 5 December 2016]
9. WK Kellogg Foundation. Logic model development guide: Using logic models to bring together planning, evaluation, and action. Battle Creek, MI: the Foundation; 2004 [cited March 1, 2012]. https://www.wkkf.org/resource-directory/resources/2004/01/logic-model-development-guide [new link 5 November 2020]
10. Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: The four levels. 3rd ed. San Francisco: Berrett-Koehler; 2006.
11. US National Library of Medicine. MedlinePlus®: Trusted Health Information for You. [Internet] Bethesda, MD: NLM; [cited 3 March 2012]. http://medlineplus.gov
12. Rogers, EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003.
13. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: A how-to guide. Health Promot Pract, 2005 Apr 26; 6(2):134- 147.
14. Olney, CA, Barnes S. Planning and evaluating health information outreach projects Booklet 3: Collecting and analyzing evaluation data. Seattle, WA: Network of the National Library of Medicine Outreach Evaluation Resource Center; 2012.
15. Lavinghouze SR, Price AW. Impact and value: Telling your program's story [Internet]. Atlanta, GA: Centers for Disease Control and Prevention; 2007. [cited 2 Mar 2012] http://www.cdc.gov/oralHealth/publications/library/pdf/success_story_workbook.pdf

Appendix 1 - How to Use a Logic Model at Different Stages of Your Project

Proposal

Logic Model Benefit

• Allows the proposal writers to plan thoroughly
• Helps the writers anticipate funders' questions and prepare answers for inclusion in the proposal
• Provides a structure for writing the proposal

Who Should Be Involved

• Principal investigator
• Project implementers
• Evaluation consultant (if available)

Planning

Logic Model Benefit

• Brings all stakeholders into the planning process
• Builds relationships between outreach and community-based organization (CBO) staff
• Allows different stakeholder groups to consider assumptions and identify potential barriers
• Helps outreach staff find ways to integrate into the CBO's services and projects
• Provides a benchmark plan against which progress can be evaluated

Who Should Be Involved

Representatives from the following stakeholder groups:

• All partners on the outreach team And if possible,
• Agency staff and volunteers
• Clients
• Evaluation consultant (if available)
• Advisory committee, if one exists
• Funders

Implementation

Logic Model Benefit

• Allows group to check project progress against the plan
• Provides convenient method for updating new team members
• Allows group to re-examine assumptions and influences and revise plan
• Allows alterations in short- and long-term outcomes based on actual implementation

Note: Good timing for this review might be quarterly or whenever reports are due to a funding agency

Who Should Be Involved

• Outreach team
• CBO partners (anyone involved in project)
• Advisory committee, if one exists

Reporting Stage

Logic Model Benefit

Provides a structure for reporting evaluation data. Reporting suggestions:

• Compare actual implementation against plan
• Report evidence of outcome achievement
• Add unexpected outcomes

Who Should Be Involved

• Outreach team
• Evaluation consultant, if available

Note: Feedback about final report should be obtained from CBO partners and, if possible, participants

Appendix 2 - Examples of Data Sources

Source: People

• Participants
• Noncompleters
• Project implementers (e.g., workshop facilitators, help desk staff )
• Administrators or agency staff
• Volunteers
• Experts (for instance, consultants trained to review training materials or websites)
• Advisory committees
• Key informants (anyone with special knowledge about a project, organization, or community)
• Community leaders
• Funders

Source: Existing Information

• Project records, such as attendance sheets from workshop sessions
• Meeting minutes
• Written material kept by people involved in a program, such as journals or notes from a training session
• Email or bulletin board discussions
• Existing databases, such as demographic information about participants who have received services from an agency
• Web traffic reports
• Newspaper articles

Source: Observations

• Observations of users navigating websites
• Videotapes of group discussions or meetings
• Products created by participants or stakeholders, such as handouts or items developed by organizational staff to promote consumer health websites
• Newsletters
• Pictures, such as photos taken of students helping their parents use MedlinePlus

Taylor-Powell E, Steele S. Collecting Evaluation Data: an Overview of Sources and Methods. [Internet]. Madison, WI: University of Wisconsin-Extension, 1996 [cited 24 February 2012] http://www.racialequitytools.org/resourcefiles/G3658_4.pdf [Link updated July 18, 2019]

Appendix 3 - Examples of Evaluation Methods

Method: Surveys

Questionnaires administered in a standardized manner to collect primarily quantitative data.

Method: Interviews

Question-and-answer sessions with individuals. Some interviews are highly structured while others may be very unstructured.

Method: Focus Groups

Interviews with a group of people. Usually individuals who are chosen for a group share some similarities, such as age, profession, level of computer experience, or role in an organization (managerial, support staff, or volunteer).

Method: Testimonials

Brief accounts that individuals may give about an experience. For instance, you may ask individuals to give a brief account of how they used MedlinePlus the week following a training session.

Method: Logs

Descriptions of events, such as the number of visitors to a health fair exhibit, or participants at a training session or a daily list of health topics brought by library users to a public library reference desk.

Method: Document Analysis

Review and summary of written material pertinent to a project. Examples of documents include organizational newsletters, meeting minutes, or existing evaluation documents created during an organizational self-study.

Method: Tests

Exams of individuals' skills or knowledge. Tests may have forced-choice questions (multiple choice or true-false) or open-ended questions such as "name three different types of information you can find at MedlinePlus."

Method: Reflective Exercises

These are questions that are designed to get participants to reflect on experiences. They may be used on a one-time basis (like at the end of a training session) or they may be used to structure participants' entries to a journal or bulletin board discussion.

Method: Expert or Peer Review

One or a group of people review products or presentations to judge their quality. The people may either be experts such as public health experts reviewing health information promotional materials) or they may be peers (such as outreach librarians observing and rating their colleagues' consumer health training sessions). Usually the review process is structured to assure systematic and thorough feedback.

Taylor-Powell E, Steele S. Collecting Evaluation Data: an Overview of Sources and Methods. [Internet]. Madison, WI: University of Wisconsin-Extension, 1996 [cited 24 February 2012] http://www.racialequitytools.org/resourcefiles/G3658_4.pdf [Link updated July 18, 2019]

Toolkit: Senior Day Care Center Outreach Project

The following case study is designed to demonstrate how to develop a logic model and an evaluation plan. For this case, activity directors from an agency that provides day care services to older adults partner with health sciences librarians to design a health information outreach project targeted for participants in their program. In this project, MedlinePlus training is offered to program participants, agency staff, and family members of the older adults. The goal of the program is to improve older adults' access to consumer health information, both by improving their skills and training others who can help them. 

We first present completed worksheets, based on the case example described here. The first worksheet presents a logic model for this project. The second worksheet presents a process assessment plan. The third worksheet presents three objectives and evaluation plans for those objectives. 

Blank versions of these worksheets are included for you to use in your project planning. 

Worksheet 1 - Logic Model, example

Program: Outreach for Older Adults
Goal: Help increase older adults' access to and use of high-quality consumer health information

Inputs	Activities Column 1	Activities Column 2	Outcomes Column 1	Outcomes Column 2	Outcomes Column 3
What we invest	What we do	Who we reach	Why we do it: Short-term results	Why we do it: Intermediate results	Why we do it: Long-Term results
Health science librarians to conduct training Senior center partners Internet access at senior centers Senior center staff to support educational activities	Six training sessions for older adults enrolled in the day care program Two training sessions per senior center staff member on accessing information on MedlinePlus (M+) or other NLM resources Monthly follow-up training for senior center staff Two monthly training sessions for family members (or other loved ones) at varying times (day, evening)	50% of older adults in the program receive training 80% of day care center staff members receive training 80% of family members receive training	Older adults and senior center staff members improve knowledge health information resources Older adults improve knowledge about their prescription drugs	Older adults will use online health information resources to research future health concerns Activity staff will become a resource for older adults to help them get health information to prepare for doctor's visits	Older adults improve communication with physicians about health concerns M+ training becomes a regular part of the older adult day care program activities Providing M+ assistance will be part of one staff member's job description M+ training will be incorporated into orientation for new staff

Assumptions

• Staff will be interested in helping older adults with online health information research
• Family members will be willing to attend training sessions

External Factors

• (+)The senior center recently upgraded its computers
• (-) There appears to be considerable turnover in mid-level agency staff

Worksheet 2 - Process Assessment Questions and Methods, example

Process Questions	Information to collect	Methods
To what extent were you able to implement your project as planned?	How well did the librarians and center staff follow procedures in the plan? What factors increased or decreased the quality of delivery?	Focused staff feedback sessions Observations of activities
To what extent were you able to conduct specific activities as they were planned?	How many training sessions were offered?	Counts of training sessions for older adults and family members Checklists for staff to report what resources they demonstrated or taught
How much community interest and activity did your project generate?	How many older adults and family members attended the training sessions? How many older adults requested assistance?	Attendance counts for training sessions Feedback forms from participants asking them to evaluate their experience Activity staff assistance counts
To what extent did you reach your intended community?	What proportion of older adults participated?	Percentage of senior day care participants
How effective were your recruitment strategies for attracting community members?	What strategies worked well to attract older adults and family members and what barriers impacted recruitment? What strategies helped you maintain participant involvement as needed and what barriers did you face?	Written feedback forms asking users what attracted them to activities Counts of participants who completed all activities (e.g., all sessions of a multi-day training) Feedback sessions with project staff Interviews with participants
What situational factors in the environment, community, or organizations affected project implementation?	What influenced librarians’ and center staff members’ ability to implement the project? What influenced older adults’ reactions to the program or their ability to participate?	Focus groups with senior center staff Focus groups with older adults Interviews with librarians

Worksheet 3 - Objectives and Methods for Assessing Them, examples

Objective 1: At least 80% of older adults who receive training on MedlinePlus will report one or more things they learned about their health concerns at the end of their last training session.

Measurable Indicator: Number of older adults who can give examples of what they learned
Target: 80% will be able to report at least one thing they have learned
Time frame: Immediately after the last training session

Data Source	Evalutation Method	Data Collection Timing
Participating older adults	Individual interviews by facilitators during the lunch session that follows the class	Immediately after last training session

Objective 2: At least 50% of trained older adults will report discussing information from MedlinePlus with their health care providers when interviewed three months after training.

Measurable Indicator: % of trained older adults who report discussing information from MedlinePlus with their health care providers
Target: 50% will report discussing information from MedlinePlus with their health care providers
Time frame: Three months after training

Data Source	Evalutation Method	Data Collection Timing
Participating older adults	Survey administered by outreach team member (The survey will be read to older adults if necessary)	Three months after last training session

Objective 3: By the end of the project, at least 25% of older adults in the lunch program will have one family member who has attended MedlinePlus training.

Measurable Indicator: % of older adults with a family member trained on MedlinePlus
Target: 25% of older adults will have a trained family member
Time frame: By the end of the project

Data Source	Evalutation Method	Data Collection Timing
Training attendance records	Family members will be asked to identify themselves on training session attendance sheets	Beginning of each training session

Blank Worksheet 1 - Logic Model

Booklet 2, Worksheet 1: Logic Model 

Blank Worksheet 2 - Process Assessment Questions and Methods

Booklet 2, Worksheet 2: Process Assessment 

Blank Worksheet 3 - Objectives and Methods for Assessing Them

Booklet 2, Worksheet 3: Objectives and Methods

Checklist

Step 1 - Plan Your Program with a Logic Model

• Develop a logic model to show how your planned work (inputs and activities) will lead to your intended results (short-term, intermediate, and long-term outcomes).
• List your assumptions underlying your logic model.
• Identify the positive external influences that will support your project and the negative external influences that may pose barriers.

Step 2 - Use Your Logic Model for Process Assessment

• Confirm community readiness and access to inputs.
• Design audience analysis as part of each structured activity.
• Collect information to monitor progress.
• Make project adjustments along the way.

Step 3 - Use Your Logic Model to Develop an Outcomes Assessment Plan

• Identify indicators for outcomes on your logic model (indicators are observable behaviors, such as responses to a survey question or performance on a quiz).
• For each indicator, write an objective that includes a target (criterion for success) and a time frame for achieving that target.
• Establish where you will collect information about the indicators (e.g., among participants or from documents such as attendance records), how you will collect it (survey, interview, document review) and when (before and after the training session; one month after the training session).