Planning & Evaluating
GETTING STARTED WITH
Title Page & Verso information
This booklet is part of the Planning and Evaluating Health Information Outreach Projects series designed to supplement Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach . This series also supports evaluation workshops offered through the Outreach Evaluation Resource Center of the National Network of Libraries of Medicine. The goal of the series is to present step-by-step planning and evaluation methods.
The series is aimed at librarians, particularly those from the health sciences sphere, and representatives from community organizations who are interested in conducting health information outreach projects. We consider "health information outreach" to be promotional and educational activities designed to enhance community members' abilities to find and use health information. A goal of these activities often is to equip members of a specific group or community to better address questions about their own health or the health of family, peers, patients, or clients. Such outreach often focuses on online health information resources such as the websites produced by the National Library of Medicine. Projects may also include other sources and formats of health information.
We strongly endorse partnerships among organizations from a variety of environments, including health sciences libraries, hospital libraries, community-based organizations and public libraries. We also encourage broad participation of members of target outreach populations in the design, implementation, and evaluation of the outreach project. We try to describe planning and evaluation methods that accommodate this participatory approach to communitybased outreach. Still, we may sound like we are talking to project leaders. In writing these booklets we have made the assumption that one person or a small group of people will be in charge of initiating an outreach project, writing a clear project plan, and managing the evaluation process.
Booklet 1 in the series, Getting Started with Community Assessment, is designed to help you collect community information to assess need for health information outreach and the feasibility of conducting an outreach project. Community assessment also yields contextual information about a community that will help you set realistic program goals and design effective strategies. It describes three phases of community assessment:
The second booklet, Planning Outcomes-Based Outreach Projects, is intended for those who need guidance in designing a good evaluation plan. By addressing evaluation in the planning stage, you are committing to doing it and you are more likely to make it integral to the overall project. The booklet describes how to do the following:
The third booklet, Collecting and Analyzing Evaluation Data, presents steps for quantitative methods (methods for collecting and summarizing numerical data) and qualitative methods (specifically focusing on methods for summarizing text-based data.) For both types of data, we present the following steps:
Finally, we believe evaluation is meant to be useful to those implementing a project. Our booklets adhere to the Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation . Utility standards, listed first because they are considered the most important, specify that evaluation findings should serve the information needs of the intended users, primarily those implementing a project and those invested in the project's success. Feasibility standards direct evaluation to be cost-effective, credible to the different groups who will use evaluation information, and minimally disruptive to the project. Propriety standards uphold evaluation that is conducted ethically, legally, and with regard to the welfare of those involved in or affected by the evaluation. Accuracy standards indicate that evaluation should provide technically adequate information for evaluating a project. Finally, the accountability standards encourage adequate documentation of program purposes, procedures, and results. We sincerely hope that you find these booklets useful. We welcome your comments, which you can email to one of the authors: Cindy Olney at firstname.lastname@example.org or Susan Barnes at email@example.com.
Return to Table of Contents
We deeply appreciate Cathy Burroughs' groundbreaking work, Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach, and thank her for her guidance in developing the Planning and Evaluating Health Information Outreach Projects series as a supplement to her publication. We also are grateful to our colleagues who provided feedback for the first edition of the series.
To update the series, we were fortunate to work with four reviewers who brought valuable different viewpoints to their critiques of the booklets. We want to thank our reviewers for their insightful suggestions:
This project has been funded in whole with federal funds from the Department of Health and Human Services, National Institutes of Health, National Library of Medicine, under Contract No. HHS-N-276-2011-00008-C with the University of Washington.
Return to Table of Contents
"As we look ahead into the next century, leaders will be those who empower others." - Bill Gates1
The next century is here, and there is a great need for leaders to help others use health information resources and technology tools to manage their health. The importance of health information use is recognized in the Healthy People 2020's (HP2020) Health Communication and Health Information Technology (Health IT) topic area, which states that "the effective use of communication and technology by health care and public health professionals can bring about an age of patient- and public-centered health information and services . Anticipated outcomes are improved health care quality, safety, and efficiency; enhanced public health information infrastructure; increased support care in the community and at home; and improved health skills, knowledge, and clinical and consumer decision-making.
The complexity of the health information environment, however, is also recognized in HP 2020: "With the increasing complexity of health information and health care settings, most people need additional information, skills, and supportive relationships to meet their health needs." Although Internet use has become commonplace , findings from the Pew Internet & American Life Project and other research projects show potential barriers to effectively using health information resources and health IT:
The Pew Internet and American Life Project also notes trends that can be leveraged to promote health information use:
Thus, there are plenty of opportunities for leaders in the movement to increase and enhance the use of health information and technology. Although HP2020 identifies health care providers and public health professionals as leading the charge in health information and IT use among patients, there is evidence that health professionals themselves need support in navigating the complex health IT environment . There also is a need to reach beyond the populations seen regularly by health professionals to general consumers who do not regularly use health care systems for various reasons. Librarians and information professionals in partnership with staff from other community-based organizations, therefore, can play critical roles in the health information revolution. One of the most important of these roles is to reach out to health professionals and consumers to provide assistance and training with navigating the health IT environment. To develop good strategies for doing that, begin with a community assessment to learn about a community's health information needs as well as its resources that can support outreach efforts. This booklet is designed to help carry out that assessment.
The term "outreach" specifically refers to formal activities designed to reach populations (or communities) outside of one's own institution . As Dutcher and Hamasu noted, "community-based outreach requires that we rely on partners from the local community to serve as our guides through the process of developing relationships, identifying needs and becoming trusted associates." .
In this booklet, we present a systematic approach to community assessment, which is outlined in Figure 1. Community assessment allows you to explore a potential outreach community and form alliances in the process. Your primary goal is to piece together a picture of a community so that you can effectively plan a project. However, the community assessment process also brings you into contact with community members and leaders, as well as those who serve the community in many capacities. The community assessment process is an important step in building relationships that will support a health information outreach project. Finally, the information you gain through community assessment, assuming your findings support a need for health information outreach, will also help you advocate for support from others who can provide you with the resources you need for an effective project.
If you are new to community-based health information outreach, you may not be ready to actually plan a project. In fact, you may want to immerse yourself in a community of interest to you (often referred to as a target community). Dutcher and Hamasu  described a panel discussion among representatives of five different communities of the types often targeted in health information outreach efforts. The panelists suggested those who want to engage in health information outreach begin visiting and getting to know the target community before they have a well-developed plan. Get to know the leaders. Engage in community events. Begin to learn and build trust with community members. Figure 2 gives some examples of how to start exploring a community that interests you. It is important to keep a record of the contacts you make and what you learn in this preliminary investigation. The Toolkit provides a worksheet for keeping track of community contacts.
On the other hand, you may have already connected with members of a certain community but want to explore ways to help them improve their health information use. If you would like ideas for effective community-based outreach strategies, a good source is the Journal of the Medical Library Association http://www.mlanet.org/publications/jmla/. You also can read about health outreach initiatives, which may help you understand how to reach your target community. PubMed Central http://pubmedcentral.gov  is an excellent source of open-access journal articles about health outreach. You also could read brief descriptions about health information outreach projects funded by the National Library of Medicine by visiting the National Network of Libraries of Medicine Projects Database at http://nnlm.gov/ funding/database.html .
Conducting a community assessment is a lot like completing a jigsaw puzzle: You have to locate pieces of information and put them together to form a picture of your target community. The goals of community assessment, adapted from Harris , are the following:
The community assessment process outlined in this booklet is modified from a three-step model of needs assessment described by Wilkins and Altschuld :
Return to Table of Contents
Community assessment begins by defining a community of interest. Possibly you have an idea of an outreach project and you want to see how well it will work in a given community. Maybe you recently learned of a group that would benefit by becoming more skilled users of health information resources. Or maybe you became aware of funding opportunities to reach specific groups. Then again, your idea may be rooted in a mandate from a higher authority, like your institution or a government entity, to provide service to a specific group.
Once you have identified your community of interest, called the "target community," there are a number of steps to take to organize your community assessment project. Be sure to allow yourself sufficient time to get organized, which includes doing the following steps:
Project stakeholders are those who have a "stake" in the success of your program. Different stakeholder groups may have very different interests and values. Your team of advisors should include representatives of all stakeholder groups, because you will need their ongoing support if you move forward with a project. To build their support, you need to create a picture of the community that answers stakeholders' questions as well as your own. Stakeholder groups may include:
For an example of potential stakeholders for a fictitious project, see Figure 3.
Representatives of stakeholder groups can help you identify the major issues facing your health information outreach project and determine the scope of the community assessment project. Those from the community can serve as "key informants," who can help you understand the target community and help you gain access to community members. These stakeholder-advisors also may know of local databases that can help you or refer you to experts who have worked in their communities and can share effective strategies for outreach. Finally, representatives from each stakeholder group will ensure that you address the concerns and values of those who have a stake in the program.
It also is a good idea to learn from others who have worked in similar target communities. You can find examples of health information outreach and health promotion projects conducted by others in the academic literature. As noted previously, PubMed Central is an excellent source for articles about health information outreach and health promotion projects.
Most health information outreach projects have goals related to behavior change. For example, we often want people to start researching their own health issues or to become more critical in choosing online health information. Behavior change is complicated: There are a number of psychological, social, and environmental influences that can make or break behavior change. The goal of community assessment is to identify these influences.
Theoretical models of behavior change organize findings from individual studies into a cohesive theory of cause and effect. They can be invaluable guides in helping you understand the type of information you need to plan a community-based outreach project. Many public health initiatives are based on behavior change models, a number of which are described in Measuring the Difference .
For community-based outreach projects, we prefer the Diffusion of Innovation model . This model incorporates not only the psychological influences over behavior change, but also the social, interpersonal, and environmental factors as well. This model specifically describes the process by which people adopt anything new: products, resources, habits, or policies. These new things are referred to as "innovations."
The "newness" of a product or resource is relative to the individual. When individuals start using a new consumer health database or change how they use online health information, they are adopting an innovation. For example, if you introduce MedlinePlus to a group of parish nurses and they begin to use it to provide customized health information to their patients, they have "adopted" the innovation.
The process by which individuals adopt an innovation (called the innovation-decision process) is illustrated in Figure 4. In the Knowledge stage, individuals first become aware of an innovation. In the Persuasion stage, they seek more information to form a positive or negative attitude about the innovation. In the Decision stage, they decide whether to try it. If the experience is positive, they start using the innovation. However, they continue in the Confirmation stage to gather information about the new product and evaluate it, continuing to compare it against alternatives .
Five innovation traits affect individuals' adoption. [See Figure 5 for descriptions.] For example, people will be more likely to adopt MedlinePlus if they perceive it is more convenient or yields better information compared with their alternative sources of health information (high relative advantage). If use of MedlinePlus is compatible with their computer use and their use of health information, it will be more likely integrated into the community (high compatibility). If community members find MedlinePlus easy to navigate (low complexity), they will be more inclined to adopt it. The fewer resources it takes to be able to try out MedlinePlus (trialability), the more likely they will be to try it. If they observe others using MedlinePlus and benefiting from its use (observability), they will be more compelled to use the website.
People gather information about an innovation through all stages of adoption, relying on media, and social channels of communication. Media and experts are useful in creating initial knowledge about an innovation, but social interactions with peers carry more influence over attitudes and ultimate decisions to adopt something new. Adoption of innovations is greatly influenced by positive reports from community "opinion leaders." These are members in the community whom others rely on for judgments and advice on innovations. Opinion leaders, if they like your product, provide very inexpensive promotion of innovations. They become role models and vocal advocates to the community, which will pay attention to what they say .
The spread of innovations within a community occurs in waves. Innovators, the first 2.5% of users, are the gatekeepers who introduce innovations to a community. An example might be a high school librarian who decides what health information sources to bookmark on the school's computers. Early adopters are the second wave of 13.5% of users. This early adopter group is often sought after in public health initiatives as "local missionaries" for an innovation because their opinions are respected by the larger community and they serve as role models. They are said to be more "integrated" into a social system, meaning they are closer to the "average" user than an innovator. The "train-the-trainer" strategy that is used in many health information outreach projects is based on the "early adopter" factor in Diffusion of Innovation: You train people who you believe can reach others.
Promoting health information resources is similar to lighting a fire. If you can get early adopters to use your product, its use will eventually spread spontaneously from them to the rest of the community with little effort on your part.
Diffusion of Innovation has been described here in detail because it gives you an idea of the type of information you need to gather about a community to plan an effective outreach project. Figure 6 presents some strategies for identifying innovators and early adopters. Figure 8 provides evaluation questions that you might ask to understand how to introduce an innovation into a community.
Once you have identified your team of advisors and brought yourself up to speed on relevant literature and theoretical models, it is time to take an inventory of what you and your team already know. If possible, you should convene an advisory group or task force meeting for a structured conversation about what is known and what needs to be known about your target community in relation to health information outreach.
To organize your discussion, you might take advantage of a strategic planning tool called a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis. A SWOT analysis is typically used to assess an organization's internal strengths and weaknesses and the opportunities and threats to the organization in the external target community.
To apply this tool in health information outreach project planning, we suggest a slight variation on the typical SWOT tool. We suggest that you gather your team of advisors to identify what you know about the project team's internal strengths and weaknesses for working with a specific community and the strengths and barriers in the community to providing health information outreach. Throughout the discussion, take note of the information you need to gather for a more complete picture of strengths, weaknesses, opportunities and threats. Figure 7 shows a way to adapt the traditional SWOT tool for a planning discussion of a community assessment.
If organizing an advisory meeting is beyond the scope of your resources, you should then talk individually to your advisors. You might start by completing the SWOT template on your own or with your immediate project team. You then could share it one-to-one with your project advisors for their input.
Your final task in Step One is to articulate evaluation questions. Evaluation questions help you plan a thorough community assessment while helping you avoid unnecessary data collection. Evaluation questions are guided by your theoretical model of choice and by advice from your team of advisors. Figure 8 provides a list of potential evaluation questions.
In the preparation stage, it is a good idea to find out if an office or board from your organization must review your community assessment plan before you actually start to collect data. Universities and academic medical centers require that some data collection procedures be reviewed by Institutional Review Boards. Other organizations, such as public schools or public health departments, may have procedures in place to review and authorize projects that include evaluation methods such as surveys. It is best to find out about all review requirements early in your community assessment planning, or you may find that your project will be significantly delayed.
Return to Table of Contents
There are two basic approaches to attaining information about your community of interest. You can collect the primary data, which is information collected directly from the community members. Chances are you will have to collect some primary data for your health information project because it is "customized" to your specific target community. We recommend, however, that you start by looking for data collected by other groups or researchers (called secondary data) that may provide you with valuable information about your target community. Some secondary data sources include the following:
Figure 9, provides some examples of secondary data sources that may be useful when conducting community assessments. Although relatively convenient, secondary data sources have their limits. They were not designed to provide information about your specific project or community. They also may not have up-to-date information. You seldom can get adequate details about the actual lives of community members that you can glean with more customized data collection procedures.
Once you have exhausted the usefulness of secondary data sources, it is time to recheck your evaluation questions to see what you still need to answer and plan a strategy for filling the missing puzzle pieces of your community pictures. Listed below are some typical methods for collecting community assessment data. Some of the most popular methods for data collections are the following:
Also, see Appendices 2 and 3 in Booklet 2  for a list of data sources and methods. Another good source for designing community assessment methods is the website Community Toolbox. (See specifically the resources under Part B in the Table of Contents at http://ctb.ku.edu/en/tablecontents/ index.aspx .)
Figure 10 provides a sample table that you could adapt for planning your data collection process. Before you begin data collection, be sure you do the following:
Once you have collected your data, it is time to move to Step Three.
Return to Table of Contents
Data analysis is a systematic process of summarizing data so that it can be interpreted and used for decision-making. Booklet 3  in this series explains in detail how to analyze data, specifically data collected through questionnaires and interviews. Readers are encouraged to review that booklet for a more detailed description of simple data analysis.
In this section, we will talk about interpreting and applying data after they have been summarized. We do not recommend that you wait until the end of your community assessment project to analyze data. As described in Step Two, we recommend that you collect secondary data first, interpret what you have learned and then move on to collect primary data. We also recommend that you do ongoing analysis with primary data. When you have a site visit, try to describe and interpret what you have learned. If you conduct a short questionnaire with some local groups, summarize the findings soon after you collect them. You also can share preliminary findings with team advisors to get their insights.
Once you have completed all data collection, you will want to put all the pieces together in a report that presents a cohesive picture of the community. Many people produce a formal written report of findings for a community assessment. If you were funded for your community assessment, your funder may require such a formal report. You probably will receive guidelines from funders about how to organize and present findings. You also may want to share your findings through a scholarly publication. A consultation of author guidelines and a review of a sample of the journal articles will give you an idea of how to use these formats.
It is important, however, that you use your data to make decisions about health information outreach and to guide your project planning. Succinct report formats that directly answer project staff and stakeholder questions are more likely to be read and used. You may decide to present key findings in a written executive summary or a PowerPoint presentation. To choose important findings, think about what different stakeholder groups will want to know:
Figure 11, provides you with an outline of the types of information you might want to present to your stakeholders for discussion.
Once you have presented findings to your team of advisors, it is time to revisit your original SWOT analysis to see whether you have a more complete analysis of your target community and your team's ability to provide health information outreach. Figure 12, presents a more traditional version of a SWOT analysis template, which might be more useful in Step 3 discussions with your team of advisors.
Return to Table of Contents
Once you have completed the three steps of a community assessment, you and your project advisors should have an understanding of the feasibility and usefulness of conducting health information outreach in your community of interest. Your team should be able to articulate the changes you would like to see in how community members use online health information. The findings should help you design a project that leads to changes of importance to you, your stakeholders, community residents, and those who serve them. These changes, or "outcomes," may include the following:
Your community assessment should also provide you with good information to guide effective strategies for helping community residents achieve these outcomes. These strategies will be sensitive to the culture of the individuals and will target their needs. They will be designed to build on the current daily experiences and needs of the community. You may know, for example, that there are diabetes support groups in the area where you can teach about online resources. Or you may have discovered a group of volunteers who want to be able to help their neighbors or friends with health information needs.
At this point, it is time to develop a project plan. Booklet 2  in this series will help you develop a project plan customized to achieve the desired outcomes you can now articulate based on your community assessment.
As noted earlier in this booklet, the community assessment process may help you find community-based partners to help implement your health information outreach project.
This is an ideal situation. A community-based partner can provide you with access to residents, demonstrate your legitimacy to community members, share financial and human resources, and point you toward venues for teaching and demonstrating online resources. If you are fortunate enough to find a partner, it is important to define the roles and responsibilities of your organization and the partnering organizations. You should formalize an agreement with a written Memorandum of Agreement (MoA) that is signed by representatives of partnering organizations. For guidelines and a sample MoA, check out the Urban Indian Health Institute's Resources for Partnership page at http://www.uihi.org/ projects/health-equity/partnership/ .
Return to Table of Contents
Return to Table of Contents
Return to Table of Contents
The following case is based on an actual outreach project conducted in Texas near the Mexican border . Students, faculty, and staff at a health careers high school partnered with librarians from a health sciences library to train the school community on use of MedlinePlus. The National Library of Medicine provided funding to the health sciences library to develop outreach projects for the Hispanic communities in the Lower Rio Grande Valley.
We start the case example at an early stage in the project. In fact, we start with the pre-community assessment networking activities when the health science librarian, charged with developing outreach projects in the region, started to network and form partnerships in the community. This librarian had just moved to the area to manage a regional academic health center library and was looking for potential partners to conduct an outreach project. In the networking process, she met the high school librarians who eventually entered into a partnership resulting in a high school peer tutor project in which students were taught to promote MedlinePlus to peers, educators, and community members.
This appendix provides two sample worksheets for recording observations. Worksheet 1 provides a method for recording contacts you make when you are first exploring a community (before you actually start a formal community assessment). Worksheet 2 demonstrates a method for recording important information gathered during one or multiple visits to an outreach site.
We provide two versions of each worksheet. First, we show these worksheets filled in with information related to our case example to give you an idea of how to use them. These completed worksheets are followed by template versions that you can easily adapt for your own projects.
|Networking activity and time/date/place||Summarize results of your networking event (ideas for partnerships, outreach sites, etc.)||Future contact(If yes, write time/date/place)||Main contact(name, phone, email)|
|Visited with medical director and main administrator of the clinic. They seemed willing to allow us to demonstrate MedlinePlus to patients in their waiting room.||No date scheduled||Mr. Director
|South County Hospital
|Met with directors of public relations and patient education. They told us of various health fairs scheduled for the coming year where we can have a booth.||Contact organizers for St. Bernard Health Fair. Deadline to enter health fair: 9/30/2000||
|Health Careers High
|Met with head librarian for a school library serving a school district of career-oriented magnet schools, including two health careers high schools||Demonstration to library staff on [date TBA] at High School Library||Ms. Head Librarian
|Learned about this from colleague yesterday. Still seeking a contact for this agency|
|Rio Grande Health
|Will attend its November luncheon|
|Date of Visit:||November 11, 2012|
|Outreach Partners:||High School Library and Health Sciences Library|
|Target Community:||The target community consists of students and faculty at Health Careers High School. A secondary target community is composed of the family members of these students. We anticipate that students will access MedlinePlus material for their families.|
|Primary Contact(s):||Ms. Head Librarian; Ms. Administrative Assistant, 101-010-1010|
|Activities:||Demonstration and discussion with all three librarians. Three focus groups: one with juniors and seniors; one with freshmen and sophomores; one with faculty. Interview with school principal.|
Who are the primary contacts for the project?
Who has the authority to make priorities and enforce plans?
The superintendent and principal are the administrators with the final authority to decide whether this outreach project can be conducted at the health careers high school. Both of them met with the health science and school librarians, attended a MedlinePlus demonstration to students, and expressed their enthusiasm to have Health Careers High be involved in a project with the health science library. The superintendent wants the head school librarian to present the project to the school board at its next meeting.
Who is the best day-to-day contact (the person who is there frequently and returns phone calls)?
Who seem to be the potential innovators and early adopters?
Who will be the opinion leaders for your resources?
Resources and assets
How do different groups in the community respond to learning about your online resources?
Can you identify possible co-facilitators whom you will train to help you with outreach activities?
Did the people you talked to seem to offer advice on how to involve the community? Did they have ideas of where your online resource could be taught?
Current status of health information access
How are community members now getting health information?
How do they feel about the quality of the information they get?
Will the resources you are introducing be better or more difficult to use compared with their different approaches?
Current status of computer experience
What groups are experienced with computers and the Internet?
What groups are learning to use the computer?
What groups are likely to have a difficult time using the computer or the Internet? Are there other community members who can help them?
Level of computer access
Where can community members get computer access?
How many have access from home?
Describe any technology centers available to the community.
Is there any type of training or assistance to residents who want to use the computers?
Describe other aspects of the community that might affect the outreach project
What do both partners have to offer?
Is either partner in the collaboration attempting to secure funding?
What is needed for outreach to occur?
Do you foresee any challenges to completing an outreach project here?
Return to Table of Contents
Download Blank Worksheet 1
Return to Table of Contents
Download Blank Worksheet 2
Return to Table of Contents
Return to Table of Contents