Skip all navigation and go to page content
NN/LM Home About OERC | Contact Us | Feedback |OERC Sitemap | Help | Bookmark and Share

Public Health Outreach Evaluation Task Force

RML Public Health Outreach Evaluation Task Force Final Report -- June, 2003

Excerpted from Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century, Institute of Medicine (2002):

A public health professional is a person educated in public health or a related discipline who is employed to improve health through a population focus. The committee believes that well-educated public health professionals, who have a real understanding of the multiple determinants of health and their interactions (the ecological model), are critical to shaping new knowledge, programs and policies relevant to both individual health and health care, and to population health in the coming century. These professionals must have a broad range of skills and information. They must be able to understand and apply new advances in science (e.g., genetics), information, and computer science technology to public health practice and learning (i.e., public health informatics). They must be proficient in communication in order to interact effectively with multiple audiences, to understand and incorporate the needs and perspectives of culturally diverse communities in public health interventions and research, and to inform policy.

Task Force Members
Neil Rambo, Chair
Associate Director
NN/LM Pacific Northwest Region
University of Washington
Seattle, WA

David Carney, MPH
Maryland Department of Health and Mental Hygiene
Baltimore, MD

Betsy Kelly
Assessment and Evaluation Liaison
NN/LM Midcontinental Region
University of Utah
Office Location: Washington University
St. Louis, MO

Julie Kwan
Library Network Coordinator
NN/LM Pacific Southwest Region
University of California, Los Angeles
Los Angeles, CA

Kay McCloskey
Public Health Liaison
NN/LM Midcontinental Region
University of Utah
Salt Lake City, UT

Jocelyn Rankin, Ph.D.
Centers for Disease Control and Prevention
CDC Information Center
Atlanta, GA

Caroline Sparks, Ph.D.
Associate Professor, Department of Prevention and Community Health
Deputy Director, Prevention Research Center
School of Public Health and Health Services
George Washington University
Washington, D.C.

Virginia (Ginny) Tanji, MEd
Medical Librarian
Library Resource Center
John A. Burns School of Medicine
Honolulu, HI

Resource Staff
Cathy Burroughs
Assistant Director
Outreach Evaluation Resource Center
NN/LM Pacific Northwest Region
Seattle, WA

Keith Cogdill, Ph.D.
Outreach Librarian
NN/LM National Network Office
National Library of Medicine
Bethesda, MD

Angela Ruffin, Ph.D.
NN/LM National Network Office
National Library of Medicine
Bethesda, MD

Task Force Charge
I. Recommend plans for achieving the desired outcomes for outreach to public health departments identified at the May 17, 2002 RML Directors Meeting, which include increases in the numbers of: 1) links to online health information resources by health departments; 2) public health department workforce trained; and 3) outreach collaborations by NN/LM with other agencies.

a. Refine the outcomes that will be measured.
b. Determine how the outcomes will be measured or counted
c. Determine how the data will be collected and analyzed.
d. Determine a timeline for data collection, including defining baseline and/or continuous and/or follow-up measures.
e. Determine how the data will be synthesized, reported and used.

II. Recommend ways to increase effectiveness in reaching the outcomes being measured. That is, how can the NN/LM regions overcome barriers and make the most effective progress toward the outcomes of interest? Knowledge or suggestions about best practices might be drawn from our collective RML experience, the audience or institutions we are trying to reach, our contacts/partners, and the published literature.

a. Determine what barriers or problems that currently exist in reaching the outcomes.
b. Determine what can be done to address these issues.
c. Determine ways to reasonably and efficiently identify and collect suggested strategies and "best practices".
d. Determine ways to disseminate findings to the RMLs in a timely way that will facilitate their use in the areas of outreach being assessed.

The task force split into two subgroups, each to address one of the charges. The first group, to address the first charge (Recommend plans for achieving the desired outcomes for outreach to public health departments…) included Julie Kwan, Betsy Kelly, Kay McCloskey, and Caroline Sparks. The second group addressed the second charge (Recommend ways to increase effectiveness in reaching the outcomes being measured) and included David Carney, Jocelyn Rankin, and Ginny Tanji. Neil Rambo, as task force chair, participated in both groups. The resource staff participated in both groups as they were able to. The whole task force participated in two conference calls, Group 1 worked through three additional conference calls, and Group 2 through an additional two, for a total of seven conference calls.

Group 1, with the urging of Caroline Sparks, quickly adopted the approach of developing a logic model as a coherent framework to explicate what we mean by outreach to the public health workforce. The work of this group focused on developing and refining a logic model for the three goals or objectives selected at the May 2002 RML Directors Meeting.

Group 2 considered the environment of public health practice and how to target NN/LM outreach to improve the effectiveness of interventions. The group considered such questions as the role of workforce readiness in outreach effectiveness, the need to ensure compatibility between the intervention and the need, and the need for a service network or networks to be available before people can be referred.

The task force recommendations are intended to guide outreach to the public health practice community by establishing uniform expectations of what is to be accomplished, resulting in a baseline of knowledge about workforce information needs and how to most effectively meet them. The recommendations are a starting point only. They delineate a minimum to be met in Year 3. Some RMLs may have the resources and opportunities to go beyond these recommendations.

Plans for achieving the desired outcomes for outreach to the public health workforce (task force charge #1) are detailed in the program logic model and appendix developed by the task force. For each of the three outreach objectives that received the most votes at the May 2002 RML Directors meeting, the logic model delineates the resources, activities, project management milestones, outputs, and the desired short- and long-term outcomes. These recommended actions apply specifically to Year 3. Follow on actions for Years 4 and 5 will need to be developed later.

Refer to the logic model for details. The summary recommendations are:

Objective 1. Increase links to NLM resources on public health agency web sites.
Each RML will develop a plan to promote adding links to NLM resources to public health agency web sites, at least at the state level. This endeavor can be based on an analysis of agency websites for links to NLM resources, but this is optional and not required. All agencies found to lack links to NLM will be contacted and encouraged to add them.

The logic model appendix explains what is meant by "public health agency". Recommended points of contact at state health departments include: Distance Learning Coordinators, Health Alert Network Coordinators, deputy secretaries, division directors (of epidemiology or nursing or environmental health), or training or workforce development coordinators. Network member libraries may have easier entrée to local public health agencies, which can be sponsored by RMLs.

In the process of accomplishing Obj. 1, contacts will be made and there will be opportunities for raising awareness and establishing relationships.

Objective 2. Improve skills in use of NLM resources by public health professionals.
Those public health workforce subgroups that are most likely to benefit from being trained to use NLM resources to meet their professional information needs will be targeted. A public health training workgroup will be formed to develop training modules to be used by each RML to train the public health workforce. The workgroup will include librarians who are knowledgeable of public health workforce information needs and uses, and experienced in training public health workers. Public health professionals will also be represented on the workgroup. NNO staff will provide project management leadership for development of the training modules and will be the communication link between the workgroup and Outreach Coordinators. Each RML will offer the training at least twice.

Some RMLs will be able to accomplish more, and learn more, than is suggested for this objective. This is to be encouraged. An attempt will be made to share the knowledge and expertise of the training workgroup about workforce information needs with RML staff. This will be explored as part of the interface between the workgroup and the Outreach Coordinators..

Objective 3. Increase relationships and collaboration among public health agencies and health science libraries.
The recommended activity is actually preliminary to increasing relationships and collaboration and thought to be a necessary foundation to build on. Each RML will assess what library services are available to the state level public health workforce. In doing so, each RML will gain an understanding of service network relationships and gaps that exist in access to services. Each RML will recommend how the NN/LM can work with other networks to fill identified gaps.
A task force member pointed out that in order to refer public health professionals to library services there have to be services to refer them to. It is not well understood what library services are available to public health workers in most states, or through what networks. Nor do we know much about the nature or extent of the current or potential use of library service.
The assessment outlined for this objective will be the basis for increasing relationships among public health agencies and libraries. As with Objective #1, the process will open opportunities for raising awareness and establishing relationships.

Left to be done for all three objectives, is consideration of the collection, analysis, synthesis, and reporting of data resulting from these activities, as well as how to evaluate the desired outcomes.

Recommendations of ways to increase effectiveness in reaching the outcomes (task force charge #2) are woven into the activities and project management milestones of the logic model. Ways to decrease barriers and increase effectiveness are based on the synthesis of lessons learned from the NLM-funded public health outreach projects as presented at the Apr. 2001 forum at NLM. These include:

1. Not all of the public health workforce has a need for knowledge-based information resources (e.g., journal literature), in their work. Most NN/LM outreach staff are trained as medical librarians and are knowledgeable in using the literature and related tools (e.g., PubMed). Identifying segments of the workforce that have a need for this science-based information is an approach that is likely to increase the chances of success. For example, those engaged in research, program design, and policy analysis are likely users. Others with educational backgrounds and professional training that included use of the literature may have developed a pattern of continuing to use the literature of their field as a normal course of doing their work.

2. Assessing information needs may have to be an iterative process leading to a refinement of objectives. Taking the time to understand information needs and developing a response that addresses those needs is worth the investment.

3. Short-term outcomes may be 1) increased awareness of information resources and their application; 2) increased access and decreased barriers to use of information resources; and, 3) increased skills in using information resources.

4. Long-term outcomes may be 1) a change in information seeking behavior, and use of information resources, leading to improved job performance; 2) the establishment of library networks to serve the public health workforce; and 3) other collaborations between agencies and libraries.

5. The support of leaders is important in underscoring the importance of using information resources as an instance of implementing evidence-based practice. Reaching and getting the support of leaders should be part of outreach intervention planning.

1. Rambo, N. Zenan, JS. Alpi KM. Burroughs CM. Cahn MA. Public Health Outreach Forum: lessons learned. Bull Med Libr Assoc. 2001 October; 89 (4): 403406.

Full text HTML and PDF formats can be found at

2. O'Carroll, P. Rambo, N. Information Needs and Uses of the Public Health Workforce -- Washington, 1997-1998. MMWR Weekly February 18, 2000; 49(06);118-120.

Full text HTML at

3. W.K. Kellog Foundation Logic Model Development Guide , October 2000.

4. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. K. Gebbie, L. Rosenstock, LM. Hernandez, Editors, Committee on Educating Public Health Professionals for the 21st Century. Institute of Medicine. 2003.