For those of you interested in attending the 2007 American Evaluation Association Conference in Baltimore (November 7-10), information is now online at the AEA conference Web site.  I also recommend AEA’s pre- and post-conference professional development workshops, which are offered Nov. 5-7 and Nov. 11. Workshop descriptions and cost information are available here.  Many of these workshops have been taught by the same instructors year after year and have received high evaluations. The quality can vary, however — so maybe some of you who have previously attended the AEA conference can add their comments about the workshops they liked.

Because so many RMLs conduct online surveys, I thought I would mention a couple of books recently recommended in discussions at EvalTalk, the American Evaluation Association’s listserv. The first is Don Dillman’s book “Mail and Internet Surveys” (updated 2007, Sage). Don Dillman is probably the best known survey researcher – that is, he researches the best ways to conduct survey research. I have used this particular book and found it to be very useful, although I have not seen this edition. The second book is by Sue and Ritter called “Conducting Online Surveys” (2007, Sage). One listserv member said he liked the book because (among other things) it addressed issues related to response rates – a concern that most of us have regarding online surveys. If you are a member of AEA, you may get similar information from an upcoming issue of New Directions in Evaluation which deals directly with use of online surveys in evaluation. This issue is edited by Sue and Ritter and is due out in September. New Directions in Evaluation is free to AEA members.

Imholz S, Arns JW. Worth Their Weight: An Assessment of the Evolving Field of Library Valuation. New York: Americans for Libraries Council, 2007.

“Worth Their Weight” takes stock of the field of library valuation, defined as the process of assessing the value of a library to its community in actual dollars and cents. The report was issued by Americans for Libraries Council (ALC), “a nonprofit organization dedicated to increasing innovation and investment in the nation’s libraries.” The report describes different valuation models adapted from business and the nonprofit sector, such as social return-on-investment, triple-bottom line accounting, corporate social responsibility reports, and the balanced scorecard. To provide an overview of the valuation field, the report includes summaries of 17 public library valuation and impact studies (with links to the full reports). These summaries include detailed descriptions of the methods used, including actual surveys employed by the libraries. Finally, the report suggests ways to build the field of valuation and apply findings for use in library advocacy. The report relates specifically to public libraries, but the information has applicability to hospital and health science library valuation.
For a quick overview of the report, check out the fact sheets at the ACT for Libraries web site.

Langille, L, MacKenzie, T. Navigating the road to success: A systematic approach to preparing competitive grant proposals. EBLIP 2007; 2 (1): 23-31.

This article gives pragmatic advice about writing competitive grant proposals, with information organized around 11 basic principles of grant preparation including “address a specific audience,” “be innovative,” “involve stakeholders” and “define your objectives and outcomes.” The tips are so concrete, an applicant could create a checklist of characteristics to include in the proposal. The article emphasizes writing to two audiences: the funding agency and reviewers. While the article specifically describes preparation of a research grant proposal, many of the principles would apply to any grant application. RML staff members who teach proposal writing to network members or provide one-to-one help to network members with applications might find the article can help to articulate characteristics of a good proposal. Those reviewing grant applications also will find good criteria for judging applications. The article includes a list of online sources to support proposal writing and project planning. It also presents a timeline, so applicants understand the lead time they need to truly prepare good proposals.

Apparently Survey Monkey has undergone a major re-design. You might want to take a look before you need to use it again.

In my quick tour around the updated site, I noticed the list manager feature is much improved. It stores your recipient list and messages with the survey, so there is less opportunity for error (like sending your survey to someone else’s list!). You now create and store your recipient list with the survey and access it under the collect button: just hit the collect button for a specific survey, and you’ll find the distribution list for that survey. (You create your distribution list there too.)

The other major change I discovered is that inactive surveys are archived and it can take up to 24 hours to restore data from archived surveys. (I couldn’t find any notations about how long surveys are inactive before they are archived.) Only the data takes time to restore, though. You do not have to restore questionnaires if you want to copy them for the basis of a new survey. When you hit the “create survey” button, a list appears of the surveys in the account that can be copied and archived surveys are included in that list.

It would be great if others using Survey Monkey could give us all a heads-up on changes you find (for better or worse) by inserting comments to this post. Instructions for adding comments can be found by clicking on “How to Participate” under “About OERC Blog” to the left.

Brophy, P. Narrative-Based Practice. EBLIP. 2007. 2:(1):149-158.

Given, L. Evidence-Based Practice and Qualitative Research: A Primer for Library and Information Professionals. EBLIP. 2007. 2:(1):15-21

In the movement promoting evidence-based library and information practice — defined as use of formalized strategies for including evidence in daily practice – the definition of the body of knowledge constituting “best evidence” continues to evolve. In the two articles cited above, which share the same issue of EBLIP, authors Brophy and Given argue for the inclusion of qualitative studies in that body of knowledge. While quantitative randomized-controlled trial (RCT) studies have long been considered the gold standard for producing evidence in the disciplines embracing evidence-based practice, Brophy and Given both argue that social fields like librarianship must look to qualitative studies to answer questions of “why” and “in what context.” They also argue that we cannot fully understand social context without methods that emphasize listening to people, observing behavior and reviewing textual and pictorial documents.

Brophy’s article, a commentary, promotes use of a database of high quality narratives (or stories) to inform practice – something he calls “narrative-based practice.” He writes “We are more likely to find meaning in the telling of how things have been experienced by others than in the formality of arid statistics and measures.” (p.156) Thus, he believes that narratives must be presented along with statistics to help managers with their “evidence-based” decision making.

Given’s article presents a more informative treatment of qualitative research with examples of its three primary methodologies: interviews; observation; and analysis of textual data (e.g., participant-created documents like journals or existing texts like policy manuals or meeting minutes). She also discusses some standard criteria for assessing qualitative research – which differs considerably from criteria for judging quantitative research.

Any number of articles have been published that argue for the legitimacy of qualitative research, but Brophy and Given go a step further. They believe that qualitative studies are essential to the development of a complete body of knowledge for informing practice.

The Joint Commission (the health care organizations’ accrediting agency formerly known as JCAHO), has launched a public policy initiative to address broad issues that may undermine patient safety and quality of health care. It has published two papers that address important concerns related to patient safety and quality of care: linguistic and cultural diversity of patients; and low health literacy. A recent publication titled Hospitals, Language, and Culture: Snapshot of a Nation. A Report of Findings presents findings from a qualitative study of how 60 hospitals dealt with linguistic and cultural diversity. A list of national strategies that emerged from this study are presented here. A white paper, “What did the Doctor Say? Improving Health Literacy to Improve Patient Safety” addresses the problem of health literacy and strategies for meeting patients’ communication needs.

The Joint Commission recognizes that the problems rising from diversity and low health literacy must be resolved through engagement of “multiple publics.” And medical librarians represent one “public” that should be engaged. Yet neither publication specifically mentions librarians, who could play a key role in providing access to information needed to support the strategies suggested in the publication. It appears that librarians may have to advocate for their role.

Fortunately, the MLA’s 2006-2007 priorities include a broad media effort to inform health care administrators of “librarians’ value (ROI) in providing consumer health information and patient education that improves patient safety, welfare and empowerment.” Another MLA priority — research defining the relationship between consumer health information and health information literacy on patient outcomes, safety and health care costs — will provide an “evidence-based” foundation for advocacy. The Joint Commission’s public policy initiative (and the accreditation standards that result from it) could pave the way for collaboration between health care organizations facing the challenges of caring for a diverse patient population and medical librarians who can support their efforts through access to much-needed information.

MLA News is running a two-part series about conducting needs assessments this spring. The first article, Needs Assessment for a Consumer Health Library Service, Part One: The Planning Process appears in the March 2007 issue. The second part of the series, Needs Assessment for a Consumer Health Library Service, Part Two: Implementation and Analysis, will appear in the May 2007 issue of MLA News. Part One focuses on planning a needs assessment. It lists some good examples of methods – surveys, key informant interviews, etc. It also gives examples of the different types of stakeholders and audiences who might be interviewed or surveyed as well as examples of existing information (publications, databases) that are useful sources of data in needs assessments. The article provides a lot of ideas in a small amount of news space.

The Aspen Institute Roundtable on Comprehensive Community Initiatives has a Measures for Community Success database with descriptions of over 100 evaluation tools and methods for evaluating community initiatives. Many of these tools have been designed for very specific types of initiatives, such as child development or substance abuse prevention. However, the resources are useful for getting ideas for community-based indicators or for evaluation of collaborations. There are some entries that appear to have more generic applications, such as the Collaboration Index and Collaborative Assessment of Capacity. (Note: I could not find these online, so they may be more context-specific than their descriptions indicate.) Some of these entries will lead you to names of experts who might have articles of interest to you.

This is mainly a database with descriptions. If you find something of interest, you will have to dig for more information, by either doing a literature search on the instrument or the author’s name or calling the contact information listed in each entry. However, it seems to be a rich resource for learning how to work with community partners.

Source: Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale.Journal of Medical Internet Research 2006; 8(4): e27.

The eHealth Literacy Scale is designed to measure consumers’ knowledge, skill, and comfort with finding, evaluating, and using electronic health resources. A scale is a measurement instrument designed for research and evaluation that is comprised of several (usually 3 or more) items. A participant’s responses to these items are combined into one score (e.g. by averaging or summing) to provide one measure of a specific concept – in this case, eHealth Literacy. A reliable scale is one that is consistent or stable — characteristics evaluated through a variety of methods. For instance, all items in this scale are supposed to be measuring the same concept, so the developers checked to see if participants’ answers were consistent across all of the items. Norman and Skinner also ran a factor analysis, which tests to see if the 8 items are related to one “theme.” This statistical method looks at patterns of responses and can indicate how many themes (known as factors) are needed to explain variations in how people responded to the questions. (The researchers name the factors by looking at items that statistics show belong together.) For the eHealth Literacy Scale, one factor seems to be adequate, which further supports its reliability. Finally, the developers tested to see if participants’ answers remained consistent (or stable) when they completed the scale on several occasions.

Since many of us do not have the skills to test measurement scales, it is nice to have one with a track record available in the literature. (Norman and Skinner provide the scale in this article.) One thing to remember, however, is that reliability is just one element of validity. Reliability tells us nothing about whether or not this scale is actually measuring eHealth Literacy (scales can be consistently wrong). Hopefully, Skinner and Norman or others will provide future publications that show evidence for the eHealth Literacy Scale’s validity. However, that should not prevent the rest of us from using it. In evaluation, we seldom make decisions based on one source of information, so we just need to pay attention to the scale’s findings in our studies to see if they corroborate other evaluation findings. If they do, you probably can feel comfortable using the data along with your other evaluation findings. If they do not, you can explore the inconsistencies and possibly gain a deeper understanding of the program you are evaluating.