SurveyMonkey’s April newsletter reports their new Bounce Report feature:

“Sometimes when sending survey invitations through our collector, the email addresses may bounce the message back to you because the email is invalid, the receiving server is too busy, the receiving email inbox is full, and so on.

Now when sending your survey invitations through our Email Invitation collector, the messages are delivered by our email server. If the message is undeliverable, the email will be considered a Hard Bounced email in the Edit Recipients portion of the collector.

You now have the ability to do the following:

• View the Bounced emails.
• Export them from the list.
• Remove them from the list.

This will help to ensure that your lists are current and contain valid emails for future survey response collections.

To learn more about the new Bounce Report feature, please refer to the following Help Topics:
Check Bounced Emails
Hard Bounce Tutorial“

The American Evaluation Association held its annual meeting in Baltimore this year and, as usual, the workshops were great and the program was packed with useful information aimed at evaluators with varied levels of expertise. Baltimore itself was wonderful; somehow they managed to provide very nice weather even in November! Of the sessions I attended, my very favorites were the ones with Brian Yates from American University, who is a sort of evangelist about “cost-inclusive” evaluation. Apparently, his outlook that “costs are at least as important to measure as outcomes” is not universally accepted amongst all evaluators. His theme is that all outcomes have costs and some outcomes are monetary. One of his presentations was about starting a cost study, the other was titled “Costs Are All That Matters.” I also took some workshops:

Quantitative Methods I spent two days in this workshop, which was quite fast-paced–the instructor claimed to be providing us with a semester’s worth of information! Participants who had never had methods courses were scrambling a bit to keep up; I have had methods courses but not since the last century, so I experienced the workshop as a challenging review. One of the main things that I took from the workshop was the instructor’s delightful phrase, “reasonable people will disagree.” It turns out that statistics can be so complicated that not even the statisticians always agree about when and how to use which approaches. For example, there is controversy about whether hypothesis testing is the best way to improve knowledge. Our instructor suggested that, since it’s established tradition for research, we should combine hypothesis testing with other approaches. Here are some other, probably more useful points I noted:

• One person’s “6″ might be another person’s “3″
• More than 10 choices will be hard for a respondent to manage. With fewer response choices, results tend to be more reliable
• Always label the middle in a range of rating choices (and the two ends, also)
• Variance in program outcomes can be very important to stakeholders; evaluators are looking to find out whether a program actually had an effect or whether an observed change was simply due to chance (or due to something other than the program)
• Inferential statistics, which are used to generalize to populations of interest (ie, should I bring more people into my program?) are based on the assumption that the population in question is normally distributed
• A number of statistical tests have their origins in agricultural research and were never really intended for use with humans

Our instructor ended the 2-day session with this warning: “I’ve taught you enough now that you’re really dangerous. Don’t try to do this without expert guidance.” True enough, but a class like this will help to communicate with experts.

Online Survey Research This was a half-day class that the instructors had not taught before. I empathized with them as they ran out of time and expressed the realization that it should have been a full-day class. Some of my notes:

• Survey objectives are not the same as program objectives. Surveys can’t fix problems with programs
• The instructors cited a 27% response rate as a “very good” one
• Research has shown that people write longer responses to open-ended questions in online questionnaires than they do on printed ones

SurveyMonkey has recently introduced three short tutorial videos that are open to anyone to view, without need for passwords or logging in. Their plan is to introduce more videos over the upcoming months. (These videos, by the way, are really nice examples of how Camtasia can be used)

These first three tutorials demonstrate the following areas of SurveyMonkey:

• Getting Started (the fundamentals and a tour of the website)
• Survey Design (the basics of designing a survey using SurveyMonkey)
• Using Collectors (how to collect responses; use of settings and restrictions)

A lot of what these videos cover will already be known to those who have used SurveyMonkey, but they can serve as a good resource for classes or for introducing newbie colleagues to the resource. I did learn some very useful items from the tutorials, though, such as:

• the difference between the “exactly one” and “at least one” settings for multiple-choice questions
• how to use rating questions to force respondents to rank their choices
• the bad news about what happens when respondents leave a SurveyMonkey survey uncompleted and return to it later

(…and that bad news is that the information about the partially-completed survey is cookie-dependent, stored by the web browser on the respondent’s computer. To go back and complete a survey, the respondent must be using the same browser on the same computer–and not have cleared the cookies!)

Perley CM, Gentry GA, Fleming S, Sen KM. Conducting a user-centered information needs assessment: the Via Christi Libraries’ experience.  J Med Libr Assoc  2007 Apr; 95(2):173-181.

This article provides a good example of a needs assessment using multiple evaluation methods. Librarians at the Via Christi Libraries in Wichita, Kansas, provide information services to all employees of the Via Christi Regional Medical Center (VCRMC) and needed to develop a strategic plan to meet the expanding use of their services and increasing cost of providing access.  This article provides detailed descriptions of how the researchers used a self-administered survey, telephone survey, and focus groups to gather information of increasing depth among users, and includes appendices with survey and focus group questions. The samples used in the project were not random, but the researchers used many venues to capture a solid cross section of their user population; and the multi-method approach allowed them to corroborate findings across different perspectives.  They also described how they used the findings to develop a strategic plan and listed their “lessons learned” about doing needs assessment.  This is not a “how to conduct a needs assessment” article and the findings are the main point of the piece.  But their concrete description of their methods provides added value to their article.

The goal of this study is to develop a research plan and proposal to measure the value of the health sciences library, library services, and the librarian. The idea for a study on the value of libraries came from a discussion of the MAR’s RAC meeting held at New York University Medical Center on November 17, 2006. Out of that discussion emerged the issue of establishing the value of health sciences libraries to administrators in hospitals, academic health sciences centers, and other organizations where librarians work.

One frequently cited study in the RAC discussion was the “Rochester study”.  In discussing both the need for a value of libraries study, and the impact of the Rochester study - cited over 95 times since its publication in 1992 - the RAC came to the conclusion that before engaging in the research study itself considerable time, effort and thinking needed to go into what a research study would look like.

Objectives of the Value of Libraries Planning Study:

1. To inform the research plan by seeking input from health sciences libraries of various types and sizes and in various geographic locations within MAR
2. To determine the value domain(s) to be measured: patient care (information for providers, patients), education, research, administration, community health
3. To determine the library products or services to be evaluated, and to identify key activities that are directly related to the quality of those products or services.
4. To identify and outline the outcomes to be measured.
5. To develop a study design and timeframe for the study.
6. To determine MAR participants in the initial research study.
7. To develop a plan that is replicable, and to suggest methods for replication.
8. To determine likely sources of funding for the research study.

Please visit the web site for further details on the study, a listing of committee members, and the project timeline,  http://nnlm.gov/mar/about/value.html

The North Suburban Library System (north suburban Chicago, IL) has created a Return on Investment (ROI) Calculator on their website in keeping with their theme for the month of July, 2007: Dollars and Sense: Why Libraries are a Good Investment. For more information about this tool, visit:http://www.nsls.info/articles/detail.aspx?articleID=143.

Shaping Outcomes, an online course on outcomes-based planning and evaluation, will be available free to museum and library professionals this summer and fall. The instructor-mediated course, which will help participants improve program designs and evaluations, was developed through a cooperative agreement between the Institute of Museum and Library Services (IMLS) and Indiana University-Purdue University, Indianapolis (IUPUI). Through the approximately five-week course, participants will work at their own pace to learn outcomes-based planning and evaluation concepts and apply the concepts to a program or a project at their own institutions. A special course for those interested in teaching Shaping Outcomes or incorporating it into their own curricula will be offered in October 2007. Those interested in learning more about Shaping Outcomes or registering for one of the courses should visit http://www.shapingoutcomes.org or e-mail outcomes@iupui.edu.

For those of you interested in attending the 2007 American Evaluation Association Conference in Baltimore (November 7-10), information is now online at the AEA conference Web site.  I also recommend AEA’s pre- and post-conference professional development workshops, which are offered Nov. 5-7 and Nov. 11. Workshop descriptions and cost information are available here.  Many of these workshops have been taught by the same instructors year after year and have received high evaluations. The quality can vary, however — so maybe some of you who have previously attended the AEA conference can add their comments about the workshops they liked.

Because so many RMLs conduct online surveys, I thought I would mention a couple of books recently recommended in discussions at EvalTalk, the American Evaluation Association’s listserv. The first is Don Dillman’s book “Mail and Internet Surveys” (updated 2007, Sage). Don Dillman is probably the best known survey researcher – that is, he researches the best ways to conduct survey research. I have used this particular book and found it to be very useful, although I have not seen this edition. The second book is by Sue and Ritter called “Conducting Online Surveys” (2007, Sage). One listserv member said he liked the book because (among other things) it addressed issues related to response rates – a concern that most of us have regarding online surveys. If you are a member of AEA, you may get similar information from an upcoming issue of New Directions in Evaluation which deals directly with use of online surveys in evaluation. This issue is edited by Sue and Ritter and is due out in September. New Directions in Evaluation is free to AEA members.

Imholz S, Arns JW. Worth Their Weight: An Assessment of the Evolving Field of Library Valuation. New York: Americans for Libraries Council, 2007.

“Worth Their Weight” takes stock of the field of library valuation, defined as the process of assessing the value of a library to its community in actual dollars and cents. The report was issued by Americans for Libraries Council (ALC), “a nonprofit organization dedicated to increasing innovation and investment in the nation’s libraries.” The report describes different valuation models adapted from business and the nonprofit sector, such as social return-on-investment, triple-bottom line accounting, corporate social responsibility reports, and the balanced scorecard. To provide an overview of the valuation field, the report includes summaries of 17 public library valuation and impact studies (with links to the full reports). These summaries include detailed descriptions of the methods used, including actual surveys employed by the libraries. Finally, the report suggests ways to build the field of valuation and apply findings for use in library advocacy. The report relates specifically to public libraries, but the information has applicability to hospital and health science library valuation.
For a quick overview of the report, check out the fact sheets at the ACT for Libraries web site.