The Oregon Program Evaluators Network (OPEN) held its Annual Conference last week in Portland. OPEN is a regional affiliate of the American Evaluation Association, with members from government agencies, nonprofits, universities, and private consulting firms. Their annual meeting primarily attracts evaluators from western Oregon and southwestern Washington (Vancouver, WA down through Eugene, OR) but there was at least one international participant and several of us from Seattle. This was a very interesting meeting and I’ll provide my subjective take-aways from it in this post and the next two posts.

The opening keynote speaker was Dr. Debra Rog, 2009 president of the American Evaluation Association, and her talk was titled, “When Background Becomes Foreground: The Importance of Context in Evaluation.” She mentioned ongoing discussions in the evaluation field about whether randomized studies can actually be considered a “gold standard”–they’re great when you can do them, but they’re not always appropriate (they may not be practical or ethical). She spoke of realist evaluation, pointing out that programs are embedded in layers of social reality. For example, power and privilege can influence programs (and the evaluation of those programs) in fundamental ways. Often there are only two “degrees of separation” between the people in power and the people who are providing data–this can lead to a lack of openness and honesty. Context can be difficult to identify and this speaks to the importance of including multiple voices and views. She provided a great insight from her own experience about sharing evaluation results with stakeholders: decision makers are busy and don’t want nuances, they want the bottom line.

The afternoon featured six workshops running in two sets of three (here’s a copy of the agenda). I attended these sessions:

• “What Program Evaluators Can Learn from How We Performance Auditors Do Our Work” and
• “Groovy Graphics: Visual Displays of Quantitative Information”

Handouts and powerpoints from the afternoon workshops will be made available at the OPEN web site.

You can compile all the statistics in the world to demonstrate the effectiveness of your program – but it’s the stories behind the statistics that will be remembered. The workbook Impact and Value: Telling Your Program’s Story provides valuable tips and examples for developing success stories to demonstrate program achievements. This workbook goes beyond describing how to collect anecdotes of individuals who have benefited from your program (although anecdotes are used effectively in the workbook examples). Rather, the workbook shows how to frame your project’s successes in a story format that is easily communicated and remembered. The authors give outlines for a variety of presentation formats, from short elevator speeches delivered to high-powered stakeholders to 2-page write-ups for various audiences. The workbook includes a useful template for developing success stories. Impact and Value: Telling Your Program’s Story is a good resource for those who want to use their evaluation results for effective program advocacy.

This workbook, published by the CDC, is available for download here.

Citation: Lavinghouze SR, Price AW. Impact and Value: Telling your Program’s Story. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Division of Oral Health, 2007.

Someone recently asked me if SurveyMonkey forms are accessible to those with functional limitations and disabilities. In fact, SurveyMonkey received Section 508 certification in June 2008. According to the company’s Web site, they are the only commercial online survey application that has this certification.

While SurveyMonkey software automatically formats surveys to be accessible, there are a few practices that we need to follow to make sure SurveyMonkey questionnaires are user-friendly with screen-readers and other visual technologies. For instance, don’t add extra html coding to your questionnaire (e.g., to bold-face or italicize words) because screen-readers may read parts of html coding as text. Also, SurveyMonkey’s default color schemes are configured for maximum contrast to help low-vision users. Creating your own color schemes may make your forms less readable for this population. You can find more tips from SurveyMonkey for creating screen-reader friendly forms at this link.

SurveyMonkey’s newsletter reports that SurveyMonkey surveys are now optimized for use on iPhones. The June 2009 newsletter states:

“Because it is a device with a modern, standards-compliant browser, any respondent can receive a link to your survey and access it directly on their iPhone.”

Furthermore, SurveyMonkey is currently working to make their surveys optimized on other media or hand-held devices.
In addition, you now have the ability to do the following:

• Create and download custom charts to enhance the presentation of your survey data.
• Import these graphics into your own presentation software such as PowerPoint, Word, etc.

To learn more about the updates, you can visit the following topic in the help center: Creating Custom Charts

I spent the earlier part of the week (June 15-17) in Atlanta attending the AEA/CDC Summer Evaluation Institute and, as usual, came away with some great information.  I’ll be adding some separate blog entries about the sessions I attended, but I thought I would give a rundown on this particular event.  The Summer Evaluation Institute is conducted jointly by the American Evaluation Association and the Centers for Disease Control and Prevention, so many presenters and attendees were from the CDC – but those of us who attend the AEA conference or other evaluation training events found familiar names on the roll of presenters. The Summer Evaluation Institute differs from the AEA conference in that it is totally training-oriented – offering a limited number of educational sessions between 8:30 and 4:00 pm over 2.5 days.  So you don’t feel conflicted over all the options of a conference and you have plenty of downtime to meet and network with colleagues. As you might expect, there is an emphasis on health-related evaluation in many of the sessions, but that emphasis appears more in the examples used by instructors – the evaluation techniques themselves are applicable across disciplines.  The cost is reasonable.  This year, the cost was $395 for AEA members (and CDC employees) and a little more for non-members. (Sorry I can’t be more specific: the fee is no longer listed at the AEA Web site now that the event is over).  That fee includes three keynote speeches, a choice of training sessions each morning and “breakout” sessions in the afternoon. (I’m not sure how “training sessions” differed from “breakout sessions,” other than length of time – the training sessions were about an hour longer than the breakout sessions).  It also includes breakfast and lunch on most days.  Beginner workshops were offered on June 14 for an additional cost:  “Quantitative Methods for Evaluation;” and “Introduction to Evaluation.”  The Summer Evaluation Institute is held annually, so if you think you might be interested in the 2010 event, check out the AEA web site (eval.org) starting in March.

A new government Web site, Data.gov, may prove to be a good tool for locating existing data from federal agencies, particularly for those of us doing needs or community assessment.  The Web site is the public’s “one-stop shop” for raw data from economic, healthcare, environmental, and other government agencies.  Along with raw data, the site provides tools for compiling raw data into more analyzable formats (e.g. tables, maps) and widgets (interactive tools with single-service purposes, like showing users the latest news). My quick browsing of the Web site gives me the impression that it is a work in progress.   However, the “about” page says that the catalog of datasets will continue to grow and that the site will be improved based on public feedback. 

Here is a link to the blog entry about Data.gov from the Office of Management and Budget: www.whitehouse.gov/omb/blog/09/05/21/DemocratizingData/ 

Read even more about the American Evaluation Association meeting at the Eagle Dawg Blog, where Nikki Detmar has summarized her Ten thousand four hundred and thirty one words of notes.  Nikki attended many different sessions from the ones I went to, and where we were both in the audience for a session, Nikki took more detailed notes than I did.  She’s a fast typist who uses her laptop for notes; I’m a codger who writes with a pen in cursive scrawls on pieces of lined notebook paper.  Also, the Eagle Dawg Blog is just an all-around good read for Nikki’s perspectives on life, the universe, health informatics, and medical librarianship.

This was a half-day workshop on Sunday morning, November 9, ably taught by Jennifer Camacho Catrambone, Ruth M. Rothstein CORE Center, Chicago. Nonparametric statistics are those that are used with ordinal or nominal data, when data are skewed, or when sample sizes are small.

In contrast, parametric statistics are designed to be used with a minimum sample size of 30 subjects per group.   Dependent variables are expected to be interval-level; categorical (nominal) dependent variables are excluded (although independent variables are often categorical).

The Chi Square test is an example of a nonparametric test of association between variables. The workshop handout lists numerous others and provides descriptions and assumptions. The class was full of information, but note to self: don’t take statistics classes after spending four days in conference sessions–the brain is tired.

Going to meetings is hard work!  Especially meetings like the American Evaluation Association annual meeting, which is chock full of interesting sessions that make you think.  Saturday was a very full day, and quite rewarding.

Fine-tuning Evaluation Methodologies for Innovative Distance Education Programs (Debora Goetz Goldberg, John James Cotter, Virginia Commonwealth University)

VCU Medical School offers a PhD in Health Related Sciences via distance education that combines on-campus learning, asynchronous discussions, synchronous chat, podcasting, and other approaches.  Program evaluation followed these steps: define quality (support, course structure, curriculum, instruction), select important areas to review (were goals met, what skills were developed, was advising adequate, was IT adequate, overall program), identify data collection sources (course evaluations, followup assessments, interviews with instructors, feedback from students’ employers), collect and analyze data.  Findings showed areas where the curriculum needed adjustment, where technology could be enhanced (for example, offering streaming videos of lectures), and where supplementary use of teaching assistants was needed.  The supplementary TAs worked with students in the statistics course.

Evaluation of an Interactive Computer-based Instruction in Six Universities: Lessons Learned (Rama Radhakrishna, Marvin Hall, Kemirembe Olive, Pennsylvania State University)

In a USDA-sponsored (with institutional matching funds) project, Penn State collaborated with five other land-grant universities to develop and offer a 1-semester agronomy course that comprised 11 interactive modules.  Development took two years and addressed the funding agency’s desire for collaborative courses that make collective use of expertise, share resources, and reduce duplication of effort.  Each module featured 20 knowledge questions plus items about the modules’ navigability, design, and layout.  Pre- and post-tests showed knowledge gain.  The project showed that multi-institutional collaboration can work, although it can be challenging.  In this case, IRB review was needed (because human subjects–the students–were involved) and the crop scientists were unfamiliar with that process.

The Use of a Participatory Multimethod Approach in Evaluating a Distance Education Program in Two Developing Countries (Charles Potter, Sabrina Liccardo, University of the Witwatersrand)

This radio-based series of English lessons for school children in South Africa and Bangladesh has grown significantly since it began in 1992.  In 1995 it was reaching 72,000 learners and as of 2005 it was reaching 1,800,000.  Evaluation has involved questionnaires, observations, focus groups, and photography; results have been used to report progress to stakeholders and to identify areas for improvement.

Building Evaluation Practice Into Online Teaching: An Action Research Approach to the Process Evaluation of New Courses (Juna Z Snow, InnovatEd Consulting)

The author has developed a Student Performance Portfolio that has been used with two online teacher education courses.  The portfolios allow students to conduct ongoing evaluation of their work and of the course, and include weekly goals, activities and time spent, with reflections on assignments and performance.   Students submit their portfolios each week.  To get the most from the portfolios, it is important to conduct ongoing content analysis and be responsive to students.

Incorporating Cellular Telephones into a Random-digit-dialed Survey to Evaluate a Media Campaign (Lance Potter and Andrea Piesse, Westat; Rebekah Rhoades and Laura Beebe, University of Oklahoma)

When both cellphones and landlines are included in telephone surveys, different sampling frames must be constructed for groups who are cellphone-only, who are landline-only, and who have both.  A tobacco intervention study found one significant difference between the 3 groups:  those who have both cellphones and landlines smoke less–a difference theorized to stem from income and educational characteristics.  The sociology of cellphones is different from landlines.  For example, if a cellphone on a counter or a desk rings and the cellphone’s owner is not present, no one else will answer the phone.  In addition, many cellphone contracts require cellphone owners to pay for calls they receive; these cellphone owners will not want to use their “minutes” up by answering survey questions.  This issue could be addressed by offering gift cards to participants or by conducting surveys on weekends.

The Growing Cell Phone-Only Population in Telephone Survey Research: Evaluators Beware (Joyce Wolfe, Brett Zollinger, Fort Hays State University)

Telephones have been fundamental tools for survey research, and cell phones are introducing new variables to be considered.  At one time more than 90% of households had landlines but now almost 16% of telephone users are cellphone-only.  The size of the cellphone-only population is projected to increase.  Whether there are significant differences between groups of people who have landlines and those who only use cellphones is a topic of debate.  The cellphone-only population tends to be young, unmarried, renters, and lower income (and more likely to have financial barriers to treatment).  Samples of cell phone numbers can be obtained, but it is illegal to use automatic dialers with these numbers.  In addition, more screening is needed because cell phones are linked to individuals rather than to households or geographic locations, and individuals can range in age down to elementary school students.

Perspectives on a Promising Practices Evaluation (Susan Ladd, Rosanne Farris, Jan Jernigan, Belinda Minta, Centers for Disease Control and Prevention; Pam Williams-Piehota, RTI International)

The Centers for Disease Control and Prevention’s Division for Heart Disease and Stroke Prevention (DHDSP) has conducted evaluations of heart disease and stroke interventions to identify effective interventions and promising practices, with the intention of building evaluation capacity at the state level.  Lessons learned included:  collaboration and comprehensive evaluation planning is time-consuming; better evaluability assessments are needed; periodic reaffirmation of commitments and expectations is necessary.

Rapid Evaluation of Promising Asthma Programs in Schools (Marian Huhman, Dana Keener,Centers for Disease Control and Prevention)

The CDC’s Division of Adolescent and School Health (DASH) funds school-based programs for asthma management and uses a rapid evaluation model to help schools assess program impacts.  These evaluations are intended to be completed within one year, with two days devoted to conducting a site’s evaluability assessment and six months devoted to data collection.  These evaluations focus on short-term outcomes.

Best of the Worst Practices: What Every New Evaluator Should Know and Avoid in Evaluation Practice (Dymaneke Mitchell, National-Louis University; Amber Golden, Florida A&M University; Roderick L Harris, Sedgwick County Health Department; Nia K Davis, University of New Orleans)

Panel presenters discussed lessons they learned from their evaluation experiences in the American Evaluation Association/Duquesne University Graduate Education Diversity Internship program.  The experiences and lessons included the difficulties faced by an evaluator who is working with a group that they feel sympathetic toward.  It is hard to be an objective evaluator if you want to help the program succeed.  In working with nonprofits it is important to develop patience with ambiguity, to clarify short and long term goals, and align goals with organizational readiness.  Strong negotiating skills are needed, along with a focus on building trust and credibility.  Evaluation seems to be 10% science and 90% relationships.  It is challenging to manage stakeholders’ diverse and sometimes conflicting agendas.

Ethics and Evaluation: Respectful Evaluation with Underserved Communities
This excellent and thought-provoking session featured three presentations that were based on chapters in the recently-published book, The Handbook of Social Research Ethics by DM Mertens and PE Ginsberg (Sage, 2008).

1.  Ethical Responsibilities in Evaluations with Diverse Populations: A Critical Race Theory (CRT) Perspective
(Veronica Thomas, Howard University)

In traditional social science research, white men are normative.  Critical Race Theory (CRT) is from the critical theory approach which views scholarship as a means to critique and change society and to counteract discrimination and oppression.  In the traditional positivist approach, research is explanatory.  CRT uses a critical lens to foreground oppressed populations and form conclusions and recommendations that promote social equity and justice.  IRBs, with their positivist emphasis on value-free research, can feature a lack of concern for community impacts of projects.

2. Researching Ourselves Back to Life (Joan LaFrance, Mekinak Consulting)

Frustration has built up for many years among Native populations from their sense of being abused by researchers.  The traditional IRB approach to human subject protection can fail to address the question of whose voice speaks with authority about Aboriginal experiences.  Tribal members are beginning to understand that they can define the degree to which they make themselves available.  Five tribes have developed their own IRBs–capacity-building is needed for more tribes to do this.  Tribal approaches involve inclusive review teams, a clear definition of who is expert, broader reporting, an understanding of data ownership and publication approval needs, and a negotiation of how stories will be told.  Different ways of knowing are accepted:  traditional (knowledge of the past), empirical (evidence-based), and revealed (knowledge that comes through channels other than the intellect).

3.  Re-Conceptualizing Ethical Concerns in Underserved Communities (Katrina Bledsoe, Walter R McDonald and Associates Inc; Rodney Hopson, Duquesne University)

Underserved communities are those that suffer from a lack of resources that would allow them to thrive.  There is a need to reconceptualize traditional views of research and ethics.  Unintentional ethical violations have grown from inappropriate methods, use of data, and dissemination of results.  There is a power differential between researchers and participants and, in randomized controlled trials, a problem from the assumption of population homogeneity.  In a new philosophical perspective, evaluators will consider culture, history, community consent, social responsibility, and the differences between what is meaningful statistically and what is meaningful to a community.

Friday at the American Evaluation Association meeting had a technology slant in most of the sessions I chose.

Blogging to the Beat of a New Drum: The Use of Blogs and Web Analytics in Evaluation (Cary Johnson and Stephen Hulme, Brigham Young University)

This roundtable was a discussion about possible uses of blogs and web analytics to assist in evaluations.  Content analysis could be used to study blog conversations; blogs could be used to identify survey participants.  Issues include lack of trust, lack of time, access to computers & network, extraneous data, uneven representation, lack of comfort with written expression, influence of social desirability, and lack of knowledge about people who are “most likely to blog.”  Video blogging, in which participants can click and speak, can be used to collect student feedback in classes but it is challenging to analyze the data.  Google Analytics is a free tool that can be used to view percentages of which users are looking at posts, who selects “read more” and how long they stay, and top search terms.

Voicethread: A New Way to Evaluate (Stephen Hulme and Tonya Tripp, Brigham Young University)

Voicethread.com is a new website where you can upload videos, images, documents, and presentations.  Viewers can then make audio and video comments.  This can make it possible to gather richer data than via a  survey or interview.

Can You Hear Me Now? Use of Audience Response Systems to Evaluate Educational Programming (Karen Ballard, University of Arkansas)

This was a hands-on demonstration of a personal/classroom/audience response (“clicker”) system.  The Vanderbilt Center for Teaching maintains a bibliography of articles about clickers and provides a guide for using them in teaching.

Evaluation Dashboards: Practical Solutions for Reporting Results (Veena Pankaj and Ehren Reed, Innovation Network Inc.)

In the context of evaluation, a dashboard is a performance monitoring tool that provides a quick view of how well goals are being met.  Dashboards, borrowed from the corporate world, are useful in the nonprofit world. They display indicators and targets, and use simple visuals (such as color codes) to illustrate levels of achievement.  Although specialized software is available to help create them, Excel also works.  More information is available at Dashboard for Nonprofits.

Walking the Talk: Evaluation Is as Evaluation Does (Matt Gladden, Michael Schooley, Rashon Lane, Centers for Disease Control and Prevention)

Representatives from CDC’s Division for Heart Disease and Stroke Prevention presented an overview of their strategies to build an organizational culture that values and routinely uses evaluation techniques. Evaluation capacity building approaches include planning evaluations, conducting evaluations, using results, and supporting evaluation through consultation, training, and resources.  Lessons learned included: establish systems that support evaluation; define boundaries and priorities; balance ability to implement a project with its potential benefit; recognize that an evaluator’s role is to identify issues but not necessarily solve them; differentiate between long-term and workplan goals.