LGBTQIA+ Health Pathway

• Evaluation questions help define what to measure and provide clarity and direction to the project.
• Process questions relate to the implementation of the program and ask questions about the input, activities, and outputs columns of the logic model.
  • Example LGBTQIA+ Process Evaluation Questions
• Outcome questions relate to the effects of the program and relate to the short, intermediate, and long-term columns of the logic model.
  • Example LGBTQIA+ Outcome Evaluation Questions

• Evaluation design influences the validity of the results.
• Most NNLM grant projects will use a non-experimental or quasi-experimental design.
  • Quasi-experimental evaluations include surveys of a comparison group - individuals not participating in the program but with similar characteristics to the participants - to isolate the impact of the program from other external factors that may change attitudes or behaviors.
  • Non-experimental evaluations only survey program participants.
  • If comparison groups are part of your quasi-experimental design, use a 'do no harm' approach that makes the program available to those in the comparison group after the evaluation period has ended.
• More information on Evaluation Design

Ethical Considerations

Consider the Belmont Report principles to examine LGBTQIA+ population ethical considerations:

• Respect for Persons:
  • Individual determination - that is the ability of an individual to make decisions about themselves- is important for evaluation.
  • When evaluating programs with participants who are part of the LGBTQIA+ community, consider the effect simply participating in the evaluation may have on them.
    • If someone feels participating in the evaluation requires them to express their identity in a way they are not comfortable sharing publicly or requires a level of personal information sharing they are simply not comfortable with, they may feel coerced to respond a certain way they may not otherwise agree with.
  • When involving community members in the evaluation or obtaining consent for your evaluation, try to minimize the pressures individuals feel to act or respond a certain way.
    • Allowing anonymous decision making and feedback, when possible, can reduce the pressures individuals feel to behave a certain way.
• Beneficence
  • How will your evaluation balance the burden that participants take to complete the evaluation with the benefits of completing the evaluation?
  • In the process of completing the evaluation, are you asking your participants to do things or participate in a way that makes them uncomfortable?
  • Participatory evaluation can increase the benefits of the evaluation to the community.
    • Including the perspectives of multiple stakeholders and community members allows for program participants to reduce the burdens of the evaluation, while increasing the benefits LGBTQIA+ community members feel.
• Justice
  • Consider how your evaluation would exacerbate or improve the recognition of health needs specific to the LGBTQIA+ community.
  • In communities where members of the LGBTQIA+ community feel they must keep their identity or orientation hidden so as to avoid discrimination, justice can be difficult to assess when discrimination is entrenched in the systems, structures, and norms of the society.
  • Try to assess how the program affects different groups of people within your program.
    • Consider that there may be groups who are less visible in the LGBTQIA+ community than others, including those who do not speak English, older adults, individuals with disabilities, low-income households, and medically vulnerable people.

Trauma-Informed Evaluation

• Asking someone about trauma is asking that person to recall potentially difficult events from their past.
• If absolutely necessary for the evaluation to ask questions about potentially traumatic events, incorporate a trauma-informed approach to collect data in a sensitive way.
• Amherst H. Wilder Foundation Fact Sheet on Trauma-Informed Evaluation
• More information on Trauma-Informed Evaluation

• It may be beneficial to complete a human-subjects research training, online or in person, if you are working with vulnerable populations or have more questions about ethical review or the ethical review process.
• The Institutional Review Board (IRB) is an administrative body established to protect the rights and welfare of human research subjects recruited to participate in research activities.
  • The IRB is charged with the responsibility of reviewing all research, prior to its initiation (whether funded or not), involving human participants.
  • The IRB is concerned with protecting the welfare, rights, and privacy of human subjects.
  • The IRB has authority to approve, disapprove, monitor, and require modifications in all research activities that fall within its jurisdiction as specified by both the federal regulations and institutional policy.
• Click here for more information on the IRB, including contact information for your local IRB
• Depending on the nature of the evaluation, the IRB may exempt a program from approval, but an initial review by the Board is recommended for all programs.
• If you are working in a community or district with its own public health department, first check with the local public health department before consulting the state-level public health department.
  • If using health records from health care facilities, review patient privacy laws in your area.
  • Most information from care facilities will be in electronic records, which have various safeguards to prevent unethical use.
  • The use and transfer of protected health information (PHI) is regulated through multiple government policies, including the Health Information Technology for Economic and Clinical Health (HITECH) Act and the Health Insurance Portability and Accountability Act (HIPAA).

• Ensure the privacy and confidentiality of all participants. Informed consent from all participants is required.
• Select a sampling strategy that will allow for reporting on program outcomes.
• Consider whether quantitative and/or qualitative data collection methods are appropriate for reporting on program outcomes.

Privacy/Confidentiality

• Privacy and confidentiality of collected data should be ensured.
• In systems of inequality, confidentiality is especially important since the data collected could be used in a harmful manner to perpetuate stigmas against individuals or groups of people.
• It is the ethical responsibility of the evaluator to preserve confidentiality, and the steps taken to preserve confidentiality should be clearly communicated to participants. As an evaluator, ask yourself the following questions when preparing for data collection:
  • Are the evaluators/program staff properly briefed on confidentiality and anonymity?
  • What steps can I take to ensure confidentiality and anonymity?
  • How might my evaluation topic affect confidentiality and anonymity?
  • How might my evaluation methods affect confidentiality and anonymity?
  • What other factors might affect my ability to preserve confidentiality and anonymity?
  • How do I intend to store the data? (It cannot be accessed by anyone outside the evaluation)
  • How can I ensure confidentiality and anonymity in funded research
• Consider how you will secure data collected to protect sensitive information.
  • Data stored electronically should, at minimum, be password-protected.

Sampling Strategies

• Sampling is the way that individuals are selected from a larger group to complete data collection activities.
• Generally, sampling strategies can be divided into non-probability sampling and probability (random) sampling.
• Sample size calculators are useful for determining the number of data points that are needed to detect a statistically significant difference.
• More information on sampling strategies

Choosing a Sampling Strategy

• Is a sample needed or is your program population small enough that it is feasible to survey everyone?
  • For a program with 20 participants, for example, the total number of participants is small enough that it is likely more work to create a truly representative sample.
  • However, if a program is operating in all libraries in the region, collecting data from a sample of participants will likely save time and money and produce similar results as surveying everyone.
• If a sample will be selected, will it be necessary to conduct probability (statistical) sampling?
  • If it is not feasible to compile a list of sampling units, random selection (required for statistical samples) will not be possible.
    • For example, you would need a full roster of participants in all libraries to create a truly random sample. If program confidentiality rules prevent this, a random selection of participants would not be possible.
  • In addition, if you do not intend to generalize to the full participant population (i.e. you won't make conclusions about all participants based on data from a sample of participants), probability sampling is not necessary. Non-probability samples may provide enough information and are less cumbersome to select.

Non-probability Sampling

Probability Sampling

Data Collection Methods

Quantitative Analysis

• Quantitative data are information gathered in numeric form.
• Analysis of quantitative data requires statistical methods.
• Results are typically summarized in graphs, tables, or charts.
• More information on quantitative analysis.

Qualitative Analysis

• Qualitative data are information gathered in non-numeric form, usually in text or narrative form.
• Analysis of qualitative data relies heavily on interpretation.
• Qualitative data analysis can often answer the 'why' or 'how' of evaluation questions.
• More information on qualitative analysis.

• Share information gathered in your evaluation with stakeholders to ensure they understand your program successes and challenges.
• Align stakeholders around the program and allow for planning for future iterations or versions of the program.
• Evaluations can be summarized as evaluation reports, Power Point Presentations, dashboards, static infographics, and/or 2-page briefs depending on the needs of stakeholders.
  • More information on data visualization and communication strategies

Meeting LGBTQIA+ Stakeholder Needs

• Evaluations that are working to reduce disparities within the LGBTQIA+ community need to be transparent and fully detail the evaluation process from start to finish.
  • Evaluations can raise awareness, mobilize supporters, and start the difficult process of social change.
  • However, evaluations can also reinforce negative stereotypes and further divide or blame groups for negative outcomes.
• All outcomes found should be placed within the larger context, including the influence of systemic injustices and systemic issues like government policies, resource allocation, and community prejudices against members of the LGBTQIA+ community.
  • Providing concrete, well-examined data to understand the outcomes in the context of the society in which they take place can move forward positive change.
• Review of findings with community members and stakeholders can help ensure that information is presented in a respectful way.
  • Determining responses to the outcomes found in the evaluation should include the direct involvement and decision making of multiple stakeholder groups and community members.
  • Be open to other interpretations or responses to your evaluation findings. If a program is renewed for subsequent program cycles, then using the findings of the evaluation can improve how responsive the program is to the needs of program participants.
• Disseminating information from the evaluation back to the community that you worked with can help ensure accountability and clarify subsequent actions.
  • It is a sign of respect to include the community that you worked with, and an all too often forgotten piece of evaluation.
  • One method of sharing findings is to hold a seminar with community members where you go over findings and hand out paper copies of the evaluation report to community members.