Good practices in conducting in-house evaluations
- Assign a staff member to be the facilitator of the process: create the questionnaire in Qualtrics or another tool that is sufficiently secure. It is advisable that capable staff be the assigned individuals to provide program leadership with aggregated data/findings and that all data is de-identified before doing so.
- If your assessment will not be conducted anonymously, be deliberate about ensuring confidentiality of the information shared with you. When using online instruments to collect information, or when conducting interviews, limit the number of individuals who have access to the data. Disclose the identity only where it is absolutely necessary for program decision-making purposes; such as in evaluating if there is a need for a change in mentors.
- Best practices suggest that evaluation questionnaires be conducted anonymously. This would be our recommendation as well. While that makes it difficult to assess change or program success at the individual level, it offers a higher chance of honest response and the aggregation of the data for analysis can still provide useful information.
- Do not ask for identifying demographic or professional characteristics if your intention is to anonymously evaluate. If you ask for such information, separate it in the analysis.
- Recognize that analyzing small number of records will always be a challenge when meeting the best practice thresholds for anonymity and confidentiality described above. If the program has a small number of participants, consider alternative methods for evaluation such as conducting focus group discussions that are facilitated by an outside consultant who can then aggregate the themes at a high-level. Check OAA resources for support.
- After a number of years, a longitudinal analysis is possible whereby change over time may be more distinguishable. In addition, some program outcomes may be better recognized through a longitudinal analysis of success indicators in areas such as promotion, retention, productivity and wellness. If this will be a program goal, decide early what your success indicators are in each of these areas; for example, how will wellness indicators be identified and measured. Obtaining direct data from the department may be useful evidence for the program success in some of these areas, such as promotion and retention.