Guidance for the fair and effective interpretation and use of CEQ data.
These are discussed in our Exemplars of Excellence in Student Education. We will continue to expand this set of indicators, to make our view of an individual’s teaching as holistic, robust and accurate as possible.
Unconscious bias can affect how students complete CEQs, and how staff interpret CEQ data. There is research which shows that there is evidence of bias against women and ethnic minorities in the completion of course evaluation surveys. However, awareness of unconscious bias can help to mitigate its effect on data. The following is guidance given to students and staff on unconscious bias. There are further resources relating to unconscious bias at the bottom of this page.
Our staff and students are our greatest assets and all members of the University community should expect to be able to excel, and to be respected and valued for their unique perspectives and contributions. For these reasons we are very sensitive to the potential for bias in responses to Course Enhancement Questionnaires.
Unconscious bias refers to a bias that we are unaware of, and which happens outside of our control. It is a bias that happens automatically and is triggered by our brain making rapid judgements and assessments of people and situations, influenced by our background, cultural environment and personal experiences. (Equality Challenge Unit: 2013 Unconscious bias in higher education)
The experiences, values, and attitudes we have come across in our lives shapes our view of the world. They can influence the decisions we make, the relationships we have, and the opinions we ourselves hold.
Sometimes we are aware when our judgement is clouded. For example, a new father might say, “I have the cutest baby in the world - but I'm biased.”
However, in many cases, our biases can be “unconscious”, and much more difficult to acknowledge. We can overlook influence of this bias if we don't realise that we hold certain views, and consider why we hold them.
Bias can creep into any situation where individuals have the power to influence the outcome through their decisions and actions. Bias can be costly. It can cause us to make decisions that are not objective, miss opportunities and limit potential.
It is possible to be unconsciously biased against people based on, for example:
This list is not exhaustive and other biases can and do exist.
When being asked to evaluate your course, ask yourself questions about your thought process:
There should be an element of professional judgement and common sense on the part of those using CEQ data. This is to ensure that the appropriate circumstances are taken into account when interpreting the data. These may include:
All of these can introduce conscious and/or unconscious bias, which should be considered when using the data and any free text comments.
CEQ data is indicative. The difference between a score of 4.3 and a 4.5 is likely to be smaller than the random variations in the data. Differences of less than 1 in CEQ data should not be over-interpreted.
Remember that a single positive or negative result is a snapshot of a particular year group and a particular instance of the course. Unless there has been a known change, success or failure, data should be studied over a number of years to look for sustained quality evaluation.
It is also important to remember that CEQs undertaken for staff who only teach once or twice on a particular course can provide data on those particular classes only. There can be no longitudinal evaluation data for staff who only teach in the short term, and interpreting these data can therefore be particularly problematic.
In relation to individual teaching staff questions, all members of teaching staff involved in the delivery of the course will be included in the survey. If there are more than 10 members of teaching staff involved in the delivery of the course, it is advised that the Head of School or their delegate agrees how this will be managed across the School.
In April 2017, Professor Dave Robertson (Vice-Principal & Head of College of Science & Engineering) convened a Group to explore semester 1 Course Enhancement Questionnaire data, and set the question ‘Is there any evidence of gender bias and, if there is, to what extent?’. This Group consisted of academic staff from each of the three Colleges, including colleagues with an equality remit and union representation.
The analysis of the data shows a small difference in the rating of female and male lecturers, with female lecturers being rated higher than male lecturers. The size of this effect is negligible. It was agreed that the effect, such as it is, could have many explanations other than a gender bias (for example, a confounding effect of class size). Given the small effect size, it was unanimously agreed that there was no useful action that could or should be taken.
Therefore the group recommended that no further analysis of the current (semester 1) data be undertaken in relation to gender bias. While recognising the research into gender bias in student questionnaires, the Group noted the importance of the exercise carried out to look at our own data, and recommended that it continues to be analysed in a similar way at appropriate intervals (to be determined through a Policy review).
A Task Group undertook a review of the Policy over summer 2017 and agreed that the interval for the analysis of staff gender data should be annually when the full academic years’ CEQ results are available, starting in 2018. This action will be taken forward by the Student Surveys Unit.