Staff

How should Course Enhancement Questionnaire data be used?

Guidance for the fair and effective interpretation and use of CEQ data.

Evaluate CEQ data along with other indicators of teaching quality

These are discussed in our Exemplars of Excellence in Student Education. We will continue to expand this set of indicators, to make our view of an individual’s teaching as holistic, robust and accurate as possible.

Take account of possible bias (conscious or unconscious)

Unconscious bias can affect how students complete CEQs, and how staff interpret CEQ data. There is research which shows that there is evidence of bias against women and ethnic minorities in the completion of course evaluation surveys. However, awareness of unconscious bias can help to mitigate its effect on data. The following is guidance given to students and staff on unconscious bias. There are further resources relating to unconscious bias at the bottom of this page.

Our staff and students are our greatest assets and all members of the University community should expect to be able to excel, and to be respected and valued for their unique perspectives and contributions. For these reasons we are very sensitive to the potential for bias in responses to Course Enhancement Questionnaires.

What is unconscious bias?

Unconscious bias refers to a bias that we are unaware of, and which happens outside of our control. It is a bias that happens automatically and is triggered by our brain making rapid judgements and assessments of people and situations, influenced by our background, cultural environment and personal experiences. (Equality Challenge Unit: 2013 Unconscious bias in higher education)

The experiences, values, and attitudes we have come across in our lives shapes our view of the world. They can influence the decisions we make, the relationships we have, and the opinions we ourselves hold.

Sometimes we are aware when our judgement is clouded. For example, a new father might say, “I have the cutest baby in the world - but I'm biased.”

However, in many cases, our biases can be “unconscious”, and much more difficult to acknowledge. We can overlook influence of this bias if we don't realise that we hold certain views, and consider why we hold them.

Bias can creep into any situation where individuals have the power to influence the outcome through their decisions and actions. Bias can be costly. It can cause us to make decisions that are not objective, miss opportunities and limit potential. 

Who might I be biased towards?

It is possible to be unconsciously biased against people based on, for example:

  • Gender
  • Race
  • Ethnicity
  • Religion
  • Sexual orientation
  • Nationality
  • Economic background
  • Political affiliation
  • Age
  • Disability

This list is not exhaustive and other biases can and do exist.        

How can unconscious bias be expressed?

  • Bias against people with any of the above characteristics
  • Popular stereotypes regarding any of the above characteristics
  • Opinions based on generalities or one-off experiences
  • Bias caused by long-held values, for example based on our parents’ opinions
  • Bias based on our earliest encounters with the world

How can I identify my biases?

When being asked to evaluate your course, ask yourself questions about your thought process:

  • Is my opinion influenced by any of the above characteristics?
  • What would someone who disagrees with me say?
  • What is preventing me from changing my mind? 

What are some ways I can manage my biases?

  • Evaluate and challenge the first opinion that comes to your mind.
  • Consider whether your decision is based on values or facts.
  • Talk to others with differing viewpoints.
  • Try to remain as objective as you can.

 What are some examples?

  • A first year student is asked what she thought of her first literature course at University, and all her feedback is negative. Inwardly she feels a male tutor would have given better comments on her essays, because her high school teachers, who were male, did.
  • A student feels a female tutor favours only female students and scores the entire course negatively as a result.               
  • An Erasmus student praises a tutor with the same accent as him while overlooking the instructor's poor attendance and inconsistent marking.

 

Take the nature and content of the course into account

There should be an element of professional judgement and common sense on the part of those using CEQ data. This is to ensure that the appropriate circumstances are taken into account when interpreting the data. These may include:

  • Pattern of teaching
    • How much contact has the tutor had with students on the course?
    • Is it team taught or individually taught?
  • Nature of teaching
    • Is the course taught by lectures?
    • Workshops?
    • Labs?
    • Practicals?
  • Perceived popularity of the course
    • e.g. Is it generally thought of as exciting, or vital-but-dull?
  • Practical limitations
    • Was it a large or small class?
    • What was the venue?
    • What time of day did the classes take place?

All of these can introduce conscious and/or unconscious bias, which should be considered when using the data and any free text comments.

Remember that the numbers do not provide fine discrimination

CEQ data is indicative.  The difference between a score of 4.3 and a 4.5 is likely to be smaller than the random variations in the data.  Differences of less than 1 in CEQ data should not be over-interpreted.

Remember that a single positive or negative result is a snapshot of a particular year group and a particular instance of the course.  Unless there has been a known change, success or failure, data should be studied over a number of years to look for sustained quality evaluation.

It is also important to remember that CEQs undertaken for staff who only teach once or twice on a particular course can provide data on those particular classes only. There can be no longitudinal evaluation data for staff who only teach in the short term, and interpreting these data can therefore be particularly problematic.

In relation to individual teaching staff questions, all members of teaching staff involved in the delivery of the course will be included in the survey. If there are more than 10 members of teaching staff involved in the delivery of the course, it is advised that the Head of School or their delegate agrees how this will be managed across the School. 

Data analysis already undertaken on the subject of bias in student surveys

In April 2017, Professor Dave Robertson (Vice-Principal & Head of College of Science & Engineering) convened a Group to explore semester 1 Course Enhancement Questionnaire data, and set the question ‘Is there any evidence of gender bias and, if there is, to what extent?’. This Group consisted of academic staff from each of the three Colleges, including colleagues with an equality remit and union representation.

The analysis of the data shows a small difference in the rating of female and male lecturers, with female lecturers being rated higher than male lecturers. The size of this effect is negligible. It was agreed that the effect, such as it is, could have many explanations other than a gender bias (for example, a confounding effect of class size). Given the small effect size, it was unanimously agreed that there was no useful action that could or should be taken.

Therefore the group recommended that no further analysis of the current (semester 1) data be undertaken in relation to gender bias. While recognising the research into gender bias in student questionnaires, the Group noted the importance of the exercise carried out to look at our own data, and recommended that it continues to be analysed in a similar way at appropriate intervals (to be determined through a Policy review).

A Task Group undertook a review of the Policy over summer 2017 and agreed that the interval for the analysis of staff gender data should be annually when the full academic years’ CEQ results are available, starting in 2018.  This action will be taken forward by the Student Surveys Unit. 

Further resources

IAD - Feedback on your teaching

HR - Overcoming unconscious bias

HR - Academic staff guidance for annual review

University of Edinburgh Dignity and Respect Policy