Delivery methods

This page examines the advantages and disadvantages of various forms of delivery for assessments, paper based (OMR), delivery using standalone applications or web based delivery (Perception, VLEs).

Paper-based delivery

Paper based delivery, using Optical Mark Reading is the simplest way to involve computers in the assessment process. Assessments are still delivered to the student's on paper, with an Optical Mark Reader tallying the exam marks.

Advantages

• The most obvious advantages of paper based delivery are that no special facilities are needed to run the exam. A normal exam hall without computers is perfectly suitable. This allows existing invigilation and exam management practices to be used. The use of paper based assessments is also familiar to the students.
• As the papers are marked by a machine the processing is totally reliable and objective. The settings on the software used to read the papers can be adjusted to accept or reject marks made with different degrees of emphasis, so that if a student has made one response and then rubbed it out in favour of another, the machine will correctly detect the darker mark. Any queries are highlighted to be checked by the operator who can if necessary intervene and correct an error.
• Machine processing of results provides detailed and powerful statistical analysis of the questions which can be used to improve question design in future tests.
• Marking is usually swift so results are usually available swiftly for both students and staff.

Disadvantages

• OMR systems require specialist software and scanners, and also customised pre-printed student response forms which must be purchased for each type of exam. Perhaps less obviously these pre-printed forms include a magnetic code which can only be decoded by its partner scanner, so departments should be cautious about sharing forms with other departments who use a different scanner. Usually a trained operator is necessary to process the papers.
• OMR systems use a pre-printed form for students to mark with their responses, this limits the variety and style of questions that can be used, when compared with other forms of assessment.

Network delivery

Network based delivery involves students using a specialised software application to take the assessment. The assessment itself may be distributed across a local network, or may be copied to each machine used. The software presents the assessment and then marks it automatically.

Advantage

• An assessment set up using a specialised delivery application can deliver complex tests easily. Such applications make it easy to use multimedia and simulations as part of the assessment, with the required files being available on a local server, CD-ROM, or hard disk. Unlike an OMR system such assessments can be marked immediately and automatically by the delivery system, and can involve complex and intellectually demanding question styles.

Disadvantages

• The most obvious disadvantage of such a system is the requirement for specialist software to be available. At best this may require the student to have access to a CD, at worst it requires a specially prepared lab to be available. Not only does this restrict which labs might be used to delivery an assessment, it also means that the use of such systems for formative tests requires a lot more preparation than web based delivery. Unless all University labs have been installed with the required software a student will not be able to use the tests from every machine, and they are unlikely to be able to access the materials from home. The requirement for specific software may also mean than more of a learning curve is required before the student is comfortable with the system, though this can be minimised by good design, or familiarisation outside the assessment context.
• Such systems do allow the individual examiner to construct their own assessments, but the software is often complex and a trained designer usually needs to be involved. Preparation of these materials would usually be expected to take several months.

Web-based delivery

Web -based delivery involves using an exam stored on a server, and then available to students through a web browser. Exams/tests are completed online and marked immediately by the server software, the marks being stored for later use. Perception and the quiz tools within VLEs use this model.

Advantages

• A Web based system is generally easy to set up and use. The presentation of the tests to students involves free software with which they are already familiar and does not require the purchase of specialist reader software, or the installation of particular programs on the student's computer. This makes it especially suitable for formative tests which the student can take from any lab or from home without needing any additional resources.
• Many different types of questions can be used, including some which would be difficult or impossible to provide without the use of a computer. Questions can include multimedia elements, and can be designed to use variables so that very many different questions can be automatically generated from one question. This can be particularly useful when the assessment is formative because students can re-take the test and will be presented with similar, but different, questions on each attempt. The use of variables can also be useful if a summative exam is held in a room where there is some possibility that neighbouring students could easily see each other's monitors.
• As with other types of CAA, machine processing of results provides detailed and powerful statistical analysis of the questions which can be used to improve question design in future tests. Marking is also fast, reliable and objective, and students can if desired, see their result before leaving the test room.

Disadvantages

• Computer Assisted assessment is not appropriate for all circumstances, so time must be devoted to ensuring that the overall balance of assessments in a course is appropriate. There may be some students for whom a computer based test is particularly nerve wracking (but there may also be others for whom it is much less nerve wracking).
• The design and development of questions is crucial. This is also true for paper based exams and tests, but because the analysis of questions is provided it tends to be much more immediately obvious if there has been poor question design in a test which uses CAA. Once questions have been designed they need to be authored using the chosen software package. This can be time consuming but is rarely difficult, and support staff can often undertake this role after a minimal amount of training.
• If the test is a summative examination there may be difficulty in finding a computer lab large enough to accommodate all the students in a class, but there are usually a number of options that are possible.
• Some staff also have concerns about the security aspects of using computer based examinations, but again there are different options available - use of randomisation in questions, or to do with layout in labs, or to do with how authentication to the test is set up.