Review of assessment of all taught courses II
The December Senate received a report on assessment from Teaching Committee. The debate at Senate was wide-ranging and inconclusive, though there were some relatively clear conclusions. This paper provides a summary of those conclusions and identifies some ways forward. The statistical research is being completed, though there has been some delay in extracting the data, and a further paper will be prepared before Teaching Committee.
The following points were agreed:
There should be a common scale for reporting marks across the University.
The common scale for reporting marks should be the 100-point scale.
Departments should continue to be allowed to mark on whatever marking scale that they find most appropriate, including the 16-point scale, provided (a) that they than convert the marks to the 100-point scale for reporting purposes, and (b) that they provide clear documentation on the use of marking scales in student handbooks.
The following points were not agreed:
The qualified median should be adopted as a basis for determining degree classification. Although there was no decision on the point a number of Senators argued either that faculties/departments should be free to determine the basis on which degrees should be classified, or that there should be a combination of methods. It was agreed to postpone the decision on this point pending further research and a recommendation from teaching committee.
Departments return marks on three points within the scale, again to determine achievement within a particular class, and to distinguish between basic, solid and outstanding performance within the particular class i.e. basic 2ii. Senators argued that departments should be free to return marks as real numbers and not be confined to 16-19 marks.
The following points were not conclusively discussed:
There should be a standard definition of marks on the 100-point reporting scale, which might be as follows:
The critical issue here concerned the recommendation to define a Pass at 35-39. This marks a significant change to past practice, but provides a mark for the third class degree that is more consistent with national practice.
There should be a common scale for converting marks to the 16-point scale to the 100-point scale, which should be used by those departments continuing to convert marks to the 100-point scale. The difficulty here concerns the scale above 70, which is used in different ways by departments in different disciplines.
This paper considers the contested and unresolved issues and makes recommendations about each. It considers, in turn, the following issues:
The method for determining degree classification
The standard definition of marks, including the threshold for a third and the marks for a pass degree.
The use and form of a conversion scale.
The manner in which marks should be returned.
The method for determining degree class
The consultation on assessment had revealed disagreement on the issue of which method should be used for determining degree class. Eleven departments and two faculties expressed a preference for the median, eleven departments and one faculty expressed a preference for the mean, eleven departments and one faculty expressed no view, preferring to wait upon further research, while nine preferred to use a combination of methods.
This division was reflected in Senate and it is clear that Senate will not agree to a single method for determining degree class.
It is also apparent that the divisions between departments and faculties reflects the same difference in approach to assessment as that which underpinned the debate about the 16-point scale. Those departments that wished to continue with the 16-point scale preferred, on the whole, to use the median or the distribution method to determine degree class, whilst those departments that wished to move to the 100-point scale wished to use the mean.
The advocates of the median argue that:
It more accurately reflects a students overall performance;
One or very few, high or low marks do not have a marked influence on degree classification.
The determining factor is the class of the median mark and so the difficulty of using a fine scale consistently is reduced.
The opponents of the median argue, in contrast, that:
The effects of very high or very low marks are minimised.
This is a problem in those parts of the university where there is a considerable variation in the marks awarded to a student in different parts of the course, leading to a non-normal distribution of marks. They would argue further that advocates of the median needn't worry, because where a student has a normal distribution of marks, the mean and median would be the same.
The median as the primary determinant of degree class. It is clear, from the consultation and the debate as Senate, that advocates of the mean will not accept the median. This is precisely because effects of very high and very low marks are minimised, and because there is frequently considerable variation in the marks awarded to a student in different parts of the course.
The mean as the primary determinant of degree class. It is not so clear that advocates of the median would not accept the mean, provided borderline students could be identified and provided that the current discretionary regulation was retained. The parts of the university that favour the median are those parts where a student tends to have a near normal distribution of marks, and the mean and the median are similar.
A combination of methods e.g. combining mean and median. A significant minority of the university is in favour of this approach. This would be unproblematic for those departments in which the mean and median converge, but it would be extremely difficult to work in those departments where the mean and the median diverge. Either the median or the distribution method could be used in those departments to determine borderline cases, but the mean would need to be the primary determinant with the median becoming a factor not a determinant in the consideration of borderline cases.
A plurality of methods. This would involve the university working with more than one method for determining degree class, with some departments using the mean and some using the median. The regulations permitting this choice would need to be drafted with some care and departments would need to choose a particular method. This pluralist option would need to be regulated and departments would need to submit their scheme for approval by a sub-committee of Learning and Teaching Committee. The scheme to be used for a given degree would need to be declared in the regulations for that degree.
The major and obvious difficulties would arrive in those departments and for those students that have dual degrees. The department of philosophy, in its response to the initial consultation, argued that such a scheme 'would be an administrative nightmare, and might lead to a situation in which two students in this department with the same runs of marks would receive different degree classifications'. Dual degrees may be a minority, but any system must be compatible with the needs of those minorities, where they represent a significant component in that system. In such circumstances it would seem to me that both the mean and the median should operate. Students would therefore be entitled to the degree class provided by either the mean or the median.
I would recommend option B, with the mean as the primary determinant of degree class, with a clear identification of borderline categories and with departments using clear and transparent methods for determining such borderlines. To facilitate the use of the median in determining borderlines this should be provided through student services.
The standard definition of marks, including the threshold for a third and the marks for a pass degree
There are two questions that could be considered here. The first is whether the university should continue to award pass degrees, and the second is the mark needed for the award of a pass degree.
I think there is a strong case for abolishing pass degrees. I would view it as a consolation prize, as a residual award available only to those who had failed to reach a satisfactory standard. It would seem to me more appropriate to say that such students had failed.
There is an equally strong case for arguing that the third class degree should be awarded, as they are in a significant number of universities, for students who have achieved 40% or more.
If the university were to retain the pass degree I would propose a mark of 35%, as the threshold.
The use and form of a conversion scale
A significant number of respondents to the consultation expressed a preference for the 16-point scale, and many expressed an intention to continue using it for the purposes of marking, even if they then had to convert for reporting. A common conversion scale across the university would facilitate this process and would enable markers to retain the emphasis on a 'basic 2ii' or a 'good 2i' rather than moving towards the fine scale of marking implied in the 100 point scale.
It would be relatively easy to produce such a conversion scale up to 70%. Thereafter the different approaches to marking first class papers would provide a very real problem. Annex A provides two attempts to deal with the problem. The first, scale A, divides the first class category into three marks, corresponding to the three marks currently allocated to a first class performance on the 16 point scale. The second, scale B, allocates a further two marks within the first class category to reflect the traditions of marking that apply across the different sections of the university.
I think there is a strong case for introducing such a conversion scale and for using the five categories in the first class range. In many parts of the university there is still a preference for focusing on where a performance falls within a class i.e. basic, solid, good, rather than focusing on the specific mark within a 100.
The manner in which marks should be returned.
The strength of the case for focusing on class, or place in a class, rather than the fine grain of a particular mark, led to the proposition that all marks should be reported out of 100, but only at three points within each class, or five, in the case of a first class mark. There was little support for this proposition at Senate. For some this could be seen as a last-ditch attempt to defend the virtues of the 16-point scale, but for others it seemed innumerate, irrational, and even lunatic. I am prepared to surrender to that judgement.
Annex 1 - A common conversion scale