Network Learning - Can it meet the Quality Challenge?
Professor Mike Holcombe,
Deputy Dean of Engineering,
Department of Computer Science,
211, Portobello Street,
University of Sheffield
Sheffield, S1 4DP.
0114 222 1802
Key words: quality assurance, academic standards, networked learning, distance learning.
The issues of quality assurance in the provision of education and the question of academic standards have emerged as major issues in higher and further education in the past decade. Recent developments such as the Dearing Report (and the Garrick report in Scotland), the Teaching Quality Assessment processes carried out by the Higher Education funding councils and the work on Continuation Audit and the Graduate Standards Programme by the Quality Assurance Agency are focusing public attention onto the important matters of whether there is confidence that the educational provision of an increasingly diverse system is of the quality that is being demanded by employers, governments and students. These are questions that will be a major challenge to all sections of the educational community but in particular the rapid growth of novel means of delivery such as virtual universities and networked learning environments create a number of new and essentially different problems to solve.
In this paper we look at some of these issues in the light of experiences of quality audit and assessment based on traditional modes of learning and examine how the emergence of networked learning affects our perceptions of quality. We also consider how the networked educational provision could develop in order to give confidence to consumers, purchasers and society in general that the quality of provision is of an acceptable standard.
vv v v v v
One important aspect that has to be considered of any publicly available service or product is its quality. This is true as much for education as for other sectors of the economy. The public perception of the quality and standard of awards obtained through networked learning schemes must not be compromised if networked learning is to flourish and make a significant impact in the campaign to build a highly skilled and educated society. Worries about the comparability of, for example, an MSc obtained through distance learning over the Internet with a traditionally delivered degree with essentially similar academic content must be addressed.
Questions such as the following have to be answered:
how can the awarding institutions be assured that the work of the student is of an adequate standard for the award of credits, degrees and other qualifications?
how can the quality agencies, (e.g. QAA, formerly the Higher Education Quality Council, HEQC), be assured that the institution's quality mechanisms and culture can effectively ensure the quality of their educational provision?
In this paper I reflect on my experience as an assessor and auditor for HEFCE, QAA and others, as well as an educator involved in distance learning, and try to examine these issues. A number of important problems and potential weaknesses in the current provision of this type of learning are discussed. A set of quality guidelines are proposed in an attempt to identify organisational, educational and quality assurance procedures and principles that could go some way to creating confidence in the public at large that the networked learning provision being developed in many universities and colleges is of a high standard and the qualifications being achieved are worth having. Failure to achieve this could do permanent damage to the development of networked learning.
vv v v v v
1. The quality challenge
There has, over the past decade or so, been a major expansion in the higher education provision. This expansion has involved new types of academic subject, new categories of student and, more recently, new styles of delivery. The expansion has been world wide and a university may now have many students located on many campuses in many countries and it may have many students located on no campus. Technology has evolved very rapidly to a point where a student could study for a qualification without leaving their home, utilising electronic communication technologies for receiving information, for submitting work for assessment and for discussing issues and their work with peers or tutors. With this plethora of provision and qualification questions are being asked about the relative standing of some of these activities vis a vis the traditional university degree and its learning environment. A number of "high profile" reports in the press have highlighted the fact that some of this activity is of dubious quality both in an intellectual sense and from the perspective of value for money for the student. Many employers, both in the UK and abroad, are also criticising the quality of some of the graduates that they are recruiting from these programmes and this has led to calls for a more rigorous approach to the assessment of the quality of these degrees.
What is quality?
The OECD report (8) defined the quality of an educational provision in terms of three broad criteria:
vaccess - meaning the measurement of the numbers who start and successfully complete a course of study;
vquality - the success of graduates/diplomates in later life that was instigated by their educational experiences;
vthe efficiency of the educational system - its cost-effectiveness.
These definitions are a useful starting point to more practical and observable metrics for monitoring quality and using this information to improve and develop the educational provision. Clearly, the second criterion cannot be measured in a short enough timescale to provide an effective feedback mechanism. The other two criteria, however, can be developed into three broad categories (8) :
vthe quality and quantity of resource available in the institution. These are attributes that can be defined and measured in such a way as to provide comparators and thus standards.
vselectivity or exclusion - it is widely believed that the best education is obtained when the fewest students are in contact with the greatest educational resource.
vthe "silent", critical and creative conversation that takes place within the learners mind which is stimulated and supported by their learning environment. This will involve, in the best situations:
wconversing with students, staff, experts, business, the community;
wexchanging ideas with others;
winteracting with resources;
wlearning by doing, solving problems, critical evaluation, arguing.
In fact, argument - the ability to marshall knowledge, understanding, critical and rational analysis with confidence to produced a well-founded, structured and incisive position - should be the key educational attainment we wish our students to develop together with the humility and professionalism to accept when a protagonist has the better case. This can only really be achieved in the context of personal dialogue with one's peers and mentors.
The first important stage in the development of a unified, nationally organised, quality assessment system for teaching was introduced by the Committee of Vice-Chancellors and Principals (CVCP) in the first phase of academic audit. This sought to establish what quality assurance mechanisms universities had to support their main teaching functions. In subsequent phases of this activity the Higher Education Quality Council (HEQC) and more recently the Quality Assurance Agency (QAA) have developed and extended this activity to include all of the Higher Education sector and the many and varied collaborative arrangements that UK institutions have set up throughout the world, these include franchise and validation arrangements with many other organisations. Academic Audits (and the more recent Continuation Audits) involve visits by trained academics to an institution to examine its quality structures, procedures and to look for any evidence that they are working appropriately. The outcome of an audit visit is a report describing their findings, appreciating good practice and making a number of recommendations that they believe should be considered by the institution in an endeavour to improve the quality of their educational provision..
Up to the present the main emphasis of these audits is on the quality processes. Under Continuation Audit there are two fundamental questions that focus the investigations. They are (4):
1. Can you tell us how you know whether you are discharging effectively your responsibility for the standard of each award granted in your name and for the quality of the education provided by you to enable students to attain that standard?
2. Can you convince us that the evidence that you are relying on for this purpose is sufficient, valid and reliable?
So, now, institutions are assumed to have in place an overarching quality process which is implemented throughout the organisation. The audit team has to critically examine this and evaluate its effectiveness. In doing this the team will look for evidence that the system works and achieves its objectives. When it comes to an institution that has a significant collaborative provision, perhaps franchising their degrees to associated colleges, here or abroad, or putting its seal of approval on another institutionís qualifications through validation, the team will be very interested in examining how the lines of communication between the various institutions allow for effective quality assurance mechanisms to operate. This is often an area where significant problems can be identified which may be a serious cause of concern. The QAA is also actively visiting partner institutions overseas to examine the workings of the quality processes from both ends.
Areas of activity that come under the audit scrutiny are:
1. The institution's quality strategy - how is it defined and how does it work?
2. The quality and standards of programmes and awards - is the institution satisfied that it has effective mechanisms for ensuring that its academic objectives are achieved and it is maintaining academic quality and standards at an appropriate level?
3. The learning infrastructure - what are the quality assurance mechanisms for ensuring the quality of the educational provision that involves information for students, tutorial support, laboratories and library facilities, staff development, complaints procedures and many other day to day aspects of learning?
4. Internal and external communications - are there appropriate policies in operation to ensure that all aspects of communications, between students and staff, between staff and administrative sections, between the institution and the outside world etc. are effective and timely?
Another, currently parallel, activity that has developed over the last few years is that of Teaching Quality Assessment (TQA), undertaken initially by the Higher Education funding councils (HEFCE, SHEFC, HEFCW) and since subsumed into QAA. This involves visits by panels of subject specialists to examine the learning environment in an individual department. Panel members will examine the course curricula, the teaching materials and will experience teaching sessions, they will meet staff and students and, hopefully, come away with an accurate picture of the quality of education in that department, defined according to a structured template of criteria.
Aspects of the subject provision examined include:
1. Curriculum design, content and delivery.
2. Teaching learning and assessment.
3. Student progression and achievement.
4. Student support and guidance.
5. Learning resources.
6. Quality assurance and enhancement.
vv v v v v
2. Networked learning
The definition of networked learning has to be broad enough to include most of the current provision as well as that which is likely to come on stream in the near future. The recent technological changes that underpin communications and information technology have transformed possibilities in many different activities. For education it promises an environment where individual students can access information from almost anywhere through telephone and satellite media as well as the traditional means of books and personal contact. In a similar way, a student can contribute to a discussion from a remote location through electronic mail, video conferencing and the Internet. A student can submit work electronically from a remote location for assessment by a tutor or perhaps a computer, and can receive feedback on these assignments through the same medium. Groups of students can work collaboratively on a group project and the management of these activities by a tutor can also be organised from a remote location. Examinations could be sat using the same media. In theory, a student could go through an entire degree course without leaving their homes or meeting a fellow student or teacher "in the flesh".
The relationship between the student and their institution will change, in fact a student may have many institutions as they take advantage of the many opportunities that are available to them. A university will become more of a gateway to learning, providing signposts to the most suitable learning resources that will be available from almost anywhere. Rather than the student going to the teacher the teacher will go to the student, albeit through the medium of the communications technologies. There will be the freedom to choose from many different learning resources and for students wishing to develop their knowledge in specialist areas they will be able to "move" to an institution that provides that speciality - which may be a different one from where they started. Thus the concept of lifelong learning can be embedded in the network learning provision.
Within the context of the many modes of networked learning there are a number of aspects which might, at first sight, seem to be unique to that style. For example the use of electronic mail for conversations and instruction, the use of bulletin boards and computer conferencing for debate are all methods that are also now used within the traditional style of education. However, these are all backed up by more personal contact and interaction between students and teachers. It is sometimes forgotten that this collaborative style of learning is enormously important for most people. The lone network learner may be at a disadvantage in this context. Some subjects are inherently difficult to teach remotely and rely on the direct interaction with teachers and peers. This is true, for example, of learning computer programming. In this context there are two teachers, the expert and the computer compiler. The former is, hopefully, sensitive and encouraging, the latter is rigid and merciless. Struggling to achieve success without substantial help from a human tutor can be fatal. It is like trying to learn the French horn over the telephone!
The process of one student explaining to a fellow student some tricky concept or solution can give great benefits to the former as well as the latter. This peer interaction is, in fact, a major, if largely unrecognised, component of any learning process. Thus we must ensure that mechanisms exists to allow distant students to talk to one another. When there are major differences in time zones this may be difficult. If the numbers of students involved is large it will require very substantial human and technical resources. Without providing mechanisms for this interaction the remote student's learning environment will be severely restricted.
My own experiences of remotely supervising an MSc student on a distance learning degree have not been entirely successful. In one case the student totally ignored my advice about the direction of her project and eventually produced a hopeless discussion of the software system she was using which was more akin to marketing hype than scholarship. This might have been avoided if we had met.
vv v v v v
3. Quality issues for networked learning
The internet is a complex, chaotic system with few boundaries and, as such, it presents us with many opportunities, but also many dangers. The relationship between the student and his/her institution will be very different. In a world of credit transfer it is conceivable that a networked student could pick and mix their education by choosing modules available over the network from many different providers, and, if successful in receiving credits for their endeavours, to cash them in with an educational provider willing to accept them for an award. In such a free market it is likely that some institutions will start marketing their modules/credits in an aggressive manner, looking for mass markets in an effort to generate income. This may well be at the expense of quality and academic standards.
It is an important responsibility of any institution prepared to accept credits obtained from elsewhere as a component towards its awards, to assure itself of their standard and reliability. The institutions with the most to loose are those with the highest reputations. If many of these credits are obtained through distance learning programmes and networked learning the difficulties in assessing their quality and standing are immense. Equally, any network learning provider has to provide assurance to both the general public as well as other institutions that their educational provision and academic standards are of the highest quality. Recognising credits from institutions whose quality assurance mechanisms are either unknown or suspect will be risky. In traditional education, for example at the postgraduate level, the ability to monitor students with uncertain qualifications closely has been of vital importance (their fate will inform future selection policy). If this is not achieved then in the medium to long term their reputations will suffer and once lost these are difficult to regain. Furthermore, in the UK, the quality assurance agencies will want to look very closely at these practices and it will be very difficult to convince sceptical auditors that all aspects of the provision are being closely monitored by the awarding institution. It is not enough to claim, for example, that the University of X has been rated "excellent" for this subject and so we can take their quality for granted. Agencies such as QAA repeatedly comment that reliance on external quality assurance mechanisms such as TQA and RAE are not a substitute for the institutionís own procedures, (8).
There are a number of quality issues that we need to address. These can be broadly divided into:
1. The generic issues concerning the overall quality strategy and provision of an appropriate learning and assessment process that might be the subject of an audit;
2. The specific subject based learning provision and those quality aspects that would be the subject of a teaching quality assessment;
3. The issues of the academic standards of the award and how these are defined and maintained.
The overall quality strategy
This needs to address the aspirations of the institution and to provide suitable mechanisms for defining and reviewing the quality process as well as ensuring that that process is fully and enthusiastically implemented throughout the institution. This must apply to the way the institution presents itself to the outside world and to prospective students and employers, to the way it selects and admits students to its courses and to the learning provision that it provides. It is important that curriculum design is carried out in a responsible and careful manner with external consultation wherever possible, that the courses offered can be delivered with the resources available and that the quality of provision is not compromised by lack of adequate planning or the capabilities of the available technology or of those delivering it. Having a wonderful course which cannot be delivered properly because the electronic infrastructure is inadequate, either at the university end or at the studentís end is a recipe for disaster. All of these issues must be part of the focus of the quality strategy.
The quality strategy must also address the provision of higher and research degrees, (6). Here the tradition is of a much closer individual relationship between the teacher and the student, almost a partnership in many cases, particularly in PhD degrees. In many subjects the existence of a group research culture is a vital part of the research environment and the community of researchers provides a rich support and evaluation culture which may not be easy to create using electronic means only.
The subject based learning provision
The resources required to satisfactorily learn must be clearly identified and planned. In many situations the course will be based on existing books or course notes that have been developed for traditional methods of education. In other cases these will be supplemented or even replaced by electronic material developed or collected for the specific purpose of supporting the course. The quality of much of this material, for example WWW archives, can be high, equally, though, it can be very poor.
Consider the issue of electronic scholarly materials. Grycz (2) identifies a number of important quality issues from the perspective of the librarian. In the traditional world of printed and bound scholarly texts the concept of quality sprang from the reputation of the publisher, the look and feel of the book itself - a well printed book on high quality paper, properly bound was some sort of guarantee that the contents were good. The main provision in the traditional library is stocked and maintained by professional librarians in consultation with academics. Many of the producers of the electronic and WWW based materials are amateurs, there is not always a referee or review process for this material. Now the responsibility is with the academic. Decisions may be made on the basis of electronic availability rather than suitability. The components selected may not provide a balanced or representative account of the subject. There is a world of difference between the considered authority of a major scholar in the field who has invested a lifetime of understanding and involvement in a subject in his/her book to the construction of a, possibly random, collection of web sites chosen by an inexperienced academic. The cost of accessing officially published electronic sources may also be an issue and, to keep fees down, free web sites might be preferred.
In many of the most highly rated institutions involved in the HEFCE Teaching Quality Assessment programme the key factor that established excellence was the existence of effective small group teaching. This ranged from the frequent one to one tutorials and supervisions for almost all students that take place at universities such as Oxford and Cambridge to the slightly larger classes at the traditional redbricks. These tutorials have the aims of stimulating, challenging and supporting students in their studies and, when done well, represent one of the most effective ways of interacting with students in a creative learning environment. It is expensive, of course, but planned well and in the context of a curriculum organised in a complementary manner it is surprising how many students can be accommodated in such an environment. How can networked learning compete with this? There have been some attempts to replicate this activity using electronic mail, computer supported cooperative work (CSCW) systems and video conferencing. In a small group, of 2 or 3 it is probably possible to go some way towards creating a similar learning environment. With larger groups it is probably impossible. The problem with large groups is that of involving all the participants. This is difficult if they cannot be seen, unless there is some mechanism for tracking each personís contribution. So can a mass networked learning system ever aspire to these levels of excellence?
The academic standards of the award
The academic standard of an award is inextricably linked to the summative assessment for that programme. Although progress has been slow in establishing an agreed framework for defining the academic standard of an award (5) there are a number of observations that can be made. The concept of ígraduatenessí, whilst appealing is inadequate to reflect the very wide diversity of provision that there is in the UK, let alone the world. It is disingenuous to suggest that a degree in a subject from one institution is equivalent in standard to that of another. There is a league table, even if it is unofficial. The resource base, which affects standards as well as quality, includes libraries, laboratories but above all staff and is very uneven amongst institutions. The quality of students, measured on the convenient yardsticks of school performance is also very variable between institutions. To anyone who has been an external examiner or an assessor it is quite apparent that the graduates from one particular department can do much more than graduates from another one. The standard of examination papers, project work and viva performance we see varies tremendously from institution to institution. Many employers, and the general public know, and understand this, furthermore they have perceptions as to which institutions are best in this respect. These impressions may not be based on much direct, published evidence, but they do exist and we have to recognise that. Press reports on academic scandals involving some institutions and their franchising operations merely reinforce these views.
In the UK and elsewhere, the legitimacy of an institutionís academic standards are founded on the external examiner system. Experienced and conscientious examiners will demand the highest standards possible in the programmes they monitor. They will carefully scrutinise all the assessed work, be it formal examination papers and scripts, coursework or projects. They will wish to assure themselves that the examining process is fair and that students receive the awards they deserve. In the best examples of this the examiners will meet some or all of the students, perhaps to participate in viva voce examinations in order to deal with borderline cases or, in the case of postgraduate degrees to examine theses. This direct contact between examiner and student is an important issue when we come to networked and distance learning programmes.
Here the big issue, in situations where the student is remote from the centre of the examining process, is - is the studentís work their own? As education becomes more valued and more expensive the pressures on a student to succeed increase. We have noticed a very significant increase in the amount of cheating that goes on - specifically copying or stealing coursework that is to be submitted for assessment. This is now compounded by the availability of material in electronic form through the Internet. The temptation to plunder these resources and pass them off as your own is considerable. It is up to the lecturer to recognise this plagiarism and to police it rigorously. This is somewhat easier when you can interview the student directly and regularly in person. At the end of a network link, however, who knows who is there and what they have done?
With novel types of examination, such as computer marked question papers and computer based assessment of various kinds, the ability of the examiner to assure themselves that the candidateís work is indeed theirs becomes almost impossible. This is a major concern for the academic standards of network learning programmes.
With higher degrees it has the potential of becoming a scandal. I have heard of distance learning masters degrees where all the assessment is coursework based and there is no viva examination for the dissertation. There is no possibility that an auditor/assessor could be assured that this arrangement could be adequate to ensure that the right person was being awarded the degree!
v v v v v v
4. Some conclusions and recommendations
From the preceding analysis there are a number of specific issues that arise in the context of the quality assurance of networked learning. We will concentrate on these, but noting also that many of the normal quality strategies, processes and policies which apply to all types of higher education will also be appropriate in this context.
The following guidelines are suggested in order to provide a framework for answering the quality question for networked learning.
·Students should be provided with as many mechanisms as possible for interacting with their peers. This may take the form of purely electronic means such as electronic discussion groups but could also involve summer schools and other means by which they can meet and discuss their subject and learn from others.
·Staff must maintain regular contact with individual students to reinforce their learning, to inspire them, to challenge them and to ascertain that they are proceeding satisfactorily.
·Learning resources must be of the highest quality, this does not just mean that they are beautifully presented but the content must be exemplary and should incorporate the finest scholarship. The lone student has enough obstacles to learning - they should expect the best from their scholarly materials.
·Assessment must be rigorous and fair. It is highly undesirable to rely solely on coursework for taught degree programmes. The identity of who has done the work has to be transparent and plagiarism and cheating must be eliminated. This will probably involve the setting of supervised examinations which will have to be located and managed with scrupulous fairness.
·Major pieces of work such as postgraduate dissertations and theses (and final year projects in high quality undergraduate degrees) must be examined stringently and defended by the student in front of competent examiners.
·The accreditation of prior learning through the recognition of credits awarded by other institutions must be carried out with great care. If there is some doubt further evidence of achievement must be sought.
·The over-riding principle is to ensure that the quality of the experience of the student and the academic standards of the awards is not compromised by virtue of the means of delivery. If this happens the promising advantages of networked learning and lifelong learning, of liberating education from the centralised campus, will evaporate.
vv v v v v
In conclusion, if the purpose of networked learning is to distribute the opportunity for suitably qualified people to study in a high quality learning environment for awards of a good academic standing then, provided that the issues raised above are addressed adequately, its future will be a success. Many more individuals will be able to benefit from it than would be the case with traditional forms of delivery.
However, if the purpose of networked learning is to make higher education available to the maximum number with the minimum cost then it will be a disaster. The awards received by students may well be of little value and the learning experience a frustrating and fraudulent one.
vv v v v v
1. Department of Education and Employment, Higher education in the Learning Society, Report of the National Committee of Enquiry into Higher Education, (Chair: Sir Ron Dearing), HMSO, 1997.
2. C.J.Grycz, Quality Indicators in a Networked Scholarly Environment: New Roles for Stakeholders, in Scholarship in the New Information Environment, Ed. C. Hughes, Research Libraries Group, USA. 1996.
3. HEFCE, Teaching quality assurance guidelines, HEFCE, 1995.
4. HEQC, Guidelines on Quality Assurance, 102 pp., HEQC, 1996.
5. HEQC, Graduate Standards Programme - Draft report and recommendations, 103 pp. HEQC, 1996.
6. HEQC, Guidelines on the Quality Assurance of Research Degrees, 28 pp., HEQC, 1996.
7. OECD, Information Technology and the Future of Post-Secondary Education, OECD Documents, Paris, 1996.
8. QAA, Report on an academic audit of the University of Sheffield, QAA, 1997.