2015 examination results by Umalusi; National Certificate Vocational certification backlog: SITA progress report

Meeting report

The Committee Secretary tendered the apologies of the Minister of Higher Education and Training, Dr Blade Nzimande, and the Director General of the Department of Higher Education and Training (DHET).

Briefing by the Council for Quality Assurance in General and Further Education and Training (Umalusi)
Dr Mafu Rakometsi, Chief Executive Officer, Umalusi, reminded the Committee that Section 17 of the General and Further Education and Training Quality Assurance (GENFETQA) Act stipulates that the Council must conduct assessments, free from any irregularity that may jeopardise the integrity of the assessment, comply with the requirements prescribed by the Council, apply the standards prescribed by the Council and comply with every other condition determined by the Council. The framework for quality assurance of assessment includes the evaluation and accreditation of assessment bodies, periodic inspection of assessment systems, on-going monitoring of assessment systems, monitoring and moderation of School Based Assessment (SBA), moderation of marking and standardisation of assessment outcomes.

Ms Faith Ramotlhale, Senior Manager, Umalusi, said the purpose of the moderation of question papers is to ensure that the question papers are of the required standard, and that papers are relatively fair, reliable, and representative of an adequate sample of the curriculum and of relevant conceptual domains. The criteria for moderation of question papers relies on several factors such as technicality, internal moderation, and content coverage, cognitive demands, marking guidelines and language and bias.

She noted that in the 2015 year, there had been improvements in that there were no leakages of question papers for the Independent Examinations Board (IEB), BAA (independent board) and DHET examinations. 98% of the question papers for the General Education and Training Certificate (GETC) were approved, while 97% of IEB papers were approved on compliance ratings.

The directives for compliance that Umalusi had set. All three assessment bodies should improve the quality and standards of internal moderation and increase the percentages of question paper approved at first moderation. The DHET should improve on the quality of marking guidelines. The quality of the question papers for the GETC should be improved, as there had been a low standard of papers since November 2013. A review of outdated and underspecified curricula for the National Accredited Technical Education Diploma (NATED) subjects was needed. Umalusi said the DHET needed to develop an assessment framework for the NATED N2 and N3 subjects to address the cognitive demands and the weighting of topics. DHET is also required to review its tracking system for question paper moderation, to ensure that correct versions of question papers, marking guidelines, internal moderator reports and assessment frameworks are submitted to Umalusi for external moderation. The Department has to adhere to its management plan for the setting and submission of question papers for moderation.

The internal assessment for the School Based Assessment (SBA) refers to an assessment conducted by the provider, whose outcome counts towards the achievement of the qualifications. The purpose of the internal assessment is to ascertain the degree to which the assessment body is attempting to ensure standardisation across providers, to check the standard and quality of the task, to establish the extent and quality of internal moderation and feedback and verify the reliability and validity of the assessment outcomes. The site-based assessments for GETCs showed areas of good practice and a year-on-year improvement regarding the administration of SBA. Umalusi had implemented a self-evaluation instrument and visited centres to support and improve the implementation of SBA tasks. The directives for compliance for GETCs would address the finding that assessors do not understand the use of rubrics and marking guidelines. The DHET and IEB should develop a strategy to monitor and support the implementation of the SBA tasks, to improve the quality of teaching and learning. The BAA must implement a strategy to ensure that training providers implement all SBA tasks.

The directives for compliance for the internal assessments of National Certificate Vocational (NCV) internal continuous assessment (ICASS) requires the DHET to improve training on the internalisation of Subject and Assessment Guidelines (SAGs) to promote effective planning, facilitation and conduct of assessment. Lecturers required training in setting and moderation too. The DHET is also required to enforce the implementation of the NATED ICASS guidelines at all sites to ensure that appropriate assessment task are developed and marks are correctly converted and captured. BAA and IEB are required to improve poor compliance with examination regulations, by improving inadequate storage facilities for question papers with poor security arrangements, poor administration regarding examination regulations and the use of cell phones during examinations.

The monitoring of examinations for Technical and Vocational Education and Training (TVET) colleges showed that there has been an improvement of system in the conduct of examinations and measures to ensure the security of examination materials. There had also been effective training of chief invigilators and a favourable environment for the writing of examinations. However, the DHET should ensure that any corrections to papers must be sent on time at the various examination centres. The DHET should also ensure that all examinations have adequate and reliable security for the storage of examination materials.

The marking phase is also monitored for the GETC. The results showed that marking centres were generally well organised and suitable for the task, there were adequate and experienced security personnel at the gates and entrances to the venues and training of marking personnel was generally thorough and effective. Areas where concerns were raised in relation to Assessment Bodies included ensuring the availability of the management plans, adequate security and poor systems to manage and report irregularities. During the monitoring of TVET colleges' marking phase, Umalusi found that the DHET had not ensured that the approved marking guidelines were received in good time at all marking centres, and the appointment of markers had not been completed well in advance.

The verification of marking was done in order to determine the standard and quality of marking and ensure that marking is conducted in accordance with agreed practices. Umalusi had held several discussions, including a memorandum discussion, panel discussion and the marking of exemplar scripts, all aimed at improving the quality of the marking guidelines. DHET must ensure that it improves the quality of Afrikaans versions of most marking guidelines. It must develop a system to ensure that the final memorandum are discussed and cascaded at provincial level and the BAA should prioritise the training of markers before the actual commencement of marking. The marking guideline discussion and verification of marking at TVETs improved, as more markers came prepared to the marking centre and in most subjects there was good adherence to the marking guidelines and markers kept notes to enhance the quality of chief marker and internal moderation reports. Umalusi did feel that the DHET should improve the quality of internal moderation and comply with the 10% internal moderation of all scripts in all subjects.

The current status was described. The subject structures are being verified, candidate registration has been verified during the state of readiness visits, the generation of mark sheets and capturing of marks has been monitored, standardisation data and sheets have been received and verified, as well adjustments and statistical moderation captured.

Statistical moderation was done for all bodies. In GETC this showed some room for improvement. The IEB should develop its internal quality assurance of standardisation booklet to ensure that accurate data on graphs and statistics is submitted. The BAA should document the processes and procedures outlining standard operating procedures, with regards to capturing and results. The scope of standardisation in 2015 showed that 59 engineering studies and four business studies were standardised by NATED and 252 were standardised through a NCV. 28 engineering studies and two business studies were adjusted, mainly upwards, through NATED and one was adjusted mainly downwards. 127 subjects were adjusted upwards through the NC (V) and two were adjusted downwards.

Dr Rakometsi concluded that the quality assurance process followed Umalusi criteria, and that Umalusi would subject these to constant review and refinement to ensure that they are in line with current trends in assessment and examinations. Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes, to make this a meaningful qualification. The out-dated and underspecified curriculum and lack of subject assessment guidelines for the NATED programmes created problems for appropriate quality assurance to be conducted. He did commend the DHET for radical actions taken to curb paper leakages in the NATED examinations. Finally he noted that Umalusi was appreciative of the DHET's concerted efforts in delivering credible examinations in 2015.

Discussion
Mr E Siwela (ANC) said that the presentation showed some improvements from the previous year as Umalusi had ensured credible results. He asked Umalusi what had caused the decrease of the students’ final marks. He asked what the IEB had done to ensure that there is an improvement with the students’ marks.

Dr B Bozzoli (DA) said that there seemed to be a problem with the employment of qualified examination markers and particularly that the quality of the markers who are employed in TVET colleges is poor. She asked what type of sanctions were imposed for TVET colleges that failed to comply with the law or improve on their marks.

Ms M Nkadimeng (ANC) was appreciative of the improvement from the marks released in early 2015; and the fewer leakages of examination papers. She asked how Umalusi would handle its internal problems in order to provide training to promote effective planning, implement NATED guidelines and provide the required training for lecturers in setting and moderation.

Mr M Mbatha (EFF) said that the way in which Umalusi was doing its business seemed to be based on the perceptions of what the DHET was doing. He was disappointed that the TVET colleges have still not improved on their marks and overall performance. The examination markers must be trained and prepared on time, and benchmarks had to be set for the correct quality of markers. He asked if whether the system was secure enough to upgrade the standards.

Ms S Mchunu (ANC) congratulated Umalusi on the improvements made. She commented that the GETC percentages remained static so she wanted to know what mechanisms had been put in place to improve the marks. It was important that assessors understand the rubrics they were using, lest a lack of understanding compromise the quality. She pointed out that examination assessors often arrived late to examination sessions, and that this undermined the work of Umalusi.

Mr C Kekana (ANC) asked why Kenya and Botswana had been chosen as a comparison and as a quality control measure for assessment benchmarks. He suggested that a comparison with other developed countries, such as Algeria and Angola, would have been more useful comparisons, as they were doing much better than SA.

Dr Rakometsi replied that since the report had only recently been compiled for the Minister, he might not be able to answer all questions fully at this stage. Umalusi relies on both qualitative and quantitative reports. It had looked at the previous years’ reports to help it to determine the proper adjustments, as well as the external and internal moderators. Experts, not Umalusi employees, were hired to help determine the proper standards of subjects. He agreed that the quality of markers is important. Umalusi had the right not to approve assessors if the quality of the assessors was poor, and he felt that examinations leakages were often caused by poor assessors. Training of examination markers is an important aspect for improving the standards and results. Umalusi compares itself not only with Kenya and Botswana but also with the international world as it participated on both international and African bodies. Umalusi also uses the Denmark benchmark, It had not been able to compare and use a benchmark similar to that of Angola and Mozambique because of a language barrier.

Professor Volmink said that the IEB is a more established and stabilised system, hence fewer subjects were standardised and did not have too many adjustments.

Mr Feizel Toefy, Chief Director, DHET, stated that resources are a problem. He added that standardisation was one method that could be used to fix the system, and could cover lecturers, quality of work and infrastructure.

Another DHET representative added that provincial support mechanisms are needed to help deal with the teaching and learning of assessment rubrics. The problem with examinations was that there was one examiner and one moderator for each examination paper, so a process was needed to scrutinise the system. The appointment process for markers is currently being assessed by the DHET. DHET is currently looking into appointing markers as chief markers; but for this, all chief markers would have to be registered with the South African Council for Educators (SACE). She added that the time frame given to markers for marking examination papers is short and many markers have to be employed to reach the deadline. However, the DHET is committed to ensuring the appointment of markers is done early in the year, to ensure that there will be enough markers by the time of the exams.

Dr Bozzoli said she understands the comments on standardisation but cautioned that DHET should not depend on standardisation alone to fix a weak system.

Dr Rakometsi agreed with Dr Bozzoli that the system was not standardised at present.

Mr Kekana accepted the reasons given for comparing benchmarks with Kenya and Botswana but repeated that Umalusi must be eager to learn from countries that are doing better and are developed.

Professor Volmink replied that Umalusi tries to use the same rubric for all its subjects.

Ms Zodwa Modimakwane, Executive Manager: Quality Assurance and Monitoring, GETC, stated that the DHET does not have permanent GETC markers.

Professor Volmink stated that because Umalusi takes quality assurance seriously, the monitoring of marking, standardisation and moderation had been introduced to improve the system.

Dr Rakometsi assured Members that their inputs were taken seriously and Umalusi will reflect on them.

The Chairperson commended Umalusi for the improvements made during the examinations. She congratulated the DHET for its innovative strategies and making resources available. She noted that despite the apparent call to do away with the moderation of systems, Umalusi would surely be able to deal with the matter.

Backlog in NC(V) certificates: State Information Technology Agency (SITA) briefing

Dr Setumo Mohapi, Chief Executive Officer, SITA, said the project approach to eliminate the backlog in the processing of NC (V) certificates included expertise, business and governance processes and examination system software.

SITA had not been successful to date in meeting its primary project objective of eliminating the NC (V) certificate backlog. SITA had not been clear on what would be required to ensure that it could meet the project objectives, but had implemented some changes to ensure that these would now be met. To facilitate this, SITA was now asking the Committee to extend the project target date, to 30 June, to allow SITA to process all backlogs. He described the backlog figures as at 2 February 2016:
- In August 2015, 129   453 candidates were affected by the backlog and 236   818 certificates were outstanding. ,
- In January 2016 102   926 candidates were affected by the backlog and 181   973 certificates were outstanding.
- To date, only 20% of the affected candidates were serviced and only 23% of the certificates were produced.

There had been a positive movement in the certificate backlog numbers recently. 54   845 backlog certificates had been produced d 14   571 certificates would be unblocked from centres with payment in arrears this week, facilitated by SITA. Payment of arrears fees was facilitated by SITA on 29 January 2016. All 59   965 level 2 certificates would be processed by 10 March 2016 and all 64   969 level 3 certificates would be processed by 29 April 2016.

Since 12 August 2015, SITA had managed to capture and align Umalusi and the DHET in their review of the NC (V) certificate business process. There had been an increase in SITA technical resource capacity to keep up with DHET demands. SITA implemented capacity for exam centres and third party vendors of college management systems, so that they could submit data to DHET in real time rather than in batches. SITA also developed capacity to stage data and do data integrity checks before submitting data to the DHET. Capacity was developed for DHET, exam centres and Umalusi to use a common rule base for certificate processing.

The initial order of backlog processing had proved to be problematic and resulted in delays. There was an underestimation of the state and complexity of the Consideration Application Function and there was limited knowledge available for the NC (V) application. SITA had taken corrective measures to solve the problems. It had flattened the project governance structure, applied additional technical and business resources and was implementing QA testing and a project management portal for task management.

Discussion
Dr Bozzoli commented that the certificate backlog had been apparent between 2007 and 2009, and she felt that the CEO had not made any improvements, and many staff were allegedly fired. She asked if whether SITA had been working throughout December to try and fix the problems.

Mr Siwela said he was disappointed by SITA’s performance. The backlog should have been cleared by 31 January 2016 and he was not convinced that the matter would be improved by granting an extension. He asked the Board whether SITA was truly committed to reducing the backlog and why he challenges it was facing were not communicated earlier to the Committee.

Ms Nkadimeng agreed that there was no guarantee that the backlog would be resolved by June.

Ms Mchunu said she needed a further explanation on how SITA intended to reach its targets, and what mechanisms it would use to make improvements.

Mrs J Kilian (ANC) said she had supported the appointment of the new CEO and remembers referring to him as ‘Mr Fix-It’. She pointed out that the CEO alone was not to blame for each Board member has a role to play in ensuring the effective running of SITA, whilst the culture of the organisation is also a contributing factor to whether a body would deliver or fail with the mandate. She believed that it would be useful to hold a joint meeting with the Portfolio Committee on Telecommunications and Postal Services, since SITA reported also to this Committee.

Mr Kekana supported that proposal, for the purposes of building capacity. He asked that SITA submit monthly reports until June, and suggested that it must also send letters to students indicating that their certificates are still being processed, which they could show to prospective employers. He urged that SITA must reach their targets each year because it provided services to a huge number of students.

Dr Mohapi replied that he fully recognised that he had failed to deliver as he had promised. He was not able to answer the questions asked by the members, and was in in fact embarrassed for not being able to take control of the matter.

Mr Zeth Malele, Board Member, SITA, said the Board will work with the CEO to ensure that its 2016 targets are reached.

The Chairperson asked why the Board has not supported the CEO earlier.

Mr Malele replied that he was not aware of the problems mentioned by the CEO but would now communicate with him on a regular basis.

Ms Mchunu said the Committee had to visit SITA and investigate in order to reach agreement on what had to be done.

Ms Kilian commended the Board on its renewed commitment. She added that the public service was lucky to have an individual like Mr Mohapi who had dedicated his time to the Agency.

Mr Mbatha responded that SITA must not adopt a culture which is not sustainable; if staff members needed to be fired then it should happen.

Dr Bozzoli asked if SITA used Information Technology (IT) management systems; it seemed that its processes are managed by the Board and committees and not by IT specialists.

Ms Nkadimeng commented that the challenges in SITA must not be allowed to disturb the work which had to be done; now that the CEO was aware of the challenges he should be able to take action. She added that the chairperson of SITA must also not be afraid to ‘get his hands dirty’ if it is needed.

Mr Siwela said SITA should not continue to pay people who are not doing their work, and the Board must support the CEO.

Mr Zukile Nomvete, Acting Chairperson, SITA, thanked the Committee for the opportunity and the second opportunity to fix the matter and promised to submit monthly progress reports.

The Chairperson concluded that she too supports having a joint meeting with the Telecommunications Portfolio Committee.

Other business:
Adoption of Minutes: 27 January 2016
Mrs Kilian stated that page 2 should differentiate between the Committee Members and Minister and Deputy Minister.

She also suggested that the bullet points on page 3 could be replaced by writing full sentences in one paragraph.

Members adopted the minutes, with these amendments.

Adoption of Minutes: 28 January 2016
The minutes were adopted with no amendments.

The meeting was adjourned.