Umalusi on disclosure of the standardisation process

Meeting report

Opening Remarks
The Chairperson said that the Committee was interested to know why Umalusi had chosen to reverse it decision to disclose the standardisation decisions applied to the NSC results.

Release of the 2010 Standardisation Process
Mr Mafu Rakometsi, Chief Executive Officer: Umalusi said that after Umalusi’s reluctance to disclose NSC standardisation decisions of the Department of Basic Education (DBE), the Independent Examinations Board (IEB) and the Eksamenraad vir Christelike Onderwys (ERCO). The country had witnessed unprecedented interest in the role of Umalusi in the 2010 NSC results and uninformed and speculative comments on the credibility of Umalusi and the NSC had emerged. Umalusi had decided to release the information after extensive consultation with relevant stakeholders and interested parties. Release of the information to the public had been done through the media on the 23 February 2011. The decision had been taken to protect the integrity of Umalusi and the NSC.

While the information was confidential, there was nothing to hide. Information on the standardisation decisions by Umaluisi’s predecessors, the Joint Matriculation Board (JMB) (1918-1992) and the South African Certification Council (SAFCERT) (1992-2001) had never been released. Umalusi was a member of the International Association for Educational Assessment (IAEA) and the Association Educational Assessment in Africa (AEAA) and all members of these associations did not disclose standardisation decisions to the public.

Some of the debates and reports in the media on standardisation had been factually incorrect and it was clear that many people did not understand Umalusi’s quality assurance processes, of which standardisation was but one.

When approval was given for the release of results, Umalusi did not know what the overall pass rate would be. Umalusi standardised the written examination component of the individual subjects. The pass rate could only be determined once standardisation had been completed and after all the other assessment components were added: School Based Assessment (SBA), Oral and Practical Marks, language compensation and rules of combination and condonation. Gender, race and social standing of candidates were not taken into account when standardisation was conducted.

Umalusi discharged its mandate in terms of the General and Further Education and Training Quality Assurance Act 58 of 2001 (as amended in 2008) and standardised examination results of the IEB, ERCO, DBE and Department of Higher Education and Training (DHET) in exactly the same way.

Mr Vijayen Naidoo, Senior Manager: Quality Assurement of Assessment; Umalusi explained the processes and procedures in quality assuring the NSC examinations and general principles applied in the standardisation of examination marks.

Professor Sizwe Mabizela, Chairperson: Umalusi Council, said that on the 19 January 2011 Umalusi had indicated to this Committee that Umalusi was bound by international norms and standards to uphold the confidentiality of standardisation decisions. The unprecedented interest in Umalusi’s work and the kind of wreckless, irresponsible and ill-informed comments necessitated Umalusi to take an unprecedented decision. In an article in the Cape Times, teachers claimed that top students could not attain their history distinctions because Umalusi had adjusted the history marks downwards. They further said that there were problems with Business Studies’ marks. In fact, Umalusi had not adjusted those marks. 

It was universally accepted that judging whether a question paper was set at the appropriate level was a difficult exercise. It was only after the paper had been written and marked that a judgment could be made.

Standardisation was applied internationally to mitigate the effect of factors on the learner’s performance other than the learner’s knowledge and aptitude. When standardising, the rank order was retained; an adjustment could not be in excess of 10%; and the raw marks of the candidate could not be doubled or halved. When raw marks were adjusted, principles of scaling were applied - a fixed number of marks could not be applied to all the marks along the curve.

Standardisation was used to achieve compatibility and consistency across years and was applied to each subject individually in a linear and non-recursive manner. Umalusi did not calculate the overall pass rate on raw exam marks nor did it meet with government or any assessment body to discuss the raw marks. As discussed earlier, the overall pass mark incorporated factors which were not available to Umalusi at the stage of standardisation. 

Raw marks were accepted when ‘qualitative information’ and ‘pairs analysis’ indicated that the question paper was appropriately pitched and when performance was in line with the historical mean. In construction of the statistical Ogive Curve for standardisation of the NSC, Piecewise and Spline Polynomials were used. The Kolmogorov-Smirnov Test was used for standardising the National Certificate (vocational) (NCV), a new and modern qualification of the National Qualifications Framework.

Umalusi could not accurately report that with standardisation, raw marks were adjusted upwards or downwards. In fact, there would be upward or downward adjustment at a particular end for a particular purpose. Where it was said that a subject was adjusted upwards, in reality raw marks may have been accepted up to a certain point on the curve and a minor upward adjustment may have been applied at the top end to improve the distinction rate to keep the rate in line with performance in previous years. Learners who obtained a distinction on raw marks were not scaled downwards. Thus the statement ‘upward adjustment’ was not fair to those who did not benefit.

The standardisation decisions for each subject of the NSC for ERCO, IEB and DBE were depicted in detail in the handout: Release of the 2010 Standardisation Decisions.

ERCO NSC 2010
Raw marks were accepted for Afrikaans: first additional language, Afrikaans: home language, Agriculture Management Practice, Agricultural Science, Business Studies, Computer Application Technology, Dramatic Arts, Economics, Electrical Technology, Engineering Graphics and Design, History, Hospitality Studies, Information Technology, IsiZulu: first additional language, Life Orientation, Life Sciences, Mathematics Literacy, Mathematics P3, Mechanical Technology, Portuguese: first additional language, Religion Studies, Sesotho: first additional language, Tourism and Visual Arts.

Standardisation decisions were made on Accounting, Civil Technology, Consumer Studies, English: first additional language, English: home language, French: second additional language, Geography, Mathematics and Physical Science. (See handout for figures)

IEB NSC 2010
Raw marks were accepted for Afrikaans: home language, Computer Application Technology, Consumer Studies, Design, Dramatic Arts, Economics, English: first additional language, English: home language, Geography, History, Information Technology, IsiZulu: first additional language, Life Orientation, Life Sciences, Mathematics P3, Tourism, Dance Studies, Hospitality Studies, Music, Sepedi: first additional language, Sesotho: first additional language, Setswana: first additional language, Siswati: first additional language, Siswati: home language, isiXhosa: first additional language; IsiZulu: first additional language, German: home language, Gujurati, second additional language, Hindi: second additional language, Italian: second additional language, Modern Greek: second additional language, Portuguese: second additional language Spanish: second additional language, Tamil: second additional language, Telegu: second additional language, Urdu: home language and Equine Studies.

Standardisation decisions were made on Accounting, Afrikaans: first additional language, Business Studies, Dramatic Arts, Engineering Graphics and Design, Mathematics Literacy, Mathematics, Physical Science, Visual Arts, Arabic: second additional language, French: second additional language, German: second additional language, Hebrew: second additional language, Advanced Programme Mathematics and Sport and Exercise Science.

DBE NSC 2010
Raw marks were accepted for Afrikaans: first additional language, Afrikaans: home language, Afrikaans: second additional language, Agriculture Management Practice, Agricultural Technology, Business Studies, Computer Application Technology, Dance Studies, Design, Dramatic Arts, Economics, Engineering Graphics and Design, English: second additional language, History, Hospitality Studies, Information Technology, IsiZulu: first additional language, isiZulu: second additional language, isiXhosa: home language; isiXhosa: second additional language, Life Orientation, Mathematics Literacy,  Mathematics P3, Mechanical Technology, Music, Physical Science, Sepedi: home language, Sepedi: first additional language, Sesotho: first additional language, Sesotho: second additional language,  Setswana: first additional language, Setswana: second additional language, Siswati: first additional language, Tshivenda: first additional language, Tourism, Visual Arts, Xitsonga: first additional language, Nautical Science; and Maritime Economics.

Standardisation decisions were made on Accounting, Agricultural Science, Civil Technology, Consumer Studies, Electrical Technology, English: home language, English: first additional language, Geography, isiNdebele: home language, IsiZulu: home language, isiXhosa: first additional language, Life Sciences, Mathematics, Religion Studies, Sesotho: home language, Setswana: home language, Siswati: home language, Tshivenda: home language, Xitsonga: home language
Where a paper was set by DBE and written by an ERCO learner – when assessment bodies entered into agreements with each other - standardisation decisions made by DBE on that paper were carried over to ERCO. Similarly, IEB papers used by DBE (nautical sciences and equine studies) and by ERCO (Portuguese: first additional language and French: second additional language) were standardized by IEB and carried across to the respective assessment bodies.

A problem with home languages was where learners were disadvantaged by not being able to take their home-spoken language at home language level when their school did not offer the language at this level. They then had to take a non-home language as a home language. This contributed to the high failure rate of English: home language and Afrikaans: home language. Failure of a home language meant that the learner failed to obtain the NSC. The Committee Members were urged to apply their minds to this inequity. Another problem with the African languages was that the quality of the examination paper was variable and the learning material was generally inadequate. Umalusi would be paying particular attention to all the home languages to ensure that they were pitched at the appropriate level. Additional languages tended to have high pass rates. The reason for this was that there were smaller numbers of learners and the home-spoken language tended to be written as an additional language.

Mathematics Literacy and Mathematics differed in kind but not in levels. The migration of Mathematics to Mathematics Literacy had serious ramifications for Human Resource Development in South Africa. The pass rate was not a single measure of the health of the education system and with teachers wanting 100% pass rates at school, there was a drive for students who were Mathematics competent to change from Mathematics to Mathematical Literacy. Mathematics Paper 3 was an important optional paper and a growing number of learners who were interested in future study of subjects such as engineering and medicine were choosing to write this paper.  Sport and Exercise Science was a new 2010 subject and without a historical mean, was standardised by simply applying pairs analysis.

The decisions belonged to Umalusi alone, which was a fiercely independent body. It did not compromise education to feed short term benefits of any nature and was not influenced by political games in any way. Umalusi was proud with its decision to disclose the NSC standardisation process and urged the Committee to ask for clarification on any issues which were not clear in the presentation.

Discussion
Ms N Vukuza-Linda (COPE) asked if there were women in the Umalusi Council.

Mr Rakometsi replied there was 50/50 gender presence in the Umalusi Council. The Chairperson was male and the chief deputy was female and there was an even spread of male and female throughout the council. The CEO and CFO were male, the COO was female and the senior manager positions were held by two males and two females. 

Ms Vukuza-Linda also commented that there was doubt about education in South Africa and therefore Umalusi should not expect to be healthy in an unhealthy environment. For Umalusi to be fiercely independent, it must understand that nothing was beyond reproach and that it should not take comments personally and react with anxiety. Standardising to mitigate factors other than knowledge was where Umalusi could take their lesson. The issue was not the instruments of Umalusi. It was about how to respond to the bigger national question – the people’s outcry for assurance about education for children in South Africa.

Professor Mabizela thanked the Member for wise counsel. South Africa was anxious and Umalusi did not take attacks personally. However, some startling statements were made by people who should have been better informed. Further, lack of understanding of the rules of condonation and the obsession with pass rates contributed to fundamental problems such as the shift to Mathematics Literacy when the need for the country was for Mathematics graduates.

Mr N Kganyago (UDM) asked for clarification on what a bimodal curve and what scaling up and scaling down meant.

Professor Mabizela said that in statistics the curve of a normal distribution went upwards and then downwards in a normal curve. A bimodal curve meant that in one population that same shape repeated itself at the bottom end and again at the top end of the graph – ie two humps. Scaling at plus 6 at 18, the ratio of 6/18 is worked back to 0 so that the ratio at 10 and at 17 for example, would give the same ratio result until at 0 the scale would be 0 at 0. A computer programme calculated these quantities. Umalusi would extend an invitation to the Committee to attend the standardisation meeting so that the Committee could understand standardisation thoroughly.

Mr Smiles (DA) asked if the gaps in the processes and procedures leading up to standardisation had been closed. The Committee expected Umalusi to report back on the quality of marking centres and markers in its effort to ensure that raw marks were valid when they arrived for standardisation. 

Professor Mabizela said that gaps were being addressed. However, a request by a chief marker not to be assigned to mark the section on linear programming in Mathematics still haunted him. This teacher taught the section but did not understand it. However, the problem could not be solved by being critical of teachers. By acknowledging the damage done by the apartheid system and through psycho-social healing, teachers could start to accept that they did not receive quality education and ask for assistance where required.

Mr K Dikobo (Alt) (AZAPO) commented that he felt that it was not necessary for Umalusi to publicly offer the information on standardisation but understood that in national interest and doubt about the validity of the NSC results, it was a decision which Umalusi felt was necessary.

Mr D Smiles asked if the core decision to publicly announce standardisation decisions would be continued each year.

Ms N Gina (ANC) said that it was important that the public was informed on some of the information around standardisation. It appeared that some issues which made an impact on standardisation could be picked during the processes of moderating, monitoring and verification of marking. She asked if disclosure of the standardisation decisions had caused any negative impact at higher education level and if disclose would continue in the future.

Mr Rakometsi answered to the questions that Umalusi had conducted standardisation each year and so disclose itself did not affect the outcome of the results. He hoped that leaders of higher education levels would not disadvantage learners because of the disclosure. It was a challenge to write the standardisation decisions in an understandable way, but in a maturing democracy, it was important for Umalusi to do so. Going forward it would continue to disclose standardisation decisions. At the time that Umalusi approved the quality assurance of the results, it would announce what adjustments were made. This would be before the Minister announced the final pass rate.

Mr C Moni (ANC) asked for clarity on whether adjustments for distinctions meant that nobody had failed that subject.

Mr Rakometsi replied that the historical distinction rate was around 7% and for 2010 was adjusted accordingly. This did not mean that nobody had failed. There was still an even spread at other levels.

Ms A Mashishi (ANC) asked how Umalusi decided which centres to visit in its fulfillment of quality assurance and ‘state of readiness’. On oversight visits, the Committee had witnessed chaos at centres in Mpumalanga and KZN.

Mr Rakometsi said that monitoring was performed randomly and unannounced.

Ms Mashishi also asked what criteria informed Umalusi to choose the so-called ‘experts’ who made up the internal and external bodies.

Mr Rakometsi replied that Umalusi’s criteria for experts on assessment were shared in the notes presented earlier. They had sound educational background and could deal with quality assurance. There were also experts who were responsible for the qualitative reports on subjects before they arrived for assessment. These experts had particular knowledge and understanding of the National Curriculum statement for that subject.

Mr K Dikobo said that it appeared that ERCO used quite a number of DBE examination papers. He asked how many question papers were set by DBE and if a set number of papers had to be set by DBE.

This question was not answered.

Mr Dikobo also asked what criteria there were for a fourth examination body to register and use examination papers set by the other three examination bodies.

Mr Rakometsi replied that the establishment and accreditation of an examination body followed very stringent criteria. There had to be a need for one; an adequate number of candidates; and the capacity to conduct an examination. Currently, there was no need for an additional examination body.

Ms Gina asked how Umalusi protected the security of examination papers. Learners may anticipate that the same paper would be used by DBE, IEB and ERCO.

Mr Rakometsi replied that any actions taken between the three bodies had to receive endorsement from Umalusi. Where papers were shared, there was a common date and time for writing of those papers.

Mr Dikobo said how in future, in Umalusi’s effort not to disadvantage any learners, it ensured that an unapproved question paper was not submitted for the NSC examination.

Mr Rakometsi replied that the late submissions of question papers from the different assessment bodies resulted in a mix-up between the examiner, internal moderator, ERCO and the external moderator. The paper that was rejected by Umalusi was returned and there was a mix-up in terms of which paper had been approved. This showed weakness in the system which needed to be strengthened. The late submission of papers from the assessment bodies created problems of quality assurance. Umalusi had subsequently agreed that a disclaimer would be issued against the assessment body which failed the deadline for submissions.

The Chairperson thanked Umalusi for their presentation and agreed that it was important to acknowledge what happened to teachers in the past and also for Umalusi to educate the communities on its work. The Committee would attend Umalusi’s standardisation meeting for clarification on any outstanding issues. In conclusion, she said that it was unclear how the Committee could assist with misinformation on Umalusi when the Constitution spoke to freedom of speech. She hoped that through Umalusi’s disclosure of standardisation, the critics were silenced and that Umalusi would continue to do what was best for the country.

Meeting adjourned.

Apologies were accepted from Mr ZS Makhubele (ANC), Dr W James (DA), Mr H Hoosen (ID) and Ms C Dudley (Alt/ACDP).