Statistics South Africa noted that the results of the Community Survey had been made available to the President in October 2007, although this was the first opportunity that this Committee had to address the matter. An article had been printed in The Star alleging that the institution had said that “error-ridden data” could not be used to set future policy. Members of the Committee had then given public comment. The Chairperson appealed to the members of the Portfolio Committee to deal with such matters within the Committee before making public statements, and noted that their statements could not be seen as a Committee response. Statistics SA noted that the survey was conducted in a consistent and scientific manner, had been accepted by the Statistics Council, that the author of the article had misquoted her sources, had perhaps misunderstood comments about the scientific basis of the study, and did not have the expertise to properly challenge the results.
A briefing was given on the results of the Community Survey 2007, highlighting that the overall comments from the sample interviewed had indicated that 2007 showed an improvement in services to South Africans on previous years. The process of the interviewing and selection was outlined, and the composition and recommendations of the Council were described. A pilot project had been run to identify problem areas and test the systems before the actual survey began. Although there were some concerns and difficulties, these were logistic-related, not science related. Raw data was compared to other databases, and there were some divergences. Challenges had included lack of a housing register, the reliance upon the population register, the high level of skills required, the high level of non-responses leading to an undercount. The Council had made a number of recommendations to the Minister and Statistics SA. The findings were highlighted and it was noted that the final report would be available in March. The comment was made that perhaps there needed to be a formal dedicated body responsible for reporting on impacts and that intellectual ability needed to be highlighted
Members again noted that the views expressed by parliamentarians must be understood as those of their parties, and explanations were given around the responses provided to the media. They felt it important to focus on the comment that the survey did not have a comprehensive scientific framework, asked how challenges in regard to capacity had impacted, what programmes were in place, and what the weaknesses were. The comparative figures for education were queried, as also the death rates, the population distribution and proportional figures. The difficulties around establishing addresses were highlighted, and the undercount was related to the methods used in the survey. Members felt it vital to build infrastructure to improve on the results. Members asked to what extent the Department of Home Affairs records were used, why the deadline had been moved to March, whether the results for education took into account home-schooling, the impact of the influx of foreigners, whether training institutions would be established and the procedure around the bursaries offered. Members concluded that there was no reason to believe that the statistics were inaccurate, and that the information obtained had served the purpose for which it was intended.
The Chairperson said that although Statistics South Africa (SSA) should have briefed the Committee on the 2007 Community Survey last year, that meeting could not take place. The survey results were tabled and given to the President during late October, as per the strategic plan of SSA. The President used these results as they were intended. The media reports about this survey were unfortunate, since the Portfolio Committee was supposed to be the first to deal with the survey.
He appealed to the members of the Portfolio Committee to deal with such matters within the Committee before making public statements. This could be done in a way that did not preclude them from still being accountable to their respective parties. Members’ behaviour should not undermine Parliamentary processes. In this way the Committee could interact with relevant bodies on issues instead of having to respond to media reports.
Briefing by Statistics South Africa (SSA)
Mr Pali Lehohla, Statistician-General, SSA, said that the reason for the presence of all the heads of provincial arms of SSA was that the community survey had captured the minds of South Africans. As a result their office had been inundated with calls from provinces to discuss the results of the survey and the content of irresponsible media reports.
On 8 January 2008 (two and a half months after the results of the survey became available) he had received a call from the Minister of Finance saying that there was an article about SSA on the front page of ‘The Star’ newspaper under the headline “R600m state survey bungle. Stats body says error-ridden data cannot be used to set future policy”. He said that it was worrying that the country and politicians had actually seriously believed that the Minister would have given SSA R600 million to deal with irresponsibly. He was therefore pleased that the Committee had called for an explanation. He described the statements in that article as irresponsible and ignorant. He said that the members of the Committee who had made statements to the media had clearly not read the report by the Statistics Council of South Africa, but seemed to have agreed with the comments by the media about the report.
The community survey was a scientific survey, with its results and methodology being consistent with those of the previous surveys. If the scientificity of the results were to be challenged, the author of the article was most certainly not the person to do it. The Statistics Council of South Africa (the Council) had informed the Minister and the SSA that it accepted the results and methodology of the 2007 survey.
Comments by Statistics Council of South Africa (the Council)
Mr Howard Gabriels, Chairperson, Statistics Council of South Africa, said that comments that the data was ‘error-ridden’ and could not be used for future policy making could not be attributed to him at all. The first time he had even become aware of these comments was when he himself had read the article in the newspaper. The journalist could have misunderstood statements in the report to have meant that there was no scientific basis for the survey, but Mr Gabriels had never said this nor had intended to make any implications to this effect.
The Council was an independent body consisting of twenty four members, which included academics, economists, statisticians and nine people appointed by the Premiers of the different provinces. A sub-committee, the Population Statistics Committee, had been involved in the planning and obtained detailed reports from the SSA since the beginning of the life of the Council. There was a document outlining the methods and estimations which would be used in the survey, with the Council recommending that no fewer than 274 000 people be interviewed in order to ensure that the data was reasonable. The process included the interviewing of households. There were detailed discussions on how to choose the sample to be interviewed and the Council also recommended how to deal with the situation where there were less or more than thirty enumerators. Once the sample was drawn, SSA would list the individual dwellings to be interviewed, which was obviously a huge logistical exercise. They had to list the units, which were chosen across districts, municipalities and provinces. The staff literally had to mark the individual units and then return to draw maps, which were given to the interviewers.
In February 2006 a pilot was run, with 18 000 interviews being conducted across the country. Results were run and problem areas identified. There was major testing of their systems in the pilot. The Council monitored the entire process, receiving both quarterly and monthly reports. Fieldwork was conducted using 6 000 interviewers. SSA and the Council were communicating extensively throughout this process.
Concerns that had emerged from the process were not science-related; for example, in Soweto questionnaires disappeared during a hijacking. After the interviews were conducted supervisors checked forms, which were then taken to a regional point and processed.
On 20 July the Council received the first raw data. These results were compared to other databases, for example of the SSA and the Population Register. An example of the types of problems identified was the fact that figures for people receiving social grants reflected differently in the two databases. A possible explanation was that when the person was asked if she or he received a social grant, there might be confusion with the the pension received, and therefore an answer might be given in the affirmative. This would therefore yield different results to what was already in the databases. Thus systematic error allowed for the re-weighting or results to compensate for these observations. It was in this context that reference to systematic errors was made in the report. He had never stated that the survey lacked a scientific basis.
The Council had made a number of recommendations to the Minister and SSA, for example on building statistical infrastructure in SA. The difficulty with the community survey was that there was no national housing register where every household was recorded and regularly updated. Also there was a heavy reliance on the population register. Another major problem identified was the fact that there was a high level of skills required for these purposes. Additionally, there was the problem of non-responses, which could be attributed to factors such as the interviewer having arrived at an inconvenient time, people might have been on holiday, or only a single person lived in the dwelling. SSA then provided a detailed report on the non-responses.
Mr Gabriels noted that it was also important to reduce the undercount in the census.
Mr Gabriels reiterated that he had never said that the data in the 2007 community survey was error-ridden and could not be used for future policy making. The Council had spent the better part of three years working closely with SSA on these issues.
The Chair said that it was important to note that the views expressed by the Members of Parliament in the article were those of their respective parties (South African Communist Party and Democratic Alliance) and not of the Portfolio Committee. The Portfolio Committee would only be able to comment after they had had this interaction.
Mr S Marais (DA) said that the statements had been made only on the basis of the Council’s report, which was the only information upon which they could base their comments. They were not responding to the survey when approached for a response. He added that he had to respond when asked to do so and could not be attacked for responding. The issues raised in the report were the same issues that had been raised during discussions on the SSA Annual Report. Based on information on hand at that time, SSA did not come across as being very reliable. It had however not been their intention to attack SSA.
The Chairperson said that members of the Portfolio Committee should, when making statements to the media, be careful not to make it look as if they are talking on behalf of the Committee.
Mr N Singh (IFP) said that a journalist could be forgiven for such an article based on the report by the Council. It was important to be able to have confidence in one’s data if the information was to be used for future policy-making.
Mr K Moloto (ANC) said that calls for the Statistician-General to be fired were taking it too far. The Committee should not leave it there in order to ensure that members refrain from such practices.
Presentation by SSA on the community survey
Mr Lehohla summarised the outcome of the survey by saying that results showed that on issues of basic services South Africans found that 2007 was better than 2001, which was in turn better than 1996. It was important to emphasise that this was not information provided by the President, but reflected views of the population of SA.
Mr Lehohla proceeded to read through the presentation document, setting out the purpose and areas of focus of the report. These were defined as meeting basic needs and developing their human resources. Under the methodology, he explained that the results were scrutinised by the Council, which then advised the Minister and the Statistician-General (SG) to accept the results of the survey. The Minister and SG accepted the advice of the Council
The findings of the report were contained in the presentation (see attached document) and concentrated on the population of South Africa, housing and services, education, disability and the social grant. The first release was on 24 October 2007, while the second release was scheduled for 19 March 2008. The latter date was however subject to change.
Mr Lehohla said the mass media, in the form of Ms K Maughan, had spoken to a member of Statistics SA, Professor Mostert, whom she had quoted in the article. Professor Mostert denied that he had in fact made the statements that she had attributed to him. Mr Lehohla said that the media was saying that the Minister gave SSA R600 million to make the top political authority look good. He said that the results of the survey were not designed to make anyone look good. Instead, what made SSA look good was the use of scientific methods. The problem with free press was that it could give rise to irresponsible journalism.
Mr Lehohla said that while the SSA was responsible for reporting on outputs and outcomes there was not a formal dedicated body responsible for reporting on impacts. Instead this was being done in passing by specialist groups, which were separate from Government. This function needed to be institutionalised, failing which it would be done by the likes of Ms Maughan.
Ms Maughan had delivered the SSA with a wake-up call by reminding them that they were placing their science in the political environment. This served to highlight the importance of strengthening the intellectual ability within the organisation.
Mr M Johnson (ANC) referred to the Members’ statements to the media as being mischievous and attributed these practices to the fact that election time was approaching. These practices were counter-productive. Mr Johnson referred to the comments that the survey lacked a comprehensive scientific framework. The Committee should focus on how to address this and not try to punish the SSA for it, as the focus should be on how to plan for 2011.
Mr Johnson asked if the report done at municipal level was still on track for release in March.
Mr Lehohla said that it was important for the data of SSA to be relevant to its users. In terms of the State time table SSA had already ‘missed the boat’, since the municipal report would not yet have been released before the State of the Nation address, as well as the budget address. Also, political parties would need the information in order to do their work. The data should be available to inform key moments for the State and political parties, so that when emotions ran high, arguments remained anchored in statistics.
Mr B Mnguni (ANC) asked to what extent the challenges with regard to capacity had impacted on the exercise.
Mr Lehohla answered that lack of capacity had not impacted on the results of the survey but on the persons who had worked day and night to make it happen. In addition, SSA’s risk increased, for example if something happened to a worker, there was not capacity to replace him or her. This also placed the country at risk.
An ANC member referred to the shortage of staff and suggested that the long hours people were required to work would further contribute to people seeking employment elsewhere.
An official from SSA responded that said that human resource capacity was addressed by introducing short, medium and long-term programmes. During the survey SSA employed 6 000 people. However the process was slowed down by the fact that the systems were unable to cope with the rate at which things were happening, especially since this process coincided with financial year end. However, in the end, SSA was increasing their numbers as the group employed the previous year had now been employed on a permanent basis. In addition SSA had a leadership programme in place for 2008. It was providing bursaries to ensure the proper development of skills within state institutions. Managers had to develop people, which would reduce the problem of people leaving the organisation. It was also busy training field workers. SSA would provide the Committee with more information as they progressed. At the end of March it would have completed the skills audit, which would give SSA a better idea of where it was.
Mr Gabriels added that these problems had to be highlighted in the interests of transparency. A person analysing the data should be able to understand the methods used. The Council had therefore issued suggestions with regard to the uses of the data. The weaknesses in the survey did not compromise the results, but did compromise the future of the organisation. He referred to the statistical infrastructure required and said that in presenting their annual report the Council had raised the absence of an adequate Business Register, Population Register and National Housing Register as being cause for concern. These registers were fundamental to the quality of statistics produced in South Africa.
In addition the SSA was introducing systems to ensure the adequate management of fieldworkers. If the skills problem was not addressed, the problems would continue to arise. In terms of the quality of data, this could only be improved if there was adequate investment in the registers and in skills development.
Mr Mnguni asked how the hijacking in Soweto had impacted on the results.
Dr Kefiloe Masiteng, DDG: Population and Social Statistics, SSA, said that the interviewers had gone back to the original households to re-interview them. This was possible since SSA had known exactly how many questionnaires had been lost.
Mr Moloto referred to the slides on education and asked what possible explanation there was for the drop in learners completing primary school.
Mr Lehohla responded that this had been a reference to proportions and not to absolute numbers. It was just that the proportion of learners completing secondary school had increased much more in relation to those completing primary school. Since this was not a reference to absolute numbers, this did not mean that a great number of learners was not completing primary school.
Dr Masiteng said that the number of learners completing primary school had increased from 3.5 million in 1996 to 4.1 million in 2001. The number of learners completing primary school in 2007 increased to 4.7 million.
Mr Moloto asked what could have attributed to the increase in death rates across certain age groups.
Mr Lehohla replied that the concern was that the death rate had increased for the younger members of the population. This was probably one of the effects of HIV/AIDS.
Mr J Bici (UDM) asked what method SSA had of authenticating households in municipalities, since this was very often difficult to determine.
Mr Johnson referred to the fact that very often people occupied dwellings that did not have numbering so it was difficult to ascertain the addresses.
Mr Lehohla said that the increase of the number of people interviewed in the survey was small, but this would improve once it was possible to label geographical positions more specifically. SSA was working on a tender to get the numbering on houses to correspond with their references. It was relying on municipalities for assistance in this regard.
Mr Bici asked if the figure reflecting attendance at educational institutions was too low. He asked what the SG would suggest to improve this figure.
Mr Lehohla responded that the SSA could not comment on what should be done, as this would mean dabbling in policy, which was not in its mandate. But, speaking in his personal capacity, he felt that something must be done to keep the 20-24 age group in school. He said that 96% of the 7-14 age group was in school, which was probably the case universally.
Mr Singh asked if the Council was satisfied that the results of the survey could be relied on in future owing to their methods being sound. He wanted information on the methodology used in the survey.
Mr Lehohla explained how the way in which South Africa was structured affected the results of the survey. South Africa had an undercount of 17%, while the United States had an undercount of 21%. In South Africa it was necessary for people to physically visit the individual households while the interviewees were at home (which was between 4pm and 10am the next morning). Very often the interviewer did not have the correct address. This was a problematic situation that arose from this country’s historical context. In the United States, on the other hand, the survey was conducted by sending a piece of paper to the household, where it would be completed and sent back to a central office. Completion was therefore not dependent on time of day, and it did not require an employee of the census office to be there. Mr Lehohla felt that this problem, which had arisen in 2001 and 1996 as well, would not be remedied in future surveys, as it was not possible to ensure that household members were at home while surveys were being conducted (unless the country declared a two-day holiday for this purpose - and even then one had to ensure that people would remain at home).
Dr Masiteng referred to the question on quality by saying that SSA managed to monitor the quality of the 2007 survey by drawing from lessons learned in 2001 and 1996. It was thus able to draft better questions, which were checked for completion. In the event of non-completion, these were returned to be completed. In addition many of the people brought in to assist with the survey were the same people who had worked on the previous surveys and therefore understood data, as well as its processing and analyses.
Mr Gabriels did not agree that one should simply accept that the undercount would remain high in future surveys due to existing circumstances. Instead the focus should be on putting in place the building blocks to improve on those results. These would include additional resources to ensure a better census. It was critical that SSA should build its infrastructure in order to enable it to make
decisions on a sound information base. These additional requirements would involve addressing issues of cost, geography and skills. The Council was looking forward to their next interaction with the Committee in this regard, as they refused to accept that it was impossible to improve on the previous surveys.
A Member said that the Method document was comprehensive, but after-the-fact. He asked if there was a preliminary document setting out the possible limitations.
An SSA official responded that pilots were conducted which focused on how the survey was done, the content of the questionnaire and a back-up plan, where problems were identified. The pilot highlighted obvious problems but there were others that arose in the survey that had not arisen in the pilot.
The Member wanted clarity on the issue of population distribution, since it did not seem clear whether there were more males or females.
Dr Masiteng responded that on aggregate there were more females, but there were more males in some age groups.
Mr Lehohla added that more males were born, but more ended up dying. These were proportional figures, which had to be compared within their own categories.
Mr Marais reiterated his earlier statement that comments to the media had been based on the report by the Council. It was important to address the issues raised by the Council. He assured the Committee that his statements had nothing to do with the upcoming elections.
The Chairperson agreed that it was important to deal with the issues raised in the Council report and said that perhaps the Committee should meet with the Council to discuss these issues.
Mr Singh referred to the slide on emotional disabilities and asked what this was. He also asked what type of questions were asked in order to determine if a person had emotional disabilities
Dr Masiteng explained that a determination whether a person suffered from emotional disabilities could be made by asking certain probing questions. The extent of these disabilities would be determined by looking at the extent to which it stopped a person from engaging in certain activities.
Mr Singh asked whether Department of Home Affairs identification records had been used to identify people within the identified households. He asked if other databases and surveys had been used for the purpose of conducting the survey.
Mr Lehohla explained that the survey had been conducted in the absence of a proper Population Register. He outlined a few estimation processes, for example, mid-year estimates, balancing equation and demographic modelling processes. All other data sets would be reviewed in the light of the new information obtained. He explained that the terrain of statistics was very fragmented, consisting of the various entities involved in the collection of statistics. Although the environment had since 1994 been very competitive, and many had tried to undermine the work of SSA, the organisation continued to thrive, doing work both in South Africa and abroad.
Mr Gabriels referred to the use of marketing surveys as a source of information. The problem with the Media and Products surveys was that they did not reach a huge portion of the population. This also applied to the labour force surveys. At the end of the day all institutions ended up relying on the information provided by SSA. In order to obtain reliable results, it was necessary to obtain information from a big enough sample. This was therefore a very costly exercise, making it very difficult for the private sector to survive in this arena.
Mr Singh asked why the second release date was only in March.
Mr Johnson warned against SSA becoming complacent. He asked what the constraints were which had necessitated the shift of the deadline to the end of March.
Mr Lehohla conceded that the movement of the deadline to March decreased the relevance of the survey, as one if its aims included providing data that was relevant to key moments of State affairs. SSA had collaborated with outside organisations and it had become very difficult to depend on them during the December period. The main constraint was lack of time, resulting from inability to rely on other parties involved.
The Chairperson instructed SSA to provide the Committee with a written explanation for its failure to meet the original deadline.
Ms Mokoto asked if results took into account the growth of home-schooling in the country.
Dr Masiteng said that SSA had not managed to obtain information on home-schooling, since their focus had been on education within institutions.
Ms Mokoto asked how the high influx of foreigners impacted on the survey.
Dr Masiteng said that the exact number of foreigners was impossible to determine, due to the amount of illegal immigrants in the country. It was possible to determine which of the people interviewed were from other countries as there was a question asking where the interviewee was born. However it should be remembered that illegal immigrants should not be included in the survey because of the fact that they were illegal.
Mr Lehohla said that the influx of foreigners had to be dealt with by Department of Home Affairs and the introduction of effective tracking systems. SSA merely had an audit function, which reflected information that it managed to obtain.
Mr Bici asked if the SSA felt that Home Affairs was effective. It appeared as if the Stastician General was implying that their systems had to be ‘jacked up’
Mr Lehohla responded that SSA depended on Department of Home Affairs for much of their information, for example the Population Register and the Population Movement Register. The Population Register had improved, but still needed more work. Much more had to be done to the Population Movement Register. The issue of migration was being dealt with by management of the borders. However this problem had to be dealt with at municipal level, with Department of Home Affairs merely coordinating the process.
Dr Jairo Arrow DDG: Quality and Integration, SSA, said that people were often tempted to use statistics for purposes other than those for which they were intended. The survey had met its stated objectives.
Dr Arrow referred to the questions regarding the methodology employed by the SSA. He said that the role of SSA represented a paradox, since it was an independent body subject to internal audit. It was therefore in conflict with itself. SSA had run surveys internally without the knowledge of the persons involved in the official survey, in order to test the correctness of information obtained. This practice was important as it allowed for a critical reflection of oneself.
Mr Johnson asked if SSA did not have capacity to establish its own training institution
Mr Lehohla answered that SSA was in the process of planning for this. The details would be further discussed in their Strategic Plan.
Mr Mnguni asked what impact the non-responses had on the overall results.
Mr Lehohla said that here was no indication as to the characteristics of the non-responses. It would only be problematic if there were a large amount of non-responses. Surveys were sometimes run to study the non-responses.
Mr Bici asked what guarantee SSA had that the recipients of its bursaries would return to the organisation.
Mr Lehohla replied that it became difficult to keep people if they moved into other areas of the public service after completion of their studies, since it was more difficult to control where they went from there.
Mr Gabriels said that there was no chance of the SSA becoming complacent. He noted that the mood in the discussion in the Committee was positive compared to three years ago when the first briefing had been made. There had been huge achievements, especially with regard to addressing issues raised in the audit. SSA had done well and this built confidence and credibility for the institution which was important for the country. The Council would keep the pressure on the SSA and would not accept today’s standard as the standard for the future. The Council welcomed the opportunity to engage with the Committee further. While he assured the Committee that there was no chance of the SSA becoming complacent, he added that one had to also recognise the improvements witnessed during this survey.
The Chairperson concluded that there was no reason to believe that the statistics were inaccurate. It was important to note that the Members of Parliament had made the statements to the media either in their personal capacities or on behalf of their respective parties. He further concluded that the information obtained from the survey had served the purpose for which it had been obtained. He thanked the presenters and said that the Committee would liaise further with the offices of the Council to ensure the further improvement of standards.
The meeting was adjourned.