Examination of Witnesses (Questions 1-15)
MR JOHN MARTIN AND DR BARRY MCGAW
WEDNESDAY 20 MARCH 2002
1. Can I start by saying we were fascinated by the PISA results and conclusions. As you say, they caused quite some interest and stir in the media in the UK, and of course, the Government immediately announced them as a triumph for the education policies they had been pursuing. We have two concerns about the methodology. There is an instinct in this Committee not to believe so much of the good news that seems to have come out of that. Looking at the rankings and so on, many of us who have been in the education sector for a very long time look at the relative ranking of, say, to take two countries, ourselves and Germany, and are concerned whether that reflects the reality on the ground. Some of our experience would seem to suggest perhaps otherwise. There has been some criticismand I am looking here particularly at a National Institute for Economic and Social Research paperthat would suggest that perhaps the way the survey was carried out might lead to conclusions about the UK performance being rather better than one might think.
(Dr McGaw) If I describe the way the study was done first, the member countries collectively decided what the focus would be, and whereas some of the earlier international comparisons began with an analysis of the curriculum in the relevant domain in each of the countries, and then focused on those bits that were common to all countries, in this study the countries wanted to take a different orientation. They wanted to say of 15 year olds the crucial question is not, for example, "What science have you been taught and how much have you learned?" but "What can you do with the science you have been taught?" The orientation in the three domains of reading literacy, mathematical literacy and scientific literacy was towards people's capacity to use what they had done. On slide 5 is a summary of the way in which the development was undertaken. Once the countries had agreed that they wanted to focus on reading literacy, mathematical literacy and scientific literacy, international expert groups were set up in each of those domains to define precisely what framework would be used for the test development. All countries were then invited to submit materials that would assess their 15 year olds' skills in those domains. We had an international research consortium that undertook the study itself, and they did much of the test development as well. That international consortium was based in Australia, French-speaking Belgium, Japan, the Netherlands and the US. All countries then worked through all the developed test items to check for cultural bias, and some items were removed at that time. At this stage we had many more items than were going to be used. Then there was a substantial trial of all items that remained in the list in all countries, and then detailed statistical analysis to identify whether any of those items were actually working differently in different countries, and any items working inadequately in that way were eliminated. So now we have a set of items for which we have good empirical evidence as well as good prior judgment that they are culturally fair and they are empirically comparable across the countries in the way in which they work. One of the problems that previous studies have had is in equivalence of versions in different languages. Every prior study has actually produced its original test in English and any country that needed to use the test in a language other than English had to translate into their language. There were various rules. Some studies did a forward translation and a back translation to make sure the translation matched; others did two independent translations, which were compared. All those strategies run a risk in working from a sole language source. There are two features of any task: one is the very words in which it is expressed and the other is the task itself. What you need to keep constant in the translation is the task. It is not linguistic equivalence that is crucial. You can sometimes get an apparent linguistic equivalence but alter the task, make it more difficult or easier in the other language. What we did in this study was to produce an original French and English version for every item so that we have already revealed what we think is the crucial element of the task in the two different language expressions, and then countries that were not using either the English or the French produced independent translations from the English and the French into their own language and then compared those. No international study has ever done more to ensure the comparability of instruments across nations.
3. Could I ask about the response rate. I take your point entirely about the cross-referencing with GCSE results, but what about the response rate within individual schools? You said that the UK did not meet the required target of 85 per cent in terms of the school response rate, but what about pupils within the schools? Was Britain noticeably different from other countries in terms of that response rate, and has there been any cross-checking against other indicators as to whether a differential response rate within a school is biased against a particular ability level of pupils?
(Dr McGaw) There was not a problem with the response rate within schools. That was set high as well. Wales did not participate, which reduced your overall participation rate, but the target population could be no less than 95 per cent of the whole cohort, the whole age group, and once Wales was out, that did not give the rest of you much room to move because you were close to your 5 per cent limit. There was no other problem. The British results really need a nuanced interpretation. What you have is a high average, but you also have a relatively large spread of results. Countries like Korea and Finland have even higher averages but much less spread. You do well because you have some students right at the top, good proportions of students very high, but you have reasonable proportions of students who are doing badly, much more than, say, Korea, Finland or Japan. Also, you look at the relationship with social class. If you turn to slide 20, what we have is a series of lines across that graph showing how strongly achievement, which is plotted up the left-hand side, relates to social class or social background, which is plotted across the bottom. The lowest of the lines and the steepest slope is Germany. You would not want to be a poor male student in Germany. The next line up, the one that curves downwards slightly to the right, is Italy. The next one up, which has a relatively steep slope, is the United States. The one above that, parallel with the United States, third from the top, is the UK. That says social background makes about as much difference in the US as in the UK, or the other way round, but if you go up to the top two lines, the slope is not as steep. The education systems in Finland and in Korea ameliorate the impact of social background differences.
4. Moving away from samples and methodology, in your report did you draw any conclusions as to the reasons for the particular impact of socio-economic factors in the United Kingdom, the United States and Germany? My recollection from skimming through the full reportI have not seen the executive summary, and it would be very helpful if we could have a copy of thatwas that the conclusions were fairly equivocal and the report did not come out forcefully in favour of any particular prescriptions as to ways of remedying these problems.
(Mr Martin) You must appreciate that this is an extraordinarily rich database. It contains the results for almost 270,000 students in 32 countries; there is information about the schools in which the students are located, and there is information about the family background and characteristics of each of the students. We are only just beginning to dig into this database. We have commissioned a series of research papers which are going to try to deal directly with the issues that you have just posed. The original report does not try to go into detailed explanations of why one country's system appears to be more successful, shall we say, in terms of achieving high average performance and a narrow dispersion of performance compared to others. That is the subject of a series of research reports that we have commissioned and which we hope to be publishing within the next year to 18 months, when we can try to dig into these striking differences and try to understand to what extent some characteristics of the schooling systems, of the nature of the schools, of the attitude and motivation of the students themselves, might conceivably help to explain these differences. It is really too early yet. It is not just research that the OECD is going to do; we are making this database available to everyone. There is no doubt that researchers in many countries will dig into this information and produce very important insights in the next few years to precisely address those key questions about why it happens, how it happens, and how you can change them to get better performance in your schooling systems.
5. I was alerted by your comment about the rich database, which is exactly the phrase used by our Government to describe what we now have as a result of OFSTED. We have this huge database. You were going on, very importantly, to expand on how it is going to be used. There was also a hint of the value judgments which you are interposing and overlaying. Who is going to make those value judgments? How are those going to arise in this ongoing debate on the use of this data?
(Mr Martin) The OECD will publish a series of research papers that it has commissioned, and other work, and these will all be presented to committees of delegates from all OECD member countries. They will be debated and discussed by experts, by education policy makers, before they are published. We are not in the business, as an organisation, of making strong value judgments about which system is preferable to what. What we are trying to identify is what the results are, what the factors are that can conceivably explain those results, to what extent they are factors which are amenable to operations and interventions by education policy makers or policy makers in other areas, and to what extent they are factors that appear to be outside the immediate control of education policy makers. Of course, the results that we produce will be disputable, other academics and other experts will dispute them. I am aware of the critique which Sig Prais has made of our UK results in PISA. He himself, as you know, for many years has been making detailed comparisons of the UK and Germany, and of course, he feels that the picture we have does not accord with the picture that he has been presenting for many years. But I think there are some rather good explanations of why the two pictures come out somewhat differently.
6. By the very nature of the membership of OECD, are we looking at relatively wealthy countries? Is there going to be in the longer term, do you think, any value? I am using "value judgment" in a very positive way, not in the need for total objectivity, which I understand your research mind inevitably calls upon. I am particularly interested in the primary education for all agenda. Will there be lessons for the international debate which is inevitably now going forward, not only in terms, as you rightly say, of education, but of the economics of the future of our world trade?
(Dr McGaw) Four countries that are not in the OECD participated in the first round. Brazil was one, and the Russian Federation was another, and Latvia and Liechtenstein were the other two. There is now a whole series of countries lined up to use the PISA instruments this year, and they include Albania, Argentina, Bulgaria, Chile, China, Hong Kong and so on. There is a problem as this extends. The focus is on 15 year olds in school, because that is where these students are tested. It would be extraordinarily expensive to have a home-based survey of 15 year olds, for example. Even in countries like Mexico, which is a member of the OECD, or Brazil, which participated, a lot of 15 year olds are not in school. For countries where that is true, the PISA model may not be a good one, and the tests anyway may be set at too demanding a level. There is quite a large number of students in Brazil and in Mexico who did not even achieve at level one in the reading. It is one thing to say to them, "You have a percentage of your students below level one," and that carries a message, but it does not tell them what they can do. These are not necessarily illiterates. They can read in the sense of decoding text on the page, but they cannot read at the kind of level that is needed to support further learning. We are cautious about putting this too far to other countries. I think for your interest what is needed is an assessment at the primary level.
(Mr Martin) I would also add that it is true that this is a sample confined to "the rich OECD countries" but remember that that includes a number of recent members from central and eastern Europe whose average income per head is significantly below that of the G7 countries. It is very interesting for example, if you look at the PISA results, and look at the performance of some of the central and eastern European countries such as the Czech Republic and Hungary, which do not perform too badly in some areas, certainly not if you adjust these results by what they spend on education or the size of income per capita in these countries. It is interesting for example that some of these countries perform rather well in science literacy tests.
7. Can we push you on that? I do not want this to be exclusively about the PISA study, but a great deal of evidenceand please correct me if you think this is wrong, but this is the general feelingmakes us very concerned about science education in the UK. Sir Gareth Roberts, the former Vice Chancellor of Sheffield University, is carrying out an inquiry for the Treasury into science education in the UK. The concern is that students are less interested in science, less students are opting for science, less students are carrying on science into higher education, and very worryingly, less students are staying on to take PhDs, to become the academics who will teach the next generation. So there is a real concern, but we hear there is evidence that this is across the piece of developed countries, in the United States and many other western European nations, but that it is not the case in others. Is that anecdotal or is there a good research base suggesting that there is this concern not just in the UK but in a number of western European and other nations that science is becoming less popular, and our kids are becoming less good at science?
(Dr McGaw) We do have some data on science and other university enrolments that we can give you later, but it is a problem for a lot of countries. It is a parallel of the story with teachers, that people are not choosing the teaching profession; they are not choosing the science profession in some countries, and it is partly to do with the reward structures. People go into law, people go into financial management, people no longer do postgraduate studies in economics; they do MBAs and go into the business sector because that is where the financial pages tell them the rewards are. Japan has always had a much higher enrolment in science and engineering than the US for example. The US has always had a much higher enrolment in law. It is a problem.
8. Just to back-track to the methodology of the tests, when you do the 2003 tests and the 2006 tests, are you going back to exactly the same schools that you used in 2000?
(Mr Martin) No. It is not a longitudinal sample of schools, but it will be a random sample of schools in all of the countries. We are not planning a tracking instrument to follow the schools. We have developed a plan to try to track the individual students who will be surveyed in the second PISA cycle for a period of about ten years, so that in principle we would follow them as they move through the rest of the education system into higher education and hopefully into the world of work. We have developed a very ambitious longitudinal panel to follow PISA. But we are not clear how many countries will in fact opt to do that since it is a very expensive operation to try to follow the young people and assess them in this way.
(Dr McGaw) It would be good if the United Kingdom signed on, and you might consider recommending that.
9. Given that the first wave of PISA results made a big impact in some countriesin Britain the Government said, "Look how well we are doing"; in Germany they have had a big crisis about it, and you have the Swiss example of Geneva who were bottom of the league because they gave it no prioritypresumably next year and in three years' time the schools who take part will give it a much higher priority because they now know that this is going to cause a lot of ripples. Will that mean the results get better, not because the schools are getting better but because they are now giving it a priority they did not give it before?
(Mr Martin) That is a very interesting question. I think it will not really be possible to assess it until we have completed the first three waves of the first full cyclewe have done the main focus on reading, as we did this time, then on mathematics, then on scienceto see whether there is some kind of training to pass the test. I think you put your finger on a very important aspect, that is, we did try to work with national trade unions or teacher organisations to encourage them to see PISA as a positive instrument, not as a threat, not as something that they should be negative about or that they should see as imposed on them. Obviously, there is still work to be done in different countries in order to encourage people to use it, but I must say that some teacher organisations in quite a number of countries have taken to it in a very positive way. They see this as a useful supplement to existing assessments at the national level, and which can provide them with very useful feedback.
10. Before we lose the thread of Paul's question though, you have comparisons with other surveys you have carried out. When we were in Denmark last year, they were extremely concerned because OECD did a study of early years in which Denmark came very low, bottom of the Scandinavian countries. Has there been a follow-up of that early years study by which you can judge what the impact has been, or have you not returned to that?
(Mr Martin) We have launched a second wave of the early childhood project. As I mentioned in my opening presentation, the first wave of the study, which obviously you are familiar with, covered 12 OECD countries. That is less than half of the membership. A number of other OECD countries and non-OECD countries have signed up for a second round, and we have in fact just begun the process of a second round, which will include countries such as Ireland, Brazil, and a number of countries that were not covered in the first round. We hope to deepen that analysis in the next two years. As I emphasized, this is a key starting point, and indeed, we shall be following with great interest the UK's experiments in this area, with Sure Start, to see to what extent this does or does not produce real benefits for the targeted households and targeted young children who feature in this rather exciting programme.
(Dr McGaw) I should just say that there is nothing in the early childhood study that is equivalent to a direct measure of students' learning. This is the first time we have ever done it actually.
11. I wanted to ask what you think you have learned so far in terms of the gender differences. I cannot quite understand the slide. What were your initial findings around gender differences?
(Dr McGaw) You are looking at slide 16. On slide 16 the zero column is the point at which there is no difference between males and females. Rows go out to the right if males are better and to the left if females are better. You can see for reading literacy that in every country females did better than males. The ones that are shaded slightly darker at the bottom of each set of four are the non-OECD countries. Girls do much better than boys in reading literacy. In mathematical literacy there is a tendency for boys to do better and in scientific literacy in these measures in about half the countries girls do better and in about half the countries boys do better. So the gender story used to be that boys out-performed girls in maths, girls out-performed boys in reading. The gender story now is girls out-perform boys in reading and boys do not out-perform girls much in anything, and it is getting less and less.
(Mr Martin) I have asked for copies of the Executive Summary of the PISA report to be made available to you because this actually will make it much clearer to you than any of these slides one way or the other. Barry has described some of the main patterns. There are a number of other interesting differences which are highlighted in the report and in the Executive Summary about different strategies that males and females adopt to learning, particularly in mathematical literacy and reading literacy. It is also important to note that in some countries one of the reasons why boys do well in maths is that there is a relatively small number of very high performers among boys in maths which tends to push up the average performance of boys in those countries in which boys out-perform girls in maths.
12. So it is a spread issue?
(Dr McGaw) Yes.
13. What about the discussions taking place in the UK across the whole piece, as you may know, where girls do better in single sex schools rather than in mixed schools, and also in a system that has selective education as against non-selective education. Can you comment on that?
(Dr McGaw) On the gender question, these data do not speak to the issue of whether schools are single sex or co-education, and the research evidence on that is very mixed actually. I have seen claims that girls do better if they are in single sex schools and boys do better if they are in co-educational schools, so the solution is that boys should go to school with other boys' sisters, not their own; their own should go to single sex girls' schools. The message is varied across countries and mixed in that respect. On the question of ability grouping in schools, which was your second point, our conclusion at this point about Germany is that the low level of performance in Germany and the extraordinary spread of results is a direct function of the German school streaming system.
14. I want to pick up on the point that the Chairman mentioned about the shortage of teachers and the tendency for people not to go into academia, research, etc. Measuring the performance differences of educational standards across all the different OECD countries is all very well, is it not, but at the end of the day, in the future if we are not going to have enough teachers and enough lecturers, and of quality, standards are going to go down, unless we are all going to move into distance learning? You talked earlier about the shape of the future classroom, which has not changed much for many hundreds of yearsnor have some of the schools actually. I wonder if you are doing any analysis of the consequences of the cost effects. We heard the second presentation about the relationship between wealth and earnings and training and education. What are the consequences if the downward trend continues in people staying on and becoming academics and lecturers and teachers? Is there a crunch point in the future that countries have to reckon with, and make the reward system better, otherwise we will go so down that it will take a long time to come back up?
(Dr McGaw) Yes, if education systems can no longer recruit adequate numbers of adequately qualified teachers, we could face, as we have suggested in one of our publications, what could be thought of as a meltdown, where countries will be unable to run schools in the way they presently run them. There are two ways to approach the solution. One is to say we must alter the recruitment incentives so that we can restore a teaching force such as we have always had. The other is to say maybe we need to deploy people very differently in schools; maybe we need to reconfigure schooling and say we simply cannot attract and cannot afford a teaching force of this present size, so why not think more radically about what it is that teachers do and have the people that are trained and qualified as teachers do only the things that it is essential to have their qualifications to do and remit other tasks to other people, somewhat like the increased differentiation that has occurred in the nursing profession in some countries. It could be introduced in the teaching profession. The architecture of schools works against it. It is the last of the cottage industries: we put 28 kids in a room and we put a teacher with them and they work unobserved. If you have the factory model of the old school and its rooms, it would be hard to configure differently, but that may be the only solution, and the meltdown will sharpen people's attention if it begins to occur. You can go on for just so long recruiting in Australia and in New Zealand and in South Africa, but they are desperately trying to recruit because they are running short of teachers. The United States and others sometimes solve the quantitative problem by altering the quality requirements so that you do not have to train as a teacher. You can get a job in New York now with a good degree and you can train for six weeks and start teaching, and then do some further training over two years, at the end of which you are qualified as a teacher, but you have spent two years teaching.
15. Are you suggesting the UK is nearer meltdown than any other country?
(Dr McGaw) No.
(Mr Martin) No, but I think you ought to bear in mind that detailed analysis of the demography of the teaching profession in comparisons across countries shows that there is no generalised shortage of teachers across all OECD countries. That is not to say that there are not some countries that are facing, and are likely to face serious teacher shortages in the coming years, but there is for the moment no generalised shortage of teachers across the OECD area as a whole. However, there are some particular features of the teaching profession which it is very important to bear in mind. One is that its average age is rising rapidly and in most cases it is ageing more rapidly than other parts of the public sector work force. Secondly, it is becoming very strongly feminised as a profession, from primary school, lower secondary, now into upper secondary. The only area of the teaching profession where males are still in a dominant role is in higher education, but that itself is also probably going to change. The third characteristic of the teaching profession is that if you look at the relative pay of teachers and conditions of teachers compared to either average public sector workers in general or other measures of workers in the private sector, there is a great deal of variance in teachers' pay in that relative sense and in working conditions across countries. But there is a growing worry that more teachers are dropping out early, and there are some statistics to indicate that teachers are taking early retirement in greater and greater numbers in many OECD countries; for example, that many teachers in their early fifties are leaving the profession. That has got to be a concern for any country. Secondly, another related trend to that is that in many countries they are reporting more and more disenchantment on the part of senior teachers to become school managers or principals, arguing that either the rewards are not great enough to induce them to want to take this on vis-a"-vis the responsibilities and stresses that are created with this. As I say, we have an activity which we have just begun which is trying to look at this, trying to document the facts, trying to see what countries are putting in place, as Barry said, to re-think the whole question of who becomes a teacher, how long they stay a teacher, what kinds of roles and responsibilities they have to take on. I should add that in this work that we are helped by the fact that we are doing a similar kind of investigation for the health work force in another part of my directorate's work, since the health sector has a number of similar characteristics to the teaching work forcenot all, and I would not want to over-stress that, but there are a number of similarities which are quite useful when one wants to make a comparison.
Chairman: Can I thank you for that evidence. We have learned a great deal from that. We are very limited for time, but we would like to look at the digital divide and lifelong learning.
1 Not printed. Back
2 Not printed. Back
3 Not printed. Back