CRELL: critiquing global university rankings and their methodologies

This guest entry has been kindly prepared for us by Beatrice d’Hombres and Michaela Saisana of the EU-funded Centre for Research on Lifelong Learning (CRELL) and Joint Research Centre. This entry is part of a series on the processes and politics of global university rankings (see herehere, here and here).

beatriceSince 2006, Beatrice d’Hombres has been working in the Unit of Econometrics and Statistics of the Joint Research Centre of  the European Commission. She is part of the Centre for Research on Lifelong Learning. Beatrice is an economist who completed a PhD at the University of Auvergne (France). She has a particular expertise in education economics and applied econometrics.


Michaela Saisana works for the Joint Research Centre (JRC) of the European Commission at the Unit of Econometrics and Applied Statistics. She has a PhD in Chemical Engineering and in 2004 she won the European Commission – JRC Young Scientist Prize in Statistics and Econometrics for her contribution on the robustness assessment of composite indicators and her work on sensitivity analysis.


The expansion of the access to higher education, the growing mobility of students, the need for economic rationale behind the allocation of public funds, together with the demand for higher accountability and transparency, have all contributed to raise the need for comparing university quality across countries.

The recognition of this fact has also been greatly stirred  by the publication, since 2003, of the ‘Shanghai Jiao Tong University Academic Ranking of World Universities’ (henceforth SJTU), which measures university research performance across the world. The SJTU ranking tends to reinforce the evidence that the US is well ahead of Europe in terms of cutting-edge university research.

Its rival is the ranking computed annually, since 2004, by the Times Higher Education Supplement (henceforth THES). Both these rankings are now receiving worldwide attention and constitute an occasion for national governments to comment on the relative performances of their national universities.

In France, for example, the publication of the SJTU is always associated with a surge of articles in newspapers which either bemoan  the poor performance of French universities or denounce the inadequacy of the SJTU ranking to properly assess the attractiveness of the fragmented French higher education institutions landscape (see Les Echos, 7 August 2008).

Whether the intention of the rankers or not, university rankings have followed a destiny of their own and are used by national policy makers to stimulate debates about national university systems and ultimately can lead to specific education policies orientations.

At the same time, however, these rankings are subject to a plethora of criticism. They outline that the chosen indicators are mainly based on research performance with no attempt to take into account the others missions of universities (in particular teaching), and are biased towards large, English-speaking and hard-science institutions. Whilst the limitations of the indicators underlying the THES or the SJTU rankings have been extensively discussed in the relevant literature, there has been no attempt so far to examine in depth the volatility of the university ranks to the methodological assumptions made in compiling the rankings.

crell3The purpose of the JRC/Centre for Research on Lifelong Learning (CRELL) report is to fill in this gap by quantifying how much university rankings depend on the methodology and to reveal whether the Shanghai ranking serves the purposes it is used for, and if its immediate European alternative, the British THES, can do better.

To that end, we carry out a thorough uncertainty and sensitivity analysis of the 2007 SJTU and THES rankings under a plurality of scenarios in which we activate simultaneously different sources of uncertainty. The sources cover a wide spectrum of methodological assumptions (set of selected indicators, weighting scheme, and aggregation method).

This implies that we deviate from the classic approach – also taken in the two university ranking systems – to build a composite indicator by a simple weighted summation of indicators. Subsequently, a frequency matrix of the university ranks is calculated across the different simulations. Such a multi-modeling approach and the presentation of the frequency matrix, rather than the single ranks, allows one to deal with the criticism, often made to league tables and rankings systems ,that ranks are presented as if they were calculated under conditions of certainty while this is rarely the case.  crell

The main findings of the report are the following. Both rankings are only robust in the identification of the top 15 performers on either side of the Atlantic, but unreliable on the exact ordering of all other institutes. And, even when combining all twelve indicators in a single framework, the space of the inference is too wide for about 50 universities of the 88 universities we studied and thus no meaningful rank can be estimated for those universities. Finally, the JRC report suggests that THES and SJTU rankings should be improved along two main directions:

  • first, the compilation of university rankings should always be accompanied by a robustness analysis based on a multi-modeling approach. We believe that this could constitute an additional recommendation to be added to the already 16 existing Berlin Principles;
  • second, it is necessary to revisit the set of indicators, so as to enrich it with other dimensions that are crucial to assessing university performance and which are currently missing.

Beatrice d’Hombres  and Michaela Saisana

European ambitions: towards a ‘multi-dimensional global university ranking’

Further to our recent entries on European reactions and activities in relationship to global rankings schemes:

and a forthcoming guest contribution to SHIFTmag: Europe Talks to Brussels, ranking(s) watchers should examine this new tender for a €1,100,000 (maximum) contract for the ‘Design and testing the feasibility of a Multi-dimensional Global University Ranking’, to be completed by 2011.

dgecThe Terms of Reference, which hs been issued by the European Commission, Directorate-General for Education and Culture, is particularly insightful, while this summary conveys the broad objectives of the initiative:

The new ranking to be designed and tested would aim to make it possible to compare and benchmark similar institutions within and outside the EU, both at the level of the institution as a whole and focusing on different study fields. This would help institutions to better position themselves and improve their development strategies, quality and performances. Accessible, transparent and comparable information will make it easier for stakeholders and, in particular, students to make informed choices between the different institutions and their programmes. Many existing rankings do not fulfil this purpose because they only focus on certain aspects of research and on entire institutions, rather than on individual programmes and disciplines. The project will cover all types of universities and other higher education institutions as well as research institutes.

The funding is derived out of the Lifelong Learning policy and program stream of the Commission.

Thus we see a shift, in Europe, towards the implementation of an alternative scheme to the two main global ranking schemes, supported by substantial state resources at a regional level. It will be interesting to see how this eventual scheme complements and/or overturns the other global ranking schemes that are products of media outlets, private firms, and Chinese universities.

Kris Olds

International university rankings, classifications & mappings – a view from the European University Association

Source: European University Association Newsletter, No. 20, 5 December 2008.

Note: also see ‘Multi-scalar governance technologies vs recurring revenue: the dual logics of the rankings phenomenon

Multi-scalar governance technologies vs recurring revenue: the dual logics of the rankings phenomenon

Our most recent entry (‘University Systems Ranking (USR)’: an alternative ranking framework from EU think-tank‘) is getting heavy traffic these days, a sign that the rankings phenomenon just won’t go away.  Indeed there is every sign that debates about rankings will be heating up over the next 1-2 year in particular, courtesy of the desire of stakeholders to better understand rankings, generate ‘recurring revenue’ off of rankings, and provide new governance technologies to restructure higher education and research systems.

This said I continue to be struck, as I travel to selective parts of the world for work, by the diversity of scalar emphases at play.

eiffeleu1In France, for example, the broad discourse about rankings elevates the importance of the national (i.e., French) and regional (i.e., European) scales, and only then does the university scale (which I will refer to as the institutional scale in this entry) come into play in importance terms. This situation reflects the strong role of the national state in governing and funding France’s higher education system, and France’s role in European development debates (including, at the moment, presidency of the Council of the European Union).

In UK it is the disciplinary/field and then the institutional scales that matter most, with the institutional made up of a long list of ranked disciplines/fields. Once the new Research Assessment Exercise (RAE) comes out in late 2008 we will see the institutional assess the position of each of their disciplines/fields, which will then lead to more support or relatively rapid allocation of the hatchet at the disciplinary/field level. This is in part because much national government funding (via the Higher Education Funding Council for England (HEFCE), the Scottish Funding Council (SFC), the Higher Education Funding Council for Wales (HEFCW) and the Department for Employment and Learning, Northern Ireland (DEL)) to each university is structurally dependent upon the relative rankings of each university’s position in the RAE, which is the aggregate effect of the position of the array of fields/disciplines in any one university (see this list from the University of Manchester for an example). The UK is, of course, concerned about its relative place in the two main global ranking schemes, but it doing well at the moment so the scale of concern is of a lower order than most other countries (including all other European countries). Credit rating agencies also assess and factor in rankings with respect to UK universities (e.g. see ‘Passing judgment’: the role of credit rating agencies in the global governance of UK universities‘).

In the US – supposedly the most marketized of contexts – there is highly variably concern with rankings.  Disciplines/fields ranked by media outlets like U.S. News & World Report are concerned, to be sure, but U.S. News & World Report does not allocate funding. Even the National Research Council (NRC) rankings matter less in the USA given that its effects (assuming it eventually comes out following multiple delays) are more diffuse. The NRC rankings are taken note of by deans and other senior administrators, and also faculty, albeit selectively. Again, there is no higher education system in the US – there are systems. I’ve worked in Singapore, England and the US as a faculty member and the US is by far the least addled or concerned by ranking systems, for good and for bad.

While the diversity of ranking dispositions at the national and institutional levels is heterogeneous in nature, the global rankings landscape is continuing to change, and quickly. In the remainder of this entry we’ll profile but two dimensions of the changes.

Anglo-American media networks and recurrent revenue

ustheFirst, new key media networks, largely Anglo-American private sector networks, have become intertwined.  As Inside Higher Ed put it on 24 November:

U.S. News & World Report on Friday announced a new, worldwide set of university rankings — which is really a repackaging of the international rankings produced this year in the Times Higher Education-QS World University Rankings. In some cases, U.S. News is arranging the rankings in different ways, but Robert Morse, director of rankings at the magazine, said that all data and the methodology were straight from the Times Higher’s rankings project, which is affiliated with the British publication about higher education. Asked if his magazine was just paying for reprint rights, Morse declined to discuss financial arrangements. But he said that it made sense for the magazine to look beyond the United States. “There is worldwide competition for the best faculty, best students and best research grants and researchers,” he said. He also said that, in the future, U.S. News may be involved in the methodology. Lloyd Thacker, founder of the Education Conservancy and a leading critic of U.S. News rankings, said of the magazine’s latest project: “The expansion of a business model that has profited at the expense of education is not surprising. This could challenge leaders to distinguish American higher education by providing better indicators of quality and by helping us think beyond ranking.”

This is an unexpected initiative, in some ways, given that the Times Higher Education-QS World University Rankings are already available on line and US New and World Report is simply repackaging these for sale in the American market. Yet if you adopt a market-making perspective this joint venture makes perfect sense. Annual versions of the Times Higher Education-QS World University Rankings will be reprinted in a familiar (to US readers) format, thereby enabling London-based TSL Education Ltd., London/Paris/Singapore-based QS Quacquarelli Symonds, and Washington DC-based U.S. News and World Report to generate recurring revenue with little new effort (apart from repackaging and distribution in the US). The enabling mechanism is, in this case, reprint rights fees. As we have noted before, this is a niche industry in formation, indeed.

More European angst and action

And second, at the regional level, European angst (an issue we profiled on 6 July in ‘Euro angsts, insights and actions regarding global university ranking schemes‘) about the nature and impact of rankings is leading to the production of critical reports on rankings methodologies, the sponsorship of high powered multi-stakeholder workshops, and the emergence of new proposals for European ranking schemes.

ecjrccoverSee, for example, this newly released report on rankings titled Higher Education Rankings: Robustness Issues and Critical Assessment, which is published by the European Commission Joint Research Centre, Institute for the Protection and Security of the Citizen, Centre for Research on Lifelong Learning (CRELL)

The press release is here, and a detailed abstract of the report is below:

The Academic Ranking of World Universities carried out annually by the Shanghai’s Jiao Tong University (mostly known as the ‘Shanghai ranking’) has become, beyond the intention of its developers, a reference for scholars and policy makers in the field of higher education. For example Aghion and co-workers at the Bruegel think tank use the index – together with other data collected by Bruegel researchers – for analysis of how to reform Europe’s universities, while French President Sarkozy has stressed the need for French universities to consolidate in order to promote their ranking under Jiao Tong. Given the political importance of this field the preparation of a new university ranking system is being considered by the French ministry of education.

The questions addressed in the present analysis is whether the Jiao Tong ranking serves the purposes it is used for, and whether its immediate European alternative, the British THES, can do better.

Robustness analysis of the Jiao Tong and THES ranking carried out by JRC researchers, and of an ad hoc created Jiao Tong-THES hybrid, shows that both measures fail when it comes to assessing Europe’s universities. Jiao Tong is only robust in the identification of the top performers, on either side of the Atlantic, but quite unreliable on the ordering of all other institutes. Furthermore Jiao Tong focuses only on the research performance of universities, and hence is based on the strong assumption that research is a universal proxy for education. THES is a step in the right direction in that it includes some measure of education quality, but is otherwise fragile in its ranking, undeniably biased towards British institutes and somehow inconsistent in the relation between subjective variables (from surveys) and objective data (e.g. citations).

JRC analysis is based on 88 universities for which both the THES and Jiao Tong rank were available. European universities covered by the present study thus constitute only about 0.5% of the population of Europe’s universities. Yet the fact that we are unable to reliably rank even the best European universities (apart from the 5 at the top) is a strong call for a better system, whose need is made acute by today’s policy focus on the reform of higher education. For most European students, teachers or researchers not even the Shanghai ranking – taken at face value and leaving aside the reservations raised in the present study – would tell which university is best in their own country. This is a problem for Europe, committed to make its education more comparable, its students more mobile and its researchers part of a European Research Area.

Various attempts in EU countries to address the issue of assessing higher education performance are briefly reviewed in the present study, which offers elements of analysis of which measurement problem could be addressed at the EU scale. [my emphasis]

While ostensibly “European”, does it really matter that the Times Higher Education-QS World University Ranking is produced by firms with European headquarters, while the Jiao Tong ranking is produced by an institution based in China?

The divergent logics underlying the production of discourses about rankings are also clearly visible in two related statements. At the bottom of the European Commission’s Joint Research Centre report summarized above we see “Reproduction is authorised provided the source is acknowledged”, while the Times Higher Education-QS World University Rankings, a market-making discourse, is accompanied by a lengthy copyright warning that can be viewed here.

Yet do not, for a minute, think that ‘Europe’ does not want to be ranked, or use rankings, as much if not more than any Asian or American or Australian institution. At a disciplinary/field level, for example, debates are quickly unfolding about the European Reference Index for the Humanities (ERIH), a European Science Foundation (ESF) backed initiative that has its origins in deliberations about the role of the humanities in the European Research Area. The ESF frames it this way:

Humanities research in Europe is multifaceted and rich in lively national, linguistic and intellectual traditions. Much of Europe’s Humanities scholarship is known to be first rate. However, there are specifities of Humanities research, that can make it difficult to assess and compare with other sciences. Also,  it is not possible to accurately apply to the Humanities assessment tools used to evaluate other types of research. As the transnational mobility of researchers continues to increase, so too does the transdisciplinarity of contemporary science. Humanities researchers must position themselves in changing international contexts and need a tool that offers benchmarking. This is why ERIH (European Reference Index for the Humanities) aims initially to identify, and gain more visibility for top-quality European Humanities research published in academic journals in, potentially, all European languages. It is a fully peer-reviewed, Europe-wide process, in which 15 expert panels sift and aggregate input received from funding agencies, subject associations and specialist research centres across the continent. In addition to being a reference index of the top journals in 15 areas of the Humanities, across the continent and beyond, it is intended that ERIH will be extended to include book-form publications and non-traditional formats. It is also intended that ERIH will form the backbone of a fully-fledged research information system for the Humanities.

See here for a defense of this ranking system by Michael Worton (Vice-Provost, University College London, and a member of the ERIH steering committee).  I was particularly struck by this comment:

However, the aim of the ERIH is not to assess the quality of individual outputs but to assess dissemination and impact. It can therefore provide something that the RAE cannot: it can be used for aggregate benchmarking of national research systems to determine the international standing of research carried out in a particular discipline in a particular country.

Link here for a Google weblog search on this debate, while a recent Chronicle of Higher Education article (‘New Ratings of Humanities Journals Do More Than Rank — They Rankle’) is also worth reviewing.

Thus we see a new rankings initiative emerging to enable (in theory) Europe to better codify its highly developed humanities presence on the global research landscape, but in a way that will enable national (at the intra-European scale) peaks (and presumably) valleys of quality output to be mapped for the humanities, but also for specific disciplines/fields. Imagine the governance opportunities available, at multiple scales, if this scheme is operationalized.

And finally, at the European scale again, University World News noted, on 23 November, that:

The European Union is planning to launch its own international higher education rankings, with emphasis on helping students make informed choices about where to study and encouraging their mobility. Odile Quintin, the European Commission’s Director-General of Education and Culture, announced she would call for proposals before the end of the year, with the first classification appearing in 2010.

A European classification would probably be compiled along the same lines as the German Centre for Higher Education Development Excellence Ranking.

European actors are being spurred into such action by multiple forces, some internal (including the perceived need to ‘modernize European universities in the context of Lisbon and the European Research Area), some external (Shanghai Jiao Tong; Times Higher QS), and some of a global dimension (e.g., audit culture; competition for mobile students).

eurankingsprogThis latest push is also due to the French presidency of the Council of the European Union, as noted above, which is facilitating action at the regional and national scales. See, for example, details on a Paris-based conference titled ‘International comparison of education systems: a european model?’ which was held on 13-14 November 2008. As noted in the programme, the:

objective of the conference is to bring to the fore the strengths and weaknesses of the different international and European education systems, while highlighting the need for regular and objective assessment of the reforms undertaken by European Member States by means of appropriate indicators. It will notably assist in taking stock of:
– the current state and performance of the different European education systems:
– the ability of the different European education systems to curb the rate of failure in schools,
– the relative effectiveness of amounts spent on education by the different Member States.

The programme and list of speakers is worth perusing to acquire a sense of the broad agenda being put forward.

Multi-scalar governance vs (?) recurring revenue: the emerging dual logics of the rankings phenomenon

The rankings phenomenon is here to stay. But which logics will prevail, or at least emerge as the most important in shaping the extension of audit culture into the spheres of higher education and research?  At the moment it appears that the two main logics are:

  • Creating a new niche industry to form markets and generate recurrent revenue; and,
  • Creating new multi-scalar governance technologies to open up previously opaque higher education and research systems, so as to facilitate strategic restructuring for the knowledge economy.

These dual logics are in some ways contradictory, yet in other ways they are interdependent. This is a phenomenon that also has deep roots in the emerging centres of global higher ed and research calculation that are situated in London, Shanghai, New York, Brussels, and Washington DC.  And it is underpinned by the analytical cum revenue generating technologies provided by the Scientific division of Thomson Reuters, which develops and operates the ISI Web of Knowledge.

Market-making and governance enabling…and all unfolding before our very eyes. Yet do we really know enough about the nature of the unfolding process, including the present and absent voices, that seems to be bringing these logics to the fore?

Kris Olds

‘University Systems Ranking (USR)’: an alternative ranking framework from EU think-tank

One of the hottest issues out there still continuing to attract world-wide attention is university rankings. The two highest profile ranking systems, of course, are the Shanghai Jiao Tong and the Times Higher rankings, both of which focus on what might constitute a world class university, and on the basis of that, who is ranked where. Rankings are also part of an emerging niche industry. All this of course generates a high level of institutional, national, and indeed supranational (if we count Europe in this) angst about who’s up, who’s down, and who’s managed to secure a holding position. And whilst everyone points to the flaws in these ranking systems, these two systems have nevertheless managed to capture the attention and imagination of the sector as a whole. In an earlier blog enty this year GlobalHigherEd mused over why European-level actors had not managed to produce an alternate system of university rankings which might counter the hegemony of the powerful Shanghai Jiao Tong (whose ranking system privileges the US universities) on the one hand, and act as a policy lever that Europe could pull to direct the emerging European higher education system, on the other.

Yesterday The Lisbon Council, an EU think-tank (see our entry here for a profile of this influential think-tank) released which might be considered a challenge to the Shanghai Jiao Tong and Times Higher ranking schemes – a University Systems Ranking (USR) in their report University Systems Ranking Citizens and Society in the Age of Knowledge. The difference between this ranking system and the Shanghai and Times is that it focuses on country-level data and change, and not  individual institutions.

The USR has been developed by the Human Capital Center at The Lisbon Council, Brussels (produced with support by the European Commission’s Education, Audiovisual and Culture Executive Agency) with advice from the OECD.

The report begins with the questions: why do we have university systems? What are these systems intended to do? And what do we expect them to deliver – to society, to individuals and to the world at large? The underlying message in the USR is that “a university system has a much broader mandate than producing hordes of Nobel laureates or cabals of tenure – and patent bearing professors” (p. 6).

So how is the USR different, and what might we make of this difference for the development of universities in the future? The USR is based on six criteria:

  1. Inclusiveness – number of students enrolled in the tertiary sector relative to the size of its population
  2. Access – ability of a country’s tertiary system to accept and help advance students with a low level of scholastic aptitude
  3. Effectiveness – ability of country’s education system to produce graduates with skills relevant to the country’s labour market (wage premia is the measure)
  4. Attractiveness – ability of a country’s system to attract a diverse range of foreign students (using the top 10 source countries)
  5. Age range – ability of a country’s tertiary system to function as a lifelong learning institution (share of 30-39 year olds enrolled)
  6. Responsiveness – ability of the system to reform and change – measured by speed and effectiveness with which Bologna Declaration accepted (15 of 17 countries surveyed have accepted the Bologna criteria.

These are then applied to 17 OECD countries (all but 2 signatories of the Bologna Process). A composite ranging is produced, as well as rankings on each of the criteria. So what were the outcomes for the higher education systems of these 17 countries?

Drawing upon all 6 criteria, a composite figure of USR is then produced. Australia is ranked 1st; the UK 2nd and Denmark 3rd, whilst Austria and Spain are ranked 16th and 17th respectively (see Table1 below). We can also see rankings based on specific criteria (Table 2 below).



There is much to be said for this intervention by The Lisbon Council – not the least being that it opens up debates about the role and purposes of universities. Over the past few months there have been numerous heated public interventions about this matter – from whether universities should be little more than giant patenting offices to whether they should be managers of social justice systems.

And though there are evident shortcomings (such as the lack of clarity about what might count as a university; the view that a university-based education is the most suitable form of education to produce a knowledge-based economy and society; what is the equity/access etc range within any one country, and so on), the USR does, at least, place issues like ‘lifelong learning’, ‘access’ and ‘inclusion’ on the reform agenda for universities across Europe. It also sends a message that it has a set of values that currently are not reflected in the two key ranking systems that it would like to advance.

However, the big question now is whether universities will see value in this kind of ranking system for its wider systemic, as opposed to institutional, possibilities, even if it is as a basis for discussing what are universities for and how might we produce more equitable knowledge societies and economies.

Susan Robertson and Roger Dale

Investing wisely for Australia’s future

Editor’s note: The following speech was given by Professor Ian Chubb, Vice-Chancellor of The Australian National University (ANU) on Wednesday 29 October 2008 at the National Press Club of Australia. It is reprinted in GlobalHigherEd with his kind permission.


Thank you Ken – for your welcome and introduction.

It has been some years since I last spoke at the National Press Club, and I appreciate the opportunity to do so again – particularly at the time when education reviews and research enquiries are being finalised and Government responses being prepared.

It is an important time; and there are opportunities not to be missed – and I plan to raise some of those with you today.

I suppose, before I start, I should make three things clear:

  1. I support the push for better funding for universities – accompanied by both reform and with selectively allocated additional funds for particular purposes based largely on the quality of the work we do – where we do it;
  2. I support the directions being pursued by the Government – and look forward to the outcomes of their various deliberations.
  3. I remind you that I am from ANU, that I work for ANU and that I serve ANU.  I like to think that through that role, however, I can also serve a bigger one – particular and important aspects of the national interest.

We at ANU do make a contribution to Australia and beyond.  For a start, we educate excellent students very well indeed; we rate in the top ‘A1’ band of the Federal Minister’s Teaching and Learning Performance Fund across our teaching profile – and have done so in the two years the category has been identified.   This was a surprise to some in the higher education sector, where we cherish the notion of a teaching-research nexus.  Notwithstanding the mantra, some had anticipated that better teaching would be done in the places where there was less research – more to spend on it perhaps, more of a focus, and so on.  It was presumed that the research-intensive universities would probably see teaching as a chore. But the research-intensive universities are places where staff and students alike are learners and where all of them are just downright curious to discover what they don’t know.  And at its best, they do it together.

At ANU we continue to work to improve what we do and how we do it. We set ourselves and our students stretch targets. We aim for high standards – not throughput. And we aim to give our students a head start in their life after University.

In research we do well.  We review what we do, we rate what we do, and we manage what we do in a way that some would call relentless, though few could argue is stifling.  So I am proud that the ANU continues to lead among Australian universities in the various world rankings that are, necessarily, based largely on research performance.

We are placed at 59 in the Shanghai Jiao Tong’s most recent listings and 16th on the list produced by the UK Times Higher Education Supplement, a position we have maintained now over three consecutive years.

I am proud because the rankings reflect well on the efforts of talented people. It is useful and it is reinforcing for that reason, and possibly more usefully it tells you about your neighbourhood when the world’s universities are rated along with you using a particular set of indicators.

I am not at all boastful, however, because we all know that such rankings are constructed arbitrarily around some of the available comparable measures year on year.  That they are called ‘prestigious’ or ‘highly regarded’ is largely because they are published each year, and because we have nothing else.  They are represented as ‘qualitative’ when in fact they are only partly about quality.

This is one reason why I support the Federal Government’s Excellence in Research for Australia (ERA) proposal, because, handled well, it will provide us with an efficacious and internationally benchmarked model to judge research quality in Australia.  Then we should have something truly useful to talk about.

But let me now talk about something usefully indicative that can be drawn from the Shanghai Jiao Tong (SJT) world university rankings: the neighbourhood.

When the rankings first came out five years ago, there were seven Chinese universities in the top 500.  This year there are eighteen.  It is quite possible that in five years’ time, given both the rate and the selectivity of their additional investment, there will be 10 or so Chinese universities in the top 200 and several in the top 100.  Australia may well struggle to keep our current three (ANU, Melbourne and Sydney) in that league.  Rankings are about relative performance and positions can change because of bigger changes in the performance of others and not because your own has slipped – or it could even be the way in which institutions present their data rather than any actual change in performance.  But the outcomes send signals that can be read.

Does this all matter?  Well I think it does, but it is not the only thing that matters.

When you look at the countries that have more than one university rated in the top 100 on the SJT ranking in 2007 you can see that they are countries with a high GDP per capita.

The United States stands out because of its scale.  The United Kingdom holds par when adjusted for population size.  Australia and Canada have been lifting above their weight, but Canada is now waxing while Australia is waning in every disciplinary field.  Asian universities are rising. European universities now realise that they are being left behind – but have started asking the right questions.

But history tells us that if you’re not able to pull your weight and to be a contributor you risk being locked out as a discipline, or as a university, or as a nation. If we don’t match investment and don’t match performance, Australia could be back to somewhere near where we were before 1946 – on the outside looking in.

In a world thirsty for talent and the benefits derived from the application of that talent, strategies are changing.  Many countries are ramping up their investments in their leading universities.  They are selectively concentrating additional funding for research, for research infrastructure, research centres of excellence and international research collaborations.  They are increasing the number of professors, and developing career opportunities for post-doctoral staff while enlarging doctoral student enrolments, including international PhD enrolments.

Take the example of Canada, which has set itself a goal of ranking amongst the top 4 countries in the world in terms of R&D performance across the government, business and higher education sectors. It set a target in 2002 of increasing the admission of Masters and PhD students at Canadian universities by 5% per year through to 2010. It is providing $300 million annually to support the salary costs of 2000 research professors in Canadian universities and hospitals, seeking, in their own words, ‘to unleash the full research potential’ of these institutions by attracting the best talent in the world – and they are doing so selectively. Close to two thirds of the Chairs have been allocated to the 13 most research intensive universities amongst their nearly 70 universities and roughly equal number of Colleges. Just last week the Canadian Finance Minister commented that they must build on  ‘our knowledge advantage’ and that: “This is a critical time in Canada in terms of making sure that in our public-policy decisions that we support universities and colleges.”

Germany.  Germany has invested heavily in research and innovation, particularly in the university sector, aiming, in their own words, “to establish internationally visible research beacons in Germany.” Their strategy includes spending up to 195 million Euros each year to establish internationally visible research and training Clusters of Excellence, based at universities but collaborating with non-university institutions. Closely tied to this is an effort to the tune of 210 million Euros each year to heighten the profile and strength of ten selected universities to be amongst world’s best in their areas of demonstrable excellence.

China.  China is the world’s fasted growing supporter of research and development with its national R&D funding now third highest in the world, just behind the United States and Japan. In 1998 China instituted the 985 Project, under which its ten leading universities were given special grants in excess of US$ 124 million over three years with the explicit aim of ensuring that China had universities represented amongst the world’s best. They have now enlarged the program to cover 36 universities amongst their hundreds.

Australia has still to move – and we have choices to make.  We see what is happening elsewhere: we see mostly additional funding concentrated and selectively allocated – not overtly, at least, at the expense of core funding; we see the benefits of granting universities adequate, strategic but accountable funding (like we once had); we see overt attempts to ‘internationalise’ by drawing staff and students from the global talent pool.  There is more…and so there are many lessons to be absorbed by us – an important one is to resist the temptation to spread additional resources – it would be close to a uniquely Australian approach.

And this in a world that won’t care unless we earn the right to be at their table; it is as true for our university leaders, our staff and our students, as it is for our political or our business leaders.   As I said earlier – if we are not at the table we would be back to something like the position we were just after the Second World War.

Our approach must be different from now.  We do need reform and we don’t need more tinkering. We don’t need more money tied up in small, so-called competitive programs that only partially fund what they purport to support and are not conducive to long-term planning.

I support a policy that will help universities be different from each other and to be outstanding at what they do.  I support policy-driven differentiation, not drift, and I support additional funding allocations above a better base related to both purpose and quality of work multiplied by the quantity of work.

I do not think that there is only one right way forward. And I would be happy to see us move on from an outdated ‘one-size fits all’ approach with side deals.  But not if it were replaced by the same sort of blunt and clumsy instrument.

But while there might not be one right way, I do know the wrong way: continuing the chronic partial funding of nearly everything we do.  In fact, we presently cope with a lot that is chronic. Chronic under-investment. Chronic tinkering rather than real reform. Chronic suspicion rather than trust. Chronic erosion of capital and infrastructure rather than provision of the best possible resources to enable our most talented to do their best work here. Chronic lack of support for students who are seen as a cost rather than a  means by which Australia invests in its future.  A chronic under-valuing of staff rather than recognising that the world call on talent means temptations are rife elsewhere. And a chronic under-valuing of PhD work and scholarships rather than using the peak to build capacity. The story rolls on.

For the universities of Australia to serve Australia’s interests, we need to be different from each other, respected for what we do, and be supported for what we do well over and above core support.  And we need the policy framework to make it happen, knowing that a consequence will be differential funding as different activities cost differently.

As a start we need to articulate what universities are for, what different purposes they may serve, and how.

In a recent essay, ‘What are universities for?‘, Boulton & Lucas (2008) suggest that the enduring role of universities is to create new opportunities and promote learning that enables deep thinking beyond the superficial, handles complexity and ambiguity, and shapes the future.  They argue that it is important not to be beguiled by prospects of immediate payoff from investment in university research and teaching.

I have a duty of care in my position to help build the University’s capacity to respond to such challenges in ways that are consistent with our envisaged role.  It is my responsibility, primarily, to ensure that the people with scholarly expertise in the University have the room and resources to excel in their research, the opportunity through teaching to share their first-hand insights with their students (I note that Paul Krugman, the most recent winner of the Nobel Prize for his research in Economics, said when he began his thanks: you have to start with your teachers), and the freedom to speak out when they see the need to do so, and to put their expertise into the public arena to help inform public opinion.

Let me indicate the ways by which universities can contribute, and then suggest some options for public policy.

I work from the premise that the ability of a university to deliver its mission depends crucially on public support, appropriate regulatory frameworks and adequate funding.  Without the requisite public trust and support universities cannot survive in today’s world.

Interestingly, the available evidence from surveys of community attitudes suggests that when it comes to major social, environmental and economic issues, the public and the Government look to universities for analysis, understanding and solutions.

Some of the current areas are well known: economic uncertainty, climate change, the threat of pandemics, sources of terrorism and the potential of alternative energy, just to name a few.

One of the ways ANU engages with the broader Australian community and seeks to understand what we Australians think is via ANUpoll.

The ANUpoll, led by Professor Ian McAllister in our College of Arts and Social Sciences, differs from other opinion polls by placing public opinion in a broad policy context, and by benchmarking Australian against international opinion. It can also reveal trends in opinions over many decades, drawing on the wide range of public opinion polls conducted at ANU since the 1960s.

It tells us interesting things about Australians. The first Poll, released in April this year, revealed that Australians, by international standards, are much more positively disposed to high levels of government expenditure, particularly on health, education, the environment and police. The Poll tells us that there is a greater level of trust in government in Australia relative to other nations.

The second poll, released in September, sought the views of Australians on higher education. It found that Australians are concerned about fair and equitable access to our universities; they view university as one important way of improving the job prospects of their children, but not the only avenue to success; and they believe that the decline in public funding for universities has gone too far.

And we know from the  ANUpoll released today that concern about climate change is a big issue for the community. Global warming is perceived as a major long-term threat to the health of the planet by a large proportion of the population.  But there is no simple solution to this problem. It is one that crosses the boundaries of science, social sciences, health, economics, law, philosophy and more. It is a big challenge; and Universities have a key role to play in meeting it.

It is no coincidence that the Australian Government and state and territory governments turned to a respected academic to investigate the impact of climate change on Australia, and to propose potential solutions.  Professor Ross Garnaut in turn drew upon the work of many of his colleagues at ANU and other universities, for the latest data, for research, thinking and ideas to respond to what he identified as a ‘diabolical problem.’

Although from one discipline, Economics, Professor Garnaut’s report reflects the reality that at the heart of the climate change challenge is the need for a deep comprehension of interlaced, inseparable elements in highly complex systems. Perhaps no challenge facing us demands such an interdisciplinary approach. It is a challenge that the community expects universities to help to meet, and one that universities must help meet.

ANU is seeking to respond to that challenge with the formation of the ANU Climate Change Institute under the leadership of Professor Will Steffen.  This initiative represents a substantial effort by the University community to harness expertise across disciplines to extend knowledge about climate change – its drivers, its implications, the scope for positive responses to its impact, and possible correctives to its trajectory.

It will develop interdisciplinary research projects on climate change through the application of the University’s core capabilities around critical questions and issues.

It will develop high quality education programs aimed at meeting the national and international demand for qualified practitioners.  From 2009 ANU will offer an interdisciplinary Masters in Climate Change offered jointly between the Fenner School of Environment and Society and the Crawford School of Economics and Government. We believe it is the first of its kind in Australia.

The Climate Change institute will also engage globally, co hosting the International Alliance of Research Universities Copenhagen Climate Change Congress March 2009, and engaging with the Intergovernmental Panel on Climate Change (IPCC), the World Climate Research Programme (WCRP), and the International Geosphere-Biosphere Programme (IGBP) among others.

ANU is seeking to respond to the expectations of the Australian community and government that the national university seek to find solutions to the complex problems that confront us.  The reality is that the world’s problems are inter-connected, and universities need organisational flexibility to respond creatively to the need for new knowledge in addressing them.

While the world faces the ticking time bomb of climate change, and universities here and around the world seek new ways to address such complex problems, another time bomb is ticking for universities – Australia’s changing demography.

The Group of Eight, has released today a backgrounder on the challenge of population change. It estimates that at least one third of the annual number of Australian PhD graduates will be needed each year on average over the next decade merely to replace retirements from the academic workforce.  Currently three quarters of doctoral graduates flow into non-academic occupations, so without additional output we would see either a slowdown of doctoral supply to the broader labour market – at a time when the country is seeking to increase the capacity of the private and public sectors to absorb new knowledge – or a shortfall in academic positions, and this is without factoring in any increase in the number of institutions to meet growth in future demand for tertiary education.

It was therefore pleasing to see the interim Report of the House of Representatives Standing Committee on Industry, Science and Innovation on Research Training in Australia.  The committee is convinced, as are we, that there is a strong case for reform – and importantly, recommendations with budget implications have bi-partisan support.

The problem is sharper for fields of research from which the pull of labour market demand is strongest – such as in engineering or geology.  We should not assume that we can meet domestic shortfall readily through immigration in the future without being prepared to pay the prices that the intensifying international competition for intellectual talent is beginning to demand.

The educational attainment of the Australian workforce is considerably below that of the world’s leaders.  Two-thirds of Australia’s workforce over 25 years of age have no post-secondary qualifications, and one third have achieved less than upper-secondary education.  Only 77% of females and 68% of males aged 19 have completed Year 12 or equivalent.

To bring Australia up to an educated workforce equivalent to the world’s leaders would involve an additional 1 million people between 25 and 44 years getting tertiary education qualifications.  To achieve that lift in the domestic skills base is challenging.  Not to do it leaves a challenge of a different kind.

Additionally, for young people aged 15 to 25, that objective would require a much higher rate of participation and would mean finding new ways of promoting access and success among potential learners who lack readiness.  For equity as well as productivity purposes it is necessary to close the achievement gap without lowering educational standards.

Taken together these rising and diversifying forms of demand for learning cannot be accommodated within the current structure of tertiary education.  Greater diversification and innovation will be needed, including new types of providers, public and private, offering flexible, multi-modal access to learning opportunities.

We should not assume this will happen naturally.  Indeed we can expect resistance to it.  New incentives will be needed to overcome structural and cultural conservatism. Another reason to move from the ‘one size fits all’ approach and rather than looking for a simple solution develop a policy framework that promotes and supports difference through design rather than drift.

Twenty years ago the Australian Government expanded higher education on a foundation of three pillars:

  • An injection of additional public investment for growth in student places and new campuses
  • The provision of income-contingent loans to enable students to make a co-contribution to the costs of higher education without up-front financial barriers
  • A redistribution of existing resources from established universities to new institutions, notably through a ‘clawback’ of research funding.

The legacy of that approach is the persistence of sameness in the funding rates for teaching, the thin spreading of funding, unvalidated claims about standards of qualifications and excellence, and a levelling down of performance peaks. It was a ‘one size fits all approach’ and it was called a unified national system. In my experience over now 23 years, it was not national, rarely unified and hardly a system.

Expansion encouraged all universities to adopt broadly similar aspirations.

We are not alone.  Boulton and Lucas made that clear to us when they discussed the European dilemma: how to have research powerhouses amongst the world’s best and provide higher education for a growing proportion of the population.  They point out that “…excessive convergence towards a single model of the basic research-focused university, with a lack of differentiated purpose, structure and mission…” has resulted in at least 980 (European) universities claiming to “aspire to achieve international excellence in research.”  In the same article, they point out that:  “The US has resolved this dilemma. Fewer than 250 universities award postgraduate degrees and fewer than 100 are recognised as research intensive, with the rest devoted to teaching and scholarship.” And remember that the U.S has thousands of post-secondary institutions.

The approach in Europe and Australia, including the funding approach, probably impeded some universities from identifying and investing in niches neglected by the established research universities.

Regardless of their varying circumstances, universities have tended to use the rhetoric of excellence, rather than ‘fitness for purpose’.  But ‘excellence’ is an empty notion when used without reference to validated performance standards.

The desire of institutions to move ‘up whichever ladder’ distracts higher education from its public purposes, skews missions, and alters institutional priorities and spending to drift beyond the limits of their capacity.

We see this very clearly in Australia where the gap between the Go8 universities and others in terms of research performance has been widening, not narrowing, despite the processes and funding of the last twenty years.

Clearly it is sub-optimal and counter-productive for the country to continue diluting the public investment in proven research capacity and performance.  We certainly cannot afford to apply this flawed thinking of the past to the future expansion and diversification of tertiary education.

An unfortunate effect of rankings like the Shanghai Jiao Tong measures that are based on internationally comparable data relating primarily to research output quality is that, in a highly competitive context, they reinforce traditional academic norms and encourage what Frans Van Vught has termed the ‘reputation race’.

He noted recently that:

The average quality of the current European higher education and research system is good but its excellence is limited.  A diversification of missions and of research portfolios and funding levels, would be necessary to allow the occurrence of more European top universities.

We could say the same about the fair to average overall quality of Australian higher education and research, while noting the view strongly held in European quarters and which resonates here, that student mobility is promoted through avoidance of stratification of universities.  It is seen to be anti-egalitarian to invest in excellence – at least in intellectual endeavours, for we don’t appear to have the same reluctance in sport. We invest additionally in the best athletes and national teams because in sport, we understand the concept of investing in excellence and that high achievement encourages the others.

Now we need a new approach to meet new challenges alongside longstanding needs to enlarge educational participation and strengthen capacity.

The three pillars on which the current system was expanded twenty years ago have become unstable. The Government share of funding has shrunk.  Market sources of finance are playing a greater role.

The problem with reliance on market forces in higher education is its tendency to reduce diversity in the system, and raise costs for students and taxpayers.  The market can be afraid of difficult or intellectually challenging ideas where the payoff isn’t easily predictable or easily apparent.

Clearly we can’t and don’t want to wind back the clock to a centrally-planned model of higher education.  Equally we cannot rely simply on the market.  A more flexible regulatory and financing approach is necessary, and we need to give form to the notion of mission-based funding compacts for each university that Labor proposed ahead of the 2007 election and has indicated subsequently its intention to progress in government.  I note that Deputy Prime Minister Gillard and Minister Carr have repeatedly declared their ambitions for the universities: structurally reformed, culturally reformed, socially inclusive and internationally competitive.  Hard to argue against – possible if not easy to achieve.

It is not enough to give universities what they ask for: more money, less regulation and more autonomy.  Or for universities to expect to be given what they ask for.  Much as we might be able to argue the compelling case for better generic funding, I can’t see that we stand a chance without conceding substantial reform and improvement.

To achieve what we need, we need not just Compacts but Compacts with teeth. We need Compacts that will hold us to hard decisions, validate and use evidence to agree and provide adequate support for our strengths and not simply endorse what we say about ourselves.

Compacts will fail to provide bold and different approaches if they are tied up in second-order metrics for shallow accountability reporting.

There must be some sharp incisors to bite through the surface of universities’ claims.  I suggest that the Government should complement negotiating teams of departmental officials with people with university experience (possibly international) who can exercise the discriminating judgements that will be necessary to validate the claims of universities against their missions.

The two main components of Compacts that may be on offer are the full funding of research and greater flexibility in financing of higher education.   I see these compacts working along the lines of the recent COAG reforms of Commonwealth-State specific purpose programs, to support additional performance-based actions on top of adequate (better) funding for core activities.  These would be significant reforms and I understand they need to be mutually beneficial for universities and the supporting community.

Hence, in return for full economic costs of research, I believe it is more than reasonable that universities should be able to demonstrate better knowledge of their costs, proper pricing, avoidance of internal cross subsidies, and improved management of their estates.

In return for improved funding, greater financing flexibility, and, for some universities, ‘deregulation’, I believe universities should be prepared to expand scholarships and bursaries for needy students, extend their outreach to raise aspirations and readiness of students from disadvantaged areas, and give greater attention to student success.

In a truly differentiated system it will be necessary to provide better support for students.  We must have a system that allows talented students, regardless of their life circumstances, to go to the university that best meets their ambitions and interests.  This will mean tackling the issue of income support, including rental assistance, if we are to develop a comprehensive strategy for improving the socio-economic mix of student enrolments in a markedly differentiated university system.

The participation rate of disadvantaged groups in higher education, notably students from low socio-economic backgrounds, Indigenous Australians, and Australians from regional and remote areas, remains low.

For many of these potential students and their parents, the additional education costs that cannot be deferred in the same way as HECS constitute an insurmountable burden – living expenses remain the major financial barrier to participation. Yet the system of student income support has not been reviewed by government since 1992.

I believe the Government is heading in the right direction with its three pillars of reform:

  • Expanding participation for the purposes of social inclusion and productivity improvement.
  • Focussing on internationally benchmarked quality as the key driver of investment in research and research training. An additional benefit of which might be to dispense with ‘perceptions’ and replace them with proven performance.
  • Increasing university flexibility and harmonising competitive processes with national public priorities.

I can simply enjoin the Government to stay on track, hold to the line and not get distracted by those who will seek a weaker course.  Even if there is to be a shortfall between the investment increases we need and the capacity of the economy to afford them for the time being, it is imperative that there is no compromise on the goals we set for ourselves and the standards we set for their achievement.

Anything less would sell Australia short.

Ian Chubb