On the illogics of the Times Higher Education World Reputation Rankings (2013)

Note: you can link here for the Inside Higher Ed version of the same entry.

~~~~~~~~

Amidst all the hype and media coverage related to the just released Times Higher Education World Reputation Rankings (2013), it’s worth reflecting on just how small of a proportion of the world’s universities are captured in this exercise (see below). As I noted last November, the term ‘world university rankings’ does not reflect the reality of the exercise the rankers are engaged in; they only focus on a minuscule corner of the institutional ecosystem of the world’s universities.

The firms associated with rankings have normalized the temporal cycle of rankings despite this being an illogical exercise (unless you are interested in selling advertising space in a magazine and on a website).  As Alex Usher pointed out earlier today in ‘The Paradox of University Rankings‘ (and I quote in full):

By the time you read this, the Times Higher Education’s annual Reputation Rankings will be out, and will be the subject of much discussion on Twitter and the Interwebs and such.  Much as I enjoy most of what Phil Baty and the THE do, I find the hype around these rankings pretty tedious.

Though they are not an unalloyed good, rankings have their benefits.  They allow people to compare the inputs, outputs, and (if you’re lucky) processes and outcomes at various institutions.  Really good rankings – such as, for instance, the ones put out by CHE in Germany – even disaggregate data down to the departmental level so you can make actual apples-to-apples  comparisons by institution.

But to the extent that rankings are capturing “real” phenomena, is it realistic to think that they change every year?  Take the Academic Ranking of World Universities (ARWU), produced annually by Shanghai Jiao Tong University (full disclosure: I sit on the ARWU’s advisory board).   Those rankings, which eschew any kind of reputational surveys, and look purely at various scholarly outputs and prizes, barely move at all.  If memory serves, in the ten years since it launched, the top 50 has only had 52 institutions, and movement within the 50 has been minimal.  This is about right: changes in relative position among truly elite universities can take decades, if not centuries.

On the other hand, if you look at the Times World Reputation Rankings (found here), you’ll see that, in fact, only the position of the top 6 or so is genuinely secure.  Below about tenth position, everyone else is packed so closely together that changes in rank order are basically guaranteed, especially if the geographic origin of the survey sample were to change somewhat.  How, for instance, did UCLA move from 12th in the world to 9th overall in the THE rankings between 2011 and 2012 at the exact moment the California legislature was slashing its budget to ribbons?  Was it because of extraordinary new efforts by its faculty, or was it just a quirk of the survey sample?  And if it’s the latter, why should anyone pay attention to this ranking?

This is the paradox of rankings: the more important the thing you’re measuring, the less useful it is to measure it on an annual basis.  A reputation ranking done every five years might, over time, track some significant and meaningful changes in the global academic pecking order.  In an annual ranking, however, most changes are going to be the result of very small fluctuations or methodological quirks.  News coverage driven by those kinds of things is going to be inherently trivial.

Top100WUR2013

The real issues to ponder are not relative placement in the ranking and how the position of universities has changed, but instead why this ranking was created in the first place and whose interests it serves.

Kris Olds

The Business Side of World University Rankings

Over the last two years I’ve made the point numerous times here that world university rankings have become normalized on an annual cycle, and function as data acquisition mechanisms to drill deep into universities but in a way that encourages (seduces?) universities to provide the data for free. In reality, the data is provided at a cost given that the staff time allocated to produce the data needs to be paid for, and allocating staff time this way generates opportunity costs.

See below for the latest indicator of the business side of world university rankings. Interestingly today’s press release from Thomson Reuters (reprinted in full) makes no mention of world university rankings, nor Times Higher Education, the media outlet owned by TSL Education, which was itself acquired by Charterhouse Capital Partners in 2007. Recall that it was that Times Higher Education began working with Thomson Reuters in 2010.

The Institutional Profiles™ that are being marketed here derive data from “a combination of citation metrics from Web of KnowledgeSM, biographical information provided by institutions, and reputational data collected by Thomson Reuters Academic Reputation Survey,” all of which (apart form the citation metrics) come to the firm via the ‘Times Higher Education World University Rankings (powered by Thomson Reuters).’

Of course there is absolutely nothing wrong with providing services (for a charge) to enhance the management of universities, but would most universities (and their funding agencies) agree, from the start, to the establishment of a relationship where all data is provided for free to a centralized private authority headquartered in the US and UK, and then have this data both managed and monetized by the private authority? I’m not so sure.

This is arguably another case of universities thinking for themselves and not looking at the bigger picture. We have a nearly complete absence of collective action on this kind of developmental dynamic; one worthy of greater attention, debate, and oversight if not formal governance.

Kris Olds

<><><><>

12 Apr 2012

Thomson Reuters Improves Measurement of Universities’ Performance with New Data on Faculty Size, Reputation, Funding and Citation Measures

Comprehensive data now available in Institutional Profiles for universities such as Princeton, McGill, Nanyang Technological, University of Hong Kong and others

Philadelphia, PA, April 12, 2012 – The Intellectual Property & Science business of Thomson Reuters today announced the availability of 138 percent more performance indicators and nearly 20 percent more university data within Institutional Profiles™, the company’s online resource covering more than 500 of the world’s leading academic research institutions. This new data enables administrators and policy makers to reliably measure their institution’s performance and make international comparisons.

Using a combination of citation metrics from Web of KnowledgeSM, biographical information provided by institutions, and reputational data collected by Thomson Reuters Academic Reputation Survey, Institutional Profiles provides details on faculty size, student body, reputation, funding, and publication and citation data.

Two new performance indicators were also added to Institutional Profiles: International Diversity and Teaching Performance. These measure the global composition of staff and students, international co-authorship, and education input/output metrics, such as the ratio of students enrolled to degrees awarded in the same area. The indicators now cover 100 different areas, ensuring faculty and administrators have the most complete institutional data possible.

All of the data included in the tool has been vetted and normalized for accuracy. The latest update also includes several enhancements to existing performance indicators, such as Normalized Citation Impact. This allows for equally weighted comparisons between subject groups that have varying levels of citations.

“Institutional Profiles continues to provide answers to the questions that keep administrators up at night: ‘Beyond citation impact or mission statement, which institutions are the best collaboration partners for us to pursue? How can I understand the indicators and data that inform global rankings?’,” said Keith MacGregor, executive vice president at Thomson Reuters. “With this update, the tool provides the resources to reliably measure and compare academic and research performance in new and more complete ways, empowering strategic decision-making based on each institution’s unique needs.”

Institutional Profiles, a module within the InCites™ platform, is part of the research analytics suite of solutions provided by Thomson Reuters that supports strategic decision making and the evaluation and management of research. In addition to InCites, this suite of solutions includes consulting services, custom studies and reports, and Research in View™.

For more information, go to:
http://researchanalytics.thomsonreuters.com/institutionalprofiles/

About Thomson Reuters
Thomson Reuters is the world’s leading source of intelligent information for businesses and professionals. We combine industry expertise with innovative technology to deliver critical information to leading decision makers in the financial and risk, legal, tax and accounting, intellectual property and science and media markets, powered by the world’s most trusted news organization. With headquarters in New York and major operations in London and Eagan, Minnesota, Thomson Reuters employs approximately 60,000 people and operates in over 100 countries. For more information, go to http://www.thomsonreuters.com.

Contacts

Alyssa Velekei
Public Relations Specialist
Tel: +1 215 823 1894