Bibliometrics, global rankings, and transparency

Why do we care so much about the actual and potential uses of bibliometrics (“the generic term for data about publications,” according to the OECD), and world university ranking methodologies, but care so little about the private sector firms, and their inter-firm relations, that drive the bibliometrics/global rankings agenda forward?

This question came to mind when I was reading the 17 June 2010 issue of Nature magazine, which includes a detailed assessment of various aspects of bibliometrics, including the value of “science metrics” to assess aspects of the impact of research output (e.g., publications) as well as “individual scientific achievement”.

The Nature special issue, especially Richard Van Noorden’s survey on the “rapidly evolving ecosystem” of [biblio]metrics, is well worth a read. Even though bibliometrics can be a problematic and fraught dimension of academic life, they are rapidly becoming an accepted dimension of the governance (broadly defined) of higher education and research. Bibliometrics are generating a diverse and increasingly deep impact regarding the governance process at a range of scales, from the individual (a key focus of the Nature special issue) through to the unit/department, the university, the discipline/field, the national, the regional, and the global.

Now while the development process of this “eco-system” is rapidly changing, and a plethora of innovations are occurring regarding how different disciplines/fields should or should not utilize bibliometrics to better understand the nature and impact of knowledge production and dissemination, it is interesting to stand back and think about the non-state actors producing, for profit, this form of technology that meshes remarkably well with our contemporary audit culture.

In today’s entry, I’ve got two main points to make, before concluding with some questions to consider.

First, it seems to me that there is a disproportionate amount of research being conducted on the uses and abuses of metrics in contrast to research on who the producers of these metrics are, how these firms and their inter-firm relations operate, and how they attempt to influence the nature of academic practice around the world.

Now, I am not seeking to imply that firms such as Elsevier (producer of Scopus), Thomson Reuters (producer of the ISI Web of Knowledge), and Google (producer of Google Scholar), are necessarily generating negative impacts (see, for example, ‘Regional content expansion in Web of Science®: opening borders to exploration’, a good news news story from Thomson Reuters that we happily sought out), but I want to make the point that there is a glaring disjuncture between the volume of research conducted on bibliometrics versus research on these firms (the bibliometricians), and how these technologies are brought to life and to market. For example, a search of Thomson Reuter’s ISI Web of Knowledge for terms like Scopus, Thomson Reuters, Web of Science and bibliometrics generates a nearly endless list of articles comparing the main data bases, the innovations associated with them, and so on, but amazingly little research on Elsevier or Thomson Reuters (i.e. the firms).  From thick to thin, indeed, and somewhat analogous to the lack of substantial research available on ratings agencies such as Moody’s or Standard and Poor’s.

Second, and on a related note, the role of firms such as Elsevier and Thomson Reuters, not to mention QS Quacquarelli Symonds Ltd, and TSL Education Ltd, in fueling the global rankings phenomenon has received remarkably little attention in contrast to vigorous debates about methodologies. For example, the four main global ranking schemes, past and present:

all draw from the databases provided by Thomson Reuters and Elsevier.

One of the interesting aspects of the involvement of these firms with the rankings phenomenon is that they have helped to create a normalized expectation that rankings happen once per year, even though there is no clear (and certainly not stated) logic for such a frequency. Why not every 3-4 years, for example, perhaps in alignment with the World Cup or the Olympics? I can understand why rankings have to happen more frequently than the US’ long-delayed National Research Council (NRC) scheme, and they certainly need to happen more frequently than the years France wins the World Cup championship title (sorry…) but why rank every single year?

But, let’s think about this issue with the firms in mind versus the pros and cons of the methodologies in mind.

From a firm perspective, the annual cycle arguably needs to become normalized for it is a mechanism to extract freely provided data out of universities. This data is clearly used to rank but is also used to feed into the development of ancillary services and benchmarking capabilities that can be sold back to universities, funding councils, foundations, regional organizations (e.g., the European Commission which is intensely involved in benchmarking and now bankrolling a European ranking scheme), and the like.

QS Quacquarelli Symonds Ltd, for example, was marketing such services (see an extract, above, from a brochure) at their stand at the recent NAFSA conference in Kansas City, while Thomson Reuters has been busy developing what they deem the Global Institutional Profiles Project. This latter project is being spearheaded by Jonathon Adams, a former Leeds University staff member who established a private firm (Evidence Ltd) in the early 1990s that rode the UK’s Research Assessment Excellence (RAE) and European ERA waves before being acquired by Thomson Reuters in January 2009.

Sophisticated on-line data entry portals (see a screen grab of one above) are also being created. These portals build a free-flow (at least one one-way) pipeline between the administrative offices of hundreds of universities around the world and the firms doing the ranking.

Data demands are becoming very resource consuming for universities. For example, the QS template currently being dealt with by universities around the world shows 14 main categories with sub-categories for each: all together there are 60 data fields, of which 10 are critical to the QS ranking exercise, to be launched in October 2010. Path dependency dynamics clearly exist for once the pipelines are laid the complexity of data requests can be gradually ramped up.

A key objective, then, seems to involve using annual global rankings to update fee-generating databases, not to mention boost intra-firm knowledge bases and capabilities (for consultancies), all operational at the global scale.

In closing, is the posited disjuncture between research on bibliometrics vs research on bibliometricians and the information service firms these units are embedded within worth noting and doing something about?

Second, what is the rationale for annual rankings versus a more measured rankings window, in a temporal sense? Indeed why not synchronize all global rankings to specific years (e.g., 2010, 2014, 2018) so as to reduce strains on universities vis a vis the provision of data, and enable timely comparisons between competing schemes. A more measured pace would arguably reflect the actual pace of change within our higher education institutions versus the needs of these private firms.

And third, are firms like Thomson Reuters and Elsevier, as well as their partners (esp., QS Quacquarelli Symonds Ltd and TSL Education Ltd), being as transparent as they should be about the nature of their operations? Perhaps it would be useful to have accessible disclosures/discussions about:

  • What happens with all of the data that universities freely provide?
  • What is stipulated in the contracts between teams of rankers (e.g., Times Higher Education and Thomson Reuters)?
  • What rights do universities have regarding the open examination and use of all of the data and associated analyses created on the basis of the data universities originally provided?
  • Who should be governing, or at least observing, the relationship between these firms and the world’s universities? Is this relationship best continued on a bilateral firm to university basis? Or is the current approach inadequate? If it is perceived to be inadequate, should other types of actors be brought into the picture at the national scale (e.g., the US Department of Education or national associations of universities), the regional-scale (e.g., the European University Association), and/or the global scale (e.g., the International Association of Universities)?

In short, is it not time that the transparency agenda the world’s universities are being subjected to also be applied to the private sector firms that are driving the bibliometrics/global rankings agenda forward?

Kris Olds

From rhetoric to reality: unpacking the numbers and practices of global higher ed

ihepnov2009Numbers, partnerships, linkages, and collaboration: some key terms that seem to be bubbling up all over the place right now.

On the numbers front, the ever active Cliff Adelman released, via the Institute for Higher Education Policy (IHEP), a new report titled The Spaces Between Numbers: Getting International Data on Higher Education Straight (November 2009). As the IHEP press release notes:

The research report, The Spaces Between Numbers: Getting International Data on Higher Education Straight, reveals that U.S. graduation rates remain comparable to those of other developed countries despite news stories about our nation losing its global competitiveness because of slipping college graduation rates. The only major difference—the data most commonly highlighted, but rarely understood—is the categorization of graduation rate data. The United States measures its attainment rates by “institution” while other developed nations measure their graduation rates by “system.”

The main target audience of this new report seems to be the OECD, though we (as users) of international higher ed data can all benefit from a good dig through the report. Adelman’s core objective is facilitating the creation of a new generation of indicators, indicators that are a lot more meaningful and policy-relevant than those that currently exist.

Second, Universities UK (UUK) released a data-laden report titled The impact of universities on the UK economy. As the press release notes:

Universities in the UK now generate £59 billion for the UK economy putting the higher education sector ahead of the agricultural, advertising, pharmaceutical and postal industries, according to new figures published today.

This is the key finding of Universities UK’s latest UK-wide study of the impact of the higher education sector on the UK economy. The report – produced for Universities UK by the University of Strathclyde – updates earlier studies published in 1997, 2002 and 2006 and confirms the growing economic importance of the sector.

The study found that, in 2007/08:

  • The higher education sector spent some £19.5 billion on goods and services produced in the UK.
  • Through both direct and secondary or multiplier effects this generated over £59 billion of output and over 668,500 full time equivalent jobs throughout the economy. The equivalent figure four years ago was nearly £45 billion (25% increase).
  • The total revenue earned by universities amounted to £23.4 billion (compared with £16.87 billion in 2003/04).
  • Gross export earnings for the higher education sector were estimated to be over £5.3 billion.
  • The personal off-campus expenditure of international students and visitors amounted to £2.3 billion.

Professor Steve Smith, President of Universities UK, said: “These figures show that the higher education sector is one of the UK’s most valuable industries. Our universities are unquestionably an outstanding success story for the economy.

See pp 16-17 regarding a brief discussion of the impact of international student flows into the UK system.

These two reports are interesting examples of contributions to the debate about the meaning and significance of higher education vis a vis relative growth and decline at a global scale, and the value of a key (ostensibly under-recognized) sector of the national (in this case UK) economy.

And third, numbers, viewed from the perspectives of pattern and trend identification, were amply evident in a new Thomson Reuters’ report (CHINA: Research and Collaboration in the New Geography of Science) co-authored by the data base crunchers from Evidence Ltd., a Leeds-based firm and recent Thomson Reuters acquisition. One valuable aspect of this report is that it unpacks the broad trends, and flags key disciplinary and institutional geographies to China’s new geography of science. As someone who worked at the National University of Singapore (NUS) for four years, I can understand why NUS is now China’s No.1 institutional collaborator (see p. 9), though the why issues are not discussed in this type of broad mapping cum PR report for Evidence & Thomson Reuters.

Table4

Shifting tack, two new releases about international double and joint degrees — one (The Graduate International Collaborations Project: A North American Perspective on Joint and Dual Degree Programs) by the North American Council of Graduate Schools (CGS), and one (Joint and Double Degree Programs: An Emerging Model for Transatlantic Exchange) by the International Institute for Education (IIE) and the Freie Universität Berlin — remind us of the emerging desire to craft more focused, intense and ‘deep’ relations between universities versus the current approach which amounts to the promiscuous acquisition of hundreds if not thousands of memoranda of understanding (MoUs).

IIEFUBcoverThe IIE/Freie Universität Berlin book (link here for the table of contents) addresses various aspects of this development process:

The book seeks to provide practical recommendations on key challenges, such as communications, sustainability, curriculum design, and student recruitment. Articles are divided into six thematic sections that assess the development of collaborative degree programs from beginning to end. While the first two sections focus on the theories underpinning transatlantic degree programs and how to secure institutional support and buy-in, the third and fourth sections present perspectives on the beginning stages of a joint or double degree program and the issue of program sustainability. The last two sections focus on profiles of specific transatlantic degree programs and lessons learned from joint and double degree programs in the European context.

It is clear that international joint and double degrees are becoming a genuine phenomenon; so much so that key institutions including the IIE, the CGS, and the EU are all paying close attention to the degrees’ uses, abuses, and efficacy. Thus we should view this new book as an attempt to both promote, but in a manner that examines the many forces that shape the collaborative process across space and between institutions. International partnerships are not simple to create, yet they are being demanded by more and more stakeholders.  Why?  Dissatisfaction that the rhetoric of ‘internationalization’ does not match up to the reality, and there is a ‘deliverables’ problem.

Indeed, we hosted some senior Chinese university officials here in Madison several months ago and they used the term “ghost MoUs”, reflecting their dissatisfaction with filling filing cabinet after filing cabinet with signed MoUs that lead to absolutely nothing. In contrast, engagement via joint and double degrees, for example, or other forms of partnership (e.g., see International partnerships: a legal guide for universities), cannot help but deepen the level of connection between institutions of higher education on a number of levels. It is easy to ignore a MoU, but not so easy to ignore a bilateral scheme with clearly defined deliverables, a timetable for assessment, and a budget.

AlQudsBrandeisThe value of tangible forms of international collaboration was certainly on view when I visited Brandeis University earlier this week.  Brandeis’ partnership with Al-Quds University (in Jerusalem) links “an Arab institution in Jerusalem and a Jewish-sponsored institution in the United States in an exchange designed to foster cultural understanding and provide educational opportunities for students, faculty and staff.”  Projects undertaken via the partnership have included administrative exchanges, academic exchanges, teaching and learning projects, and partnership documentation (an important but often forgotten about activity). The level of commitment to the partnership at Brandeis was genuinely impressive.

In the end, as debates about numbers, rankings, partnerships, MoUs — internationalization more generally — show us, it is only when we start grinding through the details and ‘working at the coal face’ (like Brandeis and Al-Quds seem to be doing), though in a strategic way, can we really shift from rhetoric to reality.

Kris Olds