In Defence of Bibliometrics

I spend a fair part of my professional life trying to avoid upsetting friends and colleagues. No really, I do, honest. So, I had to think hard about a post that is bound to annoy and offend. It needs to be said however, so here goes. I think that bibliometrics are a valuable tool in assessing research and identifying ability. This is a position that has become, surprisingly, less acceptable over the last couple of years (a trend I intend to analyse). I plan therefore to mount a defence of bibliometrics and their use.

In order not to appear naive, and to tempt those who disagree with my view to read further, let me first review the main shortcomings of bibliometrics.

It is not sensible to try to capture a complex matter such as the research quality of an individual or an institution in a single number. The only way to really assess a body of work is to read it and form an independent academic judgment.

Citations, the foundation of most bibliometrics, are highly questionable as an indicator of research influence. Some citations may be negative. Reviews and synoptic works gather more citations. Citations accrue disproportionately to early speculative work. Recent work, with a shorter period of exposure is necessarily less well cited. Typical citation counts exclude important types of intellectual output. Citation counts tend to ignore the number and differential contribution of co-authors.

Citations depend upon a research community. Citation rates differ substantially from subject to subject and even within subject from specialisation to specialisation. Small closed communities can cite each others work more highly making comparison across communities difficult. Even self-citation cannot be reliably excluded.

The principal tools for gathering citation data are inaccurate. They often exclude important research venues and include venues of lesser quality. There are systematic biases with certain subjects receiving much better coverage than others. Small technical matters such as special characters and layout as well as the precise parameters of a search can have significant distorting effects. The wrong papers are associated with the wrong authors. Citations can get 'lost' as authors move institution. A single institution or unit may appear in many guises, at different addresses, which may be difficult to reconcile.

There are no agreed 'figures of merit'. Total number of published papers favours the glib. Total number of citations favours the established researcher. Highly cited papers undervalues the cumulative effect of a body of work. Journal impact factors are secondary measures indicating merit, if at all, indirectly. The ubiquitous h-index (Hirsch index: "a scholar with an index of h has published h papers each of which has been cited by others at least h times") is both unintuitive and is bounded by the number of publications thereby disadvantaging early career researchers.

The use of bibliometrics and particularly the reliance on crude measures drives behaviour in unhealthy ways. It also feeds a collective obsession with simplistic league tables which does not serve scholarship.

Many of these criticisms are fair but must be viewed in a balanced way. Firstly, summative research assessment is necessary and inevitable. Research managers, funders and others must conduct rapid assays. When faced, for example, with many hundreds of competing CVs, each with lengthy publication lists and the support of distinguished and enthusiastic referees, a shortcut method for determining who is deserving of the closest scrutiny is required. Bibliometrics may be far from perfect but they are better than the alternatives as a support for judgment.

In fact, even with inaccuracies, bibliometrics often reveal interesting truths. When I look at subject based citation league tables they generally reflect my gut sense of who the influential players in my field are. If I drill down to look at some of the more surprising inclusions I am similarly struck by how frequently they are 'undervalued' contributors. When I search for what I regard as the 'unexpected' omissions they are, more frequently than not, researchers whose reputation outstrips their achievements. In short, bibliometrics may not tell the whole truth but they tell some important truths. These truths get more compelling the coarser the grain of the analysis.

It is true that all bibliometrics require careful interpretation. When applied blindly or without subject knowledge they can mislead, but this is self evidently an unfair criticism. Nobody is arguing for a blind or ignorant application of bibliometrics. When used in support of an analysis by reasonable subject experts possessed of a sensitive appreciation of their shortcomings they can be powerful tools. No subject expert would for example expect a heavily theoretical paper, replete with complex proofs, to garner as many citations as for example a paper that provides a reference implementation of a widely applicable experimental method. This same expert should be capable therefore of adjusting their expectations with respect to bibliometrics that attempt to reflect the impact of such work.

Similarly no reasonable expert would attempt to use a single figure of merit and expect it to suffice. A basket of metrics can however be revealing, for example of low publication rates with high citations and vice versa, which are worthy of further analysis.

Bibliometric tools are improving day by day, with better coverage, fewer glitches and vastly improved usability; more papers are accessible by web crawling; and, academics are more aware of, and sensitive to, the nuances of bibliometrics. Why, in this case do I detect a rising tide of hostility to their use? Firstly, I suspect that they matter more, and this is of concern to those with a stake in the purely 'reputational' system. I have little sympathy with this. Secondly, the fear of a formulaic approach as part, for example, of the UK Research Excellence Framework, has led to a reaction that has overspilled reasonable bounds and has become an almost phobic reaction to bibliometrics. In fact, I still believe that bibliometrics, in combination with other statistical and financial data and with some light touch moderation, might have done a perfectly adequate job of a necessarily rough allocation of research monies across UK research universities, but that debate is long lost. I think in this regard we need to 'get a grip' and adopt an adult approach to using bibliometrics.

Some interesting new trends are emerging with the growth of social network metric aggregators such as peerindex and klout. New visualisations and approaches to data mining to acquire a handle on reputation and thus influence, and perhaps trustworthiness, seem set to emerge. It would be an irony to turn our back on bibliometrics at this point.