Is writing a high impact paper the same as doing good science?

We are under starters’ orders, then, and beginning to feel the pressure as expectation builds for us to achieve success and acclaim. No, not the 2012 Olympics; rather, everything in UK academic research is gearing up for the 2014 Research Excellence Framework, the mechanism by which the quality of our research over the last few years will be judged, and decisions made on how government money should be shared between institutes in the future.

Much of the focus of this exercise, certainly for those of us in scientific disciplines, is on the production of so-called ‘high impact’ papers. Each of us must submit for assessment our four ‘best’ papers of the last 5 years or so; a panel will then grade each on a scale from 1 to 4 (actually, 1* to 4*, but the stars seem superfluous to me when there is no option of an unstarred number). A 1* paper is ‘quality that is recognised nationally in terms of originality, significance and rigour’; 2* replaces ‘nationally’ with ‘internationally’; 3* is ‘internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence’; and 4* is ‘world-leading in terms of originality, significance and rigour’. (Work can also be unclassified if it’s crap, but it seems unlikely that many submissions will fall into that category.)

Of course, the £610K / $1M question is, how do you judge the quality of a piece of published research? The REF has devoted thousands of man hours and reams of text to this question, which I won’t rehash here. Suffice it to say that the most simplistic interpretation, which happens to be one that many institutions are assuming will be the major criterion applied in the REF, is, depressingly, that the quality of work is accurately reflected by the impact factor of the journal in which it appears.

I have written before on how misguided that is, but, amazingly, it appears that my blog went unheeded. Again, I don’t really want to go over old ground again, simply to state that it is commonly assumed that a 4* paper means a paper in one of the weekly tabloids; nothing else will do.

Now, I have no intention of denigrating these publications, still less of biting the hand that hosts me – clearly, they are world-leading in their standards of originality, significance, and rigour. Rather, I wanted to point out what I see as a worrying trend, which is conducting research with the objective of ‘writing a Nature paper’.

Of course, every research project probably starts with the thought that, if all goes well, if the results are unusually stunning, unambiguous, and free of messy nuance; if the stars align just so, and the editor is having a good day when I submit; then maybe, just maybe, this might get into a top journal. But the focus is on doing the science as well as possible, and deciding on an outlet for it once the results are in.

Now, however, it seems that every workshop I am invited to has the objective (very likely a condition of funding the thing) of producing a ‘high impact paper’. In other words, the template for reporting the results is decided before any analysis is begun. Ambition is no bad thing, of course, and one might as well aim high. But, if you set by writing the headline, what happens if the story doesn’t quite pan out how you hoped?

The right option is to re-assess, and perhaps take a few more pages to thoroughly explore your more complex results for a good, but CV-wise non-stellar paper in a less high-profile journal. The temptation though, is to plough on regardless, to try to squeeze that pint into a quart pot; not, in almost all cases, to fabricate anything; but perhaps to emphasise here to draw attention away from here.

Of course, playing this game is no guarantee of success: we can usually rely on the review process to sift the wheat from the chaff, even in cases where the drive and strength of personality of the group leader has managed to bring the work to the submission stage. But the wider question is, Is this a good way to do science? And the corallary: What implications does this have for assessing research quality?

Can't scientists be intellectuals too?

Is it conceited of me to fancy myself as a bit of an intellectual? I have after all, like most practising scientists, obtained a doctorate in philosophy (and my PhD certainly included a good amount of ‘Ph’). Now I am paid, essentially, to sit and think, and to write about what I have thought. And I read pretty widely, and pretty critically, fact and fiction. If nothing else, the range of posts on this blog show that I’m certainly prepared (am perhaps too keen, if anything) to pontificate on subjects well outside my nominal area of expertise. I don’t think I am in any way unusual in this. Plenty of professional scientists would also count themselves as thinkers, perhaps even philosophers, and certainly intellectuals. Yet scientific voices remain very scarce in public intellectual debate in the UK. Sure, we’ll wheel out an ‘expert’ to comment on something on which they are, indeed, expert. But the thinkers considered suitable contributors to more general discussions on the human condition, public policy, and the like, remain overwhelmingly drawn from the arts, the humanities, and the social sciences.

This is all to give a little context, then, to my rather chippy reaction to finding out that the Arts & Humanities Research Council, together with the BBC, have identified 10 new public intellectuals through their New Generation Thinkers scheme. Said reaction being: ‘So ’intelligent discussion’ programmes on TV and radio will welcome yet more Oxbridge humanities graduates with a double first in self aggrandisement and a keen interest in the sounds of their own voices, yet no concept of the distinction between fact and opinion? Whoopie-doo.’

As it happens, this is unfair. Although at least 6 of the 10 have come through Oxbridge at some point (generally as undergrads), only one of them currently works at Cambridge, and none at Oxford. And they all look to be very high-achieving, worthy winners – several have already published well-received books, and all their ‘specialist subjects’ sound interesting, from Britain in the 1790s to the Rwandan Genocide.

But the broader point, and the source of my frustration, holds: why aren’t the scientific research councils doing something similar? Listen to what the scheme offers its winners, and turn green with envy:

BBC Radio 3 and the Arts and Humanities Research Council (AHRC) today announce the 10 winners of the inaugural New Generation Thinkers Scheme – the culmination of a nationwide search for the brightest academic minds with the potential to turn their ideas into fascinating broadcasts… The New Generation Thinkers for 2011 will now work closely with dedicated mentors from the production team of Radio 3’s arts and ideas programme Night Waves (Mondays to Thursdays 10-10.45pm). And each night from Tuesday 28 June, and for nine subsequent editions of Night Waves, a New Generation Thinker will talk about an idea inspired by their research.

Sounds pretty good, eh? And when you consider that it will surely be a fast track to regular appearances on the myriad cultural comment programmes on TV and radio, all in all a pretty good gig.

The science research councils seem to still be operating on assumption that what scientists really want is instruction from communications professionals, that we are all clamouring for the opportunity to be offered a walk-on part in Horizon, rather than actually to shape public intellectual discourse.

Even the Royal Society, who fund me, have betrayed something of this attitude in the new secondment opportunity that they have introduced for research fellows, which appears to be a kind of ‘advanced work experience’ at the BBC – getting to play with editing software and the like. It does look fun, I admit (providing you already live in London, as it comes with no living expenses; interestingly, 5/10 of the New Generation Thinkers are also London-based), but it does rather stand in contrast to the AHRC scheme: work experience, as opposed to a leg-up into public intellectual life.

I’m not suggesting that we scientists are offered anything on a plate, nor that we have any special privilege to contribute to public discourse. But rather, that public discourse would be enriched by more contributions of working scientists, applying their hard-won analytical and critical skills to fields outside their area of expertise. This would be both a complement and a balance to the views expressed (often in the absence of data) by the various think-tanks, pressure groups, and assembled ‘thinkers’ that tend to dominate discussions of social policy options.

Little bit of politics...

Yesterday in the UK, large numbers of public sector workers went on strike to protest about the changes that the government wants to make to their pensions. I was catching up with this by watching Newsnight, and a peculiar rhetoricical tactic became apparent from the government representative. Apparently unrelated to this, I have recently started what I hope will be a fruitful collaboration with a pure mathematician. In a meeting the other day, I was struck once again by the different prism through which mathematicians see the world, apparent through the questions that my colleague asks about ecological phenomena.

In the spirit of seeing things mathematically, I thought I would have a go at applying some basic set theory to the Newsnight debate. Well, that’s probably a little too grand – basically, I’ve produced a pair of Venn diagrams.

This first diagram represents government rhetoric: it is unfair for the generous public sector pensions to be funded by taxpayers (implicitly: public sector workers ⊄ taxpayers). And more specifically, those who suffer most during the strikes – especially those by teachers – are that (tautological, as I am now acutely aware) demographic most beloved of politicians of all stripes: ‘hard-working parents’ (i.e., teachers ⊄ hard-working parents).

In reality, of course all public sector workers are also taxpayers (in fact, about 20% of those working in the UK are public sector employees; PDF of the Office for National Statistics source for this here). And do you know what? Plenty of public sector employees are also parents. Some of these are even – shock! – teachers. And I don’t doubt that at least a few of these work hard. (NB – the incomplete union of hard-working parents with taxpayers is intentional, but more to capture the cash economy than the wealthy tax dodgers!)

I don’t pretend to understand pensions, I can barely muster the interest to open the annual statements I get regarding mine, and certainly I’m not in a position to offer any kind of analysis over whether we really are heading for a crisis that can only be averted by making us all work longer for a lower pension at the end. But, setting up false dichotomies to make political points ought not to pass without challenge.

And with that – I’m off on my hols!