Putting facts in the way of a good story

Or: 100 cod in the North Sea? I’ve just had a nice weekend almost completely away from screens of all kinds, and was starting to think ‘perhaps it’s time to curtail some of my online activity <cough> Twitter <cough>’. But a quick scan down my timeline on a Monday morning couldn’t hurt could it?

Best laid plans and all that… One of the first things I saw was this, from @rivercottage: Just 100 cod left in North Sea. Now I’m a big fan of Hugh F-W, I own (too?) many of his cookbooks, have eaten in his canteen, and admire his campaigning work. BUT: Just 100 cod left in North Sea? Shurly shome mishtake?

I kept scanning, and the same figure kept appearing: 100 cod in the North Sea. All of these tweets linked to a Daily Telegraph story which I duly looked up when I got to work and there it was, under the headline “Just 100 cod left in North Sea” and the only marginally less ludicrous subheading “Overfishing has left fewer than 100 adult cod in the North Sea, it was reported”: a summary of a recent Cefas survey of catches at European ports. (To be fair to the Telegraph, an almost identical piece appeared in (hold your nose) the Mail, although the headline there made the made the ‘100 adult cod’ claim more clear.) Now I know people at Cefas, I trust them to do good surveys (although there’s no link to the actual report in either the Telegraph or the Mail piece, and I can't find it on the Cefas website), so could this 100 cod figure possibly be true?

Short answer: No. Of course it couldn’t.

What the survey found was that last year, “fishermen did not catch a single cod over the age of 13”, and Chris Darby of Cefas is quoted as saying that this suggests “there are fewer than 100 such fish in the whole North Sea”. This rather uncontroversial statement from Darby is the source of the headline and thus, the Twitter meme. Anyone reading the article, though, will see that the survey did find 191 million 1 year old cod, 18 million 3 year olds, and 65,300 tons of cod aged 3 or more years. So "Just 100 cod left in North Sea" is, as @BobOHara put it, an impressively wrong headline.

But what about the subheading about adult cod? Do cod mature as late as 13? No. From Fishbase figures, North Sea cod mature at 3-4 years. So a good proportion of those 18 million 3 year olds, and most of the 65,300 tons of 3+ y fish, will be adults. Subheading debunked too then.

As for the wider issue of overfishing and the state of the cod stock in the North sea, there is some information in the article, namely that the 65,300 tons of 3+ fish is a reduction from the 276,000 in 1971, “a peak year for cod population”, although no indication of whether the 2011 figure is any kind of historic low. But the number of 13+ year old fish is presented in glorious isolation: although no fish of that age were caught in 2011, I have no idea from this article when last such a fish was caught. I suspect some time ago, for a couple of reasons.

First, although the article quotes a maximum age for cod of 25y, which is indeed the figure given in Fishbase (taken, if you follow through to the references, from a 1974 Collins Guide to the Sea Fishes of Britain and North Western Europe book), looking at stock-specific data, again from Fishbase, gives one figure for North Sea cod: 11y (and this from a study published in 1959).

Second, an anecdote: way back before I started my PhD I worked for a while at Cefas. I shared an office with a guy who was interested in the age structure of cod stocks, and who used the annual growth rings on cod otoliths (inner ear bones) to get a handle on this. While I was there, someone sent in an otolith with 7 annual growth rings taken from a cod caught in the Irish Sea (as I recall; so a different stock, but still…). He was over the moon, because cod as old as 7 were so scarce in his dataset. That was 15 years ago.

Even if we accept that 13 is an attainable age for North Sea cod, then, it’s certainly a ripe old age, so saying from these data that there are 100 adult cod left in the North Sea is like saying there are 12,640 adults living in the UK based on the number of centurions.

I don’t want to downplay the perilous state of many fish stocks, nor the fact that overfishing is primarily what’s led them there. Another statistic in the Telegraph piece is sobering: the Cefas team estimated a 2011 population of 600 cod aged 12-13, of which 200 were caught. This tallies with another personal anecdote, this time from my undergraduate project working on data obtained from tagging North Sea plaice. These tags were returned to Cefas by fishermen when they caught a tagged plaice. We reckoned on a 30% return rate within a year. So: chuck a commercially important adult fish into the North Sea, and it has about a 1/3 chance of being caught within a year. That’s pretty intensive fishing. Likewise, there’s a load of good work – much of it by Cefas scientists – showing for instance how the biomass of big fish in the North Sea is massively reduced compared to predictions of the situation with no fishing.

Given this background, isn’t it perhaps a good thing to highlight fisheries in crisis? A few people I interacted with on Twitter over this seemed to think so, for example @oceanCO2 said title is certainly misleading but the article itself has context for the numbers. As you say ‘100 cod left’ does catch the eye! and @S_Danielsson wrote situation bad even if tweets are a bit oversimplified, 93% of NS cod are caught before reproductive age. Here and elsewhere both have made reasonable points, but I cannot agree that using misleading numbers to make a point is ever justified. In this specific case, just look at the responses of some in the fishing industry, for instance @LapwingPD972 who’s been fishing the North Sea for 33 years: Who are these people that say there is only 100 mature Cod left in the north sea,OMG invitation to anyone,come away on my boat to see fish to which @AjaxAH32 (skipper with 30y experience) replied I thought it was 1^st of April Brian.

This matters because the relationship between fishers, scientists and conservationists in the UK is often fraught; but listening to various talks at the Oceans of Potential meeting in Plymouth last week I really felt that real progress has been made, with a focus on the business case for sustainability which everyone can buy into. Silly headlines – seemingly designed to be picked up and circulated without thought around Twitter – set back these efforts and risk returning us to a harmful new cod war of words. More generally, you really don’t strengthen your case by simply parroting obviously nonsensical numbers.

------------------------------------------------------

Postscript (19/09/12)

Since I wrote the above, Defra have released an official myth bust statement, worth quoting in full here:

The myth: An article in The Sunday Times claimed that ‘fewer than 100 mature cod are left in the North Sea’.

The truth: This is completely wrong, in fact we know there to be around 21 million mature cod in the north sea. Cod start to mature from six months old and are fully mature at age six.  There are a small number of cod over the age of 12 years old which has always been the case in the North Sea even when fished at lower levels in the 1950s and 1960s. Cod older than 15 have never been recorded in the North Sea.

So - the Sunday Times, Telegraph and Mail were out by a cool factor of 210,000…

Judgement vs Accountability

What with one thing and another, it’s taken me a while to sit down to write this, and the event that triggered it - the furore over this year’s GCSE results – already seems like old news. But it got me thinking more broadly, and I hope those thoughts are still relevant these several news cycles later. So: on a lively Newsnight debate about the GCSEs, someone suggested that exams at 16 were unnecessary, and said something like ‘teachers are professionals, they can use their professional judgement to assess their students at that age without the need for external examining bodies’. I don’t have particularly strong views on this particular topic (although I’m happy my results did not depend on my chemistry teacher who once graded – apparently without noticing – a pile of French essays that we handed in for a joke) but the underlying issue of the (not always complementary) relationship between professional judgment and rigid accountability seems to me highly relevant to academia, in several ways.

Most obviously, of course, in teaching. In general the days of simply sticking a grade on a paper with no justification have passed, and with them rumours of dubious practices (the famous ‘chuck a pile of essays down stairs and rank them by where they fall’). This is surely a good thing, and is the least that students should expect now that they have a more personal sense of what their education is costing.

But, partly as a consequence of increasingly assertive students, I’m getting more and more questions about the marks I give for undergraduate essays. Not disputing the marks, but asking what they would have needed to do to get that 72 rather than 68, or 75 rather than 72… Now I do try to set out an explicit marking scheme, and to provide ample feedback, but sometimes it’s tempting just to say ‘I just thought it was on solid 2:1’; or ‘What do you need to do to get 80? Just write a fantastic essay!’; or ‘What makes a great essay? Not sure but I know one when I read one…’ The strict accountability introduced by rigid marking schemes can be your friend when you have 150 exam scripts to process, but when you’re marking half a dozen tutorial essays it can get in the way of a more subjective judgement.

Something similar happens in the peer review process for both papers and grant proposals. For papers, especially when acting as an editor and rejecting work without sending it for full review, I frequently justify this course of action using bits copied and pasted from the journal’s aims and scope to defend my decision in an accountable fashion. But usually what I’m really saying (except on those occasions when I’m saying: 'this is crap') is, ‘Nah, sorry, didn’t really float my boat’. Or to couch the same sentiment in more formal language, ‘In my professional judgement, I don’t think this work merits publication in journal X’. Full stop. I think this has some similarities to a GP’s diagnosis – one hopes that it is founded in a good understanding of the subject, but one need not document every single step ruling out all other possible diagnoses.

Finally, in reviewing grant proposals you can be forced to be more prescriptive than perhaps you would like. Certain boxes must be filled in, for instance on what you perceive to be the main strengths and weaknesses of the proposed work, which forces you to break down the proposal in a way which may not match your gut feeling (to use another term for professional judgement). So something that you thought was eminently fundable is scuppered because you happened to list more in the weaknesses column than in the strengths – regardless of your overall impression.

Accountability is of course absolutely essential to the process of science – the audit trail which leads from raw data to published results is arguably more important than the results themselves. But in the assessment of its worth? I’m not so sure.

My own personal Impact Factor

The editor of a well-respected ecological journal told me recently, “I am… very down on analyses that use citation or bibliographic databases as sources of data; I'm actually quite concerned that the statistical rigor most people learn in the context of analysing biological data is thrown out completely in an attempt to show usage of a particular term has been increasing in the literature!” I think he has a point, and in fact I feel the same about much that I read on bibliometrics more generally: there’s some really insightful, thoughtful and well-reasoned text, but as soon as people attempt to bring some data to the party all usual standards of analytical excellence go out the window. I see absolutely no reason to buck that trend here.

So…

The old chestnut of Journal Impact Factors has been doing the rounds again, thanks mainly to a nice post from Stephen Curry which has elicited a load of responses in the comments and on Twitter. To simplify massively: everyone agrees that IFs are a terrible way to assess individual papers (and by inference, researchers), but there’s less agreement on whether they tell you anything useful when comparing journals within a field. Go read Stephen’s post if you want the full debate.

But what’s sparked my post was a response from Peter Coles (@telescoper), called The Impact X-Factor, which proposed an idea I’d had a while back about judging papers against the IF of the journal in which they’re published. Are your papers holding up or weighing down your favourite journal? Let’s be clear from the outset: I don’t think this tells us anything especially interesting, but that needn’t put us off. So I have bitten the bullet, and present to you here my own personal impact factor. (The fact I come out of it OK in no way influenced my decision to go public.)

The IF of a journal, remember, is simply the mean number of citations to papers published in that journal over a two-year period (various fudgings and complications make it rather more opaque than that, but that’s it in essence). So for each of my papers (fortunately there aren’t too many) I’ve simply obtained (from my google scholar page, as it’s more open that ISI) the number of citations they accrued in the two years after publication. I’ve then compared this to the relevant journal IF for that period, or as close as I could get. Here are the results:

OK, bit of explanation. This simply plots the number of citations my papers got in the two years post-publication, against the relevant IF of the journal in which they were published. (The red points are papers published in the last year or so, and I’ve down-weighted IF to take account of this; I’ve excluded a couple of very recently-published papers.) The dashed line is the 1:1 line, so if my papers exactly matched the journal mean they would all fall on this line. Anything above the line is good for me, anything below it bad – the histogram in the bottom right shows the distribution of differences of my papers from this line.

I’ve fitted a simple Poisson model to the points, with and without the outlier in the top right – neither does an especially good job of explaining citations to my work, so we might as well take a mean, giving me my own personal IF of around 6.

As my editor friend suggested, there’s a whole lot wrong with this analysis. For instance, I haven’t taken account of year of publication, or any other potential contributing factors (coauthors, publicity, etc. etc.). Another obvious caveat is the lack of papers in journals with IF > 10 (I can assure you that this has not been a deliberate strategy). But back in the peloton of points which represent the ecology journals in which I’ve published most regularly, I’m reasonably confident in stating that citations to my work are unrelated to journal IF. Gratifyingly too, the papers that I rate as my best typically fall above the 1:1 line.

So there we have it. My own personal impact factor.