Science and criticism, good and bad

It was always going to be tough for whoever took over weekly science column duties in the Guardian from Ben Goldacre. His Bad Science was entertaining about very important stuff – stats, data, the importance of evidence, and so on. So it’s perfectly possible that I’m judging Philip Ball’s first effort, which appeared last Saturday, too harshly. But, it put my back up from the very first sentence. What’s especially frustrating is that I think his aim, to be “like an arts critic, but for science”, is laudable. I’ve argued before for a more prominent place for science in public intellectual debate, and this kind of informed criticism would be an essential part of that. Unfortunately, over the course of this first column Ball has shown few signs that he has any of the attributes required to be such a critic. First, he seems not to know much about how scientists work. Forgive me if I quote in full the first paragraph to illustrate this point:

Scientists don’t like being criticised. Well, who does? But I don’t mean that they don’t like it when people say they are wrong, biased, self-serving or insular. I don’t like it when people say those things either, because in my experience scientists tend to be right, fair, generous and – well, OK, they could do with getting out more. But scientists don’t like being criticised in the proper sense of the word: in the way that books and plays and music are judged, for better or worse, by critics.

OK, so what’s wrong here? Lets ignore the tired stereotype about scientists needing to ‘get out more’ (unlike, say, anyone else who has a demanding and satisfying job with no set working hours? And especially poorly judged given the continuing revelations in the Leveson Inquiry about the less than social behaviour of some in Ball’s profession). No, let’s think instead about the nature of criticism in science.

I spend a good proportion of my working life either criticising the work of others, or responding to others’ criticisms of me. And I mean criticism in the sense – more or less – that Ball favours. That is, assessing the worth of science and placing it in its proper context. It is an essential part of the process of scientific publication, and – with the exception (on paper anyway) of a few journals like PLoS ONE – is always about much more than right and wrong. Constantly, we make judgements about the novelty, significance and interpretation of a set of results. I can’t think of many scientists who would seriously argue that “science is a question of fact – either it’s right or it isn’t”, as Ball charicatures us. Put simply, methods and results sections of papers would, we hope, fall into that kind of binary category; but Introductions and Discussions are pure interpretation, frequently subjective and designed to put a particular spin on the results to convince a journal editor that they fit within the journal’s stated aims.

We all know that, don’t we? And do we (i.e., all scientists) really have, as Ball claims, “a kneejerk aversion to any claim that science is shaped by culture”? Most of the scientists I know are a bit smarter than that. One example: Greg McInerney from Microsoft Research, who last week gave a talk in the Ecology & Environment Seminar Series that I run in Sheffield, spent some time discussing the sociology of the methodological choices that people make when modelling species distributions. To my knowledge, no-one in the audience thought this to be an especially unusual or outrageous departure. More generally, the science blogosphere is full of scientists grappling with such ideas.

One can’t help but suspect that Ball is not a great participant in the thriving culture of science online, and is building his straw men from (if anything) conversations with professors emeritus rather than with those actually doing and thinking about science. This comes across too in his claim that science journalism does have a pedagogical role in expalining science to the public “in language that scientists have forsaken” – again, discounting countless scientist-run outreach projects and terrific science communication (random example of excellence from Deep Sea News) in favour of a lazy stereotype about scientists using difficult language (which we do in our technical papers, of course, because well, they’re technical).

So, I am absolutely all in favour of a broader public debate about the social context of science. I think that would be a really positive step. But Ball’s column has failed to convince me that science journalists like himself are the people to steer this debate. Rather, science online has left him and his kind somewhat out of the loop, and scientists are already having these discussions between ourselves, and communicating them directly to the public.

Interactions and main effects in simple linear models

Bet that title’s got you itching to read on! Feel free to skip this post if you think stats are boring. You’d be wrong, but I won’t judge you… Anyway, getting straight down to the nitty gritty, and assuming that if you’ve hung in until now then you’re not afraid of words like ‘linear’ and ‘model’, here’s the thing: when I learnt statistics, one thing that was drummed in to us was that, if you’re fitting a linear model which includes interactions, you can’t sensibly interpret the main effects. I’ve been telling people the same ever since. But, in almost every paper that I review or edit, and which uses such models, people do just that. My purpose here then is to explain why I think that’s wrong. And hopefully, to find out if I am wrong to think that way.

Let’s start with a contrived example. Suppose we’ve measured activity levels at different times of the day across 100 individual birds, 50 of which are larks and 50 are owls, and we get the following:

interaction plot.jpg

Of course, in analysing these data the obvious thing to do would be to fit a linear model with activity modelled as a function of hour, species, and their interaction. The fitted lines on the figure above illustrate this model. And the explanation is straightforward: activity increases towards dawn in larks, and decreases towards dawn in owls.

What doesn’t make sense is to make any statements about general differences between larks and owls, or about any general trend in activity with time from dawn, because these ‘main effects’ are completely entangled within their interaction.

Unfortunately, most statistical software packages will give an output which includes significance levels for both main effects and interactions, for instance:

coef summary.jpg

Or in Anova form (R guys – I know this is wrong, but its particular flavour of wrongness is not important for this point!):

aov summary.jpg

Both of these outputs make it look like the main effects are ‘significant’, and the coefficients even seem to tell you the direction of these effects. So the kind of interpretation I read again and again would look something like: “There is a significant interaction between activity and species (p < 0.0001). In addition, activity increases with hour from dawn (p < 0.0001), and is higher in owls than in larks (p < 0.0001).” Which, as we’ve just demonstrated, is nonsense. And just because this example has been contrived to emphasise this point, doesn’t mean it doesn’t apply equally whenever there is an interaction; sensible plotting of your data will usually reveal this.

What should you do instead? Report the significant interaction, and then describe it, using the table of coefficients (and your knowledge of how your stats package of choice uses aliasing) to calculate the slope (and intercept if you like) for each level in the interaction. Here for example, the slope for the relationship for larks is 1.04, and for owls is 1.00 (1.045 – 2.048) – and you can get confidence intervals for both easily enough. And if the interaction doesn’t seem to be important, take it out, and interpret main effects to your heart’s content.

That’s my take on it anyway. The very concept of the ‘significance’ of a main effects is meaningless in the presence of an interaction between them. And so I keep telling people. So if I’m missing something, please let me know!

The enduring pleasure of paper

I have tried very hard to go paper free. My printer is rarely turned on, almost all of my scientific reading and reviewing, editing and marking is now done on screen, and my iPad Papers library is full of PDFs destined to remain digital. Why, then, was it such a pleasure to return home yesterday to find, after a gap of a few months, a pristine, cellophane-wrapped copy of Nature on my doormat? Honestly, I really did try to kick this habit. Earlier this year my Nature subscription expired around the time that my blogging briefly reached the required frequency to qualify me for personal online access. I duly installed the Nature.com Reader on the iPad, marvelled as it filled with content, and… have barely looked at it since.

Partly, I suspect, this is due to my entrenched reading routines. Just as I need a physical copy of the Guardian every Saturday to satisfy my coffee and crossword habit, I had also established the 20 minute tram ride to work as the ideal environment in which to catch up on essential science news. Maybe down in that London or somewhere equally fancy it would be acceptable to flop out the iPad and carry on regardless. But I just can’t do that on a Sheffield tram without feeling what Logan Mountstuart would call, with spectacular rudeness, a CAUC (one for readers of William Boyd’s Any Human Heart).

But also, I know my way around the physical magazine – know where I can find the news, the books and arts section, the opinions, correspondence, news and views, all the bits I tend to read – and prefer its linear structure to the mass of interlinked stuff on the online (and App-ed) versions, which always leave me with the feeling that I may have missed something essential. Harder to ignore content when there is less choice too, I suspect.

All of which lends to the peculiar sensation of somehow being more connected, more in touch, now I’m back with the hard copy. Of course, at some point the sheer weight of back issues in my office may force me to reconsider, but for now I remain happily wedded to paper.