Roadmap | The review process | Home
   Peer review

... like toads on lily pads ...
The end of the Peer Show - scientists’ misguided attempt to solve a non-existent problem

Dr Ted Nield, NUJ FGS

Science journalist; Science & Communication Officer, Geological Society of London

ted.nield@geolsoc.org.uk

Click here to go to Discussion of this page

... rants here in a personal capacity to the 32nd IGC, Florence, August 2004 in the session “The Ethics of Peer Review”.


In 2003, the Royal Society (the de facto UK national academy of science) established its latest working party – to study best practice in the public release of scientific information, and the peer review process. The main motivation for this was the suspicion that the public was being confused by media reports emanating from non peer-reviewed “grey” publications, but reported as though they were kosher. This, the RS thought, might be undermining public trust in science.

The Royal thought that now might be time to reveal the peer review process to the public, in the hope that they would then understand the difference between reliable (peer reviewed) and unreliable science reports whenever they were faced, for example, with another story about human cloning, the NMR vaccine, intelligent underpants, or whatever. Perhaps, they thought, the public would find peer review’s very existence reassuring. Peer reviewed publications are in a sense “official” science. Surely the knowledge that they were reading officially sanctioned research would reassure – and at least help them sort out what they could safely believe and what they could take with a pinch of salt. Then, the thinking went, we might be on track to “restoring public trust”.

Now – more about peer review and all that in a moment. Meanwhile, in another part of the forest, science as a profession is itself about to become a little more officially regulated, as a result of some very similar ideas to do with public trust.

Here is a question for you. Who, exactly, is “a scientist”? What do journalists like me mean when we write “Scientists say”? Is anyone who has a degree in a scientific subject “a scientist”? What gives a person the right to be quoted as “a scientist”? Things are OK at the top end of the market - Fellows of the Royal Society, for example, have been elected by their peers. Things are OK too if the “scientist” can be seen to carry chartered status under a properly regulated system of professional formation – and has a title like Chartered Engineer (CEng), Chartered Chemist (CChem), or Chartered Geologist (CGeol) and so on.

But is it OK if these people are just paying members of such bodies? Are the election procedures to those bodies rigorous, or is it just a matter of paying the annual dues? And how can we know this, when all these scientific societies were set up in different ways over many centuries? Furthermore, many bona fide scientists who are ordinary members of learned bodies – a large number of them well respected academics, for example – either cannot or do not wish to qualify for Chartered status under the rules of their professional and scientific body. Yet should this debar them from being described as “scientists”? Here is a dilemma.

In an attempt to address this alleged problem, the UK’s Science Council (which acts as an umbrella body for many scientific learned societies and institutes) has invented a new designation – Chartered Scientist – which they hope will be popular, conferring “official status” upon its holders as “spokespersons for science”, and bringing with it distinct responsibilities that will act as a reassurance to the public that these scientists really are scientists, and can be trusted.

Thus we have two emerging trends – one towards reinforcing and packaging the peer review system for the public good on the one hand, and another towards formal recognition of the status of individuals. Together these two movements constitute what I will call the ‘official’ road to public trust.

Those who follow it are rooted in the belief that, by strengthening professional formation and the processes of scientific sanction over publications, scientists’ status will rise (in the eyes of people at large), and their work will benefit - in that if those who report science in the media fully understand the difference between peer-reviewed “white” and non-peer-reviewed “grey” literature, and mention this distinction in their writings, the public will know what stories and people to believe, and which to discount. Public trust will have been restored.

It is difficult to list the number of flaws in this plan, but any one of them is completely fatal to it. Here is the simplest example of such a flaw, a reason why the entire plan is useless, chosen because it is most easily explained in limited time. This is it: you have my word as a journalist and a gentleman that the chances of any journalist mentioning whether any quoted “scientist” is a chartered anything, are precisely nil. And at this point, you see, the whole plan falls apart. Yet this is not the most fundamental problem with the idea – it is merely a reason why it wouldn’t achieve anything at all. There is a bigger problem, more serious than mere practicality – a fundamental flaw in the very framing of the problem that scientists think they are addressing.

The fact is that scientists are NOT mistrusted at all, though some are more trusted than others. What makes the public most mistrustful, is not “science” but the Establishment, and “officialness” – of any kind.

I often quote the results of the Office of Science & Technology/Wellcome Foundation joint report (see reference). It should be the daily reading of all scientists with an interest in public communication. Ever since it appeared just before the end of the last century, we have known that most of the public put more trust in scientists than in many other professions, but that this trust is subtly moderated according to how independent the scientist in question is perceived to be.

Thus, university scientists were found to be more trustworthy than those working for the Government, or commercial companies. Naïvely perhaps, scientist spokespeople working on behalf of Green pressure groups are not seen as “tainted” by their employment as much as those working for government or companies. (There are good Public Relations reasons why this is so, which I shall explain later.)

That one caveat aside, the UK public’s attitude to the trustworthiness of scientists is exactly as it should be. The public should not mistrust scientists per se (and they don’t) but nor should the public be expected to trust them implicitly – or any more than anyone else - just because they are scientists. The public shows subtlety and maturity in its evaluation of trustworthiness, and makes commonsense adjustments according to where the scientist gets his wages.

As I have said, this finding also shows is that the people of the UK are suspicious not of scientists, but of The Establishment. Thus the more embroiled a scientist is seen to be in government, the civil service, departments of state, commercial companies etc., the less instinctively trusted he or she will be when quoted. The more independent that scientist is, on the other hand, the more trusted. Universities are widely seen as bastions against such special interest. Their apparent disinterestedness explains why university scientists are relatively more trusted.

And why are pressure-group science spokespersons relatively more trusted than commercial or government scientists? The perception of the public is that scientists working for Green pressure groups are in some sense playing the role of the underdog. They are not seen as doing what they do for love of success, status or money, but because they are passionate in the service of a cause – hence they come over as “independent free spirits” rather than as establishment goons. This is why they are relatively more trusted. This distinction is important, for a reason that brings us right back to where we started, with the peer review process.

Attempts by scientists to bolster public trust by boosting their “officialness” (by professional accreditations) are missing the point. In fact, their idea, if it were to become known to the public, would have an effect exactly 180 degrees from that intended. Normally, it is wise to leave such spectacular shootings in the foot to Governments. But Peer Review is also the process of “officialising” science, as well as the process that makes scientific publishing scientific.

Adherents of the idea that peer review, once explained, will increase public trust in science – and this includes, I would venture, every one of the Royal Society’s review group – are living in a dream world. Or, at the very least, they take a very optimistic view not only of the review system, but also of people’s likely reaction to it – should they ever succeed in explaining it to them.

Let us take likely public reaction first. The most likely attitude that non-scientists would take to peer review is this. Once its mechanisms are explained, the public’s most likely reaction would be that peer review is probably a corrupt system whereby those in authority (the scientific Establishment) stamp on unorthodox ideas and enforce a (probably false) consensus upon their subject; where old men of failing ability sit in positions of power like toads on lily pads, protecting their achievements by suppressing new ideas in print, and perverting the course of grant money away from those they regard as enemies.

Peer review can never be perfect, and surely only the most blinkered idealist would hold that it is not, from time to time, corrupted in just these very ways. But apart from that, its track record in preventing fraud or the publication of badly conducted research is far from admirable. Much of the research over whose media coverage there has been such scientific hand-wringing in recent years in the UK (Mr. Arpad Puzstai’s laboratory mice, for example, whose prognosis seemed to show that genetically modified potatoes were in some way bad for them), was published in one of the most highly reputable journals in the world – The Lancet. Many other journals, including Nature and Science, have been recently forced to retract papers that passed peer review but nevertheless turned out to be completely fraudulent. In many cases this has been because peer review is ill equipped to detect fraud; but in others, poor scientific procedure – exactly the thing that peer review is meant to spot – also survived into publication.

So – the public would be right to regard peer review as potentially corrupt and dangerous, because it is; and they would be right to believe that it often fails to do precisely what it is supposed to do, because it has.

So, science’s attempts to address its non-existent problem of public trust are taking two courses that would have exactly the opposite effect and which would undermine rather than reinforce the public’s trust in their words. Happily, for reasons I have explained in part, neither course on this misguided “official road” to public trust will cause any damage – because neither stands the slightest chance of reaching the public.

I have already mentioned one fatal flaw in the Chartership route – namely that no journalist will ever take any notice of it and if they did, it would mean nothing to the reader. And as for explaining peer review to the public, well – the notion that people at large will wish to engage with peer review sufficiently to form any strong opinion of it – even assuming they do not lose the will to live half way through the explanation – is utterly unrealistic and completely fantastic.

The truth is that peer review processes will never excite much interest – or even attention – among any body of people except scientists and publishers, and it is expecting far too much to think otherwise. Moreover most journalists, and certainly all science journalists, fully understand peer review already, do not need to be lectured further about its importance, and can be relied upon to exercise good professional judgement on the public’s behalf in how they interpret what is published. Ninety nine times out of a hundred, this is exactly what happens. The “problem” stories are those that escape into general reporting, and there may be a case for concerted firefighting by science organisations to help non-science specialists to understand the special constraints of science reporting. But that is another subject, and beyond the scope of this paper.

But could peer review be changed? Should the Royal Society should turn their question around? Forget “strengthening” peer review. Perhaps it needs radical reform?

Consider this idea. One way the peer review process could be changed so as to render it less obnoxious to public instincts, would be to remove the cloak of secrecy that often surrounds reviewers. (This practice varies from journal to journal these days, but in the classic process, the identities of those who comment on submitted papers are withheld from the author.) The invisibility cloak is, in any case, a lot less effective than Harry Potter’s, since most researchers always have a very shrewd idea of who their reviewers are. But the secrecy of reviewers lies at the root of the problem with presenting peer review to the public.

If there were a genuine case for anyone in the peer review process to remain anonymous, then it should of course be the author – though this is almost never the case. This anonymity could be defended on the ground that it would allow the reviewer to form an unbiased opinion of the work – just as, in orchestral auditions, applicants play behind a screen, so as not to allow extraneous factors (like age, gender or physical beauty) to influence their peers’ judgement.

But of course, this plan would be as unlikely to work as the current situation, and for the same reason – the world of science is too small. Everyone knows everyone.

So while potentially more defensible, this form of anonymity is also a waste of time, because it is impractical. From the PR perspective, peer review would only be totally defensible (and excite more respect) if academic publishers and reviewers adhered to the axiom that governs my profession – journalism. No journalist would ever express any opinion that they would not, if necessary, be prepared to defend in public. The reasons given for allowing anonymity to reviewers are rarely strong, and serve only to conspire with, and give oxygen to, natural moral cowardice, and present too much temptation in the way of the powerful. Worse, the very existence of anonymity lays peer review open to corruption, and will always appear odious to the general public. Moreover, it rarely works even by its own criteria.

I have heard it said by those who defend the anonymity of reviewers in some circumstances that it can help younger editorial team members who might otherwise be afraid of criticising the work of grand old men of the subject, for fear of the consequences. This argument is, to my mind, an empty one. First the principle that I hold paramount is that all secrecy in this process is hateful and this is unaffected by specifics like this. But also, young men are just as prone to vanity and hubris as old ones are to reaction. The case for anonymous authors (which is defeated only by its impracticality) is just as strong in the case of a paper by an old scientist being reviewed by a young tyro, as in the reverse case. Prejudice against the old is no more excusable than any other form of intolerance.

It is time for scientists and their reviewers to call for an end to what remains of the peer show’s futile and odious secrecy, grow up, be mature, and most of all, be prepared to defend what they write to an author’s face with humour, humility and humanity. I fundamentally reject any administrative fix that conspires to help people not to face all the consequences of anything they say or write. In whatever form, it is no more than pandering to moral cowardice – pusillanimity is the term – and there is enough of that in the world as it is.

Something worth reading

Science and the public: a review of science communication and public attitudes to science in Britain. A joint report by the Office of Science & Technology (OST) and the Wellcome Trust. ISBN 1 841290 25 4. Available free as PDF at www.wellcome.ac.uk or from the Marketing Dept., Wellcome Trust, 183 Euston Road LONDON NW1 2BE. Tel: 020 7611 8651, FAX 020 7611 8545, Email marketing@wellcome.ac.uk.

Discussion

See also: http://www.mantleplumes.org/Penrose/Anonymity.html

Monday Sept 27th, 2004: Some unrelated remarks by Ted Irving that found their way onto the website as a result of the Ed. confusing him with Ted Nield

To Sign or not to Sign.
I understand you are interested in opinions on this question. I can't remember getting a signed review when I began 50 years ago, and the modern drift towards them may be a mistake. Here are my reasons.

  1. A signed review denies you the fun of figuring out who the reviewer is. I mean this.
  2. It used to be that all reviews were unsigned. Now I name myself, feeble-mindedly following the modern trend. But I am considering not doing so again because it can weaken friendships, separate friends, even make enemies. Scientists are a sensitive lot and a frank but good and well intentioned review often irritates people who think, without reason, that they are being got at, or deprived of tenure by some conspiracy, or subjected to settling of old score. Authors have to grow-up, and a stiff unsigned review is a good place to start; authors should not be deprived of that experience; of being obliged to consider criticism for what it is itself without personal context.
  3. Every day I wonder how science ever works, such is the overwhelming force of group-think. The frank review becomes essential. I suspect unsignedness favours that. Reviewers have two duties, to speak truth as they see it and to treat the paper as confidential. A good editor recognises an incompetent or nasty review and handles it accordingly. A good editor also recognises an intemperate response from an author. So with a friendly word a good editor can help everybody, but less readily, I believe, if the review is signed.
  4. It is sometimes forgotten that the reviewer is an important person. Reviewers should have protection from harassment by an aggressive author tiresomely seeking self-justification.
  5. Reviewers should not expect thanks or favour, have no chance to gain: these things anonymity ensures.

Wednesday Sept 29th, 2004: A few further remarks by Ted Irving

I disagree with Ted N in his last paragraph about moral courage. I have a feeling he is dangerously extending his experience as a journalist too far into the realm of scientific review; they are different kettles of fish. By doing so Ted, I think you are reducing the problem to easy simplicities. An Ulsterman I knew well 50 years ago whose physical and moral courage were matters of record, told me he never signed a review of a scientific paper; the argument was the thing; personality should be discounted. Is it morally more courageous (1) to sign a weak-kneed review and make no enemies or (2) give an unsigned strong one? I don’t think it is. And by giving an unsigned strong one, what harm is done to science? None. Indeed science is well served. The latter (2) makes for better science, and that is paramount. The status of the moral courage of reviewers in scientific review is secondary to the good of science; that is the core of my disagreement with you.

Look at it in yet another way, people of high moral courage don’t always have the best judgment, and ones of lesser courage may be a brilliant scholars who words are worth diamonds. Should we be deprived of their wisdom because they are timid?

last updated 27th September, 2004
:: HOME :: MECHANISMS :: LOCALITIES :: GENERIC ::
© MantlePlumes.org