... rants here in a personal capacity to the
32nd IGC, Florence, August 2004 in the session “The
Ethics of Peer Review”.
In 2003, the Royal Society (the de facto UK national
academy of science) established its latest working
party – to study best practice in the public
release of scientific information, and the peer review
process. The main motivation for this was the suspicion
that the public was being confused by media reports
emanating from non peer-reviewed “grey”
publications, but reported as though they were kosher.
This, the RS thought, might be undermining public
trust in science.
The Royal thought that now might be time to reveal
the peer review process to the public, in the hope
that they would then understand the difference between
reliable (peer reviewed) and unreliable science reports
whenever they were faced, for example, with another
story about human cloning, the NMR vaccine, intelligent
underpants, or whatever. Perhaps, they thought, the
public would find peer review’s very existence
reassuring. Peer reviewed publications are in a sense
“official” science. Surely the knowledge
that they were reading officially sanctioned research
would reassure – and at least help them sort
out what they could safely believe and what they could
take with a pinch of salt. Then, the thinking went,
we might be on track to “restoring public trust”.
Now – more about peer review and all that in
a moment. Meanwhile, in another part of the forest,
science as a profession is itself about to become
a little more officially regulated, as a result of
some very similar ideas to do with public trust.
Here is a question for you. Who, exactly, is “a
scientist”? What do journalists like me mean
when we write “Scientists say”? Is anyone
who has a degree in a scientific subject “a
scientist”? What gives a person the right to
be quoted as “a scientist”? Things are
OK at the top end of the market - Fellows of the Royal
Society, for example, have been elected by their peers.
Things are OK too if the “scientist” can
be seen to carry chartered status under a properly
regulated system of professional formation –
and has a title like Chartered Engineer (CEng), Chartered
Chemist (CChem), or Chartered Geologist (CGeol) and
so on.
But is it OK if these people are just paying members
of such bodies? Are the election procedures to those
bodies rigorous, or is it just a matter of paying
the annual dues? And how can we know this, when all
these scientific societies were set up in different
ways over many centuries? Furthermore, many bona fide
scientists who are ordinary members of learned bodies
– a large number of them well respected academics,
for example – either cannot or do not wish to
qualify for Chartered status under the rules of their
professional and scientific body. Yet should this
debar them from being described as “scientists”?
Here is a dilemma.
In an attempt to address this alleged problem, the
UK’s Science Council (which acts as an umbrella
body for many scientific learned societies and institutes)
has invented a new designation – Chartered Scientist
– which they hope will be popular, conferring
“official status” upon its holders as
“spokespersons for science”, and bringing
with it distinct responsibilities that will act as
a reassurance to the public that these scientists
really are scientists, and can be trusted.
Thus we have two emerging trends – one towards
reinforcing and packaging the peer review system for
the public good on the one hand, and another towards
formal recognition of the status of individuals. Together
these two movements constitute what I will call the
‘official’ road to public trust.
Those who follow it are rooted in the belief that,
by strengthening professional formation and the processes
of scientific sanction over publications, scientists’
status will rise (in the eyes of people at large),
and their work will benefit - in that if those who
report science in the media fully understand the difference
between peer-reviewed “white” and non-peer-reviewed
“grey” literature, and mention this distinction
in their writings, the public will know what stories
and people to believe, and which to discount. Public
trust will have been restored.
It is difficult to list the number of flaws in this
plan, but any one of them is completely fatal to it.
Here is the simplest example of such a flaw, a reason
why the entire plan is useless, chosen because it
is most easily explained in limited time. This is
it: you have my word as a journalist and a gentleman
that the chances of any journalist mentioning whether
any quoted “scientist” is a chartered
anything, are precisely nil. And at this point, you
see, the whole plan falls apart. Yet this is not the
most fundamental problem with the idea – it
is merely a reason why it wouldn’t achieve anything
at all. There is a bigger problem, more serious than
mere practicality – a fundamental flaw in the
very framing of the problem that scientists think
they are addressing.
The fact is that scientists are NOT mistrusted at
all, though some are more trusted than others. What
makes the public most mistrustful, is not “science”
but the Establishment, and “officialness”
– of any kind.
I often quote the results of the Office of Science
& Technology/Wellcome Foundation joint report
(see reference). It should be the daily reading of
all scientists with an interest in public communication.
Ever since it appeared just before the end of the
last century, we have known that most of the public
put more trust in scientists than in many other professions,
but that this trust is subtly moderated according
to how independent the scientist in question is perceived
to be.
Thus, university scientists were found to be more
trustworthy than those working for the Government,
or commercial companies. Naïvely perhaps, scientist
spokespeople working on behalf of Green pressure groups
are not seen as “tainted” by their employment
as much as those working for government or companies.
(There are good Public Relations reasons why this
is so, which I shall explain later.)
That one caveat aside, the UK public’s attitude
to the trustworthiness of scientists is exactly as
it should be. The public should not mistrust scientists
per se (and they don’t) but nor should the public
be expected to trust them implicitly – or any
more than anyone else - just because they are scientists.
The public shows subtlety and maturity in its evaluation
of trustworthiness, and makes commonsense adjustments
according to where the scientist gets his wages.
As I have said, this finding also shows is that the
people of the UK are suspicious not of scientists,
but of The Establishment. Thus the more embroiled
a scientist is seen to be in government, the civil
service, departments of state, commercial companies
etc., the less instinctively trusted he or she will
be when quoted. The more independent that scientist
is, on the other hand, the more trusted. Universities
are widely seen as bastions against such special interest.
Their apparent disinterestedness explains why university
scientists are relatively more trusted.
And why are pressure-group science spokespersons
relatively more trusted than commercial or government
scientists? The perception of the public is that scientists
working for Green pressure groups are in some sense
playing the role of the underdog. They are not seen
as doing what they do for love of success, status
or money, but because they are passionate in the service
of a cause – hence they come over as “independent
free spirits” rather than as establishment goons.
This is why they are relatively more trusted. This
distinction is important, for a reason that brings
us right back to where we started, with the peer review
process.
Attempts by scientists to bolster public trust by
boosting their “officialness” (by professional
accreditations) are missing the point. In fact, their
idea, if it were to become known to the public, would
have an effect exactly 180 degrees from that intended.
Normally, it is wise to leave such spectacular shootings
in the foot to Governments. But Peer Review is also
the process of “officialising” science,
as well as the process that makes scientific publishing
scientific.
Adherents of the idea that peer review, once explained,
will increase public trust in science – and
this includes, I would venture, every one of the Royal
Society’s review group – are living in
a dream world. Or, at the very least, they take a
very optimistic view not only of the review system,
but also of people’s likely reaction to it –
should they ever succeed in explaining it to them.
Let us take likely public reaction first. The most
likely attitude that non-scientists would take to
peer review is this. Once its mechanisms are explained,
the public’s most likely reaction would be that
peer review is probably a corrupt system whereby those
in authority (the scientific Establishment) stamp
on unorthodox ideas and enforce a (probably false)
consensus upon their subject; where old men of failing
ability sit in positions of power like toads on lily
pads, protecting their achievements by suppressing
new ideas in print, and perverting the course of grant
money away from those they regard as enemies.
Peer review can never be perfect, and surely only
the most blinkered idealist would hold that it is
not, from time to time, corrupted in just these very
ways. But apart from that, its track record in preventing
fraud or the publication of badly conducted research
is far from admirable. Much of the research over whose
media coverage there has been such scientific hand-wringing
in recent years in the UK (Mr. Arpad Puzstai’s
laboratory mice, for example, whose prognosis seemed
to show that genetically modified potatoes were in
some way bad for them), was published in one of the
most highly reputable journals in the world –
The Lancet. Many other journals, including Nature
and Science, have been recently forced to retract
papers that passed peer review but nevertheless turned
out to be completely fraudulent. In many cases this
has been because peer review is ill equipped to detect
fraud; but in others, poor scientific procedure –
exactly the thing that peer review is meant to spot
– also survived into publication.
So – the public would be right to regard peer
review as potentially corrupt and dangerous, because
it is; and they would be right to believe that it
often fails to do precisely what it is supposed to
do, because it has.
So, science’s attempts to address its non-existent
problem of public trust are taking two courses that
would have exactly the opposite effect and which would
undermine rather than reinforce the public’s
trust in their words. Happily, for reasons I have
explained in part, neither course on this misguided
“official road” to public trust will cause
any damage – because neither stands the slightest
chance of reaching the public.
I have already mentioned one fatal flaw in the Chartership
route – namely that no journalist will ever
take any notice of it and if they did, it would mean
nothing to the reader. And as for explaining peer
review to the public, well – the notion that
people at large will wish to engage with peer review
sufficiently to form any strong opinion of it –
even assuming they do not lose the will to live half
way through the explanation – is utterly unrealistic
and completely fantastic.
The truth is that peer review processes will never
excite much interest – or even attention –
among any body of people except scientists and publishers,
and it is expecting far too much to think otherwise.
Moreover most journalists, and certainly all science
journalists, fully understand peer review already,
do not need to be lectured further about its importance,
and can be relied upon to exercise good professional
judgement on the public’s behalf in how they
interpret what is published. Ninety nine times out
of a hundred, this is exactly what happens. The “problem”
stories are those that escape into general reporting,
and there may be a case for concerted firefighting
by science organisations to help non-science specialists
to understand the special constraints of science reporting.
But that is another subject, and beyond the scope
of this paper.
But could peer review be changed? Should the Royal
Society should turn their question around? Forget
“strengthening” peer review. Perhaps it
needs radical reform?
Consider this idea. One way the peer review process
could be changed so as to render it less obnoxious
to public instincts, would be to remove the cloak
of secrecy that often surrounds reviewers. (This practice
varies from journal to journal these days, but in
the classic process, the identities of those who comment
on submitted papers are withheld from the author.)
The invisibility cloak is, in any case, a lot less
effective than Harry Potter’s, since most researchers
always have a very shrewd idea of who their reviewers
are. But the secrecy of reviewers lies at the root
of the problem with presenting peer review to the
public.
If there were a genuine case for anyone in the peer
review process to remain anonymous, then it should
of course be the author – though this is almost
never the case. This anonymity could be defended on
the ground that it would allow the reviewer to form
an unbiased opinion of the work – just as, in
orchestral auditions, applicants play behind a screen,
so as not to allow extraneous factors (like age, gender
or physical beauty) to influence their peers’
judgement.
But of course, this plan would be as unlikely to
work as the current situation, and for the same reason
– the world of science is too small. Everyone
knows everyone.
So while potentially more defensible, this form of
anonymity is also a waste of time, because it is impractical.
From the PR perspective, peer review would only be
totally defensible (and excite more respect) if academic
publishers and reviewers adhered to the axiom that
governs my profession – journalism. No journalist
would ever express any opinion that they would not,
if necessary, be prepared to defend in public. The
reasons given for allowing anonymity to reviewers
are rarely strong, and serve only to conspire with,
and give oxygen to, natural moral cowardice, and present
too much temptation in the way of the powerful. Worse,
the very existence of anonymity lays peer review open
to corruption, and will always appear odious to the
general public. Moreover, it rarely works even by
its own criteria.
I have heard it said by those who defend the anonymity
of reviewers in some circumstances that it can help
younger editorial team members who might otherwise
be afraid of criticising the work of grand old men
of the subject, for fear of the consequences. This
argument is, to my mind, an empty one. First the principle
that I hold paramount is that all secrecy in this
process is hateful and this is unaffected by specifics
like this. But also, young men are just as prone to
vanity and hubris as old ones are to reaction. The
case for anonymous authors (which is defeated only
by its impracticality) is just as strong in the case
of a paper by an old scientist being reviewed by a
young tyro, as in the reverse case. Prejudice against
the old is no more excusable than any other form of
intolerance.
It is time for scientists and their reviewers to
call for an end to what remains of the peer show’s
futile and odious secrecy, grow up, be mature, and
most of all, be prepared to defend what they write
to an author’s face with humour, humility and
humanity. I fundamentally reject any administrative
fix that conspires to help people not to face all
the consequences of anything they say or write. In
whatever form, it is no more than pandering to moral
cowardice – pusillanimity is the term –
and there is enough of that in the world as it is.