I’m about halfway through reading the long awaited (preliminary report was supposed to have been issued in the spring according to the workplan of the) “Independent Climate Change E-mail Review” (ICCER) aka the Muir Russell Report:
Timescale
The Review team issued a call for submissions on 11 February 2010, and expects to receive submissions from UEA and the public by the end of February .
These responses will require analysis and there may be follow-up questions and/or interviews. The team expect to have at least preliminary conclusions by Spring 2010.
Perhaps there was a report of “preliminary conclusions”; but, if so, to the best of my knowledge it was never made public. Although the rather curious May 26 letter from “various” suggests the possibility that – behind closed screens – some report or other might have been circulated to a very privileged minority. May 26 is certainly closer to “Spring 2010” than July 7, the actual date of Muir Russell’s report.
Then again, it’s entirely possible that they might have changed the “workplan” … after all – unlike the “overwhelming scientific consensus” no workplan has ever been written in stone. But, considering the “Review Team’s” commitment to transparency:
Transparency
The Team will operate as openly and transparently as possible. It is establishing a website which will eventually display all of the submissions received, correspondence, analyses and conclusions. The aim will be to publish all received submissions quickly, unless there are wholly exceptional reasons to delay, for example legal issues.
one might think that if there was a change to the workplan, it would have been advertised, quite prominently.
In the absence of such announced change (unless one is expected to divine such a change from the various and sundry “note(s) of the actions agreed at the meeting(s)“), and considering:
one might be forgiven for wondering why a submission supposedly dated and received on May 26 was not posted to the site until sometime in early July.
Oh, let me guess … the authors were neither members of the “public” nor of UEA, therefore the late submission from “various” did not diverge from the workplan as advertised (or even as amended).
But, as as I sometimes do (hey, this is my blog, I can do this!) I digress …
My impression from my reading thus far reminds me of the response of over-protective, over-bearing parents who have been summoned to the Principal’s office because the behaviour of their bullying little brat has been causing disturbances in the schoolyard: “Oh, but all children act this way, and you shouldn’t pick on our little Johnnie because whatever else he might have done could not possibly have been his fault”.
But every, uh, cloud, does have a silver lining. And the very unexpected silver lining to this report can be found in the words of Richard Horton, editor of The Lancet. Horton, not one of the review team members, was the author of “Appendix 5: Peer Review”.
His words are a very frank appraisal of the limitations of peer-review, and include the following:
[p. 126]:
“The importance of peer review has been invoked by climate sceptics in other domains of the climate debate. Christopher Booker has challenged Dr Rajendra Pachauri, for example, for claiming that his Intergovernmental Panel on Climate Change (IPCC) included only peer-reviewed research (4). By contrast, Booker reports that a third of IPCC sources were newspaper articles, student dissertations, even press releases. Again, the suggestion is that the peer-reviewed literature is something special and sacred to science. One can make strong and reliable assertions if those statements are underpinned by peer-reviewed science. If evidence has not been peer-reviewed, it is next to worthless.”
[p. 127]:
“Much of the concern – and, indeed, confusion – about what took place at the CRU in relation to peer review may stem from misunderstandings about what peer review is and what it can be expected to do.”
[p. 131]:
“Everyone – scientists, the public, policymakers, politicians – would like to believe that peer review is a firewall between truth and error (or dishonesty) (15). But as the editor of one leading specialist medical journal has rightly pointed out, ―There is no question that, when it comes to peer review, the reviewers themselves are the weakest (or strongest) links”
[p. 132]:
“Unfortunately, there is evidence of a lack of evidence for peer review‘s efficacy”[p. 139]:
“The best one might hope for the future of peer review is to be able to foster an environment of continuous critique of research papers before and after publication. […]
“This process of weeding out weak research from the scientific literature can be accelerated through more formal mechanisms, such as the systematic review. A systematic approach to selecting evidence focuses on the quality of scientific methods rather than the reputations of scientists and their institutions. This more rigorous approach to gathering, appraising, and summing up the totality of available evidence has been profoundly valuable to clinical medicine. There may be useful lessons here for the IPCC. Climate sceptics and climate scientists, along with their colleagues in other scientific disciplines, would likely welcome this greater rigour and scrutiny. It would certainly promote quality and strengthen accountability to a more critical public (and media) with higher expectations of science. More importantly, intensified post as well as pre publication review would put uncertainty – its extent and boundaries – at the centre of the peer review and publication process. This new emphasis on uncertainty would limit the rhetorical power of the scientific paper (53), and offer an opportunity to make continuous but constructive public criticism of research a new norm of science.” [emphases added -hro]
Unlike the Review Team members, it would appear from his introductory comments that Horton has actually read Andrew Montford’s The Hockey Stick Illusion (which should have been required reading for Muir Russell and his team).
If you don’t have time right now to delve into the 20 pages of Appendix 5, you might consider taking a look at Horton’s article in the Guardian:
Climate email inquiry: bringing democracy to science
The climategate review could spark a new culture in science research and take peer review off its pedestal
[…]
[Muir Russell’s Report] is a forensic and deeply critical analysis of what took place in the climatic research unit at the University of East Anglia over many years. Russell concludes that the university fell badly short of its scientific and public obligations. It needs radical reform.
[Horton concludes]:
[S]cientists need to take peer review off its pedestal. As an editor, I know that rigorous peer review is indispensable. But I also know that it is widely misunderstood.
Peer review is not the absolute or final arbiter of scientific quality. It does not test the validity of a piece of research. It does not guarantee truth. Peer review can improve the quality of a research paper – it tells you something about the acceptability of new findings among fellow scientists – but the prevailing myths need to be debunked. We need a more realistic understanding about what peer review can do and what it can’t. If we treat peer review as a sacred academic cow, we will continue to let the public down again and again.
Finally, scientists should be educated to embrace this new culture of science, not fear or resist it. A scientist’s training will need to include ways of engaging citizen scientists constructively, making their data more widely available, putting uncertainty at the forefront of their work, and managing public expectations about what science can do.
The Russell review is the catalyst science needs for a thorough reappraisal of some of its most enduring and fundamental assumptions. Out of that re-examination will come a stronger and more sustainable scientific enterprise – one that the public will continue to put trust in. It’s time for science to take that first step. [emphasis added -hro]
I’ll drink to that :-)
The View From Here 