No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process. In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies. Nature has published a series of articles about the worrying extent to which research results have been found wanting in this respect. The editors of Nature and the Nature life sciences research journals have also taken substantive steps to put our own houses in order, in improving the transparency and robustness of what we publish. Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility. We hope that the articles contained in this collection will help. [emphasis added -hro]
It is certainly not entirely clear from the above whether or not the Nature Group’s climate science related journals fall (perhaps conveniently?!) outside this new, improved umbrella. The absence of any mention of the “complex system” known as “climate” suggests that it will continue to be given an exemption from such enhanced scrutiny.
When one considers all that has come to light since Climategate in November, 2009, it is somewhat of an understatement to suggest that Nature‘s pattern and practice wrt the “transparency and robustness” of the climate science they publish – along with that of Science – is not a record of which either could or should be proud.
Which brings me to some facts that came to light during the discussion on my recent post.
Readers will recall that my post was precipitated, in part, by a tweet I had spotted from Intergovernmental Panel on Climate Change (IPCC) Lead Author and U.K. Met Office’s head of “climate impacts”, Richard Betts. The implicit content of Bett’s text (i.e. the subtext) found in his brief … uh … assessment strongly suggested that Betts had failed to see that which was in black and white in front of his very eyes.
According to Betts, Steve McIntyre had evidently committed the unforgivable sin of (… wait for it! …) speculation. Here (again) is the text of Betts’ tweet:
Steve McIntyre’s comment about “pressure” on Nature to accept PAGES2K seems to be entirely speculation.
Here’s McIntyre’s apparently sinful text:
The PAGES2K article has its own interesting backstory. The made-for-IPCC article was submitted to Science last July on deadline eve, thereby permitting its use in the Second Draft, where it sourced a major regional paleo reconstruction graphic. The PAGES2K submission used (in a check-kited version) the Gergis reconstruction, which it cited as being “under revision” though, at the time, it had been disappeared.
The PAGES2K submission to Science appears to have been rejected as it has never appeared in Science and a corresponding article is scheduled for publication by Nature. It sounds like there is an interesting backstory here: one presumes that IPCC would have been annoyed by Science’s failure to publish the article and that there must have been considerable pressure on Nature to accept the article. Nature appears to have accepted the PAGES2K article only on IPCC deadline eve.
So, far from being the revelation Betts had implied, McIntyre had made it quite clear that he was “speculating” – well, clear at least to one who chooses to read rather than “skim” before commenting.
McIntyre dropped by this (usually very) quiet little corner of the blogosphere, to clarify his initial observations for Betts’ edification. His comment included:
One reasonable review response might well have been that the authors should publish their regional reconstructions in specialist journals. And that the authors should publish a detailed analysis of the methodologies in a specialist journal. It seems entirely possible that Science might have taken that position in rejecting the article.
Without the looming IPCC deadline and the prominent use of PAGES2K results by IPCC, I believe that it is entirely reasonable that Nature would have taken a similar position (to my interpretation of Science) and told the authors to split the article up into manageable review pieces. Do I believe that Nature recognized the need for very rapid acceptance and selected reviewers who also recognized the problem? Yes.
Without a set of quality control standards for academic peer review, it is impossible to say whether this process met or did not meet standards. I therefore am not moralizing about whether something was “amiss” in this case. However, I recommend that readers should not presume that the journal peer review constituted serious due diligence of the PAGES2K article.
Richard Betts’ implied that it was somehow “unhealthy” to point this out. I disagree. [emphasis added -hro]
Betts may well have missed this during the course of his daily skim, because he did not respond. His (presumed) friend, fellow scientist Oliver Bothe – known as @geschichtenpost on twitter – did respond, as he subsequently announced to his twitter followers:
Come to work, get DM on twitter, lose nearly three hours of work time. Argh. Well, da code does da work. replied to https://hro001.wordpress.com/2013/04/23/questions-for-a-jewel-in-the-crown-of-u-k-and-global-science/ …
Some might <<gasp>> speculate that there is some connection between the DM [DM = Direct Message, i.e. a private message via twitter], the nearly three hours lost and Bothe’s (quite lengthy) response to McIntyre’s comment noted above. But I couldn’t possibly comment. Bothe began his reply as follows:
To start, yes I am generally unhappy with Steve McIntyre’s speculative tone that to me often implies the accusation of scientific misconduct or corruption of peer review where to me everything is in the scope of peer review (not only in climate science but in science generally). I would like to see his post-publication review submitted as comments to journals and I would appreciate if the journals would consider him as reviewer for reconstruction and proxy-papers.
What Bothe may not realize is that, in this instance, the timing is such that even if McIntyre were inclined to submit his post-publication review as a comment to one or more journals [h/t Jonathan Jones via Bishop Hill], the IPCC “rules” are such that his comments can be ignored, just as they can (according to the IPCC’s new, improved “rules”) when posted on a blog.
Bothe noted (inter alia):
On Steve McIntyre’s comment on peer review in general. Yes, I think he’s right there. But the definition of peer-review is not a priori to try to replicate the results. Maybe Paul Matthews can comment whether mathematical peer review tries to follow each step in a paper and Jonathan Jones may comment on how this is done in the physical sub-fields he’s involved in. The task to ensure the possibility of replication lies with the author. The reviewer highlights gaps. At least that’s my impression. Should that be changed? No. The replication of results is part of the post-publication evaluation and failure to do so should be communicated in comments and possibly lead to retractions or corrections.
Put differently: The authors have to describe their methods so well that an informed reader can replicate their results with her prior knowledge and access to the data. Ideally the authors provide their code (well that should be requested). The reviewers have to check that the description allows the informed reader this replication.
So I am sure that the paper was subject to peer review as tense as average in science. Could it be more thorough. I would say peer-review can always be more detailed.
To my mind, while I don’t disagree with Bothe, is it not then the case that the IPCC’s past (although one hopes not future) claims of reliance on “all peer reviewed literature”, in addition to being false, fails to take into account this built-in deficiency in both the peer review process and the IPCC process – not to mention the myth of the glory, sanctity and superiority of “peer review” as the be-all and end-all of valid “science”?
Bothe had also written:
[In response to McIntyre’s:]
One reasonable review response might well have been that the authors should publish their regional reconstructions in specialist journals. And that the authors should publish a detailed analysis of the methodologies in a specialist journal. It seems entirely possible that Science might have taken that position in rejecting the article.
To add speculation to speculation. I assume that the consortium asked Science whether they would be interested before formal submission. That’s common and even encouraged by the glamour-journals. And I can say that the idea was, in principle, to publish the reconstructions in specialist journals and to submit this synthesis paper to one of the “Letter”-journals.
OK, I think I get it, now. Bothe’s presumptions, assumptions and speculations (and, presumably, those of Betts) are kosher; but those of McIntyre – for some reason perhaps best known only to themselves – are not.
Nonetheless, in light of Bothe’s “submit this synthesis paper to one of the ‘Letter’-journals”, I do wonder if Nature Geo-Science is deemed to be a “Letter”-journal and is therefore somewhat lower on the academic publish-or-perish-but-let’s-be- sure-we-meet-IPCC-sweepstakes-deadline totem-pole than others in the Nature and/or Science respective (but, nowadays, decreasingly respected) family of publications.
Moving right along …
[McIntyre had also written:]
Without the looming IPCC deadline and the prominent use of PAGES2K results by IPCC, I believe that it is entirely reasonable that Nature would have taken a similar position (to my interpretation of Science) and told the authors to split the article up into manageable review pieces. Do I believe that Nature recognized the need for very rapid acceptance and selected reviewers who also recognized the problem? Yes.
[To which Bothe had responded:]
Again speculation to speculation. I think the synthesis provided by the PAGES2K consortium perfectly fits the scope of a progress-article in Nature Geoscience. McIntyre’s last question has to be answered with yes, but do I think that prevented a thorough peer review? No. See list of possible alternatives above. [emphasis added -hro]
A few days later, McIntyre responded to Bothe. His reply included the following:
Dr Bothe’s comment about a “Progress Article” was an interesting one. He said: “I think the synthesis provided by the PAGES2K consortium perfectly fits the scope of a progress-article in Nature Geoscience.”
I confess that I had not previously paid note to the fact the PAGES2K article was published as a “Progress Article”, rather than a research article. Nor indeed had I been previously aware of the differences between the two in academic terms. However, given Dr Bothe’s belief that PAGES2K “perfectly fits” the definition of a “Progress Article”, here is Nature’s policy on Progress Articles http://www.nature.com/ngeo/authors/content_types.html
When the discussion is focused on a developing field that might not yet be mature enough for review, a Progress article is more appropriate. Progress articles are up to 2,000 words in length, with up to 4 display items (figures, tables or boxes). References are limited to 50. Reviews and Progress articles are commissioned by the editors, but proposals including a short synopsis are welcome.
Reviews and Progress articles are always peer-reviewed to ensure factual accuracy, appropriate citations and scholarly balance. They do not include received/accepted dates.
Curiously, although the Policy states that Progress Articles “do not include received/accepted dates”, Nature Geoscience, in apparent violation of this policy, stated that the the PAGES2K article was “Received 9 December 2012; accepted 11 March 2013; published online 21 April 2013″. Reasonable people may differ on why Nature Geoscience violated this particular policy, but I presume that they wished to demonstrate that the article had been “accepted” prior to the IPCC deadline of March 15. (It is also possible that Nature doesn’t actually observe the stated policy.) [emphasis added -hro]
As McIntyre subsequently noted, “… Nature has disregarded its policy on Received/accepted dates on Progress Articles on other non-climate related occasions.”
However, this was not the only part of the Progress Article policy (and/or practice) that Nature Geoscience appears to have waived in this instance. Yes, the number of references is exactly 50 – and by my count there are only 4 display items.
But the word count is a completely different kettle of fish! At 3,617 (including the 167 word abstract, but excluding all headings and narratives included in the 4 display items), this significantly exceeds the 2,000 word limit for a Progress Article.
And it looks as though the answer to my earlier question to Richard Betts:
Or do you have any evidence that this presumed “*extra* care” included the due diligence required? IOW, did the review include any objective examination of the underlying data and methodologies by reviewers whose understanding of (and expertise in) statistics matches that of Steve and/or other contributors to Climate Audit?
a question that he did not answer in his reply, btw, is – in all likelihood and with a high level of confidence (by IPCC standards) – a definite “No”, he has no such evidence. Nor is there any evidence of his presumed “*extra* care”.
Consequently, my vote on the validity of various parties’ respective speculations, presumptions and assumptions goes to McIntyre, who had concluded his response to Bothe as follows:
Dr Bothe said that PAGES2K “perfectly fit” the definition, a definition which recommends Progress Articles for “a developing field that might not yet be mature enough for review”. I’m surprised to learn that this is Dr Bothe’s position. My own position is that the field is “mature enough for review” and that PAGES2K therefore did not qualify for the lesser due diligence of a Progress Article – particularly when it was known that IPCC planned to use it.
Now that Dr Bothe has drawn attention to the curious fact that PAGES2K was published as a “Progress Article”, I think that it is entirely possible that one or more of the Nature reviewers, like the Science reviewers, may have recognized the impossibility of careful review of seven reconstructions using multiple methods and that someone therefore had the bright idea of circumventing the problem by labeling PAGES2K as a “Progress Article”, thereby lessening the review burden. Speculation on my part, but perhaps Dr Bothe can ask the authors whether my speculation is correct.
I had closed my earlier comment with the observation “I recommend that readers should not presume that the journal peer review constituted serious due diligence of the PAGES2K article.” Given that Dr Bothe has pointed out that the PAGES2K was merely published as a “Progress Article”, the recommendation seems even more appropriate. [emphasis added -hro]
Has either Betts or Bothe had the courtesy to respond to McIntyre’s reply to Bothe? Not bloomin’ likely!
I’ve no way of knowing whether Bothe has even returned to the thread in the interim. Who knows, perhaps he’s received another DM that is keeping him somewhat preoccupied. Betts, OTOH, seemed to think it was far more important to justify his depiction of McIntyre as “tenacious”. It was evidently my fault for suggesting that the logical solution to the editors’ dilemma would have been to invite McIntyre to be a reviewer. Betts had concurred with my suggestion but then proceeded to … uh … speculate as to why they had not followed this logical path of least resistance. Go figure, eh?!
This is a slight improvement over divergence from Betts’ previous speculation via twitter:
Steve has scientific disagreements, but clearly the reviewers didn’t share those views.
In light of all of the above, it seems to me that whatever their “views” might have been, the reviewers were far from being on the same … uh … page as Steve; i.e. they weren’t even examining that which he has been analyzing. But I digress …
Following a few more tweeted questions and speculations all around, I had eventually responded to Betts:
Hmmm … so review(s) we will never see trump one that we can. […]
EPILOGUE:
In the meantime, McIntyre has identified additional flaws in the statistical underbelly of PAGES2K. However, a few hours ago, Paul Matthews reported, via Climate Audit:
Nature Geoscience has just announced a ‘Journal Club’, involving a live discussion of the PAGES2k paper on Google+ on May 9th.
See their tweets – they are inviting questions.
Also, they say that the PAGES2k is available free to download to anyone from now until May 10th.
Hmmm … I wonder if this is a customary practice for a mere Progress Article?
Oh well … never a dull moment in the climate change game, eh?!
The View From Here 
I wondered about that the word count Steve mentioned it.
Clearly it’s not a Progress Article, except in the lax way it was reviewed.
I wondered about the word count when Steve mentioned it.
That might have been better.
Pingback: These items caught my eye – 30 April 2013 | grumpydenier
Kaufman and paleo peer reviewers ought to be aware that the recent portion of varve data can be contaminated by modern agriculture, as this was a contentious issue in relation to Mann et al 2008 (Upside Down Mann) and Kaufman et al 2009. Nonetheless, Kaufman et al 2013 (PAGES), despite dozens of coauthors and peer review at the two most prominent science journals, committed precisely the same mistake as his earlier article, though the location of the contaminated data is different.
Pingback: Joelle Gergis, Data Torturer « Climate Audit