About a month ago, I had commented on a recommendation (by the Intergovernmental Panel on Climate Change (IPCC)’s Task Group on Procedures) that the current (albeit rarely practiced) “rule” that non-peer-reviewed literature is supposed be appropriately flagged should be dropped.
As I mentioned in my last post, I sent off an E-mail enquiry in an attempt to ascertain whether or not this particular recommendation had been approved, or whether the InterAcademy Council (IAC)’s recommendation that this “rule” should be enforced would be (conveniently?!) ignored by the IPCC. Here’s the E-mail I had sent to the two media contacts listed on the IPCC’s Press Release:
From: Hilary Ostrov [mailto:xxxxx]
Sent: Saturday, May 14, 2011 5:36 PM
To: ‘Nick Nuttall'; ‘Rockaya Aidara’
Subject: Follow-up re IPCC Press Release of May 13, 2011
I’m not sure which of the two of you would have primary responsibility for responding to my questioms below, so I’ve taken the liberty of writing to both of you. My questions pertain to the “Notes to Editors” section of http://www.ipcc.ch/news_and_events/docs/ipcc33/PRESS_RELEASE_Outcomes_abu_dhabi_13_may.pdf; in particular, the following excerpt:
Scientific Uncertainties, Correction of Errors and ‘Grey’ Literature
The IAC recommended that the IPCC strengthen procedures on how all literature is reviewed and considered; [...]
The 33rd Session adopted decisions relating to these including:-
• Lead authors will consider the range of scientific, technical and socio-economic views and documents, even if there is no consensus on view on the findings, as long as they are scientifically and technically valid
• However, magazines and newspapers are in principle not valid sources and that blogs, social networking sites and broadcast media are not acceptable sources of information for IPCC reports [emphasis added -hro]
As I’m sure you are aware, the IAC’s specific recommendation:
The IPCC should strengthen and enforce its procedure for the use of unpublished and non-peer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report. [emphasis added -hro]
was endorsed by the IPCC at the 32nd session. Yet the draft recommendations of the Task Group on Procedures included the following:
The TG, after consulting the WG /TFI TSUs, found that the implementation of this IAC recommendation regarding the appropriate flagging of unpublished and non-peer reviewed literature would not be practical. [emphasis added -hro]
The Procedures Task Group failed to provide any justification for this conclusion – and did not seem to be aware that this was far from being a “new” recommendation from the IAC; but rather that it is a longstanding IPCC rule/principle.
This Task Group further recommended that “ANNEX 2: PROCEDURE ON THE USE OF LITERATURE IN IPCC REPORTS” be “rewritten” in a way that specifically excludes the following (longstanding) provision of http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf (p.14):
PROCEDURES FOR THE PREPARATION, REVIEW, ACCEPTANCE, ADOPTION, APPROVAL AND PUBLICATION OF IPCC REPORTS
Adopted at the Fifteenth Session (San Jose, 15- 18 April 1999) amended at the Twentieth Session (Paris, 19-21 February 2003), Twenty-first Session (Vienna, 3 and 6-7 November 2003), and Twenty-Ninth Session (Geneva, 31 August – 4 September 2008)
PROCEDURE FOR USING NON-PUBLISHED/NON-PEER-REVIEWED SOURCES IN IPCC REPORTS
5. Treatment in IPCC Reports
Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources. These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published. [emphasis added -hro]
As far as I was able to ascertain, the May 13 Press Release was silent on this particular aspect of the IPCC’s acceptance/approval of the Task Group’s recommendations. Could one of you, therefore, kindly confirm whether or not the IPCC has accepted the recommendation of the Task Group regarding the “appropriate flagging” of “unpublished and non-peer-reviewed literature”? In other words, will the current Item 5 of Annex 2 (as noted immediately above) be enforced or not in future reports under the auspices of the IPCC?
In keeping with the IPCC’s dedication to transparency and “rapid response” vis a vis communications, I trust that I shall be favoured with your prompt reply to the questions in this E-mail.
The good news is that I did receive a prompt reply from one of the two letting me know that he had:
“sought advice on your question. Hope to have a response today. Fingers crossed.”
Of course I thanked him, and indicated that I looked forward to receiving the response. Then the not so good news – About an hour later, I received a reply:
“Ok I have asked for our experts to get a reply back within next 48 hours…let’s see, fingers crossed!”
Based on past experience, I was prepared to wait and see if this would be the longest 48 hours evaah; but in the meantime [h/t David Wojick via comment at Judith Curry's], I discovered the International Institute for Sustainable Development (IISD)’s Earth Negotiations Bulletin (ENB) Vol. 12 No. 500.
This is a 12 page pdf, btw, that is quite well-written and – for the most part – actually provides a neutral summary of the IPCC’s proceedings – as well as some insighful analysis. I recommend reading the whole thing. Believe me, it’s far more informative than anything one is likely to find on the IPCC site! But I digress …
Their bulletin answers my question (p. 4):
On sources of data and literature, delegates addressed the blurry lines between peer-reviewed, grey literature and other sources, including references to sources such as the International Energy Agency and World Bank reports. The group agreed not to flag information derived from grey literature in the reports and focus instead on ensuring the high quality of all information, placing priority on peer-reviewed literature. [emphasis added -hro]
followed by (p. 5):
• On sources of data and literature, the Panel replaces the annex on “Procedure for using non-published/non-peer reviewed sources in IPCC reports” with a new annex entitled “Procedure on the use of literature in IPCC reports.” The revised procedures place emphasis on the quality of all cited literature, with priority given to peer-reviewed scientific, technical and socioeconomic literature, if available, and detail additional responsibilities for authors, review editors, WG/TFI Co-Chairs, and the IPCC Secretariat in this regard.
A disappointing but not unexpected result, that certainly gives rise to a number of questions. However, I doubt that we shall be hearing any more claims about how the IPCC reports are “all peer reviewed”
Oh, well … I guess I should let the IPCC “media” contacts know that I now have the answer to my question. I do hope those ‘experts’ haven’t gone to too much trouble ;-)