About a month ago, I had commented on a recommendation (by the Intergovernmental Panel on Climate Change (IPCC)’s Task Group on Procedures) that the current (albeit rarely practiced) “rule” that non-peer-reviewed literature is supposed be appropriately flagged should be dropped.
As I mentioned in my last post, I sent off an E-mail enquiry in an attempt to ascertain whether or not this particular recommendation had been approved, or whether the InterAcademy Council (IAC)’s recommendation that this “rule” should be enforced would be (conveniently?!) ignored by the IPCC. Here’s the E-mail I had sent to the two media contacts listed on the IPCC’s Press Release:
Sent: Saturday, May 14, 2011 5:36 PM
To: ‘Nick Nuttall’; ‘Rockaya Aidara’
Subject: Follow-up re IPCC Press Release of May 13, 2011Hello,
I’m not sure which of the two of you would have primary responsibility for responding to my questioms below, so I’ve taken the liberty of writing to both of you. My questions pertain to the “Notes to Editors” section of http://www.ipcc.ch/news_and_events/docs/ipcc33/PRESS_RELEASE_Outcomes_abu_dhabi_13_may.pdf; in particular, the following excerpt:
Scientific Uncertainties, Correction of Errors and ‘Grey’ Literature
The IAC recommended that the IPCC strengthen procedures on how all literature is reviewed and considered; […]
The 33rd Session adopted decisions relating to these including:-
[…]
Lead authors will consider the range of scientific, technical and socio-economic views and documents, even if there is no consensus on view on the findings, as long as they are scientifically and technically valid
However, magazines and newspapers are in principle not valid sources and that blogs, social networking sites and broadcast media are not acceptable sources of information for IPCC reports [emphasis added -hro]
As I’m sure you are aware, the IAC’s specific recommendation:
The IPCC should strengthen and enforce its procedure for the use of unpublished and non-peer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report. [emphasis added -hro]
was endorsed by the IPCC at the 32nd session. Yet the draft recommendations of the Task Group on Procedures included the following:
The TG, after consulting the WG /TFI TSUs, found that the implementation of this IAC recommendation regarding the appropriate flagging of unpublished and non-peer reviewed literature would not be practical. [emphasis added -hro]
The Procedures Task Group failed to provide any justification for this conclusion – and did not seem to be aware that this was far from being a “new” recommendation from the IAC; but rather that it is a longstanding IPCC rule/principle.
This Task Group further recommended that:
“ANNEX 2: PROCEDURE ON THE USE OF LITERATURE IN IPCC REPORTS” be “rewritten” in a way that specifically excludes the following (longstanding) provision of http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf (p.14):
PROCEDURES FOR THE PREPARATION, REVIEW, ACCEPTANCE, ADOPTION, APPROVAL AND PUBLICATION OF IPCC REPORTS
Adopted at the Fifteenth Session (San Jose, 15- 18 April 1999) amended at the Twentieth Session (Paris, 19-21 February 2003), Twenty-first Session (Vienna, 3 and 6-7 November 2003), and Twenty-Ninth Session (Geneva, 31 August – 4 September 2008)
[…]
ANNEX 2 PROCEDURE FOR USING NON-PUBLISHED/NON-PEER-REVIEWED SOURCES IN IPCC REPORTS
[…]
5. Treatment in IPCC Reports
Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources. These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published. [emphasis added -hro]
As far as I was able to ascertain, the May 13 Press Release was silent on this particular aspect of the IPCC’s acceptance/approval of the Task Group’s recommendations. Could one of you, therefore, kindly confirm whether or not the IPCC has accepted the recommendation of the Task Group regarding the “appropriate flagging” of “unpublished and non-peer-reviewed literature”? In other words, will the current Item 5 of Annex 2 (as noted immediately above) be enforced or not in future reports under the auspices of the IPCC?
In keeping with the IPCC’s dedication to transparency and “rapid response” vis a vis communications, I trust that I shall be favoured with your prompt reply to the questions in this E-mail.
Thank you,
Hilary Ostrov
[sig]
The good news is that I did receive a prompt reply from one of the two letting me know that he had:
“sought advice on your question. Hope to have a response today. Fingers crossed.”
Of course I thanked him, and indicated that I looked forward to receiving the response. Then the not so good news – About an hour later, I received a reply:
“Ok I have asked for our experts to get a reply back within next 48 hours…let’s see, fingers crossed!”
Based on past experience, I was prepared to wait and see if this would be the longest 48 hours evaah; but in the meantime [h/t David Wojick via comment at Judith Curry’s], I discovered the International Institute for Sustainable Development (IISD)’s Earth Negotiations Bulletin (ENB) Vol. 12 No. 500.
This is a 12 page pdf, btw, that is quite well-written and – for the most part – actually provides a neutral summary of the IPCC’s proceedings – as well as some insighful analysis. I recommend reading the whole thing. Believe me, it’s far more informative than anything one is likely to find on the IPCC site! But I digress …
Their bulletin answers my question (p. 4):
On sources of data and literature, delegates addressed the blurry lines between peer-reviewed, grey literature and other sources, including references to sources such as the International Energy Agency and World Bank reports. The group agreed not to flag information derived from grey literature in the reports and focus instead on ensuring the high quality of all information, placing priority on peer-reviewed literature. [emphasis added -hro]
followed by (p. 5):
• On sources of data and literature, the Panel replaces the annex on “Procedure for using non-published/non-peer reviewed sources in IPCC reports” with a new annex entitled “Procedure on the use of literature in IPCC reports.” The revised procedures place emphasis on the quality of all cited literature, with priority given to peer-reviewed scientific, technical and socioeconomic literature, if available, and detail additional responsibilities for authors, review editors, WG/TFI Co-Chairs, and the IPCC Secretariat in this regard.
A disappointing but not unexpected result, that certainly gives rise to a number of questions. However, I doubt that we shall be hearing any more claims about how the IPCC reports are “all peer reviewed”
Oh, well … I guess I should let the IPCC “media” contacts know that I now have the answer to my question. I do hope those ‘experts’ haven’t gone to too much trouble ;-)
The View From Here 
Unbelievable!
Hey! Are you questioning my carefully documented account of this travesty? ;-)
But seriously … there are many elephants left in the IPCC data and literature room … and far too much … uh …”discretion” apparently left in the hands of those who wield the mice and keyboards of the IPCC reports.
Apart from the rather deafening “silence” on the use of material from advocacy organizations (compared to, oh … I dunno … the explicitly declared unacceptability of blogs as a resource), if my reading of the draft Conflict of Interest policy – and description contained in the IISD’s ENB report – is correct, there is the “partner” elephant in the CoI room: the allusive whisper – very much subject to (convenient?!) interpretation – on the specific matter of CLA, LA, [and in some instances, perhaps CA and/or RE] authorship of material cited in chapters for which they have writing/oversight responsibility.
My guess is that they will find a way to bypass this by invoking the “team-writing defense” – as articulated in the Muir Russell report.vis a vis Jones & Briffa.
In short, the IPCC may have some new, improved cloaks … but for all intents and purposes, AR5 is likely to be, well, just as bad as we thought!
All the problems the previous IPCC report had, was due to use of literature from grey literature from advocacy fronts and environmental pressure groups, or reports from prestigious non-governmental organizations authored by activists.
The culmination of the public outcry at the blatant lying about peer-reviewing, at the IPCC, was the IAC review
In the The IAC review and and in other statements that emanated from the IPCC, post Glaciergate, it was decided, much to everyone’s surprise, that the IPCC would not be able to totally let go of quoting from non-peer-reviewed grey literature sources.
The IAC report, in turn, asked only such sources to be flagged in the report. Its recommendations were already a cave-in.
And now, these people have gotten together and unilaterally decided to do away with this meagre, and pathetic requirement!
Bingo! And speaking of the IAC they still haven’t come through with the missing questionnaire responses (or an an explanation as to why out of 400 they have compiled only 232). And, since they’ve not been in the habit of responding to emails or making any announcements* to interested stakeholders, I just checked again … and all that’s there is the same 678 page pdf.
* the notable exception being the recently posted:
I’m sure that you’re as impressed with this as I am!
But, you know … if one were inclined towards the conspiracy theory of history one might be tempted to conclude that this long – and deafening – silence from the IAC (broken only by the above May 13 milquetoast “comments”) could be indicative of a deliberate withholding of “data” that – if examined closely – would reveal that the IAC’s report should have been far more damning than it was.
Then again, considering the IPCC’s well-orchestrated response on the simple (and easily implemeneted!) matter of flagging non-peer-reviewed sources, perhaps a more damning report from the IAC would have made little, if any, difference.
Who knows, eh?!
Pingback: IPCC: Screw the Rules « NoFrakkingConsensus
The bulletin you cite above is, indeed, a great resource. Unfortunately, it also provides a good example of how the myth that the IPCC relies solely on peer-reviewed material has been promulgated. If you take a look at its boilerplate description of the IPCC on page 1, it reads:
This publication has repeated this very claim every year for more than a decade (I recently documented this on my Quotes About the IPCC page).
05/18/2011 Sorry, Donna … I wasn’t “censoring” you but it wasn’t until now that I noticed this had ended up my spam-trap. But thanks for highlighting this shortcoming of the ENB reports -Hilary
Pingback: IPCC: Peer-review? what peer-review? « Shub Niggurath Climate
IPCC: Peer-review? What peer-review?
Pingback: Secret societies upset by sunshine | JunkScience Sidebar
Pingback: Greenpeace in the IPCC: Why the surprise? « Shub Niggurath Climate
Pingback: The IAC Report One Year Later « NoFrakkingConsensus
Pingback: What Would a Bad Job Look Like? | NoFrakkingConsensus