IPCC’s use of grey literature: To flag or not to flag, that is the question

In a recent post, I had noted that the Task Group (TG) charged with developing recommendations pertaining to the Intergovernmental Panel on Climate Change (IPCC)’s “Procedures” – pursuant to the 2010 review by the InterAcademy Council (IAC) – had decided to “disappear” the IPCC’s rule regarding the appropriate flagging of grey literature, on the grounds that:

implementation of this IAC recommendation regarding the appropriate flagging of unpublished and non-peer reviewed literature would not be practical. [emphasis added -hro]

The choice of wording in the above is somewhat interesting. If one didn’t know better, one would be inclined to conclude that the IAC’s recommendation regarding “appropriate flagging” was something new – rather than the … uh … reframing of a longstanding (albeit unpracticed far more often than not) IPCC “rule”.

Here’s the full IAC recommendation:

The IPCC should strengthen and enforce its procedure for the use of unpublished and non peer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report.

The “IPCC-32 decision” was as follows:

The Panel agreed with this recommendation. The Panel decided to strengthen the application of its procedures on the use of unpublished and non-peer reviewed literature. It decided to implement this recommendation and further key elements through its procedures and guidance notes. The Panel noted the General Guidance on the Use of Literature in IPCC Reports (contained in IPCC-XXXII/INF.4) as revised in General Guidance on the Use of Literature in IPCC Reports (Appendix 1 of the decision of IPCC-32) which addresses the related aspects in the IAC recommendations and decided to endorse them as a Guidance Note. The Panel urges the Co-Chairs of Working Group I, II, III and TFI to take any necessary steps to ensure that this guidance note is applied in the development of IPCC reports. [emphases added -hro]

It’s difficult to ascertain whether or not the Procedures TG actually reviewed the contents of “IPCC-XXXII/INF.4“. This 8 page October 2010 document, “REVIEW OF THE IPCC PROCESSES AND PROCEDURES Notes on (sic) the Informal Task Group on Procedures (Submitted by Mr Thomas Stocker, Co-Chair of Working Group I on behalf of the IPCC Chair, Vice-Chairs and Co-Chairs)”, preceded the “official” TG report and contains the following (pp. 2-3):

A. Procedures for preparing IPCC reports and their implementation

The IPCC Procedures (Appendix A to the Principles) provide a concise description, but there are some aspects where the authors and/or Review Editors (REs) may benefit from more guidance.
A1. Specific briefings have been given at LA meetings for the SR authors and REs on:

• Use of sources not based on peer-reviewed journal contributions (referred to in the IPCC Procedures as non-published/non-peer-reviewed literature);
A4. Non-published/non-peer-reviewed literature: a guidance note has been prepared by the WG TSUs for the ongoing Special Reports (see Annex). This will be developed further by the TSUs and the IPCC Secretariat if necessary.

This “guidance note” would appear to be contained in (p. 6):

General Guidance on the Use of Literature in IPCC Reports


The TSUs of all three IPCC Working Groups drafted this guidance document to recall the Principles Governing IPCC Work, particularly the “Procedure for using non-published/non-peer reviewed sources in IPCC Reports”, […]

Guidance on the use of non-published/non-peer-reviewed (“grey”) literature

Extract from Annex 2 of Appendix A to the Principles Governing IPCC Work:

Procedure for using non-published/non-peer-reviewed sources in IPCC Reports
5. Treatment in IPCC Reports

Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources.1 These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published.
1 Non-published sources also will be listed in the reference sections of IPCC Reports.

The remit of the “Procedures” TG was to develop recommendations on 8 matters (including “Sources of Data and Literature”), so perhaps they had too much on their plate to give thoughtful consideration to all issues within their terms of reference.

For the record, the recommendations were “Submitted by the [Procedures] Task Group Co-chairs, Oyvind Christophersen, Norway; Eduardo Calvo, Peru; and Rapporteur, Leo Meyer, The Netherlands” and the following 20 countries were (presumably) involved in the deliberations of this TG: Australia, Bolivia, Brazil, Canada, Chile, Germany, India, Iran, Maldives, Netherlands, New Zealand, Niger, Norway, Peru, Saudi Arabia, South Africa, Swaziland, Switzerland, Thailand, and USA.

All the “Procedures” TG’s recommendations are found in a 267 page .pdf compiled by the IPCC Secretariat. This document contains the recommendations of all four TG’s as well as a section of “Comments Organized by Countries and IPCC Office Holders as decided by P-32” and a third section, “Comments Organized by Task Group”.

In my previous post and in this one, I’m focussing only on the “Procedures” TG’s recommendations on “Sources of Data and Literature”. Thirteen countries provided responses/feedback (i.e. “Comments”) on these recommendations – Austria, Belgium, Canada, Germany, India, Japan, Malaysia, Netherlands, Norway, Russian Federation, Sweden, UK and USA – as did 2 of the “Office Holders as decided by P-32” (Thomas Stocker, Co-chair, WG I and Chris Field, Co-chair, WG II).

Because those who commented have, of course, referred to specific paragraphs/items of the TG’s recommendations, I have compiled a .pdf extract which will make it easier for the reader to navigate between the “comments” and the actual text pertaining to “Sources of Data and Literature”.

Of the thirteen countries and two “Office Holders” who provided feedback on “Sources of Data and Literature”, most were remarkably silent about the TG’s recommendation that the “rule” be made to disappear. The three who did comment were far from being “shocked and appalled”:


The reasons for the refusal of the IAC recommendation must be strengthened and explained more carefully as the issue of grey literature has been a major topic of the public discussion.


We agree with the new proposed Annex 2 of the Procedures (‘Procedure for using nonpublished/non-peer-reviewed sources in IPCC reports’). However, the IAC recommendation for unpublished and non-peer-reviewed literature to be appropriately flagged in the IPCC Report should still be considered if any relevant source of data and literature is deemed important/useful. This “flagging” procedure shall be considered in the implementation after taking into consideration to the additional procedures[…]


The TG says: The TG, after consulting the WG /TFI TSUs, found that the implementation of this IAC recommendation regarding the appropriate flagging of unpublished and non-peer reviewed litterature would not be practical. Would it be wise to include an explanation of why this is not practical?

YMMV, but I don’t find these objections to be particularly “strenuous”. I also find it ironic that Stocker was prepared to overlook the recommendations of the “Informal” task group [in which flagging was not considered “impractical”] which he had submitted.

It is also somewhat difficult to reconcile “Office Holder” Stocker’s introductory: “The TSUs of all three IPCC Working Groups drafted this guidance document […]” with the “Procedures” TG’s: ” … after consulting the … TSUs …”. Nor do Stocker’s “comments” on the TG’s recommendations make any mention of the longstanding “rule” regarding the use of non-peer-reviewed material in IPCC assessment reports.

Makes one wonder … do these people’s left hands know what their right hands are doing?! Or do they (not unlike Pachauri) wear so many hats that they can’t remember from one meeting to the next which one they’re supposed to be wearing?!

What a bureaucratic web they seem to weave
Perhaps ’tis themselves whom they deceive.

Which still leaves us with … IPCC and grey literature: to flag or not to flag, that is the question.

One thought on “IPCC’s use of grey literature: To flag or not to flag, that is the question

  1. While this is an important issue, from my perspective item 5 – considering the full range of views – is at least equally, if not more important. The IPCC response I find encouraging but entirely hypocritical as it seems to this outsider that there was an unwritten agenda to exclude views which conflicted with the ‘mainstream’ orthodoxy.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s