A conversation with an IPCC coordinating lead author

UPDATE 08/1/2011 11:24 AM: Richard has advised me via E-mail that he’ll be rather busy for the next few days, and will respond as soon as he can.

UPDATE 08/7/2011 09:21 PM: Richard has further advised that he still intends to respond, but hasn’t yet found the time to do so. In the meantime, some new information has come to light regarding “participants” at the WG3 session in May, and I have commented on this in a new post.

Over the last few days, here in my quiet little corner of the blogosphere, I’ve been having a “conversation” with Richard Klein, who has been (and still is!) a Coordinating Lead Author (CLA) of a number of chapters in the Intergovernmental Panel on Climate Change (IPCC)’s various Assessment Reports (AR). His most recent involvement was that of CLA for Working Group (WG) 2, Chapter 18 of AR4 (2007), as well as Contributing Author (CA) to Chapter 17 of the same WG report. In the forthcoming (2013/14) AR5, my understanding – from the available documentation – is that he will be one of two CLAs for WG 2’s “Impacts, Adaptaion and Vulnerability” Chapter 16: “Adaptation opportunities, constraints, and limits”.

Richard was kind enough to offer his thoughts in response to my critique of Damian Carrington’s defence of the IPCC at the U.K. Guardian.

As sometimes happens during the course of such virtual conversations, our discussion has veered beyond the scope of the original post. His most recent reply to me, which can be found in this comment is an example of such divergence. So, I thought it would be useful to move the discussion to this “top post” level.

As some may have gathered from my posts, my primary concerns do not relate to “the science” per se. Because, while I do have my doubts and questions regarding some claims made in the name of science, to which I believe I am entitled, I am in no position to argue “the science” – nor do I have any wish to do so. My concerns pertain to the transparency (or more to the point, the lack thereof) in the practices, processes and procedures underlying the IPCC’s so-called “gold standard” provisioning of “assessment reports”.

In short … they write the lines … and I read between ’em :-) I believe I have fully disclosed my interests, and I would invite Richard Klein (who agrees with my decision to create this new post) to correct me if I have misrepresented his.

So let the dialogue continue!

Richard Klein wrote:

– As is common practice, the list of participants will be included in the report of the session, a draft of which will be available before the next session. See here for reports of previous sessions:
[Richard’s link]

First of all, in this day and age, it seems to me that there is no excuse for such a time lapse between the event and the posting of relevant materials. Forgive my cynicism, but continuation of this “common practice” can serve no useful purpose other than the perpetuation of a fog of distance and time – in the hope that no one will remember (or bother) to check when the details are finally posted!

Furthermore, I have yet to come across any “report” from a WG session which contains a list of participants present. Panel session/meeting reports, yes … but not WG sessions. Even the IPCC draft reports give no indication as to which (or how many) participants were actually present at each day of a “session”. As far as I have been able to ascertain, there are no by-laws for the Panel (or WGs) which one would normally consult in order to determine, for example, what constitutes a quorum at a session.

Principles of work, annexes and appendices etc. are all well and good … but they are far from legally binding – and fail to confer the kind of legitimacy one can verify by consulting legally adopted by-laws. As an aside, from documents that I’ve examined, even when such “principles” are amended, apart from a notation that the overall document has been amended on specific dates, there is no indication as to what might have been amended when. Not to mention that as we already know from the experience with non-peer-reviewed material, the “rules” – such as they are – are not always adhered to! But I digress …

The Feb 23 letter of invitation – which is one of several documents pertaining to the “Eleventh Session of IPCC Working Group III and Thirty-third Session of the IPCC Abu Dhabi, United Arab Emirates, 5 – 8 and 10 – 13 May 2011” – does include the following:

Main agenda item of the Eleventh Session of IPCC Working Group III will be acceptance and approval of the IPCC Special Report on “Renewable Energy Sources and Climate Change Mitigation” and its Summary for Policymakers (SPM). The Session will begin at 10:00 hours on Thursday, 5 May 2011. After action by the Eleventh Session of Working Group III the SPM of the Special Report and the underlying assessment will be forwarded to the IPCC for its action.
[…]
Copies of the following documents are attached to this letter:

1. The provisional agenda of Working Group III Eleventh Session (WG-III:11th/Doc.1);
2. The provisional agenda of the IPCC Thirty-third Session (IPCC-XXXIII/Doc.1);
3. A registration form.
[…]
the participation of delegates with appropriate expertise is vital for the success of the session

Are you seriously suggesting that all 194 countries had delegates with “appropriate expertise” and/or that all 194 countries’ delegates were present for both the 11th session of WG 3 and the 33rd session of the IPCC? I’m sorry but even if you are making such a suggestion (well, actually you have made such a suggestion!), the wording of the annotated provisional agenda for the IPCC clearly suggests otherwise, as I had noted in my previous post:

4. ACCEPTANCE OF THE ACTIONS TAKEN AT THE 11th SESSION OF WORKING GROUP III ON THE SPECIAL REPORT ON RENEWABLE ENERGY SOURCES AND CLIMATE CHANGE MITIGATION (SRREN)

Under this agenda item, the Panel will formally accept the Summary for Policymakers of the SRREN. Section 4.3 of the IPCC procedures stipulates that “for a Summary for Policymakers approved by a Working Group to be endorsed as an IPCC Report, it must be accepted at a Session of the Panel. Because the Working Group approval process is open to all governments, Working Group approval of a Summary for Policymakers means that the Panel cannot change it. However, it is necessary for the Panel to review the Report at a Session, note any substantial disagreements, (in accordance with Principle 10 of the Principles Governing IPCC Work) and formally accept it.” [emphasis added -hro]

With all due respect, Richard, “open to all governments” does not mean – and cannot by any stretch of the English language be construed as implying – “attended by all governments”.

Interestingly, the invitation pdf does not include the actual provisional agendas (agendae?!) that were sent as attachments. This might explain why the provisional agendas I saw on May 9 were different to those I saw on May 14. As I had noted on May 14 [in the text excerpted from my May 9th post]:

Well, the “full” [1,000 page] version of the Intergovernmental Panel on Climate Change (IPCC)’s Working Group III’s “Special Report” urging the expenditure of trillions, won’t be available until May 31. And the IPCC 33rd Session isn’t due to start until tomorrow, May 10 – at least according to their Provisional Agenda. Strangely, unless the “Special Report on Renewable Energy Sources and Climate Change Mitigation (SRREN)” has a secret code name, this “Special Report” does not appear on this Agenda, nor on the “Annotated Provisional Agenda” … ooops …make that the “Provisional Annotated Agenda“.

I also took a look at the “Provisional Agenda” for the 11th Session of Working Group III – submitted by the IPCC Secretariat. (Sorry, no “Annotated” version available for this May 5-8 Session.) And there’s no indication I could detect of any item which would suggest “approval by the IPCC”.

But, one way or another, the busy beavers of WG III have somehow succeeded in gaining the IPCC’s approval of the Summary for Policy Makers (SPM) of this report. […]

To which I then remarked (May 14):

notwithstanding the May 9 Press Release claim that the SPM for the SRREN had been “approved by by member countries of the IPCC” prior to the commencement of its May 10-13 33rd session – I concluded that sometime between May 9 and May 10, the “Provisional” Agendas of Working Group III and the IPCC – both of which bore a date of Feb. 23 – must have been replaced by more currently dated “final” Agendas.

Yet this was not what I found when following the links today. I could have been mistaken when I failed to see any mention of the Approval of the SPM in either the “Provisional Agenda” or the “Provisional Annotated Agenda” for the IPCC, had it been included in either or both of these; but, I really can’t imagine how I could have missed:

4. ACCEPTANCE OF THE ACTIONS TAKEN AT THE 11th SESSION OF WORKING GROUP III ON THE SPECIAL REPORT ON RENEWABLE ENERGY SOURCES AND CLIMATE CHANGE MITIGATION (SRREN)

As a footnote, to add to the confusion – and far from engendering trust in the IPCC or making any contribution to what I would consider as “transparency” – today, when I went to http://www.ipcc.ch/scripts/_session_template.php?page=_33ipcc.htm** in search of the Provisional Agenda for WG 3, it was no longer listed in the documents for the session! It is still available via the links in my May 9 and May 14 posts. FWIW, my recollection is that this now “hidden agenda”(!) was close to the top of the list.

[** 08/1/2011 05:49 PM: even though I had retraced my steps to session 33 docs, I had inadvertently copied and pasted the URL from the page – which does not change when one selects another session from the available dropdown list. I have now changed the URL so that if you choose to follow the link, the list of docs for session 33 will should be what you see.]

Yet another instance, I’m afraid, of what you see (from the IPCC) is (not necessarily) what you (will always) get.

Even the quasi-official rapporteur has noted:

SPECIAL REPORT ON RENEWABLE ENERGY SOURCES AND CLIMATE CHANGE MITIGATION (SRREN)
This issue (IPCC-XXXIII/Doc.20) was considered by plenary on Tuesday. WGIII Co-Chairs Ottmar Edenhofer (Germany), Youba Sokona (Mali), and Ramon Pichs Madruga (Cuba) briefly summarized the SRREN SPM report.
[…]
The Panel endorsed the SRREN SPM approved by WGIII. [emphasis added -hro]

Consequently and most importantly, the only available documentation strongly supports my contention that neither you nor Carrington have any evidence in support of Carrington’s claim (which you averred is “correct“) that the SPM was “approved by all 194 countries.”

But while I’m on the subject of IPCC produced pdfs … There are a number of problems inherent in the information retrievable from the link you provided (and indeed from any of the IPCC pages). Not the least of which is that most of the pdf’s that I’ve looked at lack even basic navigation aids, i.e. “bookmarks”. Most have been generated from Word documents, which – if structured properly by a minimally competent person (i.e. one who does not use a wordprocessor like a typewriter but understands and uses “styles”) – require no additional work to generate a table of contents, i.e. navigational bookmarks from the document headings, on conversion to pdf. Please let me know if I can be of assistance in training those who are responsible for producing the voluminous documentation (such as it is) :-)

There was a (2009) “task group” on the use of electronic technologies for AR5; however, considering the wealth of currently available technologies, the recommendations (as submitted by WG1 Chair, Thomas Stocker – who seems to have a hand in many submissions to the panel) do not give one cause for optimism if one is looking for more than lip service to much needed improvement in accessibility and transparency.

Richard Klein wrote:

– As for consensus, I can’t speak for Mike Hulme but I gave you my take. The consensus that matters in the end is the one among governments, because this then provides a shared knowledge base, for example for international climate policy negotiations outside the IPCC. This was the very reason the IPCC was founded in 1988. My emphasis on political consensus should not be interpreted as me suggesting there is no shared appreciation among the authors of the science of climate change, its impacts and possible response strategies. But as an academic, consensus is not a word I would use to describe such shared appreciation. And as an IPCC author it is my role also to reflect on any conflicting parts of the relevant academic literature, and to find a way of assessing the literature that does justice to the diversity of valid views. Some might call that striving for consensus.

For the record, your take was:

It’s hard to describe this process to someone who hasn’t actually witnessed it, but it is this line-by-line approval process that results in the actual consensus that the IPCC is famous for, and which is sometimes misunderstood. The consensus is not a consensus among all authors about every issue assessed in the report; it is a consensus among governments about the summary for policymakers.

Nonetheless, I’m glad to see that “consensus” is not a word you would use to describe such a “shared appreciation” and I certainly don’t hold you responsible for what others have said! But the point is that – for whatever reason – over the years, much ado has been made by the media, by advocacy groups and by high-profile individuals of a “scientific consensus” and, of course, an “overwhelming scientific consensus“. A few examples …

Greenpeace (July 20, 2010):

Scientific consensus

There is, in fact, a broad and overwhelming scientific consensus that climate change is occurring, is caused in large part by human activities (such as burning fossil fuels), and if left unchecked will likely have disastrous consequences. Furthermore, there is solid scientific evidence that we should act now on climate change, and this is reflected in the statements by these definitive scientific authorities.

Union of Concerned Scientists (March 7, 2011):

Scientific Consensus on Global Warming

Scientific societies and scientists have released statements and studies showing the growing consensus on climate change science. A common objection to taking action to reduce our heat-trapping emissions has been uncertainty within the scientific community on whether or not global warming is happening and if it is caused by humans. However, there is now an overwhelming scientific consensus that global warming is indeed happening and humans are contributing to it.

Naomi Oreskes (in Science 3 December 2004)

The scientific consensus is clearly expressed in the reports of the Intergovernmental Panel on Climate Change (IPCC). Created in 1988 by the World Meteorological Organization and the United Nations Environmental Programme, IPCC’s purpose is to evaluate the state of climate science as a basis for informed policy action, primarily on the basis of peer-reviewed and published scientific literature (3). In its most recent assessment, IPCC states unequivocally that the consensus of scientific opinion is that Earth’s climate is being affected by human activities: “Human activities … are modifying the concentration of atmospheric constituents … that absorb or scatter radiant energy. … [M]ost of the observed warming over the last 50 years is likely to have been due to the increase in greenhouse gas concentrations” [p. 21 in (4)].

IPCC is not alone in its conclusions. In recent years, all major scientific bodies in the United States whose members’ expertise bears directly on the matter have issued similar statements.

Maybe I missed it, but I’m not aware of any IPCC affiliated scientists who have made any serious, highly-visible attempts to correct the record. And, as the saying goes, silence is acquiescence.

Richard Klein wrote:

– The use of non-peer-reviewed sources is indeed subject to the provisions in the IPCC procedures. This means that, contrary to what is sometimes asserted, IPCC chapter can cite so-called ‘grey’ literature if authors consider it of sufficiently high quality and there is no alternative source for that information in the peer-reviewed literature. The review of the Inter-Academy Council recommended the IPCC to strengthen and enforce the existing procedures. This has been done (although not quite in the way the IAC suggested, because governments considered their suggestion to be impractical), and the result is the text you refer to. In preparing the Fifth Assessment Report authors will need to follow these procedures. So hopefully what you see is what you’ll get in the next full report.

I’m not aware that any informed person has “asserted” that the IPCC is prevented from citing grey literature in its reports. But as for “governments” considering the enforcement of the flagging of non-peer-reviewed material as “impractical”, I believe you are mistaken. Assuming that by “governments” you mean the IPCC, their first response (according to the Task Group’s draft and as I had noted in April) was:

4. Sources of Data and Literature

IAC recommendation:

The IPCC should strengthen and enforce its procedure for the use of unpublished and nonpeer-reviewed literature, including providing more specific guidance on how to evaluate such information, adding guidelines on what types of literature are unacceptable, and ensuring that unpublished and non-peer-reviewed literature is appropriately flagged in the report.

IPCC-32 decision:

The Panel agreed with this recommendation. The Panel decided to strengthen the application of its procedures on the use of unpublished and non-peer reviewed literature. It decided to implement this recommendation and further key elements through its procedures and guidance notes. The Panel noted the General Guidance on the Use of Literature in IPCC Reports (contained in IPCC-XXXII/INF.4) as revised in General Guidance on the Use of Literature in IPCC Reports (Appendix 1 of the decision of IPCC-32) which addresses the related aspects in the IAC recommendations and decided to endorse them as a Guidance Note. The Panel urges the Co-Chairs of Working Group I, II, III and TFI to take any necessary steps to ensure that this guidance note is applied in the development of IPCC reports. [link added -hro]

And as I had noted, on April 23, this “General Guidance on the Use of Literature in IPCC Reports (contained in IPCC-XXXII/INF.4) as revised in General Guidance on the Use of Literature in IPCC Reports” is as follows:

General Guidance on the Use of Literature in IPCC Reports

Introduction

The TSUs of all three IPCC Working Groups drafted this guidance document to recall the Principles Governing IPCC Work, particularly the “Procedure for using non-published/non-peer reviewed sources in IPCC Reports”, […]

Guidance on the use of non-published/non-peer-reviewed (“grey”) literature

Extract from Annex 2 of Appendix A to the Principles Governing IPCC Work:

Procedure for using non-published/non-peer-reviewed sources in IPCC Reports
[…]
5. Treatment in IPCC Reports

Non-peer-reviewed sources will be listed in the reference sections of IPCC Reports. These will be integrated with references for the peer-reviewed sources.¹ These will be integrated with references to the peer reviewed sources stating how the material can be accessed, but will be followed by a statement that they are not published.
[…]
¹ Non-published sources also will be listed in the reference sections of IPCC Reports

Please note that it was the Technical Support Units (TSUs) who drafted this revised “Guidance Note”. And as I had also observed on April 23:

Of the thirteen countries and two “Office Holders” who provided feedback on “Sources of Data and Literature”, most were remarkably silent about the TG’s recommendation that the “rule” be made to disappear. The three who did comment were far from being “shocked and appalled”

Yet sometime between their revision to the “Guidance Note” and the drafting of the Task Group’s recommendation to the Panel (“governments”), the TSUs (whom the Task Group “consulted”) evidently determined that this longstanding (but rarely practiced) rule had become “impractical”.

Setting aside all the twists and turns that obviously occurred during the evolution of this “disappearance” and acceptance thereof by “governments” … what could possibly be “impractical” about flagging “non-journal” literature? Surely when the authors are drafting their respective contributions to the text, they are required to cite the source material for their claims – and provide the full source to which they are referring – as I see that Richard Tol does with his drafts. What would be so “impractical” about adding a tiny little flag – to the inline citation and/or the actual reference source – such as, oh … I dunno … “NPR”, for example?

Richard Klein wrote:

You provide a link to Donna Laframboise’s quantitative analysis of grey literature, and I was interested to see that the chapter I led in the Fourth Assessment Report received an F, with 44% of the references not peer-reviewed. However, that includes books published by academic publishers, reports by the World Bank, OECD and other reputable international organisations, as well as previous IPCC reports. I see no reason not to cite this kind of ‘grey’ literature again in the Fifth Assessment Report; indeed not doing so would mean we’d overlook an important part of the available knowledge.

First of all, in the interest of full disclosure – and in case you were unaware of this – I was one of the Citizen Auditors who participated in Donna’s study. I would not characterize it as you have above. Rather, the purpose of this study was to determine whether or not the frequently repeated claim to the effect that the IPCC reports are “all on the basis of peer-reviewed literature” could be justified. The answer that this study provided is an unequivocal “No, it cannot”.

It is worth noting that the IAC review cited an earlier study of the Third Assessment Report in which the results were similar (p. 16):

An analysis of the 14,000 references cited in the Third Assessment Report found that peer-reviewed journal articles comprised 84 percent of references in Working Group I, but comprised only 59 percent of references in Working Group II and 36 percent of references in Working Group III (Bjurström and Polk, 2010).

I don’t know if you’ve taken a look at AccessIPCC, a computerized tagging of citations and references in all 44 Chapters of AR4, using the same criteria for non-journal (i.e. non peer-reviewed) material. There are a number of other tags as well. Here’s the summary of what we found in Chapter 18, for which you were a CLA:

Adapted from AccessIPCC's WG2-CH18 Fourth Assessment Report - Objectively Uniformly Tagged

 

And speaking of “peer review”, the IAC and your chapter … It has been claimed that the IPCC reports are “peer reviewed”. Indeed much ado is often made of the number of reviewer comments received. However, this peer review is quite different from that conducted prior to publication in a “peer-reviewed journal”. The IAC notes (pp. xiii-xiv):

Peer review is an important mechanism for assuring the quality of reports. IPCC’s peer review process is elaborate, involving two formal reviews and one or more informal reviews of preliminary text.

[…]

However, the Lead Authors have the final say on the content of their chapter.

What the IAC did not note, but which is commonly known, is that the names of the comment authors are attached to their respective comments and that the “responses” to the comments are far from readily quantifiable. Notwithstanding specific instructions to those who compose the responses “on behalf of the chapter team”, such responses are very inconsistent. In AccessIPCC we tried to quantify the number of “Accepted”, “Rejected”, “Noted” etc on the Second Order Draft of AR4. The best that we could do was to quantify those that had been unambiguously “Accepted”. Here’s what we found for your Chapter 18:

Adapted from AccessIPCC's WG2-CH18

A Helpful Hint from Hilary™ that you might pass on to the powers that be for AR5: Add another column to the Word doc sent to the “chapter teams” which contains a dropdown list from which the responder (“on behalf of the chapter team”) will select the category of response (Accepted, Rejected etc). This will make life easier for the Review Editors to carry out their responsibilities, lessen the appearance of being a “knowledge monopoly” – and make the process slightly more transparent when the review comments are (finally!) made public!

And on that constructive note, I shall end for now and await Richard’s reply :-)