IPCC monopoly gatekeeper rules very “elastic”

I’ve been working with Australia’s Peter B. on various aspects of the program he is developing which will allow anyone to quickly get the big picture (and drill down into many details) of the Intergovernmental Panel on Climate Change (IPCC)’s “gold standard” 4th Assessment Report (aka FAR aka AR4). My weekend challenge has been to attempt to identify ways of quantifying the responses to reviewer comments. It really should have been an easy task.

Here’s screen capture of the “instructions” to chapter team responders:

Instructions to 'chapter team' regarding responses to Reviewer Comments

In attempting to quantify “chapter team” responses to the 34,000+ Reviewer Comments on the Second Order Draft, the first thing one notices is that the hallmark of inconsistency is indelibly stamped throughout this particular part of this Nobel award-winning “robust – almost to a fault” process.

Nonetheless, I did find it somewhat surprising that on a first pass, so to speak, of the 34,000+ Reviewer Comments a mere 7,664 (22%) responses were unambiguously identifiable as “Accepted”.

It is possible that not all the “chapter team” responders read the instructions they were given. It’s also possible, in some instances, that they may have received no instructions. One could add to the above the 668 that were unambiguously (well, after a careful process of prior elimination) identifiable as “Agreed” (even though “Agree(d)” was not an option). Assuming that the IPCC would view “Agree(d)” as “Accepted” (which may or may not be a reasonable assumption), this would bring the Accepted total to 8,332 (23.9%). Hmmm … not much of an improvement.

As I had noted a few months ago, they seem to make up the rules as they go along. They also change the rules as they go along. Here’s an example:

AR4-WG1-CH06 Comment No. 6-1115

It seems very unwise to me to waive IPCC WG1 policies on publication guidelines, especially for lead authors. For example, Osborn and Briffa 2006 did not meet the December deadline for being published or in print; it was not even mentioned in the First Draft nor was it available from TSU as part of the First Draft process. Other citations in the chapter did not meet the December deadline for being published or in press as at the December draft meeting (Osborn and Briffa 2006; Wahl et al 2006; Wahl and Ammann 2006; Hegerl et al accepted); several did not meet the February drop-dead date for providing TSU with a preprint (Wahl and Ammann 2006; Hegerl et al accepted). The version of Wahl and Ammann 2006 as accepted differeed dramatically from the version provided to TSU for both the First Order and Second Order Drafts, notably in respect to the inclusion of their calculation of MBH verification statistics confirming the results of McIntyre and McKitrick showing failure of MBH verification statistics that had previously been denied.. [Stephen McIntyre (Reviewers comment ID #: 309-12)]

The ‘chapter team’ response:

Rejected, guidlines used for preparing the draft have been followed and new guidlines do not pose problems.

There can be no doubt that this comment was Rejected (by someone who does not know how to spell “guidelines”). Yet, consider the response to another comment from a different chapter in the same Working Group report:

AR4-WG1-CH01 Comment No. 1-514

There are no firm theoretical/computational basis behind the statement on H2O / CO2 60% / 26% RF ratio. According to recent high-resolution line-by-line computations of the monochromatic fluxes of the ERBE and TIGR datasets, based on the exact analytic solution of the Schwarzchild equation, Ferenc Miskolczi in his referred paper found 9% for clear-sky CO2, and only 0.5 C ground temperature rise for 2xCO2. Details in the forthcoming TellusB article. [MIKLOS ZAGONI (Reviewers comment ID #: 300-5)]

The ‘chapter team’ response:

Figures are no longer cited. Forthcoming articles cannot be cited.

There are two problems with this response: It appears that this Reviewer’s Comment was “partially” accepted, so the response should have indicated this. But more importantly, the ‘chapter team’ responder does not seem to be aware of the “new guidlines” (sic).

Or is it simply the case that <gasp> the new, improved “guidlines” (sic) were only applicable to those papers that did not detract from the party line?

The IPCC is actually quite fond of taking leaps back to the future – when it suits their purpose, as Donna Laframboise has noted in a recent series of posts:

Does the report restrict itself to research completed by late 2005? Think again. The Stern Review is cited 26 times across 12 chapters even though that document wasn’t released until late October 2006. A particular issue of an academic journal is cited 39 times even though that issue wasn’t published until May 2007. A research paper is cited by three IPCC chapters across two working groups even though it wasn’t accepted for publication until May 2008 and didn’t appear in print until November of that year.

If this weren’t such an important matter these irregularities would surely be raw material for a comedic farce […]

Indeed. CRUdite Mike Hulme believes that “the idea of climate change is [very] plastic”. Such a “plastic” idea must require “strict rules” for gatekeepers that are very elastic.