Of peers, reviews, missing data and … communication

The American Academy for the Advancement of Science (AAAS) has been holding its annual meeting over the last few days. As with other “Big Science” organizations, one of the “hot topics” relates to communication of climate science. Seems that after 20+ years of (for all intents and purposes) having a captive media to relay their increasingly scary “settled science” consensus messages – without challenge – the crème de la crème of climate scientists are finding that they’re having difficulty “communicating” with the public.

Bottom line from their perspective: it’s all because of the big, bad skeptics (to whom, if they had their druthers, the journalists should be paying absolutely no attention).

Some of the speakers on this topic, evidently, are “blaming” the media rather than their own considerably less than optimal “peer reviewed” cover stories and/or their inadequate communication skills. As Dr. Judith Curry has observed:

Climate scientists got lazy and thought communicating that there was a consensus among the scientists was sufficient to convince the public. Now they seem annoyed that this didn’t work and are blaming the journalists

No less a “great communicator” of “climate science” than Intergovernmental Panel on Climate Change (IPCC) Chair, Rajendra K. Pachauri began sending out “blame the media” trial balloons when, on the heels of his May 2010 presentation to the InterAcademy Council (IAC), he did an about-face on his previously entrenched position on non-peer-reviewed literature (which a few months earlier he would have consigned to the “dustbin”) by declaring that:

[T]he media and other sections of society had misunderstood the role of such information, labelling it grey literature, “as if it was some form of grey muddied water flowing down the drains”.

All of these folks could have avoided all the hand-wringing and blame-shifting (not to mention whinging, whining and back-peddaling) if they’d only been paying attention at the IPCC’s Bali meeting in October 2009. That’s when Joseph Alcamo (an IPCC alumnus and at that point the “Chief Scientist” of the UNEP, a parent organization of the IPCC) told them in no uncertain terms:

[A]s policymakers and the public begin to grasp the multi-billion dollar price tag for mitigating and adapting to climate change, we should expect a sharper questioning of the science behind climate policy.

What on Gaia’s green earth is it about the words “multi-billion dollar price tag” and/or “expect a sharper questioning of the science” that these world-renowned “experts” are having such difficulty understanding?!

Speaking of “sharper questioning” … this is not an attribute one could ascribe to the proceedings of the so-called enquiries pursuant to Climategate (the authors of which could certainly not be accused of failing to get the message that no attention should be paid to anyone who was not an “approved” climate science messenger!)

But there was one report, that of the IAC which was charged with investigating the processes and procedures of the IPCC, whose authors clearly did engage in some “sharper questioning”. The duly constitued committee that produced the report on the IPCC did not ignore the contrary (and even perhaps “inconvenient”) voices of those who weren’t exactly enamoured of the IPCC (and/or its “in crowd”). And what’s even more amazing is that (eventually) they even shared their data, so that their findings and conclusions could be verified. Well, at least some of their data …

You see, the preface to the IAC’s 678 page compilation of responses to a questionnaire they had made available to all and sundry reads as follows:

Note: All questionnaire responses received during the Committee’s writing and analysis period are given below. They have been formatted for consistency and edited only to remove identifiers, including specific roles held in the various assessments. Where identifiers could not be removed without signficantly altering the text, permission to print the original response was obtained from the author. [emphasis added -hro]

As I had observed in December, there are 232 responses in this compilation (and some of them are quite telling, as Donna Laframboise has noted in a recent series of articles). According to the math I was taught, “all questionnaire responses” does not compute with the IAC report’s observation … (pp. 5-6):

The questionnaire was also posted on the Committee’s website so the general public could comment. More than 400 individuals, listed in Appendix C, provided input. The prevailing views of the questionnaire respondents about the various steps in the IPCC assessment process are summarized in this report and a compilation of all of the responses, with identifiers removed, is available from the IAC. [emphasis added -hro]

This strongly suggests that there are at least 168 missing responses. Mind you, getting the 232 was somewhat akin to pulling teeth. Although advertised as being available when the pre-publication version of the IAC report was posted (circa August 30, 2010) the 232 responses were finally (and very, very quietly) posted circa December 20, 2010.

Rather than invite yet another encore performance of sounds of silence from the IAC secretariat and/or the 5 media contacts, I was hoping that the IAC review committee chair, Dr. Shapiro, might be more responsive. So I wrote to him, on Feb. 10, enquiring about the mathematical discrepancy. I regret to advise that he may now be singing from the IAC secretariat songbook :-(

But, while I wait (and wait) …

From the IAC committee’s:

InterAcademy Council's action plan for reviewing IPCC

Action Plan – Commitee Process

Establishment of Committee
[…]
Information Gathering & Analysis
[…]
Report Preparation
[…]
Peer Review

The report is peer reviewed using guidelines established by IAC to help ensure the report meets institutional standards for objectivity, evidence, and responsiveness to the committee task. The review of the report is overseen by individuals appointed by the IAC, who are responsible for ensuring that all review comments were carefully considered.

Final Report*
[…] (emphasis added -hro)

(* Helpful hint from Hilary to IAC webmaster: unless the final report found here is really not the “final” report, you really should give some thought to removing the nifty little “In progress” image/icon that forms part of this section.)

The section on peer review does raise a few questions in my mind (and perhaps yours, too). The first of which: is what are these “guidelines established by the IAC”?

Believe it or not, these can be found (and unlike any IPCC guidelines I’ve ever seen, they’re actually written in comprehensible English) on the IAC site. In light of the recent Steig-O’Donnell controversy – and the supremacy of reviewer confidentiality (even to the point of “justified disingenuousness“), I thought that the following in the IAC’s guidelines was quite interesting:

Confidentiality
To encourage reviewers to express their views freely, review comments are treated as confidential documents and are given to the Study Panel with identifiers removed. After submitting their comments, reviewers are asked to return or destroy the draft report and to refrain from disclosing their comments or the contents of the draft.

The names and affiliations of reviewers and the review monitors will be made public in the report when it is released, along with a disclaimer that makes clear their role in the process.

Did the IAC practice what they preach in the Review of the IPCC? On page iii of the Foreword (signed by the two IAC Co-Chairs), one finds:

First written in draft form, the final report incorporates the committee’s response to an extensive IAC report review process during August 2010 that involved twelve experts plus two distinguished scientists who served as review monitors. Upon the satisfactory completion of the report review process, the IAC Board approved publication of this final report.

The short answer would appear to be yes, the IAC – at least in this instance – did practice what it preaches. For the record: The report was written by a committee of twelve, chaired by Dr. Harold T. Shapiro. The committee had the assistance of seven “staff”: four from the US National Research Council, two from The (UK) Royal Society, and one from the InterAcademy Council. The draft report was “reviewed” by twelve “internationally renowned experts” whose names and respective affiliations are listed on p. xi along with the following:

Although the reviewers provided constructive comments and suggestions, they were not asked to endorse the conclusions and recommendations, nor did they see the final draft of the report before its release.

The duties of the two review monitors (whose names and affiliations appear on p. xii) are described (p. xi) as:

being responsible for ascertaining that the independent examination of this report was carried out in accordance with IAC procedures and that all review comments were carefully considered.

Curiously absent, though, in all this prefatory “review” verbiage is any mention of the word “peer”. Is this “significant”?! Hey, your guess is as good as mine! But just for the fun of it, let’s compare three usages of “peer review”: Journals, IPCC and IAC.

Criteria	Journals	IPCC	IAC
Author name(s) available to reviewers?	Yes	Yes	Yes
Reviewer comments attributable to commenter?	No	Yes	No
Reviewer name(s) disclosed to author(s)?	No	Yes	Yes1
Responsibility to ensure comments addressed?	Yes	Yes2	Yes
Evaluation of data, analyses and statistical methods?	No	No	Yes3

^{1 But not until publication of the report
2 Nominally, “Review Editors”; however, one of the major weaknesses of the IPCC, as noted by the IAC, is the failure of Review Editors to actually perform this function.
3 See annex to IAC guidelines}

Perhaps the best that one might say about this much vaunted “gold standard” is that consistency of principles and/or implementation is far from being a strong point of “peer review”.

Of the three, only the IAC could be “accused” of approaching the thoroughness and transparency that a “sharper questioning” public expects of a purportedly “respected” institution. Well, the IAC will be deserving of such an accolade, as soon as they deliver the 162+ mysteriously missing questionnaire responses – or a reasonable explanation for their absence.