The wonderful “A billion here, a billion there, and pretty soon you’re talking real money” statement attributed to Senator Everett Dirksen may be apocryphal, but it remains a prescient warning to our nation’s leaders. At a time when Congress is throwing billions of dollars around like pocket change based on claims of scientists and engineers, a real quote of Dirksen may be equally important (Congressional Record: June 16, 1965, p. 13884):
One time in the House of Representatives [a colleague] told me a story about a proposition that a teacher put to a boy. He said, ‘Johnny, a cat fell in a well 100 feet deep. Suppose that cat climbed up 1 foot and then fell back 2 feet. How long would it take the cat to get out of the well?
Johnny worked assiduously with his slate and slate pencil for quite a while, and then when the teacher came down and said, ‘How are you getting along?’ Johnny said, ‘Teacher, if you give me another slate and a couple of slate pencils, I am pretty sure that in the next 30 minutes I can land that cat in hell.
The nation needs Johnny. In fact, it may be time we hired a team of people like Johnny for every large science-based policy proposal Congress contemplates funding.
Carbon Capture and Storage: A Known Boondoggle
Consider, for example, the $4.4 billion Congress is putting into carbon capture and sequestration (CCS) research, nearly half of that to come from the Kerry-Lieberman climate bill. As Robert Bryce points out in the New York Times, “That’s a lot of money for a technology whose adoption faces three potentially insurmountable hurdles: it greatly reduces the output of power plants; pipeline capacity to move the newly captured carbon dioxide is woefully insufficient; and the volume of waste material is staggering.”
Those of us familiar with the coal-fired power plant industry have long recognized that CCS may be slightly more than a pipe-dream, but will never be affordable or practicable for the vast majority of coal-fired plants. Yet no one in the bureaucracy has had the courage to stand up and refute this politically correct but scientifically bankrupt concept.
Lessons from a Broken Hockey Stick
Nor is CCS the only example. Perhaps the highest visibility science that needed a “Johnny” is the now infamous global warming hockey stick. Andrew Montford’s “The Hockey Stick Illusion – Climategate and the Corruption of Science” deconstructs the woeful practices leading to this canard that propelled the entire world toward economic investments that are likely as not to be entirely wasted.
In his book, Montford identifies individuals within the paleoclimatology community who argued that they would not have used the invalid data upon which the hockey stick depended. But, they did not come forward, nor were they allowed to “peer review” the work before its publication. And why was that?
It took an act of Congress to find a “Johnny” to sort out the hockey stick and explain why no one came forward. The resultant Wegman report concluded, among many other important things:
The politicization of academic scholarly work leads to confusing public debates. Scholarly papers published in peer reviewed journals are considered the archival record of research. There is usually no requirement to archive supplemental material such as code and data. Consequently, the supplementary material for academic work is often poorly documented and archived and is not sufficiently robust to withstand intense public debate. In the present example there was too much reliance on peer review, which seemed not to be sufficiently independent.
And Wegman recommended:
Especially when massive amounts of public monies and human lives are at stake, academic work should have a more intense level of scrutiny and review. It is especially the case that authors of policy-related documents like the IPCC report, Climate Change 2001: The Scientific Basis, should not be the same people as those that constructed the academic papers.
The “Plan B” Idea
Having indepdent peer review is not a new idea. In 2005, Steven Hayward suggested “Perhaps the time has come to consider competition as the means of checking the Intergovernmental Panel on Climate Change’s monopoly and generating more reliable climate science.”
The Heartland Institute made such an attempt with its “Non-Governmental International Panel on Climate Change” report and the Center for the Study of Carbon Dioxide and Global Climate has mounted an effort to continue this work. But these efforts have failed to produce a meaningful impact on the political process, largely because the authors do not operate within the “peer” community and do not publish in the peer-reviewed literature.
There are better models, however, and these could be adapted to science-based policy analyses. For example, by federal law, an investment prospectus must include sufficient specification of risk assumptions as to permit independent analysis of the proposal. In like measure, financial institutions’ annual reports require independent audit analysis before publication.
The key is independent replication of the analysis and evaluation of important assumptions.
In the cold war days, the military used an approach called “Team B.” This team (alternatively called the Red team) developed approaches to counter the efforts of Team A (the Blue team). Still in use today, it has some strengths and some weaknesses. Without question, however, it always sharpened the quality of Team A’s work, finding errors in analysis, weaknesses of assumptions and entire areas of interest that opened up Team A’s thought processes and analytical options.
Team B operated within the professional “community,” making its contributions instantly credible and mandatory for review by Team A and the eventual policy makers.
The Marshall Institute had engendered discussion of this approach for today’s scientific policy issues, as have members of the American Physical Society. In particular, Dr. William Happer, Cyrus Fogg Brackett Professor of Physics at Princeton University, member of the National Academy of Sciences, and Fellow of the American Physical Society (APS), has proposed a Team B approach within the APS. Without question, the APS has enough independent and qualified members to have been able to audit the critical work underlying the IPCC reports and inform journal publisher’s of the important analytical elements to which regular peer-reviewers should give special attention.
A Proposal
I have little confidence in Happer’s approach, however. It stands too far back from the critical scientific process and will make contributions too late in the policy process. Instead, we can take a page from the book of the existing science policy bureaucracy – environmental assessments.
Under the National Environmental Policy Act, a project using significant federal funds cannot move forward until some form of environmental assessment has been completed and approved. These assessments have, of course, been misused to delay and prevent projects. However, at their heart, they have significantly improved proposals with the potential to adversely impact the environment. Correcting to eliminate the pathologies of the process, something similar could be used in other areas.
In the context of science-based policy today, Team B would be the ne plus ultra peer-review group required for any scientific or engineering initiative (including grants and grant programs) that Congress or the Federal bureaucracy planned to fund above a trigger level, or which would have such an impact on the private sector. The trigger might be $500 million, for example.
Before Congress authorized expenditure of such funds, the bill sponsor would need to offer a Team B report that validated the basis otherwise offered to justify the expenditure. An National Science Foundation program manager wishing to fund a subject area that expands high-cost research would need a valid Team B analysis of the science to ensure the area of investigation is justified. The Department of Energy, the Interior Department and the Environmental Protection Agency would need a Team B report on their major research initiatives and regulatory proposals. Any major studies commissioned or conducted by the National Research Council (and its subsidiaries) would require a Team B review.
In every case, a Team B report would involve an independent replication of the analysis, including assumptions, selection and screening of data, logic of computer code, statistical analysis and presentation of data. The Team B report would have to demonstrate its independence, using the tools discussed in the Wegman report.
This independence is a key aspect of a Team B approach. It would not be helpful to simply pit antagonists against each other; and, of course, one cannot permit the incestuousness observed in the Mann-Jones et al network. Rather, one wants a “one-off” approach. In nearly every case, one would want a Team B to include a statistician familiar with the statistical methods used, but not familiar with the authors and perhaps only peripherally knowledgeable about the scientific subject. They can obtain the information about the subject from the subject matter specialists on the team.
The real challenge is selecting subject matter specialists. In cases where funding opportunities is controlled by cliques, there is even a greater problem. Because all Team B work should be done in the light of day, perhaps all one needs is an honest Team B. (OK, call me Diogenes.) An open process, one that access the entire planet’s available and experienced scientific community through the internet, may be the guarantor of a quality project.
The Upside of Reform
While Wegman did not believe peer review ought to be through blogs, without question, without McIntyre’s efforts, as recorded on his blog site, Mann’s corrupt hockey stick would never have been exposed. A competent Team B could harvest the intellectual contributions from an engaged web-community to good effect, especially with regard to highly technical issues. Further, the anonymity of the web would allow specialists to provide critical information without having to be worried about being outcast by the specialists’ cliques.
Surely a Team B approach would evolve and mature over time. What is certain, however, is that even the threat of a Team B review would occasion a return to trustworthy science and a fundamental return to the bedrock principles of scientific inquiry. As Wegman, Montford and McIntyre have made unutterably clear, if we need heightened oversight of financial institutions, we surely need such heightened oversight of the science upon which we found science-related policies having large economic consequences.
What a beautifully written, logically compelling article. Thanks. I’m grateful for the wonderful story from Dirkson and the concept of Johnny and his slate pads, although I’ve used evidently the same boy in various Emperor’s New Clothes scenarios.
“Independent” peer review that works hard to “disprove” a particular thesis, particularly one fraught with significant public expense and environmental risk, should be mandatory. NEPA does indeed require something like this. And, of course, the National Academy of Science has also been called upon frequently to perform this function.
However, given the collusive potential of our mass media, and the power of the Internet, I despair for the proper use of this important tool. At the micro level of a state regulatory agency, where the Department of Natural Resources employed an “independent” Power Plant Research Program to conduct a thorough energy/socio-economic/environmental impact review, the process was a sham in Maryland. The experts often tailored their comments to suit the dictates of their political bosses and, when forced to tell the truth when cross examined under oath, their negative comments were simply elided in the final DNR report.
If this kind of political pervasion can take place at the state level, it certainly will easily overtake policy inquiries at the federal level, as it already has. No one should have any confidence in the “science” presented by Congress or any Administration, for it’s used pervasively to sell soap, not truth. And virtually everyone knows it.
Until we can elect better political leadership and inculcate knowledge of the scientific method (that is, the principles of skeptical inquiry) throughout our citizenry, modern societies will continue to be bilked by scientific poseurs employed by various good intentions paving companies.
Well reasoned and logical. I share the same concerns about the Red Team being co-opted by the system, most organizations tend, over time to drift into the orbit of the conventional wisdom. And being funded by Congress makes that result more likely. I just don’t see how you can get around those problems if the Red Teams are government funded or sponsored. Maybe the best alternative would be for the various professional societies and trade organizations to sponsor these groups. Yes, I know that an energy center of analysis sponsored by the API, as an example, would likely be derided by some, but I think that these organizations could structure groups in such a way as to make them nearly independent. For example, an industry association could endow a group with say $10,000,000 and write a clearly defined mission statement. Appoint a panel of recognized experts and charge them with staffing and subject matter review. And to keep costs low, these organizations could be virtual. I don’t doubt that there are hundreds of semi-retired people who would jump at the chance to be part of such a community.
My own personal preference is that the government create a Department of Unintended Consequences, similar to the GAO. This department’s sole mission would be to opine on proposed legislation and highlight potential land mines.
The current assessments suffer from the fatal flaw that Roger Pielke Senior has frequently brought up-the authors of the assessments review areas related to their own work, and rely heavily on their own papers, giving them prominence to promote their narrow view points. What can be done to prevent the incestuous problem of scientists reviewing their own work in these assessments? One can imagine that we could just end up with assessments from each “team” which just skew to their own viewpoints the way the current assessments do.
An excellent analysis. The current “peer review” process is fundamentally flawed; it often (too often) amounts to little more than colleagues or like-minded “experts” giving the authors a “pat on the back”. The “cross-review” (A reviews B, B reviews A) evident in the IPCC AR4 is staggering. The sceptical “Team B” approach has great merit, and would be a structured formalisation of what has traditionally though entirely informally happened in the past, yet seemingly been entirely abandoned in mainstream climate science. Scepticism is the life-blood of science; without it science will degrade into a lifeless regime of mutual back-slapping and consensus.
Blogs do a great job, but carry no authoritative weight in the scientific world. I’m a true sceptic in the real sense of the word, and blogs give me (an enthusiastic amateur) the opportunity to read, learn, discuss, question and sometimes even contribute a little.
Long live internet blogging, and thanks for a great and thought-provoking article.
Transparency and broad participation have always been the keystone to uncovering biases, whether intentional or not. A Team B approach that accesses the considerable participation of those who post on the internet should not be ignored. Bringing one hundred bright minds to a task will ensure biases to the front, even those held by a Team B member. The challenge is to find an honest Team B who will be willing to admit to their own past mistakes, errors or ignorance.
[…] Climate Policy Needs a Team “B” David Schnare, MasterResource.org, 18 May 2010 […]