Circa 2010: PNAS "Expert Credibility in Climate Change"..debunked

The claim: http://www.pnas.org/content/early/2010/06/04/1003187107.abstract

***

http://theresilientearth.com/?q=content/pnas-climate-change-expert-credibility-farce

A new, purportedly scientific report in the Proceedings of the National Academy of Sciences (PNAS) is claiming that more “top” environmental scientists believe in global warming. Moreover, the report also claims that the scientists who do believe in global warming—now re-labeled anthropogenic climate change (ACC)—have higher credibility than those who do not. All of this is based on an “extensive dataset of 1,372 climate researchers and their publication and citation data.” Citing such data is like saying “most of the people who write for conservative magazines are conservatives.” In other words, the study is devoid of factual significance and possibly purposely misleading.

In an open access article, rather innocuously titled “Expert credibility in climate change,” William R. L. Anderegg, James W. Prall, Jacob Harold, and Stephen H. Schneider have attempted to denigrate those who dare to disagree with the IPCC party line. In order to provide a false sense of balance, the “researchers” refer to climate change believers as “convinced by the evidence” (CE) and skeptics as “unconvinced by the evidence” (UE). That is the only unbiased thing about the report.

The first suspicious thing is that the “extensive dataset” contains only 1,372 climate researchers on both sides of the issue. I seem to recall that the IPCC alone claimed more than that many “experts” had worked on its reports. The second suspicious thing is the 97-98% pro-AGW result, a figure more in line with the election results from a tin-pot dictatorship or Stalinist people’s republic. This result alone is an indication that the dataset is biased. Here is how the authors described their methodology:

We ranked researchers based on the total number of climate publications authored. Though our compiled researcher list is not comprehensive nor designed to be representative of the entire climate science community, we have drawn researchers from the most high-profile reports and public statements about ACC. Therefore, we have likely compiled the strongest and most credentialed researchers in CE and UE groups. Citation and publication analyses must be treated with caution in inferring scientific credibility, but we suggest that our methods and our expertise and prominence criteria provide conservative, robust, and relevant indicators of relative credibility of CE and UE groups of climate researchers.

They clearly state that they selected researchers from the “most high-profile reports and public statements,” obviously a judgment call on the author’s part—and there is no doubt which side of the debate the authors are on. The statement that they have compiled the “strongest and most credentialed researchers in CE and UE groups” is shear supposition on the author’s part. They rightly hint that counting citations and publications is not a reliable indication of scientific credibility. Since the whole point of the article is to make the claim that climate change believers are more credible than non-believers, this amounts to an admission that the whole report is based on a faulty assumption. Then, after warning that their list is not comprehensive, nor designed to be representative, and that publication analyses must be treated with caution, they blithely state that their methods are sound.

The assertion that the quality of a scientist’s work, and by extension the scientific credibility of that scientist, can be discerned by the shear volume of his publications is ludicrous. Having an academic background myself, I can tell you that there are some scientists who turn out a staggering number of papers—they seem to thrive on writing and publishing. This is fine, and since many academic institutions place exaggerated value on the number of publications a scholar has, there is a trend toward this type of activity. Everyone in academia has heard the old saw, “publish or perish.”

If you habitually read climate related literature, as I do, you will find the same authors, in various combinations, publishing papers that are slightly different versions of each other. Another factor to consider is that scientists who are also university professors often publish papers done by students under their charge. This also contributes to publication bloat. Simply put, having a large number of publications does not mean that a scholar has anything noteworthy to say. Indeed, many great scientists published only a few seminal papers.

As you can see by looking at the data sources, it is unsurprising that they would find their sample researchers to be in agreement with the IPCC—that is were they started!

We compiled these CE researchers comprehensively (i.e., all names listed) from the following lists: IPCC AR4 Working Group I Contributors (coordinating lead authors, lead authors, and contributing authors; 619 names listed), 2007 Bali Declaration (212 signers listed), Canadian Meteorological and Oceanographic Society (CMOS) 2006 statement (120 names listed), CMOS 2008 statement (130 names listed), and 37 signers of open letter protesting The Great Global Warming Swindle film errors. After removing duplicate names across these lists, we had a total of 903 names.

For UE researchers they relied on 12 lists, ranging from the Science and Environmental 1995 Leipzig Declaration (80 names), to the 2009 newspaper ad by the Cato Institute challenging President Obama’s stance on climate change (115 signers), for a total of 472 names. The fair-minded reader might wish to consider the difference in sources between the two groups and whether the skeptics would be likely to submit their work to the same journals as the CE researchers. After all, openly skeptical articles on climate change would have a hard time passing peer review when the reviewers are all AGW believers.

What’s more, the sources used for this paper—the most cited authorities pro and con—were supplied by one of the paper’s authors, Jim Prall. Prall, a self-described non-academic, runs a climate change website of his own. As it turns out, Prall has been collecting a list of people he disagrees with for some time now. Roger Pielke Jr., has described it as a “black list” for climate skeptics. Prall’s list of the most cited skeptical authors on climate science can be found here.

I see little need to expound on the type of people who keep enemies lists but, by his own admission, Prall is fed up with “the tiny minority of ‘climate skeptics’ or ‘deniers’ who try to minimize the problem, absolve humans of any major impact, or suggest there is no need to take any action.” Oh yes, Jim is certainly a disinterested, open-minded observer who’s data should be used to evaluate the expertise of climate change skeptics. This is the equivalent of stuffing a ballot box—the whole selection process used in this paper is wrong on so many levels it beggars the imagination.

The PNAS paper is a meaningless propaganda piece, pure and simple. The “research” presented was designed to produce a biased answer. If students of mine submitted similar work they would receive a failing grade. It is so biased in its data collection and analyses that it is incapable of proving anything. As Wolfgang Pauli said, “This isn’t right, it isn’t even wrong.”

***

http://www.sciencebits.com/Expert-credibility-in-climate

I recently stumbled upon one of the most meaningless papers I have ever seen, it is called “Expert credibility in climate change” by Anderegg, Prall, Harold and Schneider. The paper “proves” that the scientists advocating an anthropogenic greenhouse warming (AGW) are statistically more credible than the “unconvinced”. Their main goal is to convince people that they should join the AGW bandwagon simply because it is allegedly more credible.

In essence, the authors show that the AGW protagonists have more published papers in climate journals and more citations. The authors then carry on with an elaborate statistical analysis showing how statistically significant the results are. The first thing that popped into my mind is the story about a statistician who proved that 87.54% of all statistical research is meaningless…

With or without the fancy statistical analysis, and in fact, with or without the data, I could have told you that the scientists in the believer camp should have more papers and many more citations. But this has nothing to do with credibility. It has everything to do with the size of the groups and the way their members behave.

Since the AGW protagonists have the tendency to block the publication of papers that don’t follow their party line (and if you think otherwise, read the climategate emails), it is way easier for the AGW protagonists to have any paper get published. Just as an example, the above meaningless paper passed the peer review process of PNAS. And of course it did! It did so because it was much more likely to reach peers in the AGW protagonist group. If I would have tried myself to get a similar paper published, it would have been thrown down the stairs (and rightly so because of its meaninglessness [yes, its a new word]). But any paper my colleagues and I try to publish gets such a hostile confrontation that it is simply very hard to publish at all. The bottom line is more papers for the AGW protagonists and less papers for those who are more critical.

In fact, I have no idea how the “average” “climate expert” could have published 408 climate publications. Over say 30-40 years of activity it means a paper once every month or so. Of course, it could be that the average expert simply contributes just a little to each paper.

Let me end with a comparison.

In 1912, Wegener came out (like a few others before, I should add) with the idea that continents drift. At worst, he was mostly ignored. At best, he did get attention – he was proven to be wrong. The tide turned only in 1960′s when paleomagnetic data showed quite unequivocally that indeed the continents move (today, this can also be measured with GPS). So, for more than 50 years, if one would have carried a similar analysis to W. R. L. Anderegg et al., or to N. Oreskes (Science 306:1686, 2004), one would have reached the conclusion that the truth is with the more credible majority thinking that continents are stuck in their place.

As you can see, science is not a democracy, just by counting people, or more sophisticatedly, counting papers per expert and citations per expert, doesn’t imply that the majority or apparently more credible group is correct, irrespective of how fancy the analysis might look. Just do the science and the truth will emerge from it.

“CONSENSUS ISN’T SCIENCE AND SCIENCE ISN’T CONSENSUS”

SNIP FROM COMMENTS:

Hello Nir, I wanted to let you know that the paper you discuss is not really a ‘peer-reviewed’ paper but rather a ‘pal-reviewed’ paper. First of all, you would not be able to publish in the PNAS because it is a closed shop – members only. So, they elect people who share their story. Once you are a member of the PNAS, you have the right, four times a year, to ‘contribute’ an article, such as that of Andregg. This does not go through the traditional peer-review process of the PNAS “Direct Submission” method which is VERY strict (and what they built their rep on). But the public doesn’t know that!! All they know is the ‘pal-reviewed’ item is in the PNAS….and you’re not.

Example - http://voices.yahoo.com/prestigious-tar-sands-papers-not-truly-peer-reviewed-11092300.html?cat=9

***

http://wattsupwiththat.com/2010/08/05/expert-embarrassment-in-climate-change/

The paper ‘Expert Credibility in Climate Change,’ published in PNAS by Anderegg, the late Stephen Schneider, James Prall and Jacob Harold attempts to measure the credibility of climate scientists by counting how many papers they have published and how often their work has been cited by others.

This led to the creation of a blacklist that will be used to injure the careers of those who have signed letters or petitions that do not agree with the Al Gore/James Hansen position on climate change, and to intimidate future scientists, effectively silencing dissent.

The paper is poorly done, as I’ve explained elsewhere. They used Google Scholar instead of an academic database. They searched only in English, despite the global nature of climate science. They got names wrong. They got job titles wrong. They got incorrect numbers of publications and citations.

As I’ve mentioned, the highly respected Spencer Weart dismissed the paper as rubbish, saying it should not have been published.

But the worst part of this is the violation of the rights of those they studied. Because Prall keeps lists of skeptical scientists on his weblog, obsessively trawling through online petitions and published lists of letters, and because those lists were used as part of the research, anyone now or in the future can have at their fingertips the names of those who now or in the past dared to disagree.

The Joe Romm’s of this world have already called for this list to be used to deny funding, tenure and grants to scientists. And it will be. It doesn’t matter that the nature of the letters and petitions they signed varied widely, from outright skepticism to really innocuous questioning of the state of the science.

The paper is tagged ‘Climate Deniers.’ Now, so are they.

This is an outright violation of every ethical code of conduct for research that has ever been published.

They violate several sections of the American Sociological Association Ethical Guidelines:
“Sociologists conduct research, teach, practice, and provide service only within the boundaries of their competence, based on their education, training, supervised experience, or appropriate professional experience.”

The members of the research team were operating outside their areas of professional competence.

“Sociologists refrain from undertaking an activity when their personal circumstances may interfere with their professional work or lead to harm for a student, supervisee, human subject, client, colleague, or other person to whom they have a scientific, teaching, consulting, or other professional obligation.” The subjects of their research–the scientists on the list–risk grave harm as a result of this paper.

———–

this is for the “cockroach” on Twitter known as __debaser__

https://twitter.com/__debaser__

hes a copy and paste parrot who keeps throwing up this PNAS study as some sort of vindication of global warming science..a study and compilation of biased statistics is not science “cockroach” and consensus is not science..so here is my reprisal..much to long for 140 characters..

no further comment by me is required..its all said in the shaded sections..cheers..

401

About these ads

~ by seeker401 on March 19, 2013.