Digital Diatribes

A presentation of data on climate and other stuff

The Ultimate Peer Review

Posted by The Diatribe Guy on April 2, 2010

The following is a guest post by Bob Heiderstadt.   I, firstly, must apologize to him for not having posted it months ago when he first submitted it to me as a potential guest post.   Bob, I did not pass it along for any issues that I had with it, it is just one of those things where I put it in a folder to look at later, and just never did.

Anyway, it is not about Climate Change, per se.   But so much of the debate about climate science these days – and highlighted by the Climategate scandal – is about peer review.   Many times when a very eloquent and insightful argument is made on a blog somewhere, one of the immediate tools of defense of those dismissing the argument is because it isn’t peer reviewed.  While it seems an agreeable practice to have peer review, Climategate demonstrates a darker side to it:  when the review group is a smaller circle unable to review a work in an unbiased fashion, and instead looks for ways to discredit a paper not because of its science, but because of its conclusion, we have a problem.

Bob put together some thoughts on what an open and honest process may look like.   Such a process seems more open.   A paper should be judged by its content, scientific application, accuracy, and presentation.   Not by titles or preconceived conclusions.

The Ultimate Peer Review

Guest Post by Bob Heiderstadt

Wikipedia says “Peer review (also known as refereeing) is the process of subjecting an author’s scholarly work, research, or ideas to the scrutiny of others who are experts in the same field. Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform impartial review. Impartial review, especially of work in less narrowly defined or inter-disciplinary fields, may be difficult to accomplish; and the significance (good or bad) of an idea may never be widely appreciated among its contemporaries. Although generally considered essential to academic quality, peer review has been criticized as ineffective, slow, and misunderstood.”

A PhD can be jokingly described as “someone who knows almost everything about next to nothing.”  Seriously though, someone who has earned a PhD does know significantly more about a particular subject than the vast majority of the population.  In fact, there may only be a handful of individuals who may as knowledgeable about a particular subject, regardless of whether it is in the social sciences or the hard sciences.

No PhD, regardless of how knowledgeable he/she is in a particular subject, can be considered an expert in all of the related areas that support his/her area of expertise.  A scientist, for example, may be an expert in his/her area of science, but may have only adequate training in statistical analysis or language to analyze or express the analysis.  This is not to say that their skills are inadequate in the support areas, but it’s impossible to be an expert in everything.  This is where the assistance of others who are more knowledgeable in those support areas provides a great benefit, or added value, to the expert in preparing a study for publishing.

The internet has become the ultimate peer review, not because a paper is reviewed by a select few experts, but because it can be reviewed by anyone, from a high school dropout with an interest in the subject, but with maybe a unique insight, to the PhD who is an expert in the area of study. Granted, some comments will not be helpful, but as the analysis of Stieg, et al has demonstrated, a team of people who are not necessarily experts in that particular area of science, but may be engineers, statisticians, economists, and yes, scientists can make a valuable contribution to the review.  It is that varied experience and knowledge that can bring a different point of view to a study and point out deficiencies or irregularities in the analysis or presentation that ultimately make the study better.  This is the true essence of peer review, to try to obtain the best product possible, while winnowing out the junk.

With this in mind, I propose the following:

  1. Establish an internet web site to behave as a repository for papers submitted for review and give it a name similar to “ or .org”
  2. Establish different categories of science to be reviewed (hard sciences or soft sciences).
  3. Provide sufficient storage to accept large papers and supporting data
  4. Develop a way to pay for the web site (advertising or a fee paid by the submitting individual(s).
  5. Accept only papers submitted with supporting data and procedures (data and software)
  6. Submitted papers should be in a pdf, Microsoft Word®, or Open Office format.
  7. Allow any individual to review and comment on any paper with the following restrictions:
    • The individual must identify himself/herself to the web site and provide minimal information regarding education and/or credentials and experience.
    • Comments must be pertinent to the paper being reviewed and polite.
  8. Optionally, a reviewer could receive an automated ID for the author, permitting anonymity.
  9. Allow two different styles of comments:
    • Blog style – visible to other reviewers or optionally for the author only.
    • Document style – for detailed analysis of submitted paper (see 6 above), again with the option to post for other reviewers or directed to the author only.
  10. Allow the author to request a specific review period based on the size and complexity of the paper so that there would be adequate time to complete a review
  11. The author could respond to comments to clarify stated concerns (blog style).


Whether the final paper is presented on this site, or through some other venue, should be at the option of the author, although if published somewhere else, the appropriate reference should be made.

Since this is about peer review, if you read it, please comment on it to make it better, or comment even if you think it is a lousy idea.  That’s the whole point.


4 Responses to “The Ultimate Peer Review”

  1. Pat Moffitt said

    I would add to this a place where pure raw data could be posted. There are many studies that do not result in publication for one reason or another. However, the raw data collected may be applicable to some other unanticipated line of inquiry.

  2. […] met office suffers 25% budget cut ; The problem with peer review is when the peers do the review ; The ultimate peer review ; Climate conflicts of interest ; Follow the money Hansen […]

  3. Rereke Whakaaro said

    I suggest that submitted material be in a non-proprietary format, and preferably one that cannot be modified by a reviewer (by intent or by accident).

    I also question the need for reviewers to give details of their credentials. I have known people who have expert knowledge in a particular field, due to a personal interest – hobby, if you will – who have no formal qualifications. They often have insights, and can ask questions, that somebody following a more formal approach might miss. It would be a pity if their input was prejudged on the basis of their formal training.

    • Bob H. said

      I had given this some consideration. I’ll grant you that somebody who has a personal interest but no formal qualifications in the field may have some insights and perhaps ask questions that would reveal something the researcher may not have considered.

      On the other hand, let’s say a paper is submitted describing the sunspot cycles for the past 400 by an eminent solar scientist. Another individual who has a hobby of tracking sunspots decides to review the paper who also happens to be a PhD in statistical analysis. This individual rips the paper to shreds based on the statistics. Obviously the credentials are important to the author so that the analysis can be properly weighted and not easily dismissed.

      Now let’s say a high school dtropout who has an interest in astronomy and likes to look at sunspots has an insightful comment or question. Again it would be helpful to the author to know a little bit of the background of the reviewer to judge the quality of the comment/question. Would the author automatically dismiss the high school dropout even if the comment/question was good and accept the PhD? Not necessarily yes in either case, if the author is actually looking to make his paper better the both sets of comments should receive attention.

      How much the author is willing to accept all comments is largely going to depend on the ego of the author and whether he/she is working from a preconceived notion of what the results should be. Unfortunately, this is one element that cannot be removed from the process, although I would suspect any author who would submit a paper to this sort of review would be more open to all comments than the typical submittal to a journal.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: