The Ultimate Peer Review
Posted by The Diatribe Guy on April 2, 2010
The following is a guest post by Bob Heiderstadt. I, firstly, must apologize to him for not having posted it months ago when he first submitted it to me as a potential guest post. Bob, I did not pass it along for any issues that I had with it, it is just one of those things where I put it in a folder to look at later, and just never did.
Anyway, it is not about Climate Change, per se. But so much of the debate about climate science these days – and highlighted by the Climategate scandal – is about peer review. Many times when a very eloquent and insightful argument is made on a blog somewhere, one of the immediate tools of defense of those dismissing the argument is because it isn’t peer reviewed. While it seems an agreeable practice to have peer review, Climategate demonstrates a darker side to it: when the review group is a smaller circle unable to review a work in an unbiased fashion, and instead looks for ways to discredit a paper not because of its science, but because of its conclusion, we have a problem.
Bob put together some thoughts on what an open and honest process may look like. Such a process seems more open. A paper should be judged by its content, scientific application, accuracy, and presentation. Not by titles or preconceived conclusions.
The Ultimate Peer Review
Guest Post by Bob Heiderstadt
Wikipedia says “Peer review (also known as refereeing) is the process of subjecting an author’s scholarly work, research, or ideas to the scrutiny of others who are experts in the same field. Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform impartial review. Impartial review, especially of work in less narrowly defined or inter-disciplinary fields, may be difficult to accomplish; and the significance (good or bad) of an idea may never be widely appreciated among its contemporaries. Although generally considered essential to academic quality, peer review has been criticized as ineffective, slow, and misunderstood.”
A PhD can be jokingly described as “someone who knows almost everything about next to nothing.” Seriously though, someone who has earned a PhD does know significantly more about a particular subject than the vast majority of the population. In fact, there may only be a handful of individuals who may as knowledgeable about a particular subject, regardless of whether it is in the social sciences or the hard sciences.
No PhD, regardless of how knowledgeable he/she is in a particular subject, can be considered an expert in all of the related areas that support his/her area of expertise. A scientist, for example, may be an expert in his/her area of science, but may have only adequate training in statistical analysis or language to analyze or express the analysis. This is not to say that their skills are inadequate in the support areas, but it’s impossible to be an expert in everything. This is where the assistance of others who are more knowledgeable in those support areas provides a great benefit, or added value, to the expert in preparing a study for publishing.
The internet has become the ultimate peer review, not because a paper is reviewed by a select few experts, but because it can be reviewed by anyone, from a high school dropout with an interest in the subject, but with maybe a unique insight, to the PhD who is an expert in the area of study. Granted, some comments will not be helpful, but as the analysis of Stieg, et al has demonstrated, a team of people who are not necessarily experts in that particular area of science, but may be engineers, statisticians, economists, and yes, scientists can make a valuable contribution to the review. It is that varied experience and knowledge that can bring a different point of view to a study and point out deficiencies or irregularities in the analysis or presentation that ultimately make the study better. This is the true essence of peer review, to try to obtain the best product possible, while winnowing out the junk.
With this in mind, I propose the following:
- Establish an internet web site to behave as a repository for papers submitted for review and give it a name similar to “SciencePeerReview.com or .org”
- Establish different categories of science to be reviewed (hard sciences or soft sciences).
- Provide sufficient storage to accept large papers and supporting data
- Develop a way to pay for the web site (advertising or a fee paid by the submitting individual(s).
- Accept only papers submitted with supporting data and procedures (data and software)
- Submitted papers should be in a pdf, Microsoft Word®, or Open Office format.
- Allow any individual to review and comment on any paper with the following restrictions:
- The individual must identify himself/herself to the web site and provide minimal information regarding education and/or credentials and experience.
- Comments must be pertinent to the paper being reviewed and polite.
- Optionally, a reviewer could receive an automated ID for the author, permitting anonymity.
- Allow two different styles of comments:
- Blog style – visible to other reviewers or optionally for the author only.
- Document style – for detailed analysis of submitted paper (see 6 above), again with the option to post for other reviewers or directed to the author only.
- Allow the author to request a specific review period based on the size and complexity of the paper so that there would be adequate time to complete a review
- The author could respond to comments to clarify stated concerns (blog style).
Whether the final paper is presented on this site, or through some other venue, should be at the option of the author, although if published somewhere else, the appropriate reference should be made.
Since this is about peer review, if you read it, please comment on it to make it better, or comment even if you think it is a lousy idea. That’s the whole point.