Thursday, August 9, 2012

The important peer review comes after the peer review.

It is not a secret among scientists that the peer review system which is used as a sort of quality control prior to publication is flawed. When a scientists thinks that he or she has made a discovery the general procedure is to write up a paper containing an introduction to the questions addressed, the methods used (the "scientific method" should really be plural as in "scientific methods" because in science a wealth of different methods are used to create ever improving models of the "real world"), the results found and then a discussion in which you discuss your work and how it relates to other scientists results.

The paper is then sent to a journal. If the scientist thinks his or her paper is groundbreaking it may be sent to nature or science, but there are many other options as well. The editor of the journal may then decide whether the paper is rejected immediately (editors have a lot of power), or whether it will be sent out for a review (notice that editors cannot just go ahead and accept a paper, however fantastic it seems.).

What people refer to as "peer review" is when a paper is sent to a few scientists (usually 2-3) in the same field, who will carefully read the paper and give their opinion on it. They may say the article lacks in news-value, that the method used has flaws or that one has forgotten to discuss a certain relevant paper (often their own...). Based on the reviews from these experts the editor makes a choice, will the article be published, rejected or they encourage you to do some changes and then send it in again.

So lets say you got through all this and managed to get your paper accepted by a top-notch scientific journal. Is your conclusions now established dogma, or scientific fact. Is it written into the holy bible of science never to be changed? No! When your article has been published the real peer review process starts, your article will now be discussed by other scientists in the field. When it is published it is open for everyone (with a subscription to the journal) to read and comment upon. It may be that after a few years a fatal flaw is found in the paper and no-one will believe in it anymore.

A good example is the faster than light neutrinos where the authors did not believe in faster than light neutrinos themselves but they still published their data allowing other scientists to add their interpretations/suggestions. Pretty fast someone figured out that the results was merely caused by a measuring error in so and so, which was later confirmed. Now no-one believes in faster than light neutrinos. The lesson is that although scientists are often stubborn and stick with their theories to the bitter end, science at large has proven to be extremely flexible, and almost on a daily basis theories and hypothesizes are questioned or modified to describe reality a bit more accurately.

For those who think that science is just another religion it is also interesting to compare how often you make changes in the bible (next to never), and how often you make changes to science textbooks, the foundation of what we teach new students in a field. In neuroscience may textbooks come out with new editions every 2-4 years to keep up with all the new facts.

No comments: