Peer review

Over the years I have noticed that consumers have a very difficult time understanding the quality of a scientific report, whether it is a book, a magazine or journal article.

Scientific knowledge is based on incremental progress derived from hypothesis and experiments that get replicated enough to become common knowledge. This is a slow process. The scientific world is full of ideas that lead into hypotheses, many of which cannot be proven or even put into an experiment. So, where is the starting point? Well, with hypotheses and experiments that conducted and made public.

A peered-reviewed scientific journal article is a body of work originated in a hypothesis, that presents methods, results, discussion and a conclusions, that are “validated” by experts in that particular field of knowledge. This method of assessing scientific information is geared to improve quality and reduce bias (we all “fall in love” with our experiments and have a tendency to not see embedded problems in our work).

The peer-review system works following way (simplified): 1. An author or group of authors generate a written report of a specific area under study (a paper). 2. The paper is then submitted for publication to a peer-reviewed journal. 3. The journal editor picks anonymous experts that would read the paper and provide feedback to both the authors and the editor about the quality of the work and possible changes that could be made to the paper under review to improve the scientific content (e.g.: additional experiments, rephrasing certain statements, etc). 4. The reviewer also provides suggestions to the editor that might range from “accept the paper,” to “reject” this body of work as it is presented if modifications are not made, or reject it altogether (regardless whether modifications can be made). 5. The proposed modifications may or may not be accepted, and finally 6. The editor decides whether the paper will be published or not in her/his journal. This type of publication is what I would refer to as a journal article.

On the other hand, we have magazine articles. These reports usually would not have been reviewed by experts, thus, the writer holds the power to check statements presented as “facts.” In this situation, the writer might introduce voluntary or involuntary bias, reaching conclusions that are not necessarily supported by data.

Finally, we have books, the lowest form of scientific information. Why? Because anyone can publish a book and say whatever they see fit. Often, books with scientific themes would include references. These references could indeed support statements made, but sometimes those references are unreliable and biased themselves. The problem is that we ultimately have to trust the author about the statements contained in the book and bias may be rampant.

If an article is peer-reviewed, does it guarantee the content is accurate?? No. The peer-review system is not perfect either, it’s the best we have. Validation of certain claims may take many years to be verified, it’s replication of the same experiment that increases confidence in a certain finding.

The dissemination of scientific information is not a simple issue. Scientific reports may drive significant amounts of money and power, thus, the risk of bias is always present.