ABOVE: Journal editors should play a more active role in the manuscript review process. ©istock, Nazan Akpolat
Disclosing conflicts of interest (COI), financial or otherwise, is quite rightly a compulsory part of scientific discourse. After all, motivated reasoning and bias is often subconscious, and even honest and well-meaning scientists can be inadvertently swayed. Ironically, however, a far more insidious intrinsic COI (iCOI) is both ubiquitous and unstated in academia. Academics who fail to publish papers and secure grants can lose their labs, might be relegated to undesirable professional tasks, could experience lowering of salaries, and may ultimately be discharged. This scenario poses an omnipresent conflict for essentially all authors who have no conflicts to declare. As Upton Sinclair famously quipped, “It is difficult to get [people] to understand something when [their] salary depends upon not understanding it.”1
In their landmark paper modeling the effects of what is commonly called p-hacking, researchers from the University of Pennsylvania and the University of California, Berkeley showed that the subtle introduction of degrees of freedom rendered a p-value cutoff of 0.05 essentially meaningless, as it resulted in a 61 percent chance of a type I error.2 P-hacking, data dredging, and data mining have become pejorative terms for the common practice of looking at data from multiple angles, which is a requisite part of good science. The problem is that researchers may think they have only tested one hypothesis when they have inadvertently tested multiple hypotheses. Use of a statistical test that doesn’t account for multiple hypothesis testing then overestimates significance. This is part of what has been coined the reproducibility crisis, and the researchers presented a list of potential remedies in the paper. They made six recommendations to authors and four to reviewers. All the recommendations focused on increased rigor, with one exception that stated, “Reviewers should be more tolerant of imperfections in results.” It might seem strange that a paper focused on increasing rigor advocated greater tolerance for imperfections. The reason is that if standards of scientific publication are too high, then the pressures of iCOI drive distortion, exaggeration, and at times outright lying. It remains a publish or perish world and no one wants to perish.
Peer review is one of the sacred characteristics of modern science that makes scientific claims distinct. The validity of a claim is not based upon the authors’ authority; rather, qualified peers determine the validity of researchers’ claims based upon the data. Usually, the reviewers are anonymous, so they feel free to give an unvarnished assessment without the fear of political fallout. This process helps remedy the confirmation bias that humans seem quite incapable of escaping from, despite actively trying. This is not to say that reviewers are not prone to their own particular biases as well, but they may view the data from a different perspective than the authors. As such, what is confirmation bias for a reviewer may serve as disconfirmation bias against the authors’ claims. In this way, the thought collective has a more balanced internal dialogue.
In the legal system, District Attorneys ultimately decide who is (and who is not) prosecuted and what penalties are sought. It is for this reason that Supreme Court Justice Robert Jackson stated, “The district attorney has more control over life, liberty, and reputation than any other person in America.”3 Similarly, in a scientific peer review system, journal associate editors decide which papers get reviewed, who reviews them, and how to adjudicate the dialogue between authors and reviewers during paper modification. Thus, editors exert tremendous control over the publication process. However, some associate editors may be failing the field by mistaking detached nonparticipation for unbiased impartiality.
Peer review is essential to ensure that author claims are held to the standard of being supported by observation, interpretation, and reason. However, the peer reviewer should be no less bound by these same standards. Sometimes, however, a reviewer may dislike the findings in a paper for myriad reasons other than those based on observation and interpretation. Worse, reviewers could be biased for political or personal reasons unrelated to the science. Any paper can be killed by pulling out the old armamentarium of thought-ending clichés that facilitate "manuscriptocide." In such cases, an author may be rendered helpless.
Ideally, the authors can appeal to the associate editor to determine the validity of the reviewers’ comments. However, at least in my experience and opinion, associate editors have become far too hesitant to get involved. Rather, associate editors more often serve as a procedural jurist, facilitating dialogue between authors and reviewers, with reviewer placation remaining the metric of acceptability. This amounts to requiring unanimous approval for publication and allows a single reviewer to hang the jury, resulting in paper rejection. If the bar is too high for publication, and if we allow the perfect to be the enemy of the good, then iCOI and the need to maintain academic productivity becomes an enormous force of motivated reasoning that will drive p-hacking and other missteps, intentionally or otherwise.4
In some cases, the associate editor may recruit an additional peer reviewer to assess the paper, but this is a further evaluation of the authors’ claims, not an assessment of the disapproving reviewer’s reasoning. Rather, associate editors can and should review the reviewers. After all, editors are carefully chosen experts in their respective fields and are well qualified to evaluate both a paper and the reviewers’ comments. If a majority of reviewers are in favor of publication, overruling a single reviewer whose view is different is not a capricious action by an associate editor.
While overriding a reviewer’s comments risks replacing one bias with another and could encourage inappropriate favoritism, intrinsic bias, and cronyism, there might be other solutions. The editor could also facilitate debate amongst the reviewers, using reviewers to peer review the reasoning of the other reviewers. In this way, rather than being a passive administrator who recruits reviewers and is simply a referee as authors and reviewers hash it out, the associate editor becomes an active scientific mind in the process, which, at least in my view, is what the role entails.
It is not helpful to have associate editors who are disengaged due to a distaste for conflict or lack of time to enter the fray when needed or are under the misguided notion that their role is procedural and not scientific. Of course, associate editors are themselves susceptible to iCOI and must therefore be senior and established enough to protect against actions based on fear of retribution from vindictive reviewers who may take umbrage at being overridden or vindictive authors who may lash out for being rejected. Associate editors who jump off the bench and wrestle in the mud, when required, are desperately needed.
Disclosure of Conflicts of Interest: JZ is a cofounder of Svalinn Therapeutics. JZ also has the iCOI present for all academic scientists.
- Sinclair U. I, Candidate for Governor: And How I Got Licked. University of California Press; 1934.
- Simmons JP, et al. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359-1366.
- Jackson RH. The Federal Prosecuter. An Address Delivered at The Second Annual Conference of United States Attorneys: United States Department of Justice; 1940.
- Zimring JC, Spitalnik SL. Fallacy of perfection harms peer review. Nature. 2016;537(7618):34.