Toxic comments are an unfortunate reality in the peer review process, affecting scientists at all levels. This article delves into the systemic failures of peer review, providing insights into its pitfalls and toxic practices.

# The State of Peer Review in Computer Science

Peer review is a fundamental aspect of academic life for computer scientists and researchers. Unfortunately, it often leads to serious issues, resulting in poor decisions and negative experiences.

> Peer Review Example > > Reviewer 1: Good paper — accept > Reviewer 2: Paper OK — weak accept > Reviewer 3: Paper not so great, might be useful after changes — weak reject > > Meta Reviewer: THIS PAPER IS BROKEN BEYOND ALL REASON — **REJECT!

Such scenarios are common in the scientific community, causing emotional distress. Is the peer review system fundamentally flawed? Why do we continue to engage in this process? It's time to examine the intricacies, flaws, and some notably toxic instances of peer review! (Caution: Potentially distressing content ahead!)

In this discussion, I aim to clarify the peer review system in computer science and highlight its shortcomings. Additionally, I'll share personal experiences of misconduct within this framework.

# The Importance of Peer Review

The peer review process has evolved over time to ensure the integrity of scientific publications. Every researcher is susceptible to making mistakes, making it crucial to have independent evaluations of their work to avoid publishing inaccuracies. A significant error can have lasting repercussions, as published work is widely accessible. Thus, peer review serves as a protective measure for researchers.

Consequently, many scientists regard only peer-reviewed studies as valid scientific evidence. Here, I will highlight three primary forms of peer review:

• Conference peer review
• Journal peer review
• Review of theses, such as Ph.D. dissertations

# How Conference Peer Review Operates

The most prevalent form of peer review occurs at scientific conferences, often serving as a researcher's first exposure to the review process. Depending on the discipline, conferences can provide varied levels of feedback, ranging from superficial comments to rigorous evaluations. The reviews are typically conducted under tight deadlines, which can lead to rushed evaluations. In some cases, as many as 30% of reviews may be submitted after the deadline, often by "emergency reviewers" within 24 hours. Such rushed reviews are inherently more prone to errors than those that receive thorough consideration.

# Types of Conference Reviews

To foster constructive and critical feedback, many conferences implement a blinded review process. In a single-blind review, the identity of the reviewer is hidden to prevent retaliation against negative comments. However, some authors attempt to deduce reviewer identities based on language and terminology used, which can lead to unfounded accusations.

Reputable conferences may adopt a double-blind review process, where authors remove all identifying details from their submissions. This ensures that reviewers do not favor particular authors or institutions based on personal biases.

# The Randomness of Peer Review

Reviewer decisions can exhibit a degree of randomness. Factors such as wording and overall concepts can sway opinions, leading to a perception that a single reviewer is not a reliable source. Most conferences mitigate this by employing two to four reviewers, relying on a majority consensus to evaluate submissions.

In prestigious conferences, the introduction of a meta reviewer helps facilitate discussions when initial reviewers disagree. However, some meta reviewers can act with an inflated sense of authority, which can lead to flawed decisions. Even meta reviewer outcomes are often subject to oversight by technical program chairs, but this can introduce additional randomness, especially when managing a large volume of submissions.

To further enhance decision accuracy, some suggest involving a significantly larger pool of reviewers, potentially 30 to 50. While this approach could yield more favorable outcomes, it risks reducing reviews to mere "thumbs up" or "thumbs down" assessments, as thorough evaluations of full papers would be impractical.

# Advantages and Disadvantages of Conference Reviews

Typically, conferences utilize one or two-step review processes, limiting the opportunity for in-depth discussions between reviewers and authors. This issue persists even in more robust review systems.

Thus, do not let negative or harsh reviews demotivate you. Many reviewers volunteer their time under significant pressure, and mistakes are part of the process. Strive to comprehend the critiques and use them to enhance your work.

Some conferences prioritize high rejection rates as a measure of quality, leading to more intense reviews. While the peer review process is inherently stochastic, higher-quality papers are more likely to pass through successfully.

However, a rejection does not necessarily reflect the quality of your work. There are several instances of well-regarded papers being initially rejected at conferences.

# The Value of Journal Publications

A significant drawback of journal publications is their extended review cycles, often lasting three to six months. Nevertheless, submitting to journals remains valuable due to more comprehensive reviews focused on detailed arguments. In many journals, the dialogue between authors and reviewers can go through multiple rounds, allowing for a deeper examination of the work.

Reviewers are aware that their assessments will be presented to the authors, which incentivizes them to articulate their critiques thoroughly. Vague comments such as "not novel enough" are less common, as reviewers must provide specific examples of existing work that were overlooked.

Authors should view reviewer comments as opportunities to enhance their papers rather than personal attacks. It's essential to engage with critiques constructively, ensuring that the paper improves with each revision.

Rejection from a journal prevents resubmission unless the paper undergoes significant revisions, which is often advisable anyway.

I recommend submitting to high-ranking journals and conferences, as a desk rejection typically occurs within two weeks, providing a sense of closure. Notably, publishers like Nature offer pre-submission inquiries, allowing authors to receive feedback on their abstracts quickly.

# The Future of Ph.D. Theses

Another type of review worth discussing is the assessment of thesis work, particularly Ph.D. dissertations. Some advisors require candidates to submit accepted peer-reviewed papers as part of their thesis. Many universities even allow cumulative theses, where a series of peer-reviewed articles replaces a traditional thesis. However, this method presents challenges.

Since theses are tied to formal degrees, cumulative works must demonstrate that they are primarily authored by the submitting individual. This often necessitates co-authors to authorize their papers' inclusion in the thesis, helping to prevent misuse of research.

Despite the safeguards, there is potential for individuals to claim credit for collaborative works they did not contribute to. Consequently, some universities still mandate the submission of formal theses, with proponents arguing that this is essential to showcase genuine academic ability.

Ultimately, theses are assessed by independent reviewers who aim to evaluate the work fairly. Demonstrating that your research has resulted in high-quality peer-reviewed publications can significantly bolster your case during this evaluation process.

# The Challenges of Peer Review

The pressure to publish in peer-reviewed venues can lead authors who have faced previous rejections to seek out lower-ranked journals for quicker acceptance. While this may seem like a viable solution, it can lead to complications.

Predatory journals frequently promise rapid publication with peer review but often fail to conduct thorough evaluations. They may accept any submission without proper scrutiny, risking the author's reputation, even if they were unaware of the journal's true nature.

Similar issues arise with conferences. Predatory conferences may be challenging to identify, often held in appealing locations to attract submissions. A common warning sign is multiple conferences occurring simultaneously with similar themes and speakers. Consulting colleagues or established conference rankings can help avoid these pitfalls.

Recently, video journals have gained popularity. While they may appear rigorous, the production costs can be substantial, so authors should assess potential publication expenses before submission.

# Toxic Conduct in Peer Review

While outright misconduct is relatively rare in peer review, toxic behavior is more prevalent. The examples discussed here range from mildly unpleasant to extreme, and gathering these anecdotes has taken years of experience. Scientific fraud is uncommon, but it does exist.

One prevalent form of toxic behavior involves overly negative reviews of others' work. Reviewers may attempt to undermine submissions from competing institutions. Since many conferences have strict acceptance thresholds, negative reviews can increase the chances of one's own group's papers being accepted. Consequently, some reviewers may seize on minor flaws to reject papers, using vague language such as "not novel" or "unclear," which can be detrimental to authors.

Reviewers looking to enhance their citation metrics may also request citations of their own work or that of their colleagues, thus jeopardizing the impartiality of the review process. This practice can make it easy to identify the reviewer, undermining the anonymity intended in peer review.

As submissions have surged in recent years, some reviewers may delegate assessments to junior team members, which can lead to errors and compromise confidentiality. While these issues have been recognized, initiatives to involve early-career researchers in the review process can help alleviate reviewer workloads.

Editors and meta reviewers wield significant power and may be tempted to direct outcomes based on their convictions. However, prematurely concluding a controversial review can lead to misguided decisions, particularly if the majority opinion among reviewers is overturned.

Another troubling practice is when meta reviewers and editors assign papers solely to their own students, which can foster cliques and lead to biased evaluations. Ideally, reviewers should come from diverse institutions to ensure fairness.

# Notable Instances of Misconduct

While instances of scientific misconduct are infrequent, they can carry severe repercussions when uncovered. Plagiarism is particularly easy to detect and can lead to significant consequences. Many journals and conferences now utilize plagiarism detection software to prevent such issues.

Some authors resort to "paper mills," often utilizing lower-ranked or predatory journals to inflate their citation counts. Additionally, mutual citation networks may form among authors, which can lead to abuses of the system.

Occasionally, falsified research results emerge, and severe cases may receive media attention. While the allure of groundbreaking discoveries can be tempting, the inevitable scrutiny will reveal any inaccuracies, particularly if research cannot be replicated.

# The Surface of a Larger Issue

Many of the aforementioned misconduct cases are familiar to scientists. However, there are lesser-known instances that warrant caution. The following examples highlight troubling practices that have been reported, but I will not disclose any identifying details.

Peer review is inherently stochastic. As probability dictates, the more submissions made, the higher the chances of publication. However, this process is time-consuming, as simultaneous submissions are generally prohibited. Some groups have attempted to exploit this by submitting the same paper multiple times under different titles and abstracts, thereby increasing acceptance chances.

In one notable case, an author submitted a paper to a prestigious double-blind conference only to have it rejected narrowly. After making improvements, the paper was submitted to a well-regarded single-blind conference, only to be accused of plagiarism. An investigation revealed that an anonymous reviewer had uploaded the original submission to a public file server, leading to the confusion. Ultimately, the author was cleared of wrongdoing, but the incident highlighted the perils of the peer review process.

# Addressing These Challenges

If you encounter significant toxic behavior, you can report it confidentially to the relevant conference or journal. If you suspect a reviewer has acted improperly, communicate your concerns to the associate editor or meta reviewer. However, it's vital to approach such situations with an open mind, as seemingly biased requests may stem from legitimate grounds. Always evaluate your suspicions critically.

Utilize ArXiv! Once your ideas are published under your name, they become less vulnerable to the anonymity of peer review, safeguarding your intellectual property. ArXiv submissions are increasingly accepted by major machine learning and pattern recognition conferences, provided all co-authors agree and any necessary disclosures are made.

Note that ArXiv is not a peer-reviewed platform; papers there may undergo revisions until final publication. Avoid presenting ArXiv entries as prior art, especially if they were posted post-deadline.

Not all countermeasures must be the author's responsibility. Many conferences and journals now employ plagiarism checks, providing reviewers with both the paper and its plagiarism report for evaluation. Additionally, meta reviews and reviewer assignments are scrutinized by committees to uphold integrity.

Misconduct cases are handled discreetly to avoid damaging the reputation of science. Scientists are generally seen as trustworthy and knowledgeable. Keep in mind that misconduct is rare; more frequent are honest mistakes. Always give others the benefit of the doubt.

Unfortunately, there is no universal code of conduct outlining misconduct and associated penalties. Each case is treated individually, leading to varying consequences across communities and countries. Be aware of the regulations that apply to you, adhere to them, and actively combat toxic behavior in peer review, such as by promoting awareness among colleagues. Avoid public accusations, as these can harm your career.

If misconduct is observed, every academic institution has an arbitration body to address the issue appropriately. In some fields, publicly accusing someone of misconduct could be considered misconduct itself, potentially harming the integrity of science. Remember, peer review is complex, stakes are high, and behaviors can be easily misinterpreted. Assume everyone is acting in good faith, but be prepared for the worst.

Lastly, do not attend a conference that rejected your work to distribute your paper there. This behavior is ill-advised and can lead to negative consequences.

# Key Takeaways

This discussion has illuminated many issues inherent in peer review. However, no superior system has emerged. It's essential to recognize that there is no single standard for peer review; each scientific community creates its own criteria and continues to evolve them to enhance the process.

ArXiv serves as a robust countermeasure against unfair peer review, alleviating the pressure for early publication. However, it’s important to remember that a paper on ArXiv is not considered fully validated until it has been successfully peer-reviewed.

Toxic behavior in peer review only diminishes the quality of the process. Such actions are often motivated by a desire for unfair advantages and can perpetuate a cycle of toxicity among researchers. Therefore, it's crucial to refrain from engaging in toxic practices, even if you've previously been a victim of such behavior.

Some behaviors may seem like misconduct but can arise from unfortunate circumstances. Always give others the benefit of the doubt. If you suspect misconduct, contact your institution's arbitration body. Avoid making public accusations, as they may jeopardize your career.

Lastly, treat your colleagues with respect and aim to foster a supportive environment in the scientific community. If you found this article insightful, more essays and educational content can be found on my platforms. Your engagement on social media is appreciated as I share more research and insights in the future. This article is published under the Creative Commons 4.0 Attribution License, permitting reprints and modifications with proper attribution.