Peer review

Rebuttal

20240315_peer_review_howto

Doing a review

Should I do it?

Reviewing paper has many benefits. We can learn about new research; the process of reviewing forces us to critically examine a paper, which is what we should do for our own papers, and to explicitly write down why the paper should be or should not be published; we can also learn about the review process itself, which can help addressing reviewer’s comments for our own papers.

Of course, the primary drawback of paper reviewing is time commitment. Reviewing a paper can take several hours or days. Most scientific peer review is unpaid work. When declining, it is nice to provide a list of alternative reviewers, which can be quite helpful for the editor.

How to do peer review?

General rules are: don’t be obnoxious. Acknowledge good stuff and be constructive. Be sure to first check what is the expected for review report by the journal/conference. Then start by writing the summary.

Start the review with a summary of the manuscript. Try to write this summary, as you actively read the paper, in your own words. You can then summarize the primary contributions and the relevance and importance of the paper (note that journals like PLOS ONE explicitly asks not to evaluate the potential impact).

This leads to your overall recommendation: whether the paper can be, in principle, accepted or the flaws are big enough that the paper should be rejected. Ask yourself whether the conclusion of the paper is supported well enough by the evidence presented by the manuscript.

Younger researchers tend to be harshier as a reviewer. If you are new, try to err on the side of giving more benefit of doubt. Papers can change a lot during the review process and authors may have really good counterarguments against your concerns. Frame your job not as a judge who decides the fate of the paper, but as an anonymous sage who helps the authors to make the paper better.

The following paragraphs should support your decision. Try your best to be constructive. Starting from the biggest concerns or comments, lay out the issues you found. Explain each issue well with enough substantive evidence. “This method is not good” or “The paper does not cite many relevant papers” are not enough. Explain why the method is not good; provide specific citations that the paper missed. Ideally, provide potential ways to address your concerns.

Although you, as a reviewer, may have an outsized power in the peer review process, remember that all of your arguments should be well-supported and well-reasoned. Be especially careful not to make disparaging or personal comments. You don’t need to say “the authors clearly do not know much about this field”, you can instead just list the important papers that the paper did not cite and explain what important points have been misinterpreted by the manuscript. Just criticize the manuscript without making it about the authors.

Reflect on your biases. Are you commenting on the writing quality because the authors have non-English names? Are you examining the paper more critically because the authors are not in the top institutions or some developing countries?

Don’t be an a-hole. Being anonymous (not completely) does not mean that you can say anything! The most important thing to do is thinking in the authors’ shoes. Imagine what you would feel if you receive your review report. Would you be grateful or devastated?