Reviewing paper has many benefits. We can learn about new research; the process of reviewing forces us to critically examine a paper, which is what we should do for our own papers, and to explicitly write down why the paper should be or should not be published; we can also learn about the review process itself, which can help addressing reviewer’s comments for our own papers.
Of course, the primary drawback of paper reviewing is time commitment. Reviewing a paper can take between one to many hours. Most scientific peer review is unpaid work. When declining, it is nice to provide a list of alternative reviewers, which can be quite helpful
When reviewing, don’t be obnoxious. Acknowledge good stuff. Be constructive. When reviewing, always be sure to check what is the expected format of the review report. Starting from the final format can save a lot of time. Start reviewing a paper by starting to write the review — thinking about the final product from the beginning.
Addressing peer review
When on the receiving end, first, assume the best intention of the reviewers and don’t dismiss the reviews too quickly. Not properly understanding, or putting not enough efforts to understand the reviews is probably the most common pitfall for junior researchers. The reviewer might have actually understood the paper and what they were trying to say may be indeed insightful. They might have to write it too quickly or be in a terrible mood. Try to really understand what the reviewers meant. Second, stay positive. Even the harshest reviewers can change their minds if we can provide a robust response. Even the insurmountable comments can be addressed nicely more often than it seems. Addressing reviews feel much harder at the beginning but it gets easier as we carefully understand and address comments one by one. Third, stay polite and civil. Trying to discredit a reviewer is the last resort. If a reviewer is completely unqualified, make it evident to the editor by crafting a strong, yet polite response. Finally, consider delivering more than what is requested or needed. Demonstrating something with no possible doubts can save time (removing the necessity of further rounds of reviews) and can change reviewer’s mind.
Studies on peer review
- Single versus Double Blind Reviewing at WSDM 2017
- Dear Reviewer 2: Go F’ Yourself
“Reviewer #2 is not the problem. Reviewer #3 is.”
- Gross2021why - ex post vs ex ante
- Cortes2021inconsistency - NeurIPS 2014 experiment
- Big names or big ideas: Do peer-review panels select the best science proposals?
- The NeurIPS 2021 Consistency Experiment
- http://sunelehmann.com/2010/08/24/no-more-supporting-material/ - How can we solve “supplemental material arms race”?
- Peer review and the meat grinder by Aaron Clauset
- http://www.timeshighereducation.co.uk/story.asp?sectioncode26&storycode=417576&c1 - don’t review for free, for closed journals.
- The Nastiness Problem in Computer Science by Bertrand Meyer
- Yes, Computer Scientists Are Hypercritical by Jeannette M
- A world without referees by Larry Wasserman
- Who’s Afraid of Peer Review?
- Reviewing the reviewers
- A little bias in peer review scores can translate into big money, simulation finds
- Let’s make peer review scientific
How to review
- How to review a paper
- IOP:introduction to refereeing