Reviewing

Writing good reviews is actually quite difficult. But if you practice it well, you also get to know what pitfalls to avoid when writing your own papers.

Here is a structure you can use:

• A summary of the paper
• some positive elements
• negative elements
• Nitpicks (e.g., typos)

Always try to help the authors

When talking about negative elements – things that you would want to see improved before you would deem a paper as publishable – try and give concrete suggestions of what you might do to fix the issues you found.

Take time

Good reviews take time. I try to quickly read the paper soon after I get it for review, and make some notes or comments. I come back to it a couple of weeks later and see if a) my comments still make sense and b) Are there things that I might have missed on first reading that shed more light on the paper.

Especially on papers that I think are good (even if I think the current version is not ready for publication yet), I try to read up on related work that I am not familiar with. This tells me whether the authors have understood the topic sufficiently for me to trust their results and can be very useful in advancing my own understanding. Beware though that this can take up a LOT of time, so unfortunately I don’t manage to make the time for this often. (If you are a subreviewer with 1-2 papers to review, this step may be feasible. It may also be essential if you are starting out with reviews for the first time, and lack experience.)

Try not to sit on the fence:

A weak accept or weak reject is probably the easiest recommendation to make. Apart from a few papers that are so egregiously bad they deserve to be printed on toilet paper rolls and a vanishingly rare minority of papers that are so brilliant that they are a shoo-in (I can count on one hand the number of times I have got such papers for reviews), most papers do fall somewhere in the middle. Their fate could go either way, depending on their luck with the other reviewers.

Help the process along by taking a firm “accept” or “reject” decision rather than their weaker versions. Many conferences have an in-person PC meeting or a discussion online after all reviews have come in. Would you make a case for or against the paper in such a discussion?

Another consequence of taking a firm stance is that it sharpens your review as well - if you decide on an accept, you will find that some of the niggles in your review no longer matter as much; if you decide on a reject, the enormity of what you are doing may force you (at least it forces me) to provide sufficient objective evidence and arguments for things that you disagree with.

Some advice from others:

Alan Meier discusses components of a good review

This advice from Dave Anderson has examples of some very successful papers, together with their reviews - a good exercise in understanding reviews as well as in understanding how papers can (and should) change between iterations :-). This is also an indication of the potential of a good review - it can help make step changes in the field.

Graham Cormode’s “ How not to review a paper” discusses several dark patterns to avoid.

Some other good advice, which aligns with what has been said above:

• John Regehr
• Shriram Krishnamurthi