How to review a paper

Expectations for paper reviewing should be more clearly communicated. It is not a complicated process, but it is a time consuming one. Through my Ph.D. I have reviewed papers for 8 different conference/journal venues (and many of them multiple times) at the intersection of machine learning and robotics. I may have overdone it on reviewing, but it should feel like the responsible thing to do.

Here is the process I have converged at for reviewing (with a few knobs being fine tuned). Let me know if there is something I should add!

My Process

My process is split into two phases, a deep reading phase (1-2 hours) where I try to have an open mind and be curious (though at the end I am normally like WTF??? WHY? UGH) and a clear feedback phase (1-4 hours). I do not start writing anything in a review until I have completely read the paper and am starting to develop an understanding for its point of view and execution.

The cumulative mental energy in both of them is similar, though I find the deep reading phase to be concentrated in a shorter time period of a couple hours. It is really crucial for it to be hyper-focused, else I end up having read a paper without anything to say. Thankfully, this paper reading process is similar to what I do when a friend asks for advice on a paper. It’s less of a cognitive load to have one mode of deep paper reading, so you don’t have to switch hats between friend, collaborator, and reviewer.

The clear feedback phase takes a lot of time, but normally having done the full read (or two or three) correctly makes the actual write up of the review a formality. The clear read generates feedback and ideas in the margin that you should translate to coherent and meaningful feedback. Though, papers should generally be reviewable by you without much additional searching if you are an expert. Clear feedback takes the form of line numbers, quotes, specific questions, links to relevant papers, and more. The text documents can get quite lengthy (I have maxed out a couple review windows for RAS papers).

Reading: Markup

You’ll give a good review if you read the paper as if you would a close friend or colleague who asked for feedback. There is no reason to put on a tough face for a review because it happens to be in a conference. Also, you should and can be supportive of what authors have done well in a paper review.

If you don’t have any markup in the margins for about half of a page, you probably did not read closely enough, or you are going to nominate this paper for an Oral Presentation and it’s near perfect.

Here is the specific markup format I use (colors are your choice):

Core messages and highlights: Highlight contributions, interesting statements, positive feedback in green,
General understanding & feedback: Circle, squiggle, cross-out, exclaim, etc. at the content in a neutral color (e.g. blue)
Grammar / language: Nit pick grammar / minor confusions in red (separate these from content comments). It is important not to dock the score because the author may be non-English native, though you can lower the score when it interferes with understanding.

Setup-wise, I use PDF Expert on an iPad Pro with an Apple Pencil. Tablet or physical paper recommended strongly. Zoom in, nit pick, write questions in the margin, draw the diagrams the paper should’ve included.

Feedback: Writing the Review

There are some things that I don’t think are normal in reviews, but I think should be required:

Say how to improve the score: Reviewers stating what revisions would cause them to modify their score. Better yet, link specific targets for different potential scores. What would take weak reject to a weak accept to a strong accept? Thinking about this as a reviewer also makes you reflect on the important pieces of papers you will submit.
Be curious and open: Open comments, questions, and ideas. As a reviewer you are an expert on at least one aspect of a paper. Having a section of the review not count in the score and just be open ended comments should be encouraged. It helps with the open flow of research ideas that I view central to science.
Ask for help: Say you are confused and ask for clarification. Given that most reviewing happens in plain text, no one will ever read your body language and know you may need a little bit of help. Ask for it, it just makes the process work better. No reviewer is expect to understand 100% of a paper.

I am figuring out which template I like to use for my reviews still. They are similar, but one breaks down the review into per-section comments after laying out the key points. The other is more high level (let me know which you think is better!).

The key is to write down almost all your thoughts. It takes a ton of time to get all these thoughts down, but it is worth it. Later in the writing process, you realize the pain points of your reading and analysis. This pain points then often take your central points / weakness of the paper. Distinctly summarizing this points is crucial to a good review (writing a lot of points without linking them together is not very useful).

Your review will be way better if you take the time to edit it after you went all the way through, but I understand normally these are deadline pushes so it may slip through the cracks.

Template - Full length

You can download my template here.

This is a review in the style of how I would help a colleague or friend when preparing a paper. I think it could be a little over the top when reviewing for conferences, but all of the comments are suggestions, so the authors should be happy with more opportunities to improve.

Template - Shorter review

You can download my template here.

This is what I think most reviewers at comments do (barring the worst reviews that do more harm than good). It’s a general paper summary, and main review, and a review summary.

Choosing a template: honestly I normally do this after my first read. While my default is to do full (and some conferences give you their own format, grrr), if a paper is a clear reject you likely should do the shorter review format. It'll save you a lot of time. In addition, you can do the shorter review template for a clear accept, but go into detail if you're going to nominate it for an award or an oral presentation.

Other comments (will keep updating this with thoughts and feedback)

Good papers are easier to review (easily half the time). People get better at writing good papers with better reviews. Let’s keep that going.
Reviews without rebuttals: With papers that send the decisions directly to authors without a rebuttal phase, the reviewers should err on the side of clarity instead of interesting-ness. This is openly hard for me to do, but there is no feedback loop to correct misunderstandings.
Number your feedback: When providing feedback about a certain section, you can number each bullet point like i1, i2, … , i5 (i for introduction), which makes it 100x easier for authors to respond to your points.
Reading related works: To make clear feedback I often look at one or two related papers I was unfamiliar with, but honestly it doesn’t usually help that much. Something to consider when writing papers.