This paper kicks ass.
It's great to referee a manuscript in which I can say that. The more reports I have written, the better the papers that I get to review.
Here is how I write a report.
- Try to figure out the point of the paper. Sometimes it's easy. Sometimes it's hard. If it's too hard, I may give up.
- If I understand the point, then I decide if it's new or not (to me and the literature) and if the analysis in the paper is correct. If the point is new, correct, and interesting, then see if I can help the author improve the exposition, results, and scope of the paper. If the point is not new, or incorrect, or uninteresting, then clearly explain my position in the report.
- If the writing is bad, I won't get the point. If the technical material is wrong or sloppy, I won't get the point. If the point is correct but the technical material is sloppy or the writing is bad, then it is going to take me a lot of work. I need to decide if the point is worth the work that has to be put in.
- The subjective part here is deciding if it is an interesting point or not. The editor wants my opinion, or I would not be chosen to be the referee.
- Sometimes what the author thinks the point is turns out to be different than what I think the point should be. Sometimes the author puts many points in the paper, so I need to help the author focus on a few of them.
- I always try to help the author improve the paper, with specific suggestions, even when I think the paper should be rejected from the journal. This involves going through the manuscript carefully a few times with a pencil and pad.
- I am always clear to the editor about what I think the outcome should be, and if it's a revise and resubmit I tell the editor and author exactly what I think would lead me to suggest acceptance.
- I am never wish-washy about what I think should happen. That doesn't mean I rewrite the paper, but it does mean that I try to be specific as possible. I have been burned a few times myself by wishy-washy reviewers who don't tell me what is required to have a successful outcome. In all such cases, the paper has gone more than one round only to be rejected in the end. Probably my fault, but it did waste everyone's time.
- I always try to take the work seriously.
- I try never, never, never to be sarcastic or mean. Of course trying may be be the same as achieving. But that being said, the most useful report I ever got was full of sarcasm--intended or not. The report forced me to work on being precise and clear in that paper and in future research. Helpful in the long run, even though the paper did not get accepted at that journal.
Many of the journals I review for have double blind refereeing. But since I see so many papers at seminars and conferences, I always wonder about how blind it is. I try hard not to let the author's identity influence my refereeing.
I find that the best way to get a report done is to read the paper, think about it carefully for a while, and then start writing. I usually start by writing a short paper summary, and then by going through the manuscript with a pencil or my text editor open, noting unclear and incorrect things. I then write a few paragraphs about why I or do not think the paper is successful. Rinse, lather, and repeat a few times until I am happy with the report.