Part of being a good citizen in the research community is volunteering to review submissions to conferences. I’ve just finished a large pile of reviews. A few conferences coincided and so over the last couple of weeks I’ve reviewed eleven papers - five from CSCW, five from CollabTech, and one from UbiComp.
For those non-research readers, the process works something like this:
- A conference has a certain number of papers they can publish, but they receive a lot more (for a good conference it’s usually around about five times as many);
- There’s a committee that decides which papers get published. They’ll send out all the submissions that they receive to researchers that have volunteered to review;
- The reviewers score the papers on a scale, dependent on the conference - I’ve seen scores out of 5, 6, 7 and 10. Various conferences also have different metrics, notably CollabTech this year wanted a “coolness” score. The reviewers also provide a written review to justify their scores.
- The committee uses the scores to guide them in picking the 20% that will be published.
- The reviews are sent to the authors with the final judgment. This elicits a variety of reactions amongst the authors, most of them variations on anger and depression.
However, being a reviewer isn’t all rainbows and bacon vodka. A proper review is a lot of work. An easy review - where I know the topic - takes me a few hours of solid work. My main concern though, is whether I’m too impressionable.
After submitting a review, you get to read what the other reviewers have said. And so often it sways my opinion. Which worries me a little bit. Do I not know what I’m doing and not notice the obvious flaws or contributions? Am I not sure of myself enough and think “everyone else gave it 2 out of 5, obviously I’m wrong by giving it 4″? Incompetence and spinelessness are not particularly good options to choose from.
Post a Comment