CHI conference review process

.
I submitted no less than three papers to the upcoming CHI conference in Boulder, Colorado and I wrote a blog post about it in September, almost three months ago. CHI is of course The ACM CHI Conference on Human Factors in Computing Systems - "the premier international conference of Human-Computer Interaction". Let's just say it's a Big Thing for people in Human-Computer Interaction (HCI) and it's notoriously hard to get your papers accepted to the conference. Beyond submitting three full papers, I also reviewed five paper for the conference. This blog post is about that process (both of submitting and of reviewing paper for CHI).

Since the process is double-blind and very rigorous, I did not divulge any information at all about my contributions back in September but I can now state that two of my papers were accepted. That's a lot. No less than 2424 papers and notes (short paper) were submitted to the conference and around 25% were accepted. The acceptance rate does not tell the whole story though as this is the conference where the premier researchers in the area send their best work - so I'm really happy about the outcome and would have been happy also with "only" having had one paper accepted. I will follow up this blog post with two additional blog post about the two papers that were accepted to the conference.

Here's this year's timeline for submitting a paper to CHI:
- Sept 21 - deadline for submitting full papers (lots of work just before this deadline)
- Nov 18 - reviews ("preliminary decisions") are sent back to the authors
- Nov 23 - deadline for a rebuttals. As an author, I have the chance to answer the reviewers and explain how I plan to fix the problems they have pointed out.
- Dec 12 - notification of acceptance/rejection

I adhere to the same deadlines as a reviewer but the process is of course inside-out; I work as a reviewer during those periods when I can relax as an author. I more specifically wrote reviews and argued with other reviewers in mid-November and then agreed or butted heads with my co-reviewers in the beginning of December.

Writing three papers at the same time was very stressful back then - ages ago (September). Fortunately I was the first author of only one of these papers. After submitting, there is not much to do but to wait and hope you were allocated "good" reviewers. They don't necessarily have to agree with you and they don't even have to like your paper, but you do want them to be open to your ideas and for them to treat them fairly. I won't say more about the papers that were accepted since I will write separate blog posts about them following this one. As to the paper that was rejected, we are already making plans for how to rewrite it and where and when we will submit it to another conference. We really like the paper and think it has potential despite the fact that it was (quite narrowly) rejected. I strongly suspect that the paper could have been accepted had some other reviewers been allocated to review it - there is certainly an element of randomness and chance in the review process since there are so many reviewers and so many papers in play. It's also harder when you take a risk and write something that is outside of the fold/breaks the mould. Worse is that you can rewrite a paper according the the feedback you get and then the next time around get other reviewers who have other ideas about what is lacking in your paper. The worst case scenario would be if these new ideas contradict the feedback you got a year ago (and have since then acted upon). Submitting to a conference can be tough at times.

For the five papers I reviewed, it was often the case that there were three reviewers and a "super-reviwer" (an "associate chair" - AC) and then a second associate chair who popped in to look at the paper. So if every reviewer had five paper to review (like I had) and there are AC's that roam around and look at a considerably larger number of papers, well I guess there must altogether be almost as many reviewers involved as there are papers submitted to the conference. Perhaps there were as many as 1500 to 2000 people involved in the process of reviewing papers for this particular conference. That's a huge undertaking. I'd say the time and care that is poured into the process of judging papers in quite amazing. That does not mean the outcome will aways be the "right" one, but the effort to review these papers is not just large - it's humongous.

I reviewed papers last year and there was a paper I liked that some other reviewers were not equally fond of. I did not have time to engage in the review process after I submitted my initial review and the paper was in the end not accepted to the conference. I have since - and due to this incident - promised to make and effort and stand up for any paper I really like. This in fact happened to one of the five papers I reviewed this year. Me and reviewer #3 kind of turned on each other and it sort of became personal. I liked the paper a lot and gave it top grades while the other reviewer disliked it immensely and gave it a really low grade. And then we fought over who was right. I'm now very curious about the outcome of this process as I don't currently know if the paper in question was accepted or not.

The bad news is that I will have to work quite some with the two accepted papers during my vacation (or "vacation") since the deadline for the final, camera-ready paper is on January 6. The good news is that I will go to Boulder, Colorado and to the CHI conference in May 2017 for the first time in three years. I will for sure take the opportunity to join a workshop (or two) in the days preceding the main conference. 

.