I was recently program chair (PC) of a moderately large CS conference, EMNLP-CoNLL 2007, which received about 400 submissions, with an acceptance rate of 27% (under 17% for talks).
To avoid duplication of effort, I saved various materials and am making them available via this page to future program chairs of ACL-style conferences.
This page concerns mainly the responsibilities of the program chair, who organizes reviewing and has final say on the program. There's obviously lots more to running a conference — local arrangements, sponsorship, publications, etc. — which is not covered here.
Many items on this page will also be useful to workshop chairs, especially if they're using the START online conference management system.
The most time-consuming thing for me was writing all the invitations, policies, advice, and detailed instructions. I kept an archive of all those emails — ask me for it. You can edit the emails as you see fit, of course.
Most of my advice is actually contained in these emails. Reading through them ahead of time may also give you a good sense of what's involved in the PC job. Even the list of filenames serves as a good overview:
And here are some useful links that appear in the above emails:(categorized by recipient; chronological within category)
advice-from-last-years-chairs areachair-01-invite areachair-02-invite-reviewers areachair-02-invite-reviewers-reminder areachair-03-reviewform areachair-04-keywords areachair-05-reviewer-policies areachair-06-setup-bidding areachair-07-assign-reviewers areachair-08-assign-reviewers-resubmission areachair-09-assign-reviewers-suggestions areachair-10-reminders areachair-11-invited-speakers areachair-12-status-report areachair-13-discussion-ranking areachair-14-discussion-reminder areachair-15-discussion-ranking-followup areachair-16-revise-metareview areachair-17-bestpaper areachair-18-credits areachair-19-dinner reviewer-01-invite reviewer-02-setup reviewer-03-bid reviewer-04-assign reviewer-05-remind reviewer-06-remind-again reviewer-07-discuss reviewer-08-revise reviewer-09-thanks reviewer-10-program author-1-confirmation author-2-extension author-3-reject-without-review author-4-accept-poster author-4-accept-poster-shepherded author-4-accept-talk author-4-accept-talk-shepherded author-4-double-submission author-4-reject author-5-emptyreview author-6-accept-poster-pleaseconfirm author-7-reminders author-8-program sessionchair-1-invite sessionchair-2-instructions sessionchair-3-reminder
(public face of the conference) http://cs.jhu.edu/EMNLP-CoNLL-2007 http://cs.jhu.edu/EMNLP-CoNLL-2007/CFP.html http://cs.jhu.edu/EMNLP-CoNLL-2007/CFP.txt http://cs.jhu.edu/EMNLP-CoNLL-2007/program.html http://cs.jhu.edu/EMNLP-CoNLL-2007/sessionchairs.txt (how to configure START) http://cs.jhu.edu/EMNLP-CoNLL-2007/keywords http://cs.jhu.edu/EMNLP-CoNLL-2007/reviewer-policies http://cs.jhu.edu/EMNLP-CoNLL-2007/review-form.html (instructions to authors) http://cs.jhu.edu/EMNLP-CoNLL-2007/your-final-paper.html http://cs.jhu.edu/EMNLP-CoNLL-2007/talk.html http://cs.jhu.edu/EMNLP-CoNLL-2007/poster.html
Take the job seriously. People's careers are at stake, as are many thousands of hours of labor by reviewers and other organizers who are doing you a favor.
Avoid duplication of effort. Don't make 15 area chairs (or 400 reviewers) all figure out the same thing, or tweak the same START setting, or write separate instructions for the same task. That's a waste. If you can handle it once for everyone without much more effort, then be a good utilitarian and take it on yourself (or delegate it to one other person). This also ensures that it's done consistently and well and on time: after all, if you ask 15 people to do something, your results may vary.
Set clear expectations. You will be writing a lot of instructions to authors, reviewers, and area chairs (including the CFP and the review form). Be clear about what you want. You can thereby improve the quality of the conference and the fairness of the reviewing. Your instructions also serve to communicate the expectations of the community: you are training new authors, new reviewers, and new area chairs in how to do their jobs.
Ask people to do things way in advance. It may seem unnecessary to invite (e.g.) reviewers 3 months early, when you know that the majority of them won't submit their reviews until the last day anyway. However, when you ask them 3 months in advance, they have nice free calendars, a sense of optimism about the future, and slack to juggle your workload with their other activities. So they'll say yes to you. Let them say no to the poor saps who ask them for favors later, when they're overcommitted and worried about getting it all done.
If you wait to ask, you'll start to get "no"s — and every "no" makes your next attempt even later, triggering a vicious cycle. You will find yourself racking your brain for halfway adequate people to do the job. While you are flagellating yourself for being such a loser, your bank will repossess your house, your significant other will leave you, and your disk array will fail.
As a rule of thumb, you and your area chairs should invite your first choices for anything fully 3 months before they will need to do any real work, so that you can be sure to have those positions filled by 2 months beforehand. Just do it.
Sometimes you will get a "no", or a series of "no"s. If you are not getting any response from someone, try a reminder email or two; then try phone calls; then email apologetically that you'd love to have them, but if you haven't heard back by Monday, you'll just have to move on to the next person on your list.
Make a timetable at the start and stick to it. Here is a sample EMNLP timetable with discussion. (ACL timetables are more relaxed.) Run your timetable past last year's chairs for comment. Think carefully about how it will collide with reality. Consider adding an author response period (see discussion below).
Get help. Many things have to run smoothly. Each one may seem manageable (at least if nothing goes wrong), but you really can't do all of them, especially simultaneously. So delegate in advance (to people you can trust!). If you have a co-chair, then the two of you could divide up some of the responsibilities and delegate the rest. Some possible subchairs to recruit:
Area chairs. You know what these hard-working folks do: coordinate reviewing and selection of papers in the various topical areas. Pick them very carefully. They need broad technical experience in the area, a strong sense of communal responsibility, good judgment, an open mind, and enough stature that they will be able to attract the right reviewers. Try to ensure that the set of area chairs is reasonably well balanced by geography and gender.
Sponsorship chair. Someone to find sponsors, negotiate with them about how much they'll donate and what they want in return, arrange for their logos to appear on the proceedings/handbook/wall/etc., make sure they're happy, collect the money at the end.
Administrative assistant. There are a lot of little things to do — deal with minor special requests from authors, tweak the START configuration, etc. I wish I had paid a reliable undergraduate with good computer skills and people skills to handle these essentially secretarial tasks.
Local arrangements chair. The overall event (conference+workshops) will have a local arrangements chair who will handle most of the logistics for all events in all rooms. But you might want your own local arrangements chair to work with that person, if you have any special requirements.
For example, EMNLP-CoNLL'07 was large enough that we had parallel sessions, two large poster sessions (see below), invited talks, special food, a thank-you dinner for the organizers, etc.
Publications chair. The aclpub
software package handles a lot of the work. But some
detail-oriented person has to listen to the ACL publications
chair, pass instructions on to authors, handle questions and
special requests from authors ("I found a bug in my proof! Can I
please still fix it, I mean after I go camping?"), make
editorial choices like the exact title and color of the volume,
remind you to write the front matter, go through the proceedings
volume with an eagle eye, pay attention to missing accents on
author names in the BibTeX database or CD-ROM index pages, and
work with authors to fix any problems with their camera-ready
files.
The more papers you have, the bigger a job publications is, and the more things can go wrong. If you're running a small workshop, it's only an afternoon's worth of work. But if you're running EMNLP or CoNLL, you should delegate it.
If you're running the ACL main conference, short paper session, or student session, then the publications job is probably already delegated. Traditionally, the overall ACL publications chair will handle your volume directly, as well as coordinating publications for the whole conference.
Best paper award committee. I asked one of the area
chairs to run the discussion of who would get the best paper
award. (See the email areachair-17-bestpaper for
details.) I did this to avoid a COI, but it also saved me work.
If you do this, pick an area chair who has broad interests and who
doesn't have a dog in the fight (i.e., did not nominate a paper
from his or her own area). Or pick another senior figure with
broad interests.
Invited speaker liason. It would be reasonable to delegate arrangements for the invited speaker or speakers. This is a relatively self-contained set of tasks. It includes soliciting names of possible speakers, getting a consensus on whom to invite, issuing invitations, obtaining the title and abstract, helping to arrange travel and hotel, and otherwise playing host.
Roughly in this order:
Write to a small group of recent chairs and other wise heads. Ask if they have any advice, and warn them that you'll be asking for their feedback on some things. (Once you have your area chairs lined up, you can consult them for feedback instead.)
Set up a special email contact address for the conference. It should be a short, intuitive address that goes to you, your co-chair (if any), and your administrative assistant (if any). This address will be widely publicized, so it will eventually start attracting spam, but you can decommission it after the conference.
Now configure your email client to route all possibly conference-related email to a dedicated mail folder that you will monitor closely. That includes all email that mentions the contact address or any form of the name of the conference.Prepare to write the CFP:
Do some rough supply and demand calculations to make sure this conference is going to work as planned. For example, we'd initially hoped to make EMNLP-CoNLL'07 a bit shorter, perhaps with a half-day break between ACL and EMNLP-CoNLL. But we didn't dare, once the back of our envelope (correctly) predicted a low acceptance rate even with 3 jam-packed days using parallel sessions and extensive poster sessions.
Set the timetable.
Find out what the URL for your START site will be, so that you can mention it in the CFP. Try to do at least some minimal configuration of the START site before the CFP goes out, for the sake of early-bird authors.
Early publicity, so that people will plan their research around your submission date:
Make a conference webpage at an short, intuitive URL. It can be a stub at first. Use appropriate META tags to help search engines find your conference.
Write the Call for Papers. Use past CFPs as a model for what to include; then get feedback. Any new or unusual policies that affect authors should be mentioned in the CFP.
Put the CFP on the webpage and send it out to appropriate mailing lists. The ACL Business Office can post it to some ACL-related lists. Other useful lists include CORPORA, LINGUIST, ELSNET, the mailing lists of relevant ACL SIGs, and any other topical lists that are relevant to your particular conference (e.g., machine learning lists). I'm sure I'm omitting some!
A week or two after you send out the CFP, check that your conference webpage is the top search engine hit for the conference's name.
You might also try searching the web for lists of relevant conferences. The people who maintain these lists will probably have already seen your CFP and posted your key dates. If not, drop them an email.
Figure out how to divide the conference into topical areas (a.k.a. tracks), so that you can invite area chairs (a.k.a. track chairs). Questions to ask:
Now you can move ahead with the emails above, starting with invitations to the area chairs.
An overall to-do list. I wondered sometimes whether I should even have set up a request tracker, like RT.
A record of where you've sent the CFP or plan to send it.
A dated log for what's going on with the tracks, noting things like these for each track:
Similarly, a dated log where you record your progress inviting other kinds of chairs, keynote speakers, etc., and dealing with them once they've accepted.
A record of deadline extensions or other special treatment that's needed for particular papers.
Any assignments (of papers to tracks) that you might perhaps want to reconsider. Keep these notes during your first-pass assignment as the papers arrive, so that you can do a second pass of adjustments.
Any outstanding financial issues.
Things you'd like to say in the preface to the proceedings.
Advice for future chairs.
How not to do it: Please, please, please don't just sort the papers by the 3 reviewers' average overall recommendation! There is too much variance in these scores for n=3 to be a large enough sample. Maybe reviewer #1 tends to give high scores to everyone, reviewer #2 has warped priorities, and reviewer #3 barely read the paper or barely knows the area. Whereas another paper drew a different set of 3 reviewers.
How still not to do it: Even as a first step, don't sort the papers by average recommendation. Trust me — this noisy and uncalibrated ranking isn't even a good way to triage the papers into likely accepts, likely rejects, and borderline papers that deserve a closer look. Don't risk letting it subtly influence the final decisions, or letting it doom some actual, nuanced reviews to go unread.
What I told myself: When you're working with several hundred papers, a single paper with an average score of 3.8 may seem to merit only a shrug and a coin flip. But a single false negative might harm a poor student's confidence, delay her progress to her next project, or undermine her advisor's grant proposal or promotion case. Conversely, a single false positive wastes the time of quite a lot of people in your audience.
Thus, you'll want to be pretty confident of every decision, after appropriate discussion. This is possible because you're the kind of broadly experienced person who gets to be program chair, and more importantly, because you have picked great area chairs and given them clear instructions.
Here's a procedure that I recommend.
Collecting rankings from area chairs: I asked each area chair to send me a careful within-area ranking, with comments. Instructions were in these emails:
areachair-12-status-report areachair-13-discussion-ranking areachair-14-discussion-reminder areachair-15-discussion-ranking-followup
After all, the area chairs are the experts within each area. Not only do they understand the standards of the area and the technical content of the reviews, but also they may know their particular reviewers well enough to calibrate them. Furthermore, they are working with a lot fewer submissions than you are, so they have enough time for detail work. Where the ranking is unclear, they can ask the reviewers for further discussion, or look carefully at the submitted papers themselves.
Merging the rankings (overview): It took me a couple of days of hard work to merge these rankings and make the final 3-way decisions (talk vs. poster vs. rejection). Here is what I did, in brief. (A detailed version appears further down the page.)
PC meeting: The above procedure doesn't require any program committee meeting. EMNLP does not traditionally have such a meeting; the final decisions are made by one or two program chairs, who exchange email with the area chairs.
But if you are going to have a physical PC meeting of all the area chairs, you might be able to adapt the above procedure:
Have each area chair send you a careful ranking ahead of time, with comments, and make a spreadsheet as in step 1. The job of the PC meeting is now steps 2-7 above.
You might want to spend some of the PC meeting in "breakout groups," where each group is in charge of 2 to 4 related areas whose rankings must be revised and merged. This is probably more efficient than a single plenary session. (A plenary session cannot deal effectively with the large number of papers, and only a small fraction of the group is familiar with any given paper, anyway.)
To use the meeting time more effectively, you might perhaps try steps 2-5 on your own ahead of the meeting. That would suggest preliminary decisions and identify difficult, anomalous, or borderline cases that need discussion.
Merging the rankings (detailed version): Anyway, here's the same procedure in more detail:
Use START's Spreadsheet Maker tool to create a spreadsheet with one row per submission. (More information about this below.)
Assign a single numerical absolute "strength" to each submission.
Base your strength ratings mainly on the area chairs' absolute recommendations and relative rankings. Within an area, higher-ranked papers should usually get higher strength.
But also look at the actual reviews so that you can calibrate the area chairs against one another. Some area chairs are more conservative with recommending talk slots, but the papers in those areas should not be penalized for that reason!
I gave initial strength ratings on this scale:
1 = area chair says to reject; no need for me to look further
2 = conceivable poster, but not sure if it's ready yet
2.5 = would be a rather postery poster
3 = try to accept: would be a good poster, but could be rejected
3.5 = try hard to accept
4 = definite accept: would be a good talk, but could be a poster
4.5 = try hard to make it a talk
5 = definite talk
Where you're not sure about strength, give both an upper and a lower bound (in separate spreadsheet columns). You can refine your estimate later if the decision on this paper is still unclear.
Any brief notes you have on a paper can go in another spreadsheet column, which you'll notice if you revisit that paper later.
The above scale assumes that the best papers should be talks, not posters. That is the usual practice. But there are occasional exceptions -- a clear accept might be better suited to a poster format because it has a very specialized audience or would greatly benefit from an interactive format. I recommend that you give such a submission a high strength score for now, so that it is likely to come out as a talk, but make a note to consider switching it to a poster format at the end of the process.
Sort submissions by upper bound on strength. Break ties for now by sorting by average recommendation.
Mark the top n submissions as talks, the next m as posters, and the rest as rejects.
Examine the decision boundaries: Look more carefully at the submissions that fall near the two cutoffs (including ties), as well as submissions whose decisions would have changed if you had sorted by lower bound on strength instead of upper bound.
I read the reviews and discussions for these borderline papers. Sometimes I looked at the submitted papers themselves to understand what the reviewers disagreed about. Occasionally I asked the area chair to help me better understand the merits or demerits of a paper, or to go back to the reviewers, or even to get an additional reviewer.
Examine anomalies: Sort the submission pool by various important subscores (impact, originality, recommendation, max recommendation). For example, nearly all of the high-originality submissions will be talks, and most of the low-originality submissions will be rejects. Exceptions to these patterns are suspicious. They should get a careful look to make sure that the decision is in fact justified by other factors.
Respect area chairs: Your task is mainly to establish cutoffs within each area chair's ranking. Now, as you do that work, you might find yourself doubting some of these within-area rankings. That's useful — but as a general rule, you shouldn't reverse an area chair's ranking without his or her permission. To check for reversals, sort the submission pool by area and secondarily by the area chair's ranking. This should neatly divide each area into talks, then posters, then rejects. Consult with the area chair about any violations of this principle, i.e., where the area chair ranked x above y but you are currently giving y a more favorable 3-way decision than x.
Consider balance among areas: It is fine for some areas to have a higher acceptance rate than others. But this shouldn't happen just because one area chair is more enthusiastic, or because you were in a good mood when assessing the absolute strength of papers in that area. So check whether the per-area acceptance rates are reasonable. And check yourself for bias: are the weakest accepts in your favorite areas really stronger than the strongest rejects in other areas? (Of course, there is no objective way to compare the worth of papers in different areas, but you can ask which paper will make its own audience happier.)
Return to step 3 (repeat until convergence).
Let the area chairs know of your provisional decisions, to give them a chance to object.
How to make the spreadsheet: You can get the data from START's Spreadsheet Maker tool. I pulled out the following columns, in this order. The starred columns were ones that I added myself.
*Notes
*Decision
Submission ID
*Strength (actually 2 columns for upper & lower bounds)
*Area chair ranking
Several columns for review subscores (averaged across all 3 reviewers):
Recommendation
Appropriateness
Clarity
Originality
Soundness
References
Depth
Impact
Impact of Resources
Audience
Title
Authors [keep this column offscreen or omit it altogether]
Keywords
Submitted elsewhere
To get this, I believe that I generated two START spreadsheets — Submission Information and Review Information — with the default settings, and then joined them on submission ID. I then deleted irrelevant material, added the columns marked with * above, rearranged columns into the above order, and manually set column widths and colors to make the spreadsheet easier to work with. You may want to ask a careful administrative assistant to help with this clerical work.
Here is an example of how I filled in the Area Ranking field for the area chaired by Suzanne. The string format here is chosen carefully, so that sorting the spreadsheet on this column will separate the papers by area, and will order the papers within each area by the area chair's ranking, as needed for 5. above. The end of the field shows Suzanne's absolute recommendation, in her own words. Note that Suzanne's ranking contained ties (e.g., her rankings #2 through #5 are in a 4-way tie):
Suz 01 (of 34) accept Suz 02-05 accept Suz 02-05 accept Suz 02-05 accept Suz 02-05 accept Suz 06 probable accept Suz 07 probable accept Suz 08 clear poster Suz 09 definite poster Suz 10 definite poster Suz 11 probable poster Suz 12-13 probable poster Suz 12-13 probable poster Suz 14-15 marginal poster Suz 14-15 marginal poster Suz reject Suz reject ... [19 recommended rejections]
areachair-13-discussion-ranking for when
this is appropriate, and
author-4-accept-talk-shepherded for how it is
handled with the author.
To avoid disclosing any reviewer's identity, it is usually the area chair whom you should appoint as the "shepherd" for the conditionally accepted paper. The "shepherd" must communicate with the authors, telling them what needs to be changed, and then must determine whether the camera-ready version has met these requirements. Typically, the shepherd delegates most of this work to the reviewer(s) whose concerns need to be met.
In the CFP: "Authors are encouraged to study the actual review form for guidance." Acceptance/rejection messages also linked to the review form, to help authors interpret their scores.
In the CFP: "To encourage thorough citation of related work, the References section does not count against the 8-page submission limit for EMNLP-CoNLL 2007. Thus, your PDF file may exceed 8 pages. However, all material other than the bibliography must fall within the first 8 pages!"
In instructions to accepted authors: "The body of your camera-ready version may use up to 9 pages if this is needed to address reviewer comments, clarify or illustrate your exposition, or include new results. To encourage thorough citation of related work, the References section does not count against this 9-page limit. (Other things equal, please avoid unnecessary length out of kindness to readers and trees.)"
These policies do make the printed proceedings thicker, but not necessarily more expensive. In any case, we made the printed proceedings optional. Most people chose not to purchase them, preferring the CD-ROM and the free ACL Anthology.
Nearly 1/3 of the EMNLP-CoNLL'07 submissions were later versions of papers that had been submitted to ACL'07 two months earlier — in effect, "revise and resubmit" cases. A journal would have sent the new version back to the 3 original reviewers, instructing them all to read the new version carefully. We didn't do that, but we typically reassigned 1 reviewer from the ACL'07 discussion as well as assigning 2 or 3 new reviewers. This policy ensured that the possibly valid concerns of the original 3 reviewers would be considered, not forgotten, when assessing the new and improved submission.
TO DO: There is much more to say about the pros and cons of this policy, and some analysis to be done of how it worked in practice. Ping me if you want to hear more right away. I can also share our script for detecting resubmissions.
In addition to the Best Paper Award, I wanted to institute an "Audience Choice" award for the most worthwhile presentation at the conference. Such a prize would have rewarded authors who not only produced outstanding research but also communicated it clearly and enjoyably at the conference meeting. I didn't follow through on this idea, but if you like it too, let's talk about how you could make it work!
Some conferences have an author response period, where the authors can reply briefly to the reviews before (or during) the discussion period. The EMNLP reviewing schedule unfortunately seemed too compressed to allow this — but other conferences do have enough time.
START does support an author response period, and I'd recommend trying it. The author responses seem useful for clearing up misunderstandings; for addressing reviewers' questions or confusions (indeed, reviewers can even pose explicit questions for the author response to answer); and for keeping reviewers honest (because they know that the author will flame a shoddy review).
This is indeed being tried for EMNLP 2008, with enthusiastic support by the SIGDAT board. It may set a precedent for the other CL conferences. (I have a lengthier justification of why it's a good idea, available on request.) I'll try to update this page and the email archive when I know more about how it worked out.
TO DO: I have some notes and memories and should write them up here. Send me additional material, too.
If these happen to you, feel free to ask me how I handled them ...
http://cs.jhu.edu/~jason/advice/how-to-chair-a-conference.html
| Jason Eisner - jason@cs.jhu.edu (suggestions welcome) | Last Mod $Date: 2008/08/07 15:23:38 $ |