We provide both general guidelines and track-specific guidelines for reviewing SIGCSE Papers.
Here are some recommendations for writing reviews of submitted papers that help the authors and improve the quality of the symposium.
Papers submitted to the SIGCSE Technical Symposium are expected to be original work. If the work is not original or is not a sufficient advancement over prior work, please state so in the review or ‘confidential comments to the program committee’ section.
Your job as a reviewer is to write detailed reviews, even for excellent papers. In addition to telling the authors what you didn’t like about their paper, be sure to include what you did like.
Even if your opinion is that the paper is poorly written or poorly thought-out, you can still provide constructive criticisms to help the authors, and in the long run, the conference. Think of your goal as convincing the authors of the paper you’re reviewing to submit something else next year, but of such high quality that it will be well-reviewed and easily accepted. Give the authors advice on how to do that.
The best reviews clearly justify the reviewer’s choice of rating. The least valuable review gives a low score with no written comments. That simply tells the authors that they have been unsuccessful, with no indication of how or why.
The focus of your review should be on the content. Authors shouldn’t provide any links to supplemental materials. If they do, as a reviewer, you are not expected to review linked supplemental materials. When crafting your review, consider the most important information that is needed for the paper to stand as an independent work.
Papers that you review are supposed to be anonymous. Your review should be based on the merits of the paper, not the reputation of the authors or their institutions. Therefore, we have asked the authors to remove all identifiable references to themselves. We realize that reviewers sometimes know the work and can guess who the authors of the papers might be. If the work is not anonymized, please let the Program Chairs know and continue to provide a fair and unbiased review. If you are unable to do so, please notify the Program Chairs.
If you recognize the work, it is your responsibility to give a fair and unbiased review, using only the information in the paper. If you do not feel that you can give a fair, unbiased review of the paper and not the authors or institutions, please contact the program chairs immediately. Do NOT submit a review for the paper. The Program Chairs can reassign as necessary.
Your review should not include comments to the authors about the anonymization (or lack thereof) in the paper. If you feel that it is necessary to comment on this, please use the text box, ‘Confidential comments to the committee.’
We realize that occasionally anonymization requires the authors to remove information that affects your review (information that otherwise the paper appears to lack). As a reviewer, you can give the authors the benefit of the doubt. Use the ‘Confidential comments to the committee’ box to indicate this to us. Example “This paper should reference Matt Jadud’s work, unless those references were removed for anonymity”).
Please point out typographic and grammatical errors; if there are too many to list, please state so in your review.
Although SIGCSE requires all submitted papers to be polished work, all accepted authors get a brief opportunity to improve the presentation of their paper before camera-ready copy is due. Your detailed feedback may help improve a paper, and in a small way, improve the conference.
Do not put your recommendation for Accept or Reject in the author visible comments. The comments to authors should focus on the strengths and weaknesses of the paper and provide feedback to the authors for improvement, either for a camera-ready or for a future submission. These comments should correspond to the numeric score that you give the paper, which the other reviewers, APC, and Program Chairs will all see. Remember, you are only looking at a small fraction of the submissions; the Program Chairs consider the entire pool of submissions for a balanced and representative program.
SIGCSE uses a meta-review process after reviews have been submitted. Reviews that do not objectively, accurately, and clearly assess a paper’s suitability for publication at SIGCSE, founded in the reviewer’s disciplinary expertise and on the basis of the written paper’s originality, technical soundness, contribution to CS education, and clarity of presentation, may be deleted.
For example, an unacceptable review might:
be incoherent, unreadable, or irrelevant to the paper;
focus on the paper’s topic area or presumed authors at the expense of assessing the paper itself; or
provide no justification for its numeric ratings. (Even in “obvious” cases, reviewers should briefly justify ratings.)
Please note that a difference in rating or opinion with other reviewers or PC members will NEVER be cause for deletion of a review.
Reviewers with substandard reviews will be subject to the Recalcitrant Reviewers policy.
To help reviewers better understand the qualities of good, useful reviews, here are several example comments, organized by review category:
Please summarize the submission in 2-4 sentences in your own words. Please DO NOT copy/paste the abstract into this section.
This paper makes a very good argument in the introduction for why this course is needed. It is timely, and addresses a topic outside of the norm often seen at the SIGCSE Technical Symposium.
I can’t recall ever seeing something similar at the SIGCSE Technical Symposium.
This paper should generate a lot of discussion and have a good audience. It is a topic that many schools are trying to address (including mine.)
Good level of detail on your approach. Table 2 is very handy. Under Section 2, it seems like log analysis and auditing may fit in your column two. How will you ensure additional security emphasis is implemented?
The organization is faultless. It is very clear what the paper is going to say and how. The paper follows through with crystal clear subject headings and a logical flow of information.
The hypotheses are too obvious and the validation of them is not enough. Therefore, the contribution of this paper is quite limited.
Hard to judge given the writing organization problems, but I do not see a lot of significance here. The verification that the laboratory helped more than on-line component alone is a nice result, if it is supported by the data. Having taught this course already and collected feedback on your approach makes the paper stronger.
It is important for those who might be considering this approach to know that it can be successful. If I were considering this approach I would want to know if the students could understand the code, and how deeply I could get into the material given time constraints.
I would have liked to see some discussion and references setting this work in the context of other studies of student learning and knowledge retention. While I don’t know of other studies that have examined exactly the phenomenon this paper does, a short search in the ACM digital library turned up these examples that are relevant…
The paper could use additional proofing and polishing. I suggest finding a non-robotics person to read for both language and communication. Some sentences are poorly formed (e.g., sent. 1 of last par. in sec. 1). Some content seems misplaced (e.g., discussion of mobility in section 3).
There are three tracks for papers. Reviewers and APCs will be assigned to review papers in one of the three tracks. Please ensure that the Program Chairs know your preferences for the track(s) where you can provide the most expertise and best feedback.
Authors must choose the track that they feel best fits their submission. Review the submission using the guidelines for the track the submission is in; not the track you would prefer it to be in. The Program Chairs will not move papers between tracks.
Papers submitted to the CS Education Research track describe an empirical computing education project.
CS Education Research papers should adhere to rigorous standards, describing hypotheses, methods, and results as is typical for research studies. These normally focus on topics relevant to computing education with emphasis on educational goals and knowledge units/topics relevant to computing education with statistical rigor; methods or techniques in computing education; evaluation of pedagogical approaches; and studies of the many different populations that are engaged in computing education, including (but not limited to) students, instructors, and issues of gender, diversity, and underrepresentation. We welcome replication papers and papers that present null or negative results that meet the criteria below.
For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):
There are many resources for writing high quality papers for submission to the SIGCSE Technical Symposium. We encourage reviewers to read and evaluate papers from prior SIGCSE Technical Symposium, especially those designated as best papers, which were selected both due to content and high quality reporting. We have linked in additional resources that you may find useful as you review CS Education Research papers.
Experience Reports and Tools papers should carefully describe a computer science education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. This track accepts experience reports, teaching techniques, and pedagogical tools. All papers in this track should provide enough detail so that others could adopt the new innovation.
For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):
Curricula initiatives should describe new curricula, programs, and degrees, the motivating context before the new initiative was undertaken, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. This track may also include position papers, which are meant to engender fruitful academic discussion by presenting a defensible opinion about a CS education topic, substantiated with evidence.
For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):