ACM SIGMOD/PODS Conference: Vancouver, 2008
SIGMOD: Guidelines for Research Papers
EXPERIMENTAL REPEATABILITY REQUIREMENTS
To help published papers achieve an impact and stand as reliable reference-able works for future research, the SIGMOD 2008 reviewing process includes an assessment of the extent to which the presented experiments are repeatable by someone with access to all the required hardware, software, and test data. Thus, we attempt to establish that the code developed by the authors exists, runs correctly on well-defined inputs, and performs in a manner compatible with that presented in the paper.
Please refer to the FAQ about the Experimental Repeatability Requirements for answers to the most common questions/concerns about these requirements.
Papers that are accepted and are verified this way will be eligible to include the following text in the proceedings:
"The results in this paper were verified by the SIGMOD repeatability committee"
.If a subset of the results cannot be verified by the SIGMOD repeatability committee (e.g., for IP reasons), then the phrasing will change to:
"The results in Figures <list of figures> and Tables <list of tables> in this paper were verified by the SIGMOD repeatability committee."
If verified code and data is also made available for archiving, the following phrase may be added:
"And the code and data are available at <a site to be determined>."
If we test your code for repeatability, we will give feedback about any problems we encounter. This will not influence whether your paper is selected or not for publication.
Submission of code and data is optional for SIGMOD 2008. that is, submitting code and data will have no influence on whether your paper is accepted for SIGMOD 2008.
If you choose not to submit, however, then you are still required to submit, on December 16, 2007, a note stating: (1) the reasons you feel you could not submit; (2) how much time, if at all feasible, it would take you to satisfy these requirements; and (3) any suggestions you have about how to achieve the goal of scientific repeatability in our field.
If you choose to submit code and data for at least some of your reported experiments, please follow the requirements below.
Authors are required to upload to a special website (to be announced via email to the primary contact author on November 20th), at the latest on Dec. 16, 2007 (one month after the SIGMOD paper submission deadline):
-
The code needed to run the experiments quoted in the paper. Executable code (strongly preferred) or source code are both acceptable.
If source code is provided, any other software (libraries etc.) required in order to run the software should also be included. For third-party software that can be downloaded for free, only download URIs are required.
Authors submitting executable code are strongly encouraged to produce an image with no or very few dynamically bound libraries.
It should be possible to deploy and run the provided code on one of the platforms listed in this table.
If you do not have access to any of the platforms listed above, and you do not want to submit source code, you may consider using a virtualization platform. In order for us to test code developed on the operating system S1 via a virtualization platform, you need to provide, besides the executable code:- A download URI for the virtual machine player which we should use. The player must be available for a physical platform S2 among the ones listed in the table above.
- A download URI for the virtual image of S1 that we should run in the virtual machine player. You may also bundle your code with the image of S1 and submit them together. We strongly prefer receiving the code and an URI for S1.
If your software requires specific third-party libraries or tools, please indicate them to us as soon as possible (email Dennis Shasha shasha@cs.nyu.edu, and specify your paper id in the email), so that we may attempt to install them.
Software used to obtain figures out of raw numerical results should also be described and, if it is developed by the authors, provided.
If intellectual property or privacy issues prevent sharing the code, a description and explanation of these issues should be provided.
If some code is proprietary but other code is not, then the non-proprietary code should be provided. -
The data sets used in the experiments. If the data sets come from standard data sources unaffiliated with the authors, only a download URI is required.
If the data sets are produced by some generator, the generator itself together with the configuration used by the authors are preferable to the actual data. If intellectual property or privacy issues prevent sharing the data sets, a reduced sample or similar synthetic data set may be provided. If some data is proprietary but other data is not, then the non-proprietary data should be provided. -
A plain-text file named INSTALL, describing:
- the hardware and software platform required in order to run the experiment
- the sequence of installation steps
-
A plain-text file named HOWTO, describing for each table or figure in the paper for which code and data were supplied:
- the formats of the expected input and the interpretation of the produced output;
- which data sets to use;
- how to run the experiment. We encourage authors to provide simple, command-line-style interfaces for their code.
Confidentiality
The SIGMOD refereeing process will respect the same standard of confidentiality with respect to disclosed data and/or code, as it traditionally does with the papers themselves.
Repeatability Expectations
The experimental evaluation committee has reasonable expectations regarding the similarity to be observed between the experiment results as observed by the authors, and as observed by the committee. For instance, there is no way to ensure that the hardware used by the authors is available to the experimental evaluation team. Therefore, we do not expect measured execution times to match those reported in the paper, but rather roughly similar curve tendencies (unless the hardware used by the authors and unavailable to us enables specific optimizations, such as e.g. parallel processing on a multi-processor machine). When measuring other things than running time, such as e.g. result sizes in experiments with no randomized component, we do expect to obtain the results presented in the paper.
Repeatability Assessment vs. Double-blind Review
Authors are encouraged to anonymize their source or executable code to the extent to which this is feasible. The experimental evaluation committee does not need to know authors' identity in order to repeat their experiments.
When claiming IP or privacy issues, however, authors have to at least partially disclose their identity, for instance to name the company they work for. Such information may be disclosed to the SIGMOD chairs but it will not be disclosed to the SIGMOD PC. Thus, authors' identity remains protected from the regular SIGMOD reviewing process.
In the view of the SIGMOD officers, submitting code for testing does not constitute public release.
Archiving
To enhance the possibility that published papers will be referenced in the future, we encourage researchers to contribute the content provided for the repeatability requirement of an accepted SIGMOD paper, to an archive hosted within the ACM Digital Library.
The authors of an accepted paper will have the choice of putting their data and code in the archive or not. No data or code submitted for the repeatability requirement will be archived without the authors' consent.
CONFLICTS OF INTEREST
The submission site will also request information about conflicts of interest with program committee members. We will follow the rules for conflicts of interest established in previous years. A paper author has a conflict of interest with a Program Committee (PC) member when and only when one or more of the following conditions hold:
- The PC member has been a collaborator and/or co-author within the past two years.
- The PC member has been a co-worker in the same department or lab within the past two years.
- The PC member is or was the author's primary advisor, no matter how long ago.
- The author is or was the PC member's primary advisor, no matter how long ago.
- The PC member is a relative or close personal friend.
It is the responsibility of all authors to identify all and only their potential conflict-of-interest reviewers among the program committee members, according to this definition. It is imperative that authors allocate sufficient time in their submission process to identify and enter their conflict-of-interest reviewers. The Program Committee members are listed in the SIGMOD Organization. If you believe there are other conditions causing a PC member to have a conflict of interest with your submitted paper, please do not mark the conflict of interest during submission, but do contact the Program Committee Chair.
To determine if an author has a conflict of interest (CoI) with a member of the Experimental Repeatability Committee, the rules are the same as for CoIs with members of the research program committee. CoIs with members of the repeatability committee should be signalled by e-mail to the general chair Dennis Shasha (shasha@cs.nyu.edu) prior to submitting code on the FTP site.
Papers with incorrect or incomplete conflict of interest information as of the submission closing time will be rejected without review.
LENGTH OF RESEARCH PAPERS
All submitted research papers must be prepared according to the formatting instructions, and must be no more than twelve US letter pages. This page limit includes all parts of the paper: title, abstract, body, bibliography, and appendices. It is the author's responsibility to make the submitted paper readable, relevant, and interesting, before submission and consideration by referees. This requirement includes legibility of diagrams and quality of English. Other submissions such as tutorial and demo proposals have different page limits.
Papers violating the length, file type, or formatting requirements will be rejected without review.
DOUBLE SUBMISSIONS
A paper submitted to SIGMOD cannot be under review for any other conference or journal during the time it is being considered for SIGMOD. Furthermore, after you submit to SIGMOD 2008, you must await our response before submitting elsewhere (should your work not be accepted by SIGMOD 2008). If you submit your paper to another conference or journal either before/after submission of the paper to SIGMOD 2008, we will reject your paper without review and will notify the other conference/journal as well. This restriction applies to identical as well as to substantially similar papers.
NOVELTY REQUIREMENTS
Every SIGMOD submission is expected to have significant new technical content. This requirement is significantly more stringent for conference papers compared to journals. The submitted paper should neither be a minor modification of any prior work nor should it be material such as proofs or performance figures that may not offer substantial, new insights over such prior work. A paper is deemed to be "prior work" if it has been published in one of the following: (a) a refereed conference (including poster papers that are longer than 5 pages), or (b) a refereed journal. In addition, the paper must be novel with respect to any paper that is currently in submission or has been accepted in (a) or (b) of which any of the authors is aware (e.g., his or her own papers). Restriction on double submission is an instance of this requirement. It is understood that notational set-up, basic motivation, general examples, and references to past results may be common across papers by the same authors. However, such repeated material should not comprise a large portion of a conference paper reporting on new research. As a guideline, if identical or similar material exceeding 20% of the submitted paper has appeared in a prior work, then you should critically examine if indeed the paper has met the novelty requirement.
DISCLOSURE REQUIREMENTS
Every SIGMOD submission is required to have citations to relevant literature in "prior work", as defined above. In addition, authors are required to cite any papers by any author of the SIGMOD submission that is closely related or overlaps significantly with the current submission and: (a) is in submission (including in this conference), (b) has been accepted for publication, or (c) has been published at the time of submission anywhere else. As a guideline, an overlap is significant when it exceeds a page of the submission or when the overlap concerns content material in the SIGMOD submission, regardless of length. Since SIGMOD 2008 does double-blind reviewing, you must strictly follow the guidelines given below to ensure anonymity
The penalty for failure to comply with the disclosure requirement or obvious violation of novelty requirement is grounds for rejection without any detailed reviews.
ANONYMITY REQUIREMENTS FOR DOUBLE-BLIND REVIEWING (RESEARCH PAPERS ONLY)
All research papers submitted to SIGMOD 2008 will undergo a "double-blind" reviewing process: the program committee members and referees who review the paper will not know the identity of the authors. The double-blind reviewing is applicable for only research papers and is not applicable to other submissions. To ensure anonymity of authorship, authors must prepare their manuscript as follows:
- Authors' names and affiliations must not appear on the title page or elsewhere in the paper.
- Funding sources(s) must not be acknowledged on the title page or elsewhere in the paper.
- Omit all personal acknowledgments. Research group members or other colleagues or collaborators must not be acknowledged anywhere in the paper. There should also be no acknowledgment section in the paper.
- The submitted file should be named with the assigned submission number. For example, if assigned paper number is 352, then name your submitted file 352.pdf. Source file naming must also be done with care. For example, if your name is Jane Smith and you submit a PDF file generated from a .dvi file called Jane-Smith.dvi, one can infer your authorship by looking into the PDF file.
- Despite the anonymity requirements, you should still include relevant prior published work of your own in the references - omitting them could potentially reveal your identity by negation. We define "prior published" work as any research paper that has been published and made available in the online or printed proceedings of a referred conference or a journal prior to the SIGMOD 2008 submission date. Such "prior published" work includes longer poster papers (4 pages or more). However, you should not cite a published demonstration paper of your own or any one page poster papers of your own even if it is in "prior published" work as defined above. You must use care in referring to your prior published work. For example, if you are Jane Smith, the following text gives away the authorship of the submitted paper:
In our previous work [1], we presented two algorithms for ... In this paper, we build on that work by ...
[1] Jane Smith, "A Simple Algorithm for ...," Proceedings of ACM SIGMOD 1997, pp. 1 - 10.
[2] Jane Smith, "A More Complicated Algorithm for...," Proceedings of ACM SIGMOD 1998, pp. 34 - 44
The solution is to reference your prior published work in the third person (just as you would any other piece of work that is related to the submitted paper). This allows you to set the context for the submitted paper, while at the same time preserving anonymity:
In previous work [1,2], algorithms were presented for ... In this paper, we build on that work by ...
Bibliography
[1] Jane Smith, "A Simple Algorithm for ...," Proceedings of ACM SIGMOD 1997, pp. 1 - 10.
[2] Jane Smith, "A More Complicated Algorithm for...," Proceedings of ACM SIGMOD 1998, pp. 34 - 44
Referring to any work (including but not limited to your own work) that has been submitted elsewhere for review (and hence is as yet unpublished), or that has been accepted for publication at a referred conference, referred workshop, or a journal for which proceedings (printed or online) have been made available on or after September 15, 2007 requires a different protocol to ensure consistency with double-blind reviewing. In the body of your SIGMOD submission, you may refer to such work in the third person as "The authors have also developed closely related techniques for query optimization [15], but...". In the above example, reference 15 would then read: [15] Details omitted due to double-blind reviewing. Note that you should mention neither the authors, nor the title or venue of publication, while describing "anonymous citations" like [15]. Be sure to place all anonymous citations after the list of your regular citations. Of course, this does not mean you are not responsible to provide the details of such anonymous citations. During the paper upload phase, the submission tool will ask you to disclose (for Program Committee Chair only) full details of each such anonymous citation. Furthermore, if requested by the Program Committee Chair, you may be required to submit the anonymous versions of these papers corresponding to such citations at short notice (7-10 days). The program committee chair will contact you during the review period if this becomes necessary.
Technical reports (or URLs for downloadable versions) of your own work should not be referenced. Self-references should also be limited to only papers that are very relevant and essential for the reviewing of the submitted paper. You should not publish the papers submitted to SIGMOD 2008 online (in your home page or elsewhere) before the acceptance notification date.
Common sense and careful writing can go a long way toward preserving anonymity without diminishing the quality or impact of a paper. The goal is to preserve anonymity while still allowing the reader to fully grasp the context (related past work, including your own) of the submitted paper. In past years this goal has been achieved successfully by hundreds of papers.
It is the responsibility of authors to do their very best to preserve anonymity. Papers that do not follow the guidelines here, or otherwise potentially reveal the identity of the authors, are subject to immediate rejection.
COPYRIGHT RELEASE
Authors of accepted papers must sign an ACM copyright release form. The online conference proceedings may be published as early as April 15, 2008, so any patent applications must be resolved before that time. By submitting a paper to SIGMOD 2008, authors implicitly agree to public release of the paper as of the above date.
REVIEWING PROCESS
SIGMOD 20'08 adopts the two-phase review process established in previous years. In the first phase, each paper is assigned to two reviewers, and if a paper has at least one weak acceptance, the paper will be assigned to additional one or more reviewers in the second phase. The authors will also be selectively asked to provide feedbacks for papers that clarifications are deemed helpful for decision making by the reviewers or the PC chair in the first two weeks of February 2008.
Questions about any of the requirements or reviewing process listed on this page should be directed to the SIGMOD 2008 Program Committee Chair Dennis Shasha (shasha@cs.nyu.edu).
