TSC Policies for Replicated Computational Results (RCR)
Authors of articles nearly-accepted in TSC may now be invited to apply for an RCR certificate attached to their article. The RCR initiative aims to improve the reproducibility of computational results in the community and adds to the trustability of the computational results in the articles published in TSC. For those that are invited and accept, a further reviewer will be appointed to ensure that computational results can be replicated.
RCR Review Steps
The RCR process includes the following steps:
- RCR suitability review: When a manuscript reaches the minor-revisions stage, the Editor-in-Chief (EiC) and AE will briefly review the manuscript to determine if it is suitable for an RCR review. Authors may also request RCR review when they submit a paper, though the process may not start until after an initial round of reviews.
- RCR reviewer assignment: After determination of RCR suitability and with the agreement of the authors, the AE will assign an RCR reviewer whose sole responsibility is to replicate manuscript computational results. Unlike the other reviewers, the RCR reviewer will be known to the authors and work together with the authors during the RCR process.
- RCR review process: Replicating computational results will require a multi-faceted approach. TSC editors will advise the RCR reviewer on acceptable approaches, but ultimately the RCR reviewer has the responsibility to declare whether or not computational results in the manuscript are replicated. RCR reviewers will document the details of how results were replicated.
- RCR determination: The Reproducibility Referee may advise the authors on what additional information to add in the article in order to let readers reproduce the results faithfully. Reacting to this advice works in the same way as with standard referees: the parties iterate until the referee is satisfied. Ultimately, this referee will declare whether or not the computational results in the manuscript are reproducible.
- RCR review failure: There is some risk that RCR efforts will fail. In this case, we must acknowledge that the manuscript is not ready for publication with the presented results. During the introductory phase, the EiC will personally manage this situation if it occurs and will work with the authors to avoid rejecting the manuscript outright. As the RCR initiative matures, we anticipate that failed RCR reviews would constitute grounds for returning the manuscript back to the authors for revision, or for rejection if concerns were serious.
- Publication: A manuscript whose computational results are successfully replicated will be published with a special RCR designation as a text-only note on the bottom of the first page. A special graphic designation may become possible in the future. The RCR referee will be acknowledged in the published paper as author of the RCR review report that will appear with the published manuscript. The RCR referee's report will be published as a TSC article, immediately following the RCR-reviewed article. This report will also go through a light review process to assure that it is well written and contains required report elements.
Methods for Replicating Results
For all RCR reviews, we rely on the expertise of the RCR reviewer to make the final determination of the RCR designation. Presently we have two basic approaches for assessing replicability. The first is more desirable, but not always possible.
- Independent replication: The authors provide the RCR reviewer access to, or sufficient description of, the computational platform and/or data used to produce the manuscript results. Access could be:
- A direct transfer of software and/or data to the reviewer (or a pointer to an archive of the software and/or data), detailed instructions (preferably a script) for performing any analyses, and specification of a commonly-available computer system on which the software or data analysis can be run.
- A guest account and access to the software and/or data on the system used to produce the results.
- Detailed observation of the authors replicating the results.
- Review of computational results artifacts: In some situations, authors may not be able to readily replicate computational results. Results may be from a system that is no longer available, or may be on a computing system to which access is very limited. In these situations, careful documentation of the process used to produce results could be sufficient for an RCR designation. In this case, the results should have their own substantial verification process to give the reviewer confidence that they were performed correctly.