Here at Scientific Data, we are busy developing guidelines for our future peer reviewers. We believe strongly in the role that peer evaluation plays in the scientific process, and hope that Scientific Data will set a new standard for critical and constructive peer evaluation at a data-focused publication. Authors will be expected to convince referees that the data is worthy of wider use in the scientific community by supporting the technical rigor of the data collection procedures, the completeness of the data and its description, and alignment with existing community standards. Referees will be asked to consider these points in detail, and will be free to request additional experiments when needed to support the data.
At the same time, referees will be asked not to base their evaluation on the perceived impact or novelty of the findings associated with the datasets, and authors will be asked to remove in-depth analyses or new scientific conclusions from submitted Data Descriptors. The peer review process will remain focused on data quality and reusability, not specific interpretations.
This may be an unfamiliar perspective for some referees, so to encourage consistency in the peer-review process, we have developed a short set of questions that referees will be required to answer before they fill out their reports. These questions are designed to help educate referees about Scientific Data’s editorial standards and to encourage them to consider issues — like data completeness and reusability — which may not be emphasized at other research journals. Referees would be asked to answer each question as “yes,” “no,” or “not applicable,” and then to comment on each point in their reports.
Scientific Data referee questionnaire
Experimental Rigor and Technical Data Quality
1. Were the data produced in a rigorous and methodologically sound manner?
2. Was the technical quality of the data supported convincingly with technical validation experiments and statistical analyses of data quality or error, as needed?
3. Are the depth, coverage, size, and/or completeness of these data sufficient for the types of applications or research questions outlined by the authors?
Completeness of the Description
4. Are the methods and any data processing steps described in sufficient detail to allow others to reproduce these steps?
5. Did the authors provide all of the information needed for others to reuse this dataset, or integrate it with other data?
6. Is this Data Descriptor, in combination with any repository metadata, consistent with relevant minimum information or reporting standards?
Integrity of the Data Files and Repository Record
7. Have you confirmed that the data files deposited by the authors are complete and match the descriptions in the Data Descriptor?
8. Have these data files been deposited in the most appropriate available data repository?
This referee questionnaire will be supplemented by a “Guide for Referees” on our website, explaining Scientific Data‘s scope, criteria for publication, and peer review process in more detail. This guide will be released before our call for submissions, which is currently scheduled for September or October of this year.