Review Manager Guidelines

Header section

Title: Guidelines for Review Managers of Proposed TDWG Standards
Date version issued: 2021-11-22
Date created: 2009-05-15
Part of TDWG Standard: Not part of any standard
Abstract: The guidelines for review managers of proposed standards were originally written in 2009 by Lee Belbin and have been used by review managers of most of the standards that have been ratified since then. They were revised in 2021 based on feedback from former review managers in order to update the recommendations based on changes in technology and practical experience. They describe the best practices for carrying out the expert and public reviews required as part of the TDWG Process, the bylaws that govern how TDWG standards are ratified.
Contributors: Lee Belbin https://orcid.org/0000-0001-8900-6203; Gail Kampmeier https://orcid.org/0000-0002-5178-4170; Dag Endresen https://orcid.org/0000-0002-2352-5497; Steve Baskauf ORCID https://orcid.org/0000-0003-4365-3135
Citation: L. Belbin, G. Kampmeier, D. Endresen, and S. Baskauf. 2021. Guidelines for Review Managers of Proposed TDWG Standards. Biodiversity Information Standards (TDWG). https://www.tdwg.org/about/review-managers/

1 Responsibilities

The TDWG process for the ratification of its standards (https://www.tdwg.org/about/process/) is set forth in the TDWG Bylaws. Article 9 of the TDWG Constitution specifies how bylaws are created and modified. Ratification as a TDWG Standard should signify that a standard has a broad consensus of support in the community, rather than a simple or super majority of votes.

1.1 Author

The Author

Prepares the submission with their Task Group to conform to the TDWG Standards Documentation Specification (http://rs.tdwg.org/sds/doc/specification/).
Places the submission in the correct standards category
- Technical Specification,
- Applicability Statement, or
- Best Current Practice
Assures that the metadata information under the Summary is accurate and complete, with the understanding that this metadata will be used ‘as is’ to populate components of the TDWG web site when the submission is approved.
Presents the submission to the TDWG Executive Committee.

1.2 TDWG Executive Committee

The TDWG Executive Committee

Appoints a Review Manager
Approves submissions for Public Review
Ratifies a submission as a TDWG Standard

The Executive Committee should ensure that the appointed Review Manager is independent and free from conflicts of interest with the proposed standard and its authors. If conflicts are unavoidable, these conflicts must be stated to the Executive Committee in review manager’s acceptance of the role. The conflicts must also be described to the Author. The Executive Committee may also make recommendations for reviewers, but the final decision for selecting reviewers rests with the Review Manager.

1.3 Review Manager

The Review Manager

Manages the process of review.
Understands the process required for an effective review (Section 5.1).
Does not need to be a technical developer but does need to have a good overview of the proposed standard. A prospective user of the standard should make for an effective Review Manager.
Ensures that the submission conforms to the TDWG Standards Documentation Specification (http://rs.tdwg.org/sds/doc/specification/) and has been placed in the correct standards category.
Verifies the accuracy and completeness of the metadata included in the Summary.
Recruits expert Reviewers from the TDWG or related communities who are independent and free of conflicts of interest.
Communicates effectively with the TDWG Executive Committee, Reviewers and other interested parties.
Expedites the review process within the time requirements of the TDWG Bylaws.

1.4 Independent Reviewers

The Independent Reviewers

Are expected to have a high level of understanding of the domain and of the technical issues related to the proposed standard.
Should be free of conflicts of interest with the Author and the Author’s projects and organization, and must be capable of providing an unbiased assessment of the proposed standard. Reviewers must describe any potential conflicts to the review manager before accepting the role, who in turn must report them to the Executive Committee.
Report to the Review Manager in a timely manner.

In the interest of supporting open science and to recognize their contributions, reviewers will be identified by default when they submit their reviews. However, the option to remain anonymous will be available to the reviewers in order to promote candidness.

2 Independent Reviews

One of the first tasks for the Review Manager is to seek reviewers for the submission. Advice can be sought from the Author, relevant TDWG groups, the Executive Committee and others. It is recommended that at least three expert reviews are provided for submissions.

Because the document(s) submitted for consideration will be commented upon by reviewers, then returned to the authors for revision, it may be advisable to make them available to the reviewers in some form where changes and comments can easily be tracked. In order to document the submission, such mutable documents should be exported as PDFs and archived in GitHub where they can be easily viewed in their original form.

The Review Manager should keep a record of correspondence with potential and engaged expert reviewers that would be sufficient for a replacement Review Manager to take over if that became necessary. That record should include expert reviewer candidate names, date of solicitation, date of response, and the response itself (accept or decline). It should also include copies of key emails that contain information such as request to review, acceptance of the review, discussion of potential conflicts of interest.

2.1 Reviewer Profile

The ideal reviewer is someone who has not been directly involved in the development of the standard (independence). Reviewers must have advanced technical skills as a standard is a technical document. The ideal reviewer has written or contributed to the development of another standard. An ideal reviewer ‘gets the job done’.

While it is appropriate to seek reviewers from different communities/disciplines, this is less important than their technical abilities. Ideally, good reviewers will have experience in a range of areas/applications.

It would be hoped that at least two of the reviews would provide a comprehensive check of all of the technical details of the proposed standard. With a complex standard, this will take time.

An example of an email request to potential expert reviewers can be found in Section 5.2: Please use something like this as the email content.

The timeframe for individual reviewers to respond is up to the Review Manager. A period of around one month would be reasonable given that the review task may be substantial. Reviewers’ submissions should be timely. If a Reviewer is not able to complete the Review within a reasonable timeframe, another Reviewer may need to be engaged. Once a deadline has been passed, the Review Manager shall send a reminder and in parallel, consider seeking an additional reviewer. This way, there are minimal holdups.

2.2 Review Criteria

What criteria should you convey to reviewers to judge a submission? A TDWG Standard is composed of documents, but consideration should be given in the review of how effectively the proposed standard could be implemented.

The key issue for reviewers is that the proposed TDWG standard efficiently addresses the community need. Does the submission achieve the intended outcome in an efficient and effective manner?
Is the submission clearly written? If not, then uptake of the standard, however technically capable it is, will be diminished. Are the documents themselves structured in a way that will facilitate understanding by the broadest audience? Each standard document must clearly indicate what parts of it are normative. This may be done by designation at the start of the document (e.g. “all parts of the document excluding examples marked as non-normative”) or by indicating at the start of particular sections whether they are normative or not. See section 3.2.1 of the Standards Documentation Specification for details.
Does the submission conform to the TDWG Standards Documentation Specification?
Does the submission detail the intended audience and its needs? This should be addressed in a preamble and is required for a reviewer to evaluate how well the standard meets its goals. If the audience and desired outcomes are not identified, it is impossible to evaluate the effectiveness of the proposed standard.
Is the proposed standard technically accurate? For a complex standard, this is not an easy thing to evaluate. Some of the reviewers need to be technically capable of systematically examining the proposed standard to ensure that it is accurate. Typos, unexpanded first use of acronyms and undefined jargon in a technical document present a significant hazards to the end-users.
Does the submission build upon effective existing standards? Proposed standards should not ‘reinvent the wheel’. They should re-use well-accepted and effective standards wherever possible. Are other existing standards more appropriate for the intended purpose?
Is the overall structure of the Standard effective? If the structure of the proposed Standard is problematic, then use of the Standard may be equally problematic. Are there ways to simplify the structure? Does the structure allow for efficient/effective extensibility?
Does the standard address a large majority of end-user needs? Are the fields/parameters sufficient and clearly defined? Are the parameters/fields sufficient to address the large majority of end-user needs?

2.3 Post Expert Review

Once all expert reviews are in, it is up to the Review Manager to look at the reviews and decide into which category to place the submission.

Accept submission (as is), which would be rare,
Revisions required (which would be the norm),
Resubmit for a new review (rare) or
Decline submission (even rarer).

The reviews as they were submitted by the expert reviewers should be archived in GitHub as exported PDFs, with the Review manager taking care to remove identifiable information if a reviewer chose to be anonymous. The Review Manager may wish to combine all salient parts of reviews into a single document before presenting it to the Author and for the report to the Executive Committee.

Inform the Author what if any steps need to be taken before seeking approval from the Executive Committee to advance the submission to Public Review. Versions of submitted human-readable documents, revised in the light of reviewer comments, should be exported as PDFs and archived in GitHub. These revised documents can be made available for examination by the Executive Committee, although another format may be used for the transmittal if that format makes it easier to examine all aspects of the proposal.

As the Process dictates, the final decision for advancing the submission to Public Review is in the hands of the Executive Committee. An email should therefore then be sent to secretariat@tdwg.org for distribution to the Executive Committee. The email should explain how the revised proposal can be accessed, and request a decision concerning progress to Public Review. The Executive Committee will discuss the submission and provide a decision within 30 days.

3 Public Review

The Public Review is complementary to the individual expert reviews. Public comment must be open for at least 30 days (see Section 5.1). Anyone except the Review Manager can post comments to the Public Review. The reason for this is that it is important for the Review Manager to remain independent in the public eye. It is the Review Manager who makes the two key recommendations to the TDWG Executive Committee. The Author is free to post comments as a response to the other comments posted. The expert reviewers may also participate in the Public Review.

The submission should be made available for public viewing according to current TDWG practice, which might include publication in the organization’s journal, Biodiversity Information Science and Standards (BISS). However, the publicly-viewable submission should have the following features:

Its components should be easily viewed on a standard web browser from an open website.
It should include an introduction or landing page that briefly explains the history and status of the submission, now identified as a “Proposed Standard”. A link to the TDWG Process web page should be included for those who are unfamiliar with the standards review process.
It should enumerate the parts of the standard, which will include one or more human readable documents. If a vocabulary standard, one of the human readable documents will be a list of terms that includes the basic term metadata required by the Standards Documentation Specification. The documents may also include any normative machine-readable files, such as XML schemas.
The submission may link to informative documents that are not included in the standard but that are intended to help users better understand how to use it.
Instructions for submitting comments.

The intention is that an uninitiated member of the community can easily understand the purpose of the public review, know what is included in the submission, and understand how to contribute a review.

The Review Manager should prepare a request for comment, announcing the initiation of the public review. The announcement must include a description of how to submit a comment and the date range within which comments may be submitted (covering a minimum of 30 days). The announcement should be made in the name of the Review Manager and include an email address that commenters can use to contact the Review Manager. An example announcement can be found as Section 5.3 and links to prior announcements in Section 5.4.

When the announcement is complete, the Review Manager should post the text on the landing page of the Task Group’s repository. The Chair of the Outreach and Communications Functional Subcommittee will assist the Review Manager in publicizing the announcement. Typically, this will involve creating a news item on the TDWG website, which is accessible via a URL. The announcement itself should be sent out to email distribution lists commonly used by the community and the news item URL can be included as part of official TDWG social media posts. The Review Manager, Author, and other community members should be encouraged to forward the announcement or link to their networks in order to publicize the public review as widely as possible. As the comment period nears its end, it is advisable to follow up with a reminder via the same communication channels.

A publicly visible issue tracking system should be used in the Public Review. This will ensure that the Author and the community can view the comments and that a log of all comments is maintained for future reference. Comments are normally posted by and identified with the commenter. However, comments may also be emailed to the Review Manager in two circumstances: if the commenter is unable to use the issue tracker or if the commenter wishes to remain anonymous. Although the default is signed comments, anonymous comments may be accepted at the Review Manager’s discretion. The Review Manager should post emailed comments to the tracker on behalf of the submitter.

Public comment may result in all of the following:

Positive suggestions that will improve and enhance the utility of the Standard.
Suggestions that will add unnecessary complexity or diversions.
Recommendations that will advocate significant/fundamental changes to the submission.
Negative comments that shows bias or lack of understanding of the intent of the proposed Standard.

Note: If possible, constructive comments should be acknowledged by the Review Manager. The Author can start to examine comments, discuss issues with the Review Manager, and prepare responses or changes once the first comment is posted.

When the Public Review period is up, the Review Manager needs to inform both the secretariat@tdwg.org and the Author, and create a document that adds a recommended action against each comment, which could include (with any further explanation as necessary):

Accept comment and implement the suggestion,
Accept comment but adds no significant change to the proposal, or
Reject comment.

It is then up to the Author to respond to the Review Manager concerning all comments and associated recommendations. The Review Manager examines the Author’s responses and any updated submission. One of two actions are then selected for the submission:

A recommendation to the Executive Committee (via the Secretariat) that the standard be accepted or
A recommendation to the Author that further work on the standard is required.

A number of iterations may be required before a recommendation to accept the proposal is made to the Executive Committee.

It is reasonable for the review manager to expect a response from the Executive Committee within 30 days.

4 Appendices

TDWG ratification process

4.1 From the TDWG Process (By-laws) - Ratification of Standards

When a Task Group believes that a working draft is ready for wider review, the Task Group Convener submits the working draft and appropriate documentation as a package to the Executive Committee via the Secretary.
The Secretary notifies the Executive Committee of the submission and requests a decision on advancing the draft. This decision must be made within 30 days of receipt of the draft.
- If the response is positive, the Executive Committee appoints a Review Manager to seek appropriate independent and expert reviews.
- If the response is negative, the Secretary reports the decision with justification to the Task Group Convener.
The Review Manager submits the reviews along with a summary and recommendation as a package to the Executive Committee.
The Executive Committee may
- advance the draft as a Proposed Standard with call for Public Review, or
- decline (with justification) to advance the draft, or
- request a new revision to address specific problems.
Public Review. The Executive Committee directs that the Proposed Standard be placed in the TDWG repository, then publishes a request for comment (RFC) that describes facilities for, and duration of Public Review. A minimum of 30 days must be allowed for public comments. At the end of the comment period, the Task Group may modify the specification or its documentation as necessary. Any subsequent submission should address the comments received in the Public Review.
Accept as a Standard - A draft is accepted as a Standard and its status in the repository is updated accordingly. A specification must have been through at least one round of Public Review before the Executive Committee may accept it as a Standard.

The figure depicts the steps in reviewing and ratifying a TDWG standard. The figure is not formally part of the By-laws. See the TDWG Process web page for the entire By-laws text.

4.2 Example of an Email to Potential Expert Reviewers

Note: This template is similar to emails that have been used in the past to request Reviews. Please modify as required. The fields within braces in the text below are variables into which you may insert known details.

Dear {reviewerName}:

I’ve been appointed as the review manager for the proposed TDWG {standardName} standard.

I am seeking independent reviewers for that standard, which has been submitted for ratification. Your name has been recommended by members of the community as an ideal reviewer.

You are encouraged to identify yourself in your review so that your contribution can be recognized when the standard is published. However, reviewers have the option to remain anonymous if that would enable a more candid review.

Please feel free to contact me with any questions via email and we can arrange a voice or video call if desired. I am currently on time zone UTC +/-xxxx.

The submission’s abstract is inserted below, and I hope that you will consider undertaking this important task for the community.

Please reply to me by {$weekLaterDate} to indicate whether you will undertake the review or not.

The review itself is due {$reviewDueDate}.

“{standardTitle}”

Abstract

{standardAbstract}

Submission materials for review will be made available pending your acceptance to undertake this task. Note that the expert review is followed by a public review period. More information about the TDWG review process is available at {link-to-this-document}.

Please disclose if you have a conflict of interest with the content of the manuscript or the authors, which might affect your ability to provide an unbiased review.

{reviewManagerSignature}

4.3 Example of Public Review Announcement

{standardName} Public Review

TDWG announces the Public Review of the proposed {standardName} standard. It has been reviewed by expert reviewers and the TDWG Executive Committee, and we now seek comment from the community. The period of comment will run from {startDate} to {endDate}.

An introduction to the proposed standard with links to the standards document{s} can be accessed at {linkToLandingPage}.

The preferred method of commenting is to post a comment on the issue tracker at {issueTrackerLink}. Users who have problems using the tracker may email their comments directly to the Review Manager at {reviewManagerContactEmail}, who will post them on behalf of the commenter. Comments can be posted anonymously by request when justified and emailed to the Review Manager. All comments will be public and archived by the issue tracker system.

In accordance with the TDWG ‘Process’, a minimum of 30 days is required for public comment on the proposed standard. Comments should therefore be received by {deadlineDate}.

Please distribute this request for comment to anyone you think could or should comment on the {standardName} standard.

{It has been common practice at this point to include acknowledgements of the authors, expert reviewers, and the Executive for their work on the proposed standard.}

Sincerely, {reviewManagerName}, Review Manager