DocumentReview

From W3C Wiki

Getting early and wide review of a document is very important yet in practice, it can be challenging. Rather than defining explicit mandatory steps for getting wide and constructive review, this document provides some considerations, guidelines and best practices that groups should follow.

Feedback on this document is welcome, preferably by directly editing this document, by sending email to public-openw3c@w3.org (archive), or by tweeting to the @openw3c twitter account.

SeeAlso:


Introduction

Getting early review and feedback is important, especially reviews from those groups that have identified a document as a dependency, and the so-called horizontal groups. It is also important to get review from other stakeholders including Web developers, technology providers and implementers not active in the Working Group, external groups and Standards organisations working on related areas, etc.

In Process-2019, the focus is on actually getting evidence that wide review has occurred. The considerations, guidelines and Best Practices (BP) in this document should provide reasonable guidance for document review.

TL;DR:

When you have published a First Public Working Draft, you should work through available "self-review" materials, and request review of your results, in at least the following areas. "Long enough*" before you request a transition to CR, you should do the same again, identifying substantive specification changes since the first review.

The meaning of "Long enough*" depends on how many changes there are, and how clearly you have explained them. Pointing to 14 concise points for a small spec means a short time, pointing to 900 diffs from commits and hoping people understand them in a 300 page spec means it will take a long time to get review. If you have effectively identified issues for review during development, and got feedback on them, the review time will probably be shortened. It also depends how busy the groups are, so planning in advance is useful.

Accessibility
Work through this questionnaire then ask APA for review: public-apa@w3.org
Architecture
Ask the TAG for review; see how to work with the TAG. If you are developing javascript APIs you may also want to ask public-script-coord@w3.org, a technical discussion list shared by W3C and ECMA's TC 39.
Internationalisation
Work through the [Checklist], then request i18n review by email to www-international@w3.org (A public list watched by the Working Group). If for some reason you want a member-confidential review, you can ask on member-i18n-core@w3.org (a member-only list watched by the Working Group).
Privacy
Go through this checklist and the privacy/security checklist, then ask the Privacy Interest Group for a review: [mailto:public-privacy@w3.org public-privacy@w3.org
Security
Use your results from the privacy/security questionnaire (see privacy), then ask the Security IG for review: public-web-security@w3.org
Groups listed in your charter
If there are groups listed in your charter as depending on some specification, you should ask them too.

Read on…

You should familiarise yourself with the rest of this document. This section is just a quick reminder for when you are in the middle of doing the work.

Considerations and Best Practices

Among the considerations and best practices, regardless of the type of review (early, thorough wide review, etc.) are:

Who to ask for review?

  • Which group(s) should be asked to review a document? All group charters should include information about the groups and external liaisons that are interested in particular documents. At a minimum, those groups should be included in all review request for their related document(s). Additionally, it may be appropriate to explicitly seek review from others such as: other Consortium groups; external organizations and SSOs; stakeholders such as implementers, testing and interop experts, application developers, etc. If a document is explicitly identified as a joint deliverable with another group, it is especially important to make sure that group is included in all review requests.
  • Which Horizontal Groups should be asked to request a review? This can be a bit tricky because in some cases the set of relevant of horizontal groups (enumerated below) should be relatively obvious but in other cases it might not be. For example, Working Group Charters often explicitly identify Horizontal Group dependencies, but post-charter development may introduce features that require review by Horizontal Groups not called out in the charter. - SNZ
    • BP: if in doubt, before formally asking a horizontal group to review a document, talk to them informally (f.ex. via e-mail).

(comment): I agree that it may not be obvious that a document needs horizontal review, but I tend to take a stronger line: even if you have not identified a horizontal group dependency, you should always discuss whether a review is necessary or even request a review (saying if you think there are no issues of interest to the given group). --Addison Phillips (talk)

  • What should be done if it appears a document might be relevant to a W3C liaison but the liaison is not explicitly listed in the group's charter? The Chair(s) and Team Contacts should consult with their group, and if there is consensus within the group to contact the liaison, then that external group should be included in all review requests.
    • BP: That group should be included in review requests unless they ask not to be --CMN

When should review be requested?

  • When is a group required to seek wide review of a Process-2019 document? For a document to enter CR, the group must show the document has had wide review. The requesting group is free to determine the timing for review requests.
  • When should reviews by Horizontal Groups be scheduled? Horizontal Groups are very often resource limited. This means that they can often only do one review and may have difficulty scheduling that review in a timely way. Secondly, to do a reasonable review, they need a document that is pretty stable, but still flexible enough to allow changes where issues are identified. -SNZ
    • BP: Contact the chair of each relevant Horizontal Group to arrange a suitable time in their schedule to review the document. - SNZ
  • What circumstances should trigger a RfC? When a new feature is added to a document after its FPWD is published or after the document has already been a target of a wide review, a group should seek wide review of that new feature.
    • Other reasons to consider wide review include: some other group expresses keen interest in a document.
  • What part of a document should be reviewed and when? The general rule of thumb is to get review early and often.
    • BP: Ensure that each section or feature is reviewed when it is considered stable (this enables changes to be made early, which lowers the risk that deployed legacy will impact the possibilities for fixing something).

Making reviews happen

  • Length of the Comment Period. The duration for comments (aka comment period) depends on a number of factors including the version of the Process Document being used. As such, consult the Process being used.
    • BP: before a group asks another group to review a document, the requesting group should work with the review group to determine a mutually agreeable review timeline.
  • How to initiate a review request. Review requests should be sent to the review groups' Public mail list(s).
    • BP: please avoid cross-posting by using Bcc: if necessary.
    • BP: use the RfC template below.
  • Announce LCWD and pre-CR publications on the chairs list. The group Chair or W3C staff should announce all LCWD and pre-CR publications on the chairs list.
  • Announce all Requests for Review (f.ex. LCWD and pre-CR publications) on the public-review-announce e-mail list. This announcement may be made by a group Chair, Consortium Staff, Editor or any other designated member of a group.
    • BP: use the RfC template below.
  • Use Public e-mail lists for review feedback. The requesting group's Public mail list should be used for review feedback.
    • BP: use the requesting group's main Public mail list for comments and do not use (nor create) a list whose sole purpose is for comments.
  • How can a group get acknowledgement that some person and/or group actually reviewed a document? Naturally, this isn't an issue if the reviewing group and/or individuals in the reviewing group actually provides review comments. However, in some cases, this can be somewhat difficult to determine and achieve (f.ex the reviewing group is small, the overlap in scope is minimal, etc.).
    • BP: when asking for review, explicitly ask the reviewing group to submit all feedback, even if the feedback is something like "I read it and have no comments".

Request for Comments (RfC) Template

See the tool to assist with calls for specification review.

General Information

A RfC should include the following general information:

  • Name of the group seeking comments.
  • Deadline for comments.
  • Subject: header: include the maturity level, name of the document and document maturity level of the document
    • BP: Subject: RfC: FPWD of Name of Spec; deadline MMM DD.
  • Mail list for comments; include the mailto: address plus a link to the Public mail list archive
    • BP: avoid cross-posting to multiple lists (and if this must be done, use Bcc:), except for the i18n WG, which needs to track issues on its technical list as well. (i18n WG members cannot subscribe to all the lists of the groups they review specs for.)
  • Alternate instructions for providing commentary (if needed). Some groups use bugzilla or github as a means of communication and for tracking issues. --Addison Phillips (talk) If your main or preferred tracking location is a database, make that clear in the SOD, since moving and tracking comments between mail threads and databases can waste time for both the reviewer and the reviewed group.
  • Reply-to: header: set the Reply-to: header to the mail list where all comments should be sent
  • State that all review be acknowledged, even if a reviewer or group has reviewed the document and has "no comments".

Information for Each Document

For each document that will be reviewed, provide at least:

  • Document title
  • URL
    • BP: the contents of the document should be stable (i.e. not changed) during the review period.
    • BP: if the contents of the document might change during the review, identify the specific section(s) that might change; or consider postponing the review until after those sections are stable.
  • Maturity level: FPWD, CR, ...
  • Scope of the review: this can be the entire document, specific sections, specific issues, etc.
    • BP: Identify the specific sections of the document that are newly stabilized, as well as those already considered stable. This allows reviewers who follow "at a distance" to focus on what's new, and allows new reviewers to focus on things that are ready for review, instead of providing a lot of comments on sections that are likely to shift.
    • BP: Specifications should include information about stability and known implementation on a "section by section" basis.
  • If a review was previously requested, include the URL of a diff, using the previous version of the document as the diff's reference/base.

Example Request for Comments

Here are some example RfCs:

Horizontal Groups

The consortium's so-called Horizontal Groups are groups that include review and coordination functions in their charter. Typically, these groups consist of individuals that can provide advice and guidance (to other groups) regarding their domain expertise.

Groups with a horizontal review function, including their mail list are (alphabetical order):

Review Resources

FAQ

  • Q: When is the best time to seek early review?
    • A: For a number factors (including document maturity, implementation status, etc.), there is no best time that applies to all documents. However, for most documents, after a First Public Working Draft is published is generally a good time to seek review of a specification.
  • Q: Is some type of security and/or privacy review mandatory before a First Public Working Draft is published?
    • A: Strictly speaking, the answer is No. However, getting early review for documents with features that might have security and/or privacy implications is strongly encouraged. See the Horizontal Groups section below for information about security and privacy groups that can provide review and related review resources.
  • Q: Does a group need to prove its documents have had wide review before proceeding to CR?
    • A: The answer is yes, since Process-2019.
  • Q: How does a group prove its documents have had wide review before proceeding to CR?
    • A: See How to work with (experiment with) Wide Review -SNZ
    • A: Keep a Disposition of Comments (DoC) Document that shows review comments and acknowledgements that have been received. It is the distribution of the reviewers that shows the review has been wide.-SNZ
    • A: Ask the W3C Team if your proposed approach is likely to meet the requirements. -SNZ
  • Q: Is there such a thing as too many reviews?
    • A: Not really
  • Q: Is it possible to make too many requests for review?
    • A: unfortunately, yes there is. Multiple requests for reviews can quickly result in a situation of diminishing returns. To help address this, subsequent review requests should clearly identify the exact differences between the last review and current review.
    • A: This is also the reason that the Process clearly suggests there should be (TR) Working Drafts published when there are "significant changes that would benefit from review beyond the Working Group", rather than every day or only twice in the life of a spec…
    • A: TR Working Drafts are also useful for reviews since they provide a dated snapshot which can be recovered when the review comments are being discussed. Trying to discuss review comments against a document which has changed out of all recognition can be a frustrating and inefficient experience.

Issues

  • Is pre-CR the best term to describe a Process-2014 snapshot that precedes CR or is there a better term?
    • Proposals: 2014-LC, 2014-Snapshot, pre-CR snapshot, 2014-Review Doc, pre-CR Review Doc, ...
    • Proposal: Don't do this in order to demonstrate "wide review". A "Last call" for a document that has already been reviewed in development is appropriate. Waiting until you think the document is finished to ask for review is a bad idea. This issue has become moot with the passing of time.
  • How can a group or individual be automatically notified when a document has been published at one of the transitions that could/should initiate the start of an early or wide review? For Process-2005 documents, the transitions of interest are FPWD, LCWD and CR and for Proc-Doc-2015 documents the transitions of interest are: FPWD, pre-CR, and CR. This Issue was resolved via the creation of the public-review-announce e-mail list.
  • For all processes Working Drafts should be relevant for review - and should identify which parts are new or newly stabilized or changed, to facilitate timely review.
    • Potential solution: when a document is published on /TR, announce its publication on a Public mail list. See a related Call for Consensus to create such a list, deadline for comments is October 15].

Terminology and Abbreviations

  • pre-CR - This is a version of a Working Draft that is created to get wide review.
    • note that this is a bad way to get review. In general, features should be reviewed as they are developed. Waiting for a "Last Call" for most review means that when reviews suggest changes it is far harder to make them, due to a commonly observed and logical reluctance to break deployed systems or content. Charles McCathie Nevile (talk) 11:18, 13 October 2014 (UTC)
  • requesting group - a group that is requesting a review

Abbreviations:

  • BP = Best Practices
  • CR = Candidate Recommendation
  • RfC = Request for Comments (aka Review Request)
  • TR = Technical Report, i.e. a formal W3C publication.
  • WD = Working Draft

Enhancement Requests

  • See the Document Review Dashboard document for information about creating a dashboard type service to facilitate document reviews.
Retrieved from "https://www.w3.org/wiki/index.php?title=DocumentReview&oldid=109135"