Re: [IETF57] coach BOF

[Sally Floyd <floyd@icir.org>]

I ended up volunteering to be one of the note-takers, so here
they are, FYI...

As Margaret presented it, COACH grew out of one of the problem
resolution proposals from the problem statement working group.

- Sally

-------------------------------------------------------


COACH: A Comprehensive Approach to Quality

Agenda:

Scott Bradner, Introduction:
Existing RFC Editorial guidelines,
(no viewgraphs)

Questions he was asked to address:
What is the aim of an IETF document?
Are IETF documents any good?
There is a general view outside the IETF that IETF documents
  aren't that good, based in part on the assumptions of other
  groups (IEEE, ITU) about standards documents.
What is the trend over time, in terms of document quality?
  Not increasing.
What are the problems of bad documents?
RFC 2360: Guide for Internet Standards Writers.

----------

Bernard Aboba:
A Comprehensive Approach to Quality (10 minutes) 
draft-loughney-coach-00.txt

Poor quality is the result of insufficient resources.
What is a WG Quality Plan: a public document, at most 5 pages.
Potential components of the WQ Quality Plan: 
  charter; 
  challenge assessment; 
    Are the goals achievable?  What are the risks?
  tools plan; 
    Mailing list; web site; document production; revision control;
    issue tracking and reporting
  review plan.
    Review of charter; work item review; midterm assessment; late review
    Cross-area review is critical.
The Post Mortem: at most three pages.

John Loughney: Many WGs do this already informally.

Dave Crocker: We don't always pay attention to the operations impact. 

Rodney Hass: Question about the slide on challenge assessment.
  The challenge assessment might be a very difficult document to write,
    because different WG members have different views.
  On poor writing: How does this address poor writing skills?

Eric Gottman:  I like where this is going.  One problem is that we lose
  the history of what we have done.

David Black:  A few cautionary notes.  The challenge assessment needs to
  be viewed as a living document.  Be careful about the challenge
  assessment being turned into a weapon to kill something.

Jonathan Rosenburg:  About poor writing, he is not sure that the plan
  will help.  It takes a lot of experience to write good protocol
  specifications.  He is not sure that each working group needs 
  completely new plans - different working groups will probably
  have similar plans.

----------

Margaret Wasserman:
IETF Problem Resolution Processes 
draft-ietf-problem-process-00.txt

Problem Resolution is being addressed in the Problem working group.
She is reporting on the problem resolution process recommendations.
Near-term activities:
  COACH (on working group processes); 
  Internal education; 
  Tools improvement;
  Inter-working-group communication
Long-term activities:
  The IMPROVE WG.
Her suggestion for improving process:
  Identify promising proposals; define metrics; measure performance;
    institute change; measure the results.
  This is classic process improvement methodology.

----------

Leslie Daigle:
The BoF Process: A Critique
Personal observations.
Pathological BoF cases:
  * BoF proponents are told no, come back when you have figured us out.
  * BoF proponents pester ADs until then get a BOF, and then a WG
The BoF can be about gaming to get approval to be a WG.
Proposal for a different approach (called the oven process):
  * First get agreement on a problem statement within a small group.
  * Then have a BoF.

Scott Bradner: 
  As Area Director, he used to have a similar approach, of requiring
    several things before the BoF.
  BoF chairs are not necessarily good WG chairs.

?: He assumes that the oven process was already in play.
  But there is a problem that people will feel locked out, so the
    BoF is useful to address that.

Aaron Falk: The process of baking in the oven is subtle.  Success depends
  a lot on the right people being added to that process.

----------

John Klensin:
(no viewgraphs)
IESG Overload and Quality of WGs 
draft-klensin-overload-00.txt

(This is a report on an old proposal, from a conversation with Mike O'Dell.)

We have an incentive problem:  
  Someone is very interested in getting something done.
  There is nothing fundamentally wrong with it, so we end up just doing it.
  Some of the WGs last forever and take a lot of time.
We start too many WGs, and we don't kill them off fast enough.
To change the incentive model:
  Put a ceiling on the number of working groups, to give an incentive
    to the IESG to be more strict, and make judgements.

James Polk: Is it about the number of WGs, or the number of efforts (e.g.,
  SIP and SIPPING)?

John: It is easier to limit WGs than limiting efforts.
  
Margaret Wasserman: I have a number of concerns with this proposal.
  Any time that you set up arbitrary rules, you invite people to game the
  system some other way.
  I think we want the IETF to grow and do more things.

John:  We are in complete agreement except for two things.
  The community tells the IESG that we are not interested in solving
  this problem, so the alternative is to change the incentives.

Dave Crocker:  You are saying that a community that is highly expert
  in resource contention problems doesn't know how to apply it to
  ourselves.  The really hard part is that the set of incentives
  that will work the right way, would be incentives that people
  would apply to themselves.  We need incentives to maintain quality
  and control quantity.

Jonathan Rosenburg: The definition of what is important is hard to nail
  down, because there are many communities for which different things
  are important.

----------

Margaret Wasserman:
Decision points/milestones in the WG process 
draft-wasserman-wg-process-00.txt

Purpose of the document:
* Provide terminology and a framework.
* Improve WG chairs training.
Steps in the WG process:
Initial submission; Author refinement; WG acceptance; Editor selection;
  WG refinement; WG last call;

Scott Bradner: If there is not much support to include a document
  in the working group, silence does not mean consent.

David Black: On the importance of design teams, and the proper
  management of design teams.  Closed design teams should open up
  after a document becomes a WG item.

?: This is interesting.  Is it consensus when four people are enthusiastic,
  and everyone else in the WG passes?

Margaret: We are in no danger of saying No too firmly or too often.

----------

Brian Carpenter:
Careful Additional Review of Documents (CARD) (10 minutes) - Brian Carpenter
  http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs-01.txt
Was: SIR: Senior IETF Reviewer
New: AIR: Agreed IETF Reviewer
This proposal is targetting many of the problems cited in the problems
  document.
Basic proposal: 
* Create a team of reviewers
* Have all drafts reviews early, mid-term, and late.
* Build trust in this process.
The standards process doesn't change.
Creating the team: create a set of 200 reviewers, for 1800 reviews/year.

Bernard: Do we have 200 people to do this?  He doesn't think so.

Increasing and maintaining the team.
Triggering reviews - this comes from the WG.
What's in a review?  These need to be public reviews.

James Polk: Of the 200 reviewers, how do you find the right reviewer?

Brian: We need to do more work on this.

Mechanics: web site, http://graybeards.net/sirs/
  web tools.

Ross: I am worried about the publicness of the comments.  Politics about
  company A/company B?

Margaret Wasserman:  There are a lot of costs in trying to do this.

Harald: There are two things I would like to see:  a few samples of reviews,
  listed on a web page.  

Allison: I think these reviews could be very useful, but the perfect can
  be the enemy of the good.  I don't think that *all* documents needs to
  be reviewed, *three* times.  It might be sufficient for some documents
  to be reviewed well, once.

Henning: This is not a new mechanism.  Scientific publications have been
  doing peer reviewing for ages.  Timeliness is an issue, because we
  don't want the reviewing to add delay.
  
----------

Aaron Falk and Allison Mankin:
The Review Process in Action: The DCCP WG 

IESG reviews sometimes surface late surprises.
Early Review Proposal from Leslie and Allison at an IAB/IESG retreat:
  Get broad inter-area review well before WG and IESG last call.

The DCCP Design Review:
DCCP, planning for WG Last Call this fall.
The expert written review was for depth.
The design review in the WG is for breadth.  Perhaps this one should have
  happened earlier.
Out of scope of the meeting: WG charter, problem statement.

Eric Rescorla: Two observations: being a designated reviewer makes a big
  difference, in terms of the review actually getting done.
  It took him two days to do his review.

Spencer Dawkins:  How much does it make a difference to have
  IESG collaboration in getting reviewers?

Brian Carpenter: What if DCCP was an individual submission?  If this
  level of review, then could the RFC editor review process be not
  needed?

Aaron:  The RFC Editor gives a very different kind of review.

: Do you have sense of the time it takes the RFC editor to do a review?

Aaron: Sorry, no.

----------

The next steps will be mailing list discussion.

General comments from the mike?

Henning:  These all sound like proposals that can be done well
  on an experimental basis.  We need to try experiments, and judge
  them successful or not.

Dave Crocker:  If we don't have enough reviewers, that can be a feature.
  If one can't get competent reviews, then the work shouldn't progress
  to the next stage.

Bernard: I would agree that if you can't come up with the resources
  to finish the work, then the work should not be started.

Dave C.:  We're having trouble putting quality and timeliness in the
  work that we do now, do lets do more?