[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[no subject]
Dear Bernard and John,
My notes from this afternoon's session (I sent them earlier, but they
haven't appeared on the mailing list - sorry for the delay)...
Spencer
Comprehensive apprOACH to quality BOF (coach)
Monday, July 14 at 0900-1130
=============================
Comprehensive apprOACH to quality BOF (COACH)
Monday, July 14 at 0900-1130
=============================
CHAIRS: Bernard Aboba <aboba@internaut.com>
John Loughney <john.loughney@nokia.com>
Mailing list: ietf-quality@bogus.com
READING LIST:
http://www.ietf.org/rfc/rfc2360.txt
http://www.ietf.org/internet-drafts/draft-loughney-coach-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-problem-issue-statement-01.tx
t
http://www.ietf.org/internet-drafts/draft-ietf-problem-process-00.txt
http://www.ietf.org/internet-drafts/draft-wasserman-wg-process-00.txt
http://www.ietf.org/internet-drafts/draft-klensin-overload-00.txt
http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs-01.txt
AGENDA:
Agenda Bashing
- No changes
Introduction
Existing RFC editorial guidelines - Scott Bradner
http://www.ietf.org/rfc/rfc2360.txt
- Focused on standards-track documents providing specifications that allow
interoperable implementations
- Some documents are better than others. Some are actually good. View
outside the IETF is that our documents are not good. Some of this is
because we’re not producing IEEE/ITU documents – clearer, but not as clear
as they should be. Why do these documents make it all the way to the IESG
before we punt on them?
- We’re still producing the same number of good documents, but we’re
producing more documents. Declining percentage!
- What’s the cost? Confused working groups, confused IESG, confused
implementers.
- Why? Producing good documents is not an easy skill. Language problems
with ESL writers – not muddled thoughts, but muddled language. Complicated
protocols lead to complicated documents. Often, the working group just
isn’t thinking clearly.
- This is not a new problem. Standards-guide standards writer guidance (BCP
22) dates from mid-1990s (and it’s quite good – read it! Including examples
of really good RFCs). You want a document that won’t confuse your mother
(whether she understands it or not, she isn’t confused).
Quality: Overview and Framework
A Comprehensive Approach to Quality - Bernard Aboba
http://www.ietf.org/internet-drafts/draft-loughney-coach-00.txt
- Focused on working group quality plans.
- We need plans, and we need to do post-mortems on plans to see what works.
- The plan wasn’t to produce poor documents – we just fall short. We can’t
all be experts at everything.
- Need short quality plans, signed off by IESG and WG founders. The charter
isn’t the only plan a WG can have!
- Not just who, what, when, but HOW we’re going to meet our milestones.
- Challenge assessment, tools plan, review plan...
- Is the charter achievable? What can be done to deal with risks?
- Issue tracking and reporting – has a big impact on WG performance.
Focused discussion leading to document improvements.
- Reviews of the charter (assessing challenges), work item review, mid-term
assessments, late reviews. Cross-area reviews are important! Need help with
problems outside WG core competencies.
- Post-mortems, including recommendations for future WGs.
- Document opportunities – tracking tools, review process (including
post-mortems).
John: Good working groups already do a lot of this. We need to make what
works more visible to other working groups.
Dave Crocker: Need to pay more attention to operations impact (what does it
take to implement something, deploy something, or use something?). We do
this informally... Feedback on quality problems don’t happen for years!
Rodney Hess: How to balance views on challenges facing working groups? How
does your presentation address poor writing skills? Challenge assessment
might help in some cases.
Erik Guttman: Proposal doesn’t change anything, we’re just moving forward
with our eyes open. Does help us to remember the history of debates
succinctly – and this is the most important benefit you’re offering.
David Black: Challenge assessment needs to be a living document, and not a
source of retribution or weapon to kill new work.
Jon Rosenburg: More poor documents from inexperience than poor language
skills! What about exposure to cut-and-paste without thought, just to
produce a quality plan? Bernard: not actually a bad thing, if you
cut-and-paste a decent plan.
IETF Problem Resolution Processes - Margaret Wasserman
http://www.ietf.org/internet-drafts/draft-ietf-problem-process-00.txt
- COACH grew out of Problem WG process recommendation draft (and improved
the recommendation).
- Process draft has two categories, near-term and long-term.
- Need to identify metrics, measure what actually works, and roll them out
IETF-wide. This is classic process improvement methodology. We need to use
the state of the art here...
Starting New Work
The BOF Process: A Critique - Leslie Daigle
- Process pathologies – either sending people away until they figure out
how to interface with the IETF, or people who keep pestering ADs until they
get three BoFs, and then a WG because we’re past the BoF limit...
- Entire focus of the BoF is to create a charter that will get approved!
“Big Red Button”...
- 1-2 hour meeting may not be the best way to bring new work into the IETF.
- Need to cycle on the problem statement and supporting documents, not on
the charter! “Oven” process (for fully-baked ideas?).
- “Oven” work is to bridge work into the rest of the IETF.
- WG chair needs to remove obstacles that keep contributors from completing
the agree-upon work.
- Is this a return to BoFs from the early IETF?
Scott Bradner: I tried to require IDs and mailing lists with discussion
before approving BoFs. There needs to be a basic understanding that BoF
chairs are likely not going to be WG chairs (and isn’t the point the
technology, not the blue dots?).
Rodney Hess: I assumed the oven process was already in play? What about
IETF participants that feel “locked out”?
Aaron Falk: process of baking in the oven is subtle, especially for people
outside the IETF. IESG is often involved in trying to interface outside
work to people in the the IETF, and who you get paired with has a lot to do
with how successful you are in the IETF. Is this a responsibility that SIRs
could help with?
IESG Overload and Quality of WGs - John Klensin
http://www.ietf.org/internet-drafts/draft-klensin-overload-00.txt
- Incentives are wrong – why do something in the IETF? Especially when
there’s nothing obviously fundamentally wrong with a proposal. But
default=yes overloads the IESG, we start too many working groups and often
take too long to make them go away.
- The IESG has the incentive to fix this problem – by making real decisions
about the number of working groups that get created. Need to set a ceiling
on the number of working groups that’s lower than the number that exist
today – force the IESG to start prioritizing. We need a vicious environment
for space, time, and attention.
James Polk: Is it the number of working groups, or the number of documents?
Think SIP/SIPPING. A: We were looking for something simple, and trying to
stimulate thinking. I’d rather limit the number of working groups because
it’s easier to measure. There is no real difference, because the number of
working groups continues to fall no matter how big any working group
becomes – the goal is to lower the level of work until it is manageable.
Margaret Wasserman: This isn’t effective because it’s gameable. We don’t
want to get smaller and do fewer things. We’re just not succeeding in that
today. A: NOMCOM isn’t hearing your message from the community.
Dave Crocker: Why can’t people who understand resource contention
principles apply it? How can we have a social process that can’t be gamed?
Can’t we focus on quality bars, and fix quantity problems that way? We like
these incentives except when they apply to us...
Jon Rosenburg: The notion of “things that constitute the Internet” has
gotten huge. We need scope decisions before we can reduce the work.
The WG Process
Decision points/milestones in the WG process - Margaret Wasserman
http://www.ietf.org/internet-drafts/draft-wasserman-wg-process-00.txt
- Have gotten next-to-no feedback on this document! It’s also used for WG
chair training.
- Want to provide terminology and framework, and help new/struggling WG
chairs get a handle on what’s happening in their working groups.
- Need agreement on process steps in order to have meaningful exit criteria
for each process step.
- Need to distinguish between author outputs and WG outputs – very
different, but not all WGs distinguish.
- Silence needs to stop equaling consensus!
Scott Bradner: working group document acceptance needs “worker bee
enthusiasm” as entry criteria.
David Black: Definitely need to include design team discussion in this
document, including risks and benefits of design teams for working group
documents.
Margaret: Will recommend disbanding design teams when documents are
accepted by working group.
Ross Callum: Now that I’m a working group chair, I’m very interested in
this document! Need to get mailing list validation before WG chairs put
items on agendas for discussion. Margaret: we’re not in any danger of
saying “no” too often – at least not yet!
The Review Process
Careful Additional Review of Documents (CARD) - Brian Carpenter
http://www.ietf.org/internet-drafts/draft-carpenter-solution-sirs-01.txt
- Don’t get hung up on the acronym! AIRs, STARs, etc...
- Trying to help with concentration of power, lack of quality, even when
documents get to the IESG, and late surprises, no credit for being a
reviewer today.
- Need to create and refresh a team of reviewers, and introduce reviews at
00, during development, and pre-IESG forwarding (at a minimum).
- Published reviews, with both local and IETF-wide issues.
- Help working groups produce high quality outputs with no late surprises.
- Not proposing changes to the standards process – only inputs to IESG/RFC
Editor.
- Incentive is to achieve positive reviews.
- Best math is at least 200 reviewers needed (so, perhaps, the IESG really
is overloaded?).
Scott Bradner: Is this standards-track or everything? Everything gets
reviewed...
Bernard: Do we really have 200 reviewers? Brian: We don’t know.
- Need to increase, need to maintain the reviewing team.
- WGs trigger reviews of WG documents, authors trigger reviews of
individual submissions.
- First sentence is the critical part of the review!
James Polk: Are reviewers qualified per-area?
Allison Mankin: Is there any doubt that these reviews are public? Brian:
absolutely.
- Need some help with web tooling to really make this happen.
Ross Callum: really good idea, worried about publicness of the comments for
a vendor employee. Brian: Understand, but this comment worries me.
Margaret Wasserman: Need to understand cost of implementation. We can’t
reduce the amount of IETF work without reducing the number of IETF
participants.
Harald Alvestrand: Can we see a couple of sample reviews, to help us
evaluate this proposal?
Allison Mankin: Don’t make the perfect the enemy of the good. If
significant documents got detailed review once, that would be an INCREASE
over today! Also – need to include compliance with WG charter as part of
evaluation process. Brian: We have running code from MIB doctor process
today.
Henning: Research community has done something like this for years. What
about just reviewing DIFs? Concern about timeliness of reviews – the last
thing we should do is add delay.
The Review Process in Action: The DCCP WG - Aaron Falk
& Allison Mankin
- Want to avoid late disruptive reviews. Used DCCP as pilot process to
accomplish this.
- Collaborated with IESG in choosing reviewers.
- Stable spec, but still able to make changes before WG last call this fall.
- Were able to identify issues prior to design review this week.
- Design review tied to mailing list via WLAN access in real time.
Eric Rescola: I was one of the reviewers – being an official reviewer puts
the spurs to reviewers. Took two days to do my review, wish I had more time.
Spencer Dawkins: Does it make a difference that you worked with the IESG to
identify reviewers, compared to the SIRs draft? Aaron: just a
different-sized reviewer pool.
Brian Carpenter: could SIR make the RFC Editor review a no-op? Aaron: bar
is much higher for working group expert review.
Thomas Narten: Average time required for RFC Editor to produce Individual
submission reviews? Aaron: don’t know.
Summary and Discussion
- Limited time for discussion, please follow up on mailing list.
Henning: these proposals look like good experimental candidates – should we
design a formal experiment, instead of relying on vague recollections? Can
we name names when we’re talking about failures?
Dave Crocker: If we can’t get enough reviews – that’s a feature, because
it’s a filter. If you can’t get reviews for competent work, that’s a
symptom of a deeper problem. SIR wasn’t focused on the details, it was
trying to push toward experience based on an experiment.
--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .