Re: AD review of: draft-ietf-tewg-measure-05.txt

[Dimitri.Papadimitriou@alcatel.be]

bert,

the proposed update meets the comments i raised,

now my two cents, as it stands now i see this i-d more
as an "analysis of the requirements for tem models and
techniques" since imho the authors took an analytical
approach and in order to probably to come with more
"crispy" input for the tewg (since the objective was
here to "Document additional measurements needed for
TE (TEM)") expansion starting from the conclusion
might be a way to proceed (take this as a suggestion
to help things moving forward) if this is what the
tewg consents to do.

see also in-line...

Wijnen, Bert (Bert) wrote:
> WG,
>
> I'd like to hear/read more comments/feedback on my review.
>
> Inline responses to the comments from Jerry
>
> Thanks,
> Bert 
>
> > -----Original Message-----
> > From: Ash, Gerald R (Jerry), ALABS [mailto:gash@att.com]
> > Sent: vrijdag 25 april 2003 23:43
> > To: Wijnen, Bert (Bert); Tewg (E-mail)
> > Cc: Ash, Gerald R (Jerry), ALABS
> > Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt 
> >
> > Bert,
> >
> > Let's not throw the baby out with the bath-water.  The I-D 
> > serves a critical need and should go forward.
> Can you formulate the "critical need" that is being served?
well, i think here you ask a critical question but is your
question not just expanding on the reasons why this has
been included within the charter of the wg ? i think that
section 4.4 of RFC 3346 expand on this

> > Regarding your comment:
> > 'I would expect a CRISP set of "requirements for additional 
> > measurements, configurable/negotiable parameters/controls" 
> > ... but not the extensive exploration and text that I now see'.
> > The requirements *are* there for additional measurements, 
> > configurable/negotiable parameters/controls, but they need to 
> > be made more prominent and crisp (as you say).
> Well cedrtainly I could not see them... they are too well hidden 
> I guess :-(. So making them CRISP is certainly a good thing. 
this is what i'd suggest also (in case of re-spin), expand
section were such statement can be summarized in order to
reflect the achievements here for the tewg ...

> > I'd suggest to focus the main body of the I-D on these 
> > requirements/recommendations, and move the descriptive 
> > material to Annex(es).  The text has evolved over a long time 
> > with many comments and responses, as such it has gathered 
> > some moss along the way...  But the critical essence is 
> > there, and the requirements *need* to move forward to provide 
> > essential TE measurements.
> So.... it seems we agree that this title:
>     A Framework for Internet Traffic Engineering Measurement
> is not good and should be something aka
>     Requirements for essential TE measurements, parameters and controls
> and then of course the main body of the document should focus on
> that and be clear as to waht is a real requirement and why.
well your question make sense in terms of "ease of read and
further use" and i think the reason why "content has been
spread" is also due to what kind of requirement (functional
vs protocol) where drafted wrt to the wg expectations - does
this just not translate the complexity of the subject ? -

> > Regarding your comment:
> > 'if we were not yet doing any of it... then I wonder if this 
> > would be helpful at all.'
> > There are many important requirements/recommendations in the 
> > I-D.  As one example, the ability to collect traffic data to 
> > generate a traffic matrix (e.g., 'node-pair data', see Table 
> > in Section 10.1) is woefully lacking.  To get a flavor of the 
> > problem/need, see these papers on the topic (links to papers 
> > at http://www.research.att.com/~jrex/papers.html#tmeas):
> > A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford, 
> > G. True, "Deriving traffic demands for operational IP 
> > networks: Methodology and experience," IEEE/ACM Transactions 
> > on Networking, June 2001, pp. 265-279. (An earlier version 
> > appeared in Proc. ACM SIGCOMM, August/September 2000.)
> I'd like to see in the document something aka:
>
>    REQUIREMENT: ability to collect traffic data to generate trafix matrices
>    DESCRIPTION: describe the details as to what traffic data is needed
>    JUSTIFICATION: describe what the problem is that needs to be solved
>    PRIORITY: x
>
> Don't need to dwell too much on that... but have a concise and crisp
> writeup. Maybe RFC3216 can serve as a good example.
again, this is probably the issue but i'd like to mention that
the priority and justification may depend on a per user basis

also i took a look at this RFC and compare for instance the abstract

"This document describes the objectives for a new data definition
 language, suitable for the modeling of network management constructs,
 that can be directly mapped into SNMP and COPS-PR protocol
 operations."

well let's try and see if this is achievable here..

"For interoperable compatibility, uniform definitions across vendors
 and operators must be ensured, e.g., in the distinction between
  offered load and achieved throughput

=> so covers terminology

"To aid network dimensioning, mechanisms to collect node-pair-based 
traffic data should be developed to facilitate the derivation of 
per-service-class traffic matrix statistics.

=> associated requirements to acheive node-pair-based traffic data
   and expectation in terms of derivation mechanisms in providing
   stat's

"For service assurance, there is a need for the use of higher-order 
statistics. To preserve representative traffic detail at manageable 
sample volumes, there is a need for packet-sampled measurements.

=> requirements in terms packet-sampling measurements

"To manage large volume of measured data, use of bulk transfer and 
filtering/aggregation mechanisms may be appropriate."

if a re-spin is needed this might be a way to proceed.

> > This work shows the difficulty in deriving a traffic matrix 
> > with today's measurements.  A traffic matrix is basic to any 
> > network engineering or planning task.
> >
> > Other critical traffic/performance measurement requirements 
> > identified in the I-D are also lacking and need to go forward.
> >
> > A few more comments below.
> >
> > Jerry
> >
> >
> > > - a lot of text... but not very focused and to the point 
> > (or at least I have trouble seeing the main points)
> >
> > > - not a "framework", rather an exploration of TE measurement 
> > >  related topics (more like  a summary-introduction)
> > As above, the I-D serves a critical need.  The text can be 
> > focused on the requirements/recommendations, as suggested, 
> > and the descriptive stuff moved to an Annex.
> One could wonder how much of the descriptive stuff is really
> helpful or usefull. But if it is in an ANEX, then maybe that is
> OK. I would say however, that if we have the requirements
> crisply formulated, with description, justification and priority,
> then one would hope that a lot of additional text is not needed
> (not even in an anex). Think about the audience. It would be
> protocol developers and/or vendors who have to understand the
> requirement and be able to make modifications to their protocol
> and/or products to meet the requirement.
bert, here that's a critical point, i have mentioned it here
above as well => where the expected outcomes in terms of functions/
protocol aspects or both ?

now annex might be useful if using the proposed scheme as
per RFC 3216, expansion on the "why" (as you have mentioned
here above) might difficult to infer, for instance - let's
first see what it gives before expanding on whether it belongs
to an annex or should be dropped from the i-d - i am afraid
that if we use one short paragraph or sentence for the why
people will show up and ask for more content afterwards -

> > > - I am not an operator, but I think if I were one, then
> > >  - if we were already doing TE measurement stuff (most 
> > >    likely) then reading it seems a waste of time
> > >  - if we were not yet doing any of it... then I wonder if 
> > >    this would be helpful at all.
> > Operators have to get by with what is available, what else 
> > can we do?  This does not mean operators are already doing 
> > 'TE measurement stuff' in an adequate way, and that nothing 
> > further is required.  E.g., see comments above on derivation 
> > of a traffic matrix.
jerry, as a suggestion this might be a way to introduce the
topic, "why operators think that current TEM when available
is not performed adequately and what's expected to achieve
what you need"

> I would hope operators can also understand and support the 
> requirements. If not, then I really doubt that they are valid
> requirements. And they must understand them, so that together
> the operators can say to protocol developing WGs and to vendors
> that these are the set of requirements to be met.
>  
> > > - W.r.t. the TEM WG work item, it says:
> > >     The tewg interacts with the common control and measurement plane
> > >     working group to abstract and define those parameters, measurements,
> > >     and controls that traffic engineering needs in order to engineer
> > >     the network. 
> > >  So I would expect a CRISP set of "requirements for additional measurements,
> > >  configurable/negotiable parameters/controls" ... but not the extensive
> > >  exploration and text that I now see. Why do people (or the WG) think
> > >  that this document meets the WG deliverable for TEM ??
> > The requirements are there, and are critically needed.  They 
> > need to be brought out in a clear and crisp way, as suggested above.
> Yea.. you keep repeating that.
bert, i think the content is there ... imho we speak
about the form it took rather than the content here
(it might always be improved but this is the case for
any standard track rfc) in all the comments you have
listed i do not see any major "content" issue

section 4.4 of RFC 3346 shows that this aspect must
be covered by the wg

last, i don't want to expand too much on the below, i
think it becomes a bit touchy

thanks,
- dimitri.

> > > - W.r.t. review:
> > >  - 4 people from ATT support it. Waisum is one of them and is main editor.
> > >    Others (Nick Duffield, Bob Cole) have some of their material listed in
> > >    the doc.
> > So AT&T is stuffing the ballot box?
> That was not what I am saying. My point was more: it is clear that those
> people who have contributed support the document. Would be strange if they
> did not, would it not? But I can see now that maybe I should have made
> my statement a bit different.
>
> > I'm mentioned in acknowledgements, others in references, are we
> > disqualified to comment?  A certain set of people (*individuals*)
> > who are  the ones most interested/knowledgeable are the ones also
> > most likely to comment.
> >
> > >  - 3 EDU users commented, 2 said they found it a good doc
> > >    3rd one asked a few questions
> > >  - Raymond Zhang is positive. He is from info.net ???
> > >    is that an operator?
> > yes.
> I thought so but was not sure. Maybe Raymond can answer himself, no?
> Raymond, can you fill me in what sort of network you operate?
> Feel free to do so privately if you prefer that.
>
> > >  - Richard Tibbs gave a thumbs up, he is from oakcitusolutions.com.
> > >    What role/function does he play/have? Operator, code/tool-developer?
> > co-author.
> Ah... ok, so obvious that he would support it. Should have been in
> my first list of people above. 
> You did not answer (but better if Richard does himself) if he is
> and operator or what role plays in his daily life. The reason I ask
> this is because I want to udnerstand their background and the eyes
> and context that they probably have used to review.
>
> > >  - One hotmail user (Spyrokontigiogis) gave a thumbs up. Not that
> > >    he/she added any comment. Do we know him/her?
> > >    What role/function does he/she play/have? Operator, code/tool-developer?
> > Level-3 employee?
> Can you be more specific? Is he/she in the accounting dept? I guess not.
> But again, I want to understand the context within which the review was done.
>
> > >  - Dimitri Papadimitrou (Alcatel) asked a question/suggested some text.
> > >    I did not see if he likes the doc or not
> > did not object
> >
> > >  - Blain Christian (uunet, so maybe a real operator?) 
> > >    withdrew as co-editor
> > >    That does not sound good (in my view)
> > > - So where are the real operators that support this?
> > So *individuals* from AT&T, Infonet, France Telecom, and 
> > Level-3 (?) supported this, IETF does not have 
> > companies/operators positions.  Also, are you saying that the 
> > above operators are not 'real' operators versus other 
> > operators who you think are real operators?
> Sorry for the bad wording. When I use company names, I try to
> see how many differnt viewpoints from people active in different 
> types of networks have looked at it. From their operational experience.
> I do not consider them as speaking for their company, but indeed as
> individuals. I'd like to see comments/review/support from all 
> types of operators, that is carriers, telecom operators, cors ISPs,
> Access ISPs, small ISPs, Enterpise Networks... etc.
>
> > > My thought is that I can do two things:
> > >
> > > - send back to WG and say that this is "not good enough" and 
> > >  that I do not 
> > >  feel comfortable to present it to IESG for approval.
> > >  It does NOT meet (in my eyes) the WG deliverable for TEM.
> > > - send it to IESG with my recommendation to NOT approve for 
> > >  same reasons.
> > >
> > > So let me try the first option first and ask the WG what they 
> > > have to say to my review.
> > I find it rather disappointing that you waited this long to 
> > come down on this I-D so hard.  If you had this level of 
> > concern, I think you should have tipped your hand long ago, 
> > but perhaps this is first time you have read the draft.
> That is a valid complaint. We are aware that a lot of the AD/IESG level
> review takes place too late in the process. Not easy to fix, we (ADs)
> all have trouble to keep up with what we have on IESG agenda, let alone
> to dive into many WG docuemnts early. We try to improve... but in this
> case I missed it.
>
> Now I do understand that the WG chairs have been pushing back on this
> in the beginning quite a bit too. I believe that the doc has improved 
> quite a bit since the earlier versions... but when I review it in
> the context of the WG deliverable, then I am clearly still very
> dissappointed. If you say that all the "requirements" are hidden in
> that doc, then pls try to make them clear. It seems that needs major
> rework.
>  
> > We have been admonished many times to read the draft and comment 
> > on this list, and many folks have abided by your wish.  
> I guess it depends on what you count as "many". When I was a kid,
> we joked about people counting:  1,2,3,many. Mmm.. maybe that statement
> is not appropriate. 
> > If we discard this I-D, the TEWG charter item also will go 
> > unfulfilled.  That's bad. I think the I-D serves an important 
> > need, and can address your comments, as above.  
>
> But I'd rather have us confess that we were unable to deliver on a
> WG item than us delivering something that is not in good shape
> (or bad quality, which I think is the case).
>
> Hope this explains my posting a bit better.
> Bert
--
Papadimitriou Dimitri
E-mail : dimitri.papadimitriou@alcatel.be
Private: http://www.rc.bel.alcatel.be/~papadimd/index.html
E-mail : dpapadimitriou@psg.com
Public : http://psg.com/~dpapadimitriou/
Address: Fr. Wellesplein 1, B-2018 Antwerpen, Belgium
Phone  : +32 3 240-8491