Re: [RRG] RRG process clarification

[Robin Whittle <rw@firstpr.com.au>]

I am responding to Lixia's and Tony's message, and to Tony's
discussion with Lars Eggert about how the RRG Design Goals are still
up for discussion and change.



Hi Lixia and Tony,

I support what I think you are suggesting - a systematic approach by
which all major design decisions are evaluated and ideally some
consensus reached.

However, the only way we can reach consensus about each branch in
the tree-like or somewhat criss-crossing network of design paths is
to go to the end of each path, or combination of paths, to find out
how all those particular choices work together.

Bugs Bunny could only acquire the knowledge required to make a fully
informed choice about whether to turn left at Albuquerque by
exploring the road to the right far enough to be sure that it sucked.

I think it is natural and essential that we discuss lots of low
level details, including "incremental deployability", business
models, problematic technicalities etc. to evaluate how particular
combinations of design choices would work together.


> Our ultimate goal is to deliver a recommendation for a routing
> architecture.  Please note that our goal is NOT to deliver a
> wholly complete, fleshed out routing subsystem implementation,
> complete with protocols, specs, and running code.  Those are all
> non-goals. Although one can potentially learn a great deal from
> running implementations, they are more useful in improving designs
> than fundamental architecture.  What we should be doing is trying
> to reach consensus on the various pieces of the architecture.

I think this can only be done after most people have thoroughly
explored their way down the various divergent design paths.

I try to explore all the way down each road until I come to one or
more "showstoppers" things I think cannot be overcome.  Some people
are convinced that some approaches could never work - they see
showstoppers in those approaches without looking at them in much
detail.  Or perhaps they are convinced that other approaches
couldn't work as well as their own preferred approach, and so never
really consider them in detail or critique them on the RRG.

Even if some proposal looks moribund to me, I try to look at it
further and hope to learn something from it.

I have documented my evaluation of the various branches, by way of
"25 questions" and specific critiques.  I think it would be good if
others did the same, particularly for Ivip, for which no detailed
critique or evaluation has yet been done.  Philip Eardley made a
start and hopefully this discussion will continue.  Michael Meisel
prompted me to clearly state a business case for the fast push
mapping system.  It would be great to have continuing discussion and
critique of these technical and business aspects of Ivip.

To properly explore various design options, it isn't enough to say
why approach X would work, or work best - it is necessary to
evaluate all the half-way credible alternatives to X and show either
why they can't work, or why they are clearly inferior to X.

For instance, I think it would be good for the supporters of
LISP-NERD to point out clearly why LISP-ALT, APT, Ivip and TRRP
either can't work, or can't solve the problem as well as NERD.

Likewise, I think LISP-ALT folks should do a detailed exposition on
what is wrong with APT, Ivip and TRRP.  Ignoring these proposals is
not as helpful as stating clearly why they can't be as good as
LISP-ALT.  In doing so, a good debate would ensue and it would be
possible to understand to what extent this view was the result of
differences in understanding the problem, the future environment,
in misunderstanding APT etc.  It may also point up some previously
ignored problems in APT etc.

There's enough documentation of Ivip and of the most challenging
part of it - the fast hybrid push-pull system and its potential
business models - for someone who supports anything other than Ivip
to do a detailed critique.  Hopefully, that will happen soon.


> An architecture (a.k.a., a framework) is a set of concepts around
> how a system works.  An architecture divides the system into its
> various subsystems and describes the functions of those subsystems
> and their interactions.  Obviously, one can iterate down,
> performing hierarchical decomposition of the architecture.  If you
> pursue this far enough, then you'll find that you have an
> implementation, but again, that's not our charter.  We need only
> go so far as to provide a working architecture.

Sure, but the best design has a bunch of elements which work
unusually well together.  Such a good combination can best be found
by playful attempts at various combinations, including of ideas
which at first seem odd or undesirable.

For instance, the outer packet's source address being that of the
sending host.  I think this is very desirable in some ways - and I
only explored as a result of mistakenly thinking this was how LISP
worked.

Only once the design is done can one step back and describe it in
terms of choices between one path and another in a top-down "design
process".  That is not really a design process.  The design is
already done and this is a way of describing it.  A proper decision
process at Albuquerque can't be done when first arriving the fork in
the road.  The proper decision can be described at the fork, but the
decision can only be made after having travelled a long way down at
least one of the two roads.

I can imagine the RRG reaching some kind of consensus on a document
which describes the choices in a correctly ordered set of design
decisions.  But very often, the reason for one top-level choice is
only revealed in the finer points of the design, by all the lower
level aspects it enables.


> The level of detail that we have to deliver is certainly open to
> discussion.  Since we are not expected to deliver the engineering
> part of the solution, it is sufficient for us to provide enough
> specifics so that good, competent engineers can take the
> architecture and design the details of a solution without further
> guidance.

This sounds good to me.


> Initially, we started with putting a number of proposals on the
> table, which helped us better understand the design space. However
> we have noticed that lately the discussions have mostly focused on
> microscopic detail about the various proposals instead of the high
> level architectural issues. This is natural, we have a great deal
> of engineering experience, and we're more easily attracted to how
> the little bits work at the bottom than how things should look
> from the larger perspective.
>
> Unfortunately, our charter is to understand that larger
> perspective.

I think this is how engineers figure out the larger perspective - by
trying a variety of distinct designs, each of which makes good sense
based on whatever assumptions it starts with, and/or according to
different peoples' views of the problem and the environment in which
the solution operates.

When writing something complex, it is not typically possible to
start with anything like a coherent summary of what the final piece
will contain.  I have to dive in, write a bunch of stuff, re-arrange
it to make it better, leave it sit overnight or for months,
rearrange it some more, have some over-arching insight, rearrange it
some more and then finalise it.  Then, a month later, I might be
able to write a concise account of what I arrived at.


> There is a very broad solution space at hand, and we need to
> understand the breadth of it.  Yes, we could simply pick one
> solution and go with that, but we would not be performing our due
> diligence.

I think a broad discussion like we are having now is a crucial part
of the due diligence.

I think the RRG could fulfil its charter by evaluating a set of
separate, cohesive, proposals and arriving at a consensus that one
particular proposal was significantly more promising than the rest.
 Then, we could articulate the reasons why we think this is the
case, noting any ways we think that proposal could be improved.
This may involve using parts of other proposals.

Then the recommendation would be for an RRG-specified set of
architectural principles, which work well together for the reasons
the RRG agrees on (rough consensus) and describes in our
recommendation.  We would note that these were best exemplified by
proposal X, adding various reservations and suggestions for
improvements.

I think this recommendation would be at about the level of detail
you are planning for, but it would include more specific mention of
proposal X than I think you currently plan for.  It would recognise
that proposal X was a work in progress, with many things yet to be
decided, so it wouldn't endorse proposal X as such.  Proposal X
could subsequently change in some way the RRG would not have been
happy with.  So the recommendation would be a set of architectural
principles, building blocks, specific goals to be solved in specific
ways, recognition that some things were either non-goals or remain
goals, but have not yet been solved etc.

This way, anyone could start up some proposal Y, based on all the
desirable elements the RRG outlined (most or all of which can be
found in proposal X) but then go on and develop Y as an alternative
to whatever X develops into, while remaining true to the RRG's goals.


> The community has placed in us the responsibility to understand
> the whole of the solution space, partition it, and select the best
> path through it.
>
> Without exploring the solution space at least a bit, we won't have
> done a credible job of thinking about all of the ways that we
> could architect the system.

I think we are exploring the space quite well.  I think it has been
something like this:

  We recognised and agreed on the problems in today's BGP system:
  how it unfairly burdens DFZ router operators with costs incurred
  due to  the existence and activities of edge networks.  From that,
  we recognised that the current BGP system places increasingly
  unacceptable limits on the desires of many end-user networks to
  get address space which is suitable for multihoming, portability
  between ISPs and traffic engineering.


  We explored what could be done to soup up the BGP system to make
  it scale better:

     Conclusion:  Very little - certainly not enough to make more
     than a few tens of percent difference to a problem which is,
     at least in one measurable respect, doubling every 4 years.

        http://bgp.potaroo.net


  Therefore, explore complete replacements for the BGP system:

     Conclusion:  None of these could be introduced, and even if
     one could be, it would be challenging to come up with a
     system which worked markedly better than BGP.


  Therefore, explore overlay architectures which leave the BGP
  system going and take most of the load of end-user network
  address management and forwarding/routing from the BGP system.
  The potentially practical ones - which work for both IPv4 and
  IPv6, which not involve host upgrades or drastic replacements
  of DFZ routers, fall into two classes:

    Tunnelling systems: (map-encap) LISP, APT, Ivip TRRP.

    Address translation:  The only one being seriously discussed and
                          which works with both IPv4 and IPv6 is
                          Six-One Router.


> Further, once we have surveyed the solution space, we need to
> develop rough consensus on the approach through the solution
> space.

I think all this low-level discussion is our best way of surveying
the solution space.

I think there are attempts at high-level theoretical discussions,
such as the recent post by Peter Sherbin, but I can't see how this
helps us arrive at practical proposals which could in fact be
introduced widely without inducements or major upheavals.


> Arguing about 'incremental deployment', for example, doesn't help
> this at all.

Since the RRG Design Goals require a solution to be "incrementally
deployable" I think we need to reach consensus on what this means.

I found I had a much more stringent understanding of this term than
many other people.  I am not trying to change anyone's mind about
what "incrementally deployable" means, but if the consensus is that
it means the less restrictive meaning, then I argue that we would be
better off with a more stringent requirement such as mine.

  http://psg.com/lists/rrg/2008/msg00957.html

The likelihood that a proposal will be widely adopted without
inducements is one of the key requirements of any successful
solution.  I think the less restrictive interpretation of
"incrementally deployable" test would allow quite a few proposals
when only a few, or perhaps none of them, could actually succeed in
the wild and be adopted widely enough without inducements to make a
difference to routing scalability.


> We need to first come to some agreement on the very highest level
> branches in the decision tree (e.g., do we do map-and-encap or
> translation or ???), and not worry about the detailed leaves.
> That is up to the IETF to wrestle with.

I am sure it is for us to wrestle with.  Good designs are cohesive
and have all the bits fitting nicely together.  Ideally, each
element does two or more separate beneficial things and doesn't get
in the way of something important.

If we did a top-down design by committee, without fully exploring
what the result would be like, how its bits would fit together etc.
the result would be the IETF folks (probably us, in fact) throwing
in the towel when they realise the architecture is moribund - that
it looked good from a distance, but doesn't actually allow the
creation of anything like the best working system.

No-one could arrive at the optimal design for any complex system by
making one high level decision after another without first exploring
all (or at least many of) the options which each choice would
support or preclude.


> Getting a consensus on this very highest level branches is a
> difficult task, as it requires a good understanding of multiple
> important factors as well as their interplays, perhaps with the
> mapping system design as a centerpiece. Again, our job is to
> deliver an architecture, thus we must first reach such a
> consensus.

Thus we must continue to explore the particular designs on offer -
with each participant ideally contributing to a robust critique of
the ones they don't support.

Ignoring the other proposals and focussing on one preferred approach
is not good enough, because it is not joining the debate and it is
not allowing for the possibility that maybe the real strengths and
weaknesses of another proposal are different from what they appear
from a distance.


> As we gain consensus on more and more of these high level design
> decisions, we will then be able to put together a sound
> architecture.

I think the process at present is healthy - multiple teams of people
doing their best to design a sound architecture in their own ways.
With more thorough debate, including detailed critiques of all
proposals, some broad consensus will probably emerge on which design
package (or design direction) is best.


This is assuming there is sufficient agreement on the problem and
the current and future environment a solution must operate in -
which may not be the case.

For instance there has already been some difference of opinion on to
what extent mobility or helping to solve the IPv4 address depletion
problem should be goals for an RRG-recommended architecture.

Lars Eggert wrote about this:

> All the solution proposals on the table apear to have slightly
> different sets of problems they address, based on what the
> proponents of each consider important. The ongoing discussion
> intermingles arguments about which goals are important to address
> for a future architecture with which technical mechanisms are
> suitable to address them, and that's confusing (well, to me at
> least.)

Some people don't want to overload the RRG process with too many
goals and therefore constraints.

Other folks believe it would be wrong to define and construct a new
architecture which only solved one thing, whilst not considering
other major goals or desirable outcomes - some of which can probably
be achieved by one or more of the current proposals without
significant extra complication, but not by others.


There are fundamental uncertainties and differences of opinion on
numbers of "end-user networks" the new architecture is supposed to
serve in the decades to come.  There is no consensus on the future
numbers of end-user networks and therefore EID prefixes, within two
or three orders of magnitude!

I think there is a wide divergence of opinion on how big these
end-user networks will be and what they will be doing with their
space.  The biggest part of this divergence of opinion is probably
about whether the future definition of an "end-user network" will
include each hand-held device of billions of consumers: mobile
address space tied to their cell-phones and laptops etc.

If we had a reasonably cohesive understanding of goals, future
environment, what "end-user networks" means, how many there will be,
how they will use their space etc. AND once we have more fully
explored the strengths and weaknesses of current design proposals,
then, it would be possible to backtrack and state, with rough consensus:

   "Our first major decision is that the future architecture will be
    based on map-encap."

If we decide this (I think we will), we would also be able have
consensus about why we chose it, such as:

  "Because there are one or more viable design approaches with
   this, and because there are in-principle reasons why all other
   alternatives can't work."

And/or:

  "Because no-one was able to create a potentially practical
   proposal which wasn't map-encap."


> For people who are experienced engineers, this is somewhat
> analogous to writing a functional specification of a system.
> We need not do the entire work of detailed design.  Rather, our
> task is to specify all of the vital properties for later
> refinement.

I think this is fine.  I think we may be able to reach a consensus
on the best architecture by next March.  But only by pursuing
low-level detailed, critiques of the practical aspects of all
current proposals.


Regarding the RRG Design Goals, Tony wrote to Lars:


>> I must admit that I don't follow the RRG very closely very
>> often (it requires a full-time commitment), but I don't
>> remember the discussion around this document resulting in a
>> consensus.
>
> There was some discussion, but not a great deal of contention.
> The comments did taper off, and that was taken as a sign of rough
> consensus by yours truly.

It isn't complete consensus (I am not suggesting it needs to be),
because I wrote a bunch of stuff which suggested improvements to the
Design Goals:

  http://psg.com/lists/rrg/2007/msg00199.html

To this I would add that we need to agree on what "incrementally
deployable" means.  If it is agreed that this is as weak a concept
as some folks indicate in recent discussion, I argue we need to have
a much stronger criteria, involving likely widespread or ubiquitous
adoption without disruption and without significant inducements.

>> But I may be wrong, and RGs don't need to operate by consensus.
>> In any event, the document is definitely is a good start.
>
> We intentionally did not issue it as an RFC so that it could be
> kept open for further changes.  Suggestions are still welcome.

OK - mine are listed above.


  - Robin


--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg