Re: [RRG] Moving the problem to the global mapping system - fast-push

[Robin Whittle <rw@firstpr.com.au>]

Short version:   There are many benefits to having fast-push of
                 mapping, so it is not just a simple trade-off.

                 For instance, with Ivip, there is no need to try
                 to anticipate all the ways end-users might need
                 to detect multihoming failure, or to make decisions
                 about how to change mapping, when to restore the
                 original mapping, how long a loss of connectivity
                 constitutes and "outage" etc.  The end-users do
                 all this themselves.

                 LISP, APT and TRRP developers need to try to make
                 their systems really complex and costly in order to
                 attempt to do as big a subset as possible of the
                 things end-users might want or need.


Hi Michael,

You wrote:

>> However, Ivip's mapping distribution system is completely different
>> from how BGP works and is optimised to handle much greater flows of
>> information, faster and more reliably than BGP.
>>
>>   http://tools.ietf.org/html/draft-whittle-ivip-db-fast-push-00
> 
> Hi Robin, you often refer to Ivip's "fast" mapping distribution method
> as an unabashedly positive thing. 

It is a very good thing in terms of what it does.  The challenge, of
course, is to show that it is worth the effort, and that there is a
business case for it being built.  There is lots of work to do, but
the above ID gives a pretty good idea of what I plan.  Quite a lot
of that ID is stuff which is repeated in the summary and analysis
document:

  http://www.firstpr.com.au/ip/ivip/Ivip-summary.pdf

The gutsy stuff is on pages 33 to 48.


> However, it seems to me that you've simply picked a different
> point the trade-off space -- by increasing the speed of mapping
> distribution, you increase the amount of mapping traffic as well.

I have two comments about this.

Firstly, it is not simply a trade-off where one gets more of one
thing and less of another.  Fast push clearly involves some extra
effort compared to something slower, but the benefits include:

  Modular separation of the multihoming restoration functions.

  Reduction in the size of the mapping information.

  Reduced ITR and ETR functionality.

  Greater security through simplification and modularization.

  Better support for the TTR model of mobility: IPv4 and IPv6
  mobility with generally optimal path lengths.

For a fuller discussion of these benefits, please see page 4 of
Ivip-summary.pdf.

The modularity is absolutely crucial.  All the other schemes provide
very limited options for the end-user to have their mapping changed
- by having more elaborate mapping data than Ivip's single ETR
address, and by having a *lot* more functionality in the ITRs and
ETRs.  Yet no matter how elaborate you make this stuff, you still
won't be able to do what some or many end-users actually want.

In a recent message, Scott Brim wrote a set of questions, which I
think the developers of LISP, APT and TRRP all need to worry about,
such as:

   Who discovers the problem, and how is it dealt with?

   What does the remote ITR do in response?

   How many intermediaries and systems have to be involved?

   How long does it take for the entire path, end-to-end, to
   stabilize again?

   Also, how far is the operational change flooded?

   To multiple mapping points in anticipation?

   To just the ones with traffic flowing?

   What happens when the situation is corrected?

Also, the whole question of "How much of a break in connectivity to
an ETR constitutes an "outage" requiring ITRs to select a different
ETR?" is likely to be answered by different end-users.  However, to
to add sufficient options into the mapping system and the ITR
functionality to cover most of the end-users' divergent needs would
be completely unworkable

How could 1000 ITRs currently sending data to the one ETR monitor
its reachability on a second by second basis without overloading it
with probe packets and with its need to respond to them all?  They
can't.

The Ivip system doesn't have to be designed to provide the answers
to such questions.  For every conceivable end-user and situation,
the end-users get what they want since they do it themselves -
however they like.  (As long as they are happy with a 5 second
latency in mapping changes, and with luck, this might be reduced to
2 or 3 seconds in the future.  LISP, APT and TRRP ITRs are not going
to respond much faster.)

With Ivip, the end-user has one multihoming monitoring system -
which won't overly burden the ETR, no matter how many ITRs are
sending data to it.  They set up their system (probably provided by
another company) to change the mapping according to whatever
criteria they like, including how long a loss of connectivity
constitutes an "outage".

Similarly, how are LISP, APT or TRRP ITRs going to rapidly respond
to the initial ETR coming on-stream again - for those end-user
networks who want the traffic to go back that way, but not for
end-user networks who want to avoid the first ETR for a good while
longer, or perhaps forever?

There's no way of telling what end-users will actually want and
need, but you can be sure end-users will will sometimes, or often,
want or need to changing the mapping to some arrangement they just
thought of right now.

Only Ivip can achieve this.  All the other systems are far to slow
at getting their mapping out to ITRs - unless with ALT or TRRP the
caching times were really short, in which case this places an
unreasonable burden on many ITRs and on the global query server system.

Although with Ivip, end-users need to supply their own reachability
detection system, and typically have it make their mapping
decisions, at least they can do whatever they like - and the
map-encap system doesn't have to invest any costly ITR-ETR
functionality trying to anticipate the sorts of things end-users
will actually want to do.

So fast-push makes Ivip modular and extensible to purposes end-users
really want, and which we cannot necessarily imagine at present.
Since there is a small fee per update, with Ivip we don't have to
fuss so much over "excessive" numbers of updates.


The average reptile might think it a pretty uninspiring trade-off to
change into a warm blooded creature, wasting all that precious
energy and so having to eat more, when it could be lazing in the sun
instead.  But this is not simply a choice in a linear sliding scale
between one thing and another: it enables a bunch of things to
happen which otherwise wouldn't happen, including much more
efficient enzymes, faster and more precisely tuned processes in
neurons, more optimised proteins and more efficient organs at every
level of the organism.  That leads to greater speed in cold
climates, new brain development possibilities etc.  Birds (10,000
species) and mammals (5,400) are doing pretty well compared to
reptiles (8,200).

A trade-off usually involves some kind of compromise.  The new thing
only does one good thing and it costs something to achieve it.

If a new thing does about five really good things - major conceptual
benefits, not just marginal improvements in existing things - and
involves some costs, then its not just a trade-off, it may be a
really great idea and a key part of the design which leads the total
system to be a far better solution than all the alternatives.


I don't agree that by increasing the speed of mapping distribution
that Ivip necessarily increases the amount of mapping traffic.  If
you souped up APT's flooding technique (initially it was a BGP
flooding technique) to send the mapping data everywhere in 5
seconds, this wouldn't increase the amount of mapping data to send.

APT would be much improved by this, and perhaps people would want to
send more mapping changes, but that is a separate issue.

One thing you could do, as I do with Ivip, is only send a single ETR
address.  There's no need for multiple options, ITRs (Default
Mappers in APT) making decisions independently etc. when the
end-user can control the mapping in 5 seconds.  So that reduces the
amount of data per mapping change.  Perhaps, as a result, there
would need to be more mapping changes, so maybe there isn't an
overall saving in the quantity of data to be sent - it depends on
how people would actually want to use it.

With APT, you have no obvious way of charging end-users per mapping
update, so "profligate" mapping changes (however defined) leads you
to exactly the same problem which bedevils BGP today: unfair cost
burdens on people who derive no benefit from a change instigated by
an end-user network.  Likewise with LISP-NERD.

Ivip's fast-push structure makes it easy for the RUASes (Root Update
Authorisation Systems) to charge per update, and so gain the revenue
they need to run most of the fast-push system.  Then, there is not
such a bad problem with "profligate", since the charges will be set
to ensure the costs are more than covered.  Then the end-users whose
changes would be considered "profligate" in an uncharged system are
not being antisocial - they are simply paying at least their fair
share of the costs of the fast push system.

In fact, it is a bit trickier that that - since there are also costs
for the far end of the fast push system which is run by ISPs etc.
Also, whoever runs full database ITRs and query servers is going to
have costs due to each update.

The Ivip business case is not complete, but it does have the
potential to charge end-users per update, and so avoid the "tragedy
of the commons" situation which is a key part of today's routing
scaling problem.

The fast-push system is far more efficient at moving updates around
the planet than BGP.  This is because it has clearly defined inputs
and as many outputs as required, with minimal fast processing by
Replicators in between - whereas BGP has to be a mesh network of
independent devices which process the information and pass a
potentially new version of it to each of their neighbours.


> So even if each update can be handled more efficiently,
> there could be many, many more of them. So you may not have a win
> overall in terms of (processing per update) * (number of updates) *
> (number of devices doing the processing), even versus BGP. Notice I said
> you /may/ not -- if you can come up with some defensible worst-case
> numbers, people may be convinced. Also, people may disagree with me that
> this is a good metric. But I think you'll find yourself fighting an
> uphill battle.

It does seem to be uphill, since no other proposal attempts this.

At least Ivip has a birds-of-feather similarity to APT, in that both
involve hybrid push-pull systems, with local pull and robust notify
systems beyond wherever operators terminate the push system.

I think the benefits of fast-push are very powerful.

I think the task of fanning mapping data out reliably in a few
seconds, across the globe, even to a few hundred thousand sites,
doesn't seem impossible or impractical - given that we have this
marvellous network which can send packets from one side of the
planet to the other, quite reliably, for very low costs, in about
0.15 seconds.  To make the Replicator servers, we have cheap PCs
with multi-core CPUs, gigabytes of RAM for peanuts (the Replicator
needs little RAM and no hard drive) and gigabit Ethernet interfaces
on the motherboard.  The actual quantity of mapping data is not all
that high, even when there are a few billion end users doing
mobility, since they only generally need to send a mapping update
when they move 1000km or so.

  http://psg.com/lists/rrg/2008/msg00535.html 	
  http://psg.com/lists/rrg/2008/msg00832.html

Perhaps the fast push system would tempt people to do a lot of TE,
in the crude, non-explicit fashion which Ivip allows.  That is fine
- 5 second latency control of incoming traffic by controlling the
mapping of single IPv4 addresses or micronets of them.  Those
end-users will always be paying their way on the fast push system,
so whatever rate they do it at is good for business.  (Except as
noted above with the update burden on full database ITRs and query
servers - and on any ITRs which are currently handling traffic for
the micronets they are changing the mapping of.)


>> When trying to frame questions about something like the "mapping
>> function", as we get down to questions which provide meaningful
>> answers regarding important principles and implementation details, I
>> think the questions only tend to make sense if certain architectural
>> attributes can be assumed.
> 
> Actually, I tend to agree with you on that. I quite enjoyed your post on
> the subject. =) However, I think it's possible to set some quantitative
> constraints on possible designs. In particular, setting constraints that
> the design must be able to solve the problems we are setting out to
> solve (without re-introducing them elsewhere) seems quite reasonable to me.

Sure, that is a good constraint.

However, moving mapping changes (for instance each one due to a
change in reachability, or a desire to do TE) to a much more
efficient system than BGP does lead to a much better situation.  If
the new system was as costly and difficult as BGP, then simply
moving the changes somewhere else wouldn't achieve much.


>> My message a week ago:
>>
>>    http://psg.com/lists/rrg/2008/msg01091.html
>>    Taxonomy: 25 questions
>>
>> had several questions which teased out commonalities and differences
>> in the mapping systems of these proposals.
> 
> I saw that post, in fact, you can expect something from me or Dan in
> reply (probably Dan) in the next few days.

OK. Bill Herrin corrected various things I didn't know or got wrong
about TRRP - to which I haven't responded yet:

  http://psg.com/lists/rrg/2008/msg01094.html

If you can do the same for APT - and ideally if someone corrects any
mistakes I made about ALT and NERD - then at some stage I will
rewrite the questions and answers and put it on my site as a kind of
comparison between the current proposals.  Also, maybe someone can
suggest more questions to ask.

  - Robin


--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg