Re: [RRG] Elegance and the rejection of SHIM6 host-based multihoming

[Robin Whittle <rw@firstpr.com.au>]

Short version:   More on SHIM6's decentralises approach to
                 multihoming reachability testing, - and the
                 somewhat less decentralised approach of using
                 ITRs to do this (LISP, APT and TRRP) compared to
                 Ivip's more centralised approach.

                 Also, do we upgrade the "routing system" to
                 scalably do what it already does: provide stable
                 IP addresses to networks and hosts - or do we
                 assume this is impossible or undesirable, and
                 insist that host operating systems and applications
                 change to cope with IP addresses which could change
                 at any time?

                 SHIM6 does not provide portable address space.
                 Nor can it do reliable address referrals without
                 application changes, since it would be necessary to
                 refer with two or more IP addresses.

Hi Iljitsch,

You wrote:

>>> Well if all your hosts are multihomed your entire network is
>>> multihomed. End-users shouldn't concern themselves too much with
>>> the details of technology that aren't of immediate importance to
>>> them.
> 
>> It is important to the administrator of an end-user network whether
>> portability and multihoming (TE too) can be controlled purely at a
>> router or some other central location, or whether every host in the
>> network needs to be coordinated in some way to achieve this.
> 
> Yes, but it's unimportant whether this control is an inherent part of a
> centralized design or is simply added to a decentralized design because
> it's an additional requirement. There is nothing stopping you from
> coming up with a shim6 control protocol.

Adding a control protocol for altering the SHIM6 behavior of all
hosts in a network would be an extra layer of complexity - which is
important.  It would also involve scaling problems in larger
networks, and require either the management system to discover all
hosts, or all the hosts to be configured to auto-discover the
management system.  It would also presumably involve changes to SHIM6.


>> For instance, each SHIM6 host does its own reachability testing and
>> multihoming service restoration decisions.  This is extraordinarily
>> messy
> 
> It's also extremely effective, because this actually tells you whether
> you have connectivity, while the presence of a prefix in BGP doesn't
> tell you much because the unreachability will usually be aggregated
> away, and it's scalable. 

In principle it could be more effective than monitoring the
reachability of an edge network from a single location.  When I
suggest that most Ivip end-user networks would pay a special
multihoming monitoring company to continually monitor connectivity,
and change the mapping of their micronets if an outage occurred - I
also suggest that these companies would monitor from multiple locations.

That isn't necessarily as robust as each sending host doing its own
reachability testing, but it scales a lot better.  It also enables
the end-user network to precisely determine how their network's
reachability is probed, what criteria is used when deciding there is
an outage - and what mapping changes to make.  With SHIM6 or the
non-Ivip core-edge separation schemes, the testing and decision
making functionalities are implemented in the sending hosts, or the
ITRs, respectively.


> It also avoids the need for continuous keepalives that is present in 
> many protocols.

But how could an application protocol know it is running over SHIM6
and therefore decide it doesn't need to use keepalives?  How could
SHIM6 know what level of continual probing to conduct in the absence
of actual traffic packets?  The recent paper:

  http://conferences.sigcomm.org/hotnets/2008/papers/18.pdf

points out this difficulty: many protocols don't have a continual
flow of packets, so unless there is continual (inefficient and
unscalable) probing by the ITR (or by the SHIM6 layer of a sending
host) then the reachability problem is only discovered some time
after it occurs, when the sending host has data to send.  Then,
there would be seconds of delay while the application gets no
response and the SHIM6 layer somehow decides that the current
address doesn't work, before the application would try again (I
think this is how it would work) and the packet could be sent to
another address, via another ISP, which did work.

With Ivip, the multihoming monitoring system will detect the problem
pretty quickly - probably within seconds - and change the mapping,
again within a few seconds.


>> With SHIM6, the situation will often be worse than with LISP etc.
>> If there were 100 ITRs each handling packets from 10 hosts which are
>> sending packets to this one edge network, then this means there are
>> 1000 hosts all doing their own reachability testing, making their
>> own decisions etc.
> 
> Anarchy!  :-)

Costs, inefficiency, unscalability . . . :{


>> A system which completely isolates the hosts from multihoming,
>> portability etc. by providing them with a consistent IP address at
>> all times is arguably a lot simpler to run and debug than one which
>> pushes a lot of the complexity, responsibility and potential
>> problems into the host operating systems themselves.
> 
> Yes, but our architecture is such that it's impossible to communicate
> this type of information between the network and the hosts, so a lot of
> information is lost if you take this functionality out of the host. We
> do congestion control in hosts for a reason.

I don't understand this.


>>>> Also, people want address space which is portable between ISPs
>>>> - so they don't have to renumber their network, DNS etc. when
>>>> selecting a new ISP.
> 
>>> Shim6 could have given them that and maybe still could in the
>>> future by using a separate ID space.
> 
>> I don't see how SHIM6 could be used for providing portable address
>> space.
> 
> shim6 just negotiates extra addresses and switches to a different one
> when necessary. If you can boot strap the address negotiation without
> requiring an initially working address pair then you could simply use
> ULAs or some such as the identifiers. However, this would require an
> id->loc lookup of some sort, which is something we were reluctant to
> include in shim6.

I don't clearly understand this.  From all I know, SHIM6 does not
enable a network to have portable address space.  So moving to
another ISP requires changes to all DNS entries, and to any other
place where IP addresses are used.  This includes any ACLs, I guess.

With a stable IP address, which is portable no matter which ISP is
used, (the aim of Ivip, LISP, APT, TRRP and Six/One Router) there is
no need for changing DNS, no problem with DNS caching times and no
changes to any ACLs anywhere, such as some filtering arrangement
restricting VPN access to remote hosts with particular IP addresses,
or in particular networks.  Furthermore, IP address referrals work fine.

SHIM6 can't do IP address referrals and support multihoming at the
same time, since by referring a single IP address to some other
host, by the time that other host uses that IP address, the ISP link
that host relies upon might be down.

Address referrals with SHIM6 would involve two or more - an unknown
number - of IP addresses, so the host which receives this list would
try them until it found one which worked.  This would require
application changes.


>> I think that the administrators of end-user networks are perfectly
>> justified in seeking a solution which isolates hosts from any
>> effects of multihoming service restoration, moving to another ISP etc.
> 
> Sure, and it's perfectly reasonable to go live in the desert and pump up
> water from deep beneath the surface to water your lawn and run an air
> conditioner 24/7 to keep cool.
> 
> Some things just don't work in the long run.

You and some other folks can't imagine the Internet running
indefinitely with an architectural enhancement such as Ivip which
gives each end-user network its own, stable, portable, IP address or
subnet of addresses.  I can.

I think it is the proper role of the routing system - with proposed
enhancements - to give each host or network stable address space,
including for portability between ISPs, for multihoming and for
mobility.  I think it can be done and that it would be desirable to
do so, compared to the alternatives.

You think it can't work out this way - that there is no
architectural enhancement which could or should remove from hosts
and end-user networks the need to cope with arbitrarily changing
addresses, such as due to multihoming failure, the need to use
another ISP or mobility.


>>> Loc/id overload isn't so bad.
> 
>> Do you mean the current situation where an IP address is arguably
>> used to define both the end-point device and how the packet needs to
>> be forwarded across the Internet to reach that endpoint?  If so, it
>> is bad because it leads to a scaling problem when more and more
>> end-user networks get what they need: PI space, in the only way
>> currently possible: via a BGP advertised prefix.
> 
> Well, if we accept that then basically a loc/id separation is a way to
> avoid renumbering. Renumbering is only painful when you're actually
> doing it. Loc/id is painful for every packet and for every session...

Renumbering is disruptive, costly and in many cases completely
impractical.  A core-edge separation scheme such as Ivip does
involve extra bits of equipment and a mapping system.  It doesn't
necessarily involve encapsulation overhead and the PMTUD problems
this brings.  My page:

  http://www.firstpr.com.au/ip/ivip/

lists two approaches to forwarding, without any overhead or PMTUD
problems, for IPv4 and IPv6 - but these require upgrades to DFZ and
many other routers.  Maybe that won't be so hard in the time we have
available - and would be easier than creating more complex ITRs and
ETRs for encapsulation.

My approach is to upgrade the routing system (broadly defined) to do
what it currently does - give hosts and end-user networks stable IP
addresses - but to be able to do so scalably for 10^10 or whatever
separate end-user networks having portable, multihomable "Scalable
PI" address space.

Your approach is a retreat from the original division of labour in
the Net, on the basis that we can't possibly expect the "routing
system" to do this - therefore all hosts must change.  Host
operating systems and applications must change so they can cope with
 having a changed IP address at any time, or at least with
multihoming having two or more addresses, with no guarantee that all
 of them will be reachable.   This does involve application changes,
since with the approach you suggest, there can't be reliable
referrals via a single IP address, which is the current arrangement.


>> Since reachability testing is built into the SHIM6 functionality of
>> each host, how are end-user network administrators going to be happy
>> with this, since they are likely to have a variety of desires about
>> what constitutes an "outage", what should cause packets to flow one
>> way or another etc.?
> 
> Sorry, that doesn't make any sense to me.

A multihomed network may have ISP A which is cheap, fast and
generally reliable as its preferred link to the Net.  It may have a
slow, costly, ISP as a backup link.  With SHIM6 the reachability
testing and multihoming service restoration decision making is done
by each correspondent host, in a way which is codified into the
SHIM6 standards.  With LISP, APT, TRRP and Six/One Router, these
things are built into the ITR (or its Six/One Router equivalent -
the translation router).  The mapping system gives the end-user
network some prioritisation of whether to use ISP A's ETR or ISP B's
ETR, but it doesn't give any sophisticated control over how these
ITRs probe for connectivity, or how quickly they make a decision to
use ISP B's ETR.  Nor is there a way the end-user network can
specify how quickly to go back to using ISP A's ETR, and the probing
and decision making criteria for doing that.

With SHIM6 and the non-Ivip core-edge separation schemes, the
processes of multihoming reachability testing and service
restoration decision making are monolithically integrated into these
schemes.  With Ivip, these are completely outside the Ivip system -
a modular approach more in keeping with good engineering practice.

With Ivip, the remote hosts and the ITRs are not at all involved in
probing reachability, or making any decisions about multihoming
service restoration.  That is done by the end-user, or more likely
by some company they hire to probe their network and send mapping
changes in the event of an outage.  With Ivip, the probing and
decision making arrangements are not at all part of the Ivip system
and so can be customised without limit to the desires of each
end-user network.  The end-user network administrator could choose
the frequency of probe packets, the number of vantage points in the
Net from where these probes were sent etc.

The administrator could precisely determine the decision criteria.
For instance, if the link from ISP A occasionally became congested,
they could have the decision process adjusted so it did not change
the mapping to use (expensive) ISP B's ETR instead unless there was
a more prolonged absence of probe packet acknowledgements.

With SHIM6 or the non-Ivip core-edge separation techniques, this
sort of sophistication would not be possible.


>>> Host stack changes take a long time, but apart from that, they're
>>> actually a fairly easy way to solve problems because only a
>>> handful of organizations need to make investments (the host OS
>>> vendors).
> 
>> Yes, but there is a huge hurdle to overcome if the new system only
>> provided benefits when communicating with another upgraded system.
> 
> True.

OK.

>>>> We just want the Internet to keep working well with minimal
>>>> disruption.
> 
>>> Then switch to IPv6 before we run out of IPv4 addresses...
> 
>> That is rather disruptive, since there's almost no-one at home on
>> the IPv6 Internet.
> 
> There will be if everyone updates... The disruption will come to IPv4 as
> it runs out of addresses. At that point there will be people who are on
> v4 and not of v6 and people who are on v6 but not on v4. That's not
> good. People who add v6 today can keep running v4 so there is no
> disruption.

I don't think there is going to be a sudden crunch with IPv4 - just
an increasingly high economic value placed on address space,
together with more economic pressure to use it efficiently, and to
trade it back and forth so none of it goes unused.  I agree it can't
go on forever, but I think we are a long way from a situation where
IPv6 is attractive to most end-users.  The exceptions would be
cell-phone end-users, who have built-in applications and who can
access selected IPv4 things via HTTP gateways etc.

I think that if mobile operators do choose en-masse to give
cell-phone users public IP addresses (they may not, wanting to keep
them on NAT and make it hard for them to run their own VoIP
programs), then this will be the first substantial adoption of IPv6.


>> Our task is to come up with a system which solves the routing
>> scalability problem which ordinary folks will adopt for their own
>> immediate reasons - since most of them won't know or care about
>> routing scalability.
> 
> Right. But we should expect users to make _some_ changes to their
> routine if this is really necessary to make our efforts succesful. Maybe
> renumbering once a month is asking too much, but rejecting a solution
> just because it does some stuff in a decentralized manner is the other
> extreme. (I'm not arguing that shim6 solves everything for everyone,
> though, just that IPv6 PI was adopted way too easily.)

OK - the debate continues!

  - Robin


--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg