Re: [RRG] How to Incrementally Deploy APT

[Robin Whittle <rw@firstpr.com.au>]

Hi Dan and Michael,

Thanks for your response, in which you wrote:

>> |  APT Island 1  |
>> |     ______     |
>> |    / ISP5 \<=================\
>> |    \__,___/    |            ||
>> |   ___/         |            ||              ________________
>> |  /             | BGP      __\/__      BGP |  APT Island 2  |
>> | |   ______     | Routes  / ISP4 \  Routes |     ______     |
>> | |  / ISP1 \<===|========>\______/<========|===>/ ISP3 \    |
>> | |  \__,___/    |            /\            |    \__,___/    |
>> |_|_____|________| BGP Routes ||            |_______|________|
>>   |     |                   __\/__                  |
>>    \ ___|___               / ISP2 \              ___|___
>>     / Site1 \              \______/             / Site3 \
>>     \_______/                 /\                \_______/
>>                    BGP Routes ||
>>                            ___\/__
>>                           / Site2 \
>>                           \_______/
>>
>>  1 - Site1 can multihome with ETRs in ISP1 and ISP5.
>>
>>  2 - Site1 can't multihome with ETRs in any other ISPs,
>>      including ISP3.
>>
> 
> Point #2 is incorrect.  Site1 can multihome with ISP3.    
> 

> Perhaps this is your concern; the traffic engineering preferences
> that Site1 expresses to ISP3(see draft for details) won't apply
> to ISP1 and ISP5. This is true, since the scope of TE preferences
> do not extend between APT islands. However, packet delivery works
> just fine.

OK - ISP1, ISP5 and ISP3 all advertise a prefix in BGP for Site1's
EID space, so packets from non-upgraded networks will go to the
nearest such ISP.  This is an anycast approach, which is fine, since
the end-point is the host in Site1.

>>  5 - Therefore, APT has very limited flexibility in serving the
>>      needs of end-users who have less than 256 IP addresses,
>>      unless all APT sites join up into a single "APT Island".
> 
>>      This puts APT at a distinct disadvantage compared to LISP or
>>      Ivip, which can handle end-user networks (EID prefixes AKA
>>      micronets) as small as a single IP address, and enable the
>>      end users to use any ETR in any ISP in the world,
>>      irrespective of whether that ISP has ITRs and without any
>>      restriction on which ETRs are used by end-users with
>>      neighbouring micronets, or micronets in the same Mapped
>>      Address Block (BGP advertised prefix containing typically
>>      hundreds or thousands of micronets).
> 

> APT considers edge networks to be on the granularity of ASes that
> do not show up in the middle of an AS path.  Such ASes would have
> >256 IPs. APT is not designed to allow home users to connect
> directly to an APT island and have their packets tunneled.

OK.  While LISP, TRRP (I think) and Ivip enable micronets (EID
prefixes) down to very small sizes, APT's EID sizes are constrained
by the current filtering limits of the BGP system.

This makes me think that APT would be a lot less effective at
enabling better utilization of IPv4 than the other proposals.

> Thus you are correct in saying that during APT incremental
> deployment, a site with a PI prefix longer than /24 will not be
> reachable outside the island (unless the rest of the /24 is in
> the island, as you pointed out).  Keep in mind that today's
> Internet does not support this either.

When, for instance, 80% of ISPs use APT and form a single island,
does this mean you could have edge networks with smaller address
assignments than the 256 lower limit imposed by your need to be
compatible with BGP's /24 limit?

I think that any such smaller edge networks would not be reachable
without the problems I raised from the other 20% of networks.


> You claim that APT is at a disadvantage over LISP or Ivip.
> Correct me if I'm wrong, but don't these proposals have similar
> problems during incremental deployment?  Let's say that I am a
> lisp/ivip customer site with 111.111.111.128/25 and
> 111.111.0.0/16 is being used by a non-upgraded ISP.  How would an
> anycast ITR or a PTR announce my prefix in this case?

Ivip "anycast ITRs in the core/DFZ" advertise Mapped Address Blocks.
 These are /24 or shorter.  A typical MAB might be a /16 or a /14.

It would contain the User Address Blocks (UABs) of hundreds or more
likely thousands of separate end-users.  In Ivip, these are of
arbitrary length and starting position - they are not constrained to
be binary boundary prefixes.

Within each UAB, the end-user defines as many micronets as they
like, down to single IP addresses.  Then they control the mapping of
each micronet to any ETR in the world.

LISP's "Proxy Tunnel Routers" are the same as Ivip's "anycast ITRs
in the core/DFZ".  They advertise a prefix, and the system only
makes sense in terms of reducing the number of advertised prefixes
due to the fact (typically) that each such prefix provides address
space for many end-users, who without LISP/Ivip would either not get
any multihomable space at all, or who would get it by conventional
means of getting an ASN, PI space and therefore by adding more
advertised prefixes to the DFZ.

LISP doesn't have a concept of User Address Block and its micronets
are called "EID prefixes" - which are constrained to be binary
boundary prefixes.


>>  6 - When Site1 uses APT's TE capabilities (I recall this is
>>      part of APT) to load share incoming traffic for its
>>      prefix over the two links from the ETRs of ISP1 and ISP5,
>>      this probably does not mean that packets from Site2 will
>>      be load shared between the two links ISP4-ISP5 and
>>      ISP4-ISP1.  More likely, all the traffic will flow from
>>      ISP4 to one of these, say ISP5.
>>
>>  7 - This would make it difficult or inefficient for the APT
>>      system to somehow load share the incoming traffic over
>>      the two ETRs, since half the traffic would need to flow
>>      from ISP5 to ISP1.  (How would this actually be done?)
> 

> Actually, we wouldn't even expect that ISP5 would forward any
> traffic for Site1 to ISP1, it would just deliver it directly.

OK, so load sharing wouldn't work for traffic coming from an
upgraded network?

> Basically, the priorities and weights that APT uses for TE would
> only apply to senders within the island. 

OK.

> There might be some tricks we could play with BGP to get data
> from ISP4 to go to across the right link based on Site1's
> preferences, but that may not be advisable -- it will likely
> make a bit of a mess in the BGP announcements. Note that other
> proposals have to pick a location in a similar trade-off space.

Ivip doesn't attempt to do explicit load sharing.  If you want to
load share over multiple ETRs, it is necessary to have the incoming
traffic split over multiple IP addresses (or IPv6 /64s) and map each
one to a different ETR.  This may seem clunky, but (for a small fee
per update) you can fine-tune the starting points and lengths of
these micronets, and which ETR they are mapped to, with (ideally)
about a 5 second delay between giving the command and having all the
world's ITRs follow your instructions.


>>  8 - If:
>>
>> <snip>
>>
>>      LISP, TRRP and Ivip have no such problem with islands or
>>      long path for packets, since the ordinary BGP system takes
>>      packets directly from the ITR to the ETR, irrespective
>>      of whether the networks those BGP routers are in have any
>>      upgrades for LISP, TRRP or Ivip.


> Tell us if we're mis-characterizing your example, but it seems
> that the larger point you're getting at is this: by removing the
> intermediate ASes in the APT island from the BGP path, we are
> providing an opportunity for ISP4 to unwittingly send traffic
> along an unnecessarily long path. That's absolutely right.

> However, this seems to us to be fundamentally the same problem
> faced by LISP PTRs or Ivip anycast ITRs in the core -- the
> ITR/PTR/BR/whatever becomes a necessary extra hop in the path
> which may be less than optimal. So we don't agree that other
> proposals avoid this problem.

With Ivip, the destination host is in Sydney, and so it its ETR.

Sending hosts all over the world, in non-upgraded networks, have
their packets attracted to one or more "anycast ITRs in the core/DFZ".

If there is only one of these there will be longer paths, unless
that ITR is on the natural path for packets from the single sending
host and the destination.

The idea is that there are ITRs around the Net.

So if the sending host is in San Diego, the nearest "anycast ITR in
the core/DFZ" is in LA, then the packets go to LA in their raw
state, and are tunnelled to the ETR in Sydney.  Unless there was a
direct link from San Diego to Sydney, then this represents no extra
path length, because without Ivip, the path would have been through
LA anyway.

At the same time, a sending host in a non-upgraded network in
Edinburgh would probably find its outgoing packets being
encapsulated by an anycast ITR in London - which again would not be
a longer path than usual.

If there was a single anycast ITR in the core in Sydney, then all
would be well in our example - there would never be any longer
paths.  But that doesn't work when the end-user decamps to New York
and uses an ETR there.   So there needs to be anycast ITRs in the
core/DFZ pretty widely scattered around the Net for the system to
work with little or no extra path lengths.

The business model for this is discussed somewhat in:

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-db-fast-push-00.html

Basically, the companies how are renting out the micronet space pay
for these anycast ITRs, either by running them themselves or paying
another company who runs these ITRs for many other such Ivip address
companies.  Without a healthy set of such ITRs, the address space
would involve longer paths in many instances.   These companies
might need to charge on volume of traffic handled by the ITRs, not
just the number of IP addresses in each micronet they rent out.

I will reply to the rest of your message later.  I haven't yet read
your discussions with Bill Herrin.

Thanks again for your response.

  - Robin


--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg