[RRG] Lightweight multihoming with TTRs, DSL, HFC, WiMax etc.

[Robin Whittle <rw@firstpr.com.au>]

Short version:      How to use TTRs (originally intended for
                    mobility) as part of a multihoming system
                    for a moderate sized corporate network - in
                    which an expensive fibre link is backed up
                    by a handful of cheap DSL, HFC and WiMax
                    links.


Further to my descriptions of using Translating Tunnel Routers
(identical to ETRs to the rest of the map-encap system) for mobility,

  http://www.firstpr.com.au/ip/ivip/#mobile

here are some thoughts on their use for a light-weight approach to
multihoming.  The following example uses Ivip, but most of it
applies to LISP, APT or TRRP.  With these three, the ITRs are doing
the multihoming service failure detection and restoration decisions
- while with Ivip, the end-user or some multi-homing monitoring
company hired by the end-user detects the failure and restores the
service by issuing a mapping change.

Acme Analysis is a specialised company who prepare and sell reports
analysing a variety of high-tech industries.  They have 150 staff at
a single physical location on the outskirts of a major metropolitan
area.  At any one time, about 10% of the staff are out and about,
presenting at conferences, researching developments in other
countries etc.

Acme has a good Internet connection, via a specially installed fibre
link with a major telco/ISP "BigTel".  Most of the time, it works
fine.  The speed is more than Acme needs and the traffic costs are
most agreeable.

Acme has a contract with some UAS (User Update System) company X -
renting 8 IP addresses in some MAB (Mapped Address Block).  The MAB
is "owned" by X, and it contracts RUAS (Root Update Authorisation
Server) Y to pump mapping changes out to the fast-push network as
described in pages 33 to 48 of:

http://www.firstpr.com.au/ip/ivip/draft-whittle-ivip-db-fast-push-00.pdf

BigTel operates an ETR which delivers all traffic for Acme's 8 IP
address UAB "User Address Block" to the fibre link.  The ITR doesn't
need to know specifically about how Acme has divided its UAB in to 7
micronets.  Acme can change this arrangement at any time, without
notifying BigTel.  It does this by issuing commands to its UAS X,
via a secure web page or some machine-to-machine protocol.

Outgoing packets from Acme's network go out the fibre link, to the
ETR which is configured (along with BigTel's internal routing
system) to forward these packets to their destination, both inside
BigTel's prefixes and outside to the rest of the Internet.

As described so far, this is a single-homed arrangement.  Acme could
choose some other ETR-equipped ISP for its connectivity without
needing new address space and without renumbering its network.

Acme's main public website is handled on a server in a separate
hosting company - so that is not part of this multihoming scenario.
 The address of that server is determined by the hosting company,
and if Acme chooses another hosting company, it will use another
address and change the DNS appropriately.

Acme uses its 8 IP addresses in this way:

  0  }  Micronet A   Mailserver and nameserver.  Secure IMAP server
  1  }               so staff at home and overseas can access their
                     email account.  Also, VPN access to the LAN for
                     staff working at home and when travelling.

  2     Micronet B   HTTPS server for handling all ordering and
                     payment for the reports it sells to the public
                     and to regular customers.

  3     Micronet C   VoIP - Asterisk server etc.

  4     Micronet D   Public address of NAT box for its admin
                     division.

  5     Micronet E   Public address of NAT box for its research
                     division.

  6     Micronet F   Just for the desktop machine of the network
                     administrator

  7     Micronet G   Dogs-body, spare, test address etc.

Normally, these 7 micronets have their mapping set to the address of
the BigTel ETR.

Acme wants to multihome, but it doesn't want to spend a lot of money
to get another ISP to haul fibre to its quasi-rural premises.  Maybe
the fibre will have zero downtime this year, and the next.  But
because this can't be assured, it needs a multihoming arrangement.

Instead of getting another fibre, Acme uses phone lines which are
already in the building and an HFC cable system which runs along the
street to install:

   DSL modem 1  }  With another telco OtherTel.
   DSL modem 2  }

   HFC modem 1  )  With cable company CoAx.
   HFC modem 2  )

Each of these gives a reasonably stable, public, care-of address - a
single IP address.  (It would also be OK if the address was behind NAT.)


This being the early days of Ivip deployment, OtherTel and CoAx
either haven't got their own ETRs, or their ETRs won't work with the
dynamically assigned addresses of the above modems.  Alternatively,
maybe Ivip is ubiquitously adopted, but Acme doesn't trust these
"retail" companies OtherTel and CoAx to run ETRs or to provide
spectacularly sophisticated service.  Alternatively, OtherTel and
CoAx can do this, but but Acme doesn't want to pay for it.

The DSL and HFC services are connected and running all the time, and
the Acme router (or some specially programmed server to do this
task) is capable of setting up 2-way encrypted tunnels to some
remote TTR (Translating Tunnel Router) via these modems.

The TTR is run by International Mobility (InterMob) - the leader in
mobility systems using Ivip and TTRs.  There are other TTR
companies, and Acme might have business relationships with them as
well - in the event that InterMob's TTR system doesn't work.

Acme also hires either InterMob or some other company to monitor its
multihomed system, and to issue mapping change commands if the fibre
link fails in some way.

One day, a back-hoe digging drains for a new housing development
severs the fibre.  Estimated repair time: 3 days.

Within a few seconds, the multihoming monitoring system recognises
that the usual fibre path is not working.  (Acme's system would know
about this within a second or two, but we are not relying on the
Acme system to restore service.)

The multihoming monitoring system has been previously configured by
Acme to change the mapping for the 7 micronets so the traffic for
these micronets is tunnelled to either one TTR, or perhaps to
different TTRs, depending on where the 2-way tunnels have been set up.

The TTR is the end-point of 28 2-way encrypted tunnels established
from the Acme and of these DSL / HFC modem services.  There is one
tunnel for each micronet from each care of address of each modem.

(There could be a second TTR with the same arrangement, perhaps of
some other TTR company - so Acme is diversified in terms of access
network and TTR company.)

The tunnel and its management protocol is pretty fancy (I am yet to
describe it) and while it may be an IETF-standardized protocol, for
our purposes we will assume it is a proprietary protocol devised by
InterMob, who provide suitable software (or even a box) to its
customer Acme, in order to manage the tunnels and send and receive
traffic along them.

For instance, the tunnel protocol is not a simple TCP link.  It is
capable of piggybacking multiple small traffic packets into one
tunnel packets.  It does this with QoS, and will resend TCP traffic
packets which are lost in the tunnel, but will not resend any packet
it considers to be part of a VoIP or similar UDP flow.  (I will
write more in another message about such a specialised 2 way tunnel
protocol for mobile nodes and multihomed networks talking to TTRs.)

In normal operation, the traffic always flows on the fibre link.
The DSL and HFC tunnels are active and the TTR is ready to receive
packets from them, forwarding them to the net, just as it is ready
to receive encapsulated packets from ITRs which have an inner packet
addressed to one of these micronets.

After the fibre failure, the mapping is changed so each micronet is
mapped to either the one TTR, or to one of the various TTRs Acme's
modem care of addresses have tunnels to.  So service is restored
within 10 seconds of the failure, and most communications sessions
survive this glitch - so almost no-one notices.  This is simpler and
more robust than with LISP, APT or TRRP, where each ITR handling an
existing flow of packets - or starting a flow after the failure -
somehow has to figure out for itself that the packets shouldn't go
to the BigTel ETR, but to a InterMob TTR instead.

The DSL and HFC modems are not as fast as the fibre system -
especially in the upstream.  Nor are they as reliable.

However, Acme is free to change the mapping as it likes, and to use
any TTR it likes (of any TTR company it has a business relationship
with) with any 2-way tunnel from any of its HFC and DSL care-of
addresses.

So while the DSL service is not absolutely 100% reliable, and either
is the HFC service, since they come along different physical cables
and are run by different companies, the chance of them both going
down at the same time is remote enough not to worry about.


The advantage of this use of TTRs and cheap consumer grade
alternative last-mile services include:

1 - Much less expensive than another fibre.

2 - Probably completely diverse physical paths for the fibre, DSL
    and HFC services.  (If they share the same duct or poles, then
    get a WiMax service as well and play the same game.)

3 - Consumer-grade HFC and DSL services are fine, since there is
    no need for those companies to have an ETR, to program their
    routing system for Acme's micronets etc.  Instead, Acme hires
    InterMob's TTRs to handle all that.

4 - While the bandwidth is not ideal, it is fine for doing business
    for a few days while the fibre is repaired.

5 - Since the address space has already been split into micronets,
    these can be mapped to whatever TTRs make most sense, and the
    choice of which HFC or DSL modem the traffic flows over is
    determined by which tunnels were made to the TTR, and how those
    tunnels are managed.


Traffic costs over the DSL and HFC link may or may not be higher
than the fibre costs.  There are additional costs of traffic in the
TTR too - but both of these are only used for a few days.

As always with Ivip, Acme pays a small fee for traffic, depending on
the volume of traffic which burdens the OITRDs (Open ITRs in the DFZ
- previously known as "anycast ITRs in the core") of the RUAS which
handles the MAB in which its micronets are located.  Acme pays for
this for all its incoming traffic, but it is a small fee compared to
the costs of actual Internet access, and the fibre link, from
BigTel.  OITRDs are just servers at peering points, so it is not as
expensive for a packet to go through that as it is for the packet to
be delivered to BigTel's network and then along the fibre link Acme
pays BigTel for.

I will write more on the cost of traffic going through OITRDs and
TTRs in another message.

If the TTR concept and its links to the "mobile node" (or in this
case the multihomed network's router) is defined outside the
map-encap RFCs, then the whole concept of the TTR, mobility - and
here the TTR's role in multihoming - does not cause any extra
complexity for Ivip, or any other map-encap scheme, since the TTR
behaves like any other ETR.

It would not be possible to prevent any map-encap scheme being used
to tunnel packets to a TTR.  Nor could any access network prevent
its customers creating 2-way encrypted tunnels to TTRs.

While I depict this system with Ivip, it could be used for any
map-encap scheme in just the same way - except that in the others,
the end-user has to rely on previously defined, complex, mapping
information to allow each ITR to figure out whether to tunnel each
micronet's packets to either BigTel's ETR, or to the one or more
TTRs to which the DSL and HFC tunnels have been established.

  - Robin                http://www.firstpr.com.au/ip/ivip/



--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg