Re: Multihoming by IP Layer Address Rewriting (MILAR)

[Iljitsch van Beijnum <iljitsch@muada.com>]

On Tue, 4 Sep 2001, Peter Tattam wrote:

> As the address exchange is controlled by the sender, I see no reason why a TCP
> stack should not use a cached value while any sessions exist in the established
> state.  Any other states would need careful thought about whether the cached
> information s reliable enough.  Of course the API should be able to control the
> desired behaviour if necessary.

Agree.

> BTW, web browsers these days are getting a little smarter by using keep alive
> HTTP connections - typically no more than 5-10 smultaneous connections.  I
> recently added support for them into my web server.

I know the capability exists, but I have no idea to what extend it is
actually used...

> so what are you suggesting? new RR's to list alternative addresses?  Would this
> not be similar to reinventing AAAA records with priorities?

In my proposal there would have to be a new type of resource record if the
DNS is to be used for selecting alternative addresses. (Note that a host
may have multiple "regular" addresses, each with one or more
"alternatives".)

I originally wanted to bind these RRs to the ip6.int zone, so they are
easy to find with the "regular" address as the key. But if we hijack the
DNS resolver library, we could find this information during the forward
lookup stage.

So this would not be the same as AAAA records with priorities, which seem
like a good idea to be able to traffic engineer and load balance.

> Also, how do you plan to deal with stale DNS cache issues?

Not a problem: the DNS only lists addresses that might be valid, but does
not convey any actual reachability information. The host has to discover
if an address is reachable on its own. If proper host unreachables are
sent, this can be done both easy and fast.

> > > It doesn't tell the other end what addresses you might be coming from though.

> > Good point. But this information could easily be carried in an IP option.

> surprise surprise - that's what I'm proposing :)

But you are proposing this in both directions, which is somewhat more
problematic: you can only send an IP option if you can send period... But
you could try each of the different addresses, of course.

> > On hosts the passive approach would be appropriate. Changing to the next
> > address periodically and keeping a copy of the RTT values for each address
> > would probably do it. Maybe for routers that rewrite many sessions, the
> > overhead of sending some extra packets is worth it.

> > If we want to be really ambitious, we could come up with a TCP
> > implementation that could actively take advantage of having multiple
> > destination addresses. This TCP would load balance the traffic over the
> > different destination addresses, exploiting different bandwidth and delay
> > properties of the different paths.

> Umm.  Nope.  I don't think this will fly with the TCP people if done on an
> arbitrary basis.

Are you commenting on the "really ambitious" part or the remark above it?

> It's sure to screw around with RTT averages and muck up the retry mechanisms.

Cycling through the list of addresses will of course have an impact on TCP
operation (so it shouldn't be done too often), but it may be worth it if
you discover a faster path.

> You only want to send a packet by an alternative path as a
> last resort unless you are very sure that it will be.

That would be adequate. Better heuristics can be developed later.

> In my recent draft update, I suggested the idea of sending concurrent packets
> to all possible destinations at the start of the syn exchange.  Basically you'd
> have an election to see which path wins & pick that.  The downside is the
> duplicate traffic could send some people ballistic.

:-)  You would trigger all kinds of anti SYN-flood mechanisms. But if you
need many TCP sessions anyway (such as when requesting HTML pages) I don't
see anything wrong with it.

> In a TCP retry scenario
> which would suggest that the path may not be the optimal, you could pull the
> same trick to see if another path turns out to be better.

This would certainly not waste as many resources at the receiving end as
doing it at the time of the SYN. However, in my rewriting mechanism the
sender wouldn't normally know for which packet (to which address) an ACK
is. That would require some more deep TCP magic.

> The issue of load balancing is a thorny one & I'm not sure if any host based
> multihoming is going to be able to do that easily.

I agree that it wouldn't be easy. But since this could probably be
implemented at the sender only, there is still incentive to do it anyway.

> I have to confess I can only
> guess at how load balancing is done with BGP. Perhaps those in the know might
> enlighten me.

In normal multihoming operation there is no load balancing at the session
level. "Load balancing" in this context usually refers to trying to
balance ALL traffic over the different pipes. BGP does support link-level
load balancing when there are several pipes between two ASes: then BGP can
add routes over each link to the forwarding information base so the links
can be used concurrently. Depending on the router, they will be used in a
per packet or per destination IP address round-robin fashion.

> If we consider the possibility of hosts within a site "colluding" in the case
> of a provider path close by being down it might not take very many packets at
> all for most of the hosts in a site to shift their traffic pattern even before
> the individual connections start timing out.  The connectiions could even send
> gratuitous acks ahead of time.  It probably only takes a few to start the
> avalanche, but you would want to be damned sure that it couldn't be triggered
> by an attacker.  Controlled multicast might be a good notification mechanism
> for this.  Hmm.. we have something already in the way of router advertisements
> - why not use them properly.

This could be very cool.

> At the other end, if an ack for sent data in a TCP connection comes via a
> different path, it is a strong hint that the current path might not be the best
> and to start hunting for better paths.  This is kind of advance warning that
> something has happened at the other end.

If the other side has interesting info, why not explicitly say so in an
option?

> This of course assumes that we are talking about symmetric network paths.
> Where traffic comes in via one provider and goes out by another it might be a
> waste of time.

Asymmetric routing is very common in IPv4. However, short (and hopefully
fast) paths are likelier to be symmetric.

Iljitsch van Beijnum