Re: failure detection

[Paul Jakma <paul@clubi.ie>]

Hi Erik,

There's one other question to consider, which I've not really had a 
good answer to (I think I asked Marcelo already ;) ) AFAIR.

How is this going to interact with local site routing policy? The 
ability to express policy in routing typically depends on routing 
being able to, by and large, affect source-selection. Ie that 
applications (or things above the output-IP layer) do not 
unneccesarily go decide what source to use.

To give an example, here's another case of a PA-multihomed site which 
I had (the misfortune[1]) to setup:

A site bought an expensive, and slow, fixed circuit, in order to 
accept inbound SMTP primarily, they also procured a faster DSL 
line[2], but not suitable for hosting inbound services on. The two 
links had different costs, one was very expensive compared to the 
other, in several ways:

- the fixed circuit had much lower bitrate
- the fixed circuit, amazingly, was metered by MB (DSL wasn't)

I had to fix their site when they discovered things didn't "just 
work" (ingress filtering). They wanted all possible traffic to go 
over their cheap/fast DSL obviously.

Solution was fairly obvious:

- source route for the fixed-address range via the slow fixed circuit
- default via the DSL[3]

This worked because:

Other than responses to SMTP connections, all outbound services (SMTP 
outbound, Web proxy, etc.) used unspecified addresses.

Fairly obvious so far :), bear with me.

My question is, how would source-remote n^2 probing shim ever work 
nicely with such multi-cost setups? If shim just happens on the 
source that belongs to the expensive link, it literally could cost 
money, or high-cost for some other definition of cost.

IIRC Marcelo's answer was that shim6 should simply take routing into 
consideration when picking the source(s) to try send probes with.

But that seems very much (to me) either like reinventing the whole 
SAS process which already is implemented elsewhere in the kernel. 
Alternatively, it will require lots of extra layering violations in 
shim6, with shim asking "what sources do you think I could use?" - at 
which point, you could surely just leave the source unspecified and 
let the mechanisms which exist anyway do the work you're going to 
replicate.

Ie, with shim doing probing using different sources:

- Just picking all sources is going to make shim6 really annoying for
  administrators who have inequal-cost site links.

- Manually configuring shim to prefer certain sources would be
  tedious and simply replicating what the kernel already has in its
  routing table (possibly installed dynamically without manual
  intervention)

- Specifying mechanisms for shim to interact with kernel to obtain
  the list of addresses which fit with routing policy and shim's
  desire seems to me to:

   - be a question that can't be answered precisely other than
     "these are the sources I would use for the /best/ route(s)"

   (shim wants to try more than just that if doing n^2 probing, but
    which ones? If just the best, well that can be achieved simply
    by /not/ trying to pick different sources and just not
    specifying an address and hence letting kernel do what it would
    do anyway. If "the best, plus a few more" - which few more, the
    kernel can't tell you from its routing table, do you want the
    lowest cost and the next highest? but what if that's the
    expensive one? there's no way to know.)

  - will require more hooks and 'APIs' between different layers of
    the networking stack. IMHO, that kind of thing can leads to shim
    being hard to, and taking longer to:

    - specify
    - implement

    (particularly when there's talk of other APIs between shim and
     ULP's. I'd strongly suggest following Spencer Dawkin's advice
     and not requiring such, or even investigating this
     much initially.)

--paulj

1. For reasons we won't go into.

2. The site during procurement were told by their provider that fixed 
addresses were only available with "business" class fixed-circuits, 
and not possible with DSL. So they got a quite expensive 128k link 
just for SMTP. sigh.

3. Failover wasn't needed, because they had two VPN links back to 
head office, one using the DSL link, the other the fixed circuit, 
with the former VPN tunnel as the preferred route - keepalives were 
provided either in the tunnel, or dynamic routing protocol ran over 
it - dont remember. If DSL went down they still had their email 
delivered to them, but to the 'expensive' MX for them via the VPN.

regards,
--
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Hmmm, look at those eyes.  He's trying to hypnotize me, but not in the
good Las Vegas way.

		-- Homer Simpson
		   Mountain of Madness