Second shim

[Iljitsch van Beijnum <iljitsch@muada.com>]

I had a conversation with Tony Hain and Daniel Karrenberg yesterday,  
where we discussed the issues with the shim and very short-lived  
sessions such as single packet UDP request/reply interactions.
This ties in with the "failure of the initially chosen locators"  
issue that we talked about. I think it was Daniel who brought up the  
need for a "second shim" that sits between the application and the  
transport protocol that could handle this need.
I think that makes a lot of sense. The current shim is all about  
keeping ULP sessions flowing within the constraint of keeping the  
ULIDs stable. But in a single packet interaction there is no need to  
keep ULIDs "stable": you can just try different source/dest addresses  
and retry the transaction until it works.
I think the current presumption is that the application itself would  
do this. However, for what the shim as it is now _does_ do, the idea  
is that the shim's activities are completely hidden from applications  
and ULPs. So it makes sense to consider the possibilities of allowing  
current applications using current APIs to survive initial failures.
I'm not talking about single packet UDP stuff anymore: for our  
purposes, a TCP connection establishment is pretty much also an  
exchange of a handful of packets. It's only after the session  
establishes and the application starts sending/receiving data that  
TCP and the like are constrained by the requirement that the  
addresses remain stable. However, at this point the shim as currently  
intended can take over so that's not a problem.
It would work as follows. The "second shim" intercepts session  
establishement requests (bind(), connect() and the like) along with  
the sending of UDP packets (sendto(), sendmsg()). The second shim  
then first tries the requested action using the addresses supplied by  
the application and/or RFC 3484 address selection mechanisms. When  
this action times out, the second shim looks up additional addresses  
(see below) and tries all source/dest combinations until something  
works. At that point the whole thing is handed back to the calling  
application and the second shim gets out of the way.
Finding additional addresses is a challenge of its own, but a good  
start would be o look up the destination address supplied by the  
application in the DNS, and then look up all AAAA records for the  
resulting name. Whether these addresses are all present on the same  
host or they are for different hosts implementing the same service is  
of no importance at this point.
There is the slight caveat that hosts will now connect to a different  
address than they thought they were connecting to. A weak version of  
the second shim would leave the destination address untouched but  
only cycle through the available source addresses if the application  
specifies IN6ADDR_ANY (yes, yes, I know, this doesn't exist as such).  
In practice we'll probably want some management hooks that allow the  
administrator to specify whcih application/protocols get which  
treatment.
Thoughts?