[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [RRG] Sprite & IPTM while PMTU probing is in progress



Hi Fred,

Thanks for your response.  You wrote, in part:

> On- and off-list discussions have explored the idea of requiring
> a 2KB EMTU_R on all ETRs and accommodating all 1500-byte and
> smaller packets; even if fragmentation is needed and the inner
> DF=1. This would uphold the "principle of least surprise" to the
> SH, but the issue comes in knowing the EMTU_R of the ETR and in
> avoiding excessive fragmentation.

I don't clearly understand this.  Iljitsch has not been able to
convince me - or anyone else AFAIK - that it it will be practical to
insist on massive hardware upgrades broad enough for the wide range
of location I think ITRs and ETRs need to be located. (Host to host,
Man - Coast to coast. http://www.firstpr.com.au/ip/ivip/tv-ad/)

Maybe some people envisage a more restricted range, or are more
upbeat about the prospects of a global jumboframes and gigabit
Ethernet upgrade.

For the purposes of discussing sprite-mtu and IPTM, I assume that
this upgrade is not practical.


> Since the ITR cannot know the EMTU_R of the ETR a priori unless
> there is some spec that says: "all ETRs MUST configure an EMTU_R
> of at least X bytes", the ITR should not simply fragment the
> outer packets (or, allow the network to fragment them) since they
> could black-hole. 

You and most other people on this list have much more experience in
these matters than I do, but I don't see why fragmenting packets for
a few seconds will lead to a "black hole".

I recognise that fragmentation involves several costs:

1 - More work for the routers at each end.

2 - Two packets rather than one being handled by all
    routers en-route.

3 - More data being sent, due to an extra packet's worth of
    overhead.

4 - Most seriously, I think, a greater chance of loss of the total
    packet due to its delivery depending on two packets, rather than
    one.

    Each packet has some risk of being lost, and the risk is
    a little higher than would otherwise be the case, since the
    two packets take up more time on the wire and more router
    resources than the single original packet.

5 - Fragmentation failures due to 16 bit ID wraparound.  You
    mentioned this being a problem with IPv4 and high packet rates
    with longer delays, causing the wrong packets to be reassembled,
    due to the 16 bit counter  wrapping and them having the same
    sequence number.

Tacking point 5 first, in message 608, quoting Iljitsch, I asked
whether this could be largely resolved by a shorter time window for
reassembly in the ETR:

 - - - -

> I don't have any references, but in short, the issue is that you
> have a 16 bit ID space with a reassembly timeout of something
> like a few minutes. This means you can only send 65536 packets
> during that "few minute" window or you'll incorrectly reassemble
> fragments from different packets if you lose a fragment. This is
> especially problematic if the fragmented packets belong to a
> tunnel because in that case the IP source/dest addresses are
> always the same.

In a fresh system such as an ITR-ETR scheme, perhaps a workaround
for this would be to set the maximum reassembly time at the ETR to
something very much shorter, such as 1, 2 or 3 seconds?

That could be a pain where the ETR is an additional function in a
server or router with a pre-existing TCP/IP stack.  Then it would
probably not be possible to shorten the time just for the ITR to ETR
packets.

 - - - -

Can anyone comment on the prospects of this being an acceptable
solution?

In an ITR-ETR scheme, we probably don't want to sit around waiting
for packets which somehow get lost and arrive more than a second
late via some circuitous route through Manangatang.

I will read:

  IPv4 Reassembly Errors at High Data Rates
  http://tools.ietf.org/html/rfc4963

The other points don't, to me, constitute a "black hole" or a
serious problem.  They seem to involve marginally greater efforts,
marginally less efficiency and marginally greater chances of the
final packet not being delivered.

To me, this is a far better thing to do, for a few seconds or so
into a potentially long flow, than dropping packets.


> There are also other factors to consider, including that the ITR
> may not have ultimate control over the setting of the ip_id. And,
> the ETR may not be able to receive non-initial fragments in the
> first place. (These factors can be mitigated by the placement of
> the ITR and ETR in some use cases, however.)

I don't clearly understand these points.  Can you explain them further?


>> A "long" inner packet is one which, once encapsulated, would
>> exceed the ITR's current best estimate of the PMTU.  This would
>> initially be a default such as 1280 bytes.
>> 
>> If this default value is above the minimum for the protocol,
>> eg. 576 for IPv4, then this value of PMTU to the "core" of the
>> net must be available to every ITR and ETR and would be part of
>> the specification of the ITR-ETR scheme.
> 
> The pathMTU cannot be known a priori; all that can be known a
> priori is the EMTU_R of the ETR if there is a specification for
> the minimum size.

Yes.  I envisage some value around 1280 - at least something so that
a 1500, 1520 etc. byte packet would only be fragmented into two packets.


>> The default value would be replaced by a higher value once the
>> probe process was complete.  The pattern would be something
>> like this, assuming the SH's initial idea of PMTU to the
>> Destination Host was 1500.  I will assume an ENCAPS overhead of
>> 20 bytes (as with IPv4 Ivip, though other ITR-ETR schemes have
>> higher overheads) and that all ITRs and ETRs are located so
>> they have an MTU of at least 1280 from the DFZ.
> 
> Do you mean to say "pathMTU" or "linkMTU"? In terms of "pathMTU",
> do we need to consider links with configurable linkMTUs that
> might have either mis-configured or overly- conservative values?

I don't understand this clearly enough to respond at present.

(My example is not in quotes - just your response.)

   Inner      ITR action on           SH's idea  DH gets
   packet     outer packet            of PMTU
   length     following encapsul-     to DH
              ation of inner packet

                                      1500


   200       Send outer packet -      1500       The packet
             the length is less than
             1280.

  1500       Fragment packet and      1500       The packet
             commence probing                    (Less efficient
                                                 and more error-
                                                 prone tunnel with
                                                 2 packets instead
                                                 of 1, but this is
                                                 only for a few
                                                 seconds, I hope.)

  1500       Fragment packet and      1500       The packet
             continue probing

             ... etc.

> This could black hole and appear as congestion-related loss to
> the SH.

As noted above, I foresee only marginal efficiency and reliability
impact, where you see unacceptable packet losses.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

  1400       Send outer packet -      1500       The packet
             the length is <= 1460.
             (This length would not
             necessarily be sent - it
             is just to show that the
             ITR will now send longer
             packets without frag-
             mentation than before.)

  1500       Drop the packet and send
             the SH a PTB message
             with value 1440.         1440       Nothing, but the
                                                 ITR is usually
                                                 close to the SH,
                                                 and it doesn't
                                                 take long for...


> Depending on the placement of the ITR, this PTB might not make it
> back to the SH.

If there is a filter blocking PTB messages from the ITR to the SH,
then I think ordinary, non-RFC4821, PMTUD would be clobbered anyway
- with or without an ITR-ETR scheme  This situation would not
persist, I think when an ITR-ETR scheme was widely deployed, as I
discussed in:

  http://psg.com/lists/rrg/2007/msg00636.html


  1440      Send the outer packet -  1440        The packet
            the length is <= 1460                (Now the tunnel
                                                 is handling
                                                 optimal length
                                                 packets.)

>> This pattern would continue unless the ITR, with periodic
>> probing, decides that the PMTU is less than 1460 (it might do
>> this quickly if it got a PTB message from a router in a new,
>> more MTU-challenged, path to the ETR), and if the SH sends a
>> packet which would be too big for the new lower value of PMTU.
>> Then the ITR would send another another PTB message to the SH,
>> with a lower value than 1440.

> There is not strictly any periodic probing needed to detect 
> pathMTU reductions, since the data packets serve as virtual 
> probes. The data packets will be lost and might be considered by
> the SH as congestion-related loss if the PTB can't be translated
> by the ITR and sent back to the SH. 

In my IPTM proposal, the ITR would only get PTB messages from
routers in the tunnel if the outer source address was that of the
ITR.  This would be the case if IPTM was applied to LISP, eFIT-APT
or TRRP.  Ivip uses the sending host's address in the outer header,
so the ITR would never get a PTB message if the PMTU to the ETR
suddenly became lower than what it had assumed, based on previous
explicit probes acknowledge by the ETR.

This would cause a black hole for all packets which, once
encapsulated, were longer than the new, lower, PMTU between the ITR
and ETR.  To guard against this, the ITR needs to periodically probe
an ETR to which it is continually sending long packets.  There could
be other techniques, such as an ITR-ETR protocol which enables the
ITR to receive acknowledgement of packets it sends, but this gets
pretty complex, and I am hoping to avoid such things.

> But, the ITR will be able to return the correct PTB when the SH
> retransmits. This is the same as for sprite-mtu.

Once the ITR has a correct idea of the new, lower, PMTU to the ETR,
it can drop (IPTM) or send (sprite-mtu) the packet, and generate a
PTB to the SH when the SH next sends a packet which, once
encapsulated, would be too long for that new PMTU.


>> Alternatively, occasional probing by the ITR might discover a
>> higher value of PMTU to this ETR, and the SH could discover
>> this increase by trying its luck with a larger packet - and
>> either having it accepted, or rejected with a PTB containing
>> the new higher value, minus 20.
> 
> SHs that don't implement RFC4821 will have to wait for a long
> time before trying a larger packet (RFCs 1191 and 1981 say 10min,
> I believe). 

Indeed.  This is why my critique of sprite-mtu seems to be
important.  As I understand it, the SH first gets a rather low value
in the PTB message from the ITR, at a part of the exchange at which
an IPTM ITR would have simply fragmented the packet without any PTB,
and commenced or continued probing the ETR.

If it takes at least 10 minutes for a non-RFC4821 compliant host to
try sending larger packets, then this is a long time for the
communication to be restricted to the shorter packets.

I am not assuming widespread adoption of RFC4821 at any time.  It
looks really complex to implement, involving applications and the
TCP layer communicating with a new function in the OS in ways which
were not originally part of the protocol stack.   Writing all this
code, for marginal immediate benefit, and then trying to debug it in
all its possible combinations of applications, live network settings
etc. sounds really, really, complex.


> SHs that implement RFC4821 can retry end-to-end probing more 
> frequently than that, since loss of a probe does not
> expose data to silent loss.

Yes.


>> I think Sprite will fragment outer packets, as does IPTM, 
>> irrespective of the DF flag of the inner packet.
> 
> That is true.

OK.


>> The criteria for which IPv4 outer packets are fragmentable is
>> complex (5.6.4).
> 
> That text borrows from the tunnel MTU and fragmentation 
> discipline set forth in RFC4213, Section 3.2. But, I don't know
> what you mean by complex?

What I meant to write was that when meant I tried to read this and
the rest of the ID in order to answer the questions I tried to pose
in my second example, I couldn't understand it clearly enough to
envisage how your proposal would operate.

My questions were something like:

  Does the ITR first send a rather low value in the PTB message (for
  the first "long" packet)?

  While probing is taking place, does the ITR generate PTBs for
  packets which are too long for its currently too low PMTU
  estimate (and sends the packet too, without fragmentation), or
  does it fragment them, like IPTM?

I think the answers are:

  Yes.

  The ITR generates PTBs and sends the packets holus bolus.


>> I am not sure how Sprite handles "large" outer packets while it
>> is probing.  Does it fragment them as IPTM does?  Or does it do
>> the following, which is the same as what it does for a "long"
>> packet once the PMTU has been reliably ascertained:
>> 
>> (5.6.5) "... admits the packet but also sends a PTB message
>> ..."
> 
> It does the latter; this borrows from RFC2003, Section 5.

OK, without carefully reading RFC2003, which I would if I had more
time right now, I understand that the ITR both sends the complete
outer packet, without fragmentation, and sends a PTB message to the SH.

>> It seems strange to me to send the packet (unfragmented, I
>> assume) while also sending back a PTB message to the sending
>> host.  Wouldn't this cause needless traffic and/or confusing
>> signals to the SH if the outer packet does in fact arrive at
>> the ETR and therefore the inner packet is delivered to the
>> destination host?
> 
> To the SH, it would appear that there is a router on the path
> returning inaccurate information. This can happen already today,
> since routers can be misconfigured, and spoofed PTBs can be sent
> from any node in the network.

It still seems strange, confusing and inefficient to me.


> SHs that implement RFC4821 should not have a problem 
> deconflicting the (suspect) PTB information from (authentic) 
> end-to-end feedback from the DH, but should benefit from the PTB
> info when the actual data is not delivered to the DH.

Yes, but I am assuming that none, or few, hosts will implement
RF4821 any time soon.


> ITRs can help the situation by sending sprites of, e.g., 1500
> bytes into the tunnel early in the process so that most if not
> all SHs that use the tunnel will see a 1500 byte or larger MTU.

Does "early in the process" mean when only shorter packets have so
far needed to be tunneled to the ETR?

If so, then the ITRs could be generating large volumes (in bytes) of
probe packets in response to only small traffic flows, and to some
flows which never in fact require PMTU knowledge, since the flows
never actually use long packets.


>> Here I will assume IPv4 only, with 1280 bytes for the default
>> PMTU for every ETR the ITR has not yet probed.  I will also
>> assume an encapsulation overhead of 20, although this would
>> typically be higher for Sprite and non-Ivip ITR-ETR schemes.
> 
> I don't understand "higher for sprite-mtu"?

This was a low-key aside.  In my second example, trying to explain
how I thought sprite-mtu might work, I kept the same 20 byte
encapsulation overhead I used in my first example, which was for
IPTM, assuming Ivip's 20 byte overhead.

Other ITR-ETR schemes, and I guess most other tunneling schemes
sprite-mtu would be applied to, have higher encapsulation overheads,
I think.


>> If the ITR sends a PTB message to the SH when the first packet
>> (or multiple packets) length exceeds the default PMTU value and
>> then, after probing, decides the PMTU is 1480, then I am
>> concerned that the SH would get contradictory values in these
>> PTB messages.
>>
>> At first the SH would be told to send packets no longer than
>> (1280 - 20 = 1260) and later, it would be told to send packets
>> no longer than (1480 - 20 = 1460).
>
> Note: in the next draft version I would like to rewrite the
> second bullet of Section 5.6.4 as:
>
> o  for IPv4/*/IPv4 tunnels, 'pathMTU' is less than MIN(EMTU_R,
>    1280+ENCAPS) bytes and the inner IPv4 packet is no larger than
>    MIN(EMTU_R-ENCAPS, 1280).

The current version is:

  o  for IPv4/*/IPv4 tunnels, 'pathMTU' is less than MIN(EMTU_R,
     1280) bytes and the inner IPv4 packet is no larger than
     MIN(EMTU_R, 1280) minus ENCAPS.  (When EMTU_R for the TFE is
     not known, 576 bytes must be assumed.)

OK.  My eyes are glazing over right now.  I will probably be able
to understand this in its full context in the future.


>> But if the SH took notice of the first PTB message, it probably
>> wouldn't send any longer packets which would trigger the
>> second. So, if my understanding of Sprite is correct, the SH
>> would experience something like this:

   Inner      ITR action on           SH's idea  DH gets
   packet     outer packet            of PMTU
   length     following encapsul-     to DH
              ation of inner packet

                                      1500


   200       Send outer packet -      1500       The packet
             the length is less than
             1280.

> Probing can start here also.

As I noted above, at this point the ITR can't reasonably assume that
it will need PMTU information to the ETR.  It is just a short packet.

Perhaps smart ITRs could have heuristics to help them decide when to
start probing earlier, based on prior statistics (of all sorts of
aspects of packets that the ITR might analyse) of shorter packets
often being followed by longer packets.  But that is not something I
would require of an IPTM ITR.


  1500       Send packet and                     Probably nothing
             commence probing                    - however, if the
             Send PTB with value                 packet was a little
             1260                     1260       shorter, such as
                                                 1440, then it may
                                                 arrive at the ETR
                                                 in one piece,
                                                 despite the SH
                                                 being told it was
                                                 too big.

> With the note above, the returned size would be 1280; not 1260.
> Also, I'm not sure as to the "probably nothing" as linkMTUs
> increase above 1500 (perhaps someone could send the IEEE
> reference that proposes the increase for 802 linkMTUs).

OK - I guess in the future it might be worth trying longer than 1500
byte packets to the ETR.

My current IPTM proposal is not aimed at optimising the
opportunities for doing so.  I will think about this whenever I
revise it.

             SH breaks the message
             into smaller packets
             and retries:

  1260       Send packet and          1260       The packet
             continue probing

             ... etc.

             Probing complete:
             PMTU to ETR decided
             to be 1460.

> By probing, do you mean by the ITR or by the SH? 

I meant the ITR sends sprite probes to the ETR.

> I am assuming that SHs will begin using RFC4821 and will
> probe the path for themselves independent of any probing
> done by the ITR.

I am not assuming hosts will be any different than they are today.
As far as I know, few, if any, implement RFC4821.

If you are assuming this, I think it would be good to make it an
explicit condition you are designing sprite-mtu to function within.


  1260       Send outer packet -      1260       The packet
             the length is <= 1460.

  1260       SH would probably keep              The packet -
             sending packets of                  but more and
             length <= 1260.  Unless             shorter
              the SH was pushy, it               packets than
             would never discover the            the ITR-ETR
             PMTU it could use was               tunnel can
             in fact 1440.                       handle.

> IMHO, SHs that use RFC4821 can be "pushy" within reason.

Yes, but I think it will be a long time before there are many such
hosts.



> I would like to add one other note about the 1280. That number
> comes from the SHOULD in RFC4213, Section 3.2.1. The reason I am
> taking the SHOULD is that there can be additional encapsulations
> on the path between the ITR and ETR (e.g., tunnel-mode IPsec) and
> we don't want to cause fragmentation for those, either. If the
> ITR and ETR are arranged such that there will be no additional 
> encapsulations on the path (and the ITR has a way of knowing
> this) then the spec could use the RFC4213, Section 3.2.1
> "configuration knob" to push the 1280 up to as much as 1480 or
> perhaps even 1500. I would not want to go any higher than this,
> since it could involve excessive fragmentation resulting in
> undetected data corruption. 

OK - I won't try to chase this up right now, but thanks for the
pointers.

> Maybe I should add something about this to the spec?

Yes, I think the more explanation of where numbers like 1280 come
from, the better.

Thanks again for this detailed discussion.


  - Robin     http://www.firstpr.com.au/ip/ivip/pmtud-frag/




--
to unsubscribe send a message to rrg-request@psg.com with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg