[idn] IDNA interoperability failures, once again

["D. J. Bernstein" <djb@cr.yp.to>]

Adam M. Costello writes:
> When an application is displaying header fields to the user,

Define your terminology. How does a UNIX program figure out whether it
is displaying something ``to the user''? (Anyone who doesn't understand
this question should read ``The UNIX Philosophy,'' by Mike Gancarz.)

One of many examples: Surely you are familiar with the ``dig'' program.
The output of ``dig soa berkeley.edu'' includes all the SOA information
for berkeley.edu. This output might be

   * piped to the ``less'' program, a generic text viewer that (usually)
     displays the results to the user; or

   * piped to a script that extracts the hostmaster address and invokes
     the ``sendmail'' program to send email; or

   * saved to a file, later viewed with ``less''; or

   * saved to a file, later piped to that script.

Yes or no: Should ``dig'' convert its results from your 7-bit encoding
to the local character set? (Assume the LANG=en_US.UTF-8 locale, so that
the conversion is at least theoretically doable.)

An answer of ``no'' means that you're failing to display the desired
non-ASCII glyphs to the user. In other words, you're missing the whole
point of IDNs.

An answer of ``yes'' creates an interoperability failure, just like the
copy-and-paste interoperability failure that you've been ignoring for
months. The mail will bounce.

If you try to dodge this question by drawing some silly distinction
between ``dig'' and other programs, I'll start asking about those other
programs. The nmh mail-handling system leaps to mind.

---D. J. Bernstein, Associate Professor, Department of Mathematics,
Statistics, and Computer Science, University of Illinois at Chicago