🚀 go-pugleaf

On Monday, in article <fg5i5l...@mid.individual.net>
     use...@andyburns.uk "Andy Burns" wrote:

> Simon Turner wrote:
> 
>> it displayed a
>> three-byte UTF-8 encoding in Niklas's malformed message
> 
> The problem didn't start when BAH quoted Niklas, the UTF-8 encoding and 
> the three byte sequence … were in BAH's message that started this 
> thread ...

Have another look.

Note that I'm reading the raw spool, so I get to see the raw message
text without any software trying to interpret it for me.

BAH's original message contained a perfectly valid UTF-8 encoding of the
Unicode character U+2026 (HORIZONTAL ELLIPSIS) -- the three binary bytes
0xE2 0x80 0xA6, i.e. "…" -- in a correctly-formatted MIME UTF-8
message.  This may not have been what BAH was expecting (her
oh-so-clever software turning the first three dots of her four-dot
ellipsis into a separate non-ASCII character), but there was no
*problem* with this until Niklas's poorly-configured slrn repeated that
UTF-8 encoding, but marking the message as having the non-existent MIME
charset "utf8".

Depending on the flexibility of your software in interpreting invalid
charsets, you may have seen Niklas's message containing a horizontal
ellipsis, or two/three bizarre binary characters.  (Incidentally, Scott
Lurndal was the first to reply to BAH's original message, but his
software isn't MIME-compliant and quoted the three-byte UTF-8 encoding
without making any attempt to tell us what the charset was: I see that
Google Groups, which is clearly quite flexible, sees it as UTF-8 anyway
and therefore displays it correctly.)

At this point BAH replied to Dave Garland and J. Clarke, both of whom
had quoted her original message, but validly (Dave's was UTF-8 and used
the same three-byte UTF-8 sequence; J.'s was "ISO-8859-1"[0] and used
the single byte 0x85).  BAH will have seen what she expected in the
quote of her original message: four dots in a row.  (The fact that the
first three of them were a single non-ASCII character is neither here
nor there.)

Then she got to Niklas's message, with its broken MIME charset
parameter, and was confronted by two/three odd characters (which you
quoted above in your valid UTF-8 message[1]); she replied to Niklas,
observing that "I did not type that weird character" (which she hadn't).

The problem *started* with Niklas's message, with its broken charset; it
got amplified when BAH quoted it, because her software converted the
three unknown-charset binary bytes 0xE2, 0x80, 0xA6 into their Unicode
equivalents U+00E2, U+0080, U+00A6 and then put those in her perfectly
valid UTF-8 reply to Niklas.  We all then got to see what BAH had seen
in Niklas's message (ish: see [1] below), and started telling BAH she
had a broken system, when we really should have been lambasting Niklas
for having a broken slrn config. 8-(


[0] Actually Windows-1252, which is a superset of ISO-8859-1: ISO-8859-1
doesn't define 0x80 - 0x9F, but Windows-1252 mostly does (five of them
are undefined); the bytes 0x80 and 0x85 have no meaning in strict
ISO-8859-1, but are the Euro sign and horizontal ellipsis respectively
in Windows-1252.  Most ISO-8859-1 implementations are actually
Windows-1252: Wiki's page on ISO-8859-1 says "For the character encoding
commonly mislabeled as "ISO-8859-1", see Windows-1252." 8-/


[1] The three UTF-8 encodings in BAH's reply to Niklas, which were
almost-but-not-quite repeated by you above, are:

 * 0xC3 0xA2 (U+00E2, "â", LATIN SMALL LETTER A WITH CIRCUMFLEX);

 * 0xC2 0x80 (U+0080, "€", <control>), which your software (and
   others, e.g. Google Groups) seem to have assumed must really be the
   "ISO-8859-1"[0] character 0x80, i.e. the Euro sign, and converted it
   into Unicode (U+20AC, EURO SIGN), which has the three-byte UTF-8
   encoding 0xE2 0x82 0xAC (as appears in your message above);

 * 0xC2 0xA6 (U+00A6, "¦", BROKEN BAR)


-- 
Simon Turner                                    DoD #0461
si...@twoplaces.co.uk
    Trust me -- I know what I'm doing!          -- Sledge Hammer