Report from the Python Language Summit

> So how many characters are in composite symbols?
What does that have to do with the fact that bytes are not text/strings/characters?

> The only sane and modern way to do Unicode is UTF-8.
Regardless of whether this is true or not, there is a lot of data in all kinds of encodings, and developers had better think about which one they are going to use when reading that data.

"So how many characters are in composite symbols?"

Characters? I'd say one. I can definitely say (as far as I understand this anyway) that it's one grapheme and one or more code points.

Perl 6 will deal with strings as sequences of Normalized Form Graphemes (NFG). There's a very interested blog post about what this means:

https://6guts.wordpress.com/2015/04/12/this-week-unicode-...

I guess the only two sane ways of handling Unicode are:
* be completely agnostic and treat stings as opaque sequences of bytes, or
* go all in and work with graphemes whenever possible.

I'll just tell all the existing systems out there to use UTF-8, even if they don't. I'm sure I can find a way to jam all of Unicode onto the Adafruit-based display board my Python code is talking to: it has a whole 64K of flash and 2K of RAM! I'm sure I can fit glyphs for all of Unicode in there and still have space for everything else it has to do!

No, not everything can use UTF-8, even in an ideal world. And such systems will *always* use different encodings, so to talk to such systems Python's enforced conversion is extremely valuable. And even when you're not, and the system you are talking to uses UTF-8 or some other Unicode variant, the enforced conversion is *still* valuable because it forces you to think about what encoding is in use, and amazingly often it's not straight UTF-8, or it's UTF-8 with extra requirements such as needing to be canonicalized or decanonicalized in a particular way or "oops we didn't say but experimentation makes it clear that $strange_canonicalization is the only way to go". (I have seen all of these on real systems, along with people claiming UTF-8 but meaning UCS-16 because they didn't know there was a difference, and vice versa -- and, in the latter cases, cursed them.)

Python says that its internal encoding is Unicode, but the nice thing about the enforced mapping to bytes to get it out anywhere else is that good code need not rely on this at all. As long as you always transcode to/from bytes when leaving the Python world, you can *completely ignore* what internal encoding the thing is using (or, rather, use the Unicode stuff like properties etc as needed inside your code, in the happy knowledge that this will not affect transfer to external systems at all, no matter what encoding they use, as long as a codec exists. And writing codecs is *really* not hard, at least not if you don't care much about performance, e.g. for one-off mappings where no codec exists in Python yet :) )

> Because the JSON spec requires that strings must be unicode?
Yes, but the reality outside is a little bit different.

> Can't you just write a custom object_hook function to pass to the decoder to solve your problem?
No, 'encoder' parameter is ignored in json.loads and everything else already gets decoded strings.

We simply switched to a third-party library instead.

> You can decode("UTF-8", "ignore") or something else to "coerce" it to some form of text though I really do value the sanity of that.
I think it will still be broken. There's a workaround that simply stores binary bytes in the lower byte of UCS-4 codepoints and it sorta works.

I'd love to fix these data sources, but they're out of my control. The vendor knows about it and they plan to base64 binary data in the future, but for now I have to work with what I have.

> You can just use byte processing for everything there and Python 3.4
Not exactly. Most of the built-in library can be used with byte sequences, but third-party libraries are often too careless.

I've fixed tons of code like this:

>def blah(p):
> if fail_to_do_something(p):
> raise SomeException(u"Failed to frobnicate %s!" % p)

It mostly works as is, but occasionally it doesn't.

So how do you put raw bytes into JSON then? How do you reasonably deal with people putting Latin-1 into comments on a website accessible via an API using JSON? Do you use an array of codepoints? Bytes? Personally, I like Python 2's json module which gives you a unicode object for UTF-8 and str for anything else (or pure ASCII). The problems there, however, come from other places such as subprocess.Popen.communicate choking on non-ASCII (so unicode object need encoded, but Latin-1 needed casted to bytes…which is the same as str, but still required for some reason), unicode strings can't be formatted with other unicode objects, but str can be (WTF logic is that?), and other pitfalls (of course, hidden until runtime, lucky me). It would be nice to move to Python 3 which seems like it fixes these problems, but this makes it sound like they just moved the ball around in some kind of shell game.

It is still broken with python 3 when I tested it about 2 or 3 months ago. It was either python 3.3 or python 3.4

I have a directory which is read just fine with python 2.7, but skips files with python 3.

Silent failure is always a STUPID idea.

If Python3 really is silently ignoring invalid filenames then that should be marked as a critical flaw.

The real world is not a perfect place with perfectly encoded strings.

Why would they be garbage?

Mission critical is not hard real-time. Mission critical means the mission fails when the program fails to read files in a directory.

Some failures are just annoying, but this one is not for me.