The future of the Python ssl module
The opening session at the 2016 Python Language Summit concerned the ssl module in the standard library. Cory Benfield and Christian Heimes described some of the problems that the module suffers from and discussed some plans for making things better.
Benfield works at Hewlett Packard Enterprise on HTTP and HTTP/2 for Python and has been involved in the pyOpenSSL project, which is an alternative to the standard library's ssl module. Heimes is a core Python developer and the co-maintainer of the ssl module. He works on identity management and public-key infrastructure for Red Hat.
The ssl module provides SSL/TLS support, Benfield explained. It uses OpenSSL internally and is important so that pip can safely download packages from the Python Package Index (PyPI) for one thing. There are alternatives to ssl, including pyOpenSSL, various wrappers of non-OpenSSL TLS implementations, as well as platform-specific variants for Python running on other frameworks (e.g. Java for Jython, .NET for IronPython).
But the ssl module is "in a dire state", Benfield said, and it has "lots of problems". Much of that comes from the way it has grown and accreted new features over the years. It is now quite complicated. For example, an __ssl.__SSLSocket is wrapped by ssl.SSLObject, which is wrapped by ssl.SSLSocket, which is a subclass of socket.socket.
In addition, the legacy API is "insane", Heimes said. The SSLSocket.getpeercert() call returns an empty dict instead of CERT_NONE when there is no peer certificate. The SSLContext object defaults to not verifying certificates; users must provide a custom factory to get verification. So, using the SSLContext() constructor is a bad way to get a context object, Benfield said. Those are just a few of the problems, he said, there are lots more.
But the ssl module also lacks features. Part of the problem is that it cannot decide if it is a high-level or low-level library, so it does a half job for both. It does not provide access to the verification chain for a certificate (i.e. the chain of signatures and certificates for the signers). Private keys can only be loaded from a file; there is no support for passing in a memory buffer or working with PKCS #11 for using hardware cryptographic tokens. The error messages when certificate validation fails are incomprehensible. And so on.
There is no "general-purpose API", Benfield said, it is really just a wrapper around OpenSSL and is tightly bound to that library. The API uses terms and concepts from OpenSSL to the point that OpenSSL cannot be switched out for a competing TLS implementation. The ssl module is also written in C, so PyPy, Jython, IronPython, and others that do not support C extensions must use something else. Beyond that, ssl in Python is stuck with only using features from OpenSSL 0.9.8, which is "ancient", because that is the latest version of OpenSSL that Apple ships in OS X.
Recently, though, the ssl module has gotten a bit better. Certificate validation by default (from PEP 476) was added and PEP 466 brought several security enhancements to Python 2.7, which has been in bug-fix-only mode for some time. Also, ssl.create_default_context() has been added so that users can avoid the SSLContext() constructor.
Benfield then went into some suggestions for the future. To start with, the standard library and third-party libraries should switch to using SSLContext objects, rather than take arguments for certificates, keys, and other verification options. The Requests module, which Benfield also works on, will start using SSLContext everywhere in an upcoming release, for example.
There is also a plan to create an abstract base class that will allow users to wrap sockets using an SSLContext to produce an SSLSocket for programs that need access to the socket itself. It would be something like a regular socket with two or three extra methods. In addition, a small set of "sane exception objects" would be defined to be used for verification and other failures, Heimes said.
The basic idea is to standardize and clarify the ssl module's API. There are some other things that could be done, but that leads to "a bikeshed we don't have time for right now", Benfield said. But there are a few new features that should help clean things up.
Adding a SSLContext.set_verify_call() would allow advanced users to provide their own verification function; most users "shouldn't touch it", but some do need it, he said. Heimes suggested that a few new types would be added to the module including an X509 class instead of the getpeercert() dictionary, a VerifyContext class to store the certificate chain information, and a PrivateKey type that would allow additional mechanisms (e.g. cryptographic tokens) for providing private keys.
After that, Benfield suggested that there be a feature moratorium for ssl. The module should simply provide the basic features needed for pip and for the standard library. Advanced users should use pyOpenSSL or other alternatives.
The ssl module maintainers would like to move to support OpenSSL 1.1.0, which has a cleaner API, and drop support for versions earlier than 1.0.2. There is a compatibility layer to provide the 1.1.0 API on 1.0.2, but there is still the problem of OS X. It ships an Apple version of 0.9.8 and does not support other versions of OpenSSL. Ned Deily noted that it has gotten "worse than that", as Apple is no longer even shipping the OpenSSL header files in the most recent OS X version.
It is not entirely clear what should be done for OS X. There are some alternative TLS libraries, such as LibreSSL or BoringSSL, that could be used as the basis for the ssl module, perhaps, but have their own sets of problems. For example, BoringSSL is tailored to support only what Google needs, Heimes said.
There is also a question about the root certificate store. Benfield was adamant that Python should never ship its own bundle of root certificates. But trusting the operating system to provide the root store leads to problems on Windows, which will try to fetch unknown root certificates from Microsoft, and other (unspecified) problems on OS X.
There is some level of confusion about what the ssl module is supposed to be, which leads to problems determining what direction it should go. Is it meant to provide building blocks for applications to use for their TLS needs or is it meant to have an API that more or less encapsulates the best practices for TLS use, someone from the audience asked.
Nick Coghlan suggested that it should simply provide enough to bootstrap pip, so that it can be used to install something more advanced. For that purpose, though, a connection provided by the underlying operating system could be used, Barry Warsaw said. But, without a root certificate bundle, that could lead to pip using unencrypted connections, which is insecure.
In the end, it was agreed that more discussions were needed and that the python-dev mailing list would be the right place for them.
| Index entries for this article | |
|---|---|
| Conference | Python Language Summit/2016 |
Posted Jun 4, 2016 0:34 UTC (Sat)
by Darkmere (subscriber, #53695)
[Link]
The same goes for Windows.
Also, the default Wrap function is horrible, so you need to use a context ( Generated with create_default_context, of course) to get proper security for socket wrapping.
Posted Jun 4, 2016 4:00 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (18 responses)
Signed,
Posted Jun 4, 2016 5:46 UTC (Sat)
by noxxi (subscriber, #4994)
[Link] (17 responses)
2.7.9 finally added support for SNI (Server Name Indication). This fixed a lot of problems.
2.7.9 also finally enabled certificate validation by default. This brought problems for applications which so far worked with invalid certificates. But in my opinion this did not introduce a new problem but instead only exposed problems which were already there, i.e. blindly trusting invalid certificates. Applications which explicitly trusted invalid certificates (like for testing) where not affected but only applications which did this accidentily and thus were open to unexpected man in the middle attacks.
Posted Jun 4, 2016 6:54 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (15 responses)
> Applications which explicitly trusted invalid certificates (like for testing) where not affected
> only applications which did this accidentily and thus were open to unexpected man in the middle attacks.
The correct way would have required:
The attitude "but these applications were already broken" is NOT acceptable.
Posted Jun 4, 2016 7:39 UTC (Sat)
by noxxi (subscriber, #4994)
[Link] (14 responses)
I wasn't aware of this. But yes, it looks like that they changed parts of the API too in an incompatible way. This of course should better not happen.
> The correct way would have required: ... small targeted patch to enable certificate validation for 2.7.9, disabled by default and activated by a special environment variable...After some time the default is flipped to "on" with env. variable to disable this behavior.
I don't think this will work. Nearly nobody will set this environment variable and they will only complain after the behavior changed permanently.
I've tried even worse nagging when changing the Perl module IO::Socket:SSL to default validation in 2012/2013. The code issued warnings for about a year that certificate validation is implicitely off by default in the users code and that this will change in the future. Lots of users complained about the nagging without understanding what this meant and often switched off validation completely as a workaround. And surprise: more complaints once the flip to secure defaults was finally done.
Posted Jun 4, 2016 8:01 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (13 responses)
An example from one of our apps - we had an HTTPS server that used certificates originating from a Brazilian state certification authority. The problem was that the root certificate was not available in Ubuntu certificate storage, so that an external watchdog box kept triggering failovers every 5 minutes. Really nice.
> I've tried even worse nagging when changing the Perl module IO::Socket:SSL to default validation in 2012/2013. The code issued warnings for about a year that certificate validation is implicitely off by default in the users code and that this will change in the future.
That was the turning point for me - I stopped writing new Python projects after that (though I do maintain several Py27 projects), switching to Go instead. I no longer trust Python developers to be able to do even basic maintenance tasks.
> And surprise: more complaints once the flip to secure defaults was finally done.
Posted Jun 4, 2016 9:07 UTC (Sat)
by noxxi (subscriber, #4994)
[Link] (12 responses)
> Sure, people do not like to rewrite perfectly functional code.
I think we have different ideas what "perfectly functional code" is. For me this includes security, for you probably not.
> we had an HTTPS server that used certificates originating from a Brazilian state certification authority. The problem was that the root certificate was not available in Ubuntu certificate storage, so that an external watchdog box kept triggering failovers every 5 minutes. Really nice.
Given that my idea of proper coding includes security I would see this as a bug in your code because you've failed to add the brazilian CA to your trust store. And that it worked before was only due to a security relevant bug in python where it failed to validate certificates. Actually you will find lots of bug reports out there where applications (even online banking apps) failed to verify certificates and it is commonly regarded a good thing if the developers fix these bugs.
Apart from that there was actually a lot of discussion before the changes were made to Python and LWN covered this discussion too in 09/2014: https://lwn.net/Articles/611243/. It is also explicitely marked as significant change in the release notes to 2.7.9, see https://www.python.org/downloads/release/python-279/.
Posted Jun 4, 2016 21:56 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (11 responses)
Security is optional and at times not needed at all.
> Apart from that there was actually a lot of discussion before the changes were made to Python and LWN covered this discussion too in 09/2014
> And that it worked before was only due to a security relevant bug in python where it failed to validate certificates.
Posted Jun 5, 2016 7:14 UTC (Sun)
by noxxi (subscriber, #4994)
[Link] (10 responses)
With this argumentation a browser which lets anybody do a man in the middle attack on HTTPS connections and thus sniff your passwords is still fully functional. I don't share this view since protection against sniffing is a major reason that somebody uses HTTPS.
> Security is optional and at times not needed at all.
I think it is fine to explicitly switch off security if you don't need it. But I don't think that security should be off by default and that the developer needs to explicitly enable it, because in this case most developers will not do it as the past has shown.
> Since the API hadn't even allowed to specify certificates.
I very much doubt this. First it was possible all the years to add certificates to the systems CA store. Then even ssl.wrap_socket in python 2.6 had the ca_certs argument which let you specify your own CA store. And modules like requests also made use if this.
> It was not a bug, but a feature....And there was absolutely no urgency to break the world after 20 or so years without SSL validation.
If you consider this as a feature you are right. If you consider this a bug (like I do) then it should be fixed as soon as possible because it is a security problem.
Posted Jun 5, 2016 7:28 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (9 responses)
Moreover, trying to "secure" browser by installing additional certificates might be a bad idea if their private keys are not managed correctly.
> I think it is fine to explicitly switch off security if you don't need it. But I don't think that security should be off by default and that the developer needs to explicitly enable it, because in this case most developers will not do it as the past has shown.
> I very much doubt this. First it was possible all the years to add certificates to the systems CA store. Then even ssl.wrap_socket in python 2.6 had the ca_certs argument which let you specify your own CA store. And modules like requests also made use if this.
So developers quite explicitly have chosen not to care about certificates and simply used the defaults.
> If you consider this as a feature you are right. If you consider this a bug (like I do) then it should be fixed as soon as possible because it is a security problem.
Would you chose to live for several months without utilities to make sure they are secure?
Posted Jun 5, 2016 8:24 UTC (Sun)
by noxxi (subscriber, #4994)
[Link] (5 responses)
How many browsers are used only inside an internal network? I think that majority will be used outside so they should be implemented with security in mind even if the security is not needed in a few specific cases.
> Moreover, trying to "secure" browser by installing additional certificates might be a bad idea if their private keys are not managed correctly.
That's correct. But there is huge difference between insecure in all cases (not checking anything) and insecure in some specific situations (private key compromised).
> Nobody cares about security if their business-critical application is hard-down.
And nobody cares about "but it was fully functional" if their business-critical passwords are compromised. Exactly for this reason it is important that applications and libraries are secure by default so that developers are aware of the problems and knowingly can decide if they disable security fully or better add a new trusted CA.
> So developers quite explicitly have chosen not to care about certificates and simply used the defaults.
It might be true that you explicitly choose to ignore security and this makes also sense in the case of a watchdog. But the ssl library is not only used by you. And I'm pretty much sure that the majority of developers just lived with the defaults while not beeing aware that these defaults are insecure. So yes, it unnecessarily broke your application but on the other hand it probably made lots of other applications more secure. There is an xkcd which describes this kind of tradeoff: https://xkcd.com/1172/
> Would you chose to live for several months without utilities to make sure they are secure?
Would you chose to live without utilities because someone hacked these, like recently happened in the Ukraine? Fortunately they are actually starting to get more security aware in this environment and laws get passed to better protect critical infrastructures. And they are actively working to improve the security.
But what they probably will not do in a SCADA environment is to upgrade to a new software version without lots of testing, i.e. unlike you in your business critical environment.
Posted Jun 5, 2016 8:42 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link]
> That's correct. But there is huge difference between insecure in all cases (not checking anything) and insecure in some specific situations (private key compromised).
If you install the certificates for these keys into your CA storage, then you open yourself to _undetectable_ MITM for anybody who obtains these keys.
> And nobody cares about "but it was fully functional" if their business-critical passwords are compromised.
> So yes, it unnecessarily broke your application but on the other hand it probably made lots of other applications more secure.
> Would you chose to live without utilities because someone hacked these, like recently happened in the Ukraine?
> But what they probably will not do in a SCADA environment is to upgrade to a new software version without lots of testing, i.e. unlike you in your business critical environment.
Posted Jun 5, 2016 23:05 UTC (Sun)
by bronson (subscriber, #4806)
[Link] (3 responses)
Posted Jun 6, 2016 5:55 UTC (Mon)
by noxxi (subscriber, #4994)
[Link] (2 responses)
The code changes do not cause DoS by themselves. The DoS is only caused in specific use cases in specific environment where the software relied on the previous default behavior and where the failure of the software after the update then caused a DoS . In other cases the code simply works as before and in yet other cases it will just break but not cause a DoS.
I don't know what exactly happened in this specific case from Cyberax where the change broke the watchdog and thus the setup of the site. But obviously a software upgrade was done on the system running a business critical application without up-front testing and maybe also without reading about the changes done by the software upgrade in detail. The release notes for Python 2.7.9 clearly stated the behavior changes.
While it is easy to complain about the changes to Python in this case I would recommend to better revise the process of installing untested software updates on critical systems, because you will find similar problems everywhere. Just look at the fallouts from CVE-2014-6321, where Microsoft had trouble to get a stable fix for a remote execution in their SChannel TLS library.
There is a reason that SCADA systems and other critical systems usually don't have the very latest system patches installed. In this environment any kind of software updates needs to be tested and approved by the vendor before they get shipped.
> to an extremely unlikely MITM?
Claiming that MITM is extremly unlikely is like claiming that the change causes an almost guaranteed DoS, i.e. exaggerating and making up "facts". Both MITM and DoS very much depend on the environment where the code is used.
But tools like firesheep have shown how easily MITM can be done (even though firesheep was for plain HTTP only the same thing would work with HTTPS when not validating certificates). And if it is possible and maybe easy too it will be done if the target is attractive. But the thing about MITM in a setup where you don't validate certificates is that you don't even realize that it happened. Thus you will not even realize if the change fixed your broken software, you will only realize if the change broke it.
Given that lots of security bugs are actually used in practice (Shellshock, Heartbleat,...) and cause serious damage I'm very much in favor of fixing such vulnerabilities fast, especially if they are trivially to exploit like this one. And yes, there will probably be collateral damage but there would be damage too if the vulnerability gets not fixed.
Posted Jun 10, 2016 1:55 UTC (Fri)
by mstone_ (subscriber, #66309)
[Link] (1 responses)
Posted Jun 16, 2016 17:29 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
We also support Ubuntu cloud AMIs that started to include Python changes that broke everything. That was fun to debug and fix...
Posted Jun 5, 2016 16:00 UTC (Sun)
by flussence (guest, #85566)
[Link] (2 responses)
Posted Jun 5, 2016 21:03 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Jun 6, 2016 18:37 UTC (Mon)
by flussence (guest, #85566)
[Link]
Posted Jun 16, 2016 12:44 UTC (Thu)
by nye (subscriber, #51576)
[Link]
And some valid certificates, seemingly at random. I have a backup script which until recently started with this rather depressing comment:
That is, this script throws an incomprehensible (at least after ~2 hours of Googling) verification error for this one crucial cert, which works in all web browsers, works in Python on Cygwin, was issued by the same CA as another certificate which does *not* throw an error in this same script, and has no obvious differences from said certificate aside from the CN and SAN. I'm presuming there must be some subtle difference somewhere in the trust chain that I couldn't see.
Fortunately the problem solved itself when they replaced the cert at the end of May. Why it threw a fit in the first place, we will never know.
This kind of thing is why I'm looking forward to when Python 2.7 finally goes out of support and can therefore at last be relied upon.
Posted Jun 4, 2016 9:14 UTC (Sat)
by morksigens (guest, #92681)
[Link] (1 responses)
I'd be interested to hear about what problems LibreSSL has. Their libtls initiative[1] looks like a good candidate for a high-level, general purpose SSL module.
Posted Jun 4, 2016 9:15 UTC (Sat)
by morksigens (guest, #92681)
[Link]
Posted Jun 4, 2016 15:03 UTC (Sat)
by robert_s (subscriber, #42402)
[Link] (8 responses)
It's a bit late - any users of requests are already using a bundled (er..) bundle of root certs (along with a bundled copy of urllib3).
Posted Jun 4, 2016 16:00 UTC (Sat)
by rahulsundaram (subscriber, #21946)
[Link] (7 responses)
Are they though? Distributions typically unbundle these.
Posted Jun 4, 2016 17:47 UTC (Sat)
by flussence (guest, #85566)
[Link] (6 responses)
Posted Jun 17, 2016 17:25 UTC (Fri)
by ceplm (subscriber, #41334)
[Link] (5 responses)
a) urllib3 is one of the reasons why requests are evil,
Anything else?
Posted Jun 21, 2016 12:30 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link] (4 responses)
Posted Jun 21, 2016 15:16 UTC (Tue)
by ceplm (subscriber, #41334)
[Link] (3 responses)
2. arrogance of maintainers who believe that they know better than anybody, so instead of using standard API, people should install for their five lines script three quarters of megabytes of their crap including their own root certificates.
Look, I don’t claim that the urllib2 API is the most beautiful one in the world, but IMHO the way how to fix standard library is to fix standard library not to do completely incompatible thing on the side. And from following the discussions about inclusion of python-request API in the stdlib I didn't get the impression that the Python community is at fault that it has been rejected.
Posted Jun 21, 2016 17:01 UTC (Tue)
by ceplm (subscriber, #41334)
[Link]
Posted Jun 22, 2016 13:40 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
Granted, Python didn't start out with a good package management system (and having gone through multiple iterations, seems to have finally decided on one[1]). But that doesn't mean that Python3 couldn't have spun modules out just as easily.
I really like that requests was not put into the standard library; it would have just stagnated it and required a new requests2 to be born.
[1]Though now Anaconda exists and some folk are claiming it as the Python Package Manager to End All Package Managers. Personally, it sets up walls around itself that are too high that it isn't a redistributable solution (hard-coded absolute paths everywhere).
Posted Jun 22, 2016 14:52 UTC (Wed)
by ceplm (subscriber, #41334)
[Link]
What I think is important is to investigate further separation of API and its implementation. Java guys manage to do that splendidly and I really like that I could switch between the implementation of the ElementTree from the Python standard library and lxml quite easily (and I see, I could do the same with lxml as an alternative implementation of SAX).
So, yes, I would be all for introduction of requests API in the standard library as a thin wrapper over the current urllib even with all limitations of that.
The future of the Python ssl module
Also worth noting is that even the improved constructor functions ( Because changing the actual constructor to do the sane thing would break compatibility ) is lacking some decisions, and defaults to TLS1.0 support by default, and not dropping support for known weak ciphers, unless your platform OpenSSL has removed them.
This leads me to expect that we'll see a function: ssl.create_real_default_context in the future, that does this better.
This is one area where Python are too afraid of breaking compatibility. But with the amount of complaints after Python3, I can sort of understand them.
The future of the Python ssl module
-- Python users.
The future of the Python ssl module
The future of the Python ssl module
Nope. Python developers completely revamped the SSL support in 2.7.9 (and these patches were backported into 2.7.8), causing problems for applications that actually cared about certificate validation and purely server applications.
Yes, they were.
These applications _worked_. After 2.7.8 minor patch they _stopped_ working. Full stop. You do NOT break applications, neither for "security" nor for "ease of maintenance".
1) A small targeted patch to enable certificate validation for 2.7.9, disabled by default and activated by a special environment variable. No wholesale SSL infrastructure replacement.
2) After some time the default is flipped to "on" with env. variable to disable this behavior.
The future of the Python ssl module
The future of the Python ssl module
That's fine. So do a major Python release (2.8) to make sure that old applications are not broken. There was no urgency to rush the fix.
Yes. That's how it should have been done. And NOT by re-doing the whole SSL API in a _minor_ _update_. It literally forced me to interrupt my vacation and spend a 100-hour work week frantically fixing stuff all over the place. Yet we've still lost one customer anyway.
Sure, people do not like to rewrite perfectly functional code. But at least after several years of warning and a new major release, breaking applications is at least justified.
The future of the Python ssl module
The future of the Python ssl module
It's quite simple - a program crashing at startup is not functional. A program that does what it needs to do is functional.
Except that the changes were widely backported into 2.7.8 since Python developers marked them as an urgent fix. See here: https://lwn.net/Articles/640697/
It was not a bug, but a feature. Since the API hadn't even allowed to specify certificates. And there was absolutely no urgency to break the world after 20 or so years without SSL validation.
The future of the Python ssl module
The future of the Python ssl module
Correct. If a browser is used only inside the internal network then it's perfectly fine to not use HTTPS authentication or even HTTPS in the first place.
Nobody cares about security if their business-critical application is hard-down.
Except that nobody did this. Watchdog doesn't care about security (only about reachability) and works only inside the trusted local network. If somebody is doing MITM in it, then it's already screwed.
Can you provide your address? I'm going to cut off your power lines because the SCADA system on your power plant is likely to be vulnerable. So you MUST get a local battery backup and a generator. Otherwise YOU'RE NOT SECURE!!!11111111!
The future of the Python ssl module
The future of the Python ssl module
Sure. That's why for a _browser_ it makes much sense to enforce HTTPS certificates. However, a lot of Python scripts _are_ designed only for internal networks.
A lot of internal systems use HTTPS just to avoid sending HTTP basic authentication passwords in the clear. However, their private keys are not managed securely so it's easy to pull them off.
However the likelihood of this resulting from a MITM attack inside a trusted LAN is way too low. This is NOT a new issue, it has been known for literally a _decade_ and simple probability of being attacked through this vector is way too low.
So is turning off the power. And no, I doubt that this whole SSL fiasco succeeded in preventing even one real-world attack.
Yes, I would.
No. There's a SECURITY (!111!oneoneone!) vulnerability, so they MUST turn off systems that have been working fine for decades in order to protect from one unlikely attack vector. No compromises are possible in the struggle for security!
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
If real-world utilities were as badly built as some of this software, security would be at the bottom of my list of priorities - below things like keeping up to date on my power socket adapters, memorizing which direction I have to turn the kitchen taps this week in order for them to put out water instead of gas, and planning an emigration to a country with some actual standards.
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
# Feb 2016
# Disable certificate checking - one of the necessary certificates fails
# verification here despite passing on other platforms and web browsers, and
# having nothing apparently wrong with it.
# Completely disabling verification is a distressingly heavy hammer, but I'm at
# my wits' end and need this backup to succeed at some point.
(And the fact that I just said that and actually mean it is why I'm looking forward to when I can go home and drink heavily.)
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
b) this looks like a packaging bug
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module
The future of the Python ssl module