[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
|
|
Subscribe / Log in / New account

CVE-less vulnerabilities

By Jake Edge
June 25, 2019

More bugs in free software are being found these days, which is good for many reasons, but there are some possible downsides to that as well. In addition, projects like OSS-Fuzz are finding lots of bugs in an automated fashion—many of which may be security relevant. The sheer number of bugs being reported is overwhelming many (most?) free-software projects, which simply do not have enough eyeballs to fix, or even triage, many of the reports they receive. A discussion about that is currently playing out on the oss-security mailing list.

Few CVEs

Alex Gaynor raised the issue in the context of CVE assignment. He noted the success that OSS-Fuzz has had ("just shy of 3000 things it counts as security bugs") since it was launched in 2016. He said that few of those bugs had been assigned CVE numbers and suggested that there may be a few reasons behind that: the difficulty of getting a CVE, the lack of an attention-seeking human motivated to get one, and the huge number of bugs being found. While CVEs are not the be-all and end-all of vulnerability handling, they do have some purposes:

CVEs are not important for their own sake. The true value is in all of the downstream processing that uses them as input: the Linux distributions that use them to figure out what fixes to backport, the docker security scanners that look for vulnerable code on the system, the corporate threat-intelligence feeds, etc.

He noted that he had done some work integrating ImageMagick and GraphicsMagick with OSS-Fuzz. He chose a random bug that was found in ImageMagick and tested it against Ubuntu 16.04 LTS; the bug was still present. Meanwhile, there are more than 100 security bugs for various projects that have been disclosed by the OSS-Fuzz team after the 90-day disclosure deadline passed without a fix. Gaynor said that he had no solution to propose but wanted to bring the issue up to spark a discussion.

The idea of picking and choosing fixes to backport "always fails in the end", Greg Kroah-Hartman said, so he suggested simply pushing users toward the latest version. That may sound a bit strange coming from the stable kernel maintainer for Linux, since there are multiple kernel versions being maintained as stable kernels. Those kernels are meant to collect up all known fixes, almost all of which are backports from the mainline, so they should contain the important fixes, regardless of whether a CVE number has been assigned to them. His intent would seem to be that each of the stable kernels has a "latest version" that he does advocate upgrading to. But Yves-Alexis Perez is not so sure that is a workable strategy for everyone:

I sympathize with this view, and I think we need to get better at updating, but I really think not all projects can be "safely" just updated to the latest version. End-users and IT admins still value stability and regressions is still a thing in a lot of cases. And once a regression is introduced, it's not always a short time before it's fixed upstream.

And before reaching the end-user, some project latest versions might depend on a lot of "latest version" of other projects. In some case the dependency tree might involve large scale changes affecting the whole system.

But if the choice is between all of the already known (possibly exploitable) bugs and an update with possible regressions, the choice should be clear, Kroah-Hartman said. He thinks that projects and users should be working hard to minimize and mitigate regressions:

Regressions always happen, we are human, but there are ways to mitigate them (testing, roll-back, preventing developers from not breaking things on purpose, etc.) And projects that do not do this type of work to prevent regressions need to learn that they should change, or users will go elsewhere.

GraphicsMagick maintainer Bob Friesenhahn said that many of the bugs found by OSS-Fuzz do not really need CVEs, "although they are bugs worthy to spend time fixing". The true security problems that he does fix are scrutinized by someone in the Debian project, he believes, so they do get CVE numbers, which are tracked by the project. There is, however, something of a disconnect between different sides of the bug finding and fixing problem:

Security bugs are often very hard to diagnose and fix. The community has become much better at producing bug reports than with helping to solve the problems found. Help with actually fixing issues is appreciated. I think that the objective should be open source software which lacks bugs and still provides a useful purpose. Finding bugs is just part of the effort.

Hanno Böck agreed that the OSS-Fuzz bugs are often not really security bugs per se, at least for programs like format parsers:

Even for the scarier looking vulns like write buffer overflows and use after free the situation is that these are usually not straightforward to exploit. All modern distributions have a combination of stack canaries, ASLR and nonexecutable memory. It's my understanding that while it's often possible to bypass those, doing so in non-scripting scenarios (e.g. in an image parser) is really hard and often impossible.

He also pointed out that finding bugs faster than they can be fixed is not a new phenomena, though it has sped up over time. Distributions may be over-promising as well, he said. Though they may "promise to backport all security fixes", they do not have enough people to actually keep up with that:

Maybe the main takeaway here is to just recognize that, and maybe distros should be more honest here and be clear what they can and can't do. And if you run a parser in a high risk environment you may not want to rely on the outdated version shipping in some LTS distribution.

The problem is not new, Gaynor agreed, but he thought that "the scale at which OSS-Fuzz has found vulnerabilities has genuinely exacerbated this problem". He also cautioned against being overly dismissive of the security implications of the bugs that OSS-Fuzz finds, pointing to two reports of vulnerabilities found using bugs that might be considered difficult or impossible to exploit.

Project responsibilities

While David A. Wheeler thought that Böck's belief that distributions are overstating their abilities was valid, he thinks that projects are part of the problem as well. He provided a list of ideas for ways that projects could help out. Many of them are based on the recommendations of the Core Infrastructure Initiative Best Practices project that he leads. The first suggestion he listed is not directly security related (thus is not part of the best practices badge), but gets to the heart of one of the problems under discussion:

Projects should work much harder at avoiding backwards-incompatible changes. Some projects (though *not* the Linux kernel) seem to take a very cavalier attitude to breaking changes. Yes, change is sometimes necessary, but projects need to work harder at providing graceful upgrades. (Slow deprecations, providing [alternative] differently-named 'new' interfaces with different semantics that let people gradually transition, and so on). IN PARTICULAR: I believe the primary reason that distros often backport, instead of using the "current" version, is because their users correctly fear backwards-incompatible changes. If projects would stop being the problem, then distros wouldn't feel the need to solve the problem.

Alan Coopersmith, who is one of the maintainers for the X.Org project, said that Wheeler's list made sense, but due to a lack of contributors X.Org cannot do all that was suggested; "and I know we're far from the only FOSS project with this problem". Part of the problem may be that companies that use the software are not doing their part: "when the people making money off the software aren't using that money to pay for this work, they can't be surprised when it doesn't get done".

There are gaps in the process of finding, analyzing, fixing, and reporting security bugs, which Marcus Meissner described. His takeaways were that better automation for bug fixing and a more comprehensive global database, with more information than what is stored with CVEs today, were the two biggest problems he sees. But Meissner also noted that CVE assignment is a manual process that could perhaps be automated. Dmitry Vyukov, developer of the syzkaller fuzzer, wondered if fuzzers should be wired up to automatically create CVEs:

So what are community thoughts on automatic CVE assignment? That would definitely get some attention to these bugs by vendors (because that's open CVEs in their products then). And this should be implementable because both OSS-Fuzz and syzbot are automated enough already. However I afraid that these CVEs may be as automatically sorted into a trashcan then :)

Alexander Potapenko noted that there is enough information in the OSS-Fuzz reports to help make a determination whether a particular bug was likely to be security relevant or not. But CVEs are only meant for actual vulnerabilities, Wheeler said. He pointed out that "many organizations have a rapid upgrade process if some software version has a CVE, and a slow process otherwise". But, as Simon McVittie pointed out, it is counterproductive to start giving CVEs to bugs that are not actually vulnerabilities:

Fast-tracking upgrades of packages with CVE fixes is only going to happen as long as it's still a rational strategy for balancing vulnerability exposure against the risk of regressions. If lots of CVE IDs get assigned to issues that aren't exploitable in the real world, then that will teach consumers of software that they can safely ignore "most" CVEs, which will tend to result in some issues that *are* exploitable being missed and not fixed on deployed systems. Everyone loses (except attackers).

As the thread started to wind down, Gaynor posted a summary of the themes he saw in the discussion, all of which seemed actionable ("which I like"), he said. The first two were to get better automation in two areas: CVE requesting/assignment, so that fuzzers could use it, and in assessing exploitability, which seems like a difficult problem, but one that perhaps would be a fertile field for academic research. The last theme is on reducing the number of vulnerabilities being added to projects:

While there's some dispute over just what % of the bugs that OSS-Fuzz and syzbot turn up are exploitable, there's no doubt that they find a _lot_ of them. Even if only 20% of OSS-Fuzz reports were truly exploitable vulnerabilities, that'd still be >600 of them. We can't produce this many vulnerabilities and then try to clean up afterwards by finding them with fuzzing -- at some point the number of vulnerabilities simply overwhelms us. Tactics for reducing vulnerabilities in the first instance, like memory safe languages, are an important part of making this problem tractable.

Overall, it was a wide-ranging, though relatively short by (some) mailing-list standards, discussion; it is still continuing, so there may be more ideas posted down the road. In the end, though, it is clear that there is a fairly large pile of security relevant bugs that have already been found—and that pile is growing steadily. Many of those did not and probably will not get CVEs. One can argue that users should simply upgrade, but it is abundantly clear that many do not do so particularly quickly without a motivating CVE fix, largely due to a concern about regressions. In addition, many of the bugs being reported from OSS-Fuzz and elsewhere are not actually being fixed due to a lack of contributors; upgrades can only be done if the bugs are being addressed.

Wheeler's point about the responsibility that projects have is an important one. But even the Linux kernel, which he highlights for its strict "no regressions" policy, is one of those code bases that many are loath to upgrade—going many months or, sometimes, years between updates to a relatively recent kernel version. Distributions play a role in that, of course, but they tend to backport fixes, which can lead to regressions popping out in other places. Their focus is often on stability, which leads to multiple kernel versions being maintained over the long run. It is all a bit of a mess.

Perhaps, though, Coopersmith's comment about companies that are making money using free software is the crux of the matter. If these problems were perceived to be truly serious—affecting bottom lines and stock prices—one can be reasonably sure there would be more effort put into solving (or at least massively reducing) them. So far that has not happened; Heartbleed may be the closest we have come.

Heartbleed resulted in the formation of the Core Infrastructure Initiative, which has made some useful contributions, though it seems to have slowed down (or stopped) over the last year or so. The initiative was a reaction to a single, serious bug; in aggregate, though, the bugs lurking out there—or still to be found and, hopefully, fixed—likely make Heartbleed pale in comparison. Getting out ahead of that horse, before it leaves the stall, much less the barn, seems appropriate somehow.


Index entries for this article
SecurityBug reporting/CVE
SecurityFuzzing

to post comments

CVE-less vulnerabilities

Posted Jun 25, 2019 22:19 UTC (Tue) by roc (subscriber, #30627) [Link] (13 responses)

Maybe companies should pay free software developers for early access to security bug fixes. E.g., project X decides that there will be a one-week window between releasing a security fix in the open repo and full disclosure of the vulnerability with exploit code. If you want more time to update your systems, you pay for earlier access to those updates.

Obviously this doesn't quite work for companies that need to distribute binaries under copyleft licenses, and it creates some poor incentives. But it might be better than not having the resources to create those fixes.

CVE-less vulnerabilities

Posted Jun 26, 2019 17:04 UTC (Wed) by misc (guest, #73730) [Link] (12 responses)

Wouldn't that have a side effect of giving less funding to a project with less security issues ?

CVE-less vulnerabilities

Posted Jun 26, 2019 20:17 UTC (Wed) by roc (subscriber, #30627) [Link]

Yes, that is one of the poor incentives.

CVE-less vulnerabilities

Posted Jun 26, 2019 20:25 UTC (Wed) by roc (subscriber, #30627) [Link] (10 responses)

Unfortunately some other popular models for funding free software also create perverse incentives. E.g. the "pay for support" model creates an incentive to make your software difficult to operate so people will have to pay for support.

CVE-less vulnerabilities

Posted Jun 27, 2019 9:01 UTC (Thu) by HenrikH (subscriber, #31152) [Link] (9 responses)

That is an excuse that I've heard many times on e.g Slashdot regarding why Red Hat introduced systemd so does there exist any evidence what so ever that "pay for support" have created unneeded complexity in the software?

CVE-less vulnerabilities

Posted Jun 27, 2019 9:24 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

I don't think we need incentives to make overcomplex software, really. We do that all on our own, with no incentive needed. :)

CVE-less vulnerabilities

Posted Jun 27, 2019 14:25 UTC (Thu) by bfields (subscriber, #19510) [Link] (1 responses)

Yeah, I mean, documentation and user interface design can be hard, and on any sufficiently complicated project may require the equivalent of multiple full-time skilled people. I don't think projects will intentionally sabotage improvements in those areas, it's more a question of whether they can fund them.

CVE-less vulnerabilities

Posted Jun 27, 2019 19:13 UTC (Thu) by HenrikH (subscriber, #31152) [Link]

And where would they find users? It's not like people will stand in line for "hey that software seams really hard to use vs the competition so let's also buy an expensive support package while we are at it!"

CVE-less vulnerabilities

Posted Jun 27, 2019 10:19 UTC (Thu) by roc (subscriber, #30627) [Link] (5 responses)

I can't point to any specific example, but the perverse incentive does exist. Hopefully most OSS developers take such pride in their work that the perverse incentive doesn't sway them!

CVE-less vulnerabilities

Posted Jun 27, 2019 19:12 UTC (Thu) by HenrikH (subscriber, #31152) [Link] (4 responses)

I have a hard time believing this, especially since no examples can be given. While such an incentive can exists in theory it's in stark contrast to the incentive to create software that are usable and good enough so that people choose to use it in the first place.

CVE-less vulnerabilities

Posted Jun 30, 2019 10:08 UTC (Sun) by nix (subscriber, #2304) [Link] (3 responses)

The automatic assumption here that most free software developers are mostly incentivized by money stands in stark contrast to almost every free software developer I have ever met. As a group we are to a large extent paid enough for free software hacking to be privileged sorts who usually don't need to worry about money, and freed from that tether most of us are interested in other things now (doing interesting things, writing code people will find useful, or just enjoying creating the software system itself).

Certainly, in my case, even a really blatant financial incentive like telling me that you'd double my salary if I did $thing (where $thing is not something I'd have been happy to do already) would not have much effect, and the financial incentive there is pointless: you could probably get me to prioritize a $thing I'd be happy to do over other things I was planning just by asking me, acting as a prod that at least one person has an expressed preference so I might as well follow it.

I suspect that if you *are* mostly incentivized by money, you've long since switched to become a VC vampire rather than a free software developer: God knows you'd have had enough opportunity to do so.

CVE-less vulnerabilities

Posted Jul 4, 2019 17:23 UTC (Thu) by Wol (subscriber, #4433) [Link] (2 responses)

> The automatic assumption here that most free software developers are mostly incentivized by money stands in stark contrast to almost every free software developer I have ever met. As a group we are to a large extent paid enough for free software hacking to be privileged sorts who usually don't need to worry about money,

While I agree with the first half of that (free software developers are not incentivized by money), I'm not sure about the second. There are probably a fair few wannabe free software developers (myself included) who lack the time and opportunity precisely because we are NOT paid enough to indulge ourselves in what we want to do (write free software).

Some people get lucky breaks - I'm not complaining - but some people don't.

Cheers,
Wol

CVE-less vulnerabilities

Posted Jul 17, 2019 18:22 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

Oh, I completely agree (I was in the same boat for fourteen years). But that's being incentivized by *something else*, and money is just something to let you eat while you do the something else. If you had enough money to do the something else, would you want more money for the sake of it? If not, you're not incentivized by money, I'd say. (An amazing number of people *are* incentivized by it: I don't understand what they get out of it, maybe they use it to keep score or something primate-status like that?)

CVE-less vulnerabilities

Posted Jul 20, 2019 11:05 UTC (Sat) by jospoortvliet (guest, #33164) [Link]

On top of that I want to challenge the perverse incentive a bit. Yes, it is there, but Red Hat also wants it’s software to be popular and used a lot, otherwise it had no large user base to draw customers from. So it isn’t a 100% bad incentive - it is ‘complicated’ and I’d argue it kind of balances out.

CVE-less vulnerabilities

Posted Jun 26, 2019 0:35 UTC (Wed) by scientes (guest, #83068) [Link] (2 responses)

The Linux Kernel benefits on its backwards compatibility goals by having extremely clear boundaries between kernel and user-land, which are based in hardware reality.

CVE-less vulnerabilities

Posted Jun 26, 2019 11:27 UTC (Wed) by gus3 (guest, #61103) [Link] (1 responses)

Until you add QEMU and hypervisors to the mix.

CVE-less vulnerabilities

Posted Jun 26, 2019 12:12 UTC (Wed) by roc (subscriber, #30627) [Link]

And BPF and tracepoints and ioctls...

But it's true that the kernel at least has the power to enforce constraints on user-space, whereas it's very difficult for user-space applications to protect themselves from other software. *cough* antivirus *barf*

Regressions

Posted Jun 26, 2019 3:19 UTC (Wed) by pabs (subscriber, #43278) [Link] (1 responses)

I came across a regression in the Linux kernel; the EDID override mechanisms in the drm subsystem are broken from 4.15 to 5.2 rc2 (also fixed in the stable releases). I can only conclude that Linux in 2017 didn't have adequate testing for the EDID override mechanism, otherwise the rework that broke this wouldn't have caused the issue.

https://bugs.freedesktop.org/show_bug.cgi?id=107583

Regressions

Posted Jun 26, 2019 16:24 UTC (Wed) by daniels (subscriber, #16193) [Link]

Yes, that's correct. If the testing was perfect, the bug wouldn't have occurred in the first place. Which is pretty generally applicable.

CVE-less vulnerabilities

Posted Jun 26, 2019 7:45 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (6 responses)

The Core Infrastructure Initiative Best Practices does not even include monitoring external dependencies to detect/fix known vulnerabilities in it's passing level! That’s why most fixes do not propagate, downstreams to no check they work with updated versions, so everyone is afraid to apply fixes, and distros exhaust themselves trying to compensate.

Just doing due diligence on fixes produced by others, is way more critical than running fuzzers on your own codebase. Who cares if your code is perfect if it pulls in hundreds of problematic third party components.

CVE-less vulnerabilities

Posted Jun 26, 2019 13:59 UTC (Wed) by NAR (subscriber, #1313) [Link] (4 responses)

monitoring external dependencies to detect/fix known vulnerabilities

How should this be done? The last project I worked on had 62 external dependencies - how do I monitor them?

CVE-less vulnerabilities

Posted Jun 26, 2019 16:04 UTC (Wed) by Sesse (subscriber, #53779) [Link]

> The last project I worked on had 62 external dependencies

Clearly you don't use Node.js. :-)

CVE-less vulnerabilities

Posted Jun 26, 2019 20:23 UTC (Wed) by roc (subscriber, #30627) [Link]

Github is providing this as a service: https://help.github.com/en/articles/about-security-alerts... (which acquired https://dependabot.com/)

There's also Greenkeeper.io and Gitlab has something similar --- https://docs.gitlab.com/ee/user/application_security/depe....

CVE-less vulnerabilities

Posted Jun 27, 2019 19:29 UTC (Thu) by k8to (guest, #15413) [Link]

It's a bear, but there are tools to automate or semi-automate this sort of thing. In the commercial space it's typically a matter of a small expenditure to get a thing that creates workflows and everything.

In the open source world, it's less clear. Some services have a no-pay tier for open source users, but it never feels very sustainable.

CVE-less vulnerabilities

Posted Jun 28, 2019 7:59 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

You write it as if some gremlin had connected to your project deep in the night and had made it use all this other code. The dependencies didn’t add themselves, someone in the project made the decision to use this other code, instead of writing the functionality himself.

This person should have evaluated the costs of keeping this code up to date, instead of forking it and maintaining it all directly. Just like he should have evaluated if the API provided by this other code is convenient, if it’s compatible licensing-wise, if it’s actively maintained in the original project and reasonably future-proof, etc.

Costs of keeping other code up to date balloon if it calls itself lots of other third party code, if its authors do not keep APIs stable and backwards-compatible, do not release (each commit is a new thing to look at), do not use semver, do not write changelogs, do not announce new versions on a mailing list. Sometimes choosing something a bit less featured with fewer keeping up to date problems is a better deal.

Anyway checking up-to-dateness can be done using announce mailing lists, release-monitoring.org, pretty much any language that uses a package manager allows querying for the latest version, everyone is moving to git nowadays so you can write your own git pull and check tags monitoring robot, etc. All the major actors (Google, Microsoft via Github, Gitlab, node…) are starting to keep registries of problematic code, and you can use those too (but, that will make you dependent on them, and pretty soon they'll start scoring projects in ecosystems they control, and force you to do whatever changes they want for free, if everyone relies on their score, and a bad score becomes the kiss of death).

Or, just choose things available in a major Linux distro, adhere to its update cycle, and update whenever it does in its devel branch.

There’s no miracle, anything like static linking, containers, vendoring, private forks, that isolates you from the version bumps of others, puts you in power. With power comes responsibility. The less you depend on the update checks of others, the more you need to perform them yourself.

As a rule, any upstream new version should be checked for security or dataloss fixes. Quite often those are not traced, and you end up updating by default, because it's less work than auditing all the code changes to identify what is needful.

The CII Best Practices badge *does* require dependency monitoring (at the silver level)

Posted Jul 1, 2019 18:39 UTC (Mon) by david.a.wheeler (subscriber, #72896) [Link]

The "passing" badge doesn't require it, but the "silver" badge (the next level) of the CII Best Practices badge certainly does require that. The dependency_monitoring criterion says:

Projects MUST monitor or periodically check their external dependencies (including convenience copies) to detect known vulnerabilities, and fix exploitable vulnerabilities or verify them as unexploitable. [dependency_monitoring]
Details: This can be done using an origin analyzer / dependency checking tool such as OWASP's Dependency-Check, Sonatype's Nexus Auditor, Black Duck's Protex, Synopsys' Protecode, and Bundler-audit (for Ruby). Some package managers include mechanisms to do this. It is acceptable if the components' vulnerability cannot be exploited, but this analysis is difficult and it is sometimes easier to simply update or fix the part.

There are related criteria to support this, especially external_dependencies, updateable_reused_components, and interfaces_current. In addition, all the test-related criteria in some sense support this, because to rapidly update a component you need to be able to be able rapidly and thoroughly test the result after the update.

In short, the best practices badge does cover this.

CVE-less vulnerabilities

Posted Jun 26, 2019 7:46 UTC (Wed) by marcH (subscriber, #57642) [Link] (2 responses)

> We can't produce this many vulnerabilities and then try to clean up afterwards by finding them with fuzzing -- at some point the number of vulnerabilities simply overwhelms us. Tactics for reducing vulnerabilities in the first instance, like memory safe languages, are an important part of making this problem tractable.

Leaving insecure and obsolete languages behind is the one of the solutions long timer, however it won't happen overnight.

If nothing else on the QA side, maybe projects could start continuously fuzzing themselves and block any change/regression that adds new failures. Writing tests and automating them is expensive, this is why it rarely gets done enough or properly, however in this particular case most of the works seems to have been done already... How much effort to automate OSS-Fuzz locally? Preferably before merge, otherwise immediately after at least (and then manual revert).

CVE-less vulnerabilities

Posted Jun 27, 2019 19:34 UTC (Thu) by k8to (guest, #15413) [Link] (1 responses)

Have you tried this in practice?

I find most of these shotgun style tests tend to work poorly, because they take longer to produce results, so time-to-merge pressures mean they end up catching problems many commits in, and many developers aren't excited about trying to figure out whose changes caused the problem. The net result is a lot of finger pointing and waste and ultimately this style of test is decommissioned.

Maybe a healthier development culture wouldn't have this problem, or maybe some projects are smaller and fuzz could produce results in minutes. Ideally there's a social hack I'm missing to solve this kind of problem more generally.

CVE-less vulnerabilities

Posted Jun 28, 2019 9:33 UTC (Fri) by marcH (subscriber, #57642) [Link]

No, I don't have any experience myself but the article states:

> projects like OSS-Fuzz are finding lots of bugs in an automated fashion—many of which may be security relevant

https://github.com/google/oss-fuzz seems pretty strong on automation and in my experience Google rarely ever pays engineers to perform repetitive tasks.

So it sounds like OSS-Fuzz finally cracked that nut. That's probably why there's a LWN article about it.

CVE-less vulnerabilities

Posted Jun 26, 2019 8:32 UTC (Wed) by Otus (subscriber, #67685) [Link]

I think distros should just drop software which cannot handle backwards compatibility well. At the very least for libraries and such that can end up having other packages depending on them and thus blocking other updates.

Yes, it takes a bit of extra work, but if the project cannot handle that work, then the software isn't ready to be distributed to millions of people.

CVE-less vulnerabilities

Posted Jun 26, 2019 18:11 UTC (Wed) by nilsmeyer (guest, #122604) [Link] (21 responses)

> Their focus is often on stability, which leads to multiple kernel versions being maintained over the long run. It is all a bit of a mess.

Stability is the stated goal, however as far as I can see most of the time a new stable kernel is released the backported / LTS series get an update too so you'll have to upgrade and reboot here as well so the only thing that is stable is the major.minor version. This approach to system administration seems to be based on lore, hearsay and branding (Debian stable anyone?) more than facts.

The fewer times you upgrade the worse the pain is going to be. The reality has changed, high uptime isn't a sign of a stable system, it's a sign of a complacent administrator. Of course there are also many corporate procedures in place that serve to arrest change as well (Change management, ITIL, ITSM etc.).

A lot of effort is spend on maintaining LTS codebases and backporting, it may make sense to redirect those efforts even at the risk of undermining broken operating procedure.

CVE-less vulnerabilities

Posted Jun 26, 2019 19:56 UTC (Wed) by corsac (subscriber, #49696) [Link] (7 responses)

Nobody (no one serious at least) care about single machine uptime. People use stable kernel *branches*, with isolated fixes, because they're stable in the long run, bugs (security or not) are fixed, and few new ones are introduced. No behavior change, no regression.

CVE-less vulnerabilities

Posted Jun 27, 2019 6:53 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (3 responses)

> People use stable kernel *branches*, with isolated fixes, because they're stable in the long run, bugs (security or not) are fixed, and few new ones are introduced. No behavior change, no regression.

Those are the stated reasons. I challenge their validity though.

CVE-less vulnerabilities

Posted Jun 27, 2019 19:41 UTC (Thu) by k8to (guest, #15413) [Link] (2 responses)

Surely you can accept that fewer changes results in fewer unexpected breakages.

You must be making some kind of argument that there's a counterbalancing force, right? The bugs that aren't fixed because of a lack of change? Something else?

It may be good practice in your mind to have a well-designed upgrade strategy, but I don't see a lot of rationale for datacenters to update their systems all the time. What problem does a strategy like upgrading every 3 months cause? It was certainly always irritating to me as a bug-fixing engineer working with these customers, but they almost never had breakages they couldn't isolate and resolve at least by rollback.

All these cattle strategies like blue/green and balloons and so on are pretty neat, and help enable faster work, but they seem better tuned towards running your own software as opposed to running off the shelf applications, especially when there's significant state involved.

Or maybe I'm just ignorant of some factors?

CVE-less vulnerabilities

Posted Jun 27, 2019 21:37 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (1 responses)

> Surely you can accept that fewer changes results in fewer unexpected breakages.

Isn't running buggy / vulnerable code also an unexpected breakage?

> You must be making some kind of argument that there's a counterbalancing force, right? The bugs that aren't fixed because of a lack of change? Something else?

My apologies, I don't think I have expressed myself well enough.

Many of the recent attention grabbing bugs also exist in earlier versions, though fixing them takes more time there since newer versions take precedence. A lot of the old code that hasn't been touched in a while may also contain errors that don't occur in newer versions since that code has been replaced. Tooling has massively improved in recent years which means that newer code is probably better tested than a lot of the old code that is working though may have issues. There are far more eyes on new code at the moment. For a developer, working on features is probably more fun than debugging and fixing code which is still a lot more fun than backporting those fixes to software versions you don't personally run - this is often paid work or in some cases automated. Long term support is usually a promise, not a guarantee.

When you encounter and report a bug for an older version the first question will usually be "can you reproduce it on current stable / devel?"

> It may be good practice in your mind to have a well-designed upgrade strategy, but I don't see a lot of rationale for datacenters to update their systems all the time.

I think there are good arguments for doing this and treating it as a part of operating routine instead of having Patch Days or regular update marathons. Doing this regularly builds institutional knowledge, establishes good patterns, encourages automation and spreads the workload over time.

> What problem does a strategy like upgrading every 3 months cause?

3 months isn't all that bad, I've seen a lot of places that do worse - most don't have an upgrade strategy or they have a strategy that is documented and audited but not executed. In my experience that date is a moving target, the longer people don't upgrade the lower their confidence is in their ability to upgrade, planned upgrades keep getting rescheduled. Once you automate these procedures you have to still do the procedure at regular intervals, otherwise you'll lose confidence in your tools or they just stop working. A lot of this work just ends up not being done since it's risky, doesn't offer an immediate reward (though it may sometimes fix user problems) and the risks don't manifest themselves.

We haven't heard many horror stories yet but I think we'll see a lot more of that in the coming years since breaking into systems has become lucrative as a criminal business model. And it's often not the Shellshocks, Heartbleeds, Spectres or Meltdowns that cause it, it's trivial and old bugs (MS Office Macros). These will only become more sophisticated over time as criminals go from targeting users directly to targeting servers.

Would you be confident running a 3 months old browser? Looking at the history of Firefox ESR it has seen almost as many upgrades as the stable series. You're upgrading anyways, why keep the old version? Because it has less features and gets less attention from developers?

> All these cattle strategies like blue/green and balloons and so on are pretty neat, and help enable faster work, but they seem better tuned towards running your own software as opposed to running off the shelf applications, especially when there's significant state involved.

This I agree with, a lot of work has been focused on the somewhat easy target of stateless software, once you introduce state people start fearing the upgrade again (or just say "let $PROPRIETARY_CLOUD worry about it"). This is even worse with commercial/proprietary software since you're entirely dependent on the vendor who of course has little incentive supporting multiple platforms or be at all friendly to other people operating the software. They have a captive audience (you can't just switch out your RDBMS). Prices often forbid having a fail-over system or heavens forbid a test platform since executives often don't understand the risk - if it hasn't failed the last year there really is no need to have a backup. It's like driving drunk to save on taxi money. Technical debt is easily externalized.

Either go fully SaaS and have SLAs with severe penalties or invest into switching to software that is less of a burden to maintain.

CVE-less vulnerabilities

Posted Jun 30, 2019 10:13 UTC (Sun) by nix (subscriber, #2304) [Link]

Would you be confident running a 3 months old browser? Looking at the history of Firefox ESR it has seen almost as many upgrades as the stable series. You're upgrading anyways, why keep the old version? Because it has less features and gets less attention from developers?
Well, perhaps the thing you're upgrading is a foundational element in the software stack (a browser might count as that: so might Qt, or the X server, or glibc, or etc etc) and you want bugfixes but a lot of things depend on this and you don't know exactly what things they might be doing, so you'd rather add new features and cleanups and the corresponding code churn, because better tooling or no better tooling, every time the code changes there is the possibility of new bugs. The old code has old bugs, sure, but you already know that those old bugs don't affect your workflow or you'd be upgrading more aggressively because the old software didn't work right for you (with the exception of security problems, of course, where the bugs will be exercised by an attacker, not by you).

So there really is an extra cost to upgrading to new-feature branches versus stable branches. (The downside is the the changes on the stable branch have probably been much less tested on that branch than they were in the new-feature branch they were likely developed against. Tradeoffs, tradeoffs...)

CVE-less vulnerabilities

Posted Jun 27, 2019 14:18 UTC (Thu) by nix (subscriber, #2304) [Link] (2 responses)

I care about single machine uptime of machines I'm using, because there's a lot of dynamic state on there (terminal emulator configs, piles of VMs I'm working in, etc etc) and every time I reboot all that dynamic state is blown away. It sometimes takes me half an hour or more to get back up to speed again after a reboot, even given that SSDs reduce the page-cache-population lags we used to have to put up with dramatically.

The cloud and microservices are all very well, but eventually you come down to the machines people are actually using, and those machines tend to have state in RAM, and that state is not all always recovered instantly and perfectly on reboot. Session saving is not a panacea :(

CVE-less vulnerabilities

Posted Jun 27, 2019 21:42 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (1 responses)

> The cloud and microservices are all very well, but eventually you come down to the machines people are actually using, and those machines tend to have state in RAM, and that state is not all always recovered instantly and perfectly on reboot. Session saving is not a panacea :(

I hear ya, having an external service or API respond slower for a few milliseconds isn't as big of a deal as the kind of disruption a single reboot of the workstation does, so I often find myself using suspend-to-ram instead of shutting down the machine.

Don't underestimate the amount of state that is only present in your mind as well...

CVE-less vulnerabilities

Posted Jun 30, 2019 10:16 UTC (Sun) by nix (subscriber, #2304) [Link]

Don't underestimate the amount of state that is only present in your mind as well...
If that gets wiped on a desktop reboot you need to get looked at fast. :) going to sleep, on the other hand...

(It's actually almost inverted: going to sleep is helpful even though it results in loss of state: the state that is lost is mostly useless and the improvements gained in knowledge and clarity of thought and just not being tired any more and so on are much greater than the loss.)

CVE-less vulnerabilities

Posted Jun 27, 2019 9:40 UTC (Thu) by jschrod (subscriber, #1646) [Link] (12 responses)

The problem ain't reboots, regressions are the problem.

E.g., the last Debian stable kernel update (to 4.9.0-9) caused timeouts in communication of Win7 clients with Samba on this Debian system. Win8.1 didn't seem to have the problem. Problems occured both with local clients (running in an VM) and external clients, so it's probably not caused by the network stack. Initial level 0 tests didn't show it -- accessing a remote directory was possible, though slow; the error only happened when one accesses a larger file, e.g. with MS Visio. Rebooting to 4.9.0-8 solved the problem immediately.

Some quick googling didn't show any others with the same problem.

So, we have a very real regression *for us* from a regular Debian stable update, where we have to triage our setup to see what has caused it. Maybe the cause is some inderdependent behavior; I'm sure it won't be easy to analyze.

Tracking feature changes in packages more aggressively would mean more of these occasions where one has to spend lots of work. Work that I would rather like to be spent for our clients than for our infrastructure. That's why companies like ours use "slow moving" dists instead of rolling releases.

CVE-less vulnerabilities

Posted Jun 27, 2019 16:28 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (4 responses)

Running the latest at least as a smokescreen would allow you to catch it at the rc level rather than after it has been backported. Backports done by stable distros are also not reliable in and of themselves anyways. A coworker told me yesterday that apparently a backport into Debian stable introduced a regression into the Python package (sorry, I don't have more details; I'll have to ask if he filed/found an issue for it).

CVE-less vulnerabilities

Posted Jun 27, 2019 21:01 UTC (Thu) by roc (subscriber, #30627) [Link] (2 responses)

If you have lots of servers, plenty of admin resources, and an architecture that hides individual server failures, running an rc kernel on a subset of your servers could make sense as way to catch regressions early and hopefully get them fixed. Though there are plenty of examples where regression reports are ignored or take a long time to fix.

But someone running a handful of Samba servers, which sounds like jschrod's case, doesn't have those things.

CVE-less vulnerabilities

Posted Jun 27, 2019 21:04 UTC (Thu) by roc (subscriber, #30627) [Link]

... in which case you're asking people to inevitably expose their users to more downtime. The answer will be, and should be, a firm "NO".

CVE-less vulnerabilities

Posted Jun 30, 2019 18:26 UTC (Sun) by jschrod (subscriber, #1646) [Link]

> But someone running a handful of Samba servers, which sounds like jschrod's case, doesn't have those things.

Yes, this is our case -- but even though we're a small company, we still do deployments of updates for important packages on canary systems first. They are used by some admins who know how to handle errors and roll back to a working system. (Canary deployments are done on VMs with snapshot/rollback capability.)

CVE-less vulnerabilities

Posted Jun 30, 2019 18:23 UTC (Sun) by jschrod (subscriber, #1646) [Link]

> Running the latest at least as a smokescreen would allow you to catch it at the rc > level rather than after it has been backported.

We caught it when we deployed the new kernel on a canary system.

For our environment (we are a small company) it isn't really needed to learn about such problems earlier. Such problems typically are sorted out within the week and don't disturb production -- here with the help of fellow lwn.net read mdg, even earlier. Usually, we can live with updating our production systems a few days later.

CVE-less vulnerabilities

Posted Jun 27, 2019 22:09 UTC (Thu) by nilsmeyer (guest, #122604) [Link] (4 responses)

> Tracking feature changes in packages more aggressively would mean more of these occasions where one has to spend lots of work. Work that I would rather like to be spent for our clients than for our infrastructure. That's why companies like ours use "slow moving" dists instead of rolling releases.

Are you speaking from experience, having used a faster moving distro before? My problem is, I don't really have any data to support my thesis either and I also don't completely practice what I preach (looking at my NAS running Ubuntu 16.04). I've been an early user of Gentoo so I had my share of grief with rolling releases.

The way I see it your slow moving distro broke your workload all the same, you were able to revert to an older version but I don't like your chances getting that bug fixed or even getting a bug report looked at. What are your options now? You're stuck on that kernel version forever, having to mitigate any issues that come up in the future form the outside?

I'm not sure how the Debian versions correspond to the LTS kernel versions, 4.9.184 is the most recent I see on kernel.org.

CVE-less vulnerabilities

Posted Jun 28, 2019 3:02 UTC (Fri) by nivedita76 (guest, #121790) [Link] (2 responses)

You think he’ll have a problem getting Debian to even look at a bug report for their stable series? Do you have any evidence to back that up, or do you not understand what a Linux distribution is?

CVE-less vulnerabilities

Posted Jun 28, 2019 17:53 UTC (Fri) by nilsmeyer (guest, #122604) [Link]

pure conjecture.

CVE-less vulnerabilities

Posted Jun 30, 2019 18:29 UTC (Sun) by jschrod (subscriber, #1646) [Link]

The problem was not in Debian's kernel update, as I suspected. Otherwise, more of those error stories would have been found. It was an incompatibility between my (quite old) smb.conf and that new kernel.

Actually, I don't know if Debian would have reacted to a bug report about that. I wouldn't have opened one without being sure that there is no problem in my installation or my configuration -- and it was in the latter.

CVE-less vulnerabilities

Posted Jun 30, 2019 18:38 UTC (Sun) by jschrod (subscriber, #1646) [Link]

> Are you speaking from experience, having used a faster moving distro before?

Yes, we used to run openSUSEs Tumbleweed. It was not worth the hassle.

The problem is actually not newly introduced errors. They are seldom enough, we were always able to handle them. The problem are incompatibilites in software upgrades that break one's files or -- especially bad -- one's workflow.

For us, it's easier to schedule some time in block where one checks such incompatibilities for a new release (e.g., currently for Debian buster), adapts workflows, and tells all co-workers about these changes in a scheduled and timely fashion. The alternative is having to handle such incompatible changes all day as surprises ("Christine called in, she cannot work any more, fix it ASAP") on top of one's normal workload.

Other companies, with more staff, may see this different than we do. (Although our customer's IT departments *are* larger, and they are even more conservative than we are.)

Samba and kernel 4.9.0-9

Posted Jun 29, 2019 5:58 UTC (Sat) by mbg (subscriber, #4940) [Link] (1 responses)

jschrod, you're not alone -- I think I hit the same thing.

Removing these options (long deprecated apparently) from my smb.conf seemed to fix it:

socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192

Samba and kernel 4.9.0-9

Posted Jun 30, 2019 18:16 UTC (Sun) by jschrod (subscriber, #1646) [Link]

Thanks a lot for this info, that solved it.

Actually, the culprit seems to be SO_RCVBUF and SO_SNDBUF, testparm meanwhile warns about them. (Maybe I should establish a monitoring plugin that runs testparms and checks for warnings...) TCP_NODELAY is actually even default.

This shows again how great the lwn.net community is!!

Thanks, Joachim

CVE-less vulnerabilities

Posted Jun 26, 2019 21:20 UTC (Wed) by rweikusat2 (subscriber, #117920) [Link] (4 responses)

Many people I have to deal with are absolutely paranoid wrt updates. This includes one entity which chose to rather keep a known remote DoS with a trivial fix than allow that trivial fix to be installed.

CVE-less vulnerabilities

Posted Jun 27, 2019 9:36 UTC (Thu) by ssiccha (guest, #131965) [Link] (1 responses)

Sounds like that entity was not at all able to assess whether the change was trivial or not.

CVE-less vulnerabilities

Posted Jun 27, 2019 12:27 UTC (Thu) by rweikusat2 (subscriber, #117920) [Link]

Well, obviously not. But even when disregarding compatibility issues or the mere fact that updating something needs a certain amount of time people aren't necessarily allowed to spend on that, "just make them all run the latest version" is not going to fly.

CVE-less vulnerabilities

Posted Jun 27, 2019 22:18 UTC (Thu) by nilsmeyer (guest, #122604) [Link]

I think this is a very common pattern simply because there often aren't any consequences to running vulnerable software. In some environments malfunctions are so common that it's hard to discern them from an attack. The risks can even be insured at reasonable rates.

Trivial fixes might not be trivial

Posted Jun 28, 2019 3:01 UTC (Fri) by dps (guest, #5725) [Link]

In a former life I maintained a something based on a way out of security support version of RHEL. Even the most trivia kernell fix required dev testing on all supported hardware, which was several days work.

QA also had to do some more testing, so kernel upgrades where not something done lightly. A CVE and sufficient CVSS was required to overcome the resistance to changing anything.

I might have declared anything which broke on current arable versions to be bugware which te to be fixed, but was not in a position to do that.

CVE-less vulnerabilities

Posted Jun 27, 2019 15:56 UTC (Thu) by JoeBuck (subscriber, #2330) [Link]

What can be done, if there aren't resources available to fix the vulnerabilities, is to have detailed documentation on which free software packages/libraries/programs (or which portions of them) need to be considered unsafe to use in an uncontrolled environment, where an untrusted party can control the input, and which are more robust. Even if code is known to be subject to buffer overruns and the like it still might be useful, but people need to know not to hook it up to the public Internet.

CVE-less vulnerabilities

Posted Jun 27, 2019 23:06 UTC (Thu) by rra (subscriber, #99804) [Link] (24 responses)

From the quote in the article from Hanno Böck:

> It's my understanding that while it's often possible to bypass those, doing so in non-scripting scenarios (e.g. in an image parser) is really hard and often impossible.

This is a (common) misunderstanding of how people use image parsing programs in practice. There is quite a lot of fully automated use out there where an attacker has essentially unlimited attempts. Think of thumbnailing of uploaded images, image resizing of profile pictures, and so forth.

Image parsers are the most common example (they are used a lot, probably more than most people expect), but versions of this misunderstanding crop up a lot. I remember recently when some people were surprised by a CVE in the file utility, thinking that at worst file might be called interactively on untrusted input, and several people said no, there were places where file was being called inside automation and an attacker could keep trying until they succeeded.

Command-line tools are useful, and UNIX is good at plugging them together in pipelines to make automated tools, and as a result, it's rare to find a program that you can guarantee is never run unattended, repeatedly, on untrusted input. You should assume all software will be run this way unless you can prove it's not.

Personally, I think it's no longer an acceptable security practice to run an image parser on untrusted input outside of a sandbox. We should keep fixing bugs in the parsers as a first line of defense, but that line of defense clearly fails frequently enough that one needs a second line of defense (a seccomp sandbox, a dedicated container, something). That's probably true of most parsers that are even mildly complex. Recent history says that it's true even of the file utility.

CVE-less vulnerabilities

Posted Jun 28, 2019 15:07 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link] (23 responses)

> Personally, I think it's no longer an acceptable security practice to run an image parser on untrusted input outside of a sandbox. We > should keep fixing bugs in the parsers as a first line of defense, but that line of defense clearly fails frequently enough that one
> needs a second line of defense (a seccomp sandbox, a dedicated container, something). That's probably true of most parsers that
> are even mildly complex.

This kind of magic thinking keeps amazing me: In order to guard against unknown bugs in 'complex' software A, we must use significantly more complex software B which is believed to be free of unknown bugs because of ... well just because ...

CVE-less vulnerabilities

Posted Jun 28, 2019 15:56 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (21 responses)

>we must use significantly more complex software B which is believed to be free of unknown bugs because of ... well just because ...

That is a not an earnest attempt at understanding a different perspective. All software including sandboxes can have bugs. However it is easier to specify access points and fix any bugs once in the sandbox that is specifically designed with security in mind compared to parsers which often have a long legacy and history of security vulnerabilities. We do have sufficient experience with sandboxes (for ex: Chrome sandbox or bubblewrap) to know both a) sandbox are not a magic solution 2) they do help to stop or mitigate vulnerabilities.

CVE-less vulnerabilities

Posted Jun 28, 2019 17:34 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link] (20 responses)

There is no way to 'understand' this 'perspective' as it's plainly self-contradictory: Compared to sandboxes, all of which have a history of security issues(!), most parsers are fairly simple. The people argueing for this just chose to believe that the 'parser' must have unknown, security-criticial bugs and chose to believe that the sandboxing software will not. This is really just a dual appeal-to-ignorance: We don't know if the 'parser' has security issues, hence, it probably has. We don't know if the sandboxing software has security issues, hence, it probably doesn't.

One should really call this a higher-order logical fallacy as it uses the exact, same falllacious reasoning to justify two contradicting conclusions.

CVE-less vulnerabilities

Posted Jun 28, 2019 17:51 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> We don't know if the 'parser' has security issues, hence, it probably has. We don't know if the sandboxing software has security issues, hence, it probably doesn't.

You're missing that the parser was written to "get a job done" which is to make a set of pixels from encoded data. The sandboxing software was written to get a job done as well: limit what can be done within an environment meant to run untrusted code. The mindset and review rigor difference between the two should be readily apparent in that I'd expect the latter to actually have a security model and documentation on how it handles that model. The former rarely has one until it has systemic problems.

CVE-less vulnerabilities

Posted Jun 28, 2019 17:52 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> There is no way to 'understand' this 'perspective'

Of course there is but in this case, you have already made up your mind and that's fine. I will just point out that in general, the industry trend is moving towards more sandboxes, not less based on empirical evidence that it is useful. You are just the outlier here.

CVE-less vulnerabilities

Posted Jun 28, 2019 18:00 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link] (1 responses)

That's now an appeal to popularity, another logical fallacy.

CVE-less vulnerabilities

Posted Jun 28, 2019 18:10 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> That's now an appeal to popularity, another logical fallacy.

How is empirical evidence an appeal to popularity? Just look at vulnerabilities in parsers vs sandboxes that protect them. It isn't that hard. If you are going to have the outlier position that goes against that evidence, make your case.

CVE-less vulnerabilities

Posted Jun 28, 2019 18:02 UTC (Fri) by excors (subscriber, #95769) [Link]

> We don't know if the 'parser' has security issues, hence, it probably has.

When we know the parser has had several hundred security issues in the past (see https://www.cvedetails.com/vulnerability-list/vendor_id-1... , most of which say "...via a crafted file"), we can be pretty sure it's going to have a lot more.

> We don't know if the sandboxing software has security issues, hence, it probably doesn't.

The sandbox doesn't need to be perfect. To exploit a sandboxed parser, you need to find a bug in the parser *and* a bug in the sandbox. That's strictly harder than finding a bug in the parser, so the sandbox makes the system more secure. It's like the most obvious example of defense in depth.

CVE-less vulnerabilities

Posted Jun 28, 2019 20:25 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> There is no way to 'understand' this 'perspective' as it's plainly self-contradictory: Compared to sandboxes, all of which have a history of security issues(!), most parsers are fairly simple.
You clearly live in a separate parallel reality. The never-ending list of critical CVEs in parsers is probably the longest list of vulnerabilities.

CVE-less vulnerabilities

Posted Jun 28, 2019 21:11 UTC (Fri) by rweikusat2 (subscriber, #117920) [Link] (2 responses)

You "clearly" haven't yet managed to move past the shtick of supplanting assertions of insanity for arguments. Parsing your second sentence yields "I [ie, you] don't know if there were more critical errors in [somehow defined] parsers than in order kinds of software". Even assuming the "but it may well be the case" (aka "probably") as given, one can only wonder what this is now supposed to express. You could even be trying to make my point for me: "As parsers are a lot simpler than sandboxes, a lot of bugs in them are eventually found and fixed, as evidenced by the very long list of CVEs relating to such bugs".

But that's really entirely besides the point. The base situation is still "it is unknown if $parser has security issues, hence it probably has" vs "it's unkown if $sandbox has security issues, hence, it's probably safe to use". These are still two appeals to ignorance used to justify both a statement "A" and the inverse statement "not A".

NB: I do not claim that the parser must be safe or that the sandbox must be unsafe, only that absolutely nothing can be validly concluded from ignorance.

CVE-less vulnerabilities

Posted Jun 28, 2019 22:39 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

The situation right now is this. If you have a C-based parser then it contains remote holes, end of story. Old C-based parsers just have the most obvious holes discovered (usualy).

Just look at libpng, libgzip, libjpeg, Apache, libcurl, ... for examples. Every one of them had multiple severe vulnerabilities in parsing.

So if you are using a C-based parser for untrusted data (which in this world means "all data") then you have at least one security hole.

Sandboxes are a way to mitigate this. They still can have bugs but there is just a handful of sandboxes, and some of them are written in languages that preclude most of regular C bugs.

Of course, if you want to be in the perfect world then you should turn off all computers tomorrow and start a Manhatattan-project style mass rewrite of C software in saner languages.

CVE-less vulnerabilities

Posted Jun 30, 2019 10:24 UTC (Sun) by nix (subscriber, #2304) [Link]

> You could even be trying to make my point for me: "As parsers are a lot simpler than sandboxes, a lot of bugs in them are eventually found and fixed, as evidenced by the very long list of CVEs relating to such bugs".

You're suggesting that finding more bugs in some piece of software is evidence that it has fewer bugs than other software in which fewer bugs were found in the first place, and you don't think this might perhaps be motivated reasoning so contorted that you could probably find singularities in it?

Thousands of image and other media parser bugs have been found (and continue to be found), even discounting the endless flood from Wireshark's network protocol dissectors. How many bugs have been found in Chrome's sandbox, which has annual events in which people basically get paid for finding holes in it? Is it as many as a dozen?

CVE-less vulnerabilities

Posted Jun 29, 2019 17:55 UTC (Sat) by sorokin (guest, #88478) [Link] (10 responses)

This were exactly my thoughts, but I couldn't find a better words to express them. Thank you!

I would like to add two more points:
1. Most parsers are single-threaded while most sandboxing environments have to deal with some forms of concurrency. Which means they are difficult to test.
2. In addition it is very easy to write tests for parsers, because normally they behave like pure functions: they have some input and they produce some output.

CVE-less vulnerabilities

Posted Jun 29, 2019 18:39 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

> 1. Most parsers are single-threaded while most sandboxing environments have to deal with some forms of concurrency. Which means they are difficult to test.
Not really. Most of sandboxing environments are thread-agnostic or single-threaded. Unless you're looking at full VMs.

> 2. In addition it is very easy to write tests for parsers, because normally they behave like pure functions: they have some input and they produce some output.
As history has shown, this is not enough.

CVE-less vulnerabilities

Posted Jun 30, 2019 7:47 UTC (Sun) by sorokin (guest, #88478) [Link] (8 responses)

> Not really. Most of sandboxing environments are thread-agnostic or single-threaded. Unless you're looking at full VMs.

OS kernel<->userspace boundary is multithreaded. x86 VMs are multithreaded. Managed languages VMs are multithreaded.

> As history has shown, this is not enough.

Enough for what? I'm telling that one type of programs is easy to write tests for and one is not.

CVE-less vulnerabilities

Posted Jun 30, 2019 7:51 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

> OS kernel<->userspace boundary is multithreaded. x86 VMs are multithreaded. Managed languages VMs are multithreaded.
No. Quite a few sandboxes prohibit launching new threads, including the one used in Chrome, for example. Managed VMs can also be single-threaded just fine - see Python or Nodejs.

> Enough for what? I'm telling that one type of programs is easy to write tests for and one is not.
Enough to avoid security bugs. The only effective way to de-louse the C programs is fuzz testing, everything else simply doesn't work. And fuzzying can't find everything.

CVE-less vulnerabilities

Posted Jun 30, 2019 10:17 UTC (Sun) by sorokin (guest, #88478) [Link] (6 responses)

> No. Quite a few sandboxes prohibit launching new threads, including the one used in Chrome, for example. Managed VMs can also be single-threaded just fine - see Python or Nodejs.

We were talking about sandboxing. Do you know any examples where people rely on python nor nodejs security to run untrusted code? I don't know of any.

I can not comment about Chrome though. I know that it is multithreaded, but perhaps its sandbox runs single-threaded code. I don't know. Ok perhaps this is a example of single-threaded sandboxed environment, still it is an exception. And overtime we will see fewer and fewer single-threaded programs.

> Enough to avoid security bugs. The only effective way to de-louse the C programs is fuzz testing, everything else simply doesn't work. And fuzzying can't find everything.

As well as you need fuzzing to test sandboxes. I would say for now fuzzing is the most effective way of finding bugs in programs.

CVE-less vulnerabilities

Posted Jun 30, 2019 17:48 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> We were talking about sandboxing. Do you know any examples where people rely on python nor nodejs security to run untrusted code? I don't know of any.
You mentioned managed languages first. So stop moving goalposts.

> I can not comment about Chrome though.
The sandboxed code is single-threaded.

You also seem to not understand the whole concept of sandboxes. Sandboxes limit the operations that the contained code can do, whether it's multithreaded or not. And it's not like regular uncontained libraries can't access threads by themselves, either directly or through injected exploit code.

> As well as you need fuzzing to test sandboxes. I would say for now fuzzing is the most effective way of finding bugs in programs.
No. The most effective way is to write software in safe languages. E.g. a sandbox in Rust: https://github.com/firecracker-microvm/firecracker - they had no CVEs so far.

The second most powerful way is to contain code in sandboxes.

CVE-less vulnerabilities

Posted Jul 1, 2019 21:13 UTC (Mon) by samuelkarp (subscriber, #131165) [Link]

> a sandbox in Rust: https://github.com/firecracker-microvm/firecracker - they had no CVEs so far.

Hey, thanks for the shout-out! This is somewhat of a sidebar, but hopefully I can help a bit. While Firecracker is used in production at AWS, it's a relatively young project. We've certainly tried to build Firecracker in a secure way (Rust is one part of that, a very limited device model is another part, and limiting privileges by default with seccomp and jailing is another), but the short amount of time it has been open-source means the security community hasn't had a lot of time to necessarily identify any issues in it. I think this may be a much stronger argument in the future as the project continues to mature and (hopefully!) security-minded folks have an opportunity to look deeper.

(disclosure: I work for AWS and work with the Firecracker team; I am helping to build firecracker-containerd, a project focused on managing container workloads inside Firecracker microVMs.)

CVE-less vulnerabilities

Posted Jul 4, 2019 17:59 UTC (Thu) by Wol (subscriber, #4433) [Link]

> > As well as you need fuzzing to test sandboxes. I would say for now fuzzing is the most effective way of finding bugs in programs.

> No. The most effective way is to write software in safe languages. E.g. a sandbox in Rust: https://github.com/firecracker-microvm/firecracker - they had no CVEs so far.

The only guaranteed secure way to write secure software is to prove it correct. Try doing that with C and its undefined behaviours - the end result will be quantum with bugs popping in and out of existence at random. And they do! I'm pretty sure there have been several occasions when an upgrade to GCC has caused bugs in software to pop INTO existence ... (because the compiler exploited a loophole in the definition of the language)

Cheers,
Wol

CVE-less vulnerabilities

Posted Jul 1, 2019 9:44 UTC (Mon) by james (subscriber, #1325) [Link] (2 responses)

We were talking about sandboxing. Do you know any examples where people rely on python nor nodejs security to run untrusted code? I don't know of any.
Well, this thread started with rra saying:
Personally, I think it's no longer an acceptable security practice to run an image parser on untrusted input outside of a sandbox.
illustrating that sandboxing isn't just for untrusted code -- it's also for mostly-trusted code that is likely to handle hostile data (and where you might not totally trust the language sandbox).

I know it's Perl, but I'd love the ability to run SpamAssassin in a sandbox (without making any complaints at all about either SpamAssassin or Perl security).

CVE-less vulnerabilities

Posted Jul 1, 2019 17:35 UTC (Mon) by sorokin (guest, #88478) [Link] (1 responses)

> illustrating that sandboxing isn't just for untrusted code -- it's also for mostly-trusted code that is likely to handle hostile data (and where you might not totally trust the language sandbox).

I completely agree with that point. In many cases this can be a completely adequate security measure.

Actually my original comment was about testing. Looking back I regret that I even responded to Cyberax who is arguing just for the sake of arguing.

CVE-less vulnerabilities

Posted Jul 4, 2019 18:03 UTC (Thu) by Wol (subscriber, #4433) [Link]

Cyberax is okay. I think we have some people here who prize belief over empirical evidence. At least you can argue with people who are persuaded by evidence.

And while "security through obscurity" is a bad thing to *rely* on - as a deliberate *extra* feature on top of other measures it *is* good. Run a parser inside a sandbox on a hardened kernel - the attacker has to first discover the security measure before he can attack it, which gives you extra opportunity to discover *him*.

Cheers,
Wol

CVE-less vulnerabilities

Posted Jul 8, 2019 5:10 UTC (Mon) by HelloWorld (guest, #56129) [Link]

If there is no bug in the parser, you're not going to be able to break out of the sandbox by sending bad input, whether or not the sandbox has bugs. OTOH if the parser does have e. g. a code execution vulnerability, then the sandbox may or may not be able to mitigate that, but even if it doesn't, that's probably no worse than running the parser without a sandbox in the first place. So no, this isn't magical thinking at all.


Copyright © 2019, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds