CVE-less vulnerabilities

Isn't running buggy / vulnerable code also an unexpected breakage?

> You must be making some kind of argument that there's a counterbalancing force, right? The bugs that aren't fixed because of a lack of change? Something else?

My apologies, I don't think I have expressed myself well enough.

Many of the recent attention grabbing bugs also exist in earlier versions, though fixing them takes more time there since newer versions take precedence. A lot of the old code that hasn't been touched in a while may also contain errors that don't occur in newer versions since that code has been replaced. Tooling has massively improved in recent years which means that newer code is probably better tested than a lot of the old code that is working though may have issues. There are far more eyes on new code at the moment. For a developer, working on features is probably more fun than debugging and fixing code which is still a lot more fun than backporting those fixes to software versions you don't personally run - this is often paid work or in some cases automated. Long term support is usually a promise, not a guarantee.

When you encounter and report a bug for an older version the first question will usually be "can you reproduce it on current stable / devel?"

> It may be good practice in your mind to have a well-designed upgrade strategy, but I don't see a lot of rationale for datacenters to update their systems all the time.

I think there are good arguments for doing this and treating it as a part of operating routine instead of having Patch Days or regular update marathons. Doing this regularly builds institutional knowledge, establishes good patterns, encourages automation and spreads the workload over time.

> What problem does a strategy like upgrading every 3 months cause?

3 months isn't all that bad, I've seen a lot of places that do worse - most don't have an upgrade strategy or they have a strategy that is documented and audited but not executed. In my experience that date is a moving target, the longer people don't upgrade the lower their confidence is in their ability to upgrade, planned upgrades keep getting rescheduled. Once you automate these procedures you have to still do the procedure at regular intervals, otherwise you'll lose confidence in your tools or they just stop working. A lot of this work just ends up not being done since it's risky, doesn't offer an immediate reward (though it may sometimes fix user problems) and the risks don't manifest themselves.

We haven't heard many horror stories yet but I think we'll see a lot more of that in the coming years since breaking into systems has become lucrative as a criminal business model. And it's often not the Shellshocks, Heartbleeds, Spectres or Meltdowns that cause it, it's trivial and old bugs (MS Office Macros). These will only become more sophisticated over time as criminals go from targeting users directly to targeting servers.

Would you be confident running a 3 months old browser? Looking at the history of Firefox ESR it has seen almost as many upgrades as the stable series. You're upgrading anyways, why keep the old version? Because it has less features and gets less attention from developers?

> All these cattle strategies like blue/green and balloons and so on are pretty neat, and help enable faster work, but they seem better tuned towards running your own software as opposed to running off the shelf applications, especially when there's significant state involved.

This I agree with, a lot of work has been focused on the somewhat easy target of stateless software, once you introduce state people start fearing the upgrade again (or just say "let $PROPRIETARY_CLOUD worry about it"). This is even worse with commercial/proprietary software since you're entirely dependent on the vendor who of course has little incentive supporting multiple platforms or be at all friendly to other people operating the software. They have a captive audience (you can't just switch out your RDBMS). Prices often forbid having a fail-over system or heavens forbid a test platform since executives often don't understand the risk - if it hasn't failed the last year there really is no need to have a backup. It's like driving drunk to save on taxi money. Technical debt is easily externalized.

Either go fully SaaS and have SLAs with severe penalties or invest into switching to software that is less of a burden to maintain.