FreeBSD and release engineering

On January 16, John Kozubik posted to the FreeBSD-hackers list and expressed his disappointment in some of the recent trends in the project. Namely, an increasingly-slow release cycle, too many overlapping "production" releases, and an estrangement between the core developers and end users when it comes to support issues like bug fixes. The list has since debated Kozubik's assessment of the situation in a heroically long thread, but while the majority agree that FreeBSD would benefit from refocusing its energies and polishing its processes, it has not yet developed a concrete plan of action.

Too many series, not enough releases

Kozubik himself is not a FreeBSD developer; he manages enterprise production environments (including rsync.net) that run almost 1,000 FreeBSD machines. As he explained in his initial message, he was disappointed to hear that the next point release of the OS, 8.3, has been pushed back to March 2012 — more than a year after 8.2. Such a long stretch between minor releases makes maintaining a stable server farm difficult, but paradoxically FreeBSD has also complicated the lives of its end users by simultaneously pushing out multiple "production releases" with different major numbers — at present including versions 7.4, 8.2, and 9.0.

Once a new "major" version is in development, he said, important patches to the previous major version start to languish, as developers lose interest in donating their time to the old code and to less-than-exciting maintenance tasks. As a result, enterprise users face a difficult choice: either go without new features and fixes in the "old" production release, or risk the instability of the "new" production release. He worries that FreeBSD may be "becoming an operating system by, and for, FreeBSD developers" — to the ultimate alienation of users.

Kozubik then outlined five underlying causes that contribute to the alienation problem, and proposes three fixes. First, there is a "widening gap of understanding" between the developers and end users, caused by developers frequently jumping between bleeding-edge snapshots of the code, rather than running the releases tagged as stable for production use. The disconnect makes discussing maintenance issues difficult. Second, maintaining multiple "production releases" simultaneously dilutes developer focus, which keeps the releases from ever truly maturing. Third, the simultaneous production releases drive away potential paid investment from enterprise customers, because they view each FreeBSD release with uncertainty.

Fourth, the slow pace of minor releases means that important fixes often sit unreleased long after they have been verified by maintainers. That not only hurts end users (by depriving them of regression fixes and security updates), but downstream projects as well. Finally, when the slow pace of minor releases is coupled with the multiple-major-releases problem, new code and fixes increasingly get bumped from the "old production" release to the "new production release," solely because the new release is what developers are interested in working on. This traps enterprise users in "the same bad choice again: make major investments in testing and people and processes every two years, or just limp along with old, buggy drivers and no support." Kozubik called this "the culture of 'that's fixed in CURRENT' or 'we built those changes into (insert next major release)'."

He suggested that the project take three steps to ameliorate the trouble. First, intentionally consider the processes and costs incurred by large FreeBSD deployments. "Think about the logistics of major upgrades and the difficulty of running snapshot releases, etc. Remember - if it's not fixed in the production release, it's NOT FIXED. Serious (large) end users have very little use for CURRENT." Second, the project should focus on just one production release at a time, and commit to a definite support schedule (Kozubik suggests five years as a production release, followed by five years as a "legacy" release). Third, in concert with the predictable major release schedule, the project should commit to doing smaller, more frequent minor updates, around three times per year.

Dissecting the problem

A majority of the people who joined in the discussion agreed that something like what Kozubik proposed would be beneficial. Opinion varied, however, on what the underlying causes are, and whether or not they can be fixed in the short term. Rich Healey observed that there are very few paid FreeBSD developers, and while volunteers have little motivation to undertake the unexciting maintenance tasks, the paid developers that exist are often contracted to implement specific new features.

Warner Losh said that no corporate sponsor has been pushing the project to keep the release process on schedule since the demise of Walnut Creek. Julian Elischer responded by suggesting that interested high-volume customers and downstream vendors could pool their resources and pay a developer to work specifically on release management — a suggestion that was met with approval.

FreeBSD release engineer Robert Watson described the dilemma as a release engineering problem — with several causes — that he and the other release team members could address. Historically, he said, the FreeBSD base and ports collection were on a single, tightly-coupled release cycle — which often resulted in ports getting very little attention. In the early days, he added, the project used CVS version control, which made branching very expensive. Last but not least, the project has come to rely on a single "head release engineer" to steer all of the release schedules, which results in bottlenecks and slipping release dates. The CVS problem has been addressed by moving to Subversion, he said, and there is growing support for de-coupling the base and ports releases, but the project still needs to fix the single-release-engineer issue. Watson suggested mentoring-in new release managers from the developer community, each of whom would take responsibility for one minor release.

On the other hand, Igor Mozolevsky argued that the FreeBSD Problem Report (PR) patch system is broken, because it effectively requires end users to undertake a nagging campaign to even get a patch examined by a developer. That drives away users and is clearly sub-optimal for the project. Not everyone agreed with that assessment, although Matthew Fuller cited an example of a manpage fix collecting dust for three years.

Adam Vande More speculated that a bounty system would motivate quicker PR merges — but Matt Olander from iXsystems pointed out that he has set one up already. Watson again suggested changes that the project could make to its existing processes, starting with replacing the GNATS issue-tracker that no one seems to like, and adopting a more formal policy for trawling PRs and triaging bug reports.

Pushback

Not everyone was on board with Kozubik's reading of the situation, however. Ivan Voras disagreed completely, saying that "the situation is actually quite good", and that "nobody would mind" if there were no more stable releases at all. "The 'releases' are for many people simply a periodical annoyance due to freezes," he said. Kozubik replied that "I could not have illustrated my point better, RE: FreeBSD becoming an OS by, and for, FreeBSD developers", and explained that most businesses will simply not use software that is not "officially" released.

Andriy Gapon defended the concept of a "by the developers, for the developers" mindset as the equivalent to being a community project. Projects that exist "for the users," he argued, tend to exist for commercial purposes, and be backed by profit-driven corporations. FreeBSD is more akin to the Debian project, he said, which is very much a "for the developers" effort, but still quite successful.

Adrian Chadd took issue with the question being raised at all, saying:

But Doug Barton quickly responded "let's do away with the whole, 'If you step up to help, your help will be accepted with open arms' myth. That's only true if the project leadership agrees with your goals." The FreeBSD project needs to do some serious thinking about things like the role of "committer" and the meaning of "release," he said, including the difference between major and minor releases, all of which are terms with no formal definition. "We also need to take a step back and ask if throwing more person-hours at the problem is the right solution, or if redesigning the system to better meet the needs of the users *as a first step* is the right answer."

Engineering release-engineering

Barton suggested defining minor "point releases" as updates only to the FreeBSD base, not the ports collection or documentation. "The other thing I think has been missing (as several have pointed out in this thread already) is any sort of planning for what should be in the next release," he added. The current release schedule is a hold-over from the trouble-filled days the project experienced in the FreeBSD 5.x era, he said, but "the pendulum has swung *way* too far in the wrong direction, such that we are now afraid to put *any* kind of plan in place for fear that it will cause the release schedule to slip."

Watson concurred, adding that ""here's been an over-swing caused by the diagnosis 'it's like herding cats' into 'cats can't be herded, so why try?'" — and asking list members if they could come up with a tentative release schedule.

The debate continues, without a consensus yet on a release schedule. For his part, Kozubik favored immediately declaring FreeBSD 7.z end-of-life, marking 8.x as "legacy," and tagging 9.x as the only production release. Not everyone agrees with Kozubik's five-production-years-plus-five-legacy-years timetable; though there is support for a five-year total support life, and most developers seem to think that making minor releases every four months is doable.

Such a schedule would be roughly in line with what the commercial Linux distributors offer, and as Freddie Cash observed, the maintenance of a production release alongside a legacy release is similar to the process used by Debian. The big question remains whether the project will be able to hash out all of the details in time to commit to a concrete release schedule for 9.x itself — or whether the new release process will have to wait for FreeBSD 10.0.