Leading items

Welcome to the LWN.net Weekly Edition for November 4, 2021

This edition contains the following feature content:

• Adding package information to ELF objects: extra metadata in executables may help developers, but not everybody is in favor of this proposed Fedora change.
• Trojan Source: tricks (no treats) with Unicode: a look at the latest heavily promoted vulnerability.
• Some 5.15 development statistics: where the code in 5.15 came from.
• Debian's which hunt: changes to the packaging of the "which" utility stir up passions in the Debian camp.
• Fedora considers removing NIS support: the writing is on the wall for an old approach to network-administration databases.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Adding package information to ELF objects

While it is often relatively straightforward to determine what package provided a binary that is misbehaving—crashing for instance—on Fedora and other Linux distributions, there are situations where it may be harder to do so. A feature recently proposed for Fedora 36—currently scheduled for the end of April 2022—would embed information into the binaries themselves to show where they came from. It is part of a multi-distribution effort to standardize how this information is stored in the binaries (and the libraries they use) to assist crash-reporting and other tools.

On October 25, Fedora program manager Ben Cotton posted the proposal to the Fedora devel mailing list; it is also available on the wiki. The basic idea is that each ELF object that gets created for an RPM package will get a .note.package ELF section added to it. That section will contain a JSON-formatted description of exactly which RPM it was distributed with. So those binaries will contain information that can tie them directly to the package, even in the absence of RPM metadata on the system.

The facility would be used by the systemd-coredump utility to log package versions when crashes occur. For regular Fedora systems, which normally have the RPM metadata available, there is no large advantage. But for other situations where Fedora-created binaries might be run—and crash—this mechanism would allow administrators and tools to recognize where exactly the binary came from.

The feature was originally proposed back in April for Fedora 35, but was rejected by the Fedora Engineering Steering Committee (FESCo) with an explicit invitation to resubmit it with "a more detailed and understandable 'Benefit to Fedora' section". So the two feature owners, Zbigniew Jędrzejewski-Szmek and Lennart Poettering, added more information to the proposal for the resubmission. The new "Benefit to Fedora" section details why it makes sense for the distribution:

A simple and reliable way to gather information about package versions of programs is added. It enhances, instead of replacing, the existing mechanisms. It is particularly useful when reporting crash dumps, but can also be used for image introspection and [forensics], license checks and version scans on containers, etc.

If we adopt this in Fedora, Fedora leads the way on implementing the standard. Fedora binaries used in any context can be easily recognized. Fedora binaries provide a better basis to build things.

If other distros adopt this, we can introspect and report on those binaries easily within the Fedora context. For example, when somebody is using a container with some programs that originate in the Debian ecosystem, we would be able to identify those programs without tools like `apt` or `dpkg-query`. Core dump [analysis] executed in the Fedora host can easily provide useful information about programs from foreign builds.

When a program crashes, there is already an identifier that can be used: the build ID that is stored in the .note.gnu.build-id ELF section. But that ID is a long hexadecimal string that is not terribly useful to a human. In addition, the ID can only be related back to the RPM it came from by using the RPM database installed on the system or by doing some sort of network query. An example in the proposal shows how the human-readable JSON in note.package might look instead:

    {
      "type": "rpm",
      "name": "hello",
      "version": "0-1.fc35.x86_64",
      "osCpe": "cpe:/o:fedoraproject:fedora:33"
    } 

The proposal notes that the "directly motivating use case is display of core dumps". The build ID could be used if the RPM database is present, but, even then, there can be problems. It is not uncommon for the package containing a crashing program to have been upgraded behind the scenes, so the installed binary is different than the running one; other mishaps are also possible, so that correspondence cannot be assured. Also, crashes that happen in environments without the database, running from initrd or a sandboxed container for example, can use the JSON note to extract the exact versions of each component involved in the crash.

For users who build their own packages, once again the human-readable information will be more useful than the build ID, which would need to be maintained in some kind of database to map the ID to the source version. In addition, binaries are sometimes pulled from Fedora for use in other distributions—and vice versa. Being able to easily find out where a binary came from will be useful in those cases too:

Whilst most distributions provide some mechanism to figure out the source build information, those mechanisms vary by distribution and may not be easy to access from a "foreign" system. Such mixing is expected with containers, flatpaks, snaps, Python binary wheels, anaconda packages, and quite often when somebody compiles a binary and puts it up on the web for other people to download.

David Cantrell had a few questions about the proposal. He wondered why Fedora should care about mixing-and-matching its binaries on other systems; if there is no way to reproduce the problem on a vanilla Fedora system, bug reports are not likely to be entirely useful. Poettering said that having the information will be useful because it will help show that the problem happened in a mixed system. That will give Fedora the opportunity to either try to reproduce the bug, perhaps in conjunction with the other distribution, or at least allow it to be "more efficient with 'not caring' for non-fedora issues".

Cantrell also asked about whether the "NEVRA" (name-epoch-version-release-architecture) package information is sufficient, because it may not be unique and wondered if the build ID plus debuginfod servers would be enough. Debian developer Luca Boccassi noted that access to the network is not a given, nor is it desirable from the sandbox that systemd-coredump runs in. Adding the URL for the debuginfod information to the package note is a possibility, as well.

There is also a privacy issue to consider Jędrzejewski-Szmek said: "querying debuginfo servers may expose information (about what is running, in what versions, what is crashing, etc.) Thus such queries need to be opt-in and under user control." He also pointed out that the Fedora Koji build system ensures that the NEVRA information is unique for the packages it creates.

Kevin Kofler had a number of objections to the feature, however. In effect, he was objecting the whole use case of running Fedora binaries in environments where the RPM database was not present. He also claimed that the licenses for the code were being violated when pulling binaries out of RPMs without providing the source code. But the licensing question is largely irrelevant, Jędrzejewski-Szmek said; in some cases there may be a license violation, but in lots of others there is not. The package information in the binary will actually help figure out when there is a problem of that sort, he continued.

Not having an RPM database results in a non-functional Fedora installation, Kofler said: "how can this not be broken?" But lacking an RPM database is rather common, especially in the container world, Daniel P. Berrangé said. It would sometimes be useful to have that database available, but it is not a high priority in container-land, which does not mean the use case is broken: "It is simply a different approach / attitude / tradeoff towards using & maintaining the software stack."

"Bloating" all of the ELF objects in Fedora to support the use case is not reasonable, though, Kofler said. The proposal notes that the overhead is around 200 bytes per ELF object, which results in an increase of 13MB if every object in the distribution had the package information added to it. Since Kofler does not seem to accept that the use case is valid, any increase to support it is unnecessary in his eyes. But Boccassi pointed out that cost is tiny to support a use case that is prevalent:

[...] it has happened, it is happening and it will keep happening, because for others it is perfectly logical and highly desirable. So one can either stay here and complain all day long that containers are bad and they are all doing them wrong, and if they only listened to reason everything would be just perfect, or one can do something to significantly improve the baseline for everybody at a cost so ridiculously negligible that if the same standard were applied to compiler updates or changing build flags or whatnot nothing would ever, ever change.

Furthermore, Poettering noted that even vanilla Fedora systems have a piece without an RPM database:

You too run a system with no RPM database – all the time, and that thing still calls itself Fedora: a dracut initrd is exactly that: built from RPMs but without any RPM db.

Thing is, there are different ways to update stuff. rpm/dnf is one thing, dracut image rebuilds is another, containers are typically updated very differently too. rpm is a useful tool (and by embedding rpm meta info into the ELF objects it becomes even stronger), but your assumption that rpm/dnf based updates is the only right way to upgrade stuff is simply neither reality nor even desirable.

A possible security issue with the proposal was also raised by Cantrell. He wondered if there might be problems with using JSON, which has been the source of some security problems in the past. "Of concern to me are encoding formats, size limits or reporting, and structure formats." Poettering said that JSON was chosen because of the "battle-tested parsers" that are already used in systemd and elsewhere. Jędrzejewski-Szmek added that "the implementation in systemd is undergoing continuous fuzzing in oss-fuzz", so there is reason to hope that many of the parser bugs have been found and fixed.

Overall, the reception was largely favorable, though there are some concerns. It is a change with fairly minor effects on binaries—200 bytes hardly seems onerous—that can help in a number of scenarios. It also work toward a cross-distribution effort. Microsoft's CBL-Mariner container distribution has added support for the feature and it will be proposed for Debian; others may well follow suit. It will be up to FESCo, of course, but the objections and concerns do not seem to offset the benefits that it will bring.

Comments (63 posted)

Trojan Source: tricks (no treats) with Unicode

A new security vulnerability that was disclosed on November 1 has some interesting properties. "Trojan Source", as it has been dubbed, is effectively an attack on human perceptions, especially as they are filtered through the tools used for source-code review. While the specifics of the flaw are new, this kind of trickery is not completely novel, but Trojan Source finds another way to confuse the humans who are in the loop.

The Trojan Source paper by Nicholas Boucher and Ross Anderson of the University of Cambridge describes the vulnerability, its impact, and the process of coordinating its disclosure in detail. In part, the flaw exploits Unicode code points that are used to switch between left-to-right and right-to-left display of bidirectional text in ways that will cause various tools to show the code in a different order than it will be processed by compilers and interpreters. That has the effect of showing code to reviewers that looks perfectly reasonable, while feeding something dangerous to language-parsing tools.

Homoglyphs

There are two separate vulnerabilities, with variants of each, that are under the Trojan Source umbrella. One is somewhat similar to the "typosquatting" problems that occur in package repositories for various languages or the deceptive URLs that are sometimes used by phishers to confuse users (e.g. replacing "l" with "I"). This "homoglyph" vulnerability (CVE-2021-42694) uses Unicode code points that are visually similar for defining function or variable names.

An example in the paper defines two functions, one with a Latin "H" and another with the Cyrillic "Н". Having both functions in the same code base might be a red flag, but the paper notes that a malicious library might define both with different effects; an attacker could then make a seemingly benign change that called the malicious version:

Although this example program appears harmless, a homoglyph attack could cause significant damage when applied against a common function, perhaps via an imported library. For example, suppose a function called hashPassword was replaced with a similar function that called and returned the same value as the original function, but only after leaking the pre-hashed password over the network.

Bidirectional attacks

The more headline grabbing vulnerability, CVE-2021-42574, is the one that exploits the bidirectional text code points. It can have various effects, but the one demonstrated on the Trojan Source web page (and in the paper) effectively extends a comment to completely contain the entire line, which otherwise appears to be reasonable:

    /* begin admins only */ if (isAdmin) {

By using some trickery with Unicode, the whole line gets commented out, removing the (seemingly rather important) test for administrative access. It is followed up by another line that looks like it contains the closing brace, but, once again, does not actually do so:

    /* end admin only */ }

There are a number of Unicode code points that can be used to effect various changes in the ordering of subsequent code points. For the example, four of these were used:

• RLO (U+202E): treat the following text as right-to-left
• LRI (U+2066): treat the following text as left-to-right, without affecting the surrounding text
• RLI (U+2067): treat the following text as right-to-left, without affecting the surrounding text
• PDI (U+2069): terminate the most recent LRI or RLI

The paper shows how those operate:

As an example, consider the following Unicode character sequence:

RLI a b c PDI

which will be displayed as:

c b a

LRI and RLI both create "isolates", which are groups of characters that get treated as a block. The PDI terminates the isolate, as does a newline, implicitly. These isolates can be nested, which the paper shows as well:

For example, consider the Unicode character sequence:

    RLI LRI a b c PDI LRI d e f PDI PDI

which will be displayed as:

    d e f a b c

As can be seen, this gives plenty of ways to obfuscate the actual sequence of code points. In the commenting-out example above, the actual sequence of code points in the source file are as follows:

    /*RLO } LRIif (isAdmin)PDI LRI begin admins only */
        ...
    /* end admin only RLO { LRI*/

The braces in the real text are reversed from what might be expected; that is because symbols that have an implicit direction, such as braces, are reversed within right-to-left sections. The end result is that an editor or code-review tool shows the "harmless" version, but the compiler, which is not particularly interested in processing format-control characters inside comments, sees both lines as comments, so any code between is run, no matter what the isAdmin flag is set to.

Note that the final isolate in both cases is unterminated within the comment; they cannot be for the attack to work, but an unterminated isolate within a comment is something that a compiler could warn about. As the paper mentions, arbitrary format-control code points in the program code itself are normally not allowed by compilers, but comments and string constants provide places where they are, and arguably should be, allowed. But, as seen, that can lead to these kinds of problems. Another example in the paper uses a Python docstring with the word "return" placed in it, seemingly innocuously; as might be guessed, though, reordering the string can lead to an immediate return from the function.

Validation and mitigation

The researchers validated these kinds of attacks by creating proof-of-concept attacks for the homoglyph flaw and several variants of the bidirectional-abusing attacks in a bunch of different languages: C, C++, C#, JavaScript, Java, Rust, Go, and Python. That code is available on GitHub and in the appendices of the paper. They also tested seven different editors and web-based code-viewers (on the three major operating systems) to see which were affected. For the most part, the homoglyph attack was displayed as "expected" (i.e. confusingly) on nearly all, while the bidirectional attack was somewhat more hit-and-miss; for example, Vim and SublimeText were both not completely "fooled" by the trick. The entire test matrix can be seen as the final appendix in the paper.

Completely disallowing the bidirectional override code points within program source is, obviously, one way to "fix" the problem, but that cure may be worse than the disease, especially for strings. Another defense mentioned in the paper is to disallow unterminated bidirectional overrides within strings and comments, which would ensure that any reordering stays within the string or comment and cannot affect adjacent code. For homoglyph attacks, compilers should emit warnings or errors, as gcc, g++, and the Rust compilers already do. Beyond that, programs that display code should make the bidirectional overrides visible in some fashion, so that code reviewers can immediately see potential problems of that sort.

In the meantime, though, there is a need to protect against these kinds of problems when using existing tools, or in languages that choose not to implement defenses. The paper recommends adding scanners to build pipelines that will refuse to continue (or at least warn) in the presence of these unterminated overrides.

Disclosure

The problems were initially disclosed to 19 organizations on July 25 under a 99-day embargo. "We met a variety of responses ranging from patching commitments and bug bounties to quick dismissal and references to legal policies." Several of those initial recipients asked that others be included, which was also done. The paper reported on the bounties that were paid, which would seem to have netted over $11,000:

Eleven of the recipients had bug bounty programs offering payment for vulnerability disclosures. Of these, five paid bounties, with an average payment of $2,246.40 and a range of $4,475.

On September 9, the flaws were disclosed to the computer emergency response team coordination center (CERT/CC) and, on October 18, the two CVE numbers were issued. At that point, the distros mailing list was contacted; its members are representatives from various Linux distributions and other Unix operating systems. The maximum embargo for problems reported to distros is 14 days, thus the public disclosure on November 1.

In addition, the researchers were curious about whether these kinds of problems were actually found in the wild. To that end, they were able to gather some data from GitHub:

We assembled a regex that identified unterminated Bidi [bidirectional] override sequences in comments and strings, and GitHub provided us with the results of this pattern run against all public commits containing non-markup language source code ingested into GitHub from January through mid October 2021.

While they did find some indication that similar techniques had been used in various ways, they found no smoking guns. For example, the Slither static-analysis tool for smart contracts checks for bidirectional overrides, noting that they can be used to swap variables passed as arguments. Another use was for obfuscating JavaScript code. In parallel to the GitHub scan, Rust project contributors did a scan of crates.io code, which is the public package repository for Rust, and found no evidence of exploitation of these flaws.

While these are serious problems, they do not seem quite the like the doomsday event that some press coverage (or at least headlines) makes it out to be. It is a clever attack, for sure, and one that could lead to some unpleasant outcomes, but there are some elements of fragility to the techniques; widespread exploitation would surely have encountered someone running an "off-brand" editor or code-review tool who would have been alerted. On the other hand, targeted attacks against a few well-chosen components might well have slipped through—and still could, perhaps.

As David A. Wheeler noted in a lengthy LWN comment, there is a fair amount of "prior art" in this area, going back to various obfuscated and underhanded code contests. The paper references others, too. There are also well-known language pitfalls that can change code in a malicious, but easy to overlook, fashion, such as the attempt to backdoor the kernel that was mentioned in the paper; it exploited the visual similarity of the = and == operators in C. None of that is meant to take anything away from what the Trojan Source researchers have achieved, of course; we all stand on the shoulders of giants, after all.

Comments (38 posted)

Some 5.15 development statistics

The 5.15 kernel was released on October 31, with the code name appropriately changed to "Trick or Treat". By that time, 12,377 non-merge changesets had been merged into the mainline, adding a net total of 332,000 lines of code. Read on for a look at where the contributions to the 5.15 kernel came from.

Long-time readers of these summaries will note that the changeset total for 5.15 is relatively low; indeed, it's worth comparing it to the kernel's release history since 5.0 came out in early 2019:

Release	Changesets	Developers
5.0	12,808	1,712
5.1	13,034	1,707
5.2	14,024	1,716
5.3	14,605	1,846
5.4	14,619	1,802
5.5	14,350	1,885
5.6	12,665	1,712
5.7	13,901	1,878
5.8	16,306	1,991
5.9	14,858	1,914
5.10	16,308	1,971
5.11	14,340	1,912
5.12	13,015	1,873
5.13	16,030	2,062
5.14	14,735	1,912
5.15	12,377	1,797

As can be seen, 5.15 is, with regard to changesets merged, the quietest development cycle we have seen in the 5.x era. Indeed, one needs to go back to the 4.7 release in August 2016, which had 12,283 changesets from 1,582 developers, to find a quieter one. The relative slowness of this cycle is especially interesting considering that 5.15 will almost certainly be the next long-term support release. Kernel developers have often worried about vendors trying to shove unready code in for LTS releases, but there is little evidence of that happening here.

Instead, perhaps, the development community is just taking a well-deserved mid-pandemic break.

Individual contributors

Meanwhile, the number of developers participating in this release, while down from recent cycles, is higher than for some others in the 5.x era. Of the 1797 developers contributing to 5.15, 251 did so for the first time; that is a relatively low number but not out of line with previous releases. The most active developers in this cycle were:

Most active 5.15 developers

By changesets Christoph Hellwig 252 2.0% Namjae Jeon 171 1.4% Takashi Iwai 161 1.3% Vladimir Oltean 138 1.1% Colin Ian King 131 1.1% Arnd Bergmann 119 1.0% Andy Shevchenko 109 0.9% Thomas Zimmermann 108 0.9% Geert Uytterhoeven 100 0.8% Michael Straube 92 0.7% Linus Walleij 91 0.7% Krzysztof Kozlowski 91 0.7% Pavel Begunkov 83 0.7% Sean Christopherson 82 0.7% Thomas Gleixner 82 0.7% Dan Carpenter 80 0.6% Phillip Potter 79 0.6% Bart Van Assche 76 0.6% Randy Dunlap 73 0.6% Christophe JAILLET 73 0.6%

By changed lines Phillip Potter 137584 17.0% Namjae Jeon 42483 5.2% Konstantin Komarov 32228 4.0% Christoph Hellwig 22177 2.7% Jin Yao 16772 2.1% Hu Haowen 13102 1.6% Trevor Wu 12407 1.5% Linus Walleij 11863 1.5% Zack Rusin 11163 1.4% Lukas Bulwahn 9211 1.1% Konrad Dybcio 8728 1.1% Takashi Iwai 7994 1.0% Vladimir Oltean 7024 0.9% Arnd Bergmann 6965 0.9% Fabio Aiuto 6864 0.8% Kumar Kartikeya Dwivedi 5493 0.7% Quinn Tran 5319 0.7% James Smart 5287 0.7% Matthew Brost 5219 0.6% Iskren Chernev 5176 0.6%

Christoph Hellwig was the top contributor of changesets this time around; he continues his extensive refactoring of code in the block and SCSI subsystems, with occasional side trips to, for example, remove set_fs() from the m68k architecture. Namjae Jeon contributed the new ksmbd filesystem implementation, Takashi Iwai continued his work as the sound subsystem maintainer, Vladimir Oltean contributed a lot of network-driver improvements, and Colin Ian King did minor cleanups throughout the kernel tree.

The developer with the most changed lines this time around was Phillip Potter, who added the rt8818eu wireless network driver. Konstantin Komarov contributed the ntfs3 filesystem driver, and Jin Yao added perf events support for a number of newer Intel subarchitectures.

The kernel project depends on its testers and reviewers as much as it depends on its developers. For the 5.15 cycle, the developers with the most test and review credits were:

Test and review credits in 5.15
Tested-by Daniel Wheeler 62 8.2% Babu Moger 24 3.2% Dvora Fuxbrumer 21 2.8% Nathan Chancellor 13 1.7% Stefano Stabellini 13 1.7% Christophe Leroy 12 1.6% Will Deacon 12 1.6% Gerald Schaefer 12 1.6% Lukasz Majczak 12 1.6% Marc Zyngier 11 1.4% Michael Schmitz 10 1.3% Marek Vasut 10 1.3% Yi Zhang 10 1.3% Guenter Roeck 9 1.2% Michael Walle 9 1.2%	Reviewed-by Christoph Hellwig 211 3.9% Rob Herring 154 2.9% Darrick J. Wong 101 1.9% Laurent Pinchart 92 1.7% David Sterba 69 1.3% Hannes Reinecke 68 1.3% Hans de Goede 60 1.1% Daniel Vetter 59 1.1% Linus Walleij 56 1.0% Alex Deucher 53 1.0% Andy Shevchenko 52 1.0% Stephen Boyd 50 0.9% Lad Prabhakar 50 0.9% José Roberto de Souza 49 0.9% Christian König 48 0.9%

Many of the people who accrue test credits in the kernel tend not to be otherwise visible within our community. What we are seeing is tags applied during the internal review process at some companies; many companies have such processes, but they don't all document that work in the patch tags. A similar thing often happens with the review tags, but that is not the case this time around. The top reviewers take on a number of roles in the kernel project; remember that the most active reviewer this time around (Christoph Hellwig) was also the most active contributor of changesets.

Employers

Work on 5.15 was supported by 213 employers that we were able to identify. The most active employers were:

Most active 5.15 employers

By changesets Intel 1216 9.8% (Unknown) 958 7.7% Google 718 5.8% (None) 687 5.6% Red Hat 633 5.1% Linaro 577 4.7% SUSE 553 4.5% AMD 504 4.1% Huawei Technologies 465 3.8% NVIDIA 368 3.0% IBM 349 2.8% (Consultant) 306 2.5% Facebook 305 2.5% Canonical 275 2.2% NXP Semiconductors 264 2.1% Oracle 255 2.1% Renesas Electronics 236 1.9% Samsung 214 1.7% Arm 193 1.6% Linutronix 147 1.2%

By lines changed (None) 166216 20.5% Intel 79492 9.8% (Unknown) 50642 6.2% Samsung 46431 5.7% Linaro 35003 4.3% Paragon Software 32228 4.0% Red Hat 24395 3.0% (Consultant) 23610 2.9% AMD 22864 2.8% NVIDIA 21705 2.7% Google 20215 2.5% MediaTek 19706 2.4% SUSE 19547 2.4% Facebook 13960 1.7% VMWare 13518 1.7% SoMainline 11581 1.4% Huawei Technologies 11070 1.4% Renesas Electronics 10557 1.3% NXP Semiconductors 10443 1.3% Broadcom 10202 1.3%

As usual, there aren't many surprises here. Paragon Software, which has not been seen on these lists before, shows up as the result of the ntfs3 contribution. Another thing worthy of note is that Huawei contributed 465 changesets to 5.15 — a substantial contribution but also big decrease from the company's 1,731 changesets in 5.14. Had Huawei sustained the 5.14 level of activity, this development cycle would have been comparable to its predecessors in terms of changesets merged. That said, almost all of the companies listed above had lower levels of contribution to 5.15 compared to 5.14.

Code longevity

Finally, it can be interesting to look at how much code from each release has survived into the current kernel. This can be done with the brute-force technique of running git blame on each file in the kernel tree, then mapping the commit that created each line to the release in which it appeared. If one does this for the current Git history, the result looks like:

The code added in most releases makes up less than 2% of the code that is found in the 5.15 kernel; 5.15 itself accounts for 2.1% of the total. At the beginning of the Git era, just before the 2.6.12 release in 2005, the kernel had about 10.8 million lines of code. Over 16 years later, about 2.3 million of those lines (just under a quarter) still exist in 5.15, making up about 7.2% of the total. If one actually looks, much of that code takes the form of individual lines reading "/*" and the like, but that is not all of it. There are, for example, still 72 files in the kernel tree that have never been changed since they were added as part of the initial commit in 2005.

As for the other releases that stand out in the plot, there is usually a fairly straightforward explanation. 4.12, for example, included one of many large dumps of amdgpu register definition header files; those tend to live forever and are, probably, read by nobody. The same is true of 5.3 and 5.9, among other releases. In other cases it is not so clear; 5.6 saw the addition of the ath11k network driver and the WireGuard VPN, but those are not enough to explain why so much code persists from that release.

How much of the new code from the 5.15 development cycle will remain in coming years is yet to be seen. As the plot above shows, though, kernel code has a certain longevity; much of it will be around until it is all rewritten in Rust someday. Expect the development community to continue adding and changing code at a high speed between now and then, though.

Comments (10 posted)

Debian's which hunt

One does not normally expect to see a great deal of angst over a one-page shell script, even on the Internet. But Debian is special, so it has been having an extended discussion over the fate of the which command that has been escalated to the Debian Technical Committee. The amount of attention that has been given to a small, nonstandard utility shines a light on Debian's governance processes and the interaction of tradition with standards.

The which command will locate an executable program within the current search path and print the result:

    $ which nethack
    /usr/bin/nethack

This long-present tool is often used at the command line to locate the binary for a program; scripts also use it for similar purposes, or to determine whether a given program is available at all. For many users, which has long been baked into muscle memory and is used reflexively at need.

For all that, which is not a standardized component on Unix-like systems; POSIX does not acknowledge its existence. For that reason, among others, there are a number of implementations of which, each differing in its own special ways. Many distributions ship the GNU version of which, for example, with its characteristic long list of options. FreeBSD has its own version. Some shells also implement which as a built-in command. Debian ships yet another version, in the form of the aforementioned one-page shell script; it is part of the debianutils package.

In August 2020, Erik Gustafsson noted that the FreeBSD version of which supports a -s flag that suppresses the printed output and sets the exit status based on the existence of the queried program. He thought that feature would be useful in Debian, and helpfully provided a patch adding that feature. Thus began the discussion of the value of which and whether Debian's version should gain more features; at one point Clint Adams, the co-maintainer of debianutils, opined that which should be removed from that package.

Fast-forward to one year later, and Boyuan Yang observed that the which command in the Debian unstable distribution now prints a deprecation warning saying that which is going away. This resulted in a fair amount of consternation (and requests for a reversion of the change) for a number of reasons, starting with the fact that many users simply expect to have which available to them. It turns out that a number of build scripts for Debian packages use which as well; as an extra annoyance, the printed deprecation warning breaks the build process for some packages. The amount of pressure applied to Adams to restore which began to increase.

Adams has a number of reasons for wanting to remove which from debianutils. The POSIX-blessed way of finding an executable program is command -v, which is consequently built into most shells. Given the standard alternative, Adams said, "surely no one competent would choose to have a package depend on `which` when a standard POSIX utility can do a better job". Given that there are numerous variants of which, each with its own advocates, it makes little sense to ship one version in a package marked "essential" (meaning that it must be installed on every Debian system). The Debian Developer's Reference manual once suggested using which in maintainer scripts, but that suggestion was removed in 2018 in favor of command -v. So the which command, Adams argued, is certainly not essential; even if Debian ships it at all, it should not be part of an essential package.

In the end, Adams argued, there are several paths that could be followed regarding which, none of which can please everybody. The path he has chosen is, in his opinion, the best one:

Another is to build a runway for people to get what they want and remove myself from the equation. This is what I have attempted to do. Now, provided anyone elects to maintain them and the FTP team allows them in, users will be able to install and use GNU which, *BSD which, busybox which, rewrite-it-in-rust which, or whatever should float someone's boat. I can think all of this is pointless, and it doesn't matter, because I am not impeding anyone else in their pursuit of which(1) apotheosis.

While most Debian developers do not appear to be opposed to the idea of gracefully moving which out of the debianutils package, many of them disagree with how that is being done. As laid out by Wookey, this change, if done with a bit more care, could avoid many of the problems that are being seen now:

A proper transition plan would mean that I would never even notice this. One which would replace another and nothing would break. That is the sort of thing I expect to happen in Debian - we are generally good at this stuff, and it's a big reason for using the distro.

In mid-September, Adrian Bunk took the matter to the Technical Committee, asking that Adams be overridden in a number of ways. Specifically, he requested a decision that which be provided by an essential package (he didn't say which one), that it not be allowed to print deprecation warnings, and that it not be provided via the "alternatives" mechanism. "Alternatives" is how Debian allows users to choose between multiple implementations of a command; it would seem to fit the bill here, given the number of versions of which that exist. Bunk said that was not an option because it could break upgrades if a maintainer script tries to use which before the package providing it is completely configured.

Bunk's message also asked for two other opinions from the Technical Committee, neither of which generated as much heat. One was to mandate that debianutils must continue to provide tempfile, which is also on the chopping block; the other was that debianutils programs cannot be moved into /usr. That last request ties into another big debate within Debian over the ongoing implementation of the /usr merge transition. The committee issued a lengthy ruling on that subject on October 18, essentially saying that packages cannot count on a merged /usr until after the upcoming Debian 12 release.

Early discussions within the Technical Committee seemed to lead to a relatively quick consensus that, indeed, debianutils must continue to provide which at least until some other essential (or "transitively essential", meaning that an essential package depends on it) package provides it. There was a disagreement over the deprecation warning, though; while most committee members seemed minded to mandate its removal, David Bremner disagreed: "I don't think people are really entitled to expect which(1) to never print to stderr". Even if the warning is annoying, he said, it doesn't rise to the level where overriding a maintainer is justified.

So the ballot that went to the committee on October 20 reflected that disagreement. It mandates that debianutils keep which for now, but gives committee members the choice of whether they would require the warning to be removed. There is no mandate when it comes to the use of alternatives; it is up to the relevant developers to decide how the transition is to be managed. The ballot does mandate keeping tempfile and includes the language on not moving files to /usr, which is consistent with the October 18 ruling.

In the resulting voting, two of the members (Bremner and Elana Hashman) voted for the resolution without the removal of the deprecation warning; all other members were in favor of the whole thing. The resolution thus passed, Adams has been overridden, and which stays within debianutils — without the warning — for now. Assuming that a home for which (and tempfile) can be found in another essential package, though, the path is clear for them to transition out of debianutils before the Debian 12 release.

All of this may seem like a big fuss over a tiny program, and that is indeed what it is. But it is also a statement of how the Debian project wants changes like this to be made. Debian developers have nearly absolute control over their packages, but there are mechanisms in place to intervene when one developer exercises that control in a way that appears damaging to the distribution as a whole. This sort of whichcraft is how Debian has managed to keep hundreds of independent-minded developers working toward a common goal for the better part of three decades.

Comments (77 posted)

Fedora considers removing NIS support

For all of you youngsters out there, the Internet has always been omnipresent, computers are something you carry in your pocket, the Unix wars are about as relevant as the War of 1812, and the term "NIS" doesn't ring a bell. But, for a certain class of Unix old-timer, NIS has a distinct place in history — and, perhaps, in still-deployed systems. So the suggestion that Fedora might drop support for NIS has proved to be a bit of a wakeup call for some.

NIS ("Network Information Service") was initially born in the depths of Sun Microsystems as "Yellow Pages". It came about in those heady times when Unix workstations were beginning to pop up in offices — and were being connected to just-installed 10Mb/s Ethernet networks via a (suitably named for the Halloween season) vampire tap. Having a network made it possible to copy around various administrative files like /etc/passwd and create an early sort of single-sign-on regime on the local network. We were all quite proud of ourselves for setting such things up.

As the number of systems grew, though, all of that copying became a little cumbersome and machines easily went out of sync. Yellow Pages was Sun's way of automating this work within a simple, centralized service. Getting a network running with it was a quick process, and adding new clients was even faster. There were occasional problems, of course, leading to the system being renamed "Yellow Plague" by some users, but as a whole, it worked quite well. That is for a value of "quite well" that discounts its total lack of access control, encryption, or defenses against malicious hosts masquerading as servers, but that was a more innocent age.

Sun eventually ran into trademark problems with the Yellow Pages name; being a Unix company, Sun had a deep understanding of the folly of getting into legal battles with telecommunications companies, so it wisely changed the name to NIS. The later NIS+ release added some security and reliability features but looked similar in many ways. Eventually, though, Sun lost interest in NIS (and just about everything else) and the system fell from its nearly dominant position in Unix shops into obscurity. It would be surprising indeed to see a new deployment adopt it now.

Linux systems still carry support for NIS, though. The pam_unix authentication module supports it, and distributions still package the various NIS utilities. At the beginning of October, though, Björn Esser suggested that Fedora, at least, might stop doing so soon. Esser is working on a project to replace pam_unix (which also receives little attention anymore) with a simpler alternative; one of the things that would make it simpler would be to drop support for NIS. He wanted to know if anybody was still actively using it.

It seems that users do still exist; they were perhaps most aptly described by Stephen John Smoogen:

The places I have seen it still being used are in Universities run by people who learned sysadmin in the 1990's and early 2000's. It is a light weight system which is simple to set up and tends to be the goto-stick for a lot of 'we put this together in 1999 with RHL6 and upgraded ever since' places.

There may be more examples of that kind of site than anybody expects. Frank Ch. Eigler was quick to ask whether any simple alternatives exist. The answer would appear to be "no". Smoogen responded that: "There is LDAP but that isn't light. There are kerberos but that isn't easy". But, he added, an awful lot of the "cool kids" just defer to one of the large service providers for authentication services — a solution that seems unlikely to appeal to anybody who has made the effort to keep NIS running for all of these years.

One relatively easy alternative that does exist, as pointed out by Tomasz Torcz, is FreeIPA. But, as Smoogen noted, it still is not as easy to manage as NIS and will be a hard sell for NIS shops: "Most of the site admins running NIS I know would change their text editor to $that_other_one before they would turn off NIS". Even so, he said that the time has come to stop using and supporting it. The removal of NIS support was posted as a Fedora 36 goal on October 21. Eigler then asked for the provision of some scripts to help sites migrate to something else; that request has gone unanswered so far.

The change proposal does note that: "For some users this change may be a bit disruptive and it may require some learning curve for switching to alternative solutions". Fedora project leader Matthew Miller responded to that statement by saying: "I've spoken with some of my sysadmin friends and universities, and they suggest that the above is enough of an understatement to feel insulting". He suggested that Fedora 36, which is currently aiming for a release in April, might be too soon for this change.

Given that nothing is calling for the urgent removal of NIS, that suggestion seems likely to be heeded. But the end of NIS support in Fedora is clearly in the works, and it will likely come sooner than some users would like. It's worth noting that RHEL 9 will not support NIS either. The Yellow Pages/NIS system has had a good run over the decades, but it would appear that the time has come to run from it.

Comments (77 posted)