Back to the drawing board for utrace?

The utrace tracing framework has had a tortuous path towards the mainline, but it always seemed like it was headed that direction. Over the past week or so, things have gotten rather murkier for the mainline inclusion of utrace. Linus Torvalds made a pronouncement that would seem to leave SystemTap without a future in the mainline—something that many had suspected for a while—but also put the future of utrace in doubt. Further discussion may have provided a way forward, but, at least in its current form, mainline utrace seems very unlikely.

The discussion resulted from a request by Frank Ch. Eigler to include utrace into linux-next. That led to a discussion about whether it was ready for linux-next—because it was likely to be merged in the next release cycle—or whether it should spend some time in another tree. Since an earlier version of utrace was in Andrew Morton's -mm tree, that was a potential path. Morton said that utrace "didn't break anything", but:

Morton also dredged up a response he had gotten from Oleg Nesterov the last time he asked, which listed various potential uses for utrace. In-kernel uses for utrace are important—new features are rarely merged without one—and an earlier utrace merge attempt ran into opposition because it lacked one. This time around, Nesterov and Roland McGrath included a rewrite of the ptrace() system call using utrace as part of the patch submission. It was hoped that rewriting the notoriously ugly ptrace() code using the cleaner utrace API would be the last hurdle for inclusion into the mainline.

But, replacing the guts of the ptrace() call, even though it may clean things up, is controversial. ptrace() is part of the kernel ABI that must be maintained—ugly or not—but cleaning it up is not without its risks, as Morton points out:

The risk is small, though, according to Eigler, because "this code has been deployed in fedora and rhel for several *years*, with millions of users. It's not some rickety experiment." Eigler also added to Nesterov's list of utrace uses as SystemTap's user-space probing is based on utrace. But SystemTap and one of the other potential uses on that list, namely reworking seccomp to use utrace, are what set Torvalds off:

Torvalds's complaint stems from the fact that utrace provides no user-space interface at all. It is purely an internal kernel API that is meant to be used by kernel code like the ptrace() rewrite, but also for kernel modules, which is part of what worries Torvalds. It provides lots of hooks that can be used by "random crazy out-of-tree crap", but doesn't provide any benefit to user space at all, he said:

One of the biggest problems with ptrace() is its signal-oriented interface. Programs using ptrace() act as the parent process of the tracee and must use wait() to detect state changes. For that reason, there can only be one ptrace() active for a particular process. So an strace of a program that is being debugged with gdb will not succeed. The ptrace() implementation using utrace would change that, but not directly, as there would still need to be a kernel piece that attached another utrace engine.

An in-kernel gdb "stub" using utrace—floated as an RFC back in November—could provide that kernel piece, but was met with a fair amount of resistance when it was proposed. The limitation that ptrace() imposes is seen as something that could, perhaps should, be lifted, but adding a relatively large, kernel-only API to do that is excessive. As Torvalds puts it:

The elephant in the room, of course, is SystemTap. It creates, builds, and loads kernel modules for doing its tracing, and uses utrace for the user-space tracing. That model is not popular with most kernel developers, especially for an out-of-tree solution—the APIs that it relies on are far too volatile. SystemTap must be updated when those interfaces change, and all of the previous versions must be maintained so that SystemTap can still be used with older kernels. Because of that, SystemTap may be out-of-sync with development kernels, which makes its utility for kernel hackers quite small.

The utrace proponents are pushing it as something useful in its own right, completely separate from its use in SystemTap, but one gets the sense that many of the kernel developers aren't quite buying that. Ted Ts'o tries to explain his concerns to Eigler

He goes on to compare the situation to that of the NVIDIA graphics drivers, which leads Kyle Moffett to propose a variation on Godwin's law: "As an LKML discussion grows longer, the probability of an unfavorable comparison involving nVidia or Microsoft approaches 1." More to the point, though, Moffett said he was uninterested in SystemTap:

Ts'o sees those features as potentially useful, but points out that they should be submitted with utrace for review. It may be that utrace in its present form does not survive that review:

Without an in-tree "killer feature" that only utrace can provide, there is going to be resistance to merging such an easily-abused API. Several suggestions were made—notably by Torvalds and Ingo Molnar—to enhance ptrace() itself to support some new features (such as multiple active calls or the ability to read/write more than a word at a time between the two processes), but that would mean scrapping much or all of the utrace work. Nesterov and McGrath, who are the ptrace() maintainers, have been largely silent throughout the discussion, but, previously, they have made it clear that they would much rather work with the utrace-based ptrace() implementation. So it is unclear when or if enhancements to the current code might happen.

Without utrace, SystemTap will have to find other ways to hook user space, but that doesn't really faze the kernel developers—particularly after Torvalds's unequivocal rejection of that approach—as there are other tracing solutions in the pipeline. Ftrace and perf events are slowly building capabilities, and are doing so in-tree. They are likely to grow the needed features to support kernel and user-space tracing a la SystemTap (and DTrace). Molnar specifically invites the SystemTap developers to collaborate:

    perf record -R -f -e irq:irq_handler_entry --filter 'irq==18 || irq==19'

It is unfortunate, in many ways, that SystemTap has gotten to this point. While it is possible that Torvalds could change his mind, he and other kernel developers find the new tracing features to be "a million times superior" to SystemTap. That could leave Red Hat holding the SystemTap bag for quite some time to come, as it will need to support it for existing, and likely future, RHEL versions. It is interesting to note that this alternate solution, based on Ftrace, etc., is also largely coming out of Red Hat.

It seems possible that utrace will be a casualty here as well. By incorporating features that were needed for SystemTap, and not providing a user-space interface, it tried to both do too much and too little. There are some potential ways forward, but its unclear whether they will be pursued. Torvalds points to the realtime tree as an example of how to get "crazy" things merged:

There are definitely lessons here, but the standard ones don't seem to apply. SystemTap and utrace were developed in the open, as free software from the outset, and were fairly often discussed on linux-kernel. SystemTap in particular was regularly criticized, to seemingly no avail. The biggest lesson—and the hardest to learn, especially after a feature has shipped—may be that ignoring the advice and complaints of the kernel developers is likely to come back and bite in the end. It is not terribly surprising, really, but that seems to be what is happening here.