Tracing unsigned modules
The reuse of one of the "tainted kernel" flags by the signed-module loading code has led to a problem using tracepoints in unsigned kernel modules. The problem is fairly easily fixed, but there was opposition to doing so, at least until a "valid" use case could be found. Kernel hackers are not particularly interested in helping out-of-tree modules, and fixing the problem was seen that way—at first, anyway.
Loadable kernel modules have been part of the kernel landscape for nearly 20 years (kernel 1.2 in 1995), but have only recently gained the ability to be verified by a cryptographic signature, so that only "approved" modules can be loaded. Red Hat kernels have had the feature for some time, though it was implemented differently than what eventually ended up in the kernel. Basically, the kernel builder can specify a key to be used to sign modules; the private key gets stored (or discarded after signing), while the public key is built into the kernel.
There are several kernel configuration parameters that govern module signing: CONFIG_MODULE_SIG controls whether the code to do signature checking is enabled at all, while CONFIG_MODULE_SIG_FORCE determines whether all modules must be signed. If CONFIG_MODULE_SIG_FORCE is not turned on (and the corresponding kernel boot parameter module.sig_enforce is not present), then modules without signatures or those using keys not available to the kernel will still be loaded. In that case, though, the kernel will be marked as tainted.
The taint flag used, though, is the same as that used when the user does a modprobe --force to force the loading of a module built for a different kernel version: TAINT_FORCED_MODULE. Force-loading a module is fairly dangerous and can lead to kernel crashes due to incompatibilities between the module's view of memory layout and the kernel's. That can lead to crashes that the kernel developers are not interested in spending time on. So, force-loading a module taints the kernel to allow those bug reports to be quickly skipped over.
But loading an unsigned module is not likely to lead to a kernel crash (or at least, not because it is unsigned), so using the TAINT_FORCED_MODULE flag in that case is not particularly fair. The tracepoint code discriminates against force-loaded modules because enabling tracepoints in mismatched modules could easily lead to a system crash. The tracepoint code does allow TAINT_CRAP (modules built from the staging tree) and TAINT_OOT_MODULE (out-of-tree modules) specifically, but tracepoints in the modules get silently disabled if there is any other taint flag.
Mathieu Desnoyers posted an RFC patch to
change that situation. It added a new TAINT_UNSIGNED_MODULE flag
that got set when those modules were loaded. It also changed the test in
the tracepoint code to allow tracing for the new taint type. It drew an
immediate NAK from Ingo Molnar, who did
not find Desnoyers's use case at all compelling: "External modules should strive to get out of the 'crap' and
'felony law breaker' categories and we should not make it
easier for them to linger in a broken state.
"
But the situation is not as simple as Molnar seems to think. There are distribution kernels that turn on signature checking, but allow users to decide whether to require signatures by using module.sig_enforce. Since it is the distribution's key that is stored in the kernel image, strict enforcement would mean that only modules built by the distribution could be loaded. That leaves out a wide variety of modules that a user might want to load: modules under development, back-ported modules from later kernels, existing modules being debugged, and so on.
Module maintainer Rusty Russell was fairly
unimpressed with the arguments given in favor of the change, at least
at first, noting that
the kernel configuration made it clear that users needed to arrange to sign
their own modules: "Then you didn't do that. You broke it, you get to
keep both pieces.
" That's not exactly the case, though, since
CONFIG_MODULE_SIG_FORCE was not set (nor was
module.sig_enforce passed on the kernel command line), so users
aren't actually required to arrange for signing. But Russell was looking
for "an actual valid use case
".
The problem essentially boils down to the fact that the kernel is lying
when it uses TAINT_FORCED_MODULE for a module that, in fact,
hasn't been forced. Steven Rostedt tried to
make that clear: "Why the hell are we setting
a FORCED_MODULE flag when no module was forced????
". He also noted that he is often the one to get bug
reports from folks whose tracepoints aren't showing up because they didn't
sign their module. As he pointed out, it is a silent failure and the
linkage between signed modules and tracepoints is not particularly obvious.
Johannes Berg was eventually able to supply the kind of use case Russell was looking for, though. In his message, he summarized the case for unsigned modules nicely:
Berg also provided another reason for loading unsigned modules: backported
kernel modules from the
kernel.org wiki to support hardware (presumably, in his case, wireless
network
hardware) features not present in the distribution-supplied drivers. He
was quite unhappy to hear those kinds of drivers, which "typically only diverge
from upstream by a few patches
", characterized as crap or
law-breaking.
Berg's use case was enough for Russell to agree to the change and to add it to his pending tree. We should see Desnoyers's final patch, which has some cosmetic changes from the RFC, in 3.15. At that point, the kernel will be able to distinguish between these two different kinds of taint and users will be able to trace modules they have loaded, signed or unsigned.
| Index entries for this article | |
|---|---|
| Kernel | Development model/Loadable modules |
| Kernel | Modules/Signed |
Posted Mar 6, 2014 6:40 UTC (Thu)
by dlang (guest, #313)
[Link] (25 responses)
It seems like what's needed is a way to tell the tracepoint mechanism to ignore it's checks and just do what you are telling it to do.
Posted Mar 6, 2014 9:05 UTC (Thu)
by iq-0 (subscriber, #36655)
[Link] (4 responses)
There is nothing "magical" about the original source tree that makes it better than any later changes, fixes or development done against that specific kernel.
If anyone wants to know if something out-of-original-build-tree gets loaded, fine. If you want to call that "tainted", fine (it would just be a different form of "taint" than was the original meaning). But that should be no indication whatsoever that something is crap (unwanted, bad, ..), and thus should be excluded from the normal kernel tracing infrastructure and such.
Posted Mar 6, 2014 11:20 UTC (Thu)
by vonbrand (subscriber, #4458)
[Link] (3 responses)
The taint flag was introduced to mark unavailable source code (binary-only drivers) or unsupported situations (kernel Oops, forcing certain operations). Out-of-tree source certainly qualifies as "not supported by the vanilla kernel crowd," they have their plates full enough already to have to chase down, configure, compile, and test external code.
Posted Mar 6, 2014 14:34 UTC (Thu)
by iq-0 (subscriber, #36655)
[Link] (2 responses)
That might be fine when people are loading stuff that they can't possibly understand (binary blobs e.g.) and when a kernel bug has been triggered in the past.
Posted Mar 6, 2014 14:39 UTC (Thu)
by dlang (guest, #313)
[Link] (1 responses)
It seems to me that the latter is the correct way to go. No matter what you do, these debugging hooks can cause grief if the modules is mismatched enough, and they also may work even with badly mismatched modules.
so give a warning "this may crash your system, based on Taint flag X, are you sure you want to do this", but allow it to be done.
Posted Mar 13, 2014 18:10 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
They're using the disstro-supplied, signed, kernel. They're rebuilding the module they're developing.
They are now STUFFED because that combo disables debugging!!!
(Why they don't want to build their own kernel I don't know, but surely there are valid circumstances ...)
Cheers,
Posted Mar 6, 2014 10:38 UTC (Thu)
by nevets (subscriber, #11875)
[Link] (17 responses)
Those modules are tainted with TAINT_OOT_MODULE. "OOT" means "Out-Of-Tree". If there's a bug and we see that taint we usually do say, "Can you reproduce that bug without the module that caused the OOT taint?". If they can't then we ignore it.
But it is far from the reason that forced modules are tainted. A module that is forced is usually one that the checksums found an incompatibility between the module and the running kernel. Something is there that can break. This means it can be anything, and we don't care if it breaks for you. As the tracing infrastructure is very intrusive, bad kernel structures can cause the tracing code to corrupt the rest of the kernel, hence why we don't allow it to work with tainted modules.
You are also missing the point completely. The bug here is that unsigned modules even from the same kernel tree will cause the force flag to be set. Try it. Build a kernel with the following:
And all your modules will be marked as FORCED, and the tracepoints in those modules will appear in the /sys/kernel/debug/tracing/events directory, but if you activate them, they wont do anything. Seems silly to let a random config break tracing, doesn't it?
Posted Mar 6, 2014 14:03 UTC (Thu)
by dlang (guest, #313)
[Link] (11 responses)
I agree, I just think the problem is in the tracing refusing to do what the admin asked rather than in the tainting.
The problem as I see it is that OOT means that you really don't know what it is. It may have exactly the same problems as a Forced module, but you don't know because you aren't checking (and as I understand it, _can't_ check to prove that it's correct or not for the generic case)
and this module isn't limited in what it can do, once it's loaded in kernelspace, it can go and muck with anything, anywhere in the kernel, so it can cause problems anywhere.
the kernel devs will do what they do with _any_ tainted kernel report, they will ask why it's tainted and if the problem can be reproduced without the tainting. It doesn't matter if it's tained for OOT, Forced, nvidia, or anything else.
The 'why is this tainted' question will discover what's going on with far better detail than you can ever hope to encode in an enumerated set of flags.
Posted Mar 6, 2014 17:11 UTC (Thu)
by nevets (subscriber, #11875)
[Link] (10 responses)
To put it another way, you are saying to let tracing crash like anything else could crash if a module is forced.
Hmm, you do have a point. We could just make it so that the only taint that fails to load tracepoints is the TAINT_PROPRIETARY_MODULE. And that one fails because of political reasons, not technical ones*.
* Some people may argue that the political reasons cause technical ones.
Posted Mar 6, 2014 19:58 UTC (Thu)
by johill (subscriber, #25196)
[Link]
The original discussion didn't really quite capture it all, I think there really are two aspects to this.
The first is that it's quite odd that a (seemingly valid) kernel configuration can result in every single module that you load being loaded as though it was forced, which is what the kernel does before the patch. I'm not sure why this part should be controversial since it just adds more detailed information ("tainted because of missing signature") up front.
The second aspect is about the tracing policy, which says that tracing is disabled for code that's tainted in certain ways (most ways, really, except a few). Since we start from the premise that unsigned modules aren't all that bad (otherwise we could just refuse loading them alltogether, or disallow the configuration that results in this), then to me at least it is strange to argue that tracing should be disabled for such modules.
The whole out-of-tree modules thing is actually a bit of a strawman, because the same happens for in-tree modules, but it's a case where it's even easier to end up with unsigned modules, and it's certainly not an uncommon thing to do (particularly with backports, which is something that some of us have to routinely support).
Now with the second aspect the other question is why tracing is disabled for certain taints to start with - after all we could argue that if the tracing code crashes for you when you have a tainted kernel then you're on your own. As I understand it Steven did this to not even run into the crashing situations (since tracing accesses the ringbuffer in low-level ways), but I would agree that you could argue for an override for allowing that to be overridden.
I tend to think that those aspects are quite unrelated though, and the bulk of the patch is really about the first aspect ("unsigned != forced") even if the reason for it was related to tracing.
Posted Mar 7, 2014 3:28 UTC (Fri)
by dlang (guest, #313)
[Link] (8 responses)
exactly, by forcing the module, the admin is saying that they know better than the kernel that this really is what they want to do (and sometimes they may even be right :-) so we should continue to feed the admin rope when they demand it, which is a pretty standard option in the kernel.
> Hmm, you do have a point. We could just make it so that the only taint that fails to load tracepoints is the TAINT_PROPRIETARY_MODULE. And that one fails because of political reasons, not technical ones*.
I'm not sure why we even need that?
What is it that we are trying to prevent by blocking tracing?
If we are trying to prevent crashing, we're already past that point.
If we are trying to make life harder for users of proprietary modules, I don't think there is really agreement on this.
Posted Mar 7, 2014 13:46 UTC (Fri)
by nevets (subscriber, #11875)
[Link] (7 responses)
That's just an added bonus but not the rational here.
The tracing code was written under the GPLv2. For it to work in a module, a lot of its code must be compiled in from the headers. The tracing headers are not just structures. The majority of the tracing C code is "magically" created via the tracing headers at the location of their use. If a proprietary module uses this tracing code, their is no question that this module is now a derived work of the tracing code, as the tracing code is now embedded inside the module. This is a clear straight forward infraction of the license that the code was written for.
As one of the major authors of the tracing code, I do not want my work being used by proprietary modules. Period!
Note, those are the modules that Ingo was calling "crap" and "felony law breakers".
Posted Mar 7, 2014 21:01 UTC (Fri)
by vonbrand (subscriber, #4458)
[Link] (6 responses)
Any actual courtroom precedent that #including a header file when compiling makes the binary (let alone the source) a derivative of said header file? Until that shows up, this is completely in the air. And I'd be surprised if it didn't end up as a point to be decided on the number of lines used that way compared to the whole.
Posted Mar 7, 2014 21:20 UTC (Fri)
by nevets (subscriber, #11875)
[Link]
Also, to make tracepoints work in a module, you must also have the following code in a .c file.
#define CREATE_TRACE_POINTS
This then does a lot of macro magic to produce a lot of code. This is not a simple interface as headers normally are. It is real code that is created, and we can see it in the module binary.
Posted Mar 7, 2014 21:31 UTC (Fri)
by nevets (subscriber, #11875)
[Link] (3 responses)
And I'd be surprised if that actually worked. I guess singing Happy Birthday in a full feature length movie, wouldn't cause any copyright issues. I mean, that song is just a very small part of the whole (movie).
Posted Mar 8, 2014 0:51 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link] (2 responses)
[1]http://arstechnica.com/tech-policy/2014/02/happy-birthday...
Posted Mar 8, 2014 1:40 UTC (Sat)
by nevets (subscriber, #11875)
[Link] (1 responses)
Posted Mar 8, 2014 23:47 UTC (Sat)
by nix (subscriber, #2304)
[Link]
Posted Mar 13, 2014 18:19 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
If I publish an anthology of poems, for each work I "borrow" I am infringing 100% of it. The court couldn't care less that it only forms 1% (or less) of my work ...
Here we have some GPL2 code, a clearly defined work, and anything using it is using 100% of it ...
Cheers,
Posted Mar 8, 2014 4:00 UTC (Sat)
by rusty (guest, #26)
[Link] (4 responses)
That's the point. CONFIG_MODULE_SIG_ALL's help says:
Sign all modules during make modules_install. Without this option,
ie. You didn't read that, and didn't sign your modules manually. This is an option for distributions to do their own special stuff. As quoted in the article: "you broke it, you get to keep both pieces".
Hope that helps,
Posted Mar 8, 2014 13:54 UTC (Sat)
by nevets (subscriber, #11875)
[Link] (3 responses)
That doesn't say: "Or your unsigned modules will not work as expected"
I think your "you broke it, you get to keep both pieces" is quite off.
I mean really. WTF does not signing a module have to do with breaking other parts of the kernel. That help message to me just says that I need to sign my module manually if I want it part of the "trusted" group. It doesn't say that parts of the module will not work because it wasn't signed.
Even if someone read that, it doesn't convey, HEY SIGN YOUR MODULE, OR WHEN YOU LOAD IT YOU WILL BE TOTALLY FSCKED AND WE WONT EVEN TELL YOU SOMETHING WENT BAD.
Posted Mar 9, 2014 0:07 UTC (Sun)
by rusty (guest, #26)
[Link] (2 responses)
Cheers,
Posted Mar 9, 2014 14:07 UTC (Sun)
by nevets (subscriber, #11875)
[Link] (1 responses)
I'm just trying to point out (perhaps loudly) that it's not broken. It's a valid config that you said "hey you broke it, you keep the pieces". Where the config is just talking about the signing of modules. No where does it say that the module will not work as expected because of it.
Posted Mar 9, 2014 21:42 UTC (Sun)
by dlang (guest, #313)
[Link]
Posted Mar 10, 2014 12:25 UTC (Mon)
by cesarb (subscriber, #6266)
[Link] (1 responses)
Backported, development, etc. modules, as long as they're compiled with the exact same compiler, headers, config options, etc., should interface properly with the rest of the kernel.
With forced modules, on the other hand, you have an ABI mismatch. You could have different-sized data structures, function parameters in a different order, inlined functions behaving completely different, and so on.
Posted Mar 10, 2014 18:11 UTC (Mon)
by dlang (guest, #313)
[Link]
That's a lot of fiddly things to get right to be safe.
and how many people would know that their compiler package wasn't updated in some way since the kernel was compiled?
> With forced modules, on the other hand, you have an ABI mismatch. You could have different-sized data structures, function parameters in a different order, inlined functions behaving completely different, and so on.
But if you know that the the ABI didn't change (or at least that the module in question doesn't use any of the changed bits), then you are safe as well.
So I still argue that it's the exact same type of risk, and the admin is telling you that they know that this is the right thing to do, no matter what the potential risk.
At that point, you should give the admin some more rope and believe them (but not trust them, thus the tainting of the kernel to save time on analyzing bugs)
Posted Mar 13, 2014 18:50 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
First, there are open source out-of-tree modules that can't be integrated because of a conflicting license. OpenAFS being the most inconic example (and ZFS-on-Linux too). This decision means that they can't be used without a complete recompilation of the kernel (boo!).
Then there's the question of development - we're writing our own filesystem based on Ceph, and we really want to use stock distro kernels to avoid maintaining tons of virtual machines for Amazon EC2. That change would make it impossible to trace these modules (and we use tracing in production to gather diagnostics!).
Tracing unsigned modules
Tracing unsigned modules
It would just mean: There is more code active than was in the original build tree, just so you know.
Tracing unsigned modules
Tracing unsigned modules
But when one loads a normal module (backported fixes, out-of-tree open source module that is pacifically for that kernel) than you only want to mention that "out-of-tree code" has been loaded. The stacktraces don't become more wrong than any in-tree code would make them.
Tracing unsigned modules
Tracing unsigned modules
Wol
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
#include <trace/events/foo.h>
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
And I'd be surprised if it didn't end up as a point to be decided on the number of lines used that way compared to the whole.
Wol
Tracing unsigned modules
> cause the force flag to be set. Try it. Build a kernel with the following:
>
> CONFIG_MODULE_SIG=y
> # CONFIG_MODULE_SIG_FORCE is not set
> # CONFIG_MODULE_SIG_ALL is not set
modules must be signed manually, using the scripts/sign-file tool.
Rusty.
Tracing unsigned modules
Tracing unsigned modules
Rusty.
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules
Tracing unsigned modules