On Rust in enterprise kernels
The players
Some context to begin: the nouveau driver handles NVIDIA GPUs. It was, for years, the product of a determined reverse-engineering effort in the absence of any help from NVIDIA; it was first merged into the mainline for the 2.6.33 release in 2009. More recently, NVIDIA has been changing its tune with regard to free-software support for its products, and has been helping with nouveau development. Ben Skeggs, who has worked on nouveau for years, recently took up employment there.
Nouveau has reached a reasonable level of functionality, but the developers in the kernel's DRM (graphics) subsystem are already looking at replacing it, at least for newer GPUs. Specifically, the Nova project has been launched to create a new driver for NVIDIA GPUs going forward. Unlike nouveau, Nova will be written in Rust. The developers involved feel that using Rust is the best way to cope with the firmware interface in newer GPUs, which can change at any time without notice. Nova is a relatively young project that may not be ready to enter the mainline for, possibly, some years.
An entirely different part of the kernel is the Virtual Function I/O (VFIO) subsystem. In short, VFIO is an interface for the control of I/O memory-management units (IOMMUs) that can be used to safely give access to a device to a user-space process. The IOMMU will ensure that the device only has access to the memory belonging to the process using it, preventing the device from (accidentally or deliberately) overwriting other parts of the system. VFIO is often used in systems running virtualized guests; each guest can be given access to the devices it needs while keeping the system as a whole secure.
To complete the cast of characters: in late September, Zhi Wang posted a 23-part patch set implementing a new "vGPU" functionality for NVIDIA GPUs. There is an increasing market for cloud-based systems with GPU access so that we can all enjoy the benefits of large language models appearing uninvited in every aspect of our lives. The vGPU patches make it easy for cloud providers to give virtual machines access to one or more GPUs and to arbitrate the use of those GPUs when contention occurs. NVIDIA clearly thinks, probably with good reason, that this feature will make its GPUs more attractive to cloud providers, which are known to purchase a nontrivial amount of hardware. Befitting NVIDIA's more upstream-friendly approach, all of this work is aimed at the mainline kernel.
Backporting worries
This new subsystem, as one might expect, is based on VFIO, meaning that the interfaces the kernel already exports for virtualized device control will work with NVIDIA GPUs. This decision is uncontroversial. But the vGPU work also depends on the nouveau driver for access to the hardware, and makes significant changes to nouveau to add the support it needs. This aspect attracted the attention of the developers behind Nova, who were concerned about basing this functionality on a driver that they are working to replace. The plan is that the chipsets to be managed by vGPU will be driven by Nova, so it is natural to want to see all of this work done there instead.
Danilo Krummrich asked what
the strategy for vGPU was in the longer term: "Is this more of a proof
of concept? Do you plan to work on Nova in general and vGPU support for
Nova?
" Jason Gunthorpe, who has seemingly had a hand in the design of
the new subsystem, answered that it "is
intended to be a real product that customers would use, it is not a proof
of concept
". He is hoping to see it merged soon, he said. He finished
by saying: "As a commercial product this will be backported extensively
to many old kernels and that is harder/impossible if it isn't exclusively
in C.
"
In other words, the new vGPU subsystem will be backported by distributors (and large-scale users) into "enterprise" kernels that, at their core, predate the introduction of Rust support into the mainline; Gunthorpe estimated that only 10% of the kernels running on the target systems are based on 6.0 or newer releases. Since the initial Rust support only landed in 6.1, and it is far from complete even now, any sort of Rust dependency will clearly make backporting vGPU harder, if indeed it can be done at all. The result could be that the vGPU subsystem doesn't find its way into those older kernels at all, which would be disappointing for many of the people (and companies) involved. Thus, Gunthorpe concluded, vGPU must remain based on nouveau for now, and the Nova project has to somehow live with that.
Krummrich acknowledged that
using nouveau for now is a reasonable approach, but worried that the
addition of Nova would lead to a duplication of functionality within the
kernel. To avoid that, he said, there needed to be an agreement that the
vGPU developers would, in the longer term, commit to helping with Nova
development and the movement of vGPU to a Nova base. Gunthorpe agreed that Nova might
someday become the base for vGPU but pointed out that, while vGPU is a
working driver that may be merged soon, Nova does not yet exist; "let's
not get too far ahead of ourselves here
".
Disagreement and options
What followed was an occasionally heated discussion where the participants did not always seem to be understanding each other. Gunthorpe suggested that a core driver (written in C) could be created with just enough functionality to allow higher-level drivers to communicate with the devices; nouveau and Nova could both sit on top of that core driver. DRM maintainer Dave Airlie, though, insisted that the lowest levels need to be in Rust:
The core has to be rust, because NVIDIA has an unstable firmware API. The unstable firmware API isn't some command marshalling, it's deep down into the depths of it, like memory sizing requirements, base message queue layout and encoding, firmware init procedures. These are all changeable at any time with no regard for upstream development, so upstream development needs to be insulated from these as much as possible. Using rust provides that insulation layer.
He suggested that there could be ways to plan the transition to make life
as easy as possible on the VFIO side. Gunthorpe answered that: "We
can't have rust in VFIO right now, we don't have that luxury. This is just
a fact, I can't change it.
". He also said that his objection to Rust
in VFIO would be somewhat reduced "if the backporting can be shown to be
solved
". Perhaps, he said, a minimal core driver could be written in
Rust as an alternative to a C version, and that could be used to see how
hard the backporting problem really is.
With regard to backporting, Greg Kroah-Hartman advised Gunthorpe
to "never make design decisions based on old ancient commercial kernels
that [don't] have any relevance to upstream kernel development today
".
If companies are interested in having vGPU support in enterprise kernels,
he said, they should just pay for the backporting of the necessary patches
into those kernels. Gunthorpe replied that he is not
ready to accept any such dependency until the kernel community has decided
that Rust support is no longer experimental. He also expressed hope that
the problem would just go away:
This argument is *way too early*. I'm deeply hoping we never have to actually have it, that by the time Nova gets merged Rust will be 100% ready upstream and there will be no issue. Please? Can that happen?
The conversation eventually just faded away. In the short term, there are no decisions that need to be made; the only way for vGPU to work in current mainline kernels is to use nouveau, so it would be surprising indeed if it were not merged in that form. Even the Nova developers have not argued against that. In the longer term, with luck, things will play out as Gunthorpe hopes and, when Nova is ready to go upstream, the course of action there will be clear as well.
Perhaps life will not be quite that easy, but there is reason for optimism anyway. While the positions on both sides appeared to be absolute, all of the developers involved have shown interest in finding solutions that would work for everybody. There is no disagreement over the long-term goals that the community is working toward, and there is reason to believe that an agreement will be found on how to get there.
Some of the themes that came up in this discussion are likely to recur,
though. Any functionality that is dependent on Rust code is going
to be a challenge to backport to ancient enterprise kernels, so merging
that code may draw opposition, even from developers who are supportive of
Rust in general. At some point, if the Rust project is going to succeed,
the kernel will have to make that leap. Expect some further discussions,
though, on just when that point should be reached.
| Index entries for this article | |
|---|---|
| Kernel | Development tools/Rust |
| Kernel | Device drivers/Nouveau |
Posted Oct 10, 2024 14:01 UTC (Thu)
by jgg (subscriber, #55211)
[Link] (1 responses)
Posted Oct 10, 2024 14:54 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
What does that look like, in your view?
Posted Oct 10, 2024 16:57 UTC (Thu)
by jhoblitt (subscriber, #77733)
[Link] (4 responses)
Posted Oct 10, 2024 17:51 UTC (Thu)
by jgg (subscriber, #55211)
[Link] (3 responses)
In general you shouldn't be making coding changes to accomodate backporting. That is severely frowned upon, but does ocassionally happen regardless.
But, there is a whole set of other grey areas where things get murky. The gcc toolchain has been kept back for alot of different reasons, including backporting.
People tend to think carefully before renaming files or moving code around. I've seen patches pushed back on because of code motion harming backporting. For instance if you send a patch to reorder functions to avoid forward declarations then there is a chance that will be refused as being too trival and too harmful. It depends on the maintainer.
I would say the basic informal agreement we seem to have is that people have to do the best technical work upstream but that with enough patches the backporting will succeed. Rust is a big, big step up of the backport requirement and I think nobody with customers and timelines funding their projects wants to be forced to be the first one that pipe cleans this. Nobody knows what this looks like, or what is possible, and there is a real fear that the project would end up useless for the real world if backporting fails.
Posted Oct 10, 2024 18:30 UTC (Thu)
by raven667 (subscriber, #5198)
[Link]
Posted Oct 12, 2024 17:26 UTC (Sat)
by marcH (subscriber, #57642)
[Link]
An example that trivial and harmful should really not depend on the maintainer. Do some maintainers want to get rid of stable kernel branches?
Posted Oct 13, 2024 0:28 UTC (Sun)
by marcH (subscriber, #57642)
[Link]
Major exaggeration to get the point across, apologies but the idea stands.
Posted Oct 10, 2024 19:48 UTC (Thu)
by geofft (subscriber, #59789)
[Link] (11 responses)
Since we're now in fact in a situation where some hardware company has what is effectively an out-of-tree driver (in that it won't be part of -stable for the versions they want to backport to, it'll be some redistributor adding the patches) and there's good reason for that driver to be in Rust, is the right answer to revive that approach? Should we look at extracting the actual Rust for Linux code in mainline into something that can be backported to old kernels?
Or maybe more simply, if vGPU is going to be backported as patches on top of an old enterprise kernel (instead of as an out-of-tree module), why can't the Rust support be backported along with it?
Posted Oct 10, 2024 20:09 UTC (Thu)
by ballombe (subscriber, #9523)
[Link] (7 responses)
Posted Oct 11, 2024 0:54 UTC (Fri)
by geofft (subscriber, #59789)
[Link] (6 responses)
So I don't think it's particularly unreasonable for a distro to package up a /usr/lib/rust-to-build-the-kernel/bin/rustc or something that's the right version of Rust for everything the kernel needs, make it clear that there's no support for this version of Rust for any other purpose (and in particular that it might get upgraded during the lifetime of the LTS in a way that /usr/bin/rustc wouldn't), and point their kernel build recipe at it. It's a larger-than-usual demand, sure, but it's not a huge demand (nobody's asking for a bunch of userspace Rust libraries along with the compiler), and if there's motivation from paying customers, doing the packaging seems like a relatively small amount of engineering time for a commercial redistributor, at least compared to implementing both the vGPU feature and the shared Rust core that it and Nova can use.
Posted Oct 11, 2024 11:35 UTC (Fri)
by pbonzini (subscriber, #60935)
[Link] (1 responses)
However, it has been common to use a separate Rust or even C++ compiler to build packages like Firefox, Libreoffice or Qt.
Posted Oct 27, 2024 3:03 UTC (Sun)
by Conan_Kudo (subscriber, #103240)
[Link]
Posted Oct 11, 2024 12:07 UTC (Fri)
by pizza (subscriber, #46)
[Link] (3 responses)
The way things are currently, you would also have to backport a significant portion (if not all) of the R4L core+specific subsystem code that's changed since the destination release, in addition to the core+subsystem code that R4L code interacts with. (and by "subsystem" I mean _every_ subsystem that your code interacts with, even indirectly...)
Posted Oct 11, 2024 19:44 UTC (Fri)
by smurf (subscriber, #17840)
[Link] (2 responses)
Posted Oct 11, 2024 23:44 UTC (Fri)
by jgg (subscriber, #55211)
[Link] (1 responses)
Posted Nov 6, 2024 14:55 UTC (Wed)
by daenzer (subscriber, #7050)
[Link]
Posted Oct 10, 2024 20:11 UTC (Thu)
by tzafrir (subscriber, #11501)
[Link]
If not, then the issue of backporting just moved elsewhere. Along with a separate pain of maintaining an out of tree module.
Posted Oct 11, 2024 0:14 UTC (Fri)
by jgg (subscriber, #55211)
[Link] (1 responses)
We might see a similar communal effort for getting a rust backporting helper for OOT as well eventually.
Posted Oct 11, 2024 7:45 UTC (Fri)
by johill (subscriber, #25196)
[Link]
However https://backports.wiki.kernel.org/ is pretty outdated/dead.
https://git.kernel.org/pub/scm/linux/kernel/git/backports...
Posted Oct 11, 2024 3:40 UTC (Fri)
by jkingweb (subscriber, #113039)
[Link] (129 responses)
I'm not much of a programmer, so maybe I'm just hopelessly ignorant, but how does Rust provide that insulation layer? Why cannot C do so as well? I understand that Rust's type system can hide gory implementation details to usually provide a simpler API, but how does that relate to firmware API changing out from under the driver's feet?
The article presents the quotation uncritically, and perhaps rightly so, but I think I might benefit from an explanation.
Posted Oct 11, 2024 5:41 UTC (Fri)
by tchernobog (subscriber, #73595)
[Link] (4 responses)
I suspect it has to do with generics and Rust macros for the generation of large swats of serialization or other code in a type-safe manner, but I am not sure.
Posted Oct 11, 2024 7:03 UTC (Fri)
by mmechri (subscriber, #95694)
[Link] (2 responses)
It seems like each firmware version may introduce changes that the driver must be able to deal with.
As a non-Rust developer, I would also be interested in learning how Rust makes it easier to deal with this.
Posted Oct 11, 2024 9:06 UTC (Fri)
by k3ninho (subscriber, #50375)
[Link]
K3n.
Posted Oct 11, 2024 10:37 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
The big thing that Rust makes easier is writing performant code that's structured as "parse external form into suitable internal structures, work with internal structures, serialize into external form".
The type system then allows you to make sure that your invariants aren't ever broken inside the internal structures (absent the use of unsafe, which you can catch and worry about during code review). Rust enums (sum type) are powerful for representing choices, because enum variants can be structs, not just integers, while the affine typing rules Rust uses mean that you can guarantee that if someone wants to get an object of a specific type, they've given you ownership of an object of a different type, which allows you to prevent double-action of many forms (double-freeing is the famous one, but any form of double action on a type that shouldn't allow it is something you can prohibit).
Posted Oct 11, 2024 15:11 UTC (Fri)
by jgg (subscriber, #55211)
[Link]
So imagine you have to make a function:
marshall_foo(int a, int b)
To produce the RPC struct to deliver to the FW.
Then FW version A has
struct foo_A {
And FW version B has
struct foo_B {
So, I can't say exactly what part of Rust is making this better. I've assumed it is built around generics to allow source code sharing and still robustly enforce the memory layout. However, any programming language has hard limits, it can't address semantic changes. In the above example you hope that 'c' can remain zero, and maybe have to deal with the reason 'a' got bigger.
Notice this is not about complex serializing or dynamic memory, or any of the other things rust is really strong at. Just knowing a giant table of offsets and sizes..
From a C perspective, I'd solve this class of problem with either a code generator or a multi-compilation scheme (similar to generics). iommu_pt and the MM are examples effectively solving a similar problem with non-semantic differences in page table entry memory layout, for instance.
Posted Oct 11, 2024 7:00 UTC (Fri)
by NYKevin (subscriber, #129325)
[Link]
* Rust has visibility restrictions and C (mostly) does not, so forcing C code to never rely on the width or layout of a type is harder (read: de facto impossible in the limit as the size of the codebase grows to infinity). I would tend to assume that this is by far the most important reason, and all of the other reasons are just bonus points.
(Nitpicking: C does support opaque structs, which is a very basic all-or-nothing visibility restriction. But opaque structs have a plethora of asterisks attached to them that Rust structs do not. Most importantly, you don't know their size at all, so you can't stack allocate them, memcpy them, or do much of anything with them other than pass pointers around. I would also tend to worry about whether it defeats inlining in some cases, but I'm not a compiler expert so I could be mistaken about that.)
Posted Oct 11, 2024 8:19 UTC (Fri)
by GhePeU (subscriber, #56133)
[Link]
Anyway I think the Nova developers have the same constraints so their reasoning must be more or less the same too.
Posted Oct 11, 2024 8:32 UTC (Fri)
by smurf (subscriber, #17840)
[Link]
Rust provides a bunch of guarantees that C does not have a concept of, which thus are not checkable at compile time. The practical effect is that a significantly large class of problems just disappears (OK not by itself, you need to heed the compiler warnings and fix them), to the tune that Asahi Lina wrote a complete and mostly-bug-free driver for Apple's M1 basically from scratch in a few weeks (while she actually needed months in real time, she also hat to write all the infrastructure).
This kind of feat is basically impossible in C.
The people who insist of that part being written in Rust are *right*. It's less work to backport the whole Rust infrastructure to an older kernel, if you really think you have to do that (in a sane world you most emphatically would not), than to write that code in C. Not if you consider the subsequent hunting of bugs.
Posted Oct 11, 2024 13:43 UTC (Fri)
by khim (subscriber, #9252)
[Link] (115 responses)
With the use of a typestate pattern. Languages that fully support a typestate can statically encode, in a type system, runtime invariants, but are, usually, very hard to use and reason about for the uninitiated. Rust also started there and if it would have stayed there it would have withered on the vine, most likely. Typestate was removed from it during Rust evolution, but it left behind Rust's borrow checker. Each variable in Rust can be in precisely two states: Surprisingly enough that's enough, in practice, to encode very convoluted rules that program that interacts with a firmware (or a physical device) have to follow. None of popular languages can do that because they lack a typestate. The simplest, cleanest example that showcases what they are lacking is growable buffer. In C one would implement it with If we want to avoid bugs in our programs, then we should ensure that no one have references to elements stashed anywhere when we try to resize the buffer. This sounds like a simple, almost trivial requirement, but Rust is almost unique among popular languages because it actually can represent this restriction in it's API. I think the only other one is Ada (via SPARK) – and Ada very explicitly and consciously picked that ability from Rust. Wouldn't that be a bit silly? We are talking, quite literally, about something that Rust was born to solve, about Rust's raison d'être, about something that shapes the whole language from the bottom to the top and makes it unique. This ability is core to what almost everyone who ever worked with Rust explicitly notes: ease of refactring. Something that even A Critical Retrospective puts under Where Rust Exceeded Expectations banner. Adding explanation about core thing that makes Rust… well… Rust – to every article? It would be like adding explanation about how planes are able to fly to every article that discusses advantages and disadvantages of different modes of transportations. Yes, planes can fly (and pay very high price for that ability), while ships and cars couldn't do that. Heck, the whole reason to add to the kernel Rust is that same ability! It Rust couldn't deliver on that promise then there are no need to even talk about adding it to the kernel!
Posted Oct 11, 2024 13:58 UTC (Fri)
by jkingweb (subscriber, #113039)
[Link]
I wasn't suggesting the article necessarily lacked anything—that's precisely why I speculated the article may not have erred in omitting more detail. Rather, it was my understanding which was clearly lacking, and apparently I wasn't the only on in this situation. I am vaguely familiar with what advantages Rust has, but the implications of those advantages are not necessarily clear to everyone all the time.
Posted Oct 11, 2024 14:19 UTC (Fri)
by MKesper (subscriber, #38539)
[Link] (112 responses)
Posted Oct 11, 2024 19:41 UTC (Fri)
by smurf (subscriber, #17840)
[Link] (110 responses)
Mwa-har-har.
Some C++ people try to tell us that this is possible and/or they're working on it.
Posted Oct 12, 2024 5:44 UTC (Sat)
by NYKevin (subscriber, #129325)
[Link] (109 responses)
For example, a large class of memory safety issues will magically vanish if you require that all buffers are statically allocated with some fixed size, and malloc et al. are not used. Nobody actually wants to write code like that, so you pretty much only see it in highly failure-sensitive domains like aerospace. But it is technically a "safe subset" of C.
But you're going to object that that is obviously not what the C++ folks are talking about. Fine. Let's talk about borrow checking. The modern Rust borrow checker did not emerge fully formed directly from Hoare's brain. It went through a lengthy period of iteration and refinement. It is possible to reinvent less powerful interpretations of borrow checking based on those earlier iterations. For example, you could just have lexically-scoped borrows (the borrow is not allowed to escape a given lexical scope) plus a syntax rule that prohibits moving a borrowed object within the borrow's lexical scope. This is not fully general, and fails to express many things that Rust can handle easily, but when you combine it with C++'s existing smart pointers, it is probably enough for many practical applications (that don't need ultra-low overhead).
Perhaps that's too weak. We probably do want the ability to put std::string_view inside of a data structure, and this would ban that as an escaping borrow. There may be ways to strengthen it that are still workable within the context of C++. But we could instead argue that std::string_view is the Wrong Abstraction for safe C++, and they should instead be using some kind of refcounted string view (i.e. each view owns a strong reference to the refcounted string), or perhaps a copy-on-delete string view of some kind (i.e. a string view that is informed when the string is about to be destructed, and allowed to copy the relevant portion of the string into a new heap allocation).
Or maybe they'll do something really crazy, such as deprecating std::string altogether and replacing it with a refcounted copy-on-write rope structure. I don't seriously think that will happen, but at least it would (probably) solve the view issue (and it would be very amusing watching people get upset about it on the internet).
Posted Oct 12, 2024 6:47 UTC (Sat)
by mb (subscriber, #50428)
[Link] (38 responses)
There is no such thing as a "safe subset" of C++ and there never will be. If you create a "safe subset", then it will be a completely different language that people will have to learn. Why bother? You would still end up with a language containing massive amount of obsolete backward compatibility stuff next to your new "safe subset" language.
Just switch to a safe language (for new developments). It's much simpler. And it's much safer.
Posted Oct 12, 2024 16:45 UTC (Sat)
by NYKevin (subscriber, #129325)
[Link]
Posted Oct 14, 2024 9:31 UTC (Mon)
by paulj (subscriber, #341)
[Link] (36 responses)
Sure, for a compute-and-complete programme, processing a finite input to give some result, Rust programmes can allocate from the stack, and/or have acyclically-linked heap objects with tight lifetimes. But... for anything long-lived with complex relationships between different data objects, isn't the pattern in Rust to use Rc/Arc? I.e., refcounting?
Posted Oct 14, 2024 10:26 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (14 responses)
Your long-lived data is often refcounted (especially in async Rust) because that's simpler to get right than rearranging so that your data doesn't have complicated lifetime relationships. But your short-lived accesses to that data usually borrow it, rather than refcounting it, because their relationship to the bit of data they need is simple.
And that's something that's not easy to express in Python - the idea that I borrow this data in a simple fashion, even though the structure as a whole is complicated and refcounted.
Posted Oct 14, 2024 11:00 UTC (Mon)
by paulj (subscriber, #341)
[Link] (13 responses)
In the little I've played with Rust, the lifetime annotations make sense on simpler stuff. But it seems to get very baroque on more complex stuff. And I havn't gotten over a certain "barrier" on how to design the not-simpler kinds of inter-linked relationships that are unavoidable in real-world stuff. E.g. a network protocol, you may have objects and state for "peers", "connections", "TLS contexts", "Streams", "message entity" (plurality of these) with m:n relationships between them. A peer may have multiple connections, messages may have a lifetime related to a /set/ of peers and not just the peer it was received from, etc. You have to use RC for at least some of these - so now you're dealing with /both/ refcounting AND lifetime annotations for the refcounting objects (in other languages you would embed the RC information in the refcounted object, but in Rust the RC object holds the refcounted object), and there are rules for getting data into and out of RCs and cells and what not. And I end up with long screeds of compiler error messages that give me flashbacks to C++ - maybe Rust's are better, but I've been traumatised. The complexity of implementing simple linked lists in Rust, without unsafe, is a bit of a red flag to me.
I think the local stack variable lifetimes possibly could be dealt with in simpler ways, perhaps with more limited annotations, or perhaps none at all but just compiler analysis. It might be more limited than Rust, but be enough to deal with typical needs for automatic data in long-lived programmes.
What I'd like, for long-lived programmes with non-simple relationships in their data model (e.g., many network protocols) is a language that did ref-counting /really/ well - ergonomically, with low programming complexity, and minimal runtime overhead. Cause refcounting is how all non-trivial, long-lived programmes, that can not tolerate scannning GC, end-up managing state.
Posted Oct 14, 2024 11:28 UTC (Mon)
by mb (subscriber, #50428)
[Link]
Rust can do that, too. It's called proc-macro
Posted Oct 14, 2024 12:26 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (10 responses)
One of the challenges I found in moving from C++ to Rust is that in C++, you model ownership in your head; you sort-of know whether a given pointer represents owning something, shared ownership, or simply referring to something someone else owns. Rust doesn't let you play fast-and-loose with ownership like this; the borrow checker means that you have to be extremely clear about who owns what, and when, using T for "this T is embedded here", Box<T> for "this T is owned, but is stored out of line", and Arc/Rc<T> for "ownership of this T is shared".
You can also write a simple linked list in 40 lines of Rust without any unsafe at all - the use of unsafe is because you want a complicated linked list that meets more interesting properties than the simple linked list implementation does.
Posted Oct 17, 2024 9:09 UTC (Thu)
by paulj (subscriber, #341)
[Link] (9 responses)
Posted Oct 17, 2024 9:39 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (8 responses)
Making a doubly-linked list in safe Rust isn't that much more complicated, although it does bring in the need for Rc, since a doubly-linked list has complex ownership (the previous pointer cannot be owning).
Posted Oct 17, 2024 10:10 UTC (Thu)
by paulj (subscriber, #341)
[Link] (2 responses)
To stay within the typed lifetime-scope of Rust, without unsafe, is somewhere between extremely complex and impossible, isn't it? (I thought impossible, but then I thought I saw a tutorial claiming it was possible once - but I can't find it back right now).
Posted Oct 17, 2024 14:23 UTC (Thu)
by farnz (subscriber, #17727)
[Link] (1 responses)
The difficult bit is the ownership model if you don't use refcounting; you want each node to be individually mutable, but you also want each node to be shared by both the previous and the next nodes. That's inevitably difficult in a "shared, or mutable, but not both" model as Rust imposes, unless you use refcounting.
Posted Oct 18, 2024 14:38 UTC (Fri)
by taladar (subscriber, #68407)
[Link]
Posted Oct 17, 2024 11:48 UTC (Thu)
by excors (subscriber, #95769)
[Link] (4 responses)
(It also explains:
> Linked lists are terrible data structures. Now of course there's several great use cases for a linked list: [...] But all of these cases are _super rare_ for anyone writing a Rust program. 99% of the time you should just use a Vec (array stack), and 99% of the other 1% of the time you should be using a VecDeque (array deque). These are blatantly superior data structures for most workloads due to less frequent allocation, lower memory overhead, true random access, and cache locality.
Generally, you probably shouldn't judge a language on the basis of how easily it can implemented linked lists. On the other hand, one of the listed "great use cases" is "You're writing a kernel/embedded thing and want to use an intrusive list", which is probably much more common amongst LWN readers than your average programmer; it's fair to have different priorities when judging languages.
But I think it's still important to consider whether your design uses (doubly-)linked lists because they really are the best data structure for your needs, or whether it's just because linked lists are really easy to implement in C (and an equivalent of Vec is hard) so they're your default choice for any kind of sequential container, and you're trying to apply the same mindset to Rust where they're really hard (but Vec is easy) and they should never be the default choice.)
Posted Oct 17, 2024 12:22 UTC (Thu)
by khim (subscriber, #9252)
[Link]
Tragedy of lists (and the reason Rust had to wait till 2015 to be viable) lies with the fact that while double-linked lists are very bad data structure today – that wasn't the case half-century ago when Computer Science was established and first courses that teach people how to program were developed. Back then computers big, but their memories (and thus programs written for these computers) were small, while RAM was fast and CPU slow. Linked lists really shined in such environment! Today computers are tiny, but have memory measured in gagabytes, programs are crazy big and CPUs are hudnreds of times faster than RAM. Today linked lists are almost never the right data structure for anything and even when they actually better than something like But people are creatures of habits: after they have learned to deal with linked lists once they tend not to ever return back and use them as long as they could. Worse yet: while it's true, today, that Thus even today, long after linked lists and all tricks associated with them stopped being important in 99% of cases, people try to apply what they know to Rust. And yes, that inevitably sends us back to one funeral at time story. Which is sad, because that means that new generation would have to relearn everything again: while linked lists are no longer useful many other lessons that “old beards” learned are still usefull… but if C/C++ would be replaced with Rust (or any other safe language) via one funeral at time way then all these lessons would be forgotten… and thus repeated.
Posted Oct 17, 2024 12:42 UTC (Thu)
by smurf (subscriber, #17840)
[Link]
The key isn't to implement a doubly-linked-list in Safe Rust, which is not exactly a workable idea for obvious lifetime reasons, but to hide them behind an appropriate abstraction.
Posted Oct 17, 2024 14:00 UTC (Thu)
by paulj (subscriber, #341)
[Link] (1 responses)
Doubly linked lists are just a well-understood form of data-structure with cycles. Cycles in data-representations in non-trivial programmes are common, even if those programmes (rightly) eschew linked-lists. The linked-list question speaks to the ability to do common kinds of data modelling.
Focusing on the linked-list part misses the point.
Posted Oct 17, 2024 15:57 UTC (Thu)
by mb (subscriber, #50428)
[Link]
I think that focussing on everything must be implemented in safe Rust is missing the point of Rust.
For implementing such basic principles as containers and lists, it's totally fine to use unsafe code.
Therefore, I don't really understand why reference cycles or linked lists are discussed here. They are not a problem. Use a safe construct such as Rc or use unsafe. It's fine.
Posted Oct 20, 2024 11:15 UTC (Sun)
by ssokolow (guest, #94568)
[Link]
...maybe because it's really the only place in the language where you have to code in a Haskell-like functional-recursive style instead of an imperative style if you want to do the fancier tricks?
(Of course, to be fair, the same thing occurs more subtly at lower levels with people misjudging the cost of a given machine language opcode and how superscalar architecture will affect costs. It just doesn't result in compiler errors.)
There's no shame in reaching for a reference-counted pointer. (I'm no C++ programmer, but I've never seen any signs of this much agonizing over
Posted Oct 14, 2024 10:31 UTC (Mon)
by excors (subscriber, #95769)
[Link] (18 responses)
Posted Oct 14, 2024 11:03 UTC (Mon)
by paulj (subscriber, #341)
[Link] (17 responses)
I'm still left a bit uneasy about the complexity on all the rules on getting data in and out of RCs and cells and what not, and implementing non-trivial relationships in data, in Rust. It's not a simple language.
Posted Oct 15, 2024 8:20 UTC (Tue)
by taladar (subscriber, #68407)
[Link] (16 responses)
The problem of ownership is complex, Rust helps you deal with it by making it explicit, other languages just let you make mistakes without providing any help with it at all.
Posted Oct 15, 2024 9:15 UTC (Tue)
by paulj (subscriber, #341)
[Link] (15 responses)
- references may be "owning" or "shared"
Rust automating these rules and enforcing them, AND allowing me to safely pass a "owning" reference to other functions, taking care of passing the ownership in the process, is nice. In my way of doing things, I will often have to use local helper functions that take "owning" references to do stuff with them. And I have to myself ensure I don't store those pointers somewhere. But, that's what I have to do anyway. Rust enforcing this, unless code is marked 'unsafe' seems nice.
But... for me... Rust's lifetime type system allows too much. I can see it would let me do more than what my own heavily constrained system allows, and with guarantees, but it also drags in a lot of complexity. Complexity I already made a conscious decision to avoid.
A language with a much more limited set of scoping rules for pointers - more like the system I've already been using for years in unsafe languages - could be simpler and just as effective, possibly.
Posted Oct 16, 2024 7:44 UTC (Wed)
by taladar (subscriber, #68407)
[Link] (14 responses)
Posted Oct 17, 2024 9:19 UTC (Thu)
by paulj (subscriber, #341)
[Link] (13 responses)
My point is more that my own system of ad-hoc, non-compiler-enforced rules for being able to manage the problem of lifetimes of objects and their references is simpler than Rusts'. In my system I basically try to have just 2 scopes for references - the very local, and then the refcounted. The latter are "safe" use-after-free issues (I don't use usually use refcounting machinery that deals with concurrency, but you could).
Rust goes further than that, and provides for arbitrary scopes of lifetimes for references to different objects. Great, lot more powerful. But... it also becomes harder to reason about when you start to make use of that power, it seems to me. I can't keep track of more than a couple of scopes - that's why my ad-hoc safe-guards in lesser languages are so simple, and I think it's part of why I struggle to get my head around Rust's error messages when I try write more interlinked, complex data-representations, in my own toy test/learn coding attempts. The fact that a lot of code ends up going back to refcounted containers suggests I might not be alone, not sure.
What I'm saying is that simple programmers like me might be better off with a "safe" (i.e. enforcing the rules) language with a more constrained, simpler, object lifetime management philosophy.
Posted Oct 17, 2024 10:36 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (12 responses)
I wonder. Could you create a "head" object who's lifetime is the same as the function that created it, and then create a whole bunch of "body" objects that share the same lifetime?
These objects can now reference each other with much fewer restrictions, because the compiler knows they share the same lifetime. So there's no problem with the head owning the next link down the line etc etc, and each link having a reference to the link above, because they'll all pass out of scope together?
Of course that means you can't access that data structure outside the scope of the function that created it, but that would be enough for a lot of purposes?
Cheers,
Posted Oct 17, 2024 11:45 UTC (Thu)
by smurf (subscriber, #17840)
[Link] (2 responses)
That would also be rather pointless for a lot of other purposes (among them: freeing a bunch of body objects, oops you suddenly have scoping problems anyway), so what's the point?
Posted Oct 17, 2024 12:25 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (1 responses)
It feels pretty simple to me - cf paulj's comment that full blown Rust just seems to blow his mind.
Does Rust actually have a memory allocator, or does it have a "new"? That was the point of my mention of alloca - as the caller goes out of scope, any memory allocated by alloca goes out of scope as well. You don't free alloca memory.
Could the Rust compiler handle "this memory is now out of scope, just forget about it"? If standard Rust rules say an object cannot own an object with a lifetime longer than itself, just dropping the object can't do any harm, can it?
You're effectively setting up a heap, with its lifetime controlled by "head". So you just drop the entire heap, because every object in the heap has had the heap lifetime constraints imposed on it.
Cheers,
Posted Oct 20, 2024 11:11 UTC (Sun)
by ssokolow (guest, #94568)
[Link]
That's the concern of the data type, not the language.
If you want a If you want to heap-allocate, you either use a type which does it internally like It does. If you want a destructor, you Rust's design does a good job of making its memory management appear more sophisticated than it really is. It's really just stack allocation, access control, destructors but no actual language-level support for constructors, and using RAII design patterns to implement everything else in library code on top of manual calls to malloc/realloc/free.
The borrow checker plays no role in machine code generation beyond rejecting invalid programs, which is why things like mrustc can exist.
Posted Oct 17, 2024 14:07 UTC (Thu)
by paulj (subscriber, #341)
[Link] (8 responses)
You could take that kind of tack and allocate stuff on the stack with alloca, sure.
Neither hierarchical allocation, nor stack allocation, address the issue of tracking validity of references though, as such. As you're implying, that requires something else - be it an ad-hoc system of rules that enforce guarantees, assuming the programmers' can hold themselves to applying those rules (and... they will fail to every now and then); or whether they are rules in the language and enforced in the compiler.
The question for me is: What is the most programmer friendly system of rules to guarantee safety?
Rust is one example of that. With a very general lifetime typing system (the most general possible?). That generality brings complexity. Yet, many Rust programmes have to step out of that compile-time lifetime-type system and use runtime ref-counting. So then the question is, if the lesson from Rust is that many many programmes simply go to runtime ref-counting for much of their scoping, would it be possible to just have a less general, simpler lifetime-typing system?
E.g., perhaps it isn't necessary at all to even need to represent lifetime types. Perhaps it would be sufficient to have 2 kinds of references - local and refcounted, with well-defined and safe conversion semantics enforced by the language.
Posted Oct 17, 2024 16:00 UTC (Thu)
by khim (subscriber, #9252)
[Link]
That's where Rust have started, more-or-less. The language which used this approach, Cyclone, is listed among many languages that “influenced” Rust. Only it's not practical: language that you are getting as a result doesn't resemble C at all! Or, more precisely: language would look like a C, but it's APIs would be entirely different. Even most trivial functions like I'm not entirely sure all the complexity that Rust have (with covariance and contravariance, reborrows and so on) is needed… technically – but it's 100% wanted. Till Rust 2018 introduced Non-lexical lifetimes Rust was famous not for its safety, but for it's needless strictness. I think almost every article till that era included a mandatory part which was telling you tales how your “fight with the borrow checker” is not in vain and how it enables safety, etc. After introduction of NLL, reborrows, HRBTs and so on people stopped complaning about that… and started complaining about complexity of the whole thing… but it's highly unlikely that people would accept anything less flexible: they want to write code and not fight with a borrow checker! Sure. Swift does that. It has quite significant performance penalty, but still much faster than many other popular languages.
Posted Oct 17, 2024 17:10 UTC (Thu)
by Wol (subscriber, #4433)
[Link] (5 responses)
But does it *have* to?
If you have a procedure-level heap, or some other data structure with a guaranteed lifetime, you apply exactly the same borrow-rules but on the heap level. If all references to the heap have the same or shorter lifetime than the heap, when the heap dies so will the references.
So you create a heap for your linked list as early as possible, and you can freely create references WITHIN that heap as much as you like. They'll die with the heap. It's not meant to be an all-singing-all-dancing structure, it's meant to have a couple of simple rules that make it easy to create moderately complex data structures. You probably wouldn't be able to store pointers in it that pointed outside of it, for example. You would have to be careful storing references to items with shorter lifetimes. But if you want to store something with loads of complex *internal* references, it would be fine precisely because of the "everything dies together" rule.
So you'd use the existing rust checking setup, just that it's a lot easier for you as the programmer to keep track of, precisely because "it's an internal self reference, I don't need to care about it".
Cheers,
Posted Oct 17, 2024 17:27 UTC (Thu)
by daroc (editor, #160859)
[Link] (1 responses)
You may be interested in Vale, a programming language that is trying a bunch of new things around lifetime-based memory safety, including compiler support for "regions", which work almost exactly how you describe. The project also has some interesting ideas about tradeoffs between different memory management strategies. I want to write an article about it at some point.
Posted Oct 18, 2024 16:33 UTC (Fri)
by paulj (subscriber, #341)
[Link]
Posted Oct 18, 2024 14:25 UTC (Fri)
by taladar (subscriber, #68407)
[Link] (2 responses)
Posted Oct 18, 2024 16:24 UTC (Fri)
by Wol (subscriber, #4433)
[Link] (1 responses)
And can't you just apply ordinary Rust rules to that? It's not intended as a way of escaping Rust's rules. It's just meant as a way of enabling the *programmer* to forget abut a lot of the rules on the basis that internal references, pointers, objects will all go invalid at the exact same time. So if A points to B and B points to A and they're in this structure you don't worry about cleanup because they both go poof and emit the magic smoke at the same time.
If A contains a pointer to C with a shorter (or longer) lifetime, Rust will need to check that A destroys the pointer as C goes out of scope, or alternatively that the entire heap cannot go out of scope until C is destroyed.
A simple structure for simple(ish) situations, and more complex structures for where simple doesn't work. And if those complex structures are hard to grasp, it helps if you've realised that the simple structures aren't up to the task (and why).
Cheers,
Posted Oct 18, 2024 16:35 UTC (Fri)
by kleptog (subscriber, #1183)
[Link]
The basic idea is you have a function with allocates an arena and keeps it alive. Within the arena objects can reference each other as much as they want, including cycles. They can also reference other objects, as long as they live longer than the arena. When your function exits, the arena is cleaned up in one go. Incompatible with destructors (though there are tricks for that), but otherwise looks like what you want.
I know them from PostgreSQL where they have an arena per query so interrupting the query safely throws away everything. There are plenty of use cases for them.
Posted Oct 18, 2024 14:23 UTC (Fri)
by taladar (subscriber, #68407)
[Link]
This is not my experience with Rust at all. Refcounting does happen occasionally but the vast, vast majority of cases doesn't use ref-counting at all in Rust. Usually it is no more than a few values (logically speaking, some of them can exist in many copies of course but the one that is referred to by the same name in the code and passed along the same code-paths) even in large programs.
Posted Oct 14, 2024 13:08 UTC (Mon)
by khim (subscriber, #9252)
[Link]
It's true that some people (even among Rust developers) actively want to make it a ref-counting language… but currently… no, it's not a refcounting language. It's very subtle and very much “something good done for entirely wrong reasons”, but the critical part is that in Rust to increment your counter you need to explicitly call Arc::clone or Rc::clone. And if you pass your smart pointer around you don't really do any expensive operations at all. This affects efficiency very significantly. Swift is trying to reach the same point from the other side and automatically eliminate ref-counting from places where compiler can prove it's not needed. It's an interesting experiment, but I'm sceptical about that approach: that advantage of Rust (every much unplanned, as shown above!) is that pesky That's simple psychology: if you see it – you want to eliminate it (when that's easy to do, it's not a crime to call
Posted Oct 17, 2024 19:16 UTC (Thu)
by sunshowers (subscriber, #170655)
[Link]
Posted Oct 12, 2024 7:00 UTC (Sat)
by smurf (subscriber, #17840)
[Link] (67 responses)
That's the point right here. Rust has exactly one of these constructs. It's called "unsafe". You can find code that uses it with grep.
C++ has myriads of conditions that are documented as Undefined Behavior. Quite a few of them are invisible without deep code analysis that borders on solving the halting problem. They're not going to go away. There's too much code out there that depends on them and too much social weight behind keeping the status quo as it is; a significant subset of the C++ community still thinks that avoiding UB is the programmer's job, not the compiler's.
Posted Oct 12, 2024 20:55 UTC (Sat)
by MaZe (subscriber, #53908)
[Link] (66 responses)
For example you'd naively expect "foo x = {};" to simply be nicer syntax for "foo x; memset(&x, 0, sizeof(x));" (at least with a lack of non-zero default initialized fields). But it's not... There's some ridiculous subtleties with structs/unions and padding and some compilers can quite literally go out of their way to set some bits to 1. It's obvious/natural that 'foo x = {}' should mean zero initialize the whole damn thing including any and all padding, and then on top of that apply any non-standard constructors/initializations.
Basically compilers chase benchmarks instead of trying to make things a little bit more predictable/sane.
Posted Oct 13, 2024 10:23 UTC (Sun)
by rschroev (subscriber, #4164)
[Link] (19 responses)
To me that is neither obvious nor natural at all. To me it feels much more natural that constructs like that serve to initialize the class/struct members; what happens to the padding is not of my concern. I don't think I've ever cared about the bits in the padding, and I can't immediately think of a reason why I would, but if I would care I would make it explicit and use something which operates not on the members but on the raw memory, i.e. memset or memcpy.
Posted Oct 13, 2024 10:52 UTC (Sun)
by johill (subscriber, #25196)
[Link] (18 responses)
However, this has bitten me in conjunction with -ftrivial-auto-var-init=pattern, where for something like
I feel like the OP is perhaps misrepresenting this quirk, but of course they might have something else in mind.
Posted Oct 13, 2024 13:10 UTC (Sun)
by MaZe (subscriber, #53908)
[Link] (17 responses)
Furthermore ={ 0 } and ={} aren't quite the same thing, but ={} isn't supported by older compilers...
As for why it matters? Take a look at the bpf kernel system call interface.
Another example is when a struct is used as a key to a (bpf) map.
There's ample other examples of C(++)-compiler-wants-to-get-you behaviours.
[I guess some sort of 'sane C' dialect that actually says a byte is unsigned 8 bits, arithmetic is twos complement, bitfields work like this, shifts works like this, alignment/padding works like this, here's a native/little/big-endian 16-bit integer, basically get rid of a lot of the undefined behaviour, add some sort of 'defer' keyword, etc... would help a lot]
Posted Oct 13, 2024 18:00 UTC (Sun)
by marcH (subscriber, #57642)
[Link] (16 responses)
It's not just "bloody annoying": it's _buggy_ because there are and always will be many instances where the developer does not think about it.
Every single discussion about C++ safety always ends up in the very same "you're just holding it wrong" argument in one form or the other. Well, guess what: yes, there is _always_ someone somewhere "holding it wrong". That's because developers are humans. Even if some humans are perfect, many others are not. When humans don't want to make mistakes, they ask _computers_ to do the tedious bits and catch errors. You know, like... a Rust compiler.
And please don't get me started on Coverity and other external checkers: I've seen first hand these to be the pinnacle of "you're holding it wrong". Notably (but not just) because of the many false positives, which invariably lead some developers with an obvious "conflict of interest" to mark real issues in their own code as false positives while more experienced people have simply no time to look and must delegate blindly.
Advocates of C++ safety live in some sort of fantasy workplace / ivory tower where none of this happens and where they muse about technical and theoretical problems while being completely disconnected from the trenches where most coding happens. This is a constant failure to realize that usability comes first, technical details second. Same thing with security: the best design on paper is worth nothing at all if it's too complicated to use.
The only, last hope for C++ safety is for a large enough number of C++ experts to spend less time on benchmarks and agree on ONE[*] safe subset and implement the corresponding static analyzer and distribute it with LLVM by default. That's a very long shot and even then some users will still find ways to hold that static analysis "wrong" in order to make their deadline. A generous dose of "unsafe" keywords on legacy code will of course be the easiest way (cause some of these keywords will be actually required) but they will find many other ways.
I think it got a bit better now but until recently we were still hearing horrors stories based on _SQL injection_! If our industry can't get past a problem that dumb and so easy to spot, catch and kill, then who's going to bet on "safe C++"?
[*] ONE subset? No, let's have C++ "profileS", plural! More choice!
Posted Oct 13, 2024 19:48 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link] (15 responses)
I sit in on SG23 (the "Safety and Security" study group for ISO C++) and participate in discussions, but have not contributed any papers (mostly I try to help provide accuracy improvements when statements about Rust come up as I have done a fair amount of Rust programming myself, something that is thankfully seeming to increase among other members as well). It is my understanding that multiple profiles is because:
- *some* progress has to be able to be made in a reasonable amount of time;
Though Sean Baxter (of Circle) et al. are working on the second item there, it is *far* more radical than things like "make sure all array/vector indexing goes through bounds-checked codepaths" that one has a hope of applying incrementally to existing codebases.
I believe the plan is to lay down guidelines for profiles so that existing code can be checked against them as well as other SGs coming to SG23 to ask "what can I do to help conform to profile X?" with their work before it is merged.
Posted Oct 14, 2024 10:19 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (14 responses)
From the outside, though, what seems to be happening is not "there is a goal of a 'one true safe profile', such that all code can be split into an 'unsafe' part and a 'safe' part, but to make progress, we're coming up with ways for you to say 'I comply only with this subset of the safety profile'", but rather "let 1,000 profiles conflict with each other, such that you cannot combine code written under different profiles safely". This has the worrying corollary that you expect 'safe' C++ to be impossible to safely compose, since there's no guarantee that two libraries will choose the same safety profile.
If it was clear that the intent is that there will eventually be one true profile that everyone uses, and that the reason for a lot of profiles existing is that we want to make progress towards the one true profile in stages, then there would be less concern.
Posted Oct 14, 2024 14:30 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (13 responses)
Posted Oct 14, 2024 16:34 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (12 responses)
For all practical purposes, there's no difference between 2 conflicting profiles and 1,000 profiles. The key bit is that you're perceived (by people showing up to ISO) as adding more profiles to "resolve" conflicts between the ultimate goal, rather than working towards a single safe C++ profile, with multiple supersets of "allowed" C++ to let you get from today's "C++ is basically unsafe" to the future's "C++ is safe" in stages.
Ultimately, what's scary is that it looks like the idea is that I won't be allowed to write code in safe C++, and mix it with safe C++ from another team, because we might choose different profiles; I can only see two paths to that:
Otherwise, you end up with the language fragmenting; if you have 3 profiles that conflict, and I write in profile 1, while you write in profile 2, the combination of our code is no longer safe C++. The only way to avoid this is to mandate that we all use the same profile; but then you've spent all that effort writing your profiles, only to see it wasted because all but 1 profile is not used.
Posted Oct 14, 2024 17:34 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (11 responses)
Where do you forsee any profiles conflicting in such an incompatible way? Sure, it could happen, but there are specific broad profiles proposed and I'm not aware of anything inherently being in conflict. I'd be surprised if anything like that were known during the design and discussion and it not be addressed.
Posted Oct 14, 2024 18:19 UTC (Mon)
by smurf (subscriber, #17840)
[Link]
What happens if I code to profile A and need to call some library function that changes the state of some variable to, let's say from shared to unshared (assuming that this is of concern for Profile A; could be anything else), but which isn't annotated with profile A's directives — because it's from library B which is coded according to profile C (or no profile at all) instead?
Answer: you spend half your productive time adding profile A's extensions to B's headers, and the other half arguing with your manager why you're even using B (and/or A) when you have to put so much nonproductive effort into it.
The result will be that you now have a fork of B. Alternately you spend even more time *really* fixing its profile-A-violations instead of just papering them over (assuming that you have its source code, which is not exactly guaranteed), sending its authors a pull request, and even more time convincing them that coding to Profile A is a good idea in the first place — which will be an uphill battle if the internals of B aren't easily convertible.
Contrast this kind of what-me-worry attitude with Rust's, where spending considerable effort to avoid "unsafe" (if at all possible) is something you do (and your manager expects you'll do, as otherwise you'd be coding in C++) because you fully expect that *every* nontrivial or not-carefully-reasoned-out use of it *will* bite you sooner or later.
Posted Oct 14, 2024 18:32 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (9 responses)
The versions I've heard about from SG23 members who pay attention to these things are cases where the different profiles assume different properties of "safe C++", such that if I, in module C, import modules A and B that each uses a different profile of "safe C++", the interactions between modules A and B through their safe interfaces, as intermediated by C, result in the safety guarantees made by their respective safety profiles being broken.
To put it in slightly more formal terms, each profile needs to be a consistent axiomatic system, such that anything that cannot be automatically proven safe using the profile's axioms is unsafe, and the human is on the hook for ensuring that the resulting code is consistent with the profile's axioms. The problem that multiple profiles introduce is that all pairs of profiles need to be relatively consistent with each other, or any complete program where two modules use different profiles has to be deemed unsafe.
I think we agree that "two modules using different profiles means your entire program is unsafe" is a bad outcome. But I'm arguing that fixing the problem of "we can't agree on axioms and theorems in our safe subset" by having multiple sets of axioms and theorems that must be relatively consistent with each other is simply expanding the amount of work you have to do, for no net gain to C++.
Posted Oct 14, 2024 20:34 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (8 responses)
Yes…they wouldn't be different if not. But I guess I'm having trouble seeing where these conflicts live given that no specific rules have been laid down to even compare yet. The boundaries of where guarantees happen is certainly something that needs to be considered. The situation is going to be messier than Rust, but it's a lot harder to get moving with billions of lines of code you can't rewrite.
I think it's more along the lines of "module A says it has bounds safety and module B says it has thread safety". Module C using them together doesn't mean B can't mess up bounds or A gets itself in a thread-related problem by the way they get used together, but the thread bugs shouldn't be in B and the bounds bugs shouldn't be in A.
Posted Oct 14, 2024 21:04 UTC (Mon)
by farnz (subscriber, #17727)
[Link] (7 responses)
Unless the profiles are subsets of a "final" target, it's very hard to avoid accidental conflicts based on assumptions; for example, module A says it has bounds safety, but the assumptions in the "bounds safety" profile happen to include "no mutation of data, including via calls into module A, from any module that does not itself have bounds safety", and module B does not have bounds safety, thus causing module A to not have bounds safety because module B calls a callback in module C that calls code in module A in a way that the bounds safety profile didn't expect to ever happen unless you have both the bounds safety and thread safety profiles on module A. As a result, module A claims to have bounds safety, but module A has messed up bounds because it was "bounds safety on the assumption of no threading from modules without bounds safety".
Now, if it looked like SG23 were doing this because they knew it would make things a lot harder, and probably put off safe C++ until C++40 or later, I'd not be so concerned; but the mathematical nature here (of axiomatic systems) means that by splitting safety into "profiles", you've got all the work you'd have to do for a single "safe C++", plus all the work involved in ensuring that all the profiles are consistent with each other in any combination - and as the number of combinations of profiles grows, that work grows, too. If you have a "bounds safety" profile and a "thread safety" profile, then you need to ensure that the combination of bounds safety and thread safety is consistent. But you also need to ensure that bounds safety plus not thread safety is consistent, and that thread safety plus not bounds safety is consistent, and so on. Add in a third profile, and now you have to ensure consistency for all three profiles at once, all three cases of 1 profile, and all 3 pairs of profiles, and it just gets worse as the profile count goes up.
Posted Oct 16, 2024 13:43 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (6 responses)
I don't think any profile can guarantee that it can't be subverted by such situations (e.g., even Rust guarantees can be broken by FFI behaviors). The way I forsee it working is that profiles get enough traction to help avoid some kind of "no C++ anymore" regulations by instead allowing "C++ guarded by profiles X, Y, Z" with ratcheting requirements as time goes by. If you need bounds safety, you need bounds safety and the bug needs to be addressed in the right location.
Thanks for the discussion, by the way. I'll keep this in mind as I participate in SG23 to help keep an eye on it.
Posted Oct 16, 2024 14:50 UTC (Wed)
by Wol (subscriber, #4433)
[Link] (5 responses)
So it's opt-in (so it won't break existing programs), but would get rid of a huge swathe of programmer logic errors. And it might even help different profiles work together ...
Cheers,
Posted Oct 16, 2024 16:09 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link] (3 responses)
Posted Oct 16, 2024 17:48 UTC (Wed)
by Wol (subscriber, #4433)
[Link] (2 responses)
Division by zero I think is simple - "if the hardware supports it, set the value to IEEE Infinity. Otherwise if the compiler detects it it's compiler error else it's a runtime error. Allow the user to override the hardware and crash".
I'd be quite happy with an answer of "The compiler can't detect it, so not our problem".
But I'm thinking of the easy wins like "signed integer overflow is UB". At present I gather most compilers optimise your naive programmers' test for it away on the grounds of "if it can't happen then the following code is dead code". Most compilers have a flag that defines it as 2s complement, I believe, though that's not the default.
So I'm guessing that I should have worded it better as "All UB that the compiler can detect and act upon needs to have that action documented, and options provided to configure sane behaviour (for the skilled-in-the-arts end-developer's definition of sane). And standard bundles of such options could/should be provided".
As they say, the whole is greater than the sum of its parts, and you will get weird interactions between optimisations. But if the "safe profiles" emphasise the end-developer's ability to know and control how the compiler is interpreting UB, we (a) might get more predictable behaviour, (b) we might get more sensible conversations about what is and is not possible, and (c) we might get fewer unpleasant surprises from those weird interactions, because end-developers will disable optimisations they consider not worth the candle.
Effectively what the linux kernel is already doing to some extent. But we want to get rid of the masses of options that you need a Ph.D. to understand, and just have a few "here this gives you a sane bundle of options". It's rather telling that most of the options linux specifies are --no-f-this, --no-f-that ... Actually, that "sane bundle of options" sounds like what -O1, 2, 3 should be doing, but I strongly get the impression it isn't.
Cheers,
Posted Oct 16, 2024 18:18 UTC (Wed)
by smurf (subscriber, #17840)
[Link]
This is an "easy win" in the sense that it's a solved problem: both gcc and clang have a '-fwrapv' option.
Most UBs are *not* "easy wins". The past discussion here should have told you that already.
Posted Oct 16, 2024 18:43 UTC (Wed)
by mathstuf (subscriber, #69389)
[Link]
And if thread 1 wins on the first byte and thread 2 on the second byte, which is "last"? Reading it gives a value that was never written. I believe the JVM handles this by saying that *a* value is returned, but it doesn't have to be a value that was ever written.
> Division by zero I think is simple - "if the hardware supports it, set the value to IEEE Infinity. Otherwise if the compiler detects it it's compiler error else it's a runtime error. Allow the user to override the hardware and crash".
Sure…assuming IEEE support and things like `-fno-fast-math`. What do you propose for integer division here?
> I'd be quite happy with an answer of "The compiler can't detect it, so not our problem".
So…can the compiler optimize based on any assumptions around this behavior? I mean, if the behavior is implementation-defined to be "I dunno", what kinds of as-if code transformations are allowed?
> Actually, that "sane bundle of options" sounds like what -O1, 2, 3 should be doing, but I strongly get the impression it isn't.
In general, the "optimization level" flags are a grabbag of heuristics and wildly varying behavior across implementations. For example, Intel's toolchains turn on (their equivalent of) `-ffast-math` on `-O1` and above.
Posted Oct 16, 2024 16:58 UTC (Wed)
by marcH (subscriber, #57642)
[Link]
https://blog.llvm.org/2011/05/what-every-c-programmer-sho...
Posted Oct 13, 2024 11:08 UTC (Sun)
by mb (subscriber, #50428)
[Link] (38 responses)
I think this is also not the case in Rust, where =Default::default() is a rough equivalent of C's ={}.
It only doesn't ever let you access the padding without unsafe. But it is still uninitialized and UB on read, unless you explicitly memset the whole thing with zeros before.
Posted Oct 13, 2024 19:50 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link] (37 responses)
Posted Oct 13, 2024 22:33 UTC (Sun)
by khim (subscriber, #9252)
[Link] (36 responses)
It's undeterminate, but not UB. Normally reading the uninitialized value in Rust is UB, but reading padding is specifically excluded: Uninitialized memory is also implicitly invalid for any type that has a restricted set of valid values. In other words, the only cases in which reading uninitialized memory is permitted are inside unions and in “padding” (the gaps between the fields/elements of a type). It's just too easy to write code that reads padding, for one reason or another, thus keeping it UB was considered to be too dangerous. But while reading padding is not an UB, the result is still some random value, and not zero.
Posted Oct 14, 2024 7:54 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (35 responses)
Imho, that's the wrong logic entirely. There is no reason it should be UB, therefore it shouldn't be.
Accessing the contents pointed to by some random address should be UB - you don't know what's there. It might be i/o, it might be some other program's data, it might not even exist (if the MMU hasn't mapped it). You can't reason about it, therefore it's UB.
But padding, uninitialised variables, etc etc are perfectly valid to dereference. You can reason about it, you're going to get random garbage back. So that shouldn't be undefined. But you can't reason about the consequences - you might want to assign it to an enum, do whatever with it that has preconditions you have no clue that this random garbage complies with. Therefore such access should be unsafe.
Principle of least surprise - can you reason about it? If so you should be able to do it.
Cheers,
Posted Oct 14, 2024 13:08 UTC (Mon)
by khim (subscriber, #9252)
[Link] (34 responses)
Thanks for showing us, yet again, why C/C++ couldn't be salavaged. Note how you have entirely ignoring everything, except your own opinion (based on how hardware works). Just because there are no reason for it to be UB from your POV doesn't mean that there are not reason for it to be UB from someone's else POV. And, indeed, that infamous Yes, it may not make much sense for the “we code for the hardware” guys, it may be somewhat problematic, but, in Rust, it's not easy to hit that corner case (just look on how many contortions I had to do to finally be able to shoot myself in the foot!) and it helps compiler developers thus it was decided that access to “normal” uninitialized variable is UB. Even if some folks think it shouldn't be. Nope. That's not how both C and Rust work. And in Rust reading uninitialised variables is UB while reading padding is not. Because that's where the best compromise for all parties involved was found. Just how much usafe? What about that pesky Except that was tried for many decades and simply doesn't work. Answer to the can you reason about it? is very much depends on the person that you are asking. But to write something in any language at least two persons should give the same answer to that question: the guy who writes the compiler (or interpreter) and the guy who uses said compiler. Simple application of principle of least surprise only works reliably when the sole user of the language is also its developer – and in that case it's very much not needed.
Posted Oct 14, 2024 15:02 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (32 responses)
> Nope. That's not how both C and Rust work. And in Rust reading uninitialised variables is UB while reading padding is not. Because that's where the best compromise for all parties involved was found.
Are you saying that the memory location is not allocated until the variable is written? Because if the memory location of the variable is only created by writing to it, then fair enough. That however seems inefficient and stupid because you're using indirection, when you expect the compiler to allocate a location.
But (and yes maybe this is "code to the hardware" - but I would have thought it was "code to the compiler") I'm not aware of any compilers that wait until a variable is written before allocating the space. When I call a function, isn't it the norm for the COMPILER to allocate a stack frame for all the local variables? (yes I know you can delay declaration, which would delay allocation of stack space.) Which means that the COMPILER allocates a memory location before the variable is accessed for the first time. Which means I may be getting complete garbage if I do a "read before write", but fair enough.
Okay, I don't know how modern gcc/clang etc work, but I'm merely describing how EVERY compiler (not many) I've ever had to dig into deep understanding of, work.
So no, this is NOT "code to the hardware". It's "code to the mental model of the compiler", and if you insist it's code to the hardware, you need to EXPLAIN WHY.
Or are you completely misunderstanding me and thinking I mean "dereference a pointer to memory" - which is not what I said and not what I meant!!! OF COURSE that's UB if you are unfortunate/stupid/incompetent enough to do that - taking the contents of the address of the pointer is perfectly okay - the compiler SHOULD have made sure that the address of the pointer points to valid memory. USING that contents to access memory is UB because using garbage as pointer to memory is obviously a stupid idea.
Cheers,
Posted Oct 14, 2024 15:21 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (17 responses)
Posted Oct 14, 2024 15:40 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (16 responses)
Going back to the "to be or not to be", if uninitialised variables are defined as containing the value "undef", about which you cannot reason, then "be || !be" would refuse to compile. But it wouldn't be UB, it would be an illegal operation on an undefined variable.
If, however, the language allows you to operate on undefined variables inside an unsafe block, then to_be_or_not_to_be() would be an unsafe function, only callable from other unsafe functions, unless it actively asserted that it itself would not return a value of "undef".
(Like SQL allows logical operations on NULL, where any NULL in the expression means the result is NULL.)
And if you have a "convert random garbage to boolean" function, that can even handle undef and poison, then to_be_or_not_to_be() would just be like any normal function - guaranteed to return a boolean, just not necessarily a predictable one.
Cheers,
Posted Oct 14, 2024 17:28 UTC (Mon)
by mathstuf (subscriber, #69389)
[Link] (15 responses)
There are codepaths which make it impossible to know if something is actually initialized. Diagnostics to that effect are madness. Rust can avoid a lot of these issues because tuples are a language-level thing and functions wanting to return multiple values don't have to fight over which one "wins" the output while the rest live as output parameters. C++ at least has tuples, but they are…cumbersome to say the least.
bool be;
Given that reading if it is actually uninitialized it is UB, the optimizer is allowed to assume that it is initialized. It may choose to emit a diagnostic that it is doing so, but it doesn't have to (and that may end up in all kinds of unwanted noise). Since the valid representations are "0" or "1", `be || !be` can be as-if implemented as `be == 1 || be == 0` which leads to UB effects of "impossible branches" being taken when it ends up being a bit-value of 2.
Posted Oct 14, 2024 18:01 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (2 responses)
… in C++. There is no such thing in Rust — unless you're using "unsafe", that is. If you do, it's your responsibility to hide 100% of that unsafe-ness from your caller, at as low a level as possible, where reasoning about the state of a variable is still easy enough to do something about it.
Posted Oct 14, 2024 19:08 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (1 responses)
Cheers,
Posted Oct 14, 2024 19:58 UTC (Mon)
by smurf (subscriber, #17840)
[Link]
That argument no longer holds when it's spread over multiple functions, or even compilation units.
Posted Oct 14, 2024 19:17 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (11 responses)
What does do_init() do? If it merely returns whether or not "be" is initialised, then how can the optimiser assume that it is initialised? That's a massive logic bug in the compiler!
If, on the other hand, it forces "be" to be a valid value, then of course the compiler can assume it's initialised. But that would be obvious from the type system, no?
Cheers,
Posted Oct 14, 2024 19:42 UTC (Mon)
by khim (subscriber, #9252)
[Link] (10 responses)
That's easy: because correct program is not supposed to read uninitialized variable it can conclude that on all branches where it's read it's successfully initilialized. Then it's responsibility of developer to fix code to ensure that Nope. When you call
Posted Oct 14, 2024 20:19 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (9 responses)
> That's easy: because correct program is not supposed to read uninitialized variable it can conclude that on all branches where it's read it's successfully initilialized. Then it's responsibility of developer to fix code to ensure that
Circular reasoning !!! Actually, completely screwy reasoning. If do_init() does not alter the value of "be", then the compiler cannot assume that the value of "be" has changed!
Let's rephrase that - "Because a SAFE *function* is not supposed to read an uninitialised variable".
to_be_or_not_to_be() knows that "be" can be 'undef'. Therefore it either (a) can apply boolean logic to 'undef' and return a true boolean, or (b) it has to return an "unsafe boolean", or (c) it's a compiler error. Whichever route is chosen is irrelevant, the fact is it's just *logic*, not hardware, and it's enforceable by the compiler. In fact, as I understand Rust, normal compiler behaviour is to give the programmer the ability to choose whichever route he wants!
All that matters is that the body of to_be_or_not_to_be() is marked as unsafe code, and the return value is either a safe "true boolean", or an unsafe boolean that could be 'undef'. At which point the calling program can take the appropriate action because IT KNOWS.
Cheers,
Posted Oct 14, 2024 20:31 UTC (Mon)
by khim (subscriber, #9252)
[Link]
Nope. Normal compiler behavior is still the same as in C: language user have to ensure program doesn't have any UBs. The big difference is that for, normal, safe, subset of Rust it's ensured by the compiler. But for unsafe Rust it's still resposibility of the developer to ensure that program doesn't violate any rules WRT UB. Nope. It couldn't be How does it know? You couldn't look on
Posted Oct 14, 2024 20:41 UTC (Mon)
by daroc (editor, #160859)
[Link] (7 responses)
Posted Oct 14, 2024 23:43 UTC (Mon)
by atnot (subscriber, #124910)
[Link] (6 responses)
I agree it may have been a worthy topic once upon a time. But when the same two people (khim and wol) have the same near-identical drawn out discussions for dozens of messages a week in every amenable thread to the point of drowning out most other discussion on the site (at least without the filter) and making zero progress on their positions over a span of at least 2 years, perhaps some more action is necessary.
Posted Oct 15, 2024 10:13 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (5 responses)
Unfortunately, I suspect the language barrier doesn't help. I think sometimes we end up arguing FOR the same thing, but because we don't understand what the other one is saying we end up arguing PAST each other.
Cheers,
Posted Oct 15, 2024 11:15 UTC (Tue)
by khim (subscriber, #9252)
[Link] (4 responses)
It could be a language barrier but I have seen similar discussions going in circles endlessly with a pair of native speakers, too, thus I suspect problem is deeper. My feeling is that it's ralated to difference between how mathematicians apply logic and laymans do it. No, that's the issue. The big problem with compiler development (and language development) lies in the fact that compiler couldn't answer any interesting questions about semantics of your program. And that's why we go in circles. Wol arguments usually come in the form of: And my answer comes in the form of: And Wol ignores the most important #2 step from my answer and goes back to “compilers should do “the right thing”… even if I have no idea how can they do that”. I have no idea why is it so hard for layman to accept that compilers are not omnipotent and compiler developers are not omnipotent either, that there are things that compilers just could never do… but that's the core issue: “we code for the hardware” guys somehow have a mental model of a compiler which is both simulatentously very simple (as discussion about how modern compilers introduce shadow variables for each assignment show) and infinitely powerful (as discussions about what compiler have to detect and report show). I have no idea how compiler can be simultaneously primitive and yet infinitely powerful, that's definitely a barrier right there, but it's not a “native speaker” vs “non-native speaker” barrier. And I continue the discussion to see if that barrier could be broken, somehow – because in that case C/C++ would have a chance! If that barrier could be broken then there's a way to reform the C/C++ “we code for the hardware” community! But so far it looks hopeless. Safe languages would still arrive, they would just have to arrive the sad way, one funeral at time way. And that would be the end of C/C++, because “one funeral at time” approach favors transition to the new language.
Posted Oct 15, 2024 16:18 UTC (Tue)
by atnot (subscriber, #124910)
[Link]
Posted Oct 15, 2024 16:42 UTC (Tue)
by intelfx (subscriber, #130118)
[Link] (2 responses)
Posted Oct 15, 2024 21:58 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (1 responses)
And when I try to argue LOGICally, if people argue straight past my logic and don't try show me what's wrong with it, I end up concluding they have to be attacking the messenger, because they are making no attempt to attack the message.
Khim is saying Science advances one funeral at a time, but it'll be his funeral not mine - if you look back over what I've written it's clear I've learnt stuff. Don't always remember it! But as I've said every now and then in the past, you need to try to understand what the other person is arguing. I get the impression khim finds that hard.
Cheers,
Posted Oct 15, 2024 22:13 UTC (Tue)
by daroc (editor, #160859)
[Link]
Posted Oct 14, 2024 19:45 UTC (Mon)
by dezgeg (subscriber, #92243)
[Link] (3 responses)
Posted Oct 14, 2024 20:32 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (2 responses)
But I'm not arguing that attempting to dereference garbage is okay. That post explicitly says the programmer chose to return garbage. No problem there. The problem comes when you attempt to USE the garbage as if it was valid. Rust would - I assume - have marked the return value as "could be garbage", and when the caller attempted to dereference it without checking, Rust would have barfed with "compile error - you can't unconditionally dereference possible garbage".
The point is, the programmer can reason about it because Rust would force them to track the fact that the return value could be garbage.
Cheers,
Posted Oct 14, 2024 20:43 UTC (Mon)
by khim (subscriber, #9252)
[Link]
Nope. Rust doesn't do that. Rust developer may use If I lie to the compiler (like I did) at that point – that's an instant UB. IOW: Rust does the exact same thing C/C++ does but mitigates the issue by making transition from “could be uninitialized” type to “I believe it's initialized now” type explicit. This couldn't be a compiler error, but sure enough, if you violate these rules and compiler can recognize it then there would be a warning. It's warning, not an error, because compiler may recognize this situation it but is not obliged to do that, it's something that it does on “best effort” basis.
Posted Oct 14, 2024 22:03 UTC (Mon)
by dezgeg (subscriber, #92243)
[Link]
int global;
Ie. direct example of an architecture where what you wrote ("But padding, uninitialised variables, etc etc are perfectly valid to dereference. You can reason about it, you're going to get random garbage back.") doesn't apply.
Posted Oct 14, 2024 20:12 UTC (Mon)
by khim (subscriber, #9252)
[Link] (9 responses)
Worse. For decades already most compilers create new location for every store to a variable. GCC started doing it around 20 years ago, LLVM did from the day one. Why would you use indirection? TreeSSA doesn't need that, you just create many copies of variable (one for each store to a variable). No. Consider the following example: Here local variable
> It's "code to the mental model of the compiler", and if you insist it's code to the hardware, you need to EXPLAIN WHY.
It's “code to the hardware” because “mental model of the compiler” couldn't explain how your program works without invoking hardware that would be executing your program. And it's only needed when you ignore rules of the language like they are described in the specification and then invent some new, entirely different specification of what you are program is doing and then assert that your POV is not just valid for today, but it would be the same 10, 20, 50 years down the road. That's… quite a presupposition. No, that's not about pointers, but about simple variables. Accessing them, if they are not initialized, is UB because normal program shouldn't do that and assuming that correct program doesn't access uninitialized variable is benefocial for the compiler. Heck, there are whole post dedicated to that issue on Ralf's blog.
Posted Oct 14, 2024 21:02 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (8 responses)
> Worse. For decades already most compilers create new location for every store to a variable. GCC started doing it around 20 years ago, LLVM did from the day one.
Ummmm ... my first reaction from your description was "Copy on write ??? What ... ???", but it's not even that!
The other thing to bear in mind is that this is a quirk of *optimising* compilers, and it completely breaks the mental model of "x = x = x", so it breaks the concept of "least surprise". And I don't buy that's "coding to the hardware". If I have a variable "x", I don't expect the compiler to create lots of "x"s behind my back!!! And while we might be dying off, there's probably plenty of people, like me! for whom this was Ph.D. research for our POST-decessors.
But. SERIOUSLY. What's wrong with either (a) pandering to the principle of least surprise and saying that dereferencing an uninitialised variable returns "rand()" (even if it's a different rand every time :-), or it returns 0x0000... The former makes intuitive sense, the latter would actually be useful, and if it's a compiler directive it's down to the programmer! which fits the Rust ethos.
If the hardware is my brain, after all, dereferencing an unitialised variable would be a runtime error, not an opportunity for optimisation and code elimination ...
Cheers,
Posted Oct 15, 2024 6:41 UTC (Tue)
by khim (subscriber, #9252)
[Link] (6 responses)
How do you call it, then? Just as you wouldn't expect CPU to create hundred accumulators in place of one that machine code uses? CPUs do that for about 30 years. Just how you wouldn't expect to have one piece of memory to contain different values simultaneously? CPUs started doing that even earlier. I'm not telling your that to shame you for not knowing, I'm explaing to you why assuming that compiler (or hardware) would work in a certain “undocumented yet obvious” way doesn't lead to something that you may trust. There are many things that are implemented both in hardware and compiler via an as-if rule… and you don't need to even know about them if you are using language in accordance to the language specification. The same with hardware. That's why I really like The Tower of Weakenings that Rust seems to embrace. With it normal developers have 90% of code that's “safe” and doesn't need to know anything about how hardware and compilers actually work. But in Weren't we just talking about how major scientific breakthroughs arrive one funeral at time? Looks like that's how safety if low-level languages would arrive, too. And that's why C/C++ wouldn't get it, in practice: language can be changed to retrofit safety into it, but who would use it in that fashion and why?
Posted Oct 15, 2024 10:09 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (5 responses)
> How do you call it, then?
Brain-dead logic.
> And that's why C/C++ wouldn't get it, in practice: language can be changed to retrofit safety into it, but who would use it in that fashion and why?
So why am I left with the impression that it's PERFECTLY ACCEPTABLE for Rust to detect UB and not do anything about it? I take - was it smurf's - comment that "we can't promise to warn on UB because we can't guarantee to detect it", but if we know it's UB?
If a programmer accesses an uninitialised variable there's a whole bunch of reasons why they might have done it. The C(++) response is to go "I don't believe you meant that" and go and do a load of stuff the programmer didn't expect.
My understanding of the ethos of Rust is that if the compiler doesn't understand what you mean it's either unsafe, or an error. You seem to be advocating that Rust behave like C(++) and just ignore it. Sorry if I've got that wrong.
Imho (in this particular case) Rust should come back at the programmer (like any sensible human being) and ask "What do you mean?". Imho there's three simple scenarios - (1) you expect the data to come from somewhere the compiler doesn't know about, (2) you forgot to explicitly request all variables are zeroed on declaration / read-before-write, or (3) you're expecting random garbage. (If there's more, just add them to the list.)
So no you don't just dump stuff into UB, you should force the programmer to explain what they mean. And if they don't, it won't compile.
Cheers,
Posted Oct 15, 2024 10:33 UTC (Tue)
by farnz (subscriber, #17727)
[Link] (3 responses)
Rust simply doesn't have UB outside of unsafe. There's no detection of UB involved at all - all the behaviour of safe Rust is supposed to be fully defined (and if it's not, that's a bug). For example, in the case of the Rust equivalent of bool be; int status = use_be(&be);, the defined behaviour is for the program to fail to compile because be is possibly used before it is initialized.
Posted Oct 15, 2024 21:40 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (2 responses)
I don't necessarily think it's the best definition, but it IS defined and it IS in-character for Rust. Which UB would not be.
Cheers,
Posted Oct 16, 2024 9:46 UTC (Wed)
by laarmen (subscriber, #63948)
[Link] (1 responses)
let b: bool = unsafe { MaybeUninit::uninit().assume_init() }; // undefined behavior! ⚠️
I lifted this from the MaybeUninit doc, including the comment. That will compile, but *is* UB.
Now, I'm of the opinion that it is perfectly reasonable for Rust to declare this UB, as the alternative makes a lot of assumptions about the underlying implementation, all for a use case that seems dubious to me.
Posted Oct 18, 2024 14:27 UTC (Fri)
by taladar (subscriber, #68407)
[Link]
Posted Oct 15, 2024 10:42 UTC (Tue)
by khim (subscriber, #9252)
[Link]
I'm not “advocating” anything, I'm just explaining how things work. Not how they “should work”. But how they, inevitably, have to work (and thus how they actually work). If out of ten choices that one like or dislike only one is actually implementable then you are getting that one whether you like it or not. We are talking the full Rust, not just “safe” Rust here. UB is UB, whether it's in safe code or unsafe code. And yes, you can trigger UB in safe Rust – and it would lead to the exact same outcome as in If programmer accesses an uninitialized variable without the use of special construct that is allowed to touch The only difference of safe Rust and Compiler may include warning for [potential] bugs in unsafe Rust, but ultimately it's resposibility of developer to fix them. Impossible. Compilers are mindless (they literally have no mind and couldn't have it) and not sensible (they don't have “a common sense” and attempts to add it inevitable lead to even worse outcome). That's something “we code for the hardware” people simply just refuse to accept for some unfathomable reason. In that case you have to use volatile read or volatile write. This is bug and it should be fixed. If you managed to do that in normal, “safe” Rust then it's bug in the compiler and it have to be fixed in compiler, if you did that in Currently that's also a bug, although there are discussions about adding such capability to the language (to permit tricks like the one used in the Using Uninitialized Memory for Fun and Profit. Currently Rust's only offer for such access is the use of asm!. Not possible, sorry. If you wrote the magic Compiler may still detect and report suspicious findings, but it couldn't be sure that it detected everything correctly thus such thing couldn't be a compile-time error, only and compile-time warning.
Posted Oct 15, 2024 6:57 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
No it doesn't. If the compiler knows at all times where your particular x lives at any given time, your mental model isn't violated. Your mental model of the apple you're going to have for tomorrow's breakfast doesn't change depending on which side of the table you put it on, does it?
Consider code like
Now why should the compiler allocate space for four variables when c can easily be stored at a's location? it's not needed any more. d doesn't even need to be stored anywhere, simply clean the stack up and jump to fn4 directly, further bypassing any sort of human-understanding-centered model (and causing much fun when debugging optimized code).
Constructing a case where a ends up in multiple locations instead of none whatsoever is left as an exercise to the reader.
Posted Oct 14, 2024 15:22 UTC (Mon)
by Wol (subscriber, #4433)
[Link]
If bool is defined as a value that can ONLY contain 1 or 0, and the location of bool contains something else (for example it's not just a single bit-field), then as I pointed out you are assigning invalid garbage to a field that cannot contain it. Either you have a coercion rule that guarantees that "be" will be understood as a boolean whatever it contains, or yes it should be a compile error or unsafe or whatever.
But imho looking at that simple example, the problem boils down to interpreting random garbage as a boolean. If the rules of the language let you do that, then I would expect it to return true - after all, the expression "(garbage) or not (garbage)" must evaluate to true if "garbage" can be interpreted as a boolean.
It's all down the language you're using, and whether that language lets you treat any random junk as a boolean. If it does, then those functions make mathematical sense and should not be UB. If the language DOESN'T let you, then it should fail to compile with an "illegal coercion" error.
And that has absolutely nothing to do with the hardware, and everything to do with mathematical logic. (Unless, of course, your DRAM behaves like a random number generator unless/until it's written to, such that (garbage) != (garbage) ...) And even there, I'd just say that you can guarantee the function will return a boolean, just not necessarily the boolean you expect ...
Cheers,
Posted Oct 13, 2024 11:50 UTC (Sun)
by khim (subscriber, #9252)
[Link] (6 responses)
True, but I would put 10% blame on compiler developers and 90% on “we code to the hardware” people. And you show why exteremely well. Which is not any different from what Rust is doing, isn't it? What is “obvious/natural” for one developers is not at all obvious/natural for other developer. And when developers refuse to “play by the rules” and use “who told you one couldn't carry the basketball and run – I tried it I could do that just fine” approach then safety remains a pipe dream. Sure, convoluted and strange rules of C and C++ with, literally, hundreds of UBs, don't help, but these rules could be changes. Except this wouldn't matter at all if people wouldn't accept them. And that is why all attempts to make C or C++ safer would fail: you could change a language, but it's almost impossible to change developers' attitude. The only known way is via Pank's principle: science progresses one funeral at a time… and it looks like language development happens in the exact same way. First we had Mel who refused to adopt assembler, then we had debate about structured programming, today “old school” developers play Mel's tricks with C/C++, loudly complain about compilers and refuse to adopt anything else… we couldn't predict whether Rust or something else would, eventually, replace C/C++ but we know it wouldn't happen peacefully, by people adopting the successor willingly and it wouldn't happen because someone mandate that C/C++ should be dropped… this leaves us with one funeral at a time… approach. That one tried, tested and works.
Posted Oct 13, 2024 12:29 UTC (Sun)
by pizza (subscriber, #46)
[Link] (5 responses)
....How dare hardware not perfectly conform to the spherical cow abstractions of higher level languages [1]
> And when developers refuse to “play by the rules” and use “who told you one couldn't carry the basketball and run – I tried it I could do that just fine” approach then safety remains a pipe dream.
Oh, you mean "if it's not explicitly disallowed then it by definition it must be allowed" attitude of nearly _every_ field of human endeavour? ("metrics" become the only thing that matter, because that's what you're judged/rewarded by..)
[1] "Higher-level" notably includes C _and_ Rust.
Posted Oct 13, 2024 14:12 UTC (Sun)
by smurf (subscriber, #17840)
[Link] (4 responses)
Well the workaround is easy[1], you disallow everything, except under proscribed circumstances.
Like Rust's "you can't write *any* data (or free a struct, or …) unless you can prove that there's no concurrent access".
1) for some value of "easy", anyway.
Posted Oct 13, 2024 19:43 UTC (Sun)
by Wol (subscriber, #4433)
[Link] (3 responses)
Or you disallow UB inasmuch as you can! And don't create new UB!
If the standard doesn't define it, you have to impose a sensible default (eg, on a 2s complement machine, you assume 2s complement), and allow a flag to change it eg --assume-no-overflow.
At which point you get a logical machine that you can reason about without repeatedly getting bitten in the bum. And the compiler writers can optimise up the wazoo for the benchmarks, but they can't introduce nasty shocks and make them the default.
Cheers,
Posted Oct 13, 2024 22:33 UTC (Sun)
by khim (subscriber, #9252)
[Link] (2 responses)
How would that work without buy-in on the compiler users side? The Rust handling of UB (which actually works fine so far) hinges not on one, but two pillars: But C/C++ community is 100% convinced that doing something about UBs is responsibility of “the other side”. Just read “What Every C Programmer Should Know About Undefined Behavior” (parts one, two, three) and What every compiler writer should know about programmers Developers demand that compiler developers should ignore the language specifications and accept program with “sensible code” as programs without UB and, as we saw, when asked what is “sensible code” immediately present something that Rust also declares as unspecified. That's very symptomatic: that example shows, again (as if articles above weren't enough), how both sides are entirely uninterested in working together toward common goal in a C/C++ world. Defining padding as containing zeros is extremely non-trivial, because many optimizations rely on ability of the compiler to “separate” a struct (or union) into a set of fields and then “assemble” them back. And because the ability to have predictable padding is very rarely needed the decision, in a Rust world, was made to go with making it unspecified. Note that's is a compromise: usually reading uninitialized value is UB in Rust (to help compiler), but reading padding is not UB. From the reference: Uninitialized memory is also implicitly invalid for any type that has a restricted set of valid values. In other words, the only cases in which reading uninitialized memory is permitted are inside unions and in “padding” (the gaps between the fields/elements of a type). This decision doesn't make any side 100% happy: compiler makers would like to make reading padding UB to simplify their job and compiler users would like to make it zero to simplify their job, but “unspecified but not UB” is good enough for both sides to, grudgingly, accept it. Yet such compromise may not ever save the world where it's always the other side that have to do the job! Nope. Logical machine that high-level language uses to define behavior of a program would always be different from “what the real hardware is does”. That's what separates high-level language from low-level language, after all. You can make it easier to understand or harder to understand, but someone who refuses to accept the fact that virtual machine used by high-level language definition even exists may always find a way to get bitten in the bum.
Posted Oct 14, 2024 13:24 UTC (Mon)
by Wol (subscriber, #4433)
[Link] (1 responses)
Well, maybe, if we didn't have stupid rules like "don't use signed integer arithmetic because overflow is undefined", the user space developers might buy in. I'm far more sympathetic to the developers when they claim there's too much UB, than I am to compiler developers trying to introduce even more UB.
In that particular case, I believe we have -fno-wrap-v or whatever the option is to tell the compiler what to do, but that should be the norm, not the exception, and it should be the default.
If the compiler devs change to saying "this is UB. This is what a user-space developer would expect, therefore we'll add a (default) option to do that, and an option that says 'take advantage of UB when compiling'", then I suspect the two sides would come together pretty quickly.
Compiler devs know all about UB because they take advantage of it. User devs don't realise it's there until they get bitten. The power is in the compiler guys hands to make the first move.
And if it really is *undefinable* data (like reading from a write-only i/o port), then user space deserves all the brickbats they'll inevitably get ... :-)
Cheers,
Posted Oct 14, 2024 15:19 UTC (Mon)
by khim (subscriber, #9252)
[Link]
This was tried, too. And, of course, what the people who tried that have found out that was that every “we code for the hardware” developer have their own idea about what compiler should do (mostly because they code for different hardware). Why? They have already done the first step and created a specification that lists what is or isn't UB. If developers have a concrete proposals – they could offer changes to it. And yes, some small perceentage of C/C++ community works with developers. But most “we code for the hardware” guys don't even bother to even look and read it… how can compiler developers would know that their changes to that list would be treated any better then what they already gave? They would probably find a way to complain even in that case… And that's precisely the problem: if language have UBs and users of said language “only realise it's there when they get bitten” then such language couldn't be made safe. Ever. Either developers have to accept and handle UBs pro-actively or language should have no UBs at all. Latter case is limiting because certain low-level things couldn't be expressed in the language without UBs thus, for low-level stuff, safe and reliable language couldn't be made unless developers think about UBs before they are bitten. And the only known way to introduce such a radical non-technical change is to create a new community. And that's the end of story: C/C++ couldn't be made safe not because there are some technical, unsolvable, issues, but because it's community refuses to accept the fact that there are technical, unsolvable, issues (that have to be worked around on social level).
Posted Oct 17, 2024 8:51 UTC (Thu)
by peter-b (subscriber, #66996)
[Link] (1 responses)
Posted Oct 17, 2024 16:39 UTC (Thu)
by kleptog (subscriber, #1183)
[Link]
> It’s not surprising that the C++ community hasn’t discovered a better way to approach safe references than the lifetime parameter model. After all, there isn’t a well-funded effort to advance C++ language-level lifetime safety. But there is in the Rust community. Rust has made valuable improvements to its lifetime safety design. Lots of effort goes into making borrow checking more permissive: The integration of mid-level IR and non-lexical lifetimes in 2016 revitalized the toolchain. Polonius[polonius] approaches dataflow analysis from the opposite direction, hoping to shake loose more improvements. Ideas like view types[view-types] and the sentinel pattern[sentinel-pattern] are being investigated. But all this activity has not discovered a mechanism that’s superior to lifetime parameters for specifying constraints. If something had been discovered, it would be integrated into the Rust language and I’d be proposing to adopt that into C++. For now, lifetime parameters are the best solution that the world has to offer.
Basically, if there was a simpler way then likely Rust developers would have already found it and implemented it, because they're actually looking for ways to make things simpler. The C++ community is barely engaging with the topic. The likely end result seems fairly obvious to me.
Posted Oct 16, 2024 4:58 UTC (Wed)
by jezuch (subscriber, #52988)
[Link]
Well, we've been hearing that for years (decades?) and it just keeps not happening. I wonder why? 🤔
Posted Oct 20, 2024 11:15 UTC (Sun)
by ssokolow (guest, #94568)
[Link]
(i.e. The impression I got was less "usually, very hard to use and reason about for the uninitiated" and more "they didn't prove useful enough to justify being a core language feature as opposed to a design pattern".)
Posted Oct 11, 2024 14:20 UTC (Fri)
by asahilina (subscriber, #166071)
[Link] (4 responses)
In C, the best you can do is multiply compile the same file with a bunch of #ifdefs declaring the conditions under which certain structs exist or have a different type. This gets verbose quickly, it's ugly to compile, and you also need to wrap every single public top-level identifier in the file so it has a unique name when compiled for each version combination. You can't mix and match code that changes by version and code that doesn't in the same file, which forces code organization that might be different from what logically makes sense. And the more of the driver you cover with this mechanism, the more awkward it gets, which encourages you to add an explicit abstraction layer with dynamic dispatch, which is then slower and reduces the compiler's ability to optimize. Or you end up covering the entire driver in the scheme, which means all the internal interfaces within the driver need the name mangling applied, or you end up wanting to put everything into fewer files so you can use static, or doing something like having all the source files #included into a single compilation unit... and then there's the bug-proneness of this all, because C doesn't force you to initialize all fields of a struct, so if a given firmware build adds a field the compiler won't check that you actually handle it in every code paths that uses that struct.
Basically, it's a bunch of bad tradeoffs. We're doing something like this for the DCP (display) driver (written in C) and it's quite messy and bug prone.
In Rust, I wrote a rather ugly but functional proc macro [1] that allows you to apply multiple compilation to any top level block in a source file, and then conditionally compile any statement or declaration. The name mangling is automatic for declarations, and to reference them you just append ::ver to the name. Firmware definitions end up looking like [2] and the code that uses them like [3] (search for "ver"). Other parts of the driver can be pulled into the multiple compilation easily without any changes to code organization, and I only use dynamic dispatch (Rust trait objects) where it makes sense to insulate a significant chunk of shared code from the versioning. Then the compiler takes it all and basically creates N variants of the driver for each firmware/GPU combination, optimizing everything together, and sharing the bits that can be shared. If I accidentally mismatch the firmware declarations and their usage (extra field, missing field, wrong type) for any version variant, the compiler will complain.
[1] https://github.com/AsahiLinux/linux/blob/asahi-wip/rust/m...
That's just the versioning story, but Rust also helps get complex firmware interfaces right in general with its rich type system. Firmware pointers can be typed with the firmware type they point to as a generic parameter, even though from the CPU/Rust's perspective they aren't pointers or references at all, just a u64. I even have lifetime-bound strong firmware pointers that guarantee that the firmware object they point to outlives the firmware object containing the pointer.
We don't have this yet, but it's also possible to use macros to verify that a firmware struct declaration has no implicit padding and only uses types that allow all bit patterns, which is important to avoid undefined behavior and portability issues, and allow you to read/write those structs from the firmware without having to use "unsafe".
Posted Oct 11, 2024 14:29 UTC (Fri)
by adobriyan (subscriber, #30858)
[Link] (2 responses)
struct S {
I've seen std::conditional_t implementation on some video (which looks almost exactly like C++ version) which makes me think they can not.
Posted Oct 11, 2024 15:11 UTC (Fri)
by farnz (subscriber, #17727)
[Link]
That's built-in, and can be done via macros, or via the existing #[cfg] attribute. Macros in general in Rust get an AST, and output a new AST, so can do pretty much anything (noting that only function-like macros get an arbitrary AST; attribute macros and derive macros must be given valid Rust source code).
But using the built-in #[cfg] mechanism, along with feature flags, you'd write that as:
This says that the field b only exists if the feature flag "config_b" is set at compile time; there's also preset cfg items, like target_has_atomic, which lets you write #[cfg(target_has_atomic = "64")] to only include a chunk of code on platforms with support for 64 bit atomics.
Posted Oct 11, 2024 15:26 UTC (Fri)
by khim (subscriber, #9252)
[Link]
You don't need macros for that because that's handled by separate facility in Rust. You probably misunderstood the precise limitation, I'm afraid. Macros in Rust are incredibly powerful, but they still work with a token trees. You can support configurable code and conditional compilation with them just fine, but they don't have access to the type information. If you go from C++ to Rust then yes, this limitation feels quite stifling. If you go from C to Rust… nope: C simply doesn't have anything similar to reflection. magic_enum couldn't be replicated in Rust, but then it doesn't work with C, either.
Posted Oct 12, 2024 1:46 UTC (Sat)
by rcampos (subscriber, #59737)
[Link]
Posted Oct 16, 2024 21:13 UTC (Wed)
by chexo4 (subscriber, #169500)
[Link] (1 responses)
I know there are various companies that offer it as a service, but I feel like the risks of security issues and difficulty auditing would be more than just… updating to a newer kernel
Posted Oct 17, 2024 6:32 UTC (Thu)
by smurf (subscriber, #17840)
[Link]
In "enterprise" space there's the lack of exhaustive automated tests, plus managers who convince themselves that a somewhat-random backport to a known kernel is somehow safer than a new "unknown" Linux version.
It's basically the same reason why Boeing went to such lengths to pretend that the 737-MAX was just another 737 — instead of a completely different aircraft that requires ex[pt]ensive re-certification. We know the results.
Posted Oct 19, 2024 0:54 UTC (Sat)
by jsakkine (subscriber, #80603)
[Link] (3 responses)
Posted Oct 19, 2024 1:07 UTC (Sat)
by intelfx (subscriber, #130118)
[Link] (2 responses)
That shouldn't be a prerequisite.
Posted Oct 19, 2024 7:07 UTC (Sat)
by smurf (subscriber, #17840)
[Link] (1 responses)
> That shouldn't be a prerequisite.
I can only assume that a few kernel maintainers more-or-less-strongly disagree.
IMHO a strong and demonstrated commitment for the GCC people to "get there", in the not-too-distant future, should go a long way towards assuaging their concerns.
NB: There are standards organizations that require at least two, sometimes three, independent implementations before they even start thinking about standardizing something.
Posted Oct 21, 2024 8:35 UTC (Mon)
by taladar (subscriber, #68407)
[Link]
Fade off
Fade off
A new policy is born...
A new policy is born...
A new policy is born...
A new policy is born...
A new policy is born...
Keep in mind that the code that turned into Rust for Linux was originally developed out-of-tree. I was not at all optimistic about it getting upstream quickly (and I'm very grateful to Miguel and the many others who got it there!); I was instead envisioning that out-of-tree module authors, who wouldn't benefit from LKML reviewing their C, might prefer to write those modules in a language where the compiler can check things for you. To that end, the original code wasn't a branch of linux.git; it used the Kbuild support for out-of-tree modules and it had conditional compilation support for kernel 4.4 onwards.
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
I think it's not uncommon for enterprise/LTS kernels to need a separate GCC to build the kernel than the normal /usr/bin/gcc, whether it's because the distro is backporting a newer stable branch that needs a newer compiler, or because you need an older kernel to be ABI-compatible, or whatever. In particular this happened in the early Spectre days when you needed a newer version of GCC for retpoline support.
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Who is doing this with Qt in distributions? I don't know of anyone doing that.
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Rust on <=6.0 kernels is possible
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
u32 a;
u32 b;
};
u32 c;
u64 a;
u32 b;
};
Insulating layer?
* Safe Rust is not capable of overflowing a buffer without help from (incorrectly-written) unsafe Rust. So if your buffer changes size out from under you, and your code does not handle that correctly, you might end up panicking the kernel, but at least you won't get your device pwned (unless there is something wrong in an unsafe block, but those are hopefully small, infrequent, and audited more carefully than the rest of the codebase).
* Rust can represent and statically check more arbitrary requirements (such as "must call the init method before you can call anything else, must not call anything after shutdown," etc.) with typestates, which are not practical in C for multiple reasons (no real ZST support, no visibility restrictions, no move-only types, etc.).
Insulating layer?
Insulating layer?
> I'm not much of a programmer, so maybe I'm just hopelessly ignorant, but how does Rust provide that insulation layer?
Insulating layer?
malloc/realloc/free while in Rust it's represented by Vec. And both languages provide two mutually incompatible sets of operations:
Insulating layer?
Insulating layer?
Insulating layer?
Clue: it will never happen. More to the point, it cannot happen, both for technical and social reasons.
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Maybe I'm just too stupid for Rust. Like I'm too stupid for Haskell. Maybe refcounting is the right answer for my lack of intelligence - but then it seems many other programmers are like me. And it seems maybe even the best Rust programmers too, given how they always end up reaching for (A)RC too.
Insulating layer?
>to be used to compute at compile time looks to be much more ergonomic (though, havn't tried Zig yet).
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
>
> Linked lists are as _niche_ and _vague_ of a data structure as a trie. Few would balk at me claiming a trie is a niche structure that your average programmer could happily never learn in an entire productive career -- and yet linked lists have some bizarre celebrity status.
> But I think it's still important to consider whether your design uses (doubly-)linked lists because they really are the best data structure for your needs, or whether it's just because linked lists are really easy to implement in C (and an equivalent of Vec is hard) so they're your default choice for any kind of sequential container, and you're trying to apply the same mindset to Rust where they're really hard (but Vec is easy) and they should never be the default choice.)
Challenges learning how to model data structures in Rust
Vec the difference is not large.Vec is the way to go and linked lists are [mostly] useless… there wasn't any point in time when linked lists stopped being useful and Vec started shining. Transition was very slow and gradual: first Vec was awful and linked lists great, then Vec have become a tiny bit less inefficient and thus become better for some tasks, while linked list have become less efficient and thus worse for these some task, etc.Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Challenges learning how to model data structures in Rust
Yes, many things *can* be implemented in safe Rust, but that doesn't mean they should.
If you can drop reference counting by implementing it in unsafe code and manually checking the rules, then do it. This is actively encouraged by the Rust community. The only thing that you must do is keep interfaces sound and do all of your internal manual safety checks.
Insulating layer?
I have tried playing with Rust. There's a lot to like. There's some stuff I dislike - the macro system especially makes me go "Wut?", Zig's approach of allowing the /same/ language to be used to compute at compile time looks to be much more ergonomic (though, havn't tried Zig yet).
The declarative macro syntax is an interesting beast because it's somehow so effective at disguising that a macro is just a match statement to simulate function overloading with some Jinja/Liquid/Twig/Moustache/Handlebars/etc.-esque templating inside, disguised by a more Bourne-like choice of sigil.
In the little I've played with Rust, the lifetime annotations make sense on simpler stuff. But it seems to get very baroque on more complex stuff.
True. On the plus side, they are still actively working to identify places where they can remove the need for weird tricks. See, for example, the Precise capturing use<..> syntax section in this week's announcement of Rust 1.82.0.
Maybe I'm just too stupid for Rust. Like I'm too stupid for Haskell. Maybe refcounting is the right answer for my lack of intelligence - but then it seems many other programmers are like me. And it seems maybe even the best Rust programmers too, given how they always end up reaching for (A)RC too.
One of Rust's greatest weaknesses has always been that its "make costs explicit" philosophy has created a language that encourages premature optimization, and people in the community concerned with writing learning materials spend an inordinate amount of time encouraging people to write quick and dirty code (eg. String instead of &str, .clone() everywhere, etc.) first, learn the more advanced stuff later, and always profile before optimizing.
std::shared_ptr.) However, for better or for worse, having people averse to it does seem to contribute to the trend that Rust implementations of things have low and stable memory usage.
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
- An "owning" reference may only ever be stored to automatic, local variables.
- A "shared" reference is ref-counted, and can be stored to multiple variables (using whatever refcounting machinery/helpers I've put in place)
Insulating layer?
Insulating layer?
Insulating layer?
Wol
Insulating layer?
Insulating layer?
Wol
Insulating layer?
Does Rust actually have a memory allocator, or does it have a "new"?
new, you give the struct at least one private member so it can't be initialized directly and write a public associated function (i.e. a public class method) which constructs and returns an instance (Named new by convention only.) ...but that doesn't automatically mean heap allocation. It's purely a matter of whether there are "correct by construction" invariants that need to be enforced.
Box<T> (std::unique_ptr in C++) or Vec<T> or you do as they do internally and use the unsafe wrappers around malloc/calloc/realloc/free in std::alloc.
Could the Rust compiler handle "this memory is now out of scope, just forget about it"?
impl Drop and, if you don't, a stack allocation will just be forgotten until the whole frame is popped and a heap allocation will go away when the owning stack object's Drop is run and frees the memory.
Insulating layer?
> E.g., perhaps it isn't necessary at all to even need to represent lifetime types. Perhaps it would be sufficient to have 2 kinds of references - local and refcounted, with well-defined and safe conversion semantics enforced by the language.
Insulating layer?
strchr are designed around ability to pass reference to local variable somewhere. It's not even possible to pass buffer that you would fill with some values to another function!Insulating layer?
Wol
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Wol
Rust has arena allocators
Insulating layer?
Insulating layer?
.clone that you don't want to see on every line..clone, if you need to!), if it's invisible – you wouldn't even think about whether what you are doing is expensive or not.Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
It also seems false that the compilers "go out of their way to set bits to 1", if anything it would seem those bits are left untouched (stack memory)?Insulating layer?
union foo {
char x;
int y;
};
...
union foo f = { 0 };
clang created code to initialize x=0 and the other three bytes to the pattern, but ={} actually does initialize it all. The actual case was harder to understand because there was a struct involved, possibly with sub-structs, but ultimately the first member was the union.
Insulating layer?
= { 0 } used to mean zero init the whole thing though...
The argument to the system call is a union of structs for the different system call subcases.
The kernel requires everything past a certain point to be zero-filled or it assumes the non-zero values have meaning (ie. future extensions) and it can't understand them and thus returns an error.
Obviously the kernel doesn't know what portion of the key is relevant, so it hashes everything - padding included, so if padding isn't 0, stuff just doesn't work (lookups fail).
Yes, the obvious answer is to make *all* padding explicit.
Of course, this is also a good idea because userspace might be 32-bit and the BPF .o code is 64-bit, so you also don't have agreement on things like sizeof(long) or its alignment. You want (or rather need) asserts on all struct sizes.
Yes, once these gotchas get you, you know better, and there are workarounds...
But it's bloody annoying that you have to think about stuff like this.
I just picked one single one...
Insulating layer?
Insulating layer?
- waiting until all of C++ is covered by a single "safe" marker is likely to take until something like C++32 (nevermind all of the things added between now and then); and
- even if a single "safe" marker were possible, deployment is nigh impossible if there's not some way to attack problems incrementally rather than "here are 3000 diagnostics for your TU, good luck".
Insulating layer?
Insulating layer?
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Wol
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Wol
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Number of profiles is kinda irrelevant here
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Wol
> Imho, that's the wrong logic entirely. There is no reason it should be UB, therefore it shouldn't be.
Insulating layer?
be || !be is very much UB in both Rust and C.be || !be? Should it guaranteed to return true or is it permitted to return false or crash? Note that both C and Rust allow all three possibilities (although rustc tries to help the programmer and make it crash when it's easily detectable, but such behavior is very much not guaranteed).Insulating layer?
Wol
Insulating layer?
Insulating layer?
Wol
Insulating layer?
int status = do_init(&be);
// Is `be` initialized or not? `init` is a function call whose implementation is not visible at compile or link time.
Insulating layer?
Insulating layer?
Wol
Insulating layer?
Insulating layer?
Wol
> If it merely returns whether or not "be" is initialised, then how can the optimiser assume that it is initialised?
Insulating layer?
read(2) nothing in the typesystem differs for return values that are larger and smaller than zero.Insulating layer?
Wol
> In fact, as I understand Rust, normal compiler behaviour is to give the programmer the ability to choose whichever route he wants!
Insulating layer?
undef. In Rust MaybeUninit<bool> can be undef, but regular bool have to be either true or false. Going from MaybeUninit<bool> to bool when it's undef (and not true or false) is an instant UB.MaybeUninit<bool:gt; and ask it whether it's initialized or not. It's still very much a resposibility of Rust user to ensure that program doesn't try to convert MaybeUninit<bool> which contains undef into normal bool.Insulating layer?
Insulating layer?
Insulating layer?
Wol
> Unfortunately, I suspect the language barrier doesn't help.
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Wol
Insulating layer?
Insulating layer?
Insulating layer?
Wol
> Rust would - I assume - have marked the return value as "could be garbage"
Insulating layer?
MaybeUninit<bool> to signal to the compiler that value may be uninitialized. And then Rust developer (and not compiler!) would decide when to go from MaybeUninit<bool> to bool (which would tell the compiler that at this point value is initialized).Insulating layer?
void f() {
int uninitialized;
global = uninitialized;
}
> Are you saying that the memory location is not allocated until the variable is written?
Insulating layer?
struct II {
int x;
int y;
};
int foo(struct II);
struct LL {
long x;
long y;
};
int bar(struct LL ll) {
struct II ii = {
.x = ll.x,
.y = ll.y
};
return foo(ii);
}
ii ii never allocated at all.Insulating layer?
Wol
> And I don't buy that's "coding to the hardware".
Insulating layer?
unsafe world there are also gradations. In Linux kernel there are certain tiny amount of code (related to RCU, e.g.) that touches everything simultaneously: the real hardware, compiler internals, and so on. But if said code is only 0.01% of the whole and the majority of developers work with the other 99.99% of code then precise rules used in that tiny part could be ignored by most developers.Insulating layer?
Wol
Insulating layer?
So why am I left with the impression that it's PERFECTLY ACCEPTABLE for Rust to detect UB and not do anything about it? I take - was it smurf's - comment that "we can't promise to warn on UB because we can't guarantee to detect it", but if we know it's UB?
Insulating layer?
Wol
Insulating layer?
Insulating layer?
> You seem to be advocating that Rust behave like C(++) and just ignore it. Sorry if I've got that wrong.
Insulating layer?
unsafe Rust.undef, then it's a bug. Period, end of story. If program includes such access then it have to be fixed, there are no any other sensible choice.unsafe Rust is decision of whose responsibility is it to fix such bug. If it's “safe” Rust then it's bug in the compiler (currently there are around 100 such bugs) and compiler developers have to fix it, if it's in unsafe Rust, then developer have to fix it.unsafe Rust, then it's bug in your code and you have to fix it.unsafe keyword then it's your responsibility to deal with UB now.Insulating layer?
a=fn1()
b=fn2(a)
c=fn3(a,b)
d=fn4(b,c)
return d
Insulating layer?
Wol
> Honestly with C/C++ a large part of the problem is the compilers themselves (and/or their developer's mindsets) and their willingness to bend over backwards to not define UB and intentionally not make it do the obvious(ly correct/natural) thing.
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
Wol
> Or you disallow UB inasmuch as you can! And don't create new UB!
Insulating layer?
Insulating layer?
Wol
> If the compiler devs change to saying "this is UB. This is what a user-space developer would expect, therefore we'll add a (default) option to do that, and an option that says 'take advantage of UB when compiling'", then I suspect the two sides would come together pretty quickly.
Insulating layer?
Insulating layer?
For example, you could just have lexically-scoped borrows (the borrow is not allowed to escape a given lexical scope) plus a syntax rule that prohibits moving a borrowed object within the borrow's lexical scope. This is not fully general, and fails to express many things that Rust can handle easily, but when you combine it with C++'s existing smart pointers, it is probably enough for many practical applications (that don't need ultra-low overhead).
Please read this excellent recent WG21 paper which addresses your suggestions in detail: P3444 "Memory Safety without Lifetime Parameters".
Insulating layer?
Insulating layer?
Insulating layer?
Insulating layer?
[2] https://github.com/AsahiLinux/linux/blob/asahi-wip/driver...
[3] https://github.com/AsahiLinux/linux/blob/asahi-wip/driver...
Insulating layer?
int a;
#ifdef CONFIG_B
int b;
#endif
};
Insulating layer?
struct S {
a: i32,
#[cfg(feature = "config_b")]
b: i32
}
Insulating layer?
std::conditional_t (and whatever Rust equivalent you saw) work with types, not with defines, and yes, that's something macros couldn't touch (neither in C/C++ nor in Rust) simply because macros work before types come into existence.Insulating layer?
Why are enterprise kernels so old?
Why are enterprise kernels so old?
Rust-GCC
Rust-GCC
Rust-GCC
Rust-GCC