Toward a more approachable Rust
The Rust programming language is designed with the principle that developers should never have to choose between performance and safety. While performance has long been the domain of the C programming language, C can often be the source of security bugs. Rust might one day be a viable replacement, but it is still maturing. This year, the project's leaders are changing their focus to make sure the Rust ecosystem can continue to grow.
C has long been
the workhorse of low-level development. It has held up remarkably well
over the years. As its creator once quipped:
"C is quirky, flawed, and an enormous success.
"
C grants programmers the freedom to write high-performance code
that is virtually unmatched by any other language. That freedom also
comes with its own pitfalls. There is even a book
devoted to the subject. That same freedom creates the opportunity
to introduce bugs that would not be possible in a higher-level
language.
Rust has a strict type system that is designed to eliminate most of the memory-management errors that are a common source of bugs in low-level languages. (Rust's feature list even includes "guaranteed memory safety".) Rust introduces some new ideas not seen in other mainstream languages that make this possible. But these unique features, combined with Rust's rich syntax, may also be a barrier to the widespread adoption of the language.
Rust aims to match the performance characteristics of C while eliminating the safety risks. This is not enough, however, to guarantee its success. The Rust project leaders have decided that the biggest challenge facing the language is improving the on-boarding process for new programmers. While Rust is still young, the first stable release was in May 2015, most of the design and technical features are set in place. In 2017, there will not be any focus on new changes to the language or the standard library:
When the Rust project conducted its 2016 survey it asked users what stood in the way of being able to use Rust. The top responses received were:
- 1 in 4: learning curve
- 1 in 7: lack of libraries
- 1 in 9: general "maturity" concerns
Some of the top priorities this year include improving documentation, for example a new version of the Rust book is under development, improving error messages generated by the compiler, and refining editor integration and other tooling. The full details are available in the 2017 roadmap.
A real-world test
Mozilla, which also sponsors the Rust project, has already started shipping Rust components in Firefox. Recently, another high-profile project had the opportunity to explore if Rust could actually replace C. The NTPsec project, which is intended to replace the Network Time Protocol (NTP) reference implementation, has a large code base written in C. Eric Raymond, who is working on NTPsec, has already completed a major cleanup of the existing code. He is also considering a complete rewrite in either Rust or Go.
In theory, Rust looked like a clear winner over Go and could even be a viable replacement for C. Rust is more "C-like", with machine code output, no garbage collection, and a stricter type system than Go. But Raymond concluded that Rust not only did not fit the project's needs, but that in its current state, Rust is unfit for production. The immaturity of the Rust ecosystem and the language syntax created a tax on productivity:
Even things that should be dirt-simple in Rust, like string concatenation, are unreasonably difficult. The language demands a huge amount of fussy, obscure ritual before you can get anything done.
The contrast with Go is extreme. By four days in of exploring Go I had mastered most of the language, had a working program and tests, and was adding features to taste.
Instead of a community design process, Raymond wonders if the Rust project might benefit from a single "Benevolent Dictator" (or a "core team" of some kind) making decisions. There is some evidence for the need for this in other parts of the language. Rust's type system lays the groundwork for concurrency, but the implementation is provided by libraries and is not part of the core language. According to the Rust documentation, this was a deliberate design decision to give programmers flexibility in their code:
Contrast that with the Go language, whose designers chose to use communicating sequential processes as the formal model to manage concurrency in the language. Signals and channels are first class primitives in Go, which makes it relatively easy to pass messages between threads.
The Rust roadmap does include plans to improve this situation. While the standard library is unlikely to change, there is a renewed focus on improving the quality of packages provided by Rust's third party "crates". Concurrency is one of the specific areas the project developers would like to improve. There is also work underway to improve the overall maturity of the ecosystem, including a ranking system for packages.
While Go's focus on programmer productivity may make it a better
fit for NTPsec, Raymond hasn't actually committed
to rewriting any of the remaining C code. In the meantime, the Rust
project is listening to its critics. With the renewed focus on
improving the language's usability, Rust is likely to become a more
competitive choice for a wider variety of use cases. But that is all in the
future. Until then, C will remain a safe bet.
| Index entries for this article | |
|---|---|
| GuestArticles | Kovsky, Eddie |
Posted Feb 23, 2017 9:52 UTC (Thu)
by oever (subscriber, #987)
[Link] (10 responses)
The borrow checker is challenging to learn, but when you understand it, you have not simply learned to appease the borrow checker, you've learned to write good code. Most of the issues that the borrow checker uncovers is unsafe cost. Code that would have been much harder to do right in C and code which I probably would not have done right in C but which might have worked well on the happy path.
Rust has many things that are easier to learn than in C. Macros are nice and sane. Building and packaging is simple. Writing tests is simple and writing code snippets in documentation is simple. In fact, Rust compiles and checks all code snippets in documentation! It even has benchmarking built into the packaging system, cargo.
Of course there are downsides. Some code is quite verbose. Rust does not have function overloading. It does not implicitly convert integer types. But these are trade-offs needed to achieve speed and safety. You have to tell the compiler exactly what you mean.
The most important aspect of Rust for me is that you can trust your code much better. There are way less hidden gotcha's in code. This makes it easier to read other people's code.
Before Rust, my favorite programming language was Haskell. Rust is less elegant in the mathematical sense, but it makes up for that by being very practical and still very fast and safe.
Posted Feb 24, 2017 8:26 UTC (Fri)
by marcH (subscriber, #57642)
[Link]
https://air.mozilla.org/introduction-to-rust-for-ocaml-pr...
Posted Mar 2, 2017 8:10 UTC (Thu)
by ras (subscriber, #33059)
[Link] (8 responses)
Which goes some way to explaining why you didn't find the learning curve high at first.
> To me the initial learning curve was not high until I hit issues with the borrow checker.
It is always is that way, isn't it?
I discovered HackerRank, and taught myself Rust by doing a number of problems there - maybe 20, maybe 50 - I'm not sure, but definitely more than the 4 days ESR spent with it. He's about my age, and he if did manage to master the language in 4 days he's a much, much smarter man than I. Haskell and it's type inference isn't something I've come across in any language I am familiar with, so the learning curve was punishing. Even so I grew to like the language, and fell in head over heels in love with the build system. Towards the end I thought I was pretty good with the borrow checker, but one a nagging doubt remained - those practice programs were all very small.
I decided to test myself by doing some serious data structure work with Rust. I general, I could not do it, at least not without lots of unsafe blocks. If I was to summarise the pain point, it would be because the borrow checkers reasoning starts and ends within the stack, mostly within the current stack frame. Use stuff on the heap and things get very difficult very quickly. Try doing tree insertion without using recursion some time - I don't think it's possible. I ended up wishing Rust had some sort of "const" feature for struct's - fields the compiler could know would never change outside of constructors and destructors. That way the bloody borrow checker extend stack lifetimes to pointers within struct's, and so could derive information about the lifetimes on stuff on the heap too.
Still, I was a relative newbie at Rust, so I went along Servo talk. Servo is Mozilla engine written entirely in Rust. In fact it's the largest Rust project and has been a big driver of the language. Naturally it uses the heap a lot, so I wanted to see how they got around code littered with unsafe's. Sadly answer was they hadn't managed it. They were hoping one the top Rust language guru's would look at the worst library and fix it. Maybe he could, but if it took someone like that a mere mortal like myself had no hope in day to day day usage. I just don't have that sort of time at my disposal.
I didn't see the point of going to all that cognitive effort of figuring out how to appease the borrow checker, when for arguably the most dangerous part of memory - the heap, you can't do it. I gave up at that point.
Posted Mar 3, 2017 1:13 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (7 responses)
Granted, I have also programmed in Haskell previously, but if anything, learning these languages has improved my code in Python, C++, and others because I can now more easily spot potential problems with pointer manipulation and the like.
There are times I fight the borrow checker, but in the end it has (so far) always been right short of the non-lexical lifetime issue (which has a path to resolution now). It lets me specify in my interface that some structure holds a resource necessary for another (e.g., a temp directory which is deleted when it goes out of scope).
What kinds of problems were you trying to solve that you needed so much unsafe code?
Posted Mar 3, 2017 2:41 UTC (Fri)
by ras (subscriber, #33059)
[Link] (6 responses)
I may not have been clear. In all the simple stuff I did there was no unsafe code, and I was never tempted reach for it. It was when I grew suspicious that I was solving only simple "Rust friendly" problems, and actively searched for places that it might fall down that the rot set in.
You don't have to look far to find buckets of unsafe code: have a read of the standard library some time. I often did that to figure out how the a rust expert would attack a problem. It was seeing all the unsafe code in there that first raised my suspicions: how was that different to what I was doing?
I gave you the answer: the problems started to arise when you use the heap, explicitly. In particular you have stuff on the heap referencing other objects on the heap. You can avoid explicit use of the heap a lot of the time (for me it was all of the time when doing simple stuff), by using Rusts inbuilt collections. These collections do use the heap internally of course, and as a consequence if you peak inside of them you will see many unsafe blocks. But a person using those collections to store simple objects won't need to use unsafe's. If you are doubting this try writing your own collection with iterators, without using Rusts collections internally. For extra bonus points don't use recursion.
One common "cheat" I've seen used several time (along with "see, you don't need unsafes in Rust") is to put the objects you want to reference into an array, and then store the index into that array. It's cheating because every index lookup does contain an unsafe block, and that unsafe block costs time (the index bounds check) at run time.
The canonical Rust solution to heap pointers to other heap objects it to wrap everything in RefCell's. Aside from leading to some truly ugly type descriptions [0] it means you are absorbing a runtime hit every time you move between pointers. One of the borrow checkers purposes in life is to allow you to move around data structures without incurring that hit by doing the checks at compile time. It works well only when everything is on the stack.
[0] From stdio.rs: Arc<ReentrantMutex<RefCell<LineWriter<Maybe<StdoutRaw>>>>>
Posted Mar 3, 2017 13:25 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (5 responses)
One other reason for unsafe is performance because Rust is also not blind to the realities of hardware. This helps it reach actual use rather than things like Haskell where things like laziness and space leaks become an issue.
I suggest you look at some of the lower libraries like Rayon which has (I believe) just a handful of unsafe bits. Everything else can be checked and verified by the compiler. Also check out the improvements to the sorting algorithms which are 90%+ safe code (last I saw it).
The type you gave describes what it is: thread-shared line buffered output that may or may not exist, can be accessed via reentrant calls. What would it be in C++? Just another pointer with a sidecar mutex. Yep, no way to screw that up.
Yes, you need unsafe somewhere because not everything does, internally, abide by the rules. But when you make those blocks and make the rules conform, the code using it can be reasoned about much more easily. Even if it's not perfect, it's still way better than what is available in C, C++, D, or Go. What else would you suggest these days?
Posted Mar 3, 2017 18:48 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Mar 4, 2017 3:50 UTC (Sat)
by ras (subscriber, #33059)
[Link] (3 responses)
I guess what I was looking for is something that I would use instead of Python. I was willing to give up simplicity for that, although it was a bit of a shock to see just how complex things became. "Code flowing off the fingers" is not something I've experienced with Rust. However, I did expect to get more than I actually received in exchange for that complexity.
Perhaps an example would help. Take a doubly linked list implementation. Lets call an element a DListElt. It has a number of operations operations you can perform on it. Lets look at two of them: DListElt.insert() and DListElt.remove(). You can call insert() only when it's not in a list, and remove() only when its in a list. It would be nice to be able to verify this at compile time (thus avoiding the overhead of run time checks, so it would be as fast a C), so for example if you try to call remove() when it's in a list, you get a compile error.
Rust can't do this. What it can do is very limited: it can attach a borrowed state to a type, but this only works for stuff in a stack frame. It's not hard to understand why this stack limitation exists: the compiler needs to know the lifetime of a borrow and it has only two sources of known lifetimes: variables on the stack whose lifetime is the same as the stack frame, statics who live forever. This means the compiler can not reason about anything on the heap. So when confronted with my dinky little DListElt type that only has a grand total of two states which can be represented by a simple boolean (borrow's state is far more complex) that lives on the heap, it can't help. Rust's does have an inbuilt LinkedList of course, but because of this it is burdened with unsafe's and run time checks for most operations.
The frustrating thing for me is it not hard to do. (I've invented some syntax for what follows, hopefully it's meaning is obvious.) Just let the programmer create his own compile time state information, and attach it to a type. Lets invent DListElt:Inuse<bool>. Remove() and insert() then become a casts of sorts. It is defined only for DListElt:Inuse(True).remove(), and returns a DListElt:Inuse(False). Similarly insert() is defined only for DListElt:Inuse(True).insert(), and returns a DListElt:Inuse(False).
Interestingly, a very early version Rust did do something like this but they abandoned the idea in favour of the current borrow syntax. I'm not sure why. I suspect that borrow is all they needed to avoid the bulk of memory corruption issues they saw and they could not come up with generic type system that was powerful enough to implement it. I also suspect most of their examples they looked at were smallish programs, because once you need complex data structures that live on the stack the borrow checkers limitations really bite hard. At least they they bit me hard, and they bit servo too.
So yes, they have come up with a language that occupies some unique niche in the (fast, safe) space. But it's not safe enough to replace Python. It is definitely safer than C / C++, and yes as you say the unsafe blocks show you where to look for the bits that are not. If you do look you will find checks in the unsafe blocks that perform at runtime the checks the compiler could not do at compile time. This is what makes it as safe a Python for simple programs that don't need their own unsafe blocks, but it comes at the cost of making it slower than C and C++.
> I suggest you look at some of the lower libraries like Rayon which has (I believe) just a handful of unsafe bits.
I'd hope not. It just an implementation of thread polling with work stealing. The person who writes the threads (ie, *you*) still has some responsibly for ensuring those threads don't stomp on each other. https://github.com/nikomatsakis/rayon contains this truly weird statement, which to sounds me like he is trying to redefine words to wriggle out of saying just that:
> While there are threadsafe types that offer similar APIs, caution is warranted because, in a threadsafe setting, other threads may "interject" modifications in ways that are not possible in sequential code. While this will never lead to a data race --- that is, you need not fear undefined behavior --- you can certainly still have bugs.
He is talking about multiple threads executing code like this: atomic.set(atomic.get() + 1). It's true only if you use the Rust definition of undefined behaviour: don't reference memory that isn't yours. It sounds like a deliberately knobbled definition to me, but one that happens to line up with what their borrow checker can guarantee, albeit for stack only.
Posted Mar 4, 2017 4:19 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
So if you hold a reference to any entry you can traverse the whole list. To guaranteed its safety for the Rust's borrow checker you need to make sure that there are no other mutating references to it.
And since it looks like you want elements to live their own livetimes, you really need to have runtime locking that tell Rust's borrow checker that it's OK. I'm not sure how typestate (that was in early version of Rust) is going to help you here.
Posted Mar 4, 2017 9:34 UTC (Sat)
by ras (subscriber, #33059)
[Link]
> So if you hold a reference to any entry you can traverse the whole list. As a point of order, no you can't. What you call the entry is called struct Node in linked_list.rs. It is private to LinkedList, and besides has no traits that could define a next() method, or even a way to get to the list head. I presume you meant "if you hold a reference to the LinkedList you can traverse it". That's certainly true (there wouldn't be much point to it otherwise). It's done that way so your next statement holds true: > To guaranteed its safety for the Rust's borrow checker you need to make sure that there are no other mutating references to it. Another point of order. There is simply no way to inform Rust's borrow checker you have borrows on all the nodes in the list. Well that's not quite true, you can use recursion. But if you restrict yourself to one stack frame you can have only borrow per local variable so borrowing an arbitrary number of objects is impossible. The implementer of LinkedList is constrained by this too, so he must bypass the borrow checker and wrap all references to Node's in unsafe's. So the guaranteed safety you mention has nothing to do with the borrow checker. It stems purely from you trusting the guy who wrote the LinkList code. In one way none of this matters. The LinkedList author succeeds in creating the illusion a borrow on the LinkList applies to all the elements on it, so the users of the library can blissfully write code without a single unsafe under that assumption. But it does have it's costs. In the current implementation a loop iterating over all nodes must first create the integrator object on the stack (copy 3 words), and then do the same pointer load the C implementation would do. As a consequence the LinkedList has 0 or 1 nodes the C version will use a fraction of the cycles. But perhaps a different implementation could approach the C version. Unfortunately, it can't hope to have the flexibility of the traditional C implementation. For example, in a C implementation if you follow pointers in the heap to a node on the linked list removing it is a O(1) operation. For that matter, so would operation you mentioned above: "if you hold a reference to any entry you can traverse the whole list". The Rust implementation doesn't provide either of these operations, at all. If it did provide a "delete this element" function it would be O(length_of_list). I don't see how a Rust library could provide an O(1) delete for a linked list. You could do it yourself of course, but then you end up where I and servo did: code littered with unsafe's. Which is exactly the point I was making when I first responded to to this series of posts. Try to use Rust to implement a largish program with complex data structures, and you will end up with code littered with unsafes because just having a borrow checker is too limited. I don't think it's a huge improvement on C. There are lots of things that are improvements of course: the wonderful build system, the doco, cargo.io, the community, play.rust-lang. It's pretty extraordinary effort really. But the language itself - it's still underdone.
Posted Mar 4, 2017 12:14 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link]
Yes, once you get down into implementing specialized structures, you'll need some unsafe, but when properly wrapped up, you shouldn't need it. I've certainly had moments where code "flowed off of the fingers" and all I was left with, once it compiled, was minor fixes rather than things like "did you remember to check for that None that can sneak in here this way?" kind of things.
> Lets call an element a DListElt. It has a number of operations operations you can perform on it. Lets look at two of them: DListElt.insert() and DListElt.remove(). You can call insert() only when it's not in a list, and remove() only when its in a list. It would be nice to be able to verify this at compile time (thus avoiding the overhead of run time checks, so it would be as fast a C), so for example if you try to call remove() when it's in a list, you get a compile error.
This is not idiomatic in Rust anyways. Instead, you make node just the data, possibly with some accessor methods. For insert and remove, you instead make new structures which represent the action and can have their own lifetime associated with it. Look at the Entry structure for a map to see an example of this.
> So yes, they have come up with a language that occupies some unique niche in the (fast, safe) space. But it's not safe enough to replace Python. It is definitely safer than C / C++, and yes as you say the unsafe blocks show you where to look for the bits that are not. If you do look you will find checks in the unsafe blocks that perform at runtime the checks the compiler could not do at compile time. This is what makes it as safe a Python for simple programs that don't need their own unsafe blocks, but it comes at the cost of making it slower than C and C++.
Ha! Python has all kinds of unsafeties from its type system that even C++ will catch. Rust has better inference, so things are more ergonomic with the type system than C++ with things like template-derived types and the like.
> I'd hope not. It just an implementation of thread polling with work stealing. The person who writes the threads (ie, *you*) still has some responsibly for ensuring those threads don't stomp on each other.
Anything stronger would (ISTM) require a proof system because you'll always have a way to escape unless you go the Erlang route and just say "nope, can't assume an OS, so you have to use Erlang threads instead of native threads" which isn't really suitable for a language that you'd like to run directly on hardware (unless the Erlang runtime is a suitable OS I suppose).
It seems to me that you're arguing from a "you need unsafe in one place, so the whole thing is invalid" whereas those kinds of things usually go into their own libraries with test suites and should be hiding all of the unsafe blocks from you. You're also expecting magic (proof systems) instead of what Rust does guarantee: write your unsafe code (usually, data structures) carefully and your use of that can be checked and enforced by the compiler with no overhead (that is, no ASan or TSan necessary).
Posted Feb 23, 2017 10:26 UTC (Thu)
by anselm (subscriber, #2796)
[Link] (1 responses)
Time for esr to fork Rust and take over?
Posted Feb 23, 2017 19:31 UTC (Thu)
by Sesse (subscriber, #53779)
[Link]
Posted Feb 23, 2017 12:03 UTC (Thu)
by peter-b (subscriber, #66996)
[Link] (21 responses)
Given the horrendous rate at which serious, exploitable defects are being routinely found in C codebases, and the complete absence of any features in C to reliably prevent them from being introduced, I believe it's irresponsible (at best) to start any new projects in C in 2017.
Posted Feb 23, 2017 13:12 UTC (Thu)
by jnareb (subscriber, #46500)
[Link] (11 responses)
Posted Feb 23, 2017 16:40 UTC (Thu)
by jubal (subscriber, #67202)
[Link] (10 responses)
Posted Feb 23, 2017 19:04 UTC (Thu)
by Beolach (guest, #77384)
[Link] (9 responses)
Posted Feb 23, 2017 19:36 UTC (Thu)
by jubal (subscriber, #67202)
[Link]
Posted Feb 23, 2017 20:55 UTC (Thu)
by pizza (subscriber, #46)
[Link] (7 responses)
Posted Feb 23, 2017 21:30 UTC (Thu)
by Beolach (guest, #77384)
[Link] (6 responses)
Posted Feb 23, 2017 23:40 UTC (Thu)
by pizza (subscriber, #46)
[Link] (5 responses)
But it is rather disingenuous to claim that NTPsec is superior to NTPd because the latter has bugs in features that the former lacks altogether.
Posted Feb 24, 2017 1:16 UTC (Fri)
by Beolach (guest, #77384)
[Link] (4 responses)
But also in general principle, I strongly believe in minimalism/KISS/UNIX philosophy. If you say my bloat is your essential feature, fine - I'm perfectly willing to believe you, and I hope you can find or create an implementation that does include your feature & works for you (and as I said above, I really do like multiple implementations, and I view monopolies as unhealthy). But I'm also going to emphasize that the converse is equally true - one person's "essential feature" is another's "bloat", and just because I'm willing to believe it is essential to you, doesn't mean I'm willing to accept it being forced on me. And from reading the bit I quoted above, I think NTPsec is being quite reasonable about listening to arguments about what is "essential". So no, I don't think it's "disingenuous to claim that NTPsec is superior to NTPd because the latter has bugs in features that the former lacks altogether."
Posted Feb 24, 2017 11:16 UTC (Fri)
by jubal (subscriber, #67202)
[Link] (3 responses)
Posted Feb 24, 2017 20:40 UTC (Fri)
by flussence (guest, #85566)
[Link] (2 responses)
Posted Feb 25, 2017 1:27 UTC (Sat)
by jubal (subscriber, #67202)
[Link] (1 responses)
Posted Feb 26, 2017 2:49 UTC (Sun)
by ncm (guest, #165)
[Link]
Posted Feb 23, 2017 14:49 UTC (Thu)
by tshow (subscriber, #6411)
[Link] (8 responses)
C definitely wasn't built with security in mind, and there are definitely problems with it, but "C codebases" are also the majority of the attack surface out there.
Posted Feb 23, 2017 15:34 UTC (Thu)
by pizza (subscriber, #46)
[Link] (7 responses)
Are you sure about that? I mean, we're in a bit of a C echo chamber here, but large-scale data breaches are almost never due to anything C-related (ie a buffer overflow) -- instead things like not changing default/hardcoded passwords, network sniffing of insecure connections, SQL insertion, cookie hijacking (and other XSS-type stuffs), and so forth -- and of course let's not forget the "disgruntled employee" who just copies everyhing onto a USB stick and walks out the door...
Posted Feb 23, 2017 17:54 UTC (Thu)
by tshow (subscriber, #6411)
[Link]
Posted Feb 24, 2017 8:26 UTC (Fri)
by peter-b (subscriber, #66996)
[Link] (2 responses)
"Almost never"
https://blog.cloudflare.com/incident-report-on-memory-lea...
Posted Feb 24, 2017 12:35 UTC (Fri)
by pizza (subscriber, #46)
[Link] (1 responses)
C buffer overflows weren't responsible for 1 billion Yahoo accounts having their credentials compromised. They weren't responsible for the multiple point-of-sale compromises that were responsible for my credit cards being revoked three times in a single calendar year (including one actual fraud attempt) Buffer overflows weren't responsible for the Federal OMB's compromise of literally millions of highly sensitive personnel files. They aren't responsible for the Classified info leaked by Snowden or Manning. More down to earth, Buffer overflows weren't responsible for the Murai botnet with upwards of half a million webcams compromised by a fixed backdoor that yielded total access. Buffer overflows have nothing to dow with folks [re]using weak passwords on every online acount. Buffer overflows don't matter at all to [spear] phishing that deliberately targets folks with keys to the kingdom, or systems with "special managment exceptions" because executives can't be bothered to abide by the same security procedures as their underlings.
...And so on.
Sure, at an individual level those overflows can (and do) suck, and we should absolutely fix them. But let's not delude ourselves into thinking that getting rid of C sill have _any_ effect on piss-poor system design and the utter fallibility of the person sitting between the keyboard and the chair.
Posted Feb 24, 2017 12:43 UTC (Fri)
by pizza (subscriber, #46)
[Link]
Posted Feb 25, 2017 5:17 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Feb 25, 2017 12:40 UTC (Sat)
by pizza (subscriber, #46)
[Link] (1 responses)
But on a more serious note -- Due to the nature of how the Cloudfare CDN caching/proxying/rewriting works, this bug seems to be more of an information leak of random data rather than something that can be realistically harnessed/targeted to extract specific, targeted information, as it is heavily dependent on:
1) The exact page contents of what you're requesting (Granted, if you can control the contents of the page being cached, you could conceivably construct pages to maximixe this)
It's not like heartbleed where you could just hammer a single, specific, targeted server again and again and adjust the parameters until you saw something interesting.
...Am I wrong in my interpretation?
Granted, with sufficient time, data capture, and a considerable amount of analysis, one could probably map the random extra stuff one was seeing to at least specific domains, which might enable some targets of opportunity -- eg a specific user session (cookie hijacking), perhaps a random user's credentials if you're really lucky, and possibly even a MITMing of an entire domain if you're able to extract its private TLS key in drips and drabs.
It's lousy, but it's not "our entire user database has been downloaded without anyone knowing" lousy.
Posted Feb 25, 2017 16:43 UTC (Sat)
by excors (subscriber, #95769)
[Link]
That part seems trivial: sign up as a Cloudflare customer and publish a page that triggers the parser bug.
> The other page requests that were served before yours and their contents, size, timing, and whatnot. Given the scale of the Cloudfare CDN, this is the sticking point..
If you want to target users of a specific site, I guess you could perhaps write a page like:
<script>
and host it via Cloudflare, and make users run that script (e.g. distribute it through an ad network).
If the user is already logged into the target site, the first request will send their credentials (cookies etc) over HTTPS, which will hopefully end up unencrypted in the Cloudflare cache server's memory. (The XHR security model assumes GETs don't trigger side effects, and it won't allow the attacker to read the cross-origin response, so that request would normally be safe). The second request will trigger the parser bug, Cloudflare will append random memory content to the response, and (since it's a same-origin request) the attacker can read that response.
Since the two requests are being made by the same user there's a reasonable chance they'll end up going through the same cache server, and since the requests are nearly simultaneous there's a reasonable chance the leaked data will come from the corresponding first request instead of being mixed up with other users' requests. Repeat a few million times until success.
Posted Feb 23, 2017 13:32 UTC (Thu)
by moltonel (guest, #45207)
[Link] (4 responses)
I'm also not sure about the "adding Rust’s ownership/lifetime annotations to C would be a huge job" argument: there are plenty of impedance mismatches when going from C to either Rust or Go, and the task of adding annotations seems rather minor compared to (for example but without holding that against Go) starting to use channels and goroutines for concurrency.
Having played with both Go and Rust, it's clear to me that Go is easier to start with, but Rust remains much more compelling due to its design, philosophy, community. I look forward to a more approachable Rust, but there's little reason to hold off starting to use Rust today.
Posted Feb 23, 2017 14:42 UTC (Thu)
by tshow (subscriber, #6411)
[Link] (1 responses)
Posted Feb 23, 2017 15:35 UTC (Thu)
by moltonel (guest, #45207)
[Link]
Posted Feb 23, 2017 16:22 UTC (Thu)
by NickeZ (guest, #100097)
[Link]
Posted Feb 27, 2017 18:04 UTC (Mon)
by davidstrauss (guest, #85867)
[Link]
Some types of concurrency, like a fork-based worker model, preclude Rust conversion. It's not only a challenge of linking and stack integration.
Posted Feb 24, 2017 4:55 UTC (Fri)
by gerdesj (subscriber, #5446)
[Link] (3 responses)
I ended up in IT (long story, short) due to the last recession in the UK. Not the 2008 shenanigans but the one in 1991ish. I should have been a Civil Engineer but I graduated at the same time as the UK building industry decided to take a bit of a time out. Before that unfortunate event, I spent bloody ages learning stuff.
I discovered the joys of how to chase stresses and strains around a structure. I also learned a few other things, not least of which was how many bloody super and subscripts a differential equation can have with regards bits of concrete and steel. Wood is a bit different but still quantitative(ish).
I don't think that you can spend four days with a new concept or system or whatever and decide that it is rubbish. I think that what Raymond demonstrated here was a malaise particular to to IT (this is in no way directed at Raymond personally).
Why on earth is there not a clear and unambiguous way of programming a computer without resorting to machine code? Why can't you clever buggers give us laymen a language we can use? You've had over 40 years to play with.
I'll learn C in the interim - it seems to be designed by Engineers (sharp edges - wear gloves).
Cheers
Posted Feb 24, 2017 5:50 UTC (Fri)
by neilbrown (subscriber, #359)
[Link]
The elephant in the room here is that you are not a rational being, any more than I am. You can fake it a lot of the time, but underneath you are instincts and intuition and hunches and past experience and approximations. All of this works amazing well when working with wide error margins. But in the digital world, if you want error margins, you need to explicitly design them in.
Well.. to be fair, a little bit of you probably is rational, but not enough to be as productive as you want without help from the other bits.
The only thing that isn't "clear and unambiguous" here is the human psyche.
Posted Feb 25, 2017 18:10 UTC (Sat)
by shmget (guest, #58347)
[Link] (1 responses)
Furthermore, in IT, contrary to civil engineering, we usually do not afford the luxury of having a 3x to 10x 'safety' factor to hide the bugs.
Posted Mar 1, 2017 14:50 UTC (Wed)
by anton (subscriber, #25547)
[Link]
Posted Feb 24, 2017 16:30 UTC (Fri)
by excors (subscriber, #95769)
[Link] (1 responses)
It seems many philosophical differences in Rust vs Go are similar to those in C++ vs Java. E.g. C++ doesn't really support strings in its core language; instead the language provides sufficiently powerful features for the standard library to implement a reasonable std::string type. In contrast, Java has explicit language support for strings: string literals are always java.lang.String objects, the "+" operator is overloaded for java.lang.String and nothing else, etc.
If you just want to use native strings, maybe the Java approach is a little nicer; you can write "foo"+"bar" and get sensible behaviour, whereas C++ needs you to express a conversion to std::string first. The bigger difference is when you *don't* want to use the native strings. You might realise that java.lang.String is UTF-16 which nowadays is a stupid representation for strings, but there's absolutely nothing you can do about it. It's a historical mistake in the language design that application developers are stuck with forever.
In C++, since there's nothing special about the standard library, you can easily replace std::string with your own better string type for your particular use case (and many large applications do so, often with multiple string types optimised for different uses). If you want to use a validity-checked UTF-8 representation, you can do that. If you want to automatically switch between ASCII and UTF-32 representations depending on the content of the string, that's fine. If you want to use a rope data structure, no problem. You have access to operator overloading and templates and implicit constructors over string literals and so on, so your new type is not disadvantaged compared to std::string.
Similarly, Go has a built-in map type with special language support. If you want something slightly different, e.g. an ordered map or a thread-safe one or a higher-performance one, then (as far as I'm aware) you have to implement a custom type that can only support syntax like foo.Get("bar") and not foo["bar"], so it's syntactically inferior to the built-in type. In C++ and Rust the standard library has hash maps, but they're not a special part of the core language, so you can reimplement them without being demoted to a second-class citizen.
For average unadventurous programs, the Java/Go approach of trusting the language designers to provide decent data structures and concurrency features is probably fine; they're pretty smart people and it'd take a lot of effort to do better. But it's likely they made tradeoffs that aren't optimal for your specific application, or decisions that turn out to be mistakes with a couple of decades of hindsight. Even if you're smart enough and willing to put in the effort to do better, the language constrains you. The C++/Rust approach of having a small but powerful core language, with most features going into libraries that application developers can rewrite and replace when necessary, removes that constraint - if your application is valuable enough to deserve the extra investment of effort then you can keep improving it and get a better result in the long term.
(But NTPsec sounds (based on very limited knowledge) like a fairly small program without demanding performance requirements that sits in Go's primary niche of networked servers, and its developers are coming from C not C++, so it probably wouldn't get the benefits of Rust's philosophy and may do better with Go.)
Posted Feb 27, 2017 18:41 UTC (Mon)
by davidstrauss (guest, #85867)
[Link]
The memory model of encapsulated data is becoming more formalized in C++17 with contiguous iterators. If code consuming a data structure expects to use such iterators (as they might choose to with std::string), you would not be able convert the underlying structure to use a rope.
This is not usually a problem for existing code, as prior standards for C++ had no concept of a data structure promising contiguous memory, but std::string and similar STL data structures have guaranteed that model in a way that hasn't been compiler-enforced (but that programmers could assume). The danger here is updating old code that relies on the properties of something like std::vector with something that doesn't guarantee contiguous iterator use. I could see this happening because the pre-C++17 code had no way to formally require such properties from a data structure (and so one might assume code not requiring it in the C++17 way doesn't need it).
Posted Mar 2, 2017 16:16 UTC (Thu)
by kjp (guest, #39639)
[Link] (6 responses)
Posted Mar 3, 2017 1:16 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link] (1 responses)
Posted Mar 4, 2017 11:45 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link]
Posted Mar 3, 2017 13:07 UTC (Fri)
by mathstuf (subscriber, #69389)
[Link]
Posted Mar 3, 2017 14:46 UTC (Fri)
by karkhaz (subscriber, #99844)
[Link] (2 responses)
The two are theoretically equivalent in terms of their expressivity, but shared memory is known to trip programmers up a lot more---deadlocks, races, all the bad concurrency bugs that don't typically occur when you're using channel-based concurrency.
Go's big win is bringing this sort of concurrency to a (hopefully soon-to-be) mainstream language. The language that came closest to this, I suppose, is Erlang, but that's still fairly niche. Channel-based concurrency is apparently the canonical style in Rust also (AFAIK), although in that language it's implemented through the library rather than as language primitives.
Posted Mar 3, 2017 16:34 UTC (Fri)
by kjp (guest, #39639)
[Link] (1 responses)
Posted Mar 3, 2017 17:36 UTC (Fri)
by karkhaz (subscriber, #99844)
[Link]
Obviously you can do anything in any language and you can write a kernel in javascript and a web application in OpenCL if you really want to. But having domain-specific linguistic constructs shape our perception of how to think about solving the problem at hand. Nothing is _stopping_ you from passing messages around in Java, but there isn't anything that makes it particularly compelling either. The Thread/Mutex/etc classes in the Standard Library are mostly modelled after pthreads, and there isn't any Standard Library interthread channel implementation AFAIK---never mind language support.
It would be great if tomorrow's programmers end up thinking about concurrency in a message-passing style, rather than shared-memory style, by default. Having languages that express these ideas as primitives will go a long way toward ensuring that this happens. Pthread-style concurrency should be something that you only learn if you have to use it, rather than the default way of doing concurrency. Go is a pioneer in this regard (well, Erlang is even more of a pioneer, but it never caught on).
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
There's also RustBelt that attempts to prove correctness of the unsafe bits if they are encapsulated correctly.
Toward a more approachable Rust
Toward a more approachable Rust
The problem here is suspicious design - your entities are both elements of a list and objects in their own right.
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Raymond wonders if the Rust project might benefit from a single "Benevolent Dictator" (or a "core team" of some kind) making decisions.
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
This is a common soundbite, but I've yet to see any proof of that.
Toward a more approachable Rust
From ESR's blog: NTPsec dodges 8 of 11 CVEs because we’d pre-hardened the code.
Toward a more approachable Rust
You can find many things on ESR's blog.
Toward a more approachable Rust
Toward a more approachable Rust
Of course. And to quote NTPsec's Removal Plans page:
Toward a more approachable Rust
If something on this list is important to you, tell us. If the complexity cost of keeping it is low, you win. If the complexity cost is high, then we will need a donation of engineering time or money to support keeping it in the codebase.
If your essential feature is on that list & you can't convince NTPsec to agree, then keep using NTP Classic. Or Chrony or OpenNTPD or whatever does still support your essential feature. Personally I really like having multiple implementations to choose from.
Toward a more approachable Rust
Toward a more approachable Rust
Can we please stop using the terminology ESR coined to help himself hijack another project? Thanks in advance.
Toward a more approachable Rust
Toward a more approachable Rust
While my opinion of ESR (both of his professional qualities and his social graces) is indeed rather low, but I don't think I have insulted anyone here.
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
2) The other page requests that were served before yours and their contents, size, timing, and whatnot. Given the scale of the Cloudfare CDN, this is the sticking point..
Toward a more approachable Rust
var xhr1 = new XMLHttpRequest();
xhr1.open("GET", "https://very-secure-site-using-cloudflare.example/");
xhr1.withCredentials = true;
xhr1.send();
var xhr2 = new XMLHttpRequest();
xhr2.open("GET", "page-that-triggers-the-parser-bug.html");
xhr2. { /* send the content back to the attacker */ }
xhr2.send();
</script>
Translation distance
Translation distance
Translation distance
Translation distance
Translation distance
Engineering
Jon
Engineering
Engineering
there is, but human are all but 'clear and ambiguous'.
There are lots of applications that can and do afford resource usages (CPU, memory consumption) beyond the necessary minimum by these or much larger factors. A problem we have in computing is that that does not necessarily help safety.
Engineering
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust
Toward a more approachable Rust