Comment by pron
17 hours ago
> I don't understand the hype about Zig?
As a long-time low-level programmer, and as someone working on a popular mainstream language, I find Zig fascinating, and I also think it addresses a long-standing problem in low-level programming. I'll get to the problem later, but the fascinating part is its use of partial evaluation (comptime) as a single coherent mechanism that replaces a myriad of other partial-evaluation mechanisms (macros, templates/generics, constexprs). That one mechanism is the core of the language, like macros are in lisps, and that design - whether you like it or not - is revolutionary. It's never been done before (other languages have partial evaluation mechanisms that are almost as general, but they're offered in addition to, not as a replacement of, other features).
> rust mostly-solved the memory management problem at compile time and without a GC
"Mostly" does a lot of work here because 1., if you look at the implementation of very efficient, possibly specialised data structures - the very thing you reach for a low-level language for - they typically require unsafe, and 2., it still suffers from the problem C++ has had for decades, which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower (huge runtimes like TCMalloc help, but not enough, because they can't move pointers). This problem, of programs that start out fast, but after five or ten years of evolution need to spend a lot of effort to remain fast, is one of the things moving collectors were designed to solve, but they require moving pointers, which doesn't work in low-level languages that are not meant to have an FFI layer between them and the hardware.
To compete with the performance of moving GCs, which allocate through bumping a pointer, like on the stack, and free memory in bulk, low-level languages need to rely on arenas (which work based on a similar principle), and Zig is the first language that makes arenas almost user-friendly and hopefully sufficiently composable to withstand program evolution. Of course, time will tell how well this works in practice.
> partial evaluation (comptime) as a single coherent mechanism that replaces a myriad of other partial-evaluation mechanisms (macros, templates/generics, constexprs).
So sad dlang never gets the credit it deserves. None of these ideas in zig are novel.
Yes. I don't really understand why Zig gets more attention than dlang. It seems that it is more to do with the personality of the authors, rather than the technicalities. I get the impression that Walter is more laissez-faire, while Andrew is more single-minded. I'm grateful to the contributions both have made.
> the very thing you reach for a low-level language for - they typically require unsafe
There's a formal proof asserting that if you keep up the safety invariants within an unsafe region then that will not infect other code, even in the presence of arbitrary other correctly-written unsafe blocks.
This means you can build abstractions on top of these low-level primitives to keep it contained, so consumer code never has to even think about or know there's unsafe blocks in it. The type system lets you build very powerful abstractions so these go a long way.
There's a lot of woo-woo scare quoting around how much you actually have to use unsafe code in Rust. It's fairly uncommon to actually have to reach for them in practice. Most of my usage ends up being things like converting a &[u8] to a &str when I know it's already valid UTF-8 so I want to skip the linear-time validity check. Very rarely do I have to build data structures with complicated pointer juggling, because there's often a library that already does what I need!
> which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower
What are you talking about? I've never encountered this and I've been using Rust for 10 years.
I works in pretty low level OS code. I promise you most of our code would be unsafe. And using unsafe in rust is less ergonomic then using zig or c++.
We could use rust. But it wouldn’t give us anything.
This hasn’t been the finding of the R4L project. Go look at their code, it’s shockingly safe outside of the parts that interact with extern “C” symbols, which naturally need to be unsafe.
> > which is that over time, as program changes and evolves over years, things tend to drift toward the more general mechanisms that rely on malloc/free on an individual objects, and the program gets slower and slower
I think the idea is that a small program can organize its allocations and data structures to minimize number of calls to malloc, e.g. with preallocated workspace structs, or slab allocation, and similar approaches. But as a program gets bigger, there's a pressure to have looser coupling, to have subsystems with simple convenient APIs which leads to them doing on-demand malloc calls internally, rather than having consumers pre-allocate their needed workspace. Because that kind of workspace management results in more complex APIs and more burden on the consumer.
That said, I don't really believe it either, at least for the kind of codebase where it would matter (scientific computing, in-memory DB server, etc). A codebase that places an emphasis on minimizing heap operations in hot codepaths can do so by consistently using workspaces and allocation-avoiding APIs. I don't think it's so difficult really, but it does take a conscious design decision to do so. But writing something like a web browser in this way could be annoying due to most data having wildly variable sizes, and zig's arena concept would be very handy -- but rust has crates like bumpalo for that purpose.
My personal mantra: "Think in FORTRAN, code in Rust/Julia/C++". But I'm mostly working on HPC-style code where I don't have to do with wildly varying input or output sizes.
> but rust has crates like bumpalo for that purpose.
Except that's not composable - not only do you need specialised data structures, but all (transitively) allocating calls need to be specialised. That's the exact same issue we have in C++, and that's the issue Zig seeks to address. BTW, just the other day there was a post here about a language with another interesting approach, but I have yet to give it a close look: https://github.com/aardappel/goose/
> But I'm mostly working on HPC-style code where I don't have to do with wildly varying input or output sizes.
There you have it. The problems arise more quickly in concurrent rather than parallel code, and when there are lots of features added over the years that touch the hot paths.
> in-memory DB server
Actually, here there can be big problems (as it's also about concurrency rather than parallelism). Last week a colleague of mine looked at Moka and saw that it could only offer half the throughput as Java's Caffeine at the same latency and RAM footprint (almost; the Java program used 5% more RAM). When he looked into it, he saw that over 40% of the program's CPU was spent on the epoch-based reclamation.
5 replies →
That's always been true in all the safe languages with unsafe escape hatches, except here these "primitives" are the main reason to reach for a low-level language in the first place - because they presumably require the control that low-level languages offer. Combining them in the same language might appeal to some and not to others who think that the high-level, safe parts are unnecessarily complicated because it needs to integrate with the low-level parts, and the low-level parts are unnecessarily complicated because they need to integrate with the safe parts. Anyway, some like this and some don't, but my point is that it's not "mostly solved".
> What are you talking about? I've never encountered this and I've been using Rust for 10 years.
Okay, but I've been doing low-level programming professionally for 25 years, and have encountered this over and over in large programs (over 500KLOC) as they evolve.
That's just not an accurate description of how you write Rust in practice. There's no separate "high level" and "low level" parts/forms of the language any more than the software development process already is all about building abstractions. You should be doing this in Zig, too.
It's just that sometimes some of the abstractions you need to build go outside what the ownership and borrowing system can model. And when you don't need to do that (which is 99% of the time) you also get all the benefits of the ownership/borrow system for free.
3 replies →
> I've been doing low-level programming professionally for 25 years
You haven't been doing any Rust though. You seem to think you can extrapolate your C++ experience to Rust. That's preposterous. The actual Rust programmers can't recognize this theoretical problem in their Rust programs.
1 reply →
> There's a formal proof asserting that if you keep up the safety invariants within an unsafe region then that will not infect other code, even in the presence of arbitrary other correctly-written unsafe blocks.
In general "unsafe" does not compose.
"if you keep up the safety invariants within an unsafe region"
This condition is doing a lot of heavy lifting.
Here's an article about the research on it which lays out the properties in simple terms: https://smallcultfollowing.com/babysteps/blog/2016/10/02/obs...
I'm curious why you think that statement is doing heavy lifting. It's much easier to write and verify that a few lines of code are correct than it is to write and verify that an entire program is correct. But that's the norm in C and Zig, and historically people haven't been very good at it. That's why we try to do it as little as possible.
7 replies →