Comment by WalterBright
3 months ago
I caveat my remarks with although I've have studed the Rust specification, I have not written a line of Rust code.
I was quite intrigued with the borrow checker, and set about learning about it. While D cannot be retrofitted with a borrow checker, it can be enhanced with it. A borrow checker has nothing tying it to the Rust syntax, so it should work.
So I implemented a borrow checker for D, and it is enabled by adding the `@live` annotation for a function, which turns on the borrow checker for that function. There are no syntax or semantic changes to the language, other than laying on a borrow checker.
Yes, it does data flow analysis, has semantic scopes, yup. It issues errors in the right places, although the error messages are rather basic.
In my personal coding style, I have gravitated towards following the borrow checker rules. I like it. But it doesn't work for everything.
It reminds me of OOP. OOP was sold as the answer to every programming problem. Many OOP languages appeared. But, eventually, things died down and OOP became just another tool in the toolbox. D and C++ support OOP, too.
I predict that over time the borrow checker will become just another tool in the toolbox, and it'll be used for algorithms and data structures where it makes sense, and other methods will be used where it doesn't.
I've been around to see a lot of fashions in programming, which is most likely why D is a bit of a polyglot language :-/
I can also say confidently that the #1 method to combat memory safety errors is array bounds checking. The #2 method is guaranteed initialization of variables. The #3 is stop doing pointer arithmetic (use arrays and ref's instead).
The language can nail that down for you (D does). What's left are memory allocation errors. Garbage collection fixes that.
As discussed multiple times, I see automatic resouce management (written this way on purpose), coupled with effects/linear/affine/dependent types for lowlevel coding as the way to go.
At least until we get AI driven systems good enough to generate straight binaries.
Rust is to be celebrated for bringing affine types into mainstream, but it doesn't need to be the only way, productivity and performance can be made into the same language.
The way Ada, D, Swift, Chapel, Linear Haskell, OCaml effects and modes, are being improved, already show the way forward.
There there is the whole formal verification and dependent type languages, but that goes even beyond Rust, in what most mainstream developers are willing to learn, the development experience is still quite ruff.
So in D, is it now natural to mix borrow checking and garbage collection? I think some kind of "gradual memory management" is the holy grail, but like gradual typing, there are technical problems
The issue is the boundary between the 2 styles/idioms -- e.g. between typed code and untyped code, you have either expensive runtime checks, or you have unsoundness
---
So I wonder if these styles of D are more like separate languages for different programs? Or are they integrated somehow?
Compared with GC, borrow checking affects every function signature
Compared with manual memory management, GC also affects every function signature.
IIRC the boundary between the standard library and programs was an issue -- i.e. does your stdlib use GC, and does your program use GC? There are 4 different combinations there
The problem is that GC is a global algorithm, i.e. heap integrity is a global property of a program, not a local one.
Likewise, type safety is a global property of a program
---
(good discussion of what programs are good for the borrow checking style -- stateless straight-line code seems to benefit most -- https://news.ycombinator.com/item?id=34410187)
> So in D, is it now natural to mix borrow checking and garbage collection?
I think "natural" is a bit loaded, there is native support in the frontend for doing both. You have to go out of your way to annotate functions with @live and it is still experimental(https://dlang.org/spec/ob.html). The garbage collection is natural and happens if you do nothing, but you can turn it off with proper annotations like @nogc(https://dlang.org/spec/function.html#nogc-functions) or using betterC(https://dlang.org/spec/betterc.html). There is also @safe, @system and @trusted(https://dlang.org/spec/memory-safe-d.html).
So natural is a stretch at the moment, but you can use all kinds of different techniques, what is needed is more community and library standardization around some solutions.
> is it now natural to mix borrow checking and garbage collection?
D is as memory safe as Rust is, when you use the garbage collector to allocate/free memory. If you don't use the GC in D, then there's a risk from:
Those last 3 is what the borrow checker handles.
In other words, with D, there is no point to using the borrow checker if one is using D's GC for memory management.
You can mix and match using the GC or manual memory allocation however it makes sense for your program. It is normal for D programmers to use both.
> D is as memory safe as Rust is, when you use the garbage collector to allocate/free memory.
Does D also protects against data race?
(I couldn't find an obvious answer after a bit of research, but I might have overlooked something)
> "gradual memory management" is the holy grail
I don't think gradual types are as holy grail as you make them out to be. In gradual typing, if I recall correctly, there was a large overhead when communicating between typed and untyped parts. But further
But lets say gradual memory management is perfect, you have to keep in mind the costs of having GC + borrow checking.
First thing, rather than focusing on perfecting GC or borrow checking, you divert your focus.
Second, you introduce an ecosystem split, with some libraries supporting GC and others supporting non-GC. E.g. you make games in C# and you want to be careful about avoiding GC, good luck finding fast enough non-GC libraries.
Not all languages using a GC are designed that way, where libraries are dependent on it. For example, in V (Vlang), none of their libraries need the GC. They also can freely mix memory management methods. There is no ecosystem split, but rather preferences.
I agree with you.
For me Rust was amazing for writing things like concurrency code. But it slowed me down significantly in tasks I would do in, say, C# or even C++. It feels like the perfect language for game engines, compilers, low-level libraries... but I wasn't too happy writing more complex game code in it using Bevy.
And you make a good point, it's the same for OOP, which is amazing for e.g. writing plugins but when shoehorned into things it's not good at, it also kills my joy.
> I can also say confidently that the #1 method to combat memory safety errors is array bounds checking. The #2 method is guaranteed initialization of variables. The #3 is stop doing pointer arithmetic (use arrays and ref's instead).
#4 safer union/enum, I do hope D gets tagged-union/pattern-matching sometimes in the future, I know about std.sumtype, but that's nowhere close to what Rust offer
Doing a sumtype is on the agenda and we'll get to it.
BTW, D has always had quite an advanced pattern matcher for types.
https://dlang.org/spec/expression.html#is_expression
https://dlang.org/spec/template.html#template_type_parameter...
Another language with such features, including sumtypes, is V (Vlang)[1].
[1] https://github.com/vlang/v/blob/master/doc/docs.md#sum-types
> So I implemented a borrow checker for D...
D's implementation of a borrow checker, is very intriguing, in terms of possibilities and putting it back into the context of a tool and not the "be all, end all".
> I can also say confidently that the #1 method to combat memory safety errors is array bounds checking. The #2 method is guaranteed initialization of variables. The #3 is stop doing pointer arithmetic (use arrays and ref's instead).
This speaks volumes from such an experienced and accomplished programmer.
Hey, thank you for spreading the joy of the borrow checker beyond Rust; awesome stuff, sounds very interesting, challenging, and useful!
One question that came to mind as a single-track-Rust-mind kind of person: in D generally or in your experience specifically, when you find that the borrow checker doesn't work for a data structure, what is the alternative memory management strategy that you choose usually? Is it garbage collection, or manual memory management without a borrow checker?
Cheers!
Personally, I frankly do not need the borrow checker. I have been writing manual memory management code for so long I have simply internalized how to avoid having problems with it. I've been called arrogant for saying this, but it's true.
But I still like the borrow checker style of programming because it makes the code easier to understand.
I find it convenient in the D compiler implementation to use the GC for the AST memory management, as the algorithms that manipulate it are easier if they needn't concern themselves with memory management. A borrow checker approach doesn't fit it comfortably, either.
Many of the data structures persist to the end of the program, as a compiler is a batch program. No memory management strategy is even necessary for those.
Alright, thank you for the answer. Cheers! <3
> I can also say confidently that the #1 method to combat memory safety errors is array bounds checking. The #2 method is guaranteed initialization of variables. The #3 is stop doing pointer arithmetic (use arrays and ref's instead).
I think these are generally considered table stake in a modern programming language? That's why people are/were excited by the borrow checker, as data races are the next prominent source of memory corruption, and one that is especially annoying to debug.