Comment by norir
5 hours ago
This piece would have been a lot more compelling if they had actually done science on selecting a language for compiler development. From what I can tell, they had an untested hypothesis that a low level systems language is necessary for a high performance compiler https://www.roc-lang.org/faq#self-hosted-compiler and from that concluded that their only choice besides rust was zig.
I know from experience that this initial assumption is wrong. Compiler performance is dominated by algorithms. The fastes managed languages tend to be at worst within a factor of two for wall time on any given algorithm. Algorithmic differences can be unbounded in their performance gaps. Zig itself is a perfect counterexample to the theory that writing a compiler in a low level systems language will lead to a fast compiler. Roc seems to compile at around 15k lines per second. That is not fast. There were evidently compilers written in ml that did 3k likes per second in 1998 https://flint.cs.yale.edu/cs421/case-for-ml.html
The zig rewrite of roc looks like the author's second compiler. Compiler and language design is a skill like any other and from my vantage point, they appear to have overcommitted to an initial design at the expense of developing their higher level design skills. In my opinion, the best thing they could do for the future of roc is stop working on their current compiler and use it to write a self hosting compiler for a much smaller subset of roc. They should be able to do that in less than 10k lines of code. They might even find that their self hosting compiler is faster than their zig based bootstrap compiler for the self hosted subset of roc. If the self hosting compiler is inadequate. Now they at least have identified a smaller useful subset of roc and can experiment with different compiler implementations in 10k likes of code rather than 300k lines of code. Then they could actually test the theory of whether or not a low level language is necessary to meet whatever arbitrary compiler performance goals they have.
By self hosting, they would also discover what roc features actually matter and they would spend much more time actually writing roc code. The features that are needed to write a self hosted compiler are all features that are generally useful. By improving the self hosted compiler, they also improve downstream programs.
Your comment is very assertive, but also doesn't offer much in the way of science.
Being able to compile ML quickly in the 90s tells you little about being able to compile Roc or some other language today because the language design enforces hard constraints on the algorithms necessary to compile it and the hardware today is much more complex. It's not hard to write a fast Pascal compiler that targets a 1980s chip with shallow pipelines. But that's not the problem being solved here.
I don't know much about Roc but it looks like it's got some amount of overloading and the linked article alludes to sophisticated algorithms to avoid heap allocating closures. Those can enforce algorithmic complexity in the compiler that is essential and can't be eliminated.
Once you're at the limits of algorithmic optimization, all that's left is reducing constant factors. I've written code in many languages in different performance regimes over the years and it's certainly the case that higher level languages, especially managed memory ones, put a hard floor in terms of how low you can go when optimizing to improve those constant factors.
I have seen in real-world code where explicit control over memory layout improved performance by more than an order of magnitude. I have friends in the game industry where much of their career is this kind of work. Those people would love to live in the luxurious world you describe where all they need to do is find a sufficiently clever algorithm and all of their performance problems will disappear.