Comment by rstuart4133
15 hours ago
> I think generators should have been stabilised first - since all the problems you need to solve to make generators work well (self referential types, return values you can name, etc) are things futures need too.
Generators are an interesting case. For example, if you implemented a Vec iterator as a generator, it becomes:
fn vec_iter(&self) {
for index in 0..self.len() {
yield &self[i];
}
}
Which is arguably easier to understand than the current event driven formulation, which required you to declare a new type to hold your state, and the code looks like:
fn next(&self) {
if (self.index >= self.vec.len()) {
None
} else {
self.index += 1;
&self.vec[self.index - 1]
}
}
Effectively the stack frame has become your type, and sequential code is always so much more compact and clearer than the event driven model. The generator could be implemented as a green thread, but you would never entertain the overhead of creating the new stack needed by the green thread implementation.
However ... the async implemented all the mechanics needed to get rid of that green thread stack allocation when the size of the stack is known, as it is in this case. The state saving stuff they created for async could be used to translate that stack to a type. It would, surprise, surprise, contain just `index` - analogous the iterator type we have to manually create for event drive code. So compiler could translate the green thread to the same implementation as the event driven code, but you get to use the compact (and very familiar) syntax of a stack machine.
I found it interesting to see what happens for a more complex generator - like something that returns every node in a tree. You can do it recursively, which is simple clear code, but you don't know the size of the stack so the trick used for the vec iterator (translating it to a type) can't be used. Or you can manually store the state you stored in the stack with a recursive implementation in a Vec<> instead. Both require a memory allocation, but they are different. One is just normal malloc that must be reallocated and moved as the allocation grows. The other can use the OS's stack implementation, that doesn't move as it grows. If you re-used stacks, the OS's stack implementation would be faster in a long running program.
Notice that the transformation from a generator to async implementation is arguably more complex than the same transformation for green threads, especially for the tree traversal.
That observation is one of the reasons I'm such a strong proponent of green threads. The other is a simpler mental model. Unlike async, you don't have to expose the inner mechanisms it depends on, like futures.
> However ... the async implemented all the mechanics needed ...
As I understand it, the implementation of async in the rust compiler grew out of the implementation for generators in nightly. Its the same continuation-passing transformation that lets you implement both await and yield in your fictional example.
> Notice that the transformation from a generator to async implementation is arguably more complex than the same transformation for green threads, especially for the tree traversal.
Yeah for sure. Another nice thing about green threads is that the compiler doesn't need to invert the call stack. I suspect you'd get smaller binaries in many cases. A lot of the complexity of async in rust comes from moving stack variables into a hidden struct as part of this transformation. For example, this function:
Emits something like this:
But y is a reference to x - which makes this struct impossible to actually write using the rust programming language. Hence pin and all that. This is a very common pattern, but the rust lifetime syntax makes this struct impossible to express.
> That observation is one of the reasons I'm such a strong proponent of green threads. The other is a simpler mental model. Unlike async, you don't have to expose the inner mechanisms it depends on, like futures.
Fair. But as I said earlier in this thread, I like the mechanism (futures) are exposed. I like that "async" is part of a function signature. I like that you need to be explicit about which functions yield, when, and where. I want programming languages to have more effect systems - for example, it would be great to have a nopanic effect. I just ... find it much easier to enjoy async in javascript.