Comment by impossiblefork

Yes, although then people came up with reasonable ways of parallelizing RNNs entirely or in part anyway. There's also the need for lookups in transformers. Queries still need to be multiplied with old keys, all of them, so RNNs don't have to be totally incompatible with parallelism. It's just that you can't have a matrix inside them like in the linear RNN h_t = Ah_t + Bx_t that you have to multiply together step after step. If A is an input-dependent scalar or something you can multiply together easily you're fine.

mLSTM layers are completely parallel (the update rule for the cell state is C_t = f_t C_{t-1} + i_t x_t where f_t and i_t are gates that can be computed from x_t alone, x_t is the input at time t and this means that you can compute F_s = \prod_{s<t} f_s as fast as a cumsum and an exponential and then compute C_t = \sum_{s<t} F_s i_t x_t, again as a cumsum, so it's as good as if though it were parallel). I think the sLSTM layers that are the other component of the xLSTM have something else like this and presumably there's some trick also to training the Kimi "delta attention" RNN.

I'm not sure whether this is hardware or optimization dependent to some degree, but I get the impression that good custom kernels are an important part of this kind of thing.

Yes, but how would it have been if the EU governments just piled on, subsidized whoever had reasonable qualifications who wanted to try, straight-up funded accelerator development so that the chips don't just become investment in US AI companies, straight up built the supercomputers for training, straight up offered access to groups that had previously had reasonable success in training some model.

I don't think it would even be that expensive. I think with EU chips we could do it for 20 billion EUR + 5 billion EUR per year. After all, an H200 doesn't cost 40k to manufacture, probably one-tenth, and if we are making the chips we can get the money circulating just as the Americans have.

Add in some state effort to create suitable training data-- maybe encourage academics to create some things that are AI-friendly, and let all the people trying have that. There would hardly be anyone in the game other than us if we went for this package.

We already have VSORA. Maybe Euclyd is ready soon too. If enough are ordered we have 85% of the inference need gone, so that whatever accelerators are freed can be repurposed for supercomputing and training? Maybe we could have some organized effort to make many of the VSORA and Euclyd chips so that the chips we need for training are freed up while we wait for our own training chips.

Edit: So in the end I think the thing we need to do is to order enough VSORA Jotunn8 cards and enough Euclyd cards that we flood the inference market so much that the price of H200eds etc. drops and we can buy them second hand and assemble huge supercomputers out of them. This may require some diplomatic effort from the commission, something like 20B EUR almost immediately and some coordination, but it's easy, it's doable and it requires only being a little bit decisive. Since these cards are faster less memory will be tied up in serving the models, so it'd have a positive effect on memory prices too, which in turn would have a positive effect on European supercomputer centers, who are currently of course constrained by high memory prices.