Comment by ph1l337

not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.