← Back to context

Comment by krackers

7 hours ago

This is very interesting since there is another notable paper which shows LLMs can recognize and generate CFGs

https://arxiv.org/abs/2305.13673

and of course a^n b^n is also classic CFG, so it's not clear why one paper had positive results while the other hand negative.

6 comments

krackers

Reply
thesz  7 hours ago

Dyck grammar (balanced brackets) are not an a^nb^n, there are several kinds of brackets.

I cannot find probability of success in paper you linked. Is it 100%? I believe it is less than 100%, because LLMs are intrinsically probabilistic machines.

  • krackers  7 hours ago

    Figure 12 shows probabilities I think, it actually does seem to be 100% at temperature 0.1 for certain pretraining runs.

    • thesz  6 hours ago

      And this Figure 12 is not about Dyck/balanced-brackets grammar. This figure is about something not properly described in the paper.