← Back to context

Comment by thethirdone

10 months ago

The sample answers for the horse race question are crazy. [0] Pretty much all the LLM really want to split 6 horses into two groups of three.

Only LLAMA 3 makes the justification that only 2 horses can be raced at a time, but then gets its modified question wrong by racing three horses. I personally would consider an answer that presumes some restriction to how the horses can be raced to be valid if it answers the restricted version correctly.

[0]: https://arxiv.org/html/2405.19616v2#S9.SS2.SSS1