Comment by rfw300
1 year ago
There's a methodological flaw here—you asked the model to generate an example, and then simulate the results on that example. For all you know, that specific (very simple) example and its output are available on the internet somewhere. Try it with five strings of other random words and see how it does.
This is key. LLMs will cheat and just look up training data whenever it can. Needs to be actively countered if one is to make it "reason".
I did exactly that, and it was very close to the actual sklearn output.