Comment by atlex2
5 months ago
Yes absolutely this! We're working on these problems at FlyShirley for our pilot training tool. My go-to is: I'm facing 160 degrees and want to face north. What's the quickest way to turn and by how much?
For small models and when attention is "taken up", these sorts of questions really send a model for a loop. Agreed - especially noticeable with small reasoning models.
I just tried this with a smaller "thinking" model (deepseek distill, running locally) and boy are you right. It keeps flipping between which direction it should turn, second guessing its thought process and then getting sidetracked with a different approach.