Comment by iLoveOncall
15 hours ago
Stupid question but isn't this useless for 99% of users? By that I mean that either your API provider supports Structured Outputs (OpenAI and Google) or it doesn't and you're SOL.
Sure the guide presents some alternatives but they're incomparably useless VS real enforced structured output.
I get that some people will run their own models or whatever and will be able to use some of the other techniques, but that's the remaining 1%.
Well, you still need to decide if structured output is the right choice.
As they point out - this might impact results where deep reasoning is required.
So you might be better off taking the unconstrained approach with feedback.
The only "solution" with the unconstrained approach is to ask the LLM to regenerate the JSON. This is definitely more expensive than whatever downside from requesting structured outputs from the API.
ESPECIALLY with situations where deep reasoning is required, since those are likely to correlate with longer JSON outputs and therefore more failure points.
Definitely. A lot of what is missing in many discussions is the absolutely essential need to have evals.
The only way to “know” what is the best (or better) approach is to have a significant number of test cases that you can measure performance against.
At the moment, for a lot of people, state of the art is “let’s try a different prompt and see if the answer on my one example is better”