Comment by ComputerGuru
20 hours ago
Speaking from experience here, flash lite models have amazing price, speed, and perform far above their size, but are susceptible to very bad instruction following and recall when either complexity or context size inch up. They’ll just forget to apply your instructions to portions of the input, and repeat parts of the input that should be returned verbatim as direct quotes but with subtle changes (breaking urls, for example).
Yes you have to continuously tune the prompts ever so subtly. 3.5 is a lot better than than 3.1 tho.
Important to remember that json schema instructions take precedence over the normal prompt, so move as much into property descriptions as possible.
This was 3.5 flash lite, actually, and after prompt tuning. It was very clearly an issue that correlated with input (JSON array) size, the more elements in the batch, the higher the error rate.
3.0 flash (not lite) handled it like a champ though, fwiw.