Comment by mattmcal
1 day ago
It's not just web chat, V4 Flash 7/31 suffers from a lot of pathological behavior in coding harnesses as well, e.g. infinite loops, hallucinations, premature termination, and invalid tool calls.
1 day ago
It's not just web chat, V4 Flash 7/31 suffers from a lot of pathological behavior in coding harnesses as well, e.g. infinite loops, hallucinations, premature termination, and invalid tool calls.
All of these flash models have this. You have to build your harness so that it deals with it. Infinite loops are solved by having an error message that says what to do differently on failure, invalid tool calls are solved by making the tool schema less strict and detect things in the runtime etc.
Hallucinations you can't fix. Gemini is a bit worse there than DeepSeek, but there's not much research on how to fix that. The only one is the CaMeL paper by Google, where you tag every prompt and result and then for every assistant response or tool call you first check where it got that data and error if you notice fabrication. This one is really annoying to implement.
With larger models the fabrication starts when the context grows or if you have too many tools, for flash models it's much earlier. We use the flash models for repetitive agentic tasks, where the prompt defines clearly what to do and how. The whole run is about 4-5 steps typically, and context size stays in the comfort zone.
Can you share what tools and processes you're using to do this?
I've been using Pi to build custom extensions and wrapping workflows in shell processes to make it more deterministic and enforce certain validations, all guided by Fable. This isn't production work, though, just playing llm factorio at home.
What you want is a bunch of sessions to replay. Something anonymized if it's not yours, and something that's not depending on state.
You replay all your sessions against your harness, and then store all logs all output, everything to a safe place.
Finally use a blind judge to check everything, and score the output.
Then fix your harness, iterate again until better until you are in a point where it's just the model's weakness. If you get to that, use a bigger model.
This never happens on the deepseek api. It’s always a different provider using lower quants.
FWIW, I haven’t experienced any of that using V4 Flash via DeepSeek in omp. What’s your coding harness and inference provider?
Same. My side projects are coded almost exclusively with the Deepseek V4 Flash 07/31 in omp, and it recovers beautifully in every case. I'm using OpenCode Zen.
OpenCode but I variously use DeepSeek API/OpenRouter/Vercel AI gateway. I'm sure it's the combo of model + inference provider that is the issue and not the model alone. DeepSeek API also has far better inference speed and reliability than the cheapest providers. That said I never seem to have these issues when using GLM 5.3 flash served by OpenRouter/Vercel.
disagree; been using flash as my exclusive model (other contributors have used other models) to build a complicated software project, a web engine. See https://github.com/gterzian/formal-web, which as you can see comes with very specific guidance explaining how to implement features.
I'm using headless Pi with my own UI and sandbox client, https://github.com/gterzian/uni03C0, as well as a bunch of Pi extensions for things like accessing Web standards and browser use via CDP for testing.
Switching to 4.1 today...
Edit: it seems they pushed the date at which they route the Pro calls to new Flash, so today I ended up paying regular Pro rates thinking I was using the new Flash; an example of how their offering is not quite as predictable as I would like it to be (the other is cache performance being unpredictable).
I have used the flash model for over 3b tokens and ofc. I saw some hallucinations and premature termination (I also get this on Astra - way more often than with deepseek v4 flash), but I never had a infinite loop (using the copilot as harness).
That may be an issue with the harness you are using, i've never, and never heard of, someone having this problem specifically with this model.
I used a lot V4 flash to implement plans built by other models, and it was honestly top notch. The thing was a workhorse, and I got none of the isuses you describe.
I was mostly using DeepSeek on Pi, connecting to their API directly (not some third party provider).
I honestly have more issues steering Sonnet properly.