Comment by criley2

6 hours ago

In my experience it never works well on any real work. In fact, I'd go the opposite, plan with the dumb model and execute with the smart model because at least the model writing the code and solving the emergent problems is capable.

In my experience (and I've been trying this a bunch): smart planner + dumb executor produces worse code with higher spend than simply using the smart planner to do both.

It's easy to understand why:

- If the planner has truly thought the issue through, properly designed the solution, solved all of the emergent problems, then the final "write" of the code is just a few more output tokens.

- If the planner has NOT truly planned the issue completely, then you're letting a substantially dumber and less capable model make significant decisions, and trusting its problem solving, without having a better model check it.

If you're highly cost conscious (paying for your own tokens and not making any money) then you have no choice but to trade your time and effort for tricks like this to save money by lowering the quality of your output.

But if your employer is paying for tokens: just use the smarter model. You save your time preventing re-work and reducing code review, you save your employer money (primarily from the cost of your own labor and reduced rework), and you get a better output every time (Opus 5.5 mogs Deepseek 4.1 flash in every single way except cost).

9 comments

criley2

Reply
gregwebs  3 hours ago

The cost is 20-40x less for Deepseek Flash v4.1. If you are just comparing to Sonnet or you aren't paying (your case) then your advice makes perfect sense.

I also agree that its a big mistake to have a flash model implement without a strong model reviewing.

I have Opus plan, Deepseek implement the code, and then review with Opus [1]. In this workflow I am saving a lot of money by having Deepseek do the implementation. Note that the review back-and-forth is fully automated [2], so it doesn't take any extra attention from me.

  [1] https://github.com/gregwebs/skills-sdlc/tree/main/skills/implement

  [2] https://github.com/gregwebs/skills-sdlc/blob/main/skills/code-review-with-followup/SKILL.md

  • TeMPOraL  3 hours ago

    > If you are just comparing to Sonnet or you aren't paying (your case) then your advice makes perfect sense.

    Also if you aren't hitting capacity.

    Off-work, I use LLMs regularly for both design/coding and non-technical work, but the volume is not enough to trip the weekly limits, and rarely enough to trip the daily limits. So I just go with whatever's current best SOTA available on my Claude & ChatGPT subscriptions and don't worry about limits. If I hit one, I do some household stuff or relax for a few hours (or just turn in for the day), and then the limit is refreshed.

  • criley2  3 hours ago

    Deepseek Flash v4.1 is only "40X cheaper" if you do not account for the time of the engineer reading the output. If Opus 5.5 high requires 1/2 of the actual engineer time, and the engineer costs $100-$200/hr, then Deepseek v4.1 is actually the more expensive model to use.

    I have tried your workflow many times, and simply letting Opus do the implementation costs much less than wasting hundreds of millions of tokens letting deepseek and opus go back and forth and back and forth. And bonus, my project finishes in 5 minutes instead of 20.

    • gregwebs  2 hours ago

      I tried the experiment of reviewing vs. not reviewing with frontier models. I consistently found that reviewing by a model with independent context finds important issues when changes are non-trivial- certainly the definition of non-trivial is getting raise as the models get better.

      I do have an /implement-simple workflow to skip the planning phase, but even that doesn't skip the review.

      Are you doing your own intensive reviews of the model code? Can you share the prompts you are using as I have?

      My bar for what models produce without human intervention is much lower defect than what a human would produce. The human interaction is mostly to guide the design and then the review burden is very low. I suspect your bar for what agents produce is lower- you are taking more of the review burden. I also suspect that you are measuring time more than actual cost since your employer is paying and that you are comparing to Sonnet rather than DeepSeek (DeepSeek 4.1 again is 20-40x cheaper than Sonnet). You mention hundreds of millions of tokens (my reviews don't use that much), but even that costs ~$1 on the DeepSeek side.

      I think you are taking exactly the right approach at your employer given the cost is free and you only have access to Anthropic models.

      2 replies →

    • Roark66  3 hours ago

      Bingo, DeepSeek (v4.1) is horribly overhyped. In all my personal benchmarks it sits below Glm5.3 Flash. Waaaay below Qwen3.8-Flash-Next a model less than half it's size.

      No, the only open weight model that really makes sense for me is Qwen3.8-Flash-Next, but it is mainly because I can run it locally with reasonable speed (prefill between 650-1400t/s generation between 22-50t/s depending on number of slots/users I configure).

      This is the first model that truly competes with Opus 4.8. I'd say it may be better than Opus 4.6 on programming.

      But it is very verbose when it comes to reasoning tokens. The more difficult the task the more verbose it is. Certain very hard tasks that take opus 4.8 400k tokens take Qwen3.8-Flash-Next 2M tokens... But it finishes them.

      And what you loose on the generation speed you get back on input caching you can keep on for weeks.

      It really depends on the workload.