Comment by senko

> Let's say I use the $200/month Claude (I don't, I use a cheaper plan, but stay with me). If I switch to a Opus 4.5 equivalent local LLM, and it makes even a single production issue that 5.5 wouldn't, that mistake will cost thousands of dollars.

You are assuming that Opus would always be superior; the cloud AI providers can be very variable. They can be very useful for months, but during certain times their quality can drop off a cliff. People online speculate that the cloud providers drop quality during this time when they are training a new model or the service is over-subscribed. You do not have that issue with a local model.

That issue of quality of service is enough many businesses may prefer a locally run model because it will be more consistent.

It's way better than Opus 4.5. See my stress test elsewhere in this thread (C -> microcode compiler with optimizations written in OCaml, generates microcode + Verilog output) - it's doing better than Opus 4.6/4.8 were when I was using them to code the original C version of this compiler early in the year. It's come up with optimizations that Claude didn't come up with back then.

> Let's say I use the $200/month Claude (I don't, I use a cheaper plan, but stay with me). If I switch to a Opus 4.5 equivalent local LLM, and it makes even a single production issue that 5.5 wouldn't, that mistake will cost thousands of dollars.

There are a few points I'd like to make here.

First, the frontier is jagged. There are errors that Qwen3.8 Flash Next make that Opus 5.5 doesn't make, but the reverese is also true: Opus 5.5 makes errors that Qwen3.8 Flash Next doesn't make. The latter will likely be fewer, but they won't be zero. Outside of really large differences in capabilities, there is no total ordering among models, only a partial one.

As an aside, do you always run Opus 5.5 at max reasoning in order to minimize errors, too? I suggest most people don't, because they have their own intuition for good enough vs. cost.

Second, Opus 5.5's strength are autonomous tasks. For interactive, HITL work, it's not a particularly great choice. This includes its preferring complex Bash commands over tool use and inscrutable Python scripts, but also often not even explaining what it does. It is not a great choice to use as a virtual pair programmer, because following along what it does and maintaining a mental model is harder than with some other models (though probably because of benchmarks that prioritize autonomous work, these and related problems have infected other models, too). You can address that to an extent with CLAUDE.md instructions, but even that has imperfect results.

Finally, if you rely on a model not to make mistakes in order to ensure correctness, your software development process is fundamentally broken. You need to assume that any model will make at least some errors and need a development process that accounts for that. The problem here is not necessarily which model creates fewer errors, but which model creates code where errors are easier to discover. Ironically, more powerful models may create more complex code, making that harder.

You can see an analogous effect with reasoning effort in FrontierBench 2.0 [1]:

"FrontierCode is one of the few coding benchmarks where writing additional (correct) code doesn’t necessarily raise a model’s score. This leads to an interesting phenomenon: models can score lower at higher reasoning effort. Because this non-monotonicity often matches user sentiment but is hard to see in other evals, we consider it a useful feature of the benchmark."

A example:

"This task comes from PostHog’s open-source Android SDK, which apps use to record analytics events. When the device is offline, the SDK writes events to disk and sends them the next time the app starts. The task is to fix a startup bug in which two modules send data at the same time: the live queue sends new events while a legacy integration, PostHogSendCachedEventsIntegration.kt, sends the cached ones. The two race, and events reach the server out of order.

"At low effort, GPT-6 Astra moves cache loading into PostHogQueue.kt, removes the sending logic from the legacy integration, and leaves the SDK’s public API untouched, which produces a passing solution.

"At max effort, the model often deletes PostHogSendCachedEventsIntegration.kt outright and spreads its responsibilities across other legacy files. This also fixes the race and passes most of the hidden behavior tests. But the deleted class is part of the SDK’s public posthog.api surface, so removing it introduces an unnecessary breaking change for every app that references it. Scope blockers, which deterministically check that the model hasn’t modified certain files, catch this change, and the agent fails the task."

[1] https://cognition.com/blog/frontier-code-2.0

> Let's say I use the $200/month Claude (I don't, I use a cheaper plan, but stay with me). If I switch to a Opus 4.5 equivalent local LLM, and it makes even a single production issue that 5.5 wouldn't, that mistake will cost thousands of dollars.

What are you talking about, you're fully dependent on an LLM (Opus or whatever) not making a mistake or else you lose thousands of dollars, in production? How is that okayed?

"It" makes a production issue? Are you not involved in this process?

Yea your math doesn't math.

Try it before you go super far into the world of this "must suck".

Most people aren't going full vibe code. If you stear these models, they are almost identical. Seriously. The major score changes in a lot of the absolute latest models are just models doing more things (often when they shouldn't). Having a slightly lower scoring model is actually better in some ways. I digress.

If your model can make a production mistake you are using it wrong. Sorry not sorry.

There are other benefits. Your data is private, it's not sold. Your privacy is kept, that's priceless. No one can degrade your workflows. Uptime is you are online whenever you are online not 3 9s. You also won't get refused asking basic security questions.

Your math doesn't math for a lot of people. It might for you, but I would encourage you to think a little more.

Not necessarily. If yor process accounts for "all models make mistakes", you are probably limiting how autonomous the model can be, and then there's not much tangible difference between models beyond that threshold.

In the competitive landscape, cost and not-getting-locked-down are also factors up for consideration.

Given the vast amounts of constant external investment money they need to keep operations, and how little moat (if any, besides influx of investor money) they have, I'd argue it's not that far fetched. Also, they are all american, so as a non-american, I also don't see it that unlikely to suddenly not being able to access these model-as-a-service services

If the planning-for-a-$2T-IPO company shuts down overnight, unexpectedly, you might want to do some actual doomsday prepping (like, canned food).