Comment by satvikpendem
8 hours ago
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
8 hours ago
How do they go away with local models? It's a bug of all LLMs not just cloud vs local. As mentioned in another comment, they did test local models here too and those failed as well.
As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?
Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.
> just context engineering like that's fucking sane at all.
Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.
And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.
The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc.
The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.
Changing the weights and controls does not alleviate the context limits causing LLMs, local or cloud, to forget over time. I was asking how local LLMs would amelioratile this but looks like there is no solution.
> Well the answer is that VC-backed companies
Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.
Local LLMs suffer from the same problems.
And I think you didn't understand what I wrote, so let me reiterate: Of course policies are equally ineffective at strictly governing the behavior of local models. Why would that change? The actual problem is that somebody is attempting to misuse them for that in the first place. There are only two reasons for it:
- They have no other options
OR
- They have no idea what they're doing
Local models solve the first problem, and let you apply the appropriate tool by virtue of the control you have. The alternative does not.
In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". To me that sounds more like the second problem, and nothing can help you there.
2 replies →
One of the biggest fixes I've seen is just getting rid of traditional sampling. But first, let me say something about quantization, just to get this out of the way.
Like, lets say you already did the sane thing, your model[1] is already either FP16 or Q8 (and quantized by a competent practitioner of the art, ex: unsloth or bartowski), and your KV context is already FP16 or Q8... which means you now are already ahead of the major companies.
Google, OpenAI, and Anthropic heavily compress both K and V to insane levels, which might not appear to be so bad on short prompts, but especially with thinking enabled, its sort of the equivalent of JPEGing a JPEG repeatedly. Every time the model thinks, and records its thoughts into the context, and then reads it back later to think more, it becomes further and further imprecise.
All models with heavy KV context compression go off the rails somewhere between a quarter and a half of a million tokens. Every. Single. One. Every team that releases a model that has a limit of a quarter of a million did this on purpose, and it was the smart thing to do.
Now, lets say you dip your toes into samplers; this includes stuff like temp, top k, top p, min p, etc. I won't describe what they do, there are already good ELI5 articles out there to help you with that. They are, however, the original samplers, and the only ones the big companies use. None of them use the newer samplers that massively outperform them.
You know what you get with most providers? Temp as a knob, and it only goes between like 0.0 and 1.5. What if you want higher? Nope! What if you want to tune the other knobs? Usually no, too (OpenAI seems to still offer it on their higher end API plans, but Google has eliminated all knobs, and Anthropic apparently removing everything but temp in the future). What if you want other samplers? Not allowed.
Even restricting yourself to normal samplers, what if I wanted temp of 100, and min_p of 0.9 and no other samplers? That produces sane results for creative writing tasks, yet I could never do this, even I was paying for some $200/mo plan at Anthropic/OpenAI/Google/etc.
All of these samplers also cause a sort of JPEGing a JPEG repeatedly sort of error, it is the third source of it (model quant and KV cache quant are the other two). Long context insanity is probably caused more by sampling error more than it does by model and KV quant error.
What other samplers are there? Llama.cpp impls dynatemp, mirostat, top-n-sigma, and some others.
The one that I think more people need to look at is top-n-sigma. Temp and top-n-sigma alone has produced results that, even on ridiculously complex and purposefully tricky prompts, let models that have 1M native context happily go to the very limit of it without any signs of tell-tale degradation.
Want to go have fun with your new found freedom? Get Qwen 3.6 27B in Q4_K_M from a reputable dealer, set llama.cpp to do Q4 KV (yep, after I just said don't do that), and then run it with `--samplers "temperature;top_n_sigma" --top-n-sigma 1.0 --temp 1000`, and then compare it to the normal recommended values of `--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00`.
You will find that a lot of problems suddenly go away: long context degradation vanishes (which Qwen 3.6 will still do inside it's 250k context, especially when both model and KV are below Q8), forgetting what you said or it said earlier goes away, losing the plot half way through goes away, overreliance on cliches (Qwen has its own form of Claudisms, but are more subtle and less grating) also goes away, hallucinations happen far less, and lazyness also goes away (Qwen 3.6 has no real lazyness defects, but Gemma 4 does).
I have tested those top-n sigma settings on code generation as well. It is better than stock, and better than the commercial offerings by any of the American providers, but I still don't think LLMs can replace human programmers: it still can't think nor reason.
But yeah, moving to more modern samplers has done more to unfuck LLMs IMO than anything else you can do.
[1]: Assuming it isn't a native 4 bit model of some kind; quantizing them correctly to work in a local inference engine without actually quantizing anything is a bit of a PITA. See unsloth's work on the QAT Gemma 4 releases.