Comment by lnenad

> You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:

> [UGC styled humorously as LLMisms]

All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information chunked into LLM-shaped pieces are for me.

I can digest LLM-shaped pieces of data very easily probably because I’ve been spending too much time with Claude, sure.

But the other side of this is that the entire human species (using LLMs) is similarly being trained to digest interrelated pieces of information/data in these specific shapes, akin to how philosophical assertions can be formulated as a syllogism and, thus, become more readily understood because of familiar epistemological cadence and shape.

Many people reject such copy/prose/data because they detect AI-generated-so-not-worth-human-attention, but I do wonder if this is preparing many millions of loosely (and tightly) associated humans and their organizations to quickly exchange and digest information.

This is not to say current LLMisms are the end, only that such detectable patterns in information delivery will make comprehension and communication more efficient (as well as more limited precisely because of such structure).

/philosophical musings about the epistemological implications of LLM-shaped conversation tics

> Three honest possibilities, and I’ll be straight with you about each

This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.

I reached point three and was nodding all along. I guess I am the NPC

This is glitch art for text, I love it

I killed my own ssh session twice with pkill -f, because the pattern matched the command line containing it.

This might be the most angry I've ever been at a HN comment that I upvoted

That's a caveat, and a real one.-

That's the nice thing about LLMs, you're always absolutely right.

You made me irrationally laugh reading this

On one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?

> For example, even if you make thinking tokens literally just

Generally speaking yes, but actually no (just randomness is suboptimal, adding steps just to add steps is suboptimal). There is a mechanism working there (in having a CoT) that is not quite clear.

The task is to optimize the efficiency of CoT. Understanding that it is not a plain "chain of thought" is the start of the problem, the solution is not there yet.

If we had the solution, there would exist no overthinking - CoT would be optimal (lean and essential plus best results).

Yeah I understand, it's my assumption that the actually/wait/but have a point. It doesn't reduce the fact that it increases the time for tasks substantially.

Especially on practical tasks. One shot prompts work better at Q6_K_XL for me. It loads a file, then analyses then second guesses itself then again then again then it tries to come up with a solution then second guess rinse and repeat. 122b is the perfect balance but it lacks quality for harder to solve stuff. I've ran DS Flash 0731 at Q4KXL, 3.8 Q6KXL, GLM 5.2 Q4KXL and they all over-reason. At least that's how it looks like to me when comparing with frontier models, even weaker ones.

Yeah, I ran into an overthinking loop with it a couple days ago on a task that shouldn't have been that hard. (It's kind of interesting to watch the internal conversation happening with it). Overall I'm impressed with it, but setting the /effort to medium is what you usually want (it defaults to xhigh). I do wonder if I had made it write out a plan if I would have avoided that though.

You need VRAM for the whole thing for optimal performance. Activation is chosen "randomly" for each token. PCIe becomes bottleneck, so much that just doing computation on CPU is likely faster.

But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).

I believe that at minimum, for usable performance, you need to be able to hold the 125B params + 51B ngrams in some sort of RAM.

Ideally VRAM, but the benefit of the MoE design is better performance with unified memory since most of that RAM is not read for every single token. So you could potentially have the model loaded in CPU RAM, and let unified memory systems page the relevant chunks on demand to VRAM, or run on a fully unified memory system and be able to achieve good speeds even with the limited memory bandwidth most of them have.

I use medium generally, that's about a minute at 20t/s and off for general chat (few seconds for a response). What kind of setup are you running it on?

Yeah 122B is the sweet spot for me as well. Even deepseek flash overthinks on stuff way too much. I think they fully rely on large reasoning turns to achieve better quality. The result of course means we wait a long time to get results even with high throughput as a lot of tokens are wasted.

This is needs ~80GB of fast memory at 4 bits per weight. Faster memory is better, but probably even something like 3090 + 64GB RAM should work (not fast, but maybe even 20-30 t/s? llama.cpp support pending).

I've got a 48c Epyc with 2x3090s and 512gb ddr4 3200. It's good enough for 25+ tps with deepseek so I'm hoping for similar performance with less overthinking.