Comment by aeneas_ory
17 hours ago
All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://github.com/ory/lumen
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Why should I trust that what you're peddling isn't snakeoil?
I literally say you should take benchmarks with a grain of salt :)
> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).
Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.
[flagged]
Only way to find out is to do some testing yourself i think.
I’m using less tokens with Lumen but I also use a bunch of other tokens hacks/skills; it’s hard to measure the impact exactly but it feels significant
I just went through a lot of benchmarking and the only thing that seemed better than rg was chunkhound, which sounds similar to this project. Actually a small Jina embedding model actually did better than voyage AI, but took a long time to index. Also chunkhound doesn’t work well with worktrees. In the end, I decided to stick with rg.
Half of what we see or consume is hype, so we should view everything with nuance. Sometimes truth lies in the middle.
[dead]
I've gotten good usage out of a structured search tool (not mine, by someone else here) called Tilth:
https://github.com/jahala/tilth
It combines search with tree sitter grammars so the results can annotate usage vs definitions, cite line number ranges, inline the actual definition if it's short, etc. Not as precise as LSP but simple (no daemon), human readable, and in many cases works without configuration.
This sounds quite similar to dirac which made a stir a few months ago:
https://github.com/dirac-run/dirac
I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.
If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.
I did the same with another project that does the same thing, called ck, and wasn't able to wring out any improved performance over just plain grep.
Yes rg + python seems to beat far fancier tools.
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days.
But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity.
The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.
Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.
1 reply →
[flagged]
A bit different but in the same domain GitNexus has been really good for me. Symbol based search and impact analysis. Once I added git hooks to auto index and added some explicit instructions it saved quite a bit for me. I have no affiliation just wanted to add that as some thing to try out.
You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)
When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.
Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253
That’s a good example, I believe it could be similar case like with RAG. I didn’t hear for a while about any recommendations to use RAG.
skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision
How does lumen compare to semble?
https://github.com/MinishLab/semble
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.
You would think, except in their own testing they found no benefit. Some measures it was even worse on.
https://blog.jetbrains.com/ai/2026/05/what-happens-when-you-...
I also found LSP like skills to usually have no advantage over rg. The agent needs to read the code to understand it; navigation is a small portion of that.
> One of: Claude Code, Cursor, Codex, or OpenCode
What makes it incompatible with Pi, Zed or any other harness?
Only the amount of free time I have to work on it - nothing fundamentally prevents it. PRs welcomed!
TBF, what difference does it make when the very harnesses you're using these "hacks" in are themselves vibe-coded?
Lumen is pretty cool, it's just local RAG, but it's a realistic approach at RAG. I would still keep tool-calling in some places though.
How does this compare to ripgrep and fzf?
> Whether it's caveman
the kinda guy who honestly thinks caveman.md reduces costs, actually adds it to his system prompt and is painstakingly reading the terse output
So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?
See my other comment on the person that commented the same thing you did!
[flagged]