Comment by apatheticonion
7 days ago
I love the Chinese models.
I use DeepSeek exclusively and now Kimi K3 offers a great planning assistant for more advanced coding tasks.
DeepSeek v4 Flash is extremely fast and is able to handle pretty much anything I've thrown at it (I use mostly Rust, PSQL, Angular and Terraform).
I self host Bifrost as my LLM gateway, though I wish LLM vendors would do monthly/daily automatic billing (like VPS providers do) rather than prepaid + auto-top up.
It's annoying maintaining a non-refundable minimum balance across vendors, I would rather be billed for my exact usage.
OpenRouter helps, but I don't really like it as a service and not a fan of the mark up.
The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.
In long form tasks, across multiple harnesses (Claude Code, OpenCode, Kimi Code, ZCode) my cache rates are typically 96-99%.
I don’t think that’s particularly out of the ordinary. Do people have different experiences with other harnesses? Which ones?
How much time do you spend on your setup vs getting a lot of stuff shipped by paying for fabel 5? For me, not using the best model is a huge opportunity cost since my company can afford it.
3 replies →
How can you be hitting cache on what I think are novel LLM prompts …
Not them but my understanding is that the harness will send a simple 'heartbeat' message to keep the cache 'warm', (see prefix caching: https://handbook.modular.com/inference-optimization/prefix-c... ) which can then be edited/changed, which does cause the user to incur a fee, but its much less than the amount they'd pay on a no-cache hit request.
The prompt is only novel the first time it’s sent. Then as it’s sent repeatedly as part of the previous context it’s cached.
Deepseek afaik has a novel architecture that is somewhat forgiving of cache shifts. I’ve been getting +90% cache hits with Zed’s agent and it’s not doing anything special regarding caching afaik.
multi turn sessions, they are typically in the high 90% hit rate across all providers without doing much of anything
I’ve never used either of those tools. Do they work well?
Can you expand more on how you are using it?
Yeah deepseek is my workhorse of choice these days, I mostly use pro because I do a lot of concurrent jobs so speed is less of a concern. When it falls over I change the model mid-chat to GPT5.5 and chuck a couple of tokens over to OpenAI and then once it's correctly found the problem I switch back to deepseek, keeping all the 5.5 analysis in context.
I have been a heavy user of k2.5/6 but they seem to have gotten slower and worse at their jobs (and the "engine overloaded" errors have increased ... ), and k3 spends a LOT of time thinking and doesn't produce noticeably better results. I do keep chucking some tasks over to moonshot every now and again to give it a chance. I figure they're under compute pressure and have probably quantised the models to cope temporarily.
But recently I've found myself mostly using deepseek v4 pro and gpt 5.5 when I need the big guns.
GLM 5.2 has been good at select tasks but really shit at others and given how sparingly I use GPT 5.5 I'm not that compelled to use it. I've yet to be impressed by any Google or Anthropic models!!
> GLM 5.2 has been good at select tasks but really shit at others
Yeah it really likes to shit the bed after thinking forever. Even if it can do some things, the quality is not quite up there in my experience.
Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.
I do understand the reasoning for it, it still sucks as a user who is doing the right thing.
As someone who is also eyeing Bifrost, I am curious to know what made you choose Bifrost at all / why you decided to use an LLM gateway. I also was considering OpenRouter, since it provides pretty much every model, with same-day releases for new models. One thing that is attractive to me about Bifrost is that if I decide to leave OpenRouter tomorrow, I can do so without touching any other part of my stack; I'm hoping self-hosting models becomes more viable, and then I can become less dependent on third party LLM providers like OpenAI/Anthropic/OpenRouter, and I would not need to worry about a model that I depend on suddenly being deprecated.
What don't you like about the service, besides the mark up?
For me, the only utility OpenRouter gives me is billing consolidation - I don't really need the routing capabilities because I use Bifrost for that.
As a router, it's not very feature rich. For example I restricted the available models to the ones I want to use however the `/models` endpoint still lists all the models, making my LLM client list the 200+ models available on the service (even though they will throw an error if I try to use them).
With Bifrost, I can also create model aliases with custom configuration - for example I can create a model alias `deepseek-v4-flash-nothink` which disables thinking. I can create `deepseek-v4-flash-caveman` which injects the caveman skill (to save tokens) etc.
Plus I can contribute to Bifrost, which I can't do with OpenRouter.
Did you look at LiteLLM at all? It seems fine but Bifrost looks interesting too.
2 replies →
Can you configure Bifrost using entirely config files without the web interface? Can it run without a database or anything stateful? I'm using LiteLLM but it is not trivial to run.
1 reply →
openrouter’s API has removed parameters specific to certain models, impairing model functionality.
For example, the image generation parameters of GPT-image-2 are largely ineffective.
Have you looked into OpenCode Zen or OpenCode Go?
https://opencode.ai/zen
Zen is nice, but they require US hosting so they don't get new Chinese models right away. There is no Kimi K3.
Go is nice for the ten minutes you can use it until your hit your cap.
Go is great if you prefer deepseek V4 flash. Then it's very hard to use up the allowance.
there is also https://cline.bot/cline-pass
basically the same deal as OC Go, lending credence to token commodification
I didn't realize that OpenRouter had a mark up. Is it a flat mark up across the board or depending on the model?
Under $20 it's a flat $0.8 fee. Over $20 it's +5.5%
+ credit card charges (+1.5%)
[flagged]
"Don't be snarky."
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
https://news.ycombinator.com/newsguidelines.html
It's more just the value for money. It doesn't really offer me anything valuable other than billing consolidation - and the markup is excessive for that use case.
If I was using the routing features it would make more sense - but they charge significantly more for that so the markup is really just for billing consolidation.
"After all, anyone can build an OpenAI API compatible server that works with every major provider, and even build it using an LLM at that!"
This but UNIRONICALLY lmao!!!