Comment by teiferer

I don’t love the “forced upon them” framing; if that’s really how people are thinking about it then maybe they should pause and reflect for a moment yhat it isn’t their money being spent. Amd the default isn’t always having the latest and greatest, it’s not paying for anything at all.

Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH my instinct is that that experiment is likely to cost more than the potential cost savings.

What I will say is that my own sense from experimenting around in a non-rigorous way is that the answer depends on how you use the tool. For actual vibecoding you should always go for the SOTA model because it will need less oversight. It’s also less likely to get stuck in a vicious loop that fruitlessly wastes tokens. But for a more hands-on approach where you move in small, carefully planned increments that you review and test in human-comprehensible chunks, smaller models may be preferable. SOTA ones don’t do that much better when working that way, and the slower inference adds a detrimental amount of friction to the work cycle.

It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate.

If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.

They should try "downgrading" to people who don't need a random number generator to do their job for them.

Luna as a doer, with a smarter model planning, can be a good compromise. Using sol for everything can be expensive without much gain, as a lot of steps don't need that sort of intelligence.

I’m building an internal tool for our company, basically an agent to help with on-call and alerts via Slack. I have evals running across a few scenarios, and my favorite models so far are Sol medium and Luna xhigh.

Sol medium has been a nice balance between intelligence and response time. Luna xhigh can achieve similar scores on the evals, but it takes noticeably longer. My impression is that the higher reasoning effort helps compensate for the lower base intelligence.

Cost is definitely a big factor, but latency and intelligence matter too. If I had the budget, I’d take Sol medium over Luna xhigh.

From using both on real scenarios, Sol is noticeably better at navigating around issues, exploring alternatives, and being creative when the obvious approach doesn’t work. That matters quite a bit when you’re investigating live alerts, where the path to the root cause isn’t always straightforward.

I choose to use Luna for most tasks because it is cost efficient, even though I get a pretty generous budget from my company.

Sometimes I will use Fable or Sol for large features/projects, or research/exploration.

I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.

It's silly to discuss it. Just do the evals.

Smaller models + more effort has strong diminishing returns, especially if your goal is to save money.

Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.

Yes. If they don't like the cost then they should fire the "leader" who introduced the AI there to begin with.

Is sol better?

Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.

Going from sol to luna is a downgrade.

It is not a question, it is a fact.

> Is sol actually worth the extra cost?

Is a question only you can answer, because it has no generic answer.

Right now, for me, being able to use sol is worth the cost, but using it all the time is not.

I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.

Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.

Team will quickly shift. People hate losing access to ai.