Comment by kamranjon
5 days ago
No benchmarks, no info on which models are used, ai generated video, just a signup page with nothing else.
Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."
The evaluator is public here: https://echo.tracerml.ai/eval/
It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.
Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.
New video is also being made.
> Echo does not disclose its per-request routing decision because that policy is the product
My honest advice: that's going to pull away a decent amount of potential customers, even though I think your idea/concept is fantastic.
For example, if we were to consider it for Canva, observerability and full transparency is critical requirement; we can't accept not knowing which model serves a request. Both for legal/contract reasons, co-ordinated capacity planning with API providers, or even just evaluating our prompts and harnesses; and debugging/tracing results that went wrong. So that renders it out of consideration; and also suggests some kind of adversarial relationship where customers aren't trusted with critical information.
I definitely understand you need to keep business value, but I don't think hiding which model a request is routed to, is the right one, or at least if you want to expand to bigger potential customers / more advanced LLM deployments.
This sounds like you want to contact sales and get a deal that includes some more NDA info? I would assume this is possible for a large enough customer.
> Echo does not disclose its per-request routing decision because that policy is the product
I have not had need for a router product thus far so excuse my ignorance if this is standard, but how could I possibly use and improve a product built on a router like this if I am not permitted to see which model served my request? If I got a bad answer back in my LLM-powered app, do I really have no way of knowing which model was responsible?
Isn't the relevant benchmark RouterBench? https://arxiv.org/html/2403.12031v2#S7
> No benchmarks, no info on which models are used, […]
The benchmarks are here → https://echo.tracerml.ai/eval/
They are not good benchmarks but at least they exist.
I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks.
In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.
I wouldn't be surprised if ≈7% of GPQA Diamond questions simply have the wrong answer in the ground truth data, so that getting such a question correct is graded as an error. Most machine-learning benchmarks are rather badly validated.
1 reply →
It's basically trying to replicate OpenRouter, which works pretty well and has a lot of nice features to abstract away any single provider, such as failover, metering, autoswitching, etc. It's actually a really smart infrastructure abstraction.
I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"
Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.
It feels a bit more like Fugu to me, which acts as a multi-LLM orchestrator (though I think Fugu combines open- and closed-weight models), but without being able to see the “secret sauce” behind how any of them decides the number of "plies" each model in the swarm gets, they all feel rather difficult to compare beyond the big public benchmarks...
https://github.com/SakanaAI/fugu
Isn't Fugu the same kind of thing as NotDiamond[1] (which I believe OpenRouter uses) except not as good?
[1] https://www.notdiamond.ai/
1 reply →
OpenRouter's model router for coding isn't really sophisticated. This is very different.
Unrelated - but reminds me of my favorite quote by tenderlove:
"microservices turn function calls into distributed computing problems"
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug
It's easy enough to copy and paste in a prompt, no?
Eval tests while giving general indicators might not be similar for each use case.
I thought login-protected apps are not allowed on Show HN.
[dead]