Comment by RussianCow
7 months ago
You can do all of that without a proxy. Just store the current state in your database or a Redis instance.
7 months ago
You can do all of that without a proxy. Just store the current state in your database or a Redis instance.
and managed from among the application servers that are greedily trying to store/retrieve this state? Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc. The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.
> Not to mention you'll have to be in the business of defining, updating and managing the schema, ensuring that upgrades to the db don't break the application servers, etc, etc.
I have to do this already with practically all software I write, so the comexity is already baked in. Sure, if you don't already have a database or cache, maybe a proxy is simpler, but otherwise it's just extra infrastructure you need to manage.
> The proxy server is the right design decision if you are truly trying to build something production worthy and you want it to scale.
I've been doing stuff like the above (not for LLMs but similar use cases) for years "at scale" without issues. But in any case, you need to store state the moment you scale beyond a single proxy server anyway. Plus, most products never achieve a scale where this discussion matters.