Is that the same as storing state in a DB and on a filesystem though?
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
I am pretty sure OpenAI and Claude actually have a higher hardware cost per request that Microsoft. Github is actually most just a standard REST API app, although with some caching and git specific operations in the background. It is not as hardware intensive as GPU-based inference per quest with huge contexts and low latency.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
Fundamentally, there's incoming requests, an application server, and a database. The devil is in the nature of those requests, and how do you process those requests and stick them in the database. And then serve them back out again. Different application, different database, but fundamentally, it's the same problem. Google managed to do this very well. Microsoft has not.
They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
Scaling ChatGPT and scaling GitHub are very different problems.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
OpenAI, Anthropic, OpenRouter, Gemeni, etc also have another escape hatch: they can arbitrarily and potentially invisibly reduce your quality of service at their discretion. They can choose to route your request to an expensive model or a cheap one. By contrast, the best Github can do is slow your request down.
They're getting increasing demand for tokens at an accelerating pace.
Is that the same as storing state in a DB and on a filesystem though?
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
> Is that the same as storing state in a DB and on a filesystem though?
Google “RLHF”.
1 reply →
I am pretty sure OpenAI and Claude actually have a higher hardware cost per request that Microsoft. Github is actually most just a standard REST API app, although with some caching and git specific operations in the background. It is not as hardware intensive as GPU-based inference per quest with huge contexts and low latency.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
Yes, but come on...
That's due to the need for ultra-fast memory and inference compute, it's not the same thing.
Fundamentally, there's incoming requests, an application server, and a database. The devil is in the nature of those requests, and how do you process those requests and stick them in the database. And then serve them back out again. Different application, different database, but fundamentally, it's the same problem. Google managed to do this very well. Microsoft has not.
They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
Scaling ChatGPT and scaling GitHub are very different problems.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
Every response from GitHub must be deterministic. OpenAI and Anthropic do not have this problem.
OpenAI, Anthropic, OpenRouter, Gemeni, etc also have another escape hatch: they can arbitrarily and potentially invisibly reduce your quality of service at their discretion. They can choose to route your request to an expensive model or a cheap one. By contrast, the best Github can do is slow your request down.