Comment by zozbot234
4 years ago
What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
4 years ago
What's preventing k8s from being useful in these larger scale deployments? GP was not very clear about that, they just said that it's not scaling enough.
K8s doesn't do anything to help you with managing the fleet of k8s hosts. At 10^5 host machines you have a much bigger problem on your hands than the one that k8s is solving for.
If you have and are solving that hyperscale infrastructure problem then whether or not you're running kubernetes doesn't actually make a damn bit of difference to how you allocate your human/financial resources.
Or summarized, at the scale where Kubernetes is advertised to make the biggest difference, it's actually irrelevant/an implementation detail.
10^5 machines cost $100M, at which point you are a huge enterprise who can afford to pay people to customize your system.
10^5 is also on the low end of hyperscale. I'd bet that google/amazon have high single-digit millions of machines.
So yes, big companies can afford to pay people to customize their systems, and do so. The infra teams at these places are enormous, even just counting job orchestration.
The argument upthread (mine) is that the companies do so, and that kubernetes is the wrong choice for them. As a result, they don't really contribute to it.
That's exactly what I'm saying. And not "can". Have to. Kubernetes or not-Kubernetes isn't the problem that huge enterprises ever have.
Is that a proper job for k8s? Isn't it supposed to be strictly for orchestrating workloads on top of that fleet?
No but that's not the point. The scale of the underlying operational problem is so much bigger that whether you're running kubernetes or some other solution on top of it doesn't actually make any material difference (kubernetes or any other option are effectively equivalent).
Thus hyperscale companies have no reason to contribute to Kubernetes. It does not affect their bottom line. Hence all of the memes about companies chasing Kubernetes are being distracted from their real goals.
1 reply →
A lot of k8s is just not optimized for beyond a relatively small scale (10k services etc.). In particular many of the components rely on a primary node to handle all the traffic rather than sharding work across a group of nodes.
Completely true. Woe unto those who learned this the hard way and have suddenly found themselves in the fulltime position of "tuning etcd for kubernetes".