Comment by cedws
2 years ago
K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway.
Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.
> K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform.
I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.
I run my own cluster too, it is managed by one terraform file which is maintained on GitHub [0]. Along with that I deploy everything on here with 1 shell script and a bunch of yaml manifests for my services. It's perfect for projects that are managed by one person (me). Everything is in git and reproducable. The only thing I am doing unconventional is that I didn't want to use github actions, so I use Kaniko to build my Docker containers inside my cluster.
0 https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne...
If you're using a K8S cluster just to deploy a few web apps then it's not really a platform that you could provide to an engineering team within a medium-large company. You could probably run your stuff on ECS.
While I love ECS you're not giving k8s enough credit. Nearly every COTS (common off the self) app has a helm chart, hardly any provide direct ECS support. If I want a simple kafka cluster or zookeeper cluster there's a supported helm chart for that, nothing is provided for ECS, you have to make that yourself.
2 replies →
> If you're using a K8S cluster just to deploy a few web apps (...)
It's really not about what I do and do not do with Kubernetes. It's on you to justify your "millions upon millions lines of code" claim because it is so outlandish and detached from reality that it says more about your work than about Kubernetes.
I repeat: I only need a few dozen lines of kustomize scripts to release whole web apps. Simple code. Easy peasy. What mess are you doing to require "millions upon millions" lines of code?
2 replies →
You're both using hyperboles that don't match the reality of the average-sized company using Kubernetes. It's neither "millions upon millions of lines of code" nor "just a few dozen lines of kustomize scripts".
This isn't my experience at all. Maybe three or four years ago?
Who exactly needs millions of lines of code?
I think they're more getting a k8s requiring a whole mess of 3rd party code to actually be useful when bringing it to prod. For EKS you end up having coredns, fluentbit, secrets store, external dns, aws ebs csi controller, aws k8s cni, etc.
And in the end it's hard to say if you've actually gained anything except now this different code manages your AWS resources like you were doing with CF or terraform.
We have all of that neatly extracted into a Terraform module. Write it once and now EKS clusters are essentially disposable.
3 replies →
kubeadm + fabric + helm got me 99% of the way there. My direct report, a junior engineer, wrote the entire helm chart from our docker-compose. It will not entirely replace our remote environment but it is nice to have something in between our SDK and remote deployed infra. Not sure what you meant by security; could you elaborate? I just needed to expose one port to the public internet.
Millions upon millions of lines of code?! What? Can you specify what you were trying to do with it?
Argo CD, Argo Rollouts, Vault, External Secrets, Cert Manager, Envoy, Velero, plus countless operators, plus a service mesh if you need it, the list goes on. If you're providing Kubernetes as a platform at any sort of scale you're going to need most of this stuff or some alternatives. This sums up to at least multiple million LOC. Then you have Kubernetes itself, containerd, etcd...
that's not much different from using the cloud PaaS offerings besides who runs that million lines and who gets the freedom/control for customization.
kinda like openshift?