Comment by alphazard

> I also know these are more fragile than the deeply self-contained nature of containers and their images

Not really. People only use Nix because it doesn't randomly break, bitrot or require arcane system setup.

Unlike containers. You really need k8s or something like it to mould Docker containers into something manageable.

The lack of adoption is because it’s not a complete operating system.

Using sel4 on a server requires complex software development to produce an operating environment in which you can actually do anything.

I’m not speaking ill of sel4; I’m a huge fan, and things like it’s take-grant capability model are extremely interesting and valuable contributions.

It’s just not a usable standalone operating system. It’s a tool kit for purpose-built appliances, or something that you could, with an enormous amount of effort, build a complete operating system on top of.

> But there's just not that much to do to connect a server application to the network, where it can access all of its resources.

If you only care to run stateless stuff that never write anything (or at least never read what they wrote) - it's comparatively easy. Still gotta deal with the thousand drivers - even on the server there are a lot of quirky stuff. But then you gotta run the database somewhere. And once you run a database you get all the problems Linus warned about. So you gotta run the database on a separate Linux box (at that point - what do you win vs. using Linux for everything?) or develop a new database tailored for SeL4 (and that's quite a bit more complex than an OS kernel). An elegant solution that only solves a narrow set of cases stands no chance over a crude solution that solves every case.

Also, with the current sexy containerized stacks it's easy to forget, but having same kind of environment on the programmer's workbench and on the sever was once Unix's main selling point. It's kinda expensive to support a separate abstraction stack for a single purpose.

You can use containers as a security measure, but I'd argue that if (when) it fails in a spectacular way (see e. g. abstract sockets for an interesting past issue) it's your fault and not a zero-day in the kernel as a sibling comment suggest. To put it a bit less harsh - containers are not just for security and containerization tools have to balance security vs usability.

The big selling points for me were what you said about simplifying deployments, but also the fact that a container uses significantly less resource overhead than a full blown virtual machine. Containers really only work if your code works in user space and doesn't need anything super low level (eg TCP network stack), but as long as you stay in user space it's amazing.

The main initial drive for me was that it let me separately run many things without a) trying to manage separate dependency sets, and b) Sharing RAM - Without having to physically allocate large amounts of memory to virtual machines; on an 8GB machine at a couple per VM that doesn’t let you get far.

"making it easier to deploy" is a rather... clinical description for fixing the "but it works on my machine!" issue. We could go into detail on how it solved that, but imo it comes down to that.

my view of docker, as a who thought it was a shallow wrapper on linux namespaces, is that it was a good fit for the average IT shop to solve the deployment friction

no more handmade scripts(or worse fully manual operations) stupid simple dockerfile scripts.. any employee would be able to understand and groups can organize around it

docker-compose tying services into their own subnet was really a cool thing though

LXD?

Because our computers have global state all over the place, and people like it, as it simplifies a lot of things.

You could see that history repeat itself in Python - "pip install something" is way easier to do that messing with virtualenvs, and even works pretty well as long as number of package is small, so it was a recommendation for a long time. Over time, as number of Python apps on same PC grew, and as the libraries gained incompatible versions, people realized it's a much better idea to keep all things isolated in its own virtualenv, and now there are tools (like "uv" and "pipx") which make it trivial to do.

But there are no default "virtualenvs" for regular OS. Containers get closest. nix tries hard, but it is facing uphill battle - it goes very much "against the grain" of *nix systems, so every build script of every used app needs to be updated to work with it. Docker is just so much easier to use.

Golang has no dynamic code loading, so a lot of times it can be used without containers. But there is still global state (/etc/pki, /etc/timezone, mime.types , /usr/share/, random Linux tools the app might call on, etc...) so some people still package it in docker.

No. Back before dynamic objects, for instance, it was easier-of course, there were other challenges at the time.

I honestly think that Dockerfile was the biggest driver. Containers as a technology are useful, for the many reasons outlined in this thread. But what Dockerfiles achieved was to make the technology accessible to much wider and much less technically deep audience. The syntax is easy to follow, the vocabulary available for the DSL is limited, and the results are immediately usable.

Oh, and the layer caching made iterative development with _very_ rapid cycles possible. That lowered the bar for entry and raised the floor for everyone to get going easier.

But back to Dockerfiles. The configuration language used made it possible for anyone[tm] to build a container image, to ship a container image and to run the container. Fire-and-forget style. (Operating the things in practice and at any scale was left as an exercise for the reader.)

And because Anyone[tm] could do it, pretty much anyone did. For good and ill alike.

I agree: I think the container image is what matters. As it turns out, getting more (or less) isolation given that image format is not a very hard problem.

> I think the important innovation of Docker was the image. It let people deploy consistent version of their software or download outside software.

What did it let people do that they couldn't already do with static linking?

ps I still miss the alternate universe where Kenton won the open source deployment battle :-)

I understand what you're saying, and I'm a fan of SEL4. But isolation isn't one of the primary points of k8s.

Containerization is after all, as you mentioned, a plugin. As is network behavior. These are things that k8s doesn't have a strong opinion on beyond compliance with the required interface. You can switch container plugin and barely notice the difference. The job of k8s is to have control loops that manage fleets of resources.

That's why containers are called "containers". They're for shipping services around like containers on boats. Isolation, especially security isolation, isn't (or at least wasn't originally) the main idea.

You manage a fleet of machines and a fleet of apps. k8s is what orchestrates that. SEL4 is a microkernel -- it runs on a single machine. From the point of view of k8s, a single machine is disposable. From the point of view of SEL4, the machine is its whole world.

So while I see your point that SEL4 could be used on k8s nodes, it performs a very different function than k8s.

The scheduler is the least interesting thing about k8s. The extensible API common to all operating environments is the real value add.

As others mentioned containers aren’t about security either, I think you’re rather missing the whole purpose of the cloud native ecosystem here.

> Kubernetes is seen as this complicated and impressive piece of software, but it's only impressive given the complexity of the APIs it is built on.

There are other reasons it's impressive. Its API and core design is incredibly well-designed and general, something many other projects could and should learn from.

But the fact that it's impressive because of the complexity of the APIs it's built on is certainly a big part of its value. It means you can use a common declarative definition to define and deploy entire distributed systems, across large clusters, handling everything from ingress via load balancers to scaling and dynamic provisioning at the node level. It's essentially a high-level abstraction for entire data centers.

seL4 overlaps with that in a pretty minimal way. Would it be better as underlying infrastructure than the Linux kernel? Perhaps, but "providing K8s functionality on top of SEL4" would require reimplementing much of what Linux and various systems on top of it currently provide. Hardly "trivial in comparison".

You're just replacing the functionality of CRI, which is already pluggable. Rest of the kubernetes is still needed.

You have solved the isolation and some storage problems for a single node. You have not solved for scaling that to 10s, 100s, 1000s of nodes. That's where Kubernetes comes in. You made a lot of good points, but "you no longer need k8s" is not one of them.

> good how?

Good because it is simple both in terms of understanding it and implementing it, and sufficient in a lot of cases.

> seL4 is not the final answer, but something close to it absolutely will be. Capability-based security is an irreducible concept at a mathematical level, meaning you can’t do better than it, at best you can match it, and its certainly not matched by anything else we’ve discovered in this space.

Security is not pure math though, it's systems and people and systems of people.

Yes but I don't see how you are addressing what I wrote. What are you getting at?

Luckily, no Dockerfile is ever as bad as old "configure" scripts were.

As long I never have to worry about configure snippets that deal with Sun's CC compiler from 1990's, or with gcc-3, I will be happy.

If you are talking about the

  RUN  foo && \
       bar && \
       baz

thing, I completely agree.

I've always wondered if there could be something like:

  LAYER
  RUN foo
  RUN bar
  RUN baz
  LAYER

to accomplish something similar, or maybe:

  RUN foo
  AND bar
  AND baz