Comment by aleqs
4 hours ago
How are you handling the actual provisioning of the agent boxes/VMs? Are you using a specific provider/cloud, or multi-provider? Are there any providers/clouds you've found are better/worse for your use cases?
Asking because I am working on an open/standard protocol/layer for provisioning cloud resources across different providers. One of the ideas is to provide a marketplace of providers/resources via one unified/standard API, which I would allow automatic selection and provisioning of VMs (and other resources) based on price/value/feature/reputation requirements/priorities.
Coder uses terraform providers to do the provisioning. I think that's the correct layer of abstraction, all major providers will be there and can be customized with the modifications needed via various other terraform features.
I've written three different solutions for platforms that solve this standard protocol layer. What has worked for us was deciding to standardize on the Kubernetes API for our services back in 2017, and then from then on, all our providers have provided either a k8s API to interact with resources, or provide a Cluster API for provisioning. For example we have two datacenter VM providers that implement the k8s API and we've been able to swap the providers with no user refactoring, it's a good solution in hindsight with how many people have k8s in their stack.
For the second big feature you're thinking of, we never really go a good solution to this problem, a lot of our workloads are long lived and not necessarily spot instance-able, but it's a very fun routing problem. Most of our need to be in multiple clouds is that we want to heavily separate our customer facing services, and keeping it away from internal workloads that support those customer APIs.
Is this public work? I'd love to take a look. This is my bread and butter.
Yeah using terraform providers is a great idea and something I have considered and aim to support. The thing I'm building also aims to handle things like payments and identity, not directly but with a plug/play facilitator model - so different providers/users can support/use different payment methods, crypto, decentralized identity, etc. That part kind of sits outside the resource provisioning part, and the resource provisioning part could in theory be terraform providers (after identity/payments is established). Resource discovery and search including - specs, pricing, region, capabilities, etc. I envision this protocol being used to build both centralized and decentralized compute/cloud resource marketplaces. In the decentralized view - regular users sharing/renting their hardware/resources with each other... lots of security/abuse risks/questions there.
I will be publicly releasing everything ~soon (I hope), it's in a pretty early stage at this point, many open questions.
For boxes.dev we use E2B - they had everything we needed at the time. I'd love a standardized API / marketplace, but more just to try out new providers with differentiated features or pricing - we probably wouldn't be switching often / using multiple (unless customers asked us for more options). There are so many sandbox providers popping up!
Our use case is pretty specific though - one persistent machine that you set up your full developer environment on, then ability to very quickly spin off full copies of that machine (filesystem + memory) as separate VMs to run agents on. Firecracker VMs + support for "forking" the box were our key requirements. Another provider that could do this was Modal, but they use gVisor and not Firecracker, which means it's harder for users to run docker inside the box.