Comment by cfontes
1 day ago
I pay for it for my team... it's expensive to be honest, but worth it IMHO. Building it would be a ok, supporting it a pain.
If you work on a small startup, not worth it... on a Big corp with a lot of bureaucracy? It's worth it I think.
We have tried self-hosting it, but it's always a pain to allocate people on infra to do it, so I gave up and started paying for it for my teams, and they love it.
From the infra side, it’s something I don’t think makes a lot of sense to self-host. Metrics are the other notable one here.
They have non-linear scaling complexity, so they work fine with bumping up resources until they don’t and now you’re looking at queueing metrics on Kafka or something equally complicated. They’re horrible to change out because a lot of the APIs aren’t compatible and literally everything uses them.
They’re also awful to maintain. Everything uses them so there’s never a window when it’s convenient to shut down the feature flag system for upgrades or to test backup restores. People are much more accepting of “the vendor will be down for 2 hours on Saturday” than “I’m taking the system down for 2 hours on Saturday”.
I'm not sure which provider you are paying for metrics, that you think has acceptable pricing.
We are on ~400k series, and switching to any hosted provider would instantly make that our single biggest infrastructure expense.
Instead we're self-hosting with less than $100/month for hosting and ~less than a hour per month of maintenance with Victoriametrics (have also switched fron singe-node to cluster over time), and it's a breeze.
The things I most disliked about our in house feature toggle system were:
1) the line between toggle and reloadable config is always fuzzy, in part because you don’t know for sure if it’s a toggle until users give their feedback. About half a percent of toggles stick around in perpetuity as either preferences or emergency relief valves for the operations team. That seems small but when you are doing CD aggressively that’s happening to you once a month or every few weeks which makes it a constant companion.
2) you need a system to track and complain about old toggles. If they survive the end of the Epic that birthed them, they become immortal. And you can see variations on Knight Capital play out when they get toggled years after they were supposed to be static.
Regarding number two, that's what I designed featureflip to handle. It tracks dead flags and will submit a pull request to your repository to remove it for you. That's always been a big pain point of mine with launch darkly.
Feels like something that should be a solved problem though.
Part of my last job was winnowing down a convoluted feature toggle system, rebuilding parts of it on top of a FOSS library that did a fraction of the same work. It had become convoluted enough though that it was materially affecting both our deployment time and our cluster density.
To be a little fair to the original author, despite not really respecting him, I had to file a couple serious PRs upstream to simplify the most tortured parts. He worked around a couple of bug/misfeatures but took the scenic route to do it. Guy “never met an abstraction he didn’t like” is not true because his favorite thing was replacing one abstraction with three. And he was one of those guys who learned big words all from books and so he used definitions nobody under 65 has ever used in conversation.
I thought LLMs were supposed to be the death knell for services like this, since you could replicate them and maintain your own code-base with little effort?
At least that's what the AI boosters keep saying.