Comment by everforward
1 day ago
From the infra side, it’s something I don’t think makes a lot of sense to self-host. Metrics are the other notable one here.
They have non-linear scaling complexity, so they work fine with bumping up resources until they don’t and now you’re looking at queueing metrics on Kafka or something equally complicated. They’re horrible to change out because a lot of the APIs aren’t compatible and literally everything uses them.
They’re also awful to maintain. Everything uses them so there’s never a window when it’s convenient to shut down the feature flag system for upgrades or to test backup restores. People are much more accepting of “the vendor will be down for 2 hours on Saturday” than “I’m taking the system down for 2 hours on Saturday”.
I'm not sure which provider you are paying for metrics, that you think has acceptable pricing.
We are on ~400k series, and switching to any hosted provider would instantly make that our single biggest infrastructure expense.
Instead we're self-hosting with less than $100/month for hosting and ~less than a hour per month of maintenance with Victoriametrics (have also switched fron singe-node to cluster over time), and it's a breeze.
The things I most disliked about our in house feature toggle system were:
1) the line between toggle and reloadable config is always fuzzy, in part because you don’t know for sure if it’s a toggle until users give their feedback. About half a percent of toggles stick around in perpetuity as either preferences or emergency relief valves for the operations team. That seems small but when you are doing CD aggressively that’s happening to you once a month or every few weeks which makes it a constant companion.
2) you need a system to track and complain about old toggles. If they survive the end of the Epic that birthed them, they become immortal. And you can see variations on Knight Capital play out when they get toggled years after they were supposed to be static.
Regarding number two, that's what I designed featureflip to handle. It tracks dead flags and will submit a pull request to your repository to remove it for you. That's always been a big pain point of mine with launch darkly.