← Back to context

Comment by fidotron

5 hours ago

> I have never seen an Ops team being rewarded for avoiding incidents

That's why their salaries are so high.

5 comments

fidotron

Reply
denysvitali  5 hours ago

Depending on the tech debt, the ops team might just be in "survival mode" and not have the time to fix every single issue.

In this particular case, they seem to be doing two things: - Phasing out the old proxy (Lua based) which is replaced by FL2 (Rust based, the one that caused the previous incident) - Reacting to an actively exploited vulnerability in React by deploying WAF rules - and they're doing them in a relatively careful way (test rules) to avoid fuckups, which caused this unknown state, which triggered the issue

  • fidotron  5 hours ago

    They deliberately ignored an internal tool that started erroring out at the given deployment and rolled it out anyway without further investigation.

    That's not deserving of sympathy.