Comment by vb-8448

Ignoring the possibility of plain ol' management incompetence, when things are changing as fast as they are the problem becomes one of lead time. Hiring competent people does not happen at the click of a finger, nor in the current climate does getting new hardware resources.

Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”).

And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication.

Scaling isn't easy. Scaling fast is a royal PITA.

need competent people to hire competent people first - it seems like their only directive is to raise $MSFT not make good tech

(talking about leadership, not my lovely msft engineers reading this)

Looking at the design of Github Actions, I don't see any competency.

Yeah, aren't they super geniuses with infra that scales with a button, etc.?

Probably contention on databases.