Comment by d3m0t3p
3 hours ago
When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ?
Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
The one thing which might have helped is the “external sandbox” detail: registering CI instances is noisy but it should be highly consistent so you could potentially alert on use outside of the expected environment (e.g. I have AWS STS sessions bound to a source IP or VPC, which really came in handy when Trivy was compromised).
I wonder whether it’d be possible to do something like export the list of legit K8s instances periodically so you’d be able to look for usage outside of those, but at that point something like that proxy approach would be less work and workload identity federation would be even easier.
True. Just based on the reporting we don't know what's going on, but the article makes it sound like those 180+ nodes were unexpected. Of course you would need to set your own limits to alerting and "what is unexpected" and you may still fail to catch something this way.