Comment by simonw
3 hours ago
> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ?
Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
True. Just based on the reporting we don't know what's going on, but the article makes it sound like those 180+ nodes were unexpected. Of course you would need to set your own limits to alerting and "what is unexpected" and you may still fail to catch something this way.
Its always CI. Whenever I've worked with good pen testers they always start with Jenkins, know that people stored creds and there and didn't treat as seriously as they treated production.