Comment by ryanscio
19 hours ago
https://x.com/rauchg/status/2045995362499076169
> A Vercel employee got compromised via the breach of an AI platform customer called http://Context.ai that he was using.
> Through a series of maneuvers that escalated from our colleague’s compromised Vercel Google Workspace account, the attacker got further access to Vercel environments.
> We do have a capability however to designate environment variables as “non-sensitive”. Unfortunately, the attacker got further access through their enumeration.
> We believe the attacking group to be highly sophisticated and, I strongly suspect, significantly accelerated by AI. They moved with surprising velocity and in-depth understanding of Vercel.
Still no email blast from Vercel alerting users, which is concerning.
> We believe the attacking group to be highly sophisticated and, I strongly suspect, significantly accelerated by AI. They moved with surprising velocity and in-depth understanding of Vercel.
Blame it on AI ... trust me... it would have never happened if it wasn't for AI.
> We believe the attacking group to be highly sophisticated and, I strongly suspect, significantly accelerated by AI.
Reads like the script of a hacker scene in CSI. "Quick, their mainframe is adapting faster than I can hack it. They must have a backdoor using AI gifs. Bleep bleep".
> Still no email blast from Vercel alerting users, which is concerning.
On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email without approval from legal or other comms teams.
But on the other hand... It's Sunday. Unless you're tuned-in to social media over the weekend, your main provider could be undergoing a meltdown while you are completely unaware. Many higher-up folks check company email over the weekend, but if they're traveling or relaxing, social media might be the furthest thing from their mind. It really bites that this is the only way to get critical information.
> On the one hand, I get that it's a Sunday, and the CEO can't just write a mass email without approval from legal or other comms teams
This is not how things work. In a crisis like this there is a war room with all stakeholders present. Doesn’t matter if it’s Sunday or 3am or Christmas.
And for this company specifically, Guillermo is not one to defer to comms or legal.
If he's not one to defer to Comms or legal, maybe this one is so bad that he's acting differently then he normally would
[dead]
> the CEO can't just write a mass email without approval from legal or other comms teams.
They can be brought in to do their job on a Sunday for an event of this relevance. They can always take next Friday off or something.
Has anyone actually gotten an email from Vercel confirming their secrets were accessed? Right now we're all operating under the hope (?) that since we haven't (yet?) gotten an email, we're not completely hosed.
Hope-based security should not be a thing. Did you rotate your secrets? Did you audit your platform for weird access patterns? Don’t sit waiting for that vercel email.
2 replies →
nope...I feel u, the "Hope-based security" is exactly what Vercel is forcing on its users right now by prioritizing social media over direct notification.
If the attacker is moving with "surprising velocity," every hour of delay on an email blast is another hour the attacker has to use those potentially stolen secrets against downstream infrastructure. Using Twitter/X as a primary disclosure channel for a "sophisticated" breach is amateur hour. If legal is the bottleneck for a mass email during an active compromise, then your incident response plan is fundamentally broken.
> the CEO can't just write a mass email without approval from legal or other comms teams
Wouldn't the CEO be... you know... the chief executive?
Sure, and the reason he is is because he DOES check stuff like this before sending it out.
Top leaders excel because they assemble a team around them they trust. You can't do everything yourself, you need to delegate. And having people in those positions also means you shouldn't be acting alone or those people will not stick around
10 replies →
I'm going down with the ship over on X.com the Everything App. There's a parcel of very important tech people that are running some playbook where posting to X.com is sufficient enough to be unimpeachable on communication, despite its rather beleaguered state and traffic.
Usually, companies have procedures for such events. But most do not.
Usually have procedures, but most don't? Say again
1 reply →
Surprising velocity? It appears the hackers had the oauth key for a month.
> an AI platform customer called http://Context.ai that he was using
Hmm? Who is the customer in this relationship? Is Vercel using a service provided by Context.ai which is hosted on Vercel?
Production network control plane must be completely isolated from the internet with a separate computer for each. The design I like best is admins have dedicated admin workstations that only ever connect to the admin network, corporate workstations, and you only ever connect to the internet from ephemeral VMs connected via RDP or similar protocol.