← Back to context

Comment by gus_

16 hours ago

so how did they exfiltrate the information without noticing? what OS was the developer using? what security measures were they using?

yesterday discussion https://news.ycombinator.com/item?id=48191680

The security measure that the developer didn't use was completely refusing to use vscode.

vscode has no security model. It's not like swiss cheese where there are holes and some of the go all the way through. vscode is all hole with some cheese on the side. There is absolutely no isolation between the front-end process, the backend size (the thing that runs in the remote or the devcontainer), and any extensions or anything that might be in a repository whose authors you "trust".

  • Or you can just refuse to use random extensions. I built my own extensions if I needed them. You're a programmer, right? The whole point of extensibility is that you, or your company, can program what you need from your IDE, without having to make a whole IDE from scratch. I have since moved on to making my own IDE, mostly because I hate Electron and its >1gb memory footprint, but vscode served me so much better than anything else for years, without installing a single rando's extension.

    • Kind of.

      A vscode workspace can trivially execute code on the machine that runs the server end of vscode. (This is how building works -- there is no sandbox unless the workspace config explicitly uses some kind of sandbox.) So the workspace can usually trivially elevate permissions to take over the vscode server, including installing extensions on it without asking you.

      In principle, there is a teeny tiny bit of isolation between the local and remote sides, so the remote side cannot trivially execute code on the local machine. But I recommend reading this rather long-standing ticket:

      https://github.com/microsoft/vscode-remote-release/issues/66...

      2 replies →

    • > You're a programmer, right?

      This is my position as well, but it's rarely received well. Usually, a response like "why would I rewrite something that's already been written and available?" By writing the code, I know how it works. I know it is not infected with crap. I know it will not in the future be infected with crap from a down stream dependency. It seems to me this really took off with node to the point that it's laughable at what people will include with no thought at all. I know component libraries have existed for many other languages before, but node just stands out to me

      1 reply →

    • > I built my own extensions if I needed them. You're a programmer, right? The whole point of extensibility is that you, or your company, can program what you need from your IDE,

      Dude, get real. We don't all have the luxury of being able to engage in endless IDE extension programming side quests just to do our day jobs. And even if we did, there's the reality that whatever you produce is probably not nearly as feature complete or bug free as the extension someone spent years writing. Hence why people want to reach for off the shelf solutions.

      3 replies →

The 3800 repos weren't exfiltrated from the compromised machine.

The malware (be it a VSCode plugin, an npm package, or whatever is next) simply slurps up all of the users private keys/tokens/env-vars it can find and sends this off somewhere covertly.

It's trivial to do this in a way to avoid detection. The small payload can be encrypted (so it can't be pattern matched) and then the destination can be one of millions of already compromised websites found via a google search and made to look like a small upload (it could even be chunked and uploaded via query parameters in a HTTP GET request).

The hackers receive the bundle of compromised tokens/keys and go look at what they give access to. Most of the time it's going to be someone's boring home network and a couple of public or private github repos. But every once in a while it's a developer who works at a big organisation (e.g. Github) with access to lots of private repos.

The hackers can then use the keys to clone all of the internal/private repos for that organisation that the compromised keys have access to. Some organisations may have alerts setup for this, but by the time they fire or are actioned upon the data will probably be downloaded. There's no re-auth or 2FA required for "git clone" in most organisations.

With this data the hackers have further options:

a) attempt to extort the company to pay a ransom on the promise of deleting the data

b) look for more access/keys/etc buried somewhere in the downloaded repos and see what else they can find with those

c) publish it for shits and giggles

d) try and make changes to further propagate the malware via similar or new attack vectors

e) analyse what has been downloaded to work out future attack vectors on the product itself

Right now Github (and others recently compromised in similar ways) will be thinking about what information is in those internal repos and what damage would it cause if that information became public, or what that information could be used to find out further down the line.

"Customer data should not be in a github repo" is all well and good, but if the customer data is actually stored in a database somewhere in AWS and there's even just one read-only access token stored somewhere in one private github repo, then there's a chance that the hackers will find that and exfiltrate the customer data that way.

Preventing the breach is hard. There will always be someone in an org who downloads and installs something on their dev machine that they shouldn't, or uses their dev machine for personal browsing, or playing games, or the company dev infra relies on something that is a known attack vector (like npm).

Preventing the exfiltration is virtually impossible. If you have a machine with access to the Internet and allow people to use a browser to google things then small payloads of data can be exfiltrated trivially. (I used to work somewhere where the dev network was air-gapped. The only way to get things onto it was typing it in, floppy or QIC-150 tape - in the days before USB memory sticks.)

Detecting the breach is nigh on impossible if the keys are not used egregiously. Sure some companies can limit access to things like Github to specific IPs, but it wouldn't take much for the malware to do something to work around this. (I can see things like a wireguard/tailscale client being embedded in malware to allow the compromised machine to be used as a proxy in such cases.)

Alerting that requires manual response is nigh on useless as by the time someone has been paged about something the horse has already bolted.

Knowing what has been taken is also a huge burden. 3800 repos that people now have to think about and decide what the implications are. Having been through something like this in the past there are plenty of times people go "I know that repo, it's fine, we can ignore that one" only for it to contain something they don't realise could be important.

These kind of attacks are going to become increasingly common as they're proven to work well and the mitigations for them are HARD. It doesn't need to be targeted at all either, you just infect a bunch of different things and see what gets sent in.

If companies continue to not pay the ransom then we're going to get a lot more things published and many companies having to apologise for all manner of things that end up being leaked.

  • > It's trivial to do this in a way to avoid detection

    I'd love to see a real example/PoC.

    Anyway, we discussed this issue in the other thread. For me, unrestricted outbound requests to any url, whether it's well known domains like api.github.com or any other domain, are a red flag.

    Why does VS need to establish outbound requests to any domain, without authorization?

    There's no magic solution, and these attacks will evolve, but I still think that restricting outbound requests is a good measure to mitigate these attacks.

    > slurps up all of the users private keys/tokens/env-vars it can find and sends this off somewhere covertly.

    Isolating applications can also mitigate the impact of these attacks. For example, you can restrict VS code to only share with the host .vscode/, .git/ and other directories. Even by project. Again, it's not bulletproof, but helps.

    • > Why does VS need to establish outbound requests to any domain, without authorization?

      I don't know but it's very standard practice in most applications, because telemetry. But VS code is one of the worst: just check open snitch when running VS code, it's constantly phoning to a bunch of IPs.

    • > but I still think that restricting outbound requests is a good measure

      It is 100% necessary, but doesn't stop most attacks quick enough.

      If you're posting to github.com/acmecompany then attackers love to do things like add their own user github.com/acemcompany and just upload your data to that. Generally it doesn't last very long, but with CI/CD they can get thousands of keys in a minute and be gone seconds later.

    • There are plenty of exfiltration examples out there that could go through known, commonly-greenlit domains. Even exfil via DNS requests has been demonstrated.

      1 reply →

    • Ah yes, sandboxing/limiting a VSCode plugin is not impossible. I was thinking in more general terms (such as post install scripts within npm/python packages). Random test code in golang packages. There's an awful lot that people don't vet because keeping up with the vetting is a huge burden which seems pointless until you're the one that gets hacked.

      The trick is to infect a plugin that has a legitimate reason for accessing the internet or running certain commands, and then coming up with ways to abuse that to exfiltrate the data. Or exfiltrating via DNS queries, or some other vector that isn't so obvious as "allow TCP/UDP connections to the whole world".

      That or just repeatedly pester a user for permissions until one user (and you only need one within the organisation) relents and grants it.

      1 reply →

  • > If companies continue to not pay the ransom then we're going to get a lot more things published

    Paying the ransom means your data still gets leaked and now you're out of money and embarrassed.

    Why would they ever, ever, delete the data?

    • If paying the ransom doesn't stop your data getting leaked, nobody will pay the ransom. There is a rational basis for the ransomers to follow through with the deletion. Even the mob did provide "protection" when they coerced you into paying for it.

      2 replies →

  • > The malware (be it a VSCode plugin, an npm package, or whatever is next)

    Not the first time we've seen a developer get popped thanks to a malicious game mod either...