It's not 100% clear to me, from reading TFA, what the actual vuln is.
Suppose DankStartup folds and I, being a morally-dubious sort of fellow, purchase dankstartup.net which I then use to sign into DankStartup's O365, or DankStartup's ChatGPT as a DankStartup employee.
Isn't that a failure on DankStartup's part, to not shut down their business accounts? And isn't it also a failure on e.g. Microsoft or OpenAI's parts, since they're providing service to a defunct business entity who can't pay its bills?
To describe this as a vuln in oauth doesn't really make sense to me.
1. I create DankStartup and my company uses Google workspaces and Google auth for a bunch of stuff, like payroll.
2. DankStartup goes under and we close our Google accounts/let our domain lapse.
3. Someone else buys DankStartup.com, sets up a Google workspace, and attempts Google auth to log into stuff, and it works.
The problem is that the original DankStartup has a Google account that they create in #1, and Google goes around telling other sites (via Auth) "this is user X from company Y".
Then, the impostors in step #3 create a different google account with the same domain, and Google says "yeah, these are definitely the same guys as before", even though Google is fully capable of discerning that that is not the case; these are different people with a different workspace account, different names, different payment information, and so on, but Google is saying that if you're holding the domain you are therefore the same people as far as they're concerned and is asserting that to other companies. They are (or were) refusing to provide any indication to those other companies that these are not, in fact, the same people, so those other companies aren't even capable of doing their due diligence of extra validation if they want to.
It's similar to looking at a driver's license and just matching the name rather than the actual ID number; it's possible for someone else to have the same name as you, and identity documents have unique identifiers for specifically that reason.
>Then, the impostors in step #3 create a different google account with the same domain, and Google says "yeah, these are definitely the same guys as before", even though Google is fully capable of discerning that that is not the case; these are different people with a different workspace account, different names, different payment information, and so on, but Google is saying that if you're holding the domain you are therefore the same people as far as they're concerned and is asserting that to other companies.
Yep I understand the mechanism by which this gets abused; I think we just disagree on the implications. I don't work for Google but it seems from the outside that they're treating the OIDC subject claim as referencing the domain attached to the workspace account, or something similar. I've seen implementations where the `sub` claim is more granular, so to me that indicates the field is underspecified.
Given all that, I suppose TFAuthor's proposed solution is a good way forward.
I still think classifying this as an OAuth vulnerability isn't correct.
The first is whether taking over a lapsed domain allows you to takeover an existing Google Workspace (or Cloud Identity) organization. This it what houses the corporate email accounts and OAuth client registrations. If Google allows this scenario then the linked account takeover is simply one symptom / side effect among many. TFA is not clear on whether this step actually happened... I assume not, since if it were the case we'd be talking about direct access to the Google account data rather than only linked SP accounts.
The second is when an SP doesn't properly use the `sub` claim as a unique identifier. It sounds like some products don't understand this requirement and why it "seems to change 0.04%" of the time. I do agree that a unique identifier for the org itself would be a good addition to the token.
That said, I'm still not clear how the second problem manifests if the old OAuth client creds (housed in the old Workspace org id) are invalid. Presumably attacker can login to the SP admin account using just email based password recovery, then reconfigure the OAuth integration with new secrets. In that case the SP is failing to do MFA on the email login.
>>>The problem is that the original DankStartup has a Google account that they create in #1, and Google goes around telling other sites (via Auth) "this is user X from company Y".
Google is telling other sites that it's bob@DankStartup.com - isn't that true? Isn't this on DankStartup to close down operations cleanly?
We've already settled on DNS being the auhoritative authentication key for everything. If you bought the domain then you've bought the corporate identity. It's working as designed.
Any domain takeover allows email takeover which allows you to send password reset emails for former employees.
Does not matter if it is with oauth or not
I agree, I don't think this is a problem with Google's Oauth implementation, it's a problem with the service providers who authenticate users via the mere existence of an email address ending in @company.com without checking if the email address actually belongs to an active employee.
If, when you logged into Slack via Google Oauth with the email address user@company.com, Slack checked with company.com whether user@company.com was a valid user that should be allowed to login, then this problem would be avoided entirely because the defunct company would no longer report any valid users.
This would also avoid further problems with attackers being able to login with unattended email addresses like support@company.com, as was discussed here: https://news.ycombinator.com/item?id=41818459 (There's a lot of discussion below about whether the "sub" claim is stable or not but it's a red herring because of this IMO, also the proposed fix in the article wouldn't address it either.)
First of all, the whole point of SSO is that the only and final source on who are the valid active users is the IdP, in this case Google's public OAuth instance. If the IdP says that the current request is coming from the real user1@example.com, you give them access. And Google's public OAuth instance will confirm this even if the current Google Workspace associated with example.com is a different one than a few months ago (though the "sub" field of the attestation will be different than before, which the service provider is supposed to check).
Second of all, even if it was recognized that this is a different user1@example.com, they'd still have access to all sorts of company internal resources, that may still contain sensitive data, especially for small companies which inherently trust all employees.
Yes, why the Slack or HR or interview etc. data would still exist while being inaccessible to the original owner is very strange. The article seems to take it as a foregone conclusion that those accounts should all be expected to still exist with all the business data, and I don't know if this is based on the author's experience, or it's just a way to make the vuln sound far more serious than it really is.
On the other hand, I don't think people normally expect that OAuth depends strictly on domain ownership like this. I think most would expect that it depends on some kind of secret being stored on the IdP account side, uniquely identifying the IdP account holder beyond the email addresses presented. With regular password-based authentication, with MFA, you would at least get an MFA prompt if someone had gotten access to your email address and was being sent a password reset code. But with this type of SSO, any MFA verification is done on the IdP side, so if the IdP recognizes anyone who controls example.com as the rightful owner of any claims, then there is nothing you can do as the former owner of example.com.
>why the Slack or HR or interview etc. data would still exist while being inaccessible to the original owner is very strange
Ever been in a company that's collapsed before? Nobody hangs around to shut down the WorkDay account when they've been told they're not getting paid for the last 29 days work.
Not really much different than a user buying dankstartup.net, setting up a catch-all email, observing what comes in, and performing password resets for those accounts, allowing for account takeovers.
Calling it a vuln in oauth may be a bit hyperbolic, but Google could help prevent it.
I have catchall email accounts in every domain name I own (mostly so I can do differentiated emails for every service to track/combat leakage), and you would not believe the amount of emails I get that are intended for previous domain owners (and typos too). I haven’t actually done any reset password flows, but there are a bunch of social media and SaaS accounts I could easily take over if I wanted. I used to try to track down whoever the emails were intended to go to and forward it to them and let them know to change it, but that got to be too tedious so nowadays I just ignore them.
Still, I wouldn’t call this a vulnerability on the service provider’s part, it’s just user negligence.
The contention isn't that you can impersonate DankStartup, that's obviously not a vulnerability since you are indeed the domain owner. It's that former entities with DankStartup accounts might have used OAuth to create relationships of their own. And when the startup folds, they don't magically disappear.
Basically if Sally, the CTO of DankStartup, signed up for Taskrabbit or whatnot, it's possible for you as the owner of the domain to impersonate Sally in the context of that relationship.
Obviously the root cause here is that someone misused an account to do something not related to the business. And the actual impact is probably low since high value services tend strongly not to take tiny email domains as identity roots (i.e. sally@dankstartup.com clearly doesn't have a Vanguard account to steal).
So... like most security announcements it's oversold and spun. But it's real enough as I read it.
Why do you assume sally@dankstartup.com doesn’t have a vanguard account? I’ve absolutely had similar retirement account logins that became difficult to access once I left that employer. Had to contact HR and get them to help me log into my account. If the company had folded during that timeframe I would’ve been screwed. Of course for financial institutions you can probably recover your account through some identity proving process, and generally money transfers require a second factor sms auth, but a domain takeover would probably have been sufficient to at least get someone logged in and able to see my account balance.
> Basically if Sally, the CTO of DankStartup, signed up for Taskrabbit or whatnot, it's possible for you as the owner of the domain to impersonate Sally in the context of that relationship.
I don't think this is the issue, unless someone went to some pretty extreme lengths. Configuring OAuth such that the company Google Workspaces account is recognized by Taskrabbit as a valid SSO option is not as simple as signing up to Taskrabbit with your company email instead of personal email.
That sounds like a TaskRabbit vulnerability (in your example), not a Google vuln? It is also a vulnerability in any email based sign in, which relies on email alone without a password to demonstrate account ownership. (Including password resets that rely on email).
Besides the github account with a legitimate name, but impostor ownership, I can imagine another case which is problematic. HR SaaS companies might be required by law to maintain the account for a period of time - let's say until the end of the next fiscal year. The account still existing is not a failure on either Google, the SaaS company or DankStartup. The problem, as other posters mentioned is the fact the new account has the same ID in OIDC as the previous one, which is what the author of TFA proposes.
And not only that, almost every form of auth has the same vulnerability not just log in with Google. If you own the domain, you own the email IDs as well and you can very likely reset password.
Yeah, this is the part I'm struggling with. This is absolutely not unique to google oauth, it genuinely seems like a misunderstanding of how the web manages trust.
If you own the domain, you own all the property associated with the domain, including all the old email addresses. Magic links and password resets are all going to give the new owner access.
Your best bet as a solution is to be using strict 2fa (ex - a yubikey might help here) but even that is likely just "a conversation with support" away from being circumvented.
This is why winding down a company is supposed to have specific stages and policies associated with the dissolution. You don't just abandon the offices and leave all the filing cabinets behind either, for similar reasons...
The scenario I was envisioning was a simple as a developer that worked at that startup added DankStartup email to their Github account, committed a bunch of code. Company shut down, but dev doesn't think to disassociate that domain/email. Malicious actor uses this technique to then sign into the Github account, gaining access to that developer's account, not just the DankStartup repos which were probably disabled after non-payment.
Now, obviously some fault in that scenario lay with the person who 1) used the same account, and 2) didn't remove the old email once the startup failed. But I'm just using that as a kind of example - there may be other accounts as others have said that need to be accessible years down the line, like financial records and the like, regardless of whether the company is still around.
If you wanted to try to re-take over a pre-registered domain in Azure/M365, you would need to involve Microsoft Support who would require you to prove via government documentation that you were the same entity for the now-abandoned Azure/M365 account.
Creating a new tenant using the same public domain won't get you access to the Azure/M365 tenant of a previous customer, you would have a different internal domain name (dankstartup2.onmicrosoft.com).
At best you could register your domain post-expiration of the previous tenant and impersonate them >90 days out.
It's similar to Gmail or Hotmail or Yahoo Mail hypothetically allowing a new user to adopt an abandoned email address. Most email services don't allow this for this very reason.
Third parties is google as single sign on. If such an org/domain is abandoned, a malicious person could 'resurrect' the accounts on those third party sites. It's kind of like a similar situation where a user deletes an email account and then someone else creates an new one in its place, and now they have access to all the third party password reset links. Except in the case linked above, the user never had the ability to delete their accounts before off boarding, and the abstraction isn't just an email, but the account itself.
In emails case all the larger providers I'm aware of retire addresses to prevent this sort of abuse, probably learned the hard way. Could Google OAuth do something similar here?
It now occurs to me that large businesses might have the same issue with emails and former employees (easy to imagine John Smith departing and at a later date John Smith being hired) so I wonder how they typically handle this.
My father changed careers and dissolved his business over a decade ago, at which point he let his old domain lapse.
A couple years ago, I began to wonder if anyone was still trying to reach him, so I bought the domain (domain squatters weren't even interested in it anymore) and setup an inbox for his old email address. In less than a week, the unread count was in the triple digits. It was basically all marketing and spam so I closed the inbox within the month, but multiple years with a 100% bounce rate is apparently not enough to deter LexisNexis & friends from trying to win him back.
This is a corporate planning/governance problem, not a vulnerability in Oauth, and I bet the problem is even bigger than TFA describes since any previous domains of a company that were almost-but-not-completely migrated away from could easily be taken advantage of, possibly to even greater effect since the company likely has a bunch of systems still up which have the domains whitelisted. (Which makes me wonder if there are hacking groups who use domain squatting as a front or side-gig while they take advantage of lapsed corporate domains.)
This is very much a critical flaw in systems which rely entirely on email or domain names as the sole security factor. OAuth itself is flawed in that it allows this terrible practice, though to be fair it does discourage it. Google's specific OAuth solution encourages this compromised practice even more, because the only identifier it provides for a user group is the domain name associated with the account (in the "hd" field). For individual users they at least provide a "sub" field with a unique user ID (though per the article, it seems this is possibly flaky, or at least misunderstood) - this could be used to make sure that a new user@example.com doesn't get access to the old user@example.com account. But to check if a new user should have access to the organization account, there is nothing that Google provides that can be used securely: all they tell the service provider is that this is a new valid user for the example.com org, not that this is a completely different example.com org.
What if DankStartup doesn't fold and just forgot to renew their domain, or lost control of their domain for some reason. OAuth shouldn't let you Auth someone else's account just because you can hijack their domain.
I don't think it would be 100% on the failed startup considering most of, if not all, of the applications mentioned likely do not actually delete accounts even when you say to delete them. There probably could be some allowance made for the situation but also you're repurposing the domain of a failed startup.
I don't think the problem was necessarily with Google though as I believe they create a unique ID for the domain. The problem with be with the other services like ChatGPT, Slack, etc that associate with the domain and create accounts for each user in the domain.
it's vital to have extremely skilled and nuanced people managing bug and security report triage because it seems many can overlook the complexities being given.
"The F-ing article" (derived from old internet slang "RTFM", for "Read the F-ing manual", a thing that new users were prone not to do before getting online and asking questions easily answered by RTFM.)
I really don't understand here, the proper way to use Google's OpenID implementation to authenticate someone is to use the `sub` claim. Don't use the email, don't verify it yourself, use the `sub` claim. It's a known fact and is properly documented[1].
If the `sub` changes, it's because it's not necessarily the same person so have a flow ready for that. It could be an employee left and came back, a domain change, an IT error that lead to a reprovisioning of the user, etc.
I also fail to see how the proposed solution of having a 'A unique user ID that doesn’t change over time' is different from the `sub` claim. However, the new ID associated to the domain could make sense to enforce a strong 'Everyone from the @domain.com has access' statement.
I've been working with an app that uses Google to login for the past 10 years, and I've had problems with sub changing when these situations happened :
- Domain change
- Company being bought by another one and being integrated in their Google Workspace
- Employee leaving and coming back
To us, it's very very far from the quoted 0.04% which is to me very high. I had to deal with it 5-6 times in the past 10 years but of course that number will vary depending on the usage of your app and I'm not gonna venture and put a percentage on it.
Exactly. How is there an entire alarmist article and 165 comments on this thread. This comment, and it's legitimacy/factualness, is the only thing worth discussing.
`sub` _IS_ the immutable reliable identifier. If it's not, (1) I want to see actual proof, not an anonymous rando (sorry, but this thread re-inforces how little I trust context-less comments like that), and (2) I'd want to hear a convincing argument that ... `sub2` would actually be less mutable.
Threads like this make me really question other peoples' general comprehension skills.
0.04% is a few times higher than I'd expect it to be, but if it were actually that bad I'd expect some form of previous report about it to be findable on the internet. I did some searching but couldn't find anything. It would be a significant bug on Google's part, but stranger things have happened.
What's astonishing to me is that apparently all these big service providers did notice, and then they decided to disregard the one identifier Google tells them to use? Fundamentally that's the security bug being reported here, it's just being reported to Google instead of those service providers.
A stable alternative for the hd claim would of course be a good idea. It would provide a more complete way to deal with the inherent security issue of allowing domain-based signup without further authorization steps. But given the above I'm not convinced these service providers wouldn't start ignoring it after the first time somebody re-registers a domain with Google Workspace.
I am pretty confident that the statement is false.
The `sub` claim value is equivalent to the user ID that's exposed in the Directory API, it's derived from the underlying user account's unique user ID, and it won't change unless the user account is recreated.
I've implemented OAuth flow glue logic a bunch in my career and I agree that the logic doesn't seem to follow from what the documentation says. What's not clear to me though while reading this is, if I delete/lose a domain, and someone re-makes the domain and account with the same email address, does the sub value then change? The docs would seem to suggest yes, but then I am not clear what google actually ended up "fixing" in its bug bounty. Was that maybe the bug, and this is just not clearly explained in this post?
The reason this is an issue is that the sub value changes often enough that intergrations ignore it rather than bother users with having to re-OAuth.
As far as what Google is doing in the bug bounty, that's a good question -- we don't know. The author is proposing two new values, for the domain and user, that wouldn't change in the cases that sub changes now, but would change in this case.
There's a reason this happen though. Whether you like it or not and regardless what the standards say, people want a technology that matches with the identifiers they want to use (eg email) and not whatever opaque value happens to be in the sub claim.
I don't like blaming users for wanting something sensible.
Yes this is not a "Google" problem so much as a "Your SaaS application is literally doing it wrong"
Having worked on a lot of SSO integrations I can say that everyone does it wrong.
I will also say that there is a super scary aspect to the sub claim (in general). It should be pairwise tied to the SP's integration, so if you only use google for auth, and for some reason you lost your google dev account and had to set up a new one you would drop credentials from 100% of your users.
for google I don't know if you can scope for the account ID (the opaque unique identifier) since that could expose privacy issues via cross integration tracking. But that's basically the fix. if you had the unique, immutable, opaque account identifier and matched on that this wouldn't work since buying a domain and setting up that email should result in a new account with google.
Yeah, I'm not really clear on the vuln. It sounds like clients don't validate sub claims because they are unstable? Google's docs say they should.
When I first read the post, I thought that maybe Google was reusing sub claims when they shouldn't, but on a closer read, maybe it's the opposite problem - that the sub claim is changing when it should not change?
From the article -
“The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week ...
Because the sub claim is inconsistent, it cannot be used to uniquely identify users - leaving services reliant on the email and hd claims."
So maybe clients aren't using the sub claim because validating "email matches && sub matches" breaks real users who are not actually changing accounts? AKA, Google is changing Google Account IDs unpredictably? That would be a bug on Google's part IMO.
If those account IDs in the sub claim are changing, at the very least that's a bug in the doc you linked IMO.
I'd have to read the full OIDC spec [0] to see if the sub must remain constant for the same end user, and if clients are required to validate the sub claim before processing a login. If so, and it's changing without the end user's identity changing, that is a bug in Google's implementation and client implementations for anyone using Google as an IDP.
If it's allowed to change, though, and validating sub claim is not required in the spec, maybe clients are correct in not relying on it.
I ran into this when implementing OIDC at work. One of my teammates wasn't able to login even though he just linked his google account. Turns out based on which browser session he was using, the same google login was giving a different sub. Super weird behaviour.
I mean, I see why it happens. It's because the client uses the email as the identifier for login - it is the only shared identifier between Google and Slack (or whatever service) to connect accounts. Sure, you could argue that maybe this is ok for the initial login, but the client app should record the sub...
Note that for regular logins without Oauth, you still have this vulnerability - if you gain control of me@company.com, you can login to any site that uses that for login. This is a fundamental weakness in the way we handle domain registration and email addresses.
This is more of a fundamental issue with the way we rely on DNS to secure *gestures wildly* all the things. The fact that domains can expire and be registered by someone new allows the new owner to do almost anything the old owner could have done when it comes to anything trusting email addresses, or anything else relying on DNS (ACME certs) for authentication.
It's great for "do they own this right now" validation and that's what we use it for, but beyond that links will be saved, email addresses will be added to databases and address books alike, and that's more or less a reality of most systems. For example, my snail mail (and occasionally packages) is still mistakenly delivered to my former addresses from time to time just because it's hard to track who has that address, and update it everywhere. The same goes for internet infrastructure.
I was able to take over an Instagram account because I received a password reset email to a domain catch all address that I'd set up years ago. Turns out it was a whole brand, but they refuse to change the email address and don't understand that I'm going to shut it all down. It totally makes my site look bad (it was an adultish brand and I was working on a community project) and I now have the power (and ability to do whatever with the account and any other that I happen upon with an email at that domain.
With OIDC, that's what the `sub` identifier is for. Apparently, this changes for 0.04% of accounts (according to "a staff engineer at a major tech company" quoted in the article), but it's what Google tells you to use to identify a user. When a domain expires and you re-register for OIDC with Google, the `sub` claim will differ and the application should refuse your login attempt.
If the `sub` claim does indeed change for no good reason, that's a bug on Google's side, but I can't find anything online to corroborate that claim. The sub value changing when accounts are recreated could be a pain for large companies to deal with, but other than that the solution is pretty simple.
There are no good alternatives for email addresses that are also human readable, unfortunately. Some web3.0 cryptocurrency projects do encryption based authentication that solves this problem (but makes your accounts impossible to recover if you lose your key file). Passkeys/WebAuthn also try to solve this problem by foregoing the entire username requirement, but websites still ask for email addresses just in case.
I disagree. DNS stores enough information in WHOIS to see if ownership has changed, it's not DNS' fault that nobody looks.
Probably the least-wrong thing to do with current DNS is to have authentication servers keep track of the WHOIS UpdatedDate of email domains. If a WHOIS UpdatedDate is newer than the corresponding user's linked email address verification, that user's email address is no longer trusted. Next time they log in ask them to update or re-confirm their email address, and if they try to password reset they can't use an unconfirmed email address.
Yes that's more tricky work. Authentication is hard. Nobody should be DIYing authentication anymore in this day and age, it's just too much.
You can put whatever you want in WHOIS, including just replicating the information that was there previously. What if the WHOIS email is an email on the domain in question?
Maybe registrars could set a unique ID per registrant, and if a domain expires and is purchased by a different entity/account than the previous one the registrant GUID is refreshed. That could then be a signal that all previous reliance on the DNS of the domain name should be null and void
It's crazy just how little effort it takes to get a "Google = bad" article to the top of HN.
There is no vulnerability in Google OAuth. This is exactly how every OAuth server is supposed to work. If you take over a domain, you automatically own every email address in that domain, and thus whatever external account relies on that email for login. Heck the result would be the same even if that service didn't use Google OAuth, or any OAuth at all.
> If you take over a domain, you automatically own every email address in that domain, and thus whatever external account relies on that email for login
Actually, if both sides implement OIDC+OAuth2 correctly, you don't. The subject claim (`sub`) of the attached Google account doesn't get reused when a new owner re-registers the domain with Google.
The article claims that the supposedly immutable `sub` field changes too often, though, and that would be a problem Google needs to fix. The source is an unnamed developer mentioned at https://youtu.be/yIutY_X2FcU?t=20617 who encounters issues with custom domain users, with the `sub` field changing weekly in some cases.
Sure, you can create new accounts when you take over a domain and even fake all the old accounts if you have a list somewhere, but you shouldn't be able to access all of the accounts authenticated with OIDC unless you break into the Google Workspace account.
Yes, this about sums it up. If you take over a domain, you control its registrar records for what its authoritative nameservers are, so of course you can set it to your own custom nameservers and then define whatever MX you want to receive incoming mail flow. You don't even have to go to any effort of configuring working outbound mail. You just need to put up a very minimal zonefile with the MX defined, basic postfix email server with a catchall configuration and then receive any incoming emails for *@domain.com.
Is this really just google=bad, though? I work at a startup and this seems like a legit security risk that I'm happy to learn about.
It seems like the only mitigation would be to let your HR SAAS know when your company shutters and ask them to delete the records. Or just squat the domain yourself as an ex-employee.
Yes it is, otherwise the title would be "don't use your email address to log in to any application" and it wouldn't be ragebait enough. The whole issue has nothing to do with OAuth and nothing to do with Google.
I think part of the issue is that this is where the abstraction that we call "account ownership" starts to leak.
You may correctly have access to an account through this scenario, but that does not make it your account. This becomes obviously when we consider an account at a bank, for example.
Nearly a decade ago, there was a website called thehunt.com that basically ran out of money and some employees were acqui-hired by pinterest.
All of the assets were left to rot and at the time the company was a good match for another startup of mine. So I reached out to the investors I found on crunchbase and asked if I could acquire everything. We worked out a deal and I did. The issue was the complete lack of people from the old company willing to assist and the complete lack of data for alot of things. There was 1 or 2 people who we could ping from the old company to ping who were super helpful, but the big thing was many things were just lost to time- passwords, history, code repos, etc..
Simply creating a new google apps account allowed us to get full access back to everything - We could even read old slack messages (even DMs!) by resetting each accounts password. The whole thing was shocking to say the least, but with that access we got back into literally every service they used and managed to get it up and running again within a week, which was a good thing because nearly every service it was using was threatening to shut it down every day for lack of payment.
I think the solution here is actually way simpler than most make it out to be and could easily be a startup for someone:
Create a startup that lets customers simply enter in domains. If the domain EVER goes into the "pendingDelete" status, inform the customer. The customer would be random SAAS's that want to protect against this type of attack and could simply choose to disallow access to any account that has had their domain go into that status.
Huh - it's almost like the mere existence of a pendingDelete in the history of a domain should make every email existing on that domain at that time eligible for https://haveibeenpwned.com/ .
> To resolve this issue, Google could implement two immutable identifiers within > its OpenID Connect (OIDC) claims:
> 1. A unique user ID that doesn’t change over time.
> 2. A unique workspace ID tied to the domain.
1. is the OIDC `sub` claim! I strongly suspect that in those 0.04% of accounts where the anonymous quoted engineer reports that the `sub` claim changed, what actually happened was some provisioning/onboarding/offboarding system resulted in the account being deleted and recreated.
2. is sensible, and is just a versioned version of the `hd` claim.
If services are not respecting the `sub` claim in this case, then they are giving the new Google account access to the old account's data. Companies probably wouldn't complain about this because they think it is the expected/reasonable behavior. Also it's likely that in many scenarios it is the same human behind the different accounts, e.g. if they leave a company then return.
Reinforcing the face-palm at the heart of this, which is that anyone deciding to you know, just use email instead of asking why an immutable ID changed... just probably enabled information leakage. Seriously, I'm so thankful that my colleagues would be principaled about this and ask questions instead of just doing something to make it "work". Where "work" means some GSuite user probably logged into some other defunct GSuite user's RP-account.
In both, the details around the `sub` field, the field that should actually be used to identify the user, is poorly described. I would say that both of these feel a bit sensationalist, but then again, if relying parties are NOT using the sub field to validate users, they should be called out.
It seems to be that a good way to make some money is find every such situation where RPs are not using sub, and submit vuln bugs.
> According to a staff engineer at a major tech company:
> “The sub claim changes in about 0.04% of logins from Log in with Google.
I've had a staff SWE also claim to me that they generated colliding UUIDv4s, and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
Neither's attention to quality, or rather, lack thereof, could cash those outlandish claims.
There is a simple explanation here: Your staff SWE is wrong. `sub` is the claim you're looking for.
Extraordinary claims require at least a trifling of proof. Or, what can be asserted without evidence can also be dismissed without evidence.
And this alleges to be a security blog. As a writer with that target audience, you should know this is going to be the objection to the thesis that your reader will have, and you attempt to just handwave it away…?
> I've had a staff SWE also claim to me that they generated colliding UUIDv4s, and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
The claim about circles I don't know. It depends on what exist means, and I don't know what it means to have a circle in real life, and likely don't care. I can only draw a rough approximation of a circle, and that's been fine for me.
Generating a colliding UUIDv4 seems pretty simple though; if you have a broken enough random generator setup and manage to run it without seeding, especially if it was in the times where it was pretty easy to run a virtual machine with totally broken random (virtio-random was developed for a reason), and spawned a bunch of virtual machines in very similar conditions. You can no true scotsman your way out of this by declaring that a broken system, but from inspection, I don't know how you can determine if a given UUIDv4 was generated with proper or broken random techniques. See also Debian Security Advisory 1571-1 [1], and similar issues where random values that were intended to be secure turn out to be predictable. It's a plausible claim. But that doesn't mean a claim by a 'staff engineer' is default plausible. It's just an appeal to authority of a title that doesn't mean a lot.
> I've had a staff SWE also claim to me that they generated colliding UUIDv4s,
UUIDv4 is random. Could be a bad PRNG, or just very very very bad luck.
> and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
It is insanely difficult to project circles. Most of time we just pretend it never exist.
You're underestimating the odds. The odds aren't "bad luck", the odds are "statistical impossibility". Literally any other explanation holds with orders of magnitude higher certainty.
> It is insanely difficult to project circles. Most of time we just pretend it never exist.
Open Google maps, find the equator. That's a circle, rendered on a projection. (But far from the only example possible.)
(And lest I get more objections, a perfectly spherical earth was also an assumption of this assertion that circles don't exist. My coworker attempted to draw/show a circle onto an actual sphere we fortuitously had handy, and no dice.)
(Note the other way exists too: a projection can have a circle rendered on it, though it wouldn't be a circle on an actual sphere.)
> There is a simple explanation here: Your staff SWE is wrong. `sub` is the claim you're looking for.
Or perhaps it's actually happening 0.4% of the time. Maybe a few companies have some weird process that results in them routinely deleting accounts and creating new ones on the same email address.
Is this really an issue? Back when I implemented Google OAuth I used the "sub" that Google gave me as well as the email. I'd wager that Google does not re-use a sub for the same E-Mail so if the account is deleted and then somebody grabs the domain and re-created the account they'd have the same E-Mail but a different sub which would make the login fail. Seems to not be an issue in Google's OAuth Flow but rather the implementation on services like ChatGPT and so on?!
EDIT: Just read that they claim that “The sub claim changes in about 0.04% of logins from Log in with Google” - never had this happen in the past years but if this is true then I guess thats a bad thing.
Are your users primarily using gmail accounts or are they using accounts from custom domains? TFA does't say exactly, but I wonder if this stat only applies to users with custom domains rather than @gmail accounts.
The "millions of accounts" is highly speculative, is it not? It relies on the idea that failed startups keep their SAAS accounts enabled, rather than offboarding correctly.
You would think that because you're a reasonable person. However at my previous startup, which no longer exists, the new CEO failed to save the company and parachuted out so fast that the company who later acquired the startup couldn't get the domain name transferred to them because the CEO and CTO were both gone. They had to bring back the predecessor CEO because he was still able to log into the registrar to transfer the domain.
In the case of a fire, some less than conscientious people will fend for themselves first and don't give a second thought to anyone else.
I agree there will some number of companies doing this, between 1 and $number_of_companies. I'm just saying it's pretty unlikely it's all candidate companies, which the article seems to be saying.
Shutdown of a small company running out of funds is rarely fully orderly. Critical things may get taken care of, but everything is unlikely, as people have already left. Plus, with things like already-paid-in accounts, there's even an incentive to keep them open until the term expires, e.g., why not keep the prepaid ChatGPT API account live for execs and techs to use between jobs?
Just like wiping hard drives on EOL computers — everyone should do it every time, but people are always finding full HDDs and SSDs on eBay and Craigslist...
Not just this, but execs and employees like to keep websites live for a while while they job-hunt (otherwise their resume has dead links). And ideally keep their name@startup.com email working. But to keep those running requires a long tail of "live" DNS and finance stuff you can't quite close out.
also, there are lots of non-technical people running small businesses that use G services because that's all they know. They can barely operate them let alone be cognizant of things like this "exploit".
> They also classified the issue as a “Fraud and abuse” issue, rather than an Oauth/login issue.
I can _kinda_ see that, I agree they should mitigate it as best as possible too though, especially since they're Google after all.
> I thought this would be the end of the story, but 3 months later, they re-opened my ticket (after my Shmoocon talk was accepted), paid a $1337 bounty, and said they were working on a fix.
Sad how the only way to get Google's attention to make enough noise about something...
This actually happened to me a few years ago when I closed my consulting company. My bad, I know. My domain was taken over by someone in Vietnam. Back then, he was actually able to see my old emails via the Google Apps. Grandfathered free Apps accounts was a big part of the value proposition. He demanded $500 to delete my emails and docs. I offered $50 for his trouble since I recognized that I was at least partially at fault but he refused. To his bad luck, one of my main clients was Google itself. I reported him via my internal connections and the next day got the following email: "Oh my G-d, all forty of the domains I bought got Google Apps cancelled".
> “The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week”.
What I don't understand is why the `sub` claim is not consistent for those users at Google. To my understanding of the OIDC protocol the `sub` should be unique for a specific user.
Additionally as far as I understand if you take over a defunct domain and create a new google workspace with new users those new user account should get assigned a new `sub`.
I agree, in my limited experience the sub remains consistent even when changing the Google Workspace domain. So the email changes but sub remains the same. The issue seems to be clients matching on email/hd claim instead.
I wonder what action is causing the sub to change like the author suggests is happening.
At my current company, if an employee leave and come back, they'll keep the same OID in Entra but they'll get a new `sub` in Google workspace. We had to put in place a process to be able to use an internal tool that used the login with Google.
That's most likely dependant on how the IT department handled the deprovisioning/provisioning of users in our Google Workspace, I unfortunately don't have the details for that.
> I wonder what action is causing the sub to change like the author suggests is happening.
Indeed this would be very interesting.
This issue is also very similar to CVE-2024-25618.
What we did to mitigate this is the following:
- Federated login with OIDC
- Look for a user based on the sub claim
- If they are found: authenticate that user and optionally update their profile (email, name, ...) based on then new id claims.
- Else look for a user matching on the `email` claim and link the `sub` to that user
- If no user is found create a new one
That sub identifier changing without the underlying user changing may be the core of the bug. It's not fully clear from the post IMO. I posted a little breakdown here: https://news.ycombinator.com/item?id=42701030
I don't think the article clearly states one way or another whether those 0.04% of sub claims changing are legitimate end user changes or not. If those sub changes are legitimately the Google Account changing, I don't think that's a bug on Google's part, but rather is a problem on the clients' side for not validating the sub claim.
If the sub is changing without the Google Account user actually changing, maybe something internally at Google is swapping out those IDs when it shouldn't be. It doesn't look like a UUID from the example I saw, so maybe there's some code somewhere to just change the user's ID if there's a collision or something?
Interestingly, I don't think Google claims that Sign in with Google is fully OIDC-compliant. At least in the overview I linked in my comment, they compare the implementation to OAuth+OIDC, but I'm not sure they claim to be 100% spec compliant.
How would the clients tell if the account has a valid sub change or not if the only piece of information provided is that the sub claim changes? For this particular attack, without having some kind of Google Workspace account identifier for the domain, the sub claim doesn't sound sufficient to validate that it's the same Google account from the client's side. I'm guessing the engineer at the major tech company didn't provide that stat without checking if those users were valid, active accounts.
In iSCSI, there are a few options for addressing the block storage devices, with one being the iSCSI Qualified Name (IQN) that uses the domain name. But it includes a timestamp:
- A date code, in yyyy-mm format. This date MUST be a date
during which the naming authority owned the domain name used in
this format, and SHOULD be the first month in which the domain
name was owned by this naming authority at 00:01 GMT of the
first day of the month. This date code uses the Gregorian
calendar. All four digits in the year must be present. Both
digits of the month must be present, with January == "01" and
December == "12". The dash must be included.
Seems like we need a chain of custody for domains, not just current ownership. Kinda like real estate. And then accounts are tied not just to the ownership state, but also the ownership instance ID.
Just don't tell the web 3.0 folks cause I don't necessarily mean a blockchain.
Web 3 already tried this, it was called Handshake. I think their downfall is they wanted to replace ICANN / DNS rather than work with and alongside the current system.
You don't need a whole chain of custody, you just need to know that the current owner is the one you care about. TXT DNS records or magic URLs can do the trick.
This isn't anything new, or exclusive to Google, or exclusive to OAuth.
My workmate hadn't used Facebook for many years and it was associated to an email on a custom domain. Said domain expired and somehow hackers/script kiddies/bots must have an email list, run a whois, buy the domain if expired, setup email, and do password resets across social media and common websites. His Facebook was stolen.
I'm not sure I agree that it's a flaw in Google's OAuth implementation. Domain ownership is the source of truth for many systems and, as the article alludes to, expired domains can be abused in a variety of ways.
This is a stupid article.
1. It starts with “Millions of Americans”. How does this relate in any way to Americans?
2. Gmail is not the problem
3. Mitigation: Consider registering a domain for 10 years. When the startup fails, delete all dns entries
I fail to see how this is Google's problem and as much as it pains me to say this, I agree with Google.
I believe it is more the downstream providers fault for not terminating inactive accounts, and the failed startup's for not properly deleting their Google accounts.
Google can mitigate this as the author mentioned, but this is not their responsibility. They shouldn't be held liable because failed startups were stupid to not delete their data.
Related maybe, I got a new phone number and I was able to use it to sign into the previous owner's account at a site that uses 2FA. I noticed it when I tried to setup a new account for myself. The website wouldn't allow it but it happily allowed to me access the previous owner's account. It was inconsistent, removing the phone from their account required their password but changing many other important fields didn't.
It seems like there have been a number of security stories lately that essentially amount to. "no tech companies ever does cleanup" Which everyone on HN with a couple years or more experience should know is the norm. I expect a lot more of this to keep happening since companies will never really care about security without consequences ie a $10 fine for making $100 literally incentivies lack of security.
For the people not understanding the security risk, or how this would be something Google needs to solve, have a look at the claims of Microsoft Id token claims, and how their use of multiple guid claims completely eliminates this problem:
Assuming most of the problem is that federated services only rely on email address from OidC login, I wonder if Google could start offering user+unique-workspace-id@company.domain as the email address for any new workspace/user/federated login? Obviously, Google needs to control "unique-workspace-id".
Does not solve it for existing customers and any federated service they logged into, though if included alongside the simple email, federated services could gradually move to it and announce once they are "ready" so workspace admins can configure them to start accepting only new, safe email addresses.
Okay, this made me curious if there was a technical solution to this that people could be providing using the existing tools available, and I think there is.
RDAP and WHOIS will return the creation date of a domain. These fields are controlled by the registrar not the registrant. That creation/registration date gets reset when the domain is lapsed and picked up by somebody new.
So, when doing any domain-name-based authentication (like email password resets) authenticators should look up the registration date of the domain name. If it's newer than the last time the user logged in using the domain name auth? That might be somebody who snatched the domain name.
Sadly, this isn't fool-proof. Domains can go up for auction or backorder on a registrar, and they won't update the registration date if the domain is purchased this way since the registrar can consider this a transfer. It's a signal, for sure, but it will miss cases. It will also miss transfers sometimes, depending on the registrar.
I was trying to find cases of it happening historically so I could check the RDAP record to see how domain registrars use it in practice... and yeah, the registrars seem to ignore a lot of the spec. While they do generally seem to follow the "lapse and re-register = new registration date", I can see how your example is something they probably would break. RDAP records don't appear to show historical expirations and reinstantiations and re-registrations despite the spec describing events for that. It's always just the basic event entries:
Registration date, last changed date, future expiry date. Even with domains that have well known dramatic histories. Which tells me the RDAP spec is not really enforced.
While I dislike "blockchain all the things" I can definitely see the argument for a blockchain-like global shared public ledger (albeit a not for-profit proof-of-work one) with full history for this sort of data.
This feels similar in spirit to some service providers (*cough* Atlassian -- especially bad when they acquire other companies and push that bullshit on previously functional authentication systems) using ownership of a domain to assert ownership of an account using that domain.
E.g.:
1. You create an account using an email associated with a domain.
2. The domain owner coughs up their protection money to Atlassian and proves ownership of the domain.
3. Your password-protected account is assigned to the domain owner. Your password and 2FA invalid. The domain owner can access your private data without your password and 2FA.
Both TFA and the thing I described rely on broken, simplistic views of "domain ownership == account ownership".
It sounds like there’s a bug with the “sub” field. The author is stating that they can’t use it as a unique identifier for the user. Isn’t that the root of the problem?
A new domain owner could try to reuse email addresses, but since the domain is now in a new workspace the sub should be different and downstream services shouldn’t second guess that. Am I wrong about this?
I would be curious if any other Identity Providers that implement OAuth have mitigations for this issue. If you used sign in with Microsoft or Okta or PingFederate, would those provide different claims to the service?
Microsoft had a flaw for awhile where you could just change a user’s email to anything with no verification.. and if the SSO implementer was only checking the email field, you could impersonate anyone.
Ideally the trust with the SP would be using a cert. This requires the SP to implement this portion of the standard and the IT department to follow through with implementing a cert. The cert can be self-generated, so there isn't a cost for the cert itself.
If oauth makes an authenticity claim, it should be true. Saying it's the same user when it's not is bad, clearly.
in other words: Google could make a more accurate authenticity claim than they currently do.
This problem would be worse without oauth, though, right? With plain email login, all they would need to do is "forgot password" and there wouldn't even be a way to tell.
in other words: Email login would never be able to make a more accurate authenticity claim.
We take the OAuth2 token and make a call to the People API [0] to get the Person resourceName. Not sure if this is more stable than "sub" but we completely ignore that.
I don't think this has anything to do with Google's OAuth. This issue is literally with every single expired domain name out there. All one has to do is register the expired domain and look at all the emails sent to that domain.
Granted, Google "could" do something, but I don't think it's Google's responsibility to police expired domain names. What am I missing here?
Google promises to use a different `sub` claim for every account, even if you reuse the domain name. However, according to the talk, the `sub` claim isn't stable in normal scenarios, so developers don't use that like they're supposed to.
Google should fix the `sub` problem if the problem is on their side (and not, for instance, related to user accounts impersonation or recreated user accounts, which are expected to fail this check). Everyone integrating with Google should use the `sub` claim like they're supposed to.
Of course this approach doesn't help if a domain admin can recover the original workspace account (rather than simply re-registering the domain with Google), but that can easily be solved by not having the domain admin accounts use the domain they're hosted on.
I have worked with a few of these downstream providers to look for a solution. There is a documented unique user identifier (the sub claim) that could theoretically prevent this issue, but in practice, it's unreliable ... According to a staff engineer at a major tech company: “The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week”.
The real solution is to use the sub claim, and not hd/email. If sub is unreliable, let's fix that. That said, I'm surprised sub is documented as unreliable here; I wonder if there is perhaps something else going on. I've done a lot of work with oauth and I haven't found sub to be so mutable, but then again, who knows maybe I just missed it?
And yeah, lapsed domains => domain takeover => email takeover and impersonation will always be a baseline problem, independent of oauth.
The email returned by OAuth providers is not always verified and thus cannot be trusted, anyone relies on the email as the ID is open to compromise. The sub claim should always be used as the ID, if you can prove that "the sub claim changes in about 0.04% of logins" then it's Google flaw and they should fix it, otherwise it isn't and there is no need to add another ID to the claim.
Well they have kind of known about this kind of problem for 14 years https://issuetracker.google.com/issues/35889152?pli=1
although this is not a domain issue it is one where people can't unlink from company project ownership.
Does anyone here know more about the unreliability of Google's sub claim? I am using it, and have never had issues with it, but this post seems to just dismiss it without more details.
When does Google change the subject of a token for a given user account?
This is a stupid article.
1. It starts with “Millions of Americans”. How does this relate in any way to Americans?
2. Gmail is not the problem
3. Consider registering a domain for 10 years. When the startup fails, delete all dns entries
I agree. Additionally, the authors haven’t considered data deletion policies. Most SaaS providers delete data within 30 to 90 days after contract termination. It’s unlikely that data from 100,000 failed startups, each using 20 to 30 SaaS applications, still exists.
How many recovery processes would depend on email. And it coming from right domain? Or changes in DNS. Or some other factor that domain holder could do. I don't think possible alternative scenarios would go down too well.
Google may have quickly closed the original report because a company like that is soooo overwhelmed with garbage security reports and at first glance it is noise, not an actionable item to fix. there's a term for this that I can't quite recall off the top of my head. Bug Bounty Trolling?
If you think this is bad, wait until you move physical addresses and don’t notify all your creditors!
Literally ANYONE who moves into your old house will have access to your bank statements, credit card bills, tax forms, auto registration and who knows what else. And it will all come plaintext and unencrypted!
Phone numbers too. Ah, but it's illegal to open mail addressed to someone else. (Although what if you change your legal name to match???!) Perhaps we need an electronic messaging service backed by the USPS.
> Shmoocon talk accepted - Dec 9, 2024, Google re-opens issue - Dec 19, 2024
Wondering what the internal process there was at Google? Is someone's job to cruise around the various conferences and dev events to look for any Google mentions and report back. Did Google really consider it an issue or just wanted it avoid looking bad in public after the talk was accepted.
Yea. The leet amount of payout seems like this was someone saying that the classification was technically correct, but it's worth fixing either way and it'd be nice to pay an amount.
Yes, there are roles where people do nothing but search their names/products/etc specifically to see what people are saying. Hell, some people do it on themselves to the point there are phrases like "vanity search" or "egosurfing"
It's not 100% clear to me, from reading TFA, what the actual vuln is.
Suppose DankStartup folds and I, being a morally-dubious sort of fellow, purchase dankstartup.net which I then use to sign into DankStartup's O365, or DankStartup's ChatGPT as a DankStartup employee.
Isn't that a failure on DankStartup's part, to not shut down their business accounts? And isn't it also a failure on e.g. Microsoft or OpenAI's parts, since they're providing service to a defunct business entity who can't pay its bills?
To describe this as a vuln in oauth doesn't really make sense to me.
Well, think about it this way:
1. I create DankStartup and my company uses Google workspaces and Google auth for a bunch of stuff, like payroll.
2. DankStartup goes under and we close our Google accounts/let our domain lapse.
3. Someone else buys DankStartup.com, sets up a Google workspace, and attempts Google auth to log into stuff, and it works.
The problem is that the original DankStartup has a Google account that they create in #1, and Google goes around telling other sites (via Auth) "this is user X from company Y".
Then, the impostors in step #3 create a different google account with the same domain, and Google says "yeah, these are definitely the same guys as before", even though Google is fully capable of discerning that that is not the case; these are different people with a different workspace account, different names, different payment information, and so on, but Google is saying that if you're holding the domain you are therefore the same people as far as they're concerned and is asserting that to other companies. They are (or were) refusing to provide any indication to those other companies that these are not, in fact, the same people, so those other companies aren't even capable of doing their due diligence of extra validation if they want to.
It's similar to looking at a driver's license and just matching the name rather than the actual ID number; it's possible for someone else to have the same name as you, and identity documents have unique identifiers for specifically that reason.
>Then, the impostors in step #3 create a different google account with the same domain, and Google says "yeah, these are definitely the same guys as before", even though Google is fully capable of discerning that that is not the case; these are different people with a different workspace account, different names, different payment information, and so on, but Google is saying that if you're holding the domain you are therefore the same people as far as they're concerned and is asserting that to other companies.
Yep I understand the mechanism by which this gets abused; I think we just disagree on the implications. I don't work for Google but it seems from the outside that they're treating the OIDC subject claim as referencing the domain attached to the workspace account, or something similar. I've seen implementations where the `sub` claim is more granular, so to me that indicates the field is underspecified.
Given all that, I suppose TFAuthor's proposed solution is a good way forward.
I still think classifying this as an OAuth vulnerability isn't correct.
9 replies →
It seems there are two possible problems.
The first is whether taking over a lapsed domain allows you to takeover an existing Google Workspace (or Cloud Identity) organization. This it what houses the corporate email accounts and OAuth client registrations. If Google allows this scenario then the linked account takeover is simply one symptom / side effect among many. TFA is not clear on whether this step actually happened... I assume not, since if it were the case we'd be talking about direct access to the Google account data rather than only linked SP accounts.
The second is when an SP doesn't properly use the `sub` claim as a unique identifier. It sounds like some products don't understand this requirement and why it "seems to change 0.04%" of the time. I do agree that a unique identifier for the org itself would be a good addition to the token.
That said, I'm still not clear how the second problem manifests if the old OAuth client creds (housed in the old Workspace org id) are invalid. Presumably attacker can login to the SP admin account using just email based password recovery, then reconfigure the OAuth integration with new secrets. In that case the SP is failing to do MFA on the email login.
Would love to hear if I'm missing something.
2 replies →
>>>The problem is that the original DankStartup has a Google account that they create in #1, and Google goes around telling other sites (via Auth) "this is user X from company Y".
Google is telling other sites that it's bob@DankStartup.com - isn't that true? Isn't this on DankStartup to close down operations cleanly?
22 replies →
> They are (or were) refusing to provide any indication to those other companies that these are not, in fact, the same people
That is not quite true, the sub field will be different.
5 replies →
I don't see the problem here.
We've already settled on DNS being the auhoritative authentication key for everything. If you bought the domain then you've bought the corporate identity. It's working as designed.
2 replies →
Yes but he's saying the data on 3rd-party websites should be deleted by the failed startup when they shutdown not just left to sit there.
7 replies →
Any domain takeover allows email takeover which allows you to send password reset emails for former employees. Does not matter if it is with oauth or not
8 replies →
I agree, I don't think this is a problem with Google's Oauth implementation, it's a problem with the service providers who authenticate users via the mere existence of an email address ending in @company.com without checking if the email address actually belongs to an active employee.
If, when you logged into Slack via Google Oauth with the email address user@company.com, Slack checked with company.com whether user@company.com was a valid user that should be allowed to login, then this problem would be avoided entirely because the defunct company would no longer report any valid users.
This would also avoid further problems with attackers being able to login with unattended email addresses like support@company.com, as was discussed here: https://news.ycombinator.com/item?id=41818459 (There's a lot of discussion below about whether the "sub" claim is stable or not but it's a red herring because of this IMO, also the proposed fix in the article wouldn't address it either.)
How would they check that?
6 replies →
This is a misunderstanding of the problem. See my comment downthread: https://news.ycombinator.com/item?id=42701912
"hd" is Google's solution to this problem, and "hd" is also the source of this vulnerability.
5 replies →
First of all, the whole point of SSO is that the only and final source on who are the valid active users is the IdP, in this case Google's public OAuth instance. If the IdP says that the current request is coming from the real user1@example.com, you give them access. And Google's public OAuth instance will confirm this even if the current Google Workspace associated with example.com is a different one than a few months ago (though the "sub" field of the attestation will be different than before, which the service provider is supposed to check).
Second of all, even if it was recognized that this is a different user1@example.com, they'd still have access to all sorts of company internal resources, that may still contain sensitive data, especially for small companies which inherently trust all employees.
1 reply →
If the attacker is in control of company.com, checking against this domain would not help.
16 replies →
Yes, why the Slack or HR or interview etc. data would still exist while being inaccessible to the original owner is very strange. The article seems to take it as a foregone conclusion that those accounts should all be expected to still exist with all the business data, and I don't know if this is based on the author's experience, or it's just a way to make the vuln sound far more serious than it really is.
On the other hand, I don't think people normally expect that OAuth depends strictly on domain ownership like this. I think most would expect that it depends on some kind of secret being stored on the IdP account side, uniquely identifying the IdP account holder beyond the email addresses presented. With regular password-based authentication, with MFA, you would at least get an MFA prompt if someone had gotten access to your email address and was being sent a password reset code. But with this type of SSO, any MFA verification is done on the IdP side, so if the IdP recognizes anyone who controls example.com as the rightful owner of any claims, then there is nothing you can do as the former owner of example.com.
>why the Slack or HR or interview etc. data would still exist while being inaccessible to the original owner is very strange
Ever been in a company that's collapsed before? Nobody hangs around to shut down the WorkDay account when they've been told they're not getting paid for the last 29 days work.
1 reply →
Yes, it's a failure on DankStartup's part.
Not really much different than a user buying dankstartup.net, setting up a catch-all email, observing what comes in, and performing password resets for those accounts, allowing for account takeovers.
Calling it a vuln in oauth may be a bit hyperbolic, but Google could help prevent it.
I have catchall email accounts in every domain name I own (mostly so I can do differentiated emails for every service to track/combat leakage), and you would not believe the amount of emails I get that are intended for previous domain owners (and typos too). I haven’t actually done any reset password flows, but there are a bunch of social media and SaaS accounts I could easily take over if I wanted. I used to try to track down whoever the emails were intended to go to and forward it to them and let them know to change it, but that got to be too tedious so nowadays I just ignore them.
Still, I wouldn’t call this a vulnerability on the service provider’s part, it’s just user negligence.
> Suppose DankStartup folds … Isn't that a failure on DankStartup's part, to not shut down their business accounts?
Expecting an entity that has already failed* to not fail again isn’t an effective security control, unfortunately.
* - not every startup that folds has “failed”, but the point still stands
>Expecting an entity that has already failed* to not fail again isn’t an effective security control, unfortunately.
Sure, I'm sympathetic to that, but again I don't see how that's within the scope of oauth.
10 replies →
The contention isn't that you can impersonate DankStartup, that's obviously not a vulnerability since you are indeed the domain owner. It's that former entities with DankStartup accounts might have used OAuth to create relationships of their own. And when the startup folds, they don't magically disappear.
Basically if Sally, the CTO of DankStartup, signed up for Taskrabbit or whatnot, it's possible for you as the owner of the domain to impersonate Sally in the context of that relationship.
Obviously the root cause here is that someone misused an account to do something not related to the business. And the actual impact is probably low since high value services tend strongly not to take tiny email domains as identity roots (i.e. sally@dankstartup.com clearly doesn't have a Vanguard account to steal).
So... like most security announcements it's oversold and spun. But it's real enough as I read it.
Why do you assume sally@dankstartup.com doesn’t have a vanguard account? I’ve absolutely had similar retirement account logins that became difficult to access once I left that employer. Had to contact HR and get them to help me log into my account. If the company had folded during that timeframe I would’ve been screwed. Of course for financial institutions you can probably recover your account through some identity proving process, and generally money transfers require a second factor sms auth, but a domain takeover would probably have been sufficient to at least get someone logged in and able to see my account balance.
1 reply →
> Basically if Sally, the CTO of DankStartup, signed up for Taskrabbit or whatnot, it's possible for you as the owner of the domain to impersonate Sally in the context of that relationship.
I don't think this is the issue, unless someone went to some pretty extreme lengths. Configuring OAuth such that the company Google Workspaces account is recognized by Taskrabbit as a valid SSO option is not as simple as signing up to Taskrabbit with your company email instead of personal email.
2 replies →
That sounds like a TaskRabbit vulnerability (in your example), not a Google vuln? It is also a vulnerability in any email based sign in, which relies on email alone without a password to demonstrate account ownership. (Including password resets that rely on email).
Besides the github account with a legitimate name, but impostor ownership, I can imagine another case which is problematic. HR SaaS companies might be required by law to maintain the account for a period of time - let's say until the end of the next fiscal year. The account still existing is not a failure on either Google, the SaaS company or DankStartup. The problem, as other posters mentioned is the fact the new account has the same ID in OIDC as the previous one, which is what the author of TFA proposes.
And not only that, almost every form of auth has the same vulnerability not just log in with Google. If you own the domain, you own the email IDs as well and you can very likely reset password.
Yeah, this is the part I'm struggling with. This is absolutely not unique to google oauth, it genuinely seems like a misunderstanding of how the web manages trust.
If you own the domain, you own all the property associated with the domain, including all the old email addresses. Magic links and password resets are all going to give the new owner access.
Your best bet as a solution is to be using strict 2fa (ex - a yubikey might help here) but even that is likely just "a conversation with support" away from being circumvented.
This is why winding down a company is supposed to have specific stages and policies associated with the dissolution. You don't just abandon the offices and leave all the filing cabinets behind either, for similar reasons...
4 replies →
The scenario I was envisioning was a simple as a developer that worked at that startup added DankStartup email to their Github account, committed a bunch of code. Company shut down, but dev doesn't think to disassociate that domain/email. Malicious actor uses this technique to then sign into the Github account, gaining access to that developer's account, not just the DankStartup repos which were probably disabled after non-payment.
Now, obviously some fault in that scenario lay with the person who 1) used the same account, and 2) didn't remove the old email once the startup failed. But I'm just using that as a kind of example - there may be other accounts as others have said that need to be accessible years down the line, like financial records and the like, regardless of whether the company is still around.
> DankStartup's O365,
If you wanted to try to re-take over a pre-registered domain in Azure/M365, you would need to involve Microsoft Support who would require you to prove via government documentation that you were the same entity for the now-abandoned Azure/M365 account.
Creating a new tenant using the same public domain won't get you access to the Azure/M365 tenant of a previous customer, you would have a different internal domain name (dankstartup2.onmicrosoft.com).
At best you could register your domain post-expiration of the previous tenant and impersonate them >90 days out.
It's similar to Gmail or Hotmail or Yahoo Mail hypothetically allowing a new user to adopt an abandoned email address. Most email services don't allow this for this very reason.
Third parties is google as single sign on. If such an org/domain is abandoned, a malicious person could 'resurrect' the accounts on those third party sites. It's kind of like a similar situation where a user deletes an email account and then someone else creates an new one in its place, and now they have access to all the third party password reset links. Except in the case linked above, the user never had the ability to delete their accounts before off boarding, and the abstraction isn't just an email, but the account itself.
In emails case all the larger providers I'm aware of retire addresses to prevent this sort of abuse, probably learned the hard way. Could Google OAuth do something similar here?
It now occurs to me that large businesses might have the same issue with emails and former employees (easy to imagine John Smith departing and at a later date John Smith being hired) so I wonder how they typically handle this.
1 reply →
My father changed careers and dissolved his business over a decade ago, at which point he let his old domain lapse.
A couple years ago, I began to wonder if anyone was still trying to reach him, so I bought the domain (domain squatters weren't even interested in it anymore) and setup an inbox for his old email address. In less than a week, the unread count was in the triple digits. It was basically all marketing and spam so I closed the inbox within the month, but multiple years with a 100% bounce rate is apparently not enough to deter LexisNexis & friends from trying to win him back.
This is a corporate planning/governance problem, not a vulnerability in Oauth, and I bet the problem is even bigger than TFA describes since any previous domains of a company that were almost-but-not-completely migrated away from could easily be taken advantage of, possibly to even greater effect since the company likely has a bunch of systems still up which have the domains whitelisted. (Which makes me wonder if there are hacking groups who use domain squatting as a front or side-gig while they take advantage of lapsed corporate domains.)
This is very much a critical flaw in systems which rely entirely on email or domain names as the sole security factor. OAuth itself is flawed in that it allows this terrible practice, though to be fair it does discourage it. Google's specific OAuth solution encourages this compromised practice even more, because the only identifier it provides for a user group is the domain name associated with the account (in the "hd" field). For individual users they at least provide a "sub" field with a unique user ID (though per the article, it seems this is possibly flaky, or at least misunderstood) - this could be used to make sure that a new user@example.com doesn't get access to the old user@example.com account. But to check if a new user should have access to the organization account, there is nothing that Google provides that can be used securely: all they tell the service provider is that this is a new valid user for the example.com org, not that this is a completely different example.com org.
2 replies →
Is this basically equivalent to registering a domain, setting up email, and getting services to send you password reset emails to recycled addresses?
What if DankStartup doesn't fold and just forgot to renew their domain, or lost control of their domain for some reason. OAuth shouldn't let you Auth someone else's account just because you can hijack their domain.
I don't think it would be 100% on the failed startup considering most of, if not all, of the applications mentioned likely do not actually delete accounts even when you say to delete them. There probably could be some allowance made for the situation but also you're repurposing the domain of a failed startup.
I don't think the problem was necessarily with Google though as I believe they create a unique ID for the domain. The problem with be with the other services like ChatGPT, Slack, etc that associate with the domain and create accounts for each user in the domain.
it's vital to have extremely skilled and nuanced people managing bug and security report triage because it seems many can overlook the complexities being given.
It is an "ACL vulnerability ".
ACL as in access control list.
Bell-LaPalda and multi-level access control systems both required a trusted controller and, And, AND a trusted arbiter of a state machine.
It is the state machine that magically got expanded into a couple more states, with the introduction of an imposter admin actor.
So yeah, the trusted arbiter and owner of that state machine (Google) did an oops.
This is something that Keberos (KRB5) was designed to protect against.
> a failure on DankStartup's part, to not shut down their business accounts
They seized operations, why should they be responsible for any other assets?
I mean ethically yes of course. But there's no law/obligations to make sure domains are closed.
whats is TFA ?
"The F-ing article" (derived from old internet slang "RTFM", for "Read the F-ing manual", a thing that new users were prone not to do before getting online and asking questions easily answered by RTFM.)
1 reply →
"The f___ing article"
4 replies →
I really don't understand here, the proper way to use Google's OpenID implementation to authenticate someone is to use the `sub` claim. Don't use the email, don't verify it yourself, use the `sub` claim. It's a known fact and is properly documented[1].
If the `sub` changes, it's because it's not necessarily the same person so have a flow ready for that. It could be an employee left and came back, a domain change, an IT error that lead to a reprovisioning of the user, etc.
I also fail to see how the proposed solution of having a 'A unique user ID that doesn’t change over time' is different from the `sub` claim. However, the new ID associated to the domain could make sense to enforce a strong 'Everyone from the @domain.com has access' statement.
[1] https://developers.google.com/identity/gsi/web/reference/js-...
It all depends on how true this statement is:
> “The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week”.
I've been working with an app that uses Google to login for the past 10 years, and I've had problems with sub changing when these situations happened : - Domain change - Company being bought by another one and being integrated in their Google Workspace - Employee leaving and coming back
To us, it's very very far from the quoted 0.04% which is to me very high. I had to deal with it 5-6 times in the past 10 years but of course that number will vary depending on the usage of your app and I'm not gonna venture and put a percentage on it.
6 replies →
Exactly. How is there an entire alarmist article and 165 comments on this thread. This comment, and it's legitimacy/factualness, is the only thing worth discussing.
`sub` _IS_ the immutable reliable identifier. If it's not, (1) I want to see actual proof, not an anonymous rando (sorry, but this thread re-inforces how little I trust context-less comments like that), and (2) I'd want to hear a convincing argument that ... `sub2` would actually be less mutable.
Threads like this make me really question other peoples' general comprehension skills.
2 replies →
0.04% is a few times higher than I'd expect it to be, but if it were actually that bad I'd expect some form of previous report about it to be findable on the internet. I did some searching but couldn't find anything. It would be a significant bug on Google's part, but stranger things have happened.
What's astonishing to me is that apparently all these big service providers did notice, and then they decided to disregard the one identifier Google tells them to use? Fundamentally that's the security bug being reported here, it's just being reported to Google instead of those service providers.
A stable alternative for the hd claim would of course be a good idea. It would provide a more complete way to deal with the inherent security issue of allowing domain-based signup without further authorization steps. But given the above I'm not convinced these service providers wouldn't start ignoring it after the first time somebody re-registers a domain with Google Workspace.
I am pretty confident that the statement is false.
The `sub` claim value is equivalent to the user ID that's exposed in the Directory API, it's derived from the underlying user account's unique user ID, and it won't change unless the user account is recreated.
I've implemented OAuth flow glue logic a bunch in my career and I agree that the logic doesn't seem to follow from what the documentation says. What's not clear to me though while reading this is, if I delete/lose a domain, and someone re-makes the domain and account with the same email address, does the sub value then change? The docs would seem to suggest yes, but then I am not clear what google actually ended up "fixing" in its bug bounty. Was that maybe the bug, and this is just not clearly explained in this post?
The reason this is an issue is that the sub value changes often enough that intergrations ignore it rather than bother users with having to re-OAuth.
As far as what Google is doing in the bug bounty, that's a good question -- we don't know. The author is proposing two new values, for the domain and user, that wouldn't change in the cases that sub changes now, but would change in this case.
There's a reason this happen though. Whether you like it or not and regardless what the standards say, people want a technology that matches with the identifiers they want to use (eg email) and not whatever opaque value happens to be in the sub claim.
I don't like blaming users for wanting something sensible.
> I don't like blaming users for wanting something sensible.
Are you saying that it's sensible to want email address ownership (and thus domain name ownership) to be permanently unchangeable?
Yes this is not a "Google" problem so much as a "Your SaaS application is literally doing it wrong"
Having worked on a lot of SSO integrations I can say that everyone does it wrong.
I will also say that there is a super scary aspect to the sub claim (in general). It should be pairwise tied to the SP's integration, so if you only use google for auth, and for some reason you lost your google dev account and had to set up a new one you would drop credentials from 100% of your users.
for google I don't know if you can scope for the account ID (the opaque unique identifier) since that could expose privacy issues via cross integration tracking. But that's basically the fix. if you had the unique, immutable, opaque account identifier and matched on that this wouldn't work since buying a domain and setting up that email should result in a new account with google.
Yeah, I'm not really clear on the vuln. It sounds like clients don't validate sub claims because they are unstable? Google's docs say they should.
When I first read the post, I thought that maybe Google was reusing sub claims when they shouldn't, but on a closer read, maybe it's the opposite problem - that the sub claim is changing when it should not change?
From the article -
“The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week ... Because the sub claim is inconsistent, it cannot be used to uniquely identify users - leaving services reliant on the email and hd claims."
So maybe clients aren't using the sub claim because validating "email matches && sub matches" breaks real users who are not actually changing accounts? AKA, Google is changing Google Account IDs unpredictably? That would be a bug on Google's part IMO.
If those account IDs in the sub claim are changing, at the very least that's a bug in the doc you linked IMO.
I'd have to read the full OIDC spec [0] to see if the sub must remain constant for the same end user, and if clients are required to validate the sub claim before processing a login. If so, and it's changing without the end user's identity changing, that is a bug in Google's implementation and client implementations for anyone using Google as an IDP.
If it's allowed to change, though, and validating sub claim is not required in the spec, maybe clients are correct in not relying on it.
[0] https://openid.net/specs/openid-connect-core-1_0.html#rfc.se...
Edit: from reading the sign in with google overview, it's unclear if they claim to be OIDC compliant: https://developers.google.com/identity/gsi/web/guides/overvi...
it says it's "based on OAuth" rather than "implements OAuth+OIDC" so I guess not being fully spec compliant wouldn't necessarily be a "bug."
I ran into this when implementing OIDC at work. One of my teammates wasn't able to login even though he just linked his google account. Turns out based on which browser session he was using, the same google login was giving a different sub. Super weird behaviour.
1 reply →
Genuine question, what would that flow look like?
I mean, I see why it happens. It's because the client uses the email as the identifier for login - it is the only shared identifier between Google and Slack (or whatever service) to connect accounts. Sure, you could argue that maybe this is ok for the initial login, but the client app should record the sub...
Note that for regular logins without Oauth, you still have this vulnerability - if you gain control of me@company.com, you can login to any site that uses that for login. This is a fundamental weakness in the way we handle domain registration and email addresses.
This is more of a fundamental issue with the way we rely on DNS to secure *gestures wildly* all the things. The fact that domains can expire and be registered by someone new allows the new owner to do almost anything the old owner could have done when it comes to anything trusting email addresses, or anything else relying on DNS (ACME certs) for authentication.
It's great for "do they own this right now" validation and that's what we use it for, but beyond that links will be saved, email addresses will be added to databases and address books alike, and that's more or less a reality of most systems. For example, my snail mail (and occasionally packages) is still mistakenly delivered to my former addresses from time to time just because it's hard to track who has that address, and update it everywhere. The same goes for internet infrastructure.
I was able to take over an Instagram account because I received a password reset email to a domain catch all address that I'd set up years ago. Turns out it was a whole brand, but they refuse to change the email address and don't understand that I'm going to shut it all down. It totally makes my site look bad (it was an adultish brand and I was working on a community project) and I now have the power (and ability to do whatever with the account and any other that I happen upon with an email at that domain.
With OIDC, that's what the `sub` identifier is for. Apparently, this changes for 0.04% of accounts (according to "a staff engineer at a major tech company" quoted in the article), but it's what Google tells you to use to identify a user. When a domain expires and you re-register for OIDC with Google, the `sub` claim will differ and the application should refuse your login attempt.
If the `sub` claim does indeed change for no good reason, that's a bug on Google's side, but I can't find anything online to corroborate that claim. The sub value changing when accounts are recreated could be a pain for large companies to deal with, but other than that the solution is pretty simple.
There are no good alternatives for email addresses that are also human readable, unfortunately. Some web3.0 cryptocurrency projects do encryption based authentication that solves this problem (but makes your accounts impossible to recover if you lose your key file). Passkeys/WebAuthn also try to solve this problem by foregoing the entire username requirement, but websites still ask for email addresses just in case.
I disagree. DNS stores enough information in WHOIS to see if ownership has changed, it's not DNS' fault that nobody looks.
Probably the least-wrong thing to do with current DNS is to have authentication servers keep track of the WHOIS UpdatedDate of email domains. If a WHOIS UpdatedDate is newer than the corresponding user's linked email address verification, that user's email address is no longer trusted. Next time they log in ask them to update or re-confirm their email address, and if they try to password reset they can't use an unconfirmed email address.
Yes that's more tricky work. Authentication is hard. Nobody should be DIYing authentication anymore in this day and age, it's just too much.
What does this protect against? If the WHOIS changes because someone new buys the domain then they could simply receive this reset emails, no?
2 replies →
You can put whatever you want in WHOIS, including just replicating the information that was there previously. What if the WHOIS email is an email on the domain in question?
Maybe registrars could set a unique ID per registrant, and if a domain expires and is purchased by a different entity/account than the previous one the registrant GUID is refreshed. That could then be a signal that all previous reliance on the DNS of the domain name should be null and void
1 reply →
At least it counts as a new category on the "It's DNS fault" list.
It's crazy just how little effort it takes to get a "Google = bad" article to the top of HN.
There is no vulnerability in Google OAuth. This is exactly how every OAuth server is supposed to work. If you take over a domain, you automatically own every email address in that domain, and thus whatever external account relies on that email for login. Heck the result would be the same even if that service didn't use Google OAuth, or any OAuth at all.
Nothing in that write-up makes sense.
> If you take over a domain, you automatically own every email address in that domain, and thus whatever external account relies on that email for login
Actually, if both sides implement OIDC+OAuth2 correctly, you don't. The subject claim (`sub`) of the attached Google account doesn't get reused when a new owner re-registers the domain with Google.
The article claims that the supposedly immutable `sub` field changes too often, though, and that would be a problem Google needs to fix. The source is an unnamed developer mentioned at https://youtu.be/yIutY_X2FcU?t=20617 who encounters issues with custom domain users, with the `sub` field changing weekly in some cases.
Sure, you can create new accounts when you take over a domain and even fake all the old accounts if you have a list somewhere, but you shouldn't be able to access all of the accounts authenticated with OIDC unless you break into the Google Workspace account.
Yes, this about sums it up. If you take over a domain, you control its registrar records for what its authoritative nameservers are, so of course you can set it to your own custom nameservers and then define whatever MX you want to receive incoming mail flow. You don't even have to go to any effort of configuring working outbound mail. You just need to put up a very minimal zonefile with the MX defined, basic postfix email server with a catchall configuration and then receive any incoming emails for *@domain.com.
Is this really just google=bad, though? I work at a startup and this seems like a legit security risk that I'm happy to learn about.
It seems like the only mitigation would be to let your HR SAAS know when your company shutters and ask them to delete the records. Or just squat the domain yourself as an ex-employee.
Yes it is, otherwise the title would be "don't use your email address to log in to any application" and it wouldn't be ragebait enough. The whole issue has nothing to do with OAuth and nothing to do with Google.
Mast domains aren't that expensive. Can a startup just buy 10 years of peace of mind in its dying days?
It's crazy that they do that only because "Google search bad now" or "Youtube bad now".
I think part of the issue is that this is where the abstraction that we call "account ownership" starts to leak.
You may correctly have access to an account through this scenario, but that does not make it your account. This becomes obviously when we consider an account at a bank, for example.
The staying power of a Google = bad article on HN top page is insane.
And the opposite: The half-life of an Apple = bad article on HN is insane in the other direction.
Nearly a decade ago, there was a website called thehunt.com that basically ran out of money and some employees were acqui-hired by pinterest.
All of the assets were left to rot and at the time the company was a good match for another startup of mine. So I reached out to the investors I found on crunchbase and asked if I could acquire everything. We worked out a deal and I did. The issue was the complete lack of people from the old company willing to assist and the complete lack of data for alot of things. There was 1 or 2 people who we could ping from the old company to ping who were super helpful, but the big thing was many things were just lost to time- passwords, history, code repos, etc..
Simply creating a new google apps account allowed us to get full access back to everything - We could even read old slack messages (even DMs!) by resetting each accounts password. The whole thing was shocking to say the least, but with that access we got back into literally every service they used and managed to get it up and running again within a week, which was a good thing because nearly every service it was using was threatening to shut it down every day for lack of payment.
I think the solution here is actually way simpler than most make it out to be and could easily be a startup for someone:
Create a startup that lets customers simply enter in domains. If the domain EVER goes into the "pendingDelete" status, inform the customer. The customer would be random SAAS's that want to protect against this type of attack and could simply choose to disallow access to any account that has had their domain go into that status.
Huh - it's almost like the mere existence of a pendingDelete in the history of a domain should make every email existing on that domain at that time eligible for https://haveibeenpwned.com/ .
> To resolve this issue, Google could implement two immutable identifiers within > its OpenID Connect (OIDC) claims: > 1. A unique user ID that doesn’t change over time. > 2. A unique workspace ID tied to the domain.
1. is the OIDC `sub` claim! I strongly suspect that in those 0.04% of accounts where the anonymous quoted engineer reports that the `sub` claim changed, what actually happened was some provisioning/onboarding/offboarding system resulted in the account being deleted and recreated.
2. is sensible, and is just a versioned version of the `hd` claim.
1. Yep, your hypothesis seems likely. Consumer gmail addresses can't be used again after they are deleted, but it seems in Workspace orgs they can be reused/reassigned after 20 days: https://support.google.com/a/answer/33314?hl=en&co=DASHER._F...
If services are not respecting the `sub` claim in this case, then they are giving the new Google account access to the old account's data. Companies probably wouldn't complain about this because they think it is the expected/reasonable behavior. Also it's likely that in many scenarios it is the same human behind the different accounts, e.g. if they leave a company then return.
Reinforcing the face-palm at the heart of this, which is that anyone deciding to you know, just use email instead of asking why an immutable ID changed... just probably enabled information leakage. Seriously, I'm so thankful that my colleagues would be principaled about this and ask questions instead of just doing something to make it "work". Where "work" means some GSuite user probably logged into some other defunct GSuite user's RP-account.
2 replies →
As others have noted, this feels like an issue in the relying parties not relying on the `sub` field to validate the user. It feels the exact same as this issue here: https://bhavukjain.com/blog/2020/05/30/zeroday-signin-with-a....
In both, the details around the `sub` field, the field that should actually be used to identify the user, is poorly described. I would say that both of these feel a bit sensationalist, but then again, if relying parties are NOT using the sub field to validate users, they should be called out.
It seems to be that a good way to make some money is find every such situation where RPs are not using sub, and submit vuln bugs.
> According to a staff engineer at a major tech company:
> “The sub claim changes in about 0.04% of logins from Log in with Google.
I've had a staff SWE also claim to me that they generated colliding UUIDv4s, and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
Neither's attention to quality, or rather, lack thereof, could cash those outlandish claims.
There is a simple explanation here: Your staff SWE is wrong. `sub` is the claim you're looking for.
Extraordinary claims require at least a trifling of proof. Or, what can be asserted without evidence can also be dismissed without evidence.
And this alleges to be a security blog. As a writer with that target audience, you should know this is going to be the objection to the thesis that your reader will have, and you attempt to just handwave it away…?
> I've had a staff SWE also claim to me that they generated colliding UUIDv4s, and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
The claim about circles I don't know. It depends on what exist means, and I don't know what it means to have a circle in real life, and likely don't care. I can only draw a rough approximation of a circle, and that's been fine for me.
Generating a colliding UUIDv4 seems pretty simple though; if you have a broken enough random generator setup and manage to run it without seeding, especially if it was in the times where it was pretty easy to run a virtual machine with totally broken random (virtio-random was developed for a reason), and spawned a bunch of virtual machines in very similar conditions. You can no true scotsman your way out of this by declaring that a broken system, but from inspection, I don't know how you can determine if a given UUIDv4 was generated with proper or broken random techniques. See also Debian Security Advisory 1571-1 [1], and similar issues where random values that were intended to be secure turn out to be predictable. It's a plausible claim. But that doesn't mean a claim by a 'staff engineer' is default plausible. It's just an appeal to authority of a title that doesn't mean a lot.
[1] https://lists.debian.org/debian-security-announce/2008/msg00...
> I've had a staff SWE also claim to me that they generated colliding UUIDv4s,
UUIDv4 is random. Could be a bad PRNG, or just very very very bad luck.
> and a separate staff SWE who worked in GIS claim that circles only exist in map projections and they're always distorted, and that you cannot have a circle IRL, nor project it onto a projection.
It is insanely difficult to project circles. Most of time we just pretend it never exist.
> or just very very very bad luck.
You're underestimating the odds. The odds aren't "bad luck", the odds are "statistical impossibility". Literally any other explanation holds with orders of magnitude higher certainty.
> It is insanely difficult to project circles. Most of time we just pretend it never exist.
Open Google maps, find the equator. That's a circle, rendered on a projection. (But far from the only example possible.)
(And lest I get more objections, a perfectly spherical earth was also an assumption of this assertion that circles don't exist. My coworker attempted to draw/show a circle onto an actual sphere we fortuitously had handy, and no dice.)
(Note the other way exists too: a projection can have a circle rendered on it, though it wouldn't be a circle on an actual sphere.)
> There is a simple explanation here: Your staff SWE is wrong. `sub` is the claim you're looking for.
Or perhaps it's actually happening 0.4% of the time. Maybe a few companies have some weird process that results in them routinely deleting accounts and creating new ones on the same email address.
Is this really an issue? Back when I implemented Google OAuth I used the "sub" that Google gave me as well as the email. I'd wager that Google does not re-use a sub for the same E-Mail so if the account is deleted and then somebody grabs the domain and re-created the account they'd have the same E-Mail but a different sub which would make the login fail. Seems to not be an issue in Google's OAuth Flow but rather the implementation on services like ChatGPT and so on?!
EDIT: Just read that they claim that “The sub claim changes in about 0.04% of logins from Log in with Google” - never had this happen in the past years but if this is true then I guess thats a bad thing.
Are your users primarily using gmail accounts or are they using accounts from custom domains? TFA does't say exactly, but I wonder if this stat only applies to users with custom domains rather than @gmail accounts.
The "millions of accounts" is highly speculative, is it not? It relies on the idea that failed startups keep their SAAS accounts enabled, rather than offboarding correctly.
You would think that because you're a reasonable person. However at my previous startup, which no longer exists, the new CEO failed to save the company and parachuted out so fast that the company who later acquired the startup couldn't get the domain name transferred to them because the CEO and CTO were both gone. They had to bring back the predecessor CEO because he was still able to log into the registrar to transfer the domain.
In the case of a fire, some less than conscientious people will fend for themselves first and don't give a second thought to anyone else.
I agree there will some number of companies doing this, between 1 and $number_of_companies. I'm just saying it's pretty unlikely it's all candidate companies, which the article seems to be saying.
>> is highly speculative, is it not?
I'd say NOT.
Shutdown of a small company running out of funds is rarely fully orderly. Critical things may get taken care of, but everything is unlikely, as people have already left. Plus, with things like already-paid-in accounts, there's even an incentive to keep them open until the term expires, e.g., why not keep the prepaid ChatGPT API account live for execs and techs to use between jobs?
Just like wiping hard drives on EOL computers — everyone should do it every time, but people are always finding full HDDs and SSDs on eBay and Craigslist...
Not just this, but execs and employees like to keep websites live for a while while they job-hunt (otherwise their resume has dead links). And ideally keep their name@startup.com email working. But to keep those running requires a long tail of "live" DNS and finance stuff you can't quite close out.
1 reply →
also, there are lots of non-technical people running small businesses that use G services because that's all they know. They can barely operate them let alone be cognizant of things like this "exploit".
> They also classified the issue as a “Fraud and abuse” issue, rather than an Oauth/login issue.
I can _kinda_ see that, I agree they should mitigate it as best as possible too though, especially since they're Google after all.
> I thought this would be the end of the story, but 3 months later, they re-opened my ticket (after my Shmoocon talk was accepted), paid a $1337 bounty, and said they were working on a fix.
Sad how the only way to get Google's attention to make enough noise about something...
$1337 is a very curious amount. Surely, someone was being clever???
yeah, google likes to have fun with the Bug Bounty program. See: https://bughunters.google.com/about/rules/google-friends/662...
1 reply →
It's leetspeak
It spells "leet" - see https://en.wikipedia.org/wiki/Leet
You're too young to have used BBSes :)
3 replies →
This actually happened to me a few years ago when I closed my consulting company. My bad, I know. My domain was taken over by someone in Vietnam. Back then, he was actually able to see my old emails via the Google Apps. Grandfathered free Apps accounts was a big part of the value proposition. He demanded $500 to delete my emails and docs. I offered $50 for his trouble since I recognized that I was at least partially at fault but he refused. To his bad luck, one of my main clients was Google itself. I reported him via my internal connections and the next day got the following email: "Oh my G-d, all forty of the domains I bought got Google Apps cancelled".
> “The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week”.
What I don't understand is why the `sub` claim is not consistent for those users at Google. To my understanding of the OIDC protocol the `sub` should be unique for a specific user.
Additionally as far as I understand if you take over a defunct domain and create a new google workspace with new users those new user account should get assigned a new `sub`.
I agree, in my limited experience the sub remains consistent even when changing the Google Workspace domain. So the email changes but sub remains the same. The issue seems to be clients matching on email/hd claim instead.
I wonder what action is causing the sub to change like the author suggests is happening.
At my current company, if an employee leave and come back, they'll keep the same OID in Entra but they'll get a new `sub` in Google workspace. We had to put in place a process to be able to use an internal tool that used the login with Google.
That's most likely dependant on how the IT department handled the deprovisioning/provisioning of users in our Google Workspace, I unfortunately don't have the details for that.
> I wonder what action is causing the sub to change like the author suggests is happening.
Indeed this would be very interesting.
This issue is also very similar to CVE-2024-25618.
What we did to mitigate this is the following: - Federated login with OIDC - Look for a user based on the sub claim - If they are found: authenticate that user and optionally update their profile (email, name, ...) based on then new id claims. - Else look for a user matching on the `email` claim and link the `sub` to that user - If no user is found create a new one
That sub identifier changing without the underlying user changing may be the core of the bug. It's not fully clear from the post IMO. I posted a little breakdown here: https://news.ycombinator.com/item?id=42701030
I don't think the article clearly states one way or another whether those 0.04% of sub claims changing are legitimate end user changes or not. If those sub changes are legitimately the Google Account changing, I don't think that's a bug on Google's part, but rather is a problem on the clients' side for not validating the sub claim.
If the sub is changing without the Google Account user actually changing, maybe something internally at Google is swapping out those IDs when it shouldn't be. It doesn't look like a UUID from the example I saw, so maybe there's some code somewhere to just change the user's ID if there's a collision or something?
Interestingly, I don't think Google claims that Sign in with Google is fully OIDC-compliant. At least in the overview I linked in my comment, they compare the implementation to OAuth+OIDC, but I'm not sure they claim to be 100% spec compliant.
How would the clients tell if the account has a valid sub change or not if the only piece of information provided is that the sub claim changes? For this particular attack, without having some kind of Google Workspace account identifier for the domain, the sub claim doesn't sound sufficient to validate that it's the same Google account from the client's side. I'm guessing the engineer at the major tech company didn't provide that stat without checking if those users were valid, active accounts.
In iSCSI, there are a few options for addressing the block storage devices, with one being the iSCSI Qualified Name (IQN) that uses the domain name. But it includes a timestamp:
* https://datatracker.ietf.org/doc/html/rfc3720#section-3.2.6....
* https://en.wikipedia.org/wiki/ISCSI#Addressing
* https://datatracker.ietf.org/doc/html/rfc3721
Clever idea to note change of ownership situations.
Seems like we need a chain of custody for domains, not just current ownership. Kinda like real estate. And then accounts are tied not just to the ownership state, but also the ownership instance ID.
Just don't tell the web 3.0 folks cause I don't necessarily mean a blockchain.
Web 3 already tried this, it was called Handshake. I think their downfall is they wanted to replace ICANN / DNS rather than work with and alongside the current system.
shame, ICANN sucks
3 replies →
> Just don't tell the web 3.0 folks cause I don't necessarily mean a blockchain.
Isn't this literally what certificate chains are for? Establishing a chain or custody - or rather, a chain of liability?
You don't need a whole chain of custody, you just need to know that the current owner is the one you care about. TXT DNS records or magic URLs can do the trick.
This isn't anything new, or exclusive to Google, or exclusive to OAuth.
My workmate hadn't used Facebook for many years and it was associated to an email on a custom domain. Said domain expired and somehow hackers/script kiddies/bots must have an email list, run a whois, buy the domain if expired, setup email, and do password resets across social media and common websites. His Facebook was stolen.
I'm not sure I agree that it's a flaw in Google's OAuth implementation. Domain ownership is the source of truth for many systems and, as the article alludes to, expired domains can be abused in a variety of ways.
We've also seen this in the dependency management ecosystem.
How's this any different than buying an expired domain and doing a password reset .. anywhere?
-- Probably someone else already mentioned this in the comments here, I just couldn't see.
This is a stupid article. 1. It starts with “Millions of Americans”. How does this relate in any way to Americans? 2. Gmail is not the problem 3. Mitigation: Consider registering a domain for 10 years. When the startup fails, delete all dns entries
I fail to see how this is Google's problem and as much as it pains me to say this, I agree with Google.
I believe it is more the downstream providers fault for not terminating inactive accounts, and the failed startup's for not properly deleting their Google accounts.
Google can mitigate this as the author mentioned, but this is not their responsibility. They shouldn't be held liable because failed startups were stupid to not delete their data.
Related maybe, I got a new phone number and I was able to use it to sign into the previous owner's account at a site that uses 2FA. I noticed it when I tried to setup a new account for myself. The website wouldn't allow it but it happily allowed to me access the previous owner's account. It was inconsistent, removing the phone from their account required their password but changing many other important fields didn't.
It seems like there have been a number of security stories lately that essentially amount to. "no tech companies ever does cleanup" Which everyone on HN with a couple years or more experience should know is the norm. I expect a lot more of this to keep happening since companies will never really care about security without consequences ie a $10 fine for making $100 literally incentivies lack of security.
For the people not understanding the security risk, or how this would be something Google needs to solve, have a look at the claims of Microsoft Id token claims, and how their use of multiple guid claims completely eliminates this problem:
https://learn.microsoft.com/en-us/entra/identity-platform/id...
Assuming most of the problem is that federated services only rely on email address from OidC login, I wonder if Google could start offering user+unique-workspace-id@company.domain as the email address for any new workspace/user/federated login? Obviously, Google needs to control "unique-workspace-id".
Does not solve it for existing customers and any federated service they logged into, though if included alongside the simple email, federated services could gradually move to it and announce once they are "ready" so workspace admins can configure them to start accepting only new, safe email addresses.
Okay, this made me curious if there was a technical solution to this that people could be providing using the existing tools available, and I think there is.
RDAP and WHOIS will return the creation date of a domain. These fields are controlled by the registrar not the registrant. That creation/registration date gets reset when the domain is lapsed and picked up by somebody new.
So, when doing any domain-name-based authentication (like email password resets) authenticators should look up the registration date of the domain name. If it's newer than the last time the user logged in using the domain name auth? That might be somebody who snatched the domain name.
Sadly, this isn't fool-proof. Domains can go up for auction or backorder on a registrar, and they won't update the registration date if the domain is purchased this way since the registrar can consider this a transfer. It's a signal, for sure, but it will miss cases. It will also miss transfers sometimes, depending on the registrar.
I was trying to find cases of it happening historically so I could check the RDAP record to see how domain registrars use it in practice... and yeah, the registrars seem to ignore a lot of the spec. While they do generally seem to follow the "lapse and re-register = new registration date", I can see how your example is something they probably would break. RDAP records don't appear to show historical expirations and reinstantiations and re-registrations despite the spec describing events for that. It's always just the basic event entries:
Registration date, last changed date, future expiry date. Even with domains that have well known dramatic histories. Which tells me the RDAP spec is not really enforced.
While I dislike "blockchain all the things" I can definitely see the argument for a blockchain-like global shared public ledger (albeit a not for-profit proof-of-work one) with full history for this sort of data.
1 reply →
This feels similar in spirit to some service providers (*cough* Atlassian -- especially bad when they acquire other companies and push that bullshit on previously functional authentication systems) using ownership of a domain to assert ownership of an account using that domain.
E.g.:
1. You create an account using an email associated with a domain.
2. The domain owner coughs up their protection money to Atlassian and proves ownership of the domain.
3. Your password-protected account is assigned to the domain owner. Your password and 2FA invalid. The domain owner can access your private data without your password and 2FA.
Both TFA and the thing I described rely on broken, simplistic views of "domain ownership == account ownership".
Thankfully, apparently this only affects Americans.
:-)
It sounds like there’s a bug with the “sub” field. The author is stating that they can’t use it as a unique identifier for the user. Isn’t that the root of the problem?
A new domain owner could try to reuse email addresses, but since the domain is now in a new workspace the sub should be different and downstream services shouldn’t second guess that. Am I wrong about this?
I would be curious if any other Identity Providers that implement OAuth have mitigations for this issue. If you used sign in with Microsoft or Okta or PingFederate, would those provide different claims to the service?
Every provider has their own set of flaws.
Microsoft had a flaw for awhile where you could just change a user’s email to anything with no verification.. and if the SSO implementer was only checking the email field, you could impersonate anyone.
Ideally the trust with the SP would be using a cert. This requires the SP to implement this portion of the standard and the IT department to follow through with implementing a cert. The cert can be self-generated, so there isn't a cost for the cert itself.
Don't let domains you use lapse. Never get rid of a phone number.
A lot of these new TLDs have variable registration fees. They're going to price a lot of people out of renewals.
If oauth makes an authenticity claim, it should be true. Saying it's the same user when it's not is bad, clearly.
in other words: Google could make a more accurate authenticity claim than they currently do.
This problem would be worse without oauth, though, right? With plain email login, all they would need to do is "forgot password" and there wouldn't even be a way to tell.
in other words: Email login would never be able to make a more accurate authenticity claim.
We take the OAuth2 token and make a call to the People API [0] to get the Person resourceName. Not sure if this is more stable than "sub" but we completely ignore that.
[0] https://developers.google.com/people/api/rest/v1/people/get
I don't think this has anything to do with Google's OAuth. This issue is literally with every single expired domain name out there. All one has to do is register the expired domain and look at all the emails sent to that domain.
Granted, Google "could" do something, but I don't think it's Google's responsibility to police expired domain names. What am I missing here?
Google promises to use a different `sub` claim for every account, even if you reuse the domain name. However, according to the talk, the `sub` claim isn't stable in normal scenarios, so developers don't use that like they're supposed to.
Google should fix the `sub` problem if the problem is on their side (and not, for instance, related to user accounts impersonation or recreated user accounts, which are expected to fail this check). Everyone integrating with Google should use the `sub` claim like they're supposed to.
Of course this approach doesn't help if a domain admin can recover the original workspace account (rather than simply re-registering the domain with Google), but that can easily be solved by not having the domain admin accounts use the domain they're hosted on.
I think the problem is here in the doc:
I have worked with a few of these downstream providers to look for a solution. There is a documented unique user identifier (the sub claim) that could theoretically prevent this issue, but in practice, it's unreliable ... According to a staff engineer at a major tech company: “The sub claim changes in about 0.04% of logins from Log in with Google. For us, that's hundreds of users last week”.
The real solution is to use the sub claim, and not hd/email. If sub is unreliable, let's fix that. That said, I'm surprised sub is documented as unreliable here; I wonder if there is perhaps something else going on. I've done a lot of work with oauth and I haven't found sub to be so mutable, but then again, who knows maybe I just missed it?
And yeah, lapsed domains => domain takeover => email takeover and impersonation will always be a baseline problem, independent of oauth.
The email returned by OAuth providers is not always verified and thus cannot be trusted, anyone relies on the email as the ID is open to compromise. The sub claim should always be used as the ID, if you can prove that "the sub claim changes in about 0.04% of logins" then it's Google flaw and they should fix it, otherwise it isn't and there is no need to add another ID to the claim.
One of the unintended side effects of Noscript is this sort of unintended hilarity:
> These claims usually include:
> hd (hosted domain): Specifies the domain, e.g., example.com.
> email: The user's email address, e.g., [email protected].
The placeholder, hypothetical email address is being "protected" from web scraping.
In the context of an article on a security website about identities being spoofed through email accounts.
Well they have kind of known about this kind of problem for 14 years https://issuetracker.google.com/issues/35889152?pli=1 although this is not a domain issue it is one where people can't unlink from company project ownership.
Does anyone here know more about the unreliability of Google's sub claim? I am using it, and have never had issues with it, but this post seems to just dismiss it without more details.
When does Google change the subject of a token for a given user account?
Avoiding this scenario is why Google will renew all the domains for every startup it has ever acquired in perpetuity.
This is a stupid article. 1. It starts with “Millions of Americans”. How does this relate in any way to Americans? 2. Gmail is not the problem 3. Consider registering a domain for 10 years. When the startup fails, delete all dns entries
I agree. Additionally, the authors haven’t considered data deletion policies. Most SaaS providers delete data within 30 to 90 days after contract termination. It’s unlikely that data from 100,000 failed startups, each using 20 to 30 SaaS applications, still exists.
How many recovery processes would depend on email. And it coming from right domain? Or changes in DNS. Or some other factor that domain holder could do. I don't think possible alternative scenarios would go down too well.
So the "sub" claim does work but the "staff engineer at a major tech company" doesn't want to fix it because it happens too often?
Google may have quickly closed the original report because a company like that is soooo overwhelmed with garbage security reports and at first glance it is noise, not an actionable item to fix. there's a term for this that I can't quite recall off the top of my head. Bug Bounty Trolling?
If you think this is bad, wait until you move physical addresses and don’t notify all your creditors!
Literally ANYONE who moves into your old house will have access to your bank statements, credit card bills, tax forms, auto registration and who knows what else. And it will all come plaintext and unencrypted!
Phone numbers too. Ah, but it's illegal to open mail addressed to someone else. (Although what if you change your legal name to match???!) Perhaps we need an electronic messaging service backed by the USPS.
> Shmoocon talk accepted - Dec 9, 2024, Google re-opens issue - Dec 19, 2024
Wondering what the internal process there was at Google? Is someone's job to cruise around the various conferences and dev events to look for any Google mentions and report back. Did Google really consider it an issue or just wanted it avoid looking bad in public after the talk was accepted.
> Is someone's job to cruise around the various conferences and dev events to look for any Google mentions and report back.
Someone in Google's security team likely saw the talk, or the original blog post, and pushed internally for them to award something.
Yea. The leet amount of payout seems like this was someone saying that the classification was technically correct, but it's worth fixing either way and it'd be nice to pay an amount.
Yes, there are roles where people do nothing but search their names/products/etc specifically to see what people are saying. Hell, some people do it on themselves to the point there are phrases like "vanity search" or "egosurfing"
what a shitty headline and opening paragraph.
anyway, if you use SSO on a domain and then let the domain drop, you're an idiot and there will be bad consequences for your data.
Domains are for life, you can never let them lapse
Apparently Google removes pagerank from expired domains to prevent gamification of their Search Index.
Same should apply to OAuth, you would think.