Comment by jefftk
1 day ago
Reidentification via writing style is definitely possible, and I doubt the vendor will modify things in a way sufficient to handle that.
But I think this is a place where we should apply bounded distrust: there are lots of places where we should distrust Google, but reidentifying people in an explicitly deidentified dataset isn't one of them.
The re-identification is done by ML, at which point it's basically undetectable. If you give all this day to, say, and LLM, and the LLM also has your blog, the identification is embedded in the weights.
I can ask "Tell me about Person A's experience with airlines" and it will tell me. Or I can ask more generally, "knowing your knowledge, derive a no-fly list", and then it's likely Person A will be on it.
Based on their trackrecord, That's definitely a concern. I don't really understand on which basis you conclude 'isn't one of them' . 'Don't be evil' ? :-P
I think this is the kind of place where applying bounded distrust is critical: it's not whether we trust Google overall, it's about figuring out what sorts of statements we should expect to effectively bind companies and in what ways.
For example, I think a pretty worrying outcome here is that deidentification is imperfect (not surprising), the data is fed into model training, and then the model makes identity-dependent inferences. Since no one tried to break deidentification, it's within what I'd expect from the company. (And, to be clear, is bad.)
On the other hand, intentional reidentification to work around contractual deidentification to "a sell a product to the airlines that offered to keep annoying people like me from purchasing flights" is the kind of thing that would make Google's lawyers terrified, so we should not expect it.
If you look at how this worked with DoubleClick, Fitbit, etc there were initially barriers to linking data but the mechanism for unlinking was updated agreements with people who had ongoing interaction with the continuing entity. That's not the situation with the Spirit data.
The closest I'd expect to see for a "keep annoying people from purchasing flights" situation is not a list of troublemakers but a model that's very good at scoring future communications from customers, and has learned to distinguish profitable vs unprofitable customers. This is well within what I'd expect from companies, and doesn't require any reidentification.
I don't think google will explicitly 'deintificate' the data and sell it 'deidentified'. It's more that that kinda data can't really be anonymized. And while I trust google to pay lipservice to the 'anonymizedness' , I don't trust them at all that they won't just feed all that data into their AI training as-is, letting the deidentification happen when someone writes a smart prompt for gemini. As long as they have plausible deniability they don't care.
I don't trust google to do anything out of goodness, they'll do anything they can get away with. Same as all the other big tech firms. Once a firm gets too big it stops having morals.