Comment by SilverRed
4 years ago
>I would add that people have generated legal images that match the hashes.
That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
The NCMEC hash list is private, and adversarial attacks require running gradient descent and being able to generate a hash value for arbitrary input.
At least one of these two things must be true: either Apple is going to upload hashes of every image on your device to someone else's server, or the database of hashes will be available somehow to your device.
Turns out there's a third option I wasn't thinking about: private set intersection.
https://news.ycombinator.com/item?id=28097683
Is it possible to narrow in on a hash using gradient descent? You can correlate distance between inputs to distance between hashes somehow?
Replying to my own question since I can’t edit anymore: it turns out “perceptual hashing,” which I didn’t know much about, has exactly this property, that small changes in the input result in small changes in the output.
Might be hard if they use a huge hash.
One thing to note is these are not typical cryptographic hashes because they have to be able to find recompressed/cropped/edited versions as well. Perhaps a hash is not an accurate way to describe it.
There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.