The phone has intelligence to detect things that look like landmarks, and does cropping/normalization and converts to a mathematical form.
Apple has a database trained on multiple photos of each landmark (or part of a landmark), to give a likelihood of a match.
Homomorphic encryption means that the encrypted mathematical form of a potential landmark from the phone can be applied to the encrypted set of landmark data, to get an encrypted result set.
The phone can then decrypt this and see the result of the query. But anyone else sees this as noise being translated to new noise, including Apple's server.
The justification for this approach is storage - the data set of landmarks can only get larger as the data set gets more comprehensive. Imagine trying to match photos for inside castles, cathedrals and museums as examples.
I don't completely understand the maths of how this works, but no, they don't.
Here's a theoretical way I wrote in another comment:
> I think they have more efficient ways, but theoretically what you could do is apply each row in your database to this encrypted value, in such a way that the encrypted value becomes the name of the POI of the best match, or otherwise junk is appended (completely changing the encrypted value) Again, the server has not read the encrypted value, it does not know which row won out. Only the client will know when it decrypts the new value.
They do something like this, using homomorphic encryption. Whatever they do, there is no doubt they incur serious performance hits.
The phone has intelligence to detect things that look like landmarks, and does cropping/normalization and converts to a mathematical form.
Apple has a database trained on multiple photos of each landmark (or part of a landmark), to give a likelihood of a match.
Homomorphic encryption means that the encrypted mathematical form of a potential landmark from the phone can be applied to the encrypted set of landmark data, to get an encrypted result set.
The phone can then decrypt this and see the result of the query. But anyone else sees this as noise being translated to new noise, including Apple's server.
The justification for this approach is storage - the data set of landmarks can only get larger as the data set gets more comprehensive. Imagine trying to match photos for inside castles, cathedrals and museums as examples.
> get an encrypted result set.
seems to me at that point, the server knows what segment of the overall dataset is being returned.
I don't completely understand the maths of how this works, but no, they don't.
Here's a theoretical way I wrote in another comment:
> I think they have more efficient ways, but theoretically what you could do is apply each row in your database to this encrypted value, in such a way that the encrypted value becomes the name of the POI of the best match, or otherwise junk is appended (completely changing the encrypted value) Again, the server has not read the encrypted value, it does not know which row won out. Only the client will know when it decrypts the new value.
They do something like this, using homomorphic encryption. Whatever they do, there is no doubt they incur serious performance hits.
You may also be interested: https://arxiv.org/abs/2406.06761
1 reply →