Comment by godelski

Datasets are actually pretty hard to create. You can see several papers specifically studying ImageNet[0] including some on fairness and how labels matter. There's also Google's famous private JFT-300M dataset[1]. JFT was specifically made with heavy tails in the distribution to better help study these areas, which is specifically the problem we're interested with here and one that is not solved in ML. Even with more uniform datasets like CIFAR there are still many features that are noisy in the latent space. This is often one of the issues with doing facial recognition and why there's issues with people with darker skin. Even if you have the same number of dark skinned people as light skinned you may be ignoring the fact that cameras often do not have high dynamic ranges and so albedo and that dynamic range play a bigger role that simply "1M white people and 1M black people". There's tons of effects like this that add up quickly (this is just an easy to understand example and one that's more near the public discourse). You can think back to how Google's image search at one point showed black people if you searched gorilla. On one hand you can think "oh got a dark color humanoid" or you can think "oh no... dear god...". That's not a mistake you want to make, even if we understand why the model made it. It is also hard to find these mistakes, especially because the specifics of them aren't shared universally across cultures because this mistake has to do with historical context.

This is still an unsolved problem in ML. Not only do we have dataset biases (as discussed above) but models can also exaggerate these biases. So even if you get a perfectly distributed dataset your model can still introduce problems.

But in either case, we don't have the same concerns in research as we have in production. While there are people researching these topics most of us are still trying to just get good at dealing with large data (and tails) in the first place. Right now the popular paradigm is "throw more data at the model." There are nuances and opinions to this why this may not be the best strategy and why we should be focusing on other aspects (opinions being key here).

Either way, "using publicly available datasets" is an answer that suggests 1) they might not understand these issues and 2) the model is going to have a ton of bias because they're just using off the shelf models. I want some confidence that these people actually understand ML instead of throwing a neural net at the problem and hitting go.

> I'm curious how to enable models without giving away my data.

Our best guess right now is homomorphic encryption. But right now this is really slow and not as accurate. There's federated learning but this has issues too. Remember, we can often reconstruct images from the dataset if we have the trained model[2]. You'll see in this reference that while the reconstructions aren't perfect, they are more than satisfactory. So right now we should probably rule out federated learning.

> Open source models owned by society?

Actually models aren't the big issue. Google and Facebook have no problem sharing their models because that isn't their secret sauce. The secret sauce is the data (like Google's proprietary JFT-300M) and the training methods (though most of the training methods are public as well as few are able to actually reproduce due to not having millions of dollars in compute).

I hope this accurately answers your questions and further expands on the reasoning behind my concerns (and specifically why I don't think the responses to me are sufficient).

[0] https://image-net.org/about.php

[1] https://arxiv.org/abs/1707.02968 (personally it bugs me that this dataset is proprietary and used in their research. Considering how datasets can allow for gaming the system I think this is harmful to the research space. We shouldn't have to just trust them. I don't think Google is being nefarious, but that's 300M images and mistakes are pretty easy to make).

[2] https://arxiv.org/abs/2003.14053