Comment by Eridrus
8 hours ago
It seems like this is measuring algorithms against the disparate impact standard of all demographic groups needing to have the same aggregate results.
Which is extra weird because the samples to this are applications, not humans, so this is subject to bias in how people apply to these positions. So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
On top of all this, there isn't even really a claim that the algorithms are picking up on anything demographic related. One of the vendors they look at pymetrics, which makes players play abstract games and uses that to pre-screen people.
In the abstract, it makes sense that monocultures are problematic since ML bias alone (in the bias vs variance sense) would just randomly harm folks in a fairly persistent way. But it's also not immediately clear that this even applies to the pymetrics example where I think they have a large assortment of games they make people play for different positions?
It's also not clear that these systems breed monocultures if the inputs into them are firm/position-specific, e.g. job descriptions.
Though honestly I would be far more interested in the validity of these measures at predicting actual on the job measures like performance reviews, etc.
> So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
Your understanding appears to be incorrect.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
This seems to strongly support they argument that effectively a single AI makes a single decision for a candidate across "all" positions they apply for rather than independently assessing them for each position.
Essentially it's more or less saying they're is one hiring manager for the entire industry and if they have a random reason they don't like you, you won't be hired for any job in the industry.
There is a single evaluation function for the industry and if it puts you a negative for any reason in the model's distribution, every job that uses it will do so.
Those are somewhat separate concerns. You could have companies making independent hiring decisions while systematically discriminating against demographic groups, and you could also have companies all use the same system that systematically disadvantages certain individuals, but it's unrelated to their demographics and instead based on things like their resume not being easy to OCR.
In this case, the claim is that both are happening: companies aren't making decisions independently and they're doing so in a way that discriminates against certain demographics. But the evidence needed for each half of the claim is different.
> There is a single evaluation function for the industry
Could this be an opportunity in disguise? Somehow learn what this function wants, maximize it, then the entire industry opens up?
Apparently the key 'hack' is to present white or hispanic.
A few years ago the same professor did an analysis of racism in police traffic stops. Compared to this current study on job applications, the conclusions from the police study were more convincing because they actually observed the words spoken by the cops in each interaction.
https://www.pnas.org/doi/10.1073/pnas.1702413114
> if a demographic group is more likely to apply to some jobs they are not qualified for
Can you expand on this? How is a whole demographic group not qualified for jobs on wide spectrum? Is this about certain industries? Certain jobs? Certain groups?