Comment by apical_dendrite
2 months ago
I'm fairly certain this was an example of overfitting and Freedman's Paradox, not deliberate cheating.
Let's say you have a completely random data set. You generate a bunch of random variables x1 through xn and a random dependent variable y. Then you poke around and see whether any of the x variables look like they might predict y, so you pick those variables and try to build a model on them. What you end up with is a model where, according to the standard tests of statistical significance, some of the xs predict the y, even though all the data is completely random.
This is a much more likely explanation for why the answer weights on the biographical assessment were so weird than some conspiracy between the contractors who developed the test, the FAA staff, and the black employee organization.
They had a dataset that was very skewed because historically there have been very few black controllers, and so was very prone to overfitting. The FAA asked the contractor to use that dataset to build a test that would serve as a rough filter, screen in good candidates, and not show a disparate impact. The contractor delivered a test that fulfilled those criteria (at least in the technical sense that it passed statistical validation). Whether or not the test actually made any sense was not their department.
> I'm fairly certain this was an example of overfitting and Freedman's Paradox, not deliberate cheating.
The answers to the biographical questionnaire - which screened out 90% of applicants - were leaked to ethnic affinity groups. If a select group of being being provided with the correct answers isn't cheating, I don't know what is.
No, that's not what happened. The guy from the black affinity group CLAIMED that he knew the answers. But he's a completely unreliable source who was pretending to know things that he didn't actually know. He also claimed to have a list of magic buzzwords that would get your application moved to the top of the pile, but if you look at the list of magic buzzwords that he provided, it was just a list of dozens of generic action verbs like "make", "manage", "organize", "analyze", etc. from a resume writing book. I'm sure it's the same thing with the biographical assessment. He was just telling people what he THOUGHT were the right answers.
> As the hiring wave approached, some of Reilly’s friends in the program encouraged her to join the National Black Coalition of Federal Aviation Employees (NBCFAE), telling her it would help improve her chances of being hired. She signed up as the February wave started. Soon, though, she became uneasy with what the organization was doing, particularly after she and the rest of the group got a voice message from FAA employee Shelton Snow:
> “I know each of you are eager very eager to apply for this job vacancy announcement and trust after tonight you will be able to do so….there is some valuable pieces of information that I have taken a screen shot of and I am going to send that to you via email. Trust and believe it will be something you will appreciate to the utmost. Keep in mind we are trying to maximize your opportunities…I am going to send it out to each of you and as you progress through the stages refer to those images so you will know which icons you should select…I am about 99 point 99 percent sure that it is exactly how you need to answer each question in order to get through the first phase.”2
> The biographical questionnaire Snow referred to as the “first phase” was an unsupervised questionnaire candidates were expected to take at home. You can take a replica copy here. Questions were chosen and weighted bizarrely, with candidates able to answer “A” to all but one question to get through.
From the first article on The scandal: https://www.tracingwoodgrains.com/p/the-faas-hiring-scandal-...
> After the 2014 biographical questionnaire was released, Snow took it a step further. As Fox Business reported (related in Rojas v. FAA), he sent voice-mail messages to NBCFAE applicants, advising them on the specific answers they needed to enter into the Biographical Assessment to avoid failing, stating that he was "about 99 point 99 percent sure that it is exactly how you need to answer each question."
You can take the bigraphical questionnaire and see the question weightings here: https://kaisoapbox.com/projects/faa_biographical_assessment/
I'm not sure where you're getting the idea that this was just "buzzwords".
9 replies →
Have you looked at the info on the test here https://kaisoapbox.com/projects/faa_biographical_assessment/? (copied from another post)
To pass the test you have to click A on all 62 questions apart from question 16 where you have to click D to say your lowest grade in school was in history. The thing's a complete travesty.
You don't have to do that to pass the test. The max score possible is 179. One can pass the test without answering either of the worst subject questions "correctly."
Also answering answer A to 23 (>20 hours/week paid employment last year of college) would logically conflict with answering A to 56 (Did not attend college).
Did you get a different question/answer than I did? For me it showed science as the only correct answer awarding 15 points.
it's science for school and history for college
I agree that it seems likely that the weird questions and their weighting came from over-fitting as you describe. The cheating allegation though, from my reading, is that the "correct" answers were leaked and then disseminated by the leakee(s). (And that this was particularly impactful because it was unlikely that you would pass the overfit test otherwise.)
When I read the IG report and saw what the guy actually said (and that his list of secret buzzwords actually turned out to be a photocopy from a resume writing book) it was pretty clear that he was bullshitting and claiming that he had inside information about the process that he didn't actually have.
The investigation says the screenshots he talked about were for USA Jobs too, not the nonsense biographical test. Seems like it ought to be pretty easy to just check if NBCFAE members passed the that test at an unusually high rate.
> I'm fairly certain this was an example of overfitting and Freedman's Paradox, not deliberate cheating.
Buddy if someone tells you the answers to a multiple choice exam and you use them, then you've deliberately cheated. That's all there is to it.