What they are doing here is selecting a number of events (particle collisions) and comparing with a simulation of the standard model (sans Higgs or whatever they are looking for). If there is a significant excess, then you found something.
One scenario, and I think what people were paranoid about, is that you could make your cuts really arbitrary, and select a region where you by accident have too few simulation events and too many data events - say "Select all events where the first muon has momentum > 15.6 GeV, but not between 20.1 GeV and 20.3 GeV, unless there are 3 jets, in which case the selection is...". You just try out so many regions until you find something. For this reason you need to keep every step of your analysis justifyable, and a BDT is somewhat more explainable than a neural net (it's essentially just a trained list of simple cuts). Another thing that is frequently done is a blind analysis, where you only look at the real data when everything is finalized and locked in.
There are other issues beyond cherry picking that determine which methods you can use. E.g. the simulation doesn't reflect all correlations between variables, and it doesn't simulate the hardware triggers perfectly - so your simulations might behave different wrt. certain cuts than your data.
Nevertheless, it is not like you are banned-banned from trying different methods. When I was in particle physics (ATLAS and D0), we tried all kinds of creative things including NNs. But you have to convince hundreds of collaborators to put their names under your paper and you have to pass internal peer-review, which can feel a bit conservative and frustrating.
This sounds quite frustrating. I'm not in particle physics, I'm just a lowly SWE, but my workplace doesn't have a problem with me burning $30k of tokens on some one off idea.
Why did CERN require so much selectivity? Was it a lack of funding? Are particle collisions particularly expensive? Are you competing for "collider time"? I am curious what the limiting factor was.
I think I would lose my mind if I needed hundreds of collaborators to support an experiment.
What they are doing here is selecting a number of events (particle collisions) and comparing with a simulation of the standard model (sans Higgs or whatever they are looking for). If there is a significant excess, then you found something.
One scenario, and I think what people were paranoid about, is that you could make your cuts really arbitrary, and select a region where you by accident have too few simulation events and too many data events - say "Select all events where the first muon has momentum > 15.6 GeV, but not between 20.1 GeV and 20.3 GeV, unless there are 3 jets, in which case the selection is...". You just try out so many regions until you find something. For this reason you need to keep every step of your analysis justifyable, and a BDT is somewhat more explainable than a neural net (it's essentially just a trained list of simple cuts). Another thing that is frequently done is a blind analysis, where you only look at the real data when everything is finalized and locked in.
There are other issues beyond cherry picking that determine which methods you can use. E.g. the simulation doesn't reflect all correlations between variables, and it doesn't simulate the hardware triggers perfectly - so your simulations might behave different wrt. certain cuts than your data.
Nevertheless, it is not like you are banned-banned from trying different methods. When I was in particle physics (ATLAS and D0), we tried all kinds of creative things including NNs. But you have to convince hundreds of collaborators to put their names under your paper and you have to pass internal peer-review, which can feel a bit conservative and frustrating.
This sounds quite frustrating. I'm not in particle physics, I'm just a lowly SWE, but my workplace doesn't have a problem with me burning $30k of tokens on some one off idea.
Why did CERN require so much selectivity? Was it a lack of funding? Are particle collisions particularly expensive? Are you competing for "collider time"? I am curious what the limiting factor was.
I think I would lose my mind if I needed hundreds of collaborators to support an experiment.
They once spent 2 months to be allowed to use imaginary numbers
Say more please!