Comment by freddealmeida
19 hours ago
Flock did not put the woman in jail. The police and DA put her in Jail. This is not a failure of the technology (do they have confidence levels?). This is a failure of how police are outsourcing their reasoning and decision making to an AI camera service that seems to fail at the tail.
But Flock gives police overconfidence to commit actions instead of having due diligence. It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
This is also comparable to self-driving car: Waymo's system has achieve the level of "less error than human on average" long time ago, but they are still only doing slow expansion. Why? Because when AI is involved, human naturally expect them to be almost 100% correct, not just barely better than human, otherwise why bother use AI (and all the troubles to legalize for AI) in the first place? So in this case, I guess the police just blindly trust the AI being 99.99% correct.
Flock isn’t an “AI” though? It doesn’t put together a case and say “yes, this is the criminal”. It’s not a crime-solving AI.
It’s a camera system, and data can be queried by police. They have to search for something. So the cops searched for a vehicle matching the description in the general area, seemed to take the first one, and called it a day.
That’s not an AI-uncanny-valley. That’s the equivalent of (pre-flock/cameras) looking at vehicle registration list and assuming the one at the top of the list must be it.
The automation-trust is a legitimate concern but it’s not applicable here.
I don't know the details on how Flock is internally designed, but I suppose AI is at least involved in the image recognition and correlation, at least to this scale. I also don't know exactly why Flock misidentifies Issacs' car in this case, because I don't find it in the article or the interview, but I assume it is due to a mislabeling of the plate number or the car damage imagery, which is related to AI. Correct me if I'm wrong.
1 reply →
Correct, but what's more, there is a lack of accountability issue that's part of it. When people are not held accountable for the screw ups, then processes don't or are very slow to change. That's how you can just call it a day, regardless of the harm done to innocent citizens.
Ahem, reality would like a word about flock not being AI.
https://cybernews.com/privacy/flock-ai-os-investigate/
1 reply →
Software developers approving AI PRs without reading them: This is the way the world works now. Get with the times.
Police approving AI PRs without reading them: This is the way the world works now. Get with the times.
> It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
Yes, but it is still your fault. “I am dumb and I was deceived” is no valid defense, least of all for a police officer.
I'm not saying the police has no fault. Using AI assist without secondary verification is never good idea, at least not in 2026. But you can't argue Flock the technology itself, and especially Flock the company, are totally innocent, just like you can't say the "guns are just tools like hammer, it only depends on who uses them", and the gun industry.
4 replies →
> It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him
Closer analogy: The dumbest student in a class tells you 2 + 2 = 5 and you believe them.
Even more apt analogy: You're the dumbest student in the class, the second-dumbest student tells you 2 + 2 = 5, but you lack the critical thinking skills necessary to know whether that student is smart or not so you just believe them.
Cops regularly frame people and get caught only because of cameras that the cops are too stupid/incompetent to disable or avoid. This goes both ways.
> It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
It’s more like normal people don’t know that 97.8% accurate isn’t 100% - despite how good of a score that would be on a test.
Law enforcement is predominantly made up of normal people.
You might see where this is going.
I would say US law enforcement are made up of mostly abnormal people with lower intellegences, higher aggressivness, and a propensity to lie and deceive. They tend to push actual normal people out for not playing ball with their own corruption.
The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
Which makes Flock cameras even worse right? It provides false confidence to law enforcement to make decisions that can change someone's life for worse. I don't understand why people hate the NSA spying on them and yet support Flock cameras.
The question is whether Flock does provide false confidence to law enforcement, or whether it provides a media hook to cover the ways in which bad police departments routinely misbehave.
A lot of people coming to this story seem to have the idea that it's unthinkable for someone to spend 13 days in jail when they have clear evidence they're innocent, so there must be some novel explanation for how this could happen. But as any criminal defense lawyer will tell you it's very thinkable. She was ordered to report to jail on a Friday afternoon, the standard stratagem cops use to ensure people they don't like spend at least a weekend behind bars. She was denied bail, even though she made no move to flee in the months where she knew the police thought she did it, because the courts in many places view mandatory pretrial detention as a punishment for presumptively guilty people.
If you give a toddler a gun and it shoots someone, it's the toddler's fault, not the gun's right? But you don't give toddlers guns. You don't say "well toddlers just need to be more careful".
Are toddlers being hired to do government work nowadays, including as police officers?
As far as reasoning abilities goes it would seem so.
Worse, fully grown incompetent and or malicious people!
The police is of course responsible, but you can't wave the responsibility of flock away when it's inconvenient for them
How is Flock responsible in this case?? There may be cases where Flock is responsible, but I don't see it here.
Would depend entirely how Flock is marketed and represented to the people who buy them and whatever training Flock provides.
The technology is not a neutral thing here. Automation bias¹ exists and is especially problematic in this context.
And if your country happens to have a police force that is historically bad at its job, giving them that power seems like a really stupid idea.
¹: https://en.wikipedia.org/wiki/Automation_bias
I very much agree with what you say on automation bias.
Certain technology just has too much potential to be misused / people repeatedly prove they can't use it properly. It also comes with risks of making authoritarian takeover easier.
However, here, it seems they used Flock a bit like a regular ANPR. The actions they took independent of Flock show there are better stories to use to criticise this technology.
While you are technically correct (which we all know from Futurama is “the best kind”), that is neither helpful nor consolation to the affected, and I wish these kinds of comments didn’t dominate the top spot in HN discussions. They’re the “guns don’t kill people, people kill people” of tech.
Yes, the system is broken even before we get to Flock (or whatever tech) but Flock makes it worse. If a bad tool is badly used to cause harm, it’s fair to criticise its existence. You can’t fix police abuse with a snap of the finger, so you need to start but taking away the tools which facilitate it.
If it were your ass which had been thrown in solitary, would you be pleading with your lawyer to “go easy on Flock. Sure, they allowed a major disruption and trauma into my life, but they’re just a tool”?
Then the right fix is to fix the system, e.g. imposing accountability. Removing tools is just a band aid. Why is the tool bad? A tool does what it's designed to do. You can use it properly or improperly. There should be no conflation of regular civilians with supposedly trained government officers.
They can have their tools back after they show they can use them responcibly.
If a kid has shown themselves to be repeatedly irresponsible, when they ask for a circular saw you don't just give it to them and turn away. You make them explain what it is for and ensure they know how to safely use it. And if they start swinging it around wildly in opposition to what you said, you take it away, they are obviously incapable of using it right.
Yes, of course the better fix is to fix the system but one thing at a time. You’re on a giant boat that has left port years ago and a leak just sprung; you should absolutely get back to port and fix the processes that allowed the leak to open but first you have to close it.
If someone released a hammer that when held a certain way would cause your hand to explode and it kept happening over and over, the solution wouldn’t be just to tell people to hold it better, it would also be to recall the tool and fix it so the issue were less likely or impossible to occur. And if it can’t be done, then maybe the hammer shouldn’t be released again.
This “it’s just a tool, the proper solution is to rework civilisation” defence is frankly as tired as it is ignorant. The way you fix the system as a whole is via a lot of small fixes.
2 replies →
The people who say there are problems with Flock are perceived as excusing the human police officers.
The people who say there are problems with the human police officers are perceived as excusing Flock.
Who is responsible for the area inside a square, the width or the height? It is not an additive relationship.
True, this isn't about flock and it's not even about outsourcing reasoning.
This is the good old police incompetence that you come across once in a while and shake your head.
Where detectives lock on one angle of a case and completely refuse to entertain any other possible explanations
of course it's the absence of common sense in the law enforcement but on the other hand this technology is thought to be faultless and it takes resource to put it right
Indeed. It’s insane to see these arguments implying the contrary. The institution committing the violence is the one responsible for the outcome.
If a random person walks into a police station and accuses a random stranger of murder, and the police put that stranger in jail based on that alone, the main problem is not the accuser.
But in this case, it wasn't a random person, it was a police-vetted system. Of course the authorities are to blame, but Flock hold a lot of responsibility, too.
It shouldn’t have been vetted then. It’s again the fault of authorities who approved that.
A scientist who uses an LLM to write a paper remains responsible for its contents. And authorities that outsource police work remain responsible for what comes from that.
1 reply →
It is always both. A product that can be abused to this extent is a bad product.
I have never released a single product that has seen people die through common user misuse.
The standards for solutions shouldn't get _lower_ as the stakes get _higher_.
If a private company can puts hundreds of tons of payload into orbit without ever causing someone to accidentally be imprisoned a company that sells cameras on a stick shouldn't either.