Comment by stuaxo
17 hours ago
"In a 2023 blog post, external, Palantir described the challenge of combining data from multiple government systems containing tens of thousands of visa applications and hundreds of thousands of accommodation offers."
This is the kind of thing GDS and other Civil Service departments build all the time, its a completely standard kind of challenge that a small team of Devs (+ supporting staff) from a departments DDAT department does day in and day out.
The output will be open source by default and use existing standards.
Yeah, this exactly. "multiple government systems", "tens of thousands", "hundreds of thousands" is the typical "part-time allocation for four people in an office" government project. This should have a budget in the low hundreds of thousands of £ at most.
Exactly. And more importantly is the fact that when done inhouse the incentives are different than when done by a contractor. The latter has an incentive to keep the client dependent.
Hundreds of thousands of documents is small enough that you can feasibly run a pen and paper office handling them. Especially since most of them do not cross-reference eachother (family applications do, but unrelated families have no such links).
That America's brightest tech minds can't solve this problem is embarrassing. (Never mind the baggage of giving a foreign, potentially adversarial nation access to something as sensitive as residency and visa information.)
This article is about the UK.
I assumed that when the GP said the UK was "giving a foreign, potentially adversarial nation access" the GP meant that the US is that "foreign, potentially adversarial nation"
I can't believe that in our timeline Europe has to think like this, but here we are.
I'm well aware.
Note that Palantir is an American company that failed to solve this problem well, and introduces an adversarial risk to the UK.
Could probably be moderately complex excel sheet. Well, hopefully not but keeping that one guy that know how it works is still cheaper than Palantir!
They did that with the COVID19 tracker.
And ran out of rows
https://www.bbc.co.uk/news/technology-54423988
The core data platform for NHS Test and Trace was not Excel based at all. It was a reasonably simple but solid AWS setup using S3, Glue and the smallest Redshift instance at the time, on top of which were Tableau/Quicksight/PowerBI dashboards. Some organisations insisted on enabling an "export to CSV" feature which was...not a good idea for so many reasons, and Public Health England (PHE) in the article found out one of those reasons the hard way.