Comment by FigurativeVoid
4 months ago
At my first gig, I had "god" level access to our production database.
All I learned is that nobody should have this level of access unless it is some sort of temporary break glass situation. It is extremely dangerous and even experienced engineers can cause irreparable data loss or some other bad outcome. In our case, some engineer accidentally sent around 10,000 invoices to customers that shouldn't have gotten them.
There are far better data access patterns. In the case of US gov data, I don't see why the DOGE team would need anything more than a read replica to query. It could even be obfuscated in some way to protect citizens' identities.
Ah, I remember a time 30 years ago when I logged accidentally into the PROD database (forgot to add the suffix "1" to the connection ID), thinking it was a Dev instance, and then issued a "truncate table CUSTOMERS"... the reaction came within 75 seconds - and restore from backing took several hours.
That's not bad. I wonder how many companies would not be able to recover from such an event in days / weeks / ever.
That's when you pray that the engineering team didn't fix the replication lag.
I've worked with older governmental systems, and chances are they are running a wide variety of systems, some of which, the oldest and most critical, are probably written in COBOL running on IBM mainframe hardware. In those environments, there is no real distinction between "database" and "application". COBOL systems are very file- and batch-oriented, and are "monolithic" in the extremist sense. The technology itself makes it impossible to give read only access to such systems.
> The technology itself makes it impossible to give read only access to such systems.
This isn't true. Mainframe COBOL systems commonly store data in VSAM files, or DB2, or IMS, or sometimes some more obscure non-IBM database (e.g. CA/Broadcom's Datacom/DB or IDMS, or Software AG's ADABAS). But whichever one they use, there are multiple ways of granting read-only access.
For example, if it is VSAM, you can configure RACF (or TopSecret or ACF2) to allow an account read (but not write) permission to those VSAM datasets. Or, you can stick DB2 in front of VSAM (on DB2 for z/OS, CREATE TABLE can refer to a pre-existing VSAM file, and make it look like a database table), and then you can have a readonly account in DB2 to give you access to that database schema. Or, there's a lot of other ways to "skin this cat", depending on exactly how the legacy app is designed, and exactly how it stores data. But, probably this is already implemented – most of these apps have read-only access for export into BI systems or whatever – and if it happens for whatever reason not to be, setting it up should only be a modest amount of work, not some multiyear megaproject.
>Or, there's a lot of other ways to "skin this cat", depending on exactly how the legacy app is designed, and exactly how it stores data. But, probably this is already implemented
Given that neither of us knows the actual systems in question, what is more likely, that it's a well-designed system or one that has organically accreted over time? It seems like you tend to believe the former, and I the latter. I suppose my view is based on the fact that, like in statmech, you enumerate all possible systems that can do a particular job, the vast majority of those solutions will not have any organizing principle and will not be amenable to surgical analysis or change.
5 replies →
This implies good faith actor, which is not the case.
You can absolutely give read only access in COBOL systems. That's just lazy administration and IT security on a shoestring budget.
These old systems need to be upgraded but Congress never approves the financing or execution of it because they're too divided and won't increase taxes on anyone especially the rich and giant corporations to do it.
[flagged]
You know that annoying thing where someone joins a new team, looks around, declares all their friction points to be easily solvable, dives in & starts making changes, and turns out to make a big giant mess?
And the reason is they don't understand the specific domain & context well enough to know what the actual hard problems are. Instead they're just pattern matching to things they do know and extrapolating. And it usually doesn't go well.
Dealing with a system that's replicating 50 years of regulatory rules is going to be that times infinity.
1 reply →
>They don't even know how to build a website that works.
What percentage of people who know how to make a "website" do you think could make an automated tax system?
>the tech industry has been the beating heart of this country
Agriculture? Construction? The heart means something without which you can't function. How did people in the 1950s survive?
1 reply →
The USG does in fact know how to build a website and it is intellectually lazy (so very lazy) to suggest otherwise. A high profile illustration of this is login.gov, which is SSO used across USG agencies. It's not possible to take a comment like this seriously, at all.
Elon Musk is also not an auditor. DOGE is not an auditing entity. You bring in accountants to audit. These are 20 y/o something programmers. How DOGE has been operating has been completely opaque and this lack of transparency just plays to the point that what someone says their goals are and what their actual goals are are not mutually exclusive, so no, Elon Musk shouldn't be allowed anywhere near these systems.
4 replies →
"fixing the government" in this case seems to mean "destroy the government" for somewhat hidden purposes.
13 replies →
DOGE literally took over the agency that competently modernized and integrated US gov technology (United States Digital Service), gutted it, and is now using that agency's pretense of needing access to data to now pilfer citizens' private information and grossly violate the constitutional separation of powers.
This is the mechanism by which this administrative coup (declared here in https://www.whitehouse.gov/presidential-actions/2025/02/ensu...) is being enacted. None of this is legal or constitutional in any way.
The rule of law is not a partisan issue nor a matter of "government efficiency". Those who aid this coup should be considered traitors.
If it ain’t broke.. move fast and break things?
All I've seen about this DOGE stuff is negativity based on hypotheticals, this is the first optimistic hypothetical I've seen so far.
It's an interesting point. As a thought exercise, tech is absolutely the core of modern America, #1 export (I assume) and a key market. Private sector influence probably can give huge amounts of low hanging fruit.
I think peoples main concerns stem from not trusting Trump (which seems odd given he's a second term president, he is objectively wanted) and not trusting Musk (which is probably fair, he's publicly and openly an arsehole).
Speed probably concerns people too, however "move fast and break things" is a pretty fundamental American tech mantra, so entirely unsurprising and usually effective.
7 replies →
> That's why all this stuff is backed up to an iron mountain.
When one of your threat vectors is a massive ball of nuclear fire right on top of the federal government in DC, your offsite backup policy is going to be absurd overkill by the standards of any other organization on this planet. That doesn't mean it's flawed.
> ...many of the people in charge don't even know how to use a website. Now for the first time, tech industry people have the opportunity to help run these computer systems, and you're afraid they're the ones who'll be incompetent and accidentally break everything?
Are you honestly suggesting that the people who built these systems, maintained them, and updated them to reflect often significant changes in rules and regulations over the course of decades somehow don't know how those systems work? If they were so damned clueless, those COBOL systems would have sputtered out and died decades ago. The fact that they've continued to run for all this time is practically prima facie evidence that the system works just fine by industry standards for that kind of legacy code.
No doubt there's plenty of stuff buried in the codebase that bugs the hell out of the developers working on it, but you get that with any complex legacy code. It's the nature of the beast. Do you think there's nothing in Google's monorepo that some of their engineers don't quite like but doesn't rise to a big enough issue to warrant refactoring right now? Any other FAANG company? Or large tech company in general?
You're writing as though a bunch of junior developers--and that describes pretty much all of the publicly known DOGE employees so far--are wizards who can just waltz right in and magic up a better solution just because they're from the "tech industry."
Setting aside the unlikely chances that those juniors--no matter how skilled or talented--have any experience with COBOL, mainframes, or even just decades-old legacy code, is anyone going to suggest that something like the federal government's payment system isn't defined by an immense amount of complex business logic so as to comply with legislative requirements? It's not something you just start playing around with.
I can't think of any tech company that would take a junior developer, toss them overboard in the middle of the freezing Atlantic, grant them sudo access, and tell them to do whatever the hell they want with critical systems before they drown and--somehow--take the ship with them. Worse yet, those juniors were chosen for ideology fervor and/or purity, so what happens when the normal review processes and experienced senior developers are pushed aside because they're in the way and part of the "deep state conspiracy" that doesn't want them to "[fix] the government" as you put it?
Not only is that a recipe for disaster for the company itself, it's a damned good way to take an otherwise talented junior developer and permanently ruin them. Instead of mentoring them so they can work well as part of a team, you're basically creating a toxic working environment that's going to turn them all feral. By the time they crawl out the other side and the public hears all about what they've been up to, what company is going to be stupid enough to a developer with "DOGE" on their resume? Beyond that, you're conflating a whole bunch of different issues here with federal software contracts and IT, while putting the tech industry on a really peculiar pedestal.
Besides, if the goal is to discover waste/fraud/abuse, the obvious answer is to hire a bunch of forensic accountants and let them dig into everything. Those are the people who actually find that kind of stuff, and they're incredibly skilled at their job. If it's there, given the time, they'll find it. But it's a slow-going process, so we instead see a bunch of engineers focusing on random transactions so they can ask themselves (1) "do I like that one?" and (2) "do I think it's legitimate?" because it's faster.
That's not exactly how you fix anything, least of all a country.
1 reply →
> What's with you people
Right?
> For decades the tech industry has been the beating heart of this country that's kept the American dream alive
By "tech industry" do you mean the consumer surveillance industry? Maybe your vision of the American dream involves inescapable corporate control, but mine certainly doesn't!
2 replies →
Never mind the direct risks, if you have "god mode" to basically any government thing, you instantly become the target of foreign intel/military operations. You can bet good money that there are entire teams, if not divisions, working around the clock to exploit this situation.
I can imagine Chinese and Russian hackers laughing at the DOGE l33t hackers.
And if I was advising the Ukranians I'd tell them to try to exploit it too, hey, if you're fighting 2 superpowers with another 1 quietly backing the fight against you, you need all the help you can get.
It is literally why we never log in as root.
I don't know an admin who hasn't, on multiple occasions, unintentionally caused irreparable damage. It is easy to do even with the best of intentions and with extreme levels of care. Any one trying to rush through a dragon's den is only going to get burned. Considering how many dragons' dens they are running into, I do not question "if" damage has been done, but "what".
I remember having some kind of C programming bug where output filenames got scrambled (string memory error probably). And output files in the same folder as the source code.
That seems innocuous, but remember then some of the output files might have the character "?" or even "*". So imagine trying to remove these files and going an asterisk too far. All gone!
I've had a company give me full admin access to their cloud account. Thankfully, I learned the lesson earlier in my career and immediately created myself of more mundane user. Break glass access is important, but definitely not as the usual level of access.
> I don't see why the DOGE team would need anything more than a read replica to query.
They shouldn't need more than limited read access. The fact that they have more access, very likely demanded and not accidentally given, is due to their intent to do more than simply query data.
I loathe working places where they just give you all the permissions because it's "easier". One risk is if something does happen, and they don't have exceptional tracing and logging, (and let's be honest, at an organization sloppy enough to hand out privileges like candy, what's the chance of that?) it's difficult or impossible to pin down the source to any individual. As a result, both responsibility and suspicion is diffuse.
The appropriate restrictions are relative to the size and momentum of the organization. It's easy to spend months setting up safeguards rather than working on product development that won't proportionally return.
Of course, this involves being honest with yourself about risk and reward, and we all have implicit incentives to disregard the risk until we get burned and learn to factor that in.
I have so many horror stories from there.
When they did decide to lock down the database, the DB admin only locked in down in the sql server client most people used. If you used some other client, you still had access. _sigh_
What DB system operates that way, that's nuts.
My favorite security anti-pattern! Locking the main doors while leaving all the windows wide open.
It's not just about the risk. It signifies that you're not dealing with an experienced database administration staff. (At a startup that might just mean one guy, but that's better than zero.
A second thought. It leads to lazy application development. Whenever you have production intervention that happens more than a few times, you should just make a feature that does it safely via application code.
I've definitely worked in places where "Move fast and break things" tended to focus on breaking things. There would be bugs that we didn't fix because "We can just fix the database when it happens." It would take 2hours to fix a bug that would cause of 10's of hours of weekly support request, but the focus would always be on building new features, of which 10% got any real usage.
I agree. Good access controls and being prevented from accessing things that I don't need access to protect me as an employee just as much as the data itself.
Meta completely restricted graph data access to requiring a specific business purpose and managerial approval tied an articulable, concrete task need.
Why should they even have read access? They're not a legal government institution, and they're being led by a private citizen that's not been elected or appointed by Congress to access our data in agencies that were made by Congress under particular rules to keep these kinds of snoops out.
Ultimately someone has root permissions. Re: federal agencies, in the United States, that someone is clearly, constitutionally, the President. Article II of the constitution vests all power of the executive in the person of the President. The President has authority to appoint agents. That same article _does also_ say the President has to "take Care that the Laws be faithfully executed", but the "Care" there is highly debated. But the idea that the President doesn't have the right to appoint Musk to get root access to federal agencies seems legally incorrect.
I'm not make a value judgement on this, it's just how it is. At a startup, the founder ultimately has root access to the database, no matter what the technical controls.
Now, maybe it's stupid, and maybe it should be some other way, but to my mind the other way is that Congress gets together and writes a law saying "the executive cannot get root access to X, Y, Z". In absence of that law, the executive can do whatever they want.
Not to be THAT GUY, but "an append-only database which cannot be modified by anyone" is something HN has spent the past 10 years saying is completely useless...
The power rests with the office. There is an important but nuanced distinction there.
And Trump can launch the nukes to blow up the world too... but building a system where he can just click a button to do so would be idiotic. Same idea with giving godmode to the guy who thinks carrying a sink and saying "Let that sink in" is hilariously clever.
There's a good balance between preventing accidents and reducing friction.
One person having "god-mode" access isn't usually that terrible.