Comment by progbits
4 days ago
I like how frank the report is, no sugarcoating. "We relied on manual error prone verification and made a mistake. We have to automate the process."
As ARIN block owner this situation is kinda scary but reading this actually makes me think it's less likely to happen again .
The road to automation is always full of outages.
I'm curious how these fellas took something like IP block allocation and turned it into an Excel based workflow.
“Workflow” is probably a bit generous to describe how they probably use Excel.
Having worked at a mom and pop ISP a couple of decades ago where we used Excel to track a lot of things, I can see how this might have happened.
To actually know who is allocated what is ultimately just a list.
And when there are only a few people who edit the list (and probably no more than 1 person at a time) you can get by with even a plain text file, but Excel is quite a bit nicer as you can do things like filtering and sorting easily, maybe even some formulas to help with things.
Building a program backed by a database might be nice, but hard to justify when the manual system has never been a problem before.
They’ve probably been thinking for a while they should, but it’s just never been enough of a pain point for them to invest the effort.
Looks like they see this incident as justification that they need a system with hard coded rules and constraints, no more manual checking.
It's ARIN, this is essentially their only job
The world's financial systems run on Excel, to a great extent.
I'm more surprised that a single person, apparently without seniority, could delete a block. IME deleting user data is usually a significant event; an IP block would especially be a big deal, especially for the IP block issuers. From the OP:
> RSD has implemented additional process controls that require a dual review for all ticketing type workflows that include a network delete.
> Only a limited set of experienced analysts are permitted to perform this function.
Great that they didn't blame the person who deleted it. ARIN seems to have put them in position where a failure was likely, eventually. Without any inside knowledge, I'd hope the culture would have any engineer leary about pressing that button without a second set of eyes reviewing it carefully and without clear authorization; I don't imagine they delete many blocks each day so it shouldn't interfere with productivity.
They've improved over the decades. At one point the authoritative database was a physical paper notebook.
You don't find this part
> We have to automate the process.
to be ominous?
I don’t. The report says part of this process relied on flat files and spreadsheets. Automating that with software is a good idea.
“Automate the process” doesn’t mean feeding everything to an LLM.
Certificate issuance was once only possible manually.
Domains too, well into the 90s.