Comment by csomar
1 day ago
Everyone was locked out in a building am staying at (40 something stories) for several hours. When I asked the concierge if I can have a look at the system, it turns out they had none. The whole thing communicated with AWS for some subscription SaaS that provided them with a front-end to register/block cards. And every tap anywhere (elevators/doors/locks) in the building communicated back with this system hosted on AWS. Absolute nightmare.
I wonder what happened to the building when us-east-1 went down.
As the parent said: “Everyone was locked out in a building am staying at (40 something stories) for several hours.”
Now I am waiting for time when they move us-east-1 physical security to run in us-east-1... Thus locking themselves out when needing some physical intervention on servers to get backup.
Facebook already got bit by this when their BGP setup pooped its pants on Oct 4, 2021
They could ask facebook for advice.
https://skeptics.stackexchange.com/questions/52448/were-face...
I wonder what happened to the building when the internet went down. How do you get into the room to reboot the router?
There’s usually a back door with a physical key. The problem can be getting ahold of one of the people with that key though!
There is probably a break-glass procedure for such cases, like, break the literal window.
3 replies →
This is in SEA. They probably operate from ap-southeast-1 or 2. But yeah, if the internet goes down, the provider service goes down or AWS goes down they are cooked.
> Absolute nightmare.
Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
Easier maybe. But significantly worse. Parts of these systems have been build and engineered to be entirely reliable with automatic hand-overs when some component fails or with alternative routings when some connection is lost.
>than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
You don't need any of that. You need one more box in the electrical closet and one password protected wifi for all the crap in the building (the actual door locks and the like) to connect to.
And when that box fails, you're looking at how long with no access? Longer than any AWS outage.
5 replies →
Its absolutely possible to have both a SaaS based control plane and continue functioning if the internet connection/control plane becomes unavailable for a period. There's presumably hardware on site anyway to forward requests to the servers which are doing access control, it wouldn't be difficult to have that hardware keep a local cache of the current configuration. Done that way you might find you can't make changes to who's authorised while the connection is unavailable, but you can still let people who were already authorised into their rooms.
> with redundant power, cooling, etc
The doors the system controls don't have any of this. Hell, the whole building doesn't have any of this. And it definitely doesn't have redundant internet connections to the cloud-based control plane.
This is fear-mongering when a passive PC running a container image on boot will suffice plenty. For updates a script that runs on boot and at regular intervals that pulls down the latest image with a 30s timeout if it can't reach the server.
What updates? That would be on a local network and have no internet connection, if done right.
2 replies →
You know what else would suffice plenty? Physical keys and mechanical locks. They worked (and still work) without electricity. The tech is mature and well-understood.
2 replies →
Those devices can be trivially power cycled, and won’t have as many issues with dodgy power. Some PC somewhere with storage is a bigger problem.
5 replies →
It's also easier to keep all the water for fighting fires in trucks that are remote, than to run high pressure water pipes to every room's ceilings, with special valves that only open when exposed to high heat. Imagine the overhead costs!
Cooling for a card access system?
A card access system requires zero cooling, it’s a DC power supply or AC transformer and a microcontroller that fits in a small unvented metal enclosure. It requires no management other than activating and deactivating badges.
There is no reason to have any of the lock and unlock functionality tied to the cloud, it’s just shitty engineering by a company who wants to extract rent from their customers.
The server running that system needs cooling, yes. You can't just shove it in a closet with zero thought and expect it to not overheat/shut down/catch fire, unless you live in the Arctic.
8 replies →
The system was not built with resiliency in mind and had no care/considerations for what a shit-show will unfurl once the system or the link goes down. I wonder if exit is regulated (you can still fully exit the building from any point using the green buttons and I think these are supposed to activate/still work even if electricity is down).
> Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.)
A isolated building somewhere in the middle of the jungle dependent for its operation on some American data-center hundreds of miles away is simply negligence. I am usually against regulations but clearly for certain things we can trust that all humans will be reasonable.
In the US, the answer is that exit would have to work in the event that AWS is down or power is out. Some exceptions exist for special cases.