All the people working in the datacenter have that level of physical access.
Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen:
- unlocked racks
- doors open
- temporary network cables and keyboards, monitors and mice attached to running systems
- systems logged in left unattended
- floor panels raised up and left open unattended exposing cabling
- meet-me rooms with interfaces exposed (gear in racks without doors)
DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18...
Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
* Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now.
* Require them to sign into and out of the site.
* Annotate which systems / compromises are in place.
- That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc)
Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen.
I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.
You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised.
There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.
Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same.
Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target.
And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal...
... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal.
All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in.
I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead.
No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS.
I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way.
And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8].
> Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
Amusingly, I read about an incident like this on one of those forums (probably ServeTheHome). It apparently happens so often that Hetzner's control panel has a special state for it. Server status: "seized by law enforcement" and the power-on button is disabled.
Next time you try that trick, use heptane (sold commercially in North America as "Un-du") instead of isopropanol. Hot knife meets butter... and it evaporates cleanly and the sticker retains most of its stickiness!
All the people working in the datacenter have that level of physical access.
Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen:
- unlocked racks
- doors open
- temporary network cables and keyboards, monitors and mice attached to running systems
- systems logged in left unattended
- floor panels raised up and left open unattended exposing cabling
- meet-me rooms with interfaces exposed (gear in racks without doors)
DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
Do tell, please; stories about "interesting characters" are often the best.
1 reply →
The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18...
Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
I would hope they at least:
* Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now.
* Require them to sign into and out of the site.
* Annotate which systems / compromises are in place.
- That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc)
Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
8 replies →
Those are some very optimistic hopes!
You would expect that at AWS, but Hetzner is a low-cost operation.
I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen.
I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.
> I can't imagine many would attempt to stop you.
You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised.
There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.
Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same.
Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target.
And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal...
... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal.
All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in.
I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead.
No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS.
I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way.
And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8].
[1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnega...
[2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8
[3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277
[4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg
[5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog
[6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU
[7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE
[8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-mega...
> Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.
Amusingly, I read about an incident like this on one of those forums (probably ServeTheHome). It apparently happens so often that Hetzner's control panel has a special state for it. Server status: "seized by law enforcement" and the power-on button is disabled.
Next time you try that trick, use heptane (sold commercially in North America as "Un-du") instead of isopropanol. Hot knife meets butter... and it evaporates cleanly and the sticker retains most of its stickiness!