I had a friend who worked in federal law enforcement who once described a vampire device that they used. It would clamp around a power cable and inject a UPS in the mix so that an electronic device could be removed without turning it off. Seemed like a useful little trick.
If nothing else, would let you move a Frogger machine.
More seriously, I have wondered if you can detect these kinds of external interference. Auto lock the machine if power/network/wifi/Bluetooth/USB conditions change.
Nabbing an unlocked laptop was how they got the Silk Road guy (though they probably already had sufficient evidence elsewhere).
One trick you could use is to abuse the fact that law enforcement often plugs in a mouse wiggler on an unlocked desktop and kill your server the moment you see a new HID device (make sure to run some kind of desktop on your server so they think they can keep the session open, best to do it in a VM).
You could also monitor the ethernet link. They can move your server but they can't move the entire network, set up an encrypted tunnel between two distant physical servers and self destruct the moment that tunnel gets disrupted.
Some computers come with gyros/accelerometers built in. My old HP laptop had some kind of head crash prevention that used that hardware. I know this, because Gnome thought it was a tablet style sensor and turned my screen upside down if I didn't disable the sensor. Maybe getting a HP server can already get you a whole bunch of movement sensors.
You could probably figure out if the server is being moved by measuring capacitance of the case, measuring accelerometers, maybe add a GPS dongle. Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. You can probably measure _something_ in the server room, though, so to make sure your LTE dongle doesn't get interrupted, also measure whatever reliable signal you can find to detect Faraday cages.
Lastly, you could put a video camera in the case on all sides and measure changes. Detecting law enforcement badges probably isn't that hard with opencv if you're dedicated enough.
You have to hide your security measures and never tell anyone, though, or they'll just leave the server as-is and use the classic rubber hose exploit to make you give up the key material.
Maybe attack the problem from a different angle: use an accelerometer. Or spend a little bit more money to add a gyro and make a real, if very low accuracy, IMU.
Easily. Bolt the machine to the floor in such a way where the case has to be opened and a trip sensor activated to actually move the machine.
You can switch my power source without noticing? Who cares. The attack is taking the machine where it is not supposed to be. That's a problem we've been solving since forever.
Wifi would probably be the easiest. Either hide a dummy AP in the house or use a combination of multiple neighbors APs. If you don't see any beacon frames from the dummy SSID for a 30 second period then lock/shred the computer.
Isn’t that kinda what they used for Ross Ulbright’s computer? I know it was a laptop but they probably didn’t want to take chances given if that thing shut down the entire thing would be encrypted?
I thought they had an attractive agent distract him for a moment while another agent grabbed his still-unlocked-and-open laptop to prevent him from locking it or closing it up. At least I think that was the cloak-and-dagger story I heard.
Someone successfully did this for copper gigabit ethernet and presented at one of the security conferences - but with a few milliseconds interruption in signal.
That is why you put in special outlets that communicate with the PC over the power line encrypted.
You would need to drill holes in the concrete wall to get to the power lines in the wall in order to take the outlet along and hope that there isn't an additional device in the breaker panel.
Its a parasitic tap that connects to the mains power cable going into the device.
It then phase locks an inverter with said mains power, allowing the mains power cable to be unplugged and the whole lot transported elsewhere on battery power.
This is a great writeup! Especially for those that may want to DIY it, the how and the why and all of that, and not have to shell out for carrier-quality Layer 1 encryption devices. Nice to see that even off-the-shelf components can do it with relative ease at those rates. Also nice to see sane sysctl tunes as well. Anything to make an adversary's day a bit harder. I low key love the explanation of old 10B5 taps, something that so well and truly dead, but the legacy carries on into everything new today.
This is actually a well-trodden area of datacenter interconnect (DCI) devices that do line-rate encryption (to crazy levels like 400G+) to protect those links that may have easily accessible fibers strung along poles, for instance, to prevent just the vampirism described in the post. Packetlight, Ciena, Infinera and others.
Really cool article, I enjoy reading through all the details behind the decision making.
Just spit-balling a little, but I wonder if Wireguard is the best tool here given that the author is only using it for a single point-to-point link and they control the devices on both ends. That CPU supports AES-NI and probably does it a lot faster than Wireguard's ChaCha20 (hard to get numbers for their server CPU, but the tiny little x86 mini PC I use as my router does AES XTS at 43Gbps according to `cryptsetup benchmark`).
You might see better performance by tunneling the vxlan connection using a different technology which can use AES-NI? Then again, Wireguard is definitely still a good tool for stuff like this, and maybe the performance penalty isn't a big deal here.
AES can only encrypt up to 64TB; after that you need to re-key. So you need a mechanism for rekeying anyway. Definitely a good idea to use a battle-tested tool like wireguard instead of rolling your own.
> I wonder if their setup achieves the same degree of transparency, because afaiui, that's just not possible involving a 802.1Q-compliant (Linux) bridge.
Can you elaborate on what is not transparent about 802.1q bridge in Linux?
I hear you on the system tuning. Whenever I change sysctl variables I always include a comment with what the default was and why the new setting is better. I don't trust sysctl copy pasta w/o decent explanations.
There's a number of "special" Ethernet addresses that a proper Ethernet bridge must never forward. The Linux bridge implements a mechanism to ignore _some_ of these constraints, but not all of them. If you ned that, you can always get to manual patching in https://github.com/torvalds/linux/blob/d42f7708e27cc68d080ac... et al.
What mitigations did you disable, specific ones you know wouldn't be a risk to what the machines were doing (mostly network, mostly kernel space)..?
Like, by disabling the mitigations does that leave the servers slightly more open to someone nefarious finding a way to use some kind of timing attack to get some knowledge of your wireguard keys?
(Genuine question as someone with very little knowledge on both wireguard and *bleed CPU flaws)
No, I actually just booted with 'mitigations=off' and called it a day. We will employ Zen4 cores on the pre-prod setup soon enough, and I'll be looking into the benefit (if any) of disabling mitigations in a more fine-grained manner there.
To "fix" performance (i.e., increase throughput by close to 35%) one has to mess with the "energy performance bias" on the (Broadwell) platform, e. g. using x86_energy_perf_policy[0] or cpupower[1]. Otherwise, the CPUs/platform firmware will select to operate in a very dissatisfactory compromise between high-ish power consumption (~90W per socket), but substantially less performance than with having all idle states disabled (= CPU in POLL at all times, resulting in ~135W per core) completely. One can tweak things to reach a sweet spot in the middle, where you can achieve ~99% of the peak performance at very sensible idle power draw (i.e., ~25W when the link isn't loaded).
With Zen3, this hardly mattered at all.
I also got to witness that using IPv4 for the wireguard "overlay" network yielded about 30% better performance than when using IPv6 with ULA prefixes.
I can't believe they were under any memory pressure, so the first three presumably made no difference, but it's also quite surprising to me that the default ondemand cpu governor was responsible for such a dramatic performance hit. Not throttling up quickly enough leading to higher latency maybe? Very interesting anyway.
Did Cisco really invent MACSec?! I thought it was cooked up by the IEEE and supported in hardware from many vendors. I imagine they all have their own bugs though, it's quite a complicated spec. I know some switch/router vendors also now offer hardware-accelerated end-to-end encryption, similar to IPsec, Nokia call their's anysec but I'm sure the other players have their own. The benefit of those is you'd get full bandwidth (e.g. Tbps).
Usually one vendor prototypes a feature then they take it to IEEE/IETF for standardization. Probably half of all network protocols were invented by Cisco.
Why MACSEC isn't the default is pretty crazy! given that is is extremely stateless (encrypting at the frame level) and counters should be pretty reliable (only go up, since there's two parties) you could take advantages of some AES and GCM modes that would pretty quickly spot injection, replay, and other attacks.
But getting back to the main topic of the paper: why not just S2S IPSec the link?
I don't recall the specifics of macsec but it's possible to build a link encryptor that adds essentially zero latency. (like... no more latency than the gate delay of a single xor gate... plus some once-an-hour packet-length delay of some rekeying traffic).
Missing attack: Cause a disruption that obviously breaks the connection while further away you get time to tap it properly.
"Oh, no, a truck run into the pole carrying the copper/fiber, it must be an accident and no intervention is going on undetected because of the outage."
What we really need is promiscuous connectivity , but fully untrusted connections. It's maddening why it's hard to communicate 2 wireless devices while they are literally sharing the same radio spectrum and multiple radios could be used to talk to each other.
Tapping is even easier if you have access to the cable end in a patch panel.
I have a computer setup with a one-way gige connection for reviewing potentially malicious content in an air-gapped manner. The transmit side transceiver needs to see an incoming signal, so I just use one of these to feed its own output back into it:
# setup a 8020 MTU on wg0 interface to account for the 80 bytes wireguard headers overhead
# 20-byte IPv4 header or 40 byte IPv6 header, 8-byte UDP header 4-byte type, 4-byte key index, 8-byte nonce, 16-byte authentication tag)
/sbin/ip li set dev wg0 mtu 8020
Shouldn't that be 8920? To go with the 9000 byte MTU on the outer interface above it.
I had a friend who worked in federal law enforcement who once described a vampire device that they used. It would clamp around a power cable and inject a UPS in the mix so that an electronic device could be removed without turning it off. Seemed like a useful little trick.
If nothing else, would let you move a Frogger machine.
More seriously, I have wondered if you can detect these kinds of external interference. Auto lock the machine if power/network/wifi/Bluetooth/USB conditions change.
Nabbing an unlocked laptop was how they got the Silk Road guy (though they probably already had sufficient evidence elsewhere).
https://arstechnica.com/tech-policy/2015/05/sunk-how-ross-ul...
One trick you could use is to abuse the fact that law enforcement often plugs in a mouse wiggler on an unlocked desktop and kill your server the moment you see a new HID device (make sure to run some kind of desktop on your server so they think they can keep the session open, best to do it in a VM).
You could also monitor the ethernet link. They can move your server but they can't move the entire network, set up an encrypted tunnel between two distant physical servers and self destruct the moment that tunnel gets disrupted.
Some computers come with gyros/accelerometers built in. My old HP laptop had some kind of head crash prevention that used that hardware. I know this, because Gnome thought it was a tablet style sensor and turned my screen upside down if I didn't disable the sensor. Maybe getting a HP server can already get you a whole bunch of movement sensors.
You could probably figure out if the server is being moved by measuring capacitance of the case, measuring accelerometers, maybe add a GPS dongle. Or you could add an LTE connector and measure any signals you may receive that you shouldn't from inside a server room. You can probably measure _something_ in the server room, though, so to make sure your LTE dongle doesn't get interrupted, also measure whatever reliable signal you can find to detect Faraday cages.
Lastly, you could put a video camera in the case on all sides and measure changes. Detecting law enforcement badges probably isn't that hard with opencv if you're dedicated enough.
You have to hide your security measures and never tell anyone, though, or they'll just leave the server as-is and use the classic rubber hose exploit to make you give up the key material.
3 replies →
Maybe attack the problem from a different angle: use an accelerometer. Or spend a little bit more money to add a gyro and make a real, if very low accuracy, IMU.
6 replies →
If they’re seizing your laptop and your laptop will only work inside your house, wouldn’t they just seize your whole house?
2 replies →
> detect these kinds of external interference.
Easily. Bolt the machine to the floor in such a way where the case has to be opened and a trip sensor activated to actually move the machine.
You can switch my power source without noticing? Who cares. The attack is taking the machine where it is not supposed to be. That's a problem we've been solving since forever.
Wifi would probably be the easiest. Either hide a dummy AP in the house or use a combination of multiple neighbors APs. If you don't see any beacon frames from the dummy SSID for a 30 second period then lock/shred the computer.
1 reply →
Isn’t that kinda what they used for Ross Ulbright’s computer? I know it was a laptop but they probably didn’t want to take chances given if that thing shut down the entire thing would be encrypted?
I thought they had an attractive agent distract him for a moment while another agent grabbed his still-unlocked-and-open laptop to prevent him from locking it or closing it up. At least I think that was the cloak-and-dagger story I heard.
2 replies →
Someone successfully did this for copper gigabit ethernet and presented at one of the security conferences - but with a few milliseconds interruption in signal.
That is why you put in special outlets that communicate with the PC over the power line encrypted.
You would need to drill holes in the concrete wall to get to the power lines in the wall in order to take the outlet along and hope that there isn't an additional device in the breaker panel.
So it would emulate a UPS?
So they could just remove the existing UPS?
what is inject a UPS?
Its a parasitic tap that connects to the mains power cable going into the device. It then phase locks an inverter with said mains power, allowing the mains power cable to be unplugged and the whole lot transported elsewhere on battery power.
8 replies →
This is a great writeup! Especially for those that may want to DIY it, the how and the why and all of that, and not have to shell out for carrier-quality Layer 1 encryption devices. Nice to see that even off-the-shelf components can do it with relative ease at those rates. Also nice to see sane sysctl tunes as well. Anything to make an adversary's day a bit harder. I low key love the explanation of old 10B5 taps, something that so well and truly dead, but the legacy carries on into everything new today.
This is actually a well-trodden area of datacenter interconnect (DCI) devices that do line-rate encryption (to crazy levels like 400G+) to protect those links that may have easily accessible fibers strung along poles, for instance, to prevent just the vampirism described in the post. Packetlight, Ciena, Infinera and others.
Really cool article, I enjoy reading through all the details behind the decision making.
Just spit-balling a little, but I wonder if Wireguard is the best tool here given that the author is only using it for a single point-to-point link and they control the devices on both ends. That CPU supports AES-NI and probably does it a lot faster than Wireguard's ChaCha20 (hard to get numbers for their server CPU, but the tiny little x86 mini PC I use as my router does AES XTS at 43Gbps according to `cryptsetup benchmark`).
You might see better performance by tunneling the vxlan connection using a different technology which can use AES-NI? Then again, Wireguard is definitely still a good tool for stuff like this, and maybe the performance penalty isn't a big deal here.
AES can only encrypt up to 64TB; after that you need to re-key. So you need a mechanism for rekeying anyway. Definitely a good idea to use a battle-tested tool like wireguard instead of rolling your own.
>AES can only encrypt up to 64TB
I've never heard that before. Are you referring to a specific mode of operation?
2 replies →
Umm... IPsec?
1 reply →
Because Wireguard is cool and AES is uncool.
I guess it depends on whether you're more concerned about transport security or cipher cycles/byte.
8 replies →
As I had posted a few weeks ago (https://www.man7.org/linux/man-pages/man8/tc-mirred.8.html
> I wonder if their setup achieves the same degree of transparency, because afaiui, that's just not possible involving a 802.1Q-compliant (Linux) bridge.
Can you elaborate on what is not transparent about 802.1q bridge in Linux?
I hear you on the system tuning. Whenever I change sysctl variables I always include a comment with what the default was and why the new setting is better. I don't trust sysctl copy pasta w/o decent explanations.
There's a number of "special" Ethernet addresses that a proper Ethernet bridge must never forward. The Linux bridge implements a mechanism to ignore _some_ of these constraints, but not all of them. If you ned that, you can always get to manual patching in https://github.com/torvalds/linux/blob/d42f7708e27cc68d080ac... et al.
1 reply →
> include a comment
You may already do this, but in general, please include the Year, Month, Kernel Version and your own Name when doing this.
What mitigations did you disable, specific ones you know wouldn't be a risk to what the machines were doing (mostly network, mostly kernel space)..?
Like, by disabling the mitigations does that leave the servers slightly more open to someone nefarious finding a way to use some kind of timing attack to get some knowledge of your wireguard keys?
(Genuine question as someone with very little knowledge on both wireguard and *bleed CPU flaws)
No, I actually just booted with 'mitigations=off' and called it a day. We will employ Zen4 cores on the pre-prod setup soon enough, and I'll be looking into the benefit (if any) of disabling mitigations in a more fine-grained manner there.
> CPU frequency scaling/idle states sorted out correctly
please elaborate
To "fix" performance (i.e., increase throughput by close to 35%) one has to mess with the "energy performance bias" on the (Broadwell) platform, e. g. using x86_energy_perf_policy[0] or cpupower[1]. Otherwise, the CPUs/platform firmware will select to operate in a very dissatisfactory compromise between high-ish power consumption (~90W per socket), but substantially less performance than with having all idle states disabled (= CPU in POLL at all times, resulting in ~135W per core) completely. One can tweak things to reach a sweet spot in the middle, where you can achieve ~99% of the peak performance at very sensible idle power draw (i.e., ~25W when the link isn't loaded).
With Zen3, this hardly mattered at all.
I also got to witness that using IPv4 for the wireguard "overlay" network yielded about 30% better performance than when using IPv6 with ULA prefixes.
[0]: https://man.archlinux.org/man/x86_energy_perf_policy.8 [1]: https://linux.die.net/man/1/cpupower
2 replies →
I'm interested in their last step, in which they set
to raise a surprisingly low performance ceiling.
I can't believe they were under any memory pressure, so the first three presumably made no difference, but it's also quite surprising to me that the default ondemand cpu governor was responsible for such a dramatic performance hit. Not throttling up quickly enough leading to higher latency maybe? Very interesting anyway.
> Not throttling up quickly enough leading to higher latency maybe?
Quite amusingly it's the load isn't big enough for the default governor to rise the clock up to max.
If they posted the cpu load it would be pretty obvious.
Did Cisco really invent MACSec?! I thought it was cooked up by the IEEE and supported in hardware from many vendors. I imagine they all have their own bugs though, it's quite a complicated spec. I know some switch/router vendors also now offer hardware-accelerated end-to-end encryption, similar to IPsec, Nokia call their's anysec but I'm sure the other players have their own. The benefit of those is you'd get full bandwidth (e.g. Tbps).
Usually one vendor prototypes a feature then they take it to IEEE/IETF for standardization. Probably half of all network protocols were invented by Cisco.
Why MACSEC isn't the default is pretty crazy! given that is is extremely stateless (encrypting at the frame level) and counters should be pretty reliable (only go up, since there's two parties) you could take advantages of some AES and GCM modes that would pretty quickly spot injection, replay, and other attacks.
But getting back to the main topic of the paper: why not just S2S IPSec the link?
> Why MACSEC isn't the default
TFA explains it pretty well. Also every encryption is adding the load and latency, so defaulting to it when it wasn't asked for isn't the best way
> why not just S2S IPSec the link?
Because IPSec is still PITA and also sucks bad performance wise against WG.
I don't recall the specifics of macsec but it's possible to build a link encryptor that adds essentially zero latency. (like... no more latency than the gate delay of a single xor gate... plus some once-an-hour packet-length delay of some rekeying traffic).
Missing attack: Cause a disruption that obviously breaks the connection while further away you get time to tap it properly.
"Oh, no, a truck run into the pole carrying the copper/fiber, it must be an accident and no intervention is going on undetected because of the outage."
What we really need is promiscuous connectivity , but fully untrusted connections. It's maddening why it's hard to communicate 2 wireless devices while they are literally sharing the same radio spectrum and multiple radios could be used to talk to each other.
Tapping is even easier if you have access to the cable end in a patch panel.
I have a computer setup with a one-way gige connection for reviewing potentially malicious content in an air-gapped manner. The transmit side transceiver needs to see an incoming signal, so I just use one of these to feed its own output back into it:
https://www.amazon.com/dp/B0B8ZHBK26
Shouldn't that be 8920? To go with the 9000 byte MTU on the outer interface above it.
That would probably add only a smidge more performance.
Sure, but the comment block says that they arrived at 8020 by subtracting 80 from 9000, which is just wrong.
I did something like this to stretch L2 as I was moving into a new home. Worked great after I realized t-mobile does not like passing IP fragments.
Got to use it again to set up a remote telescope for the eclipse.
A notebook with pandas would have had a df.plot().
9.71 Gbps with wg on a 10GBps link with sysctl tunings, custom MTUs,.
I had heard of token ring, but not 10BASE5: https://en.wikipedia.org/wiki/10BASE5
> Eth/IP/UDP/WG/IP/UDP/VXLAN/Eth/802.1q/IP/Payload
I'd love to see your IT manager's face when you propose it :D
This is not uncommon. At my last job we had double VXLAN.
I prefer to just run fiber inside of a copper gas line.
because it's easily available and unconspicuous?
asking because copper is neither cheap nor particularly hard to penetrate.