Keystroke timing obfuscation added to ssh(1)

2 years ago (undeadly.org)

Keystroke timing has been a concern for terminal I/O since the 1980s and folks were using primitive encryption with stelnet and kerberos.

Most terminal applications use buffered I/O for password entry, which is still an important security feature. In that mode, nothing is sent to the other end until the user presses return. A MiTM only "sees" one packet no matter what, and with padding they can't even infer the password length.

For a time, there was rich pickings in applications that accepted passwords in unbuffered mode. Many of them doing it so that they could echo "*" symbols, character by character, as the user typed. That simple feature looks cool, and does give the user feedback ... but would leak the keystroke rate, which is the last thing you want on password entry.

I hope we preserve buffered I/O for password entry, because it's still better than what ssh can do with some obfuscation. But it's great to see ssh add this, and will help protect content that can't be buffered, like shell and editor input.

  • Aside, I've noticed that the current technique of rendering a fixed number of asterisks independent of the password length is quite confusing to users -- "that's wrong, it's the wrong length", resulting in attempts to type in the "correct" password and this obviating the benefit of the stored password.

    Not sure how to fix that. I recall a visible hash of some form being used in the past (eg take a 2-digit hash, pair of with a smiley; I must have entered it right, it's showing me ROFL smiley), but that would aid shoulder surfed password entries, at least.

    • I've seen a GUI password input field that mutated an abstract line drawing on every keypress. Think random cross-hatching over the whole input field where the lines are nudged a little on every press.

      (Not that that's necessarily a good idea, it still gives away timing/length information to e.g. cameras.)

      6 replies →

    • I'm honestly seeing little value in asterisks with WFH and the move to passphrases. Feedback is important when you're typing a long phrase with complete precision. Plus shoulder surfing is simply not a thing when my physical security profile now involves a locked front door and a call to the police.

      24 replies →

    • I’ve seen some programs render three asterisks per key stroke. Defers human shoulder surfers from seeing the length of your password.

      I think simplest and safest solution would be a shape that rotates at random interval for each key stroke.

      Depends what problem you want to solve. Did keyboard register my press? vs did I type the same thing as last time.? have some different constraints.

    • The login fractal - a shape that is infinitely recurring, starts at a random place, and indicates entry with "zooming".

    • The browser could use a different rendering convention for autopopulated passwords. For instance, it could render a solid black bar (no characters for the user to count) or maybe the phrase "autofilled", perhaps with a strange background color / rendering convention.

  • > Keystroke timing has been a concern for terminal I/O since the 1980s and folks were using primitive encryption with stelnet and kerberos.

    I had a visual basic AI addon in the 1990's that could work out who was typing at the keyboard from their typing pattern within a few minutes of typing, which kind of rendered the logon process mute.

    Today, that can applied to touchscreen logons by tying finger pressure patterns ie size and shape of finger contact with the touchscreen to a user, and when incorporating swipes or mouse movements in the desktop OS context, its possible to have a security app which can lock a system if someone is using a device and user account which is not theirs.

    At the very least you can log every time one's GF/missus has gone through your phone.

  • Password based ssh authentication should be used approximately never.

    • That is not the only time you use passwords over ssh, e.g. I don't use a password to remote into my desktop from my laptop, but I do use one when using sudo on the desktop.

      10 replies →

    • This is naive in the extreme. There are many scenarios where passwords are needed, for bootstrapping, a disgruntled admin leaving, etc.

      There is a role for a common secret in a secure ecosystem (password, passkey)

      1 reply →

    • This is completely irrelevant to password based SSH authentication. The timing obfuscation is for the session _after_ authentication.

  • Does anyone know any SSH clients that support line-buffering of input?

    I.e. where what you type doesn't get transmitted until you hit or click return/send?

    I had one of these clients (but for telnet) back in more active MUD gaming days but haven't seen it with the few SSH clients I've used since... but always thought that would be a good defense to SSH keystroke timing data leakage, and potentially superior to this 20ms delay approach mentioned in this article, at least for some usage scenarios.

    (Although now that I think about it, ideally you might want it to also transmit when someone hits tab so you could still have linux shell autocomplete...)

    • That would only work if the ssh client could know exactly what was going on in the user session. Like, how would that work if I were editing a file with vim? Or even just typing a command into the shell (where I might need to backtrack and edit the command)?

      This doesn't seem very feasible or useful to me.

      1 reply →

    • That's more a choice a current-day shell etc does for you, wanting to control the editing experience. Run `cat` and it'll switch to line buffered mode, note how your arrow keys just input line noise, and watch the cat process with ptrace if you want to confirm it really receives the whole line in one read syscall.

  • > Most terminal applications use buffered I/O for password entry

    does the existance of this patch implicate that openssh does not behave that way?

  • i don't really type passwords into remote connected terminals anymore and haven't for some time. shrug

    while shoring up the existing holes is worthwhile, people should really be using keys or difficult to type passwords in 2023.

  • so that's why my terminal doesn't show "*" as i type my password?

    tripped me up the first time i used a shell. interesting to see why

This reminds me of professional Bridge. They split the teams with a wall and pass their cards through a window at the same time to prevent communication through timing.

https://youtube.com/watch?v=RVZLNRmO3vo

  • And yet they cheat through the screens.

    https://en.wikipedia.org/wiki/Blue_Team_(bridge)#Cheating_an...

    https://en.wikipedia.org/wiki/Fantoni_and_Nunes_cheating_sca...

    https://en.wikipedia.org/wiki/Fisher_and_Schwartz_cheating_s...

    And those are the ones we know about. :)

    As far as actually using bridge in a job interview once. I did. In bridge there's a rule where if your partner gives you a hint not via the bidding, you must take the opposite approach if logically possible. It is called "Active Ethics". I had an interviewer try to lead me by the nose to the answer way too hard, in a debugging interview. So I'd stop and check EVERYTHING I could think of first before doing what he said. I told him I was doing it after the interview, and to look up active ethics if he needed a further explanation.

    Got the job.

    • While I admire your ethics, I feel like a lot of technical job interviews are structured such that you're supposed to actively collaborate with the interviewer. The interviewer is allowed to give you hints or suggestions, and they're very interested in how well the candidate takes hints.

      And sometimes the hint can be a trick! I recently did an interview where the interviewer asked if I should use a shortcut to compare two strings, one that assumed there's only one way to normalize a string. I almost fell for it, but then I hesitated and mentioned that I was concerned about some languages where that assumption wouldn't hold. They agreed and were happy that I chose the safer approach.

      5 replies →

    • It probably wasn't the situation in your case, but I often give straightforward hints if the candidate is struggling with something that I don't want them to spend time on so we can get to the significant material.

      E.g. in an algorithms interview they get stuck on an unrelated python issue (many people interview in python but don't use it day to day), or in a system design interview they get stuck on designing extra-credit subsystem C when they haven't finished subsystems A and B.

      If they aren't getting it after a couple hints, I'll just tell them the answer or tell them to come back to it later.

      Anyway, I would be very careful if you aren't going where the interviewer is pointing you. If you think it's a trick or you want to practice Active Ethics, then I would call that out in the moment since you might be messing up the flow of the interview at best and come off as hard to work with at worst.

      1 reply →

    • Oh, I know. Attackers will continue to attack. In my opinion, professional bridge is a doomed game. Decades of added steps to prevent cheating complicate too much an already very difficult game, and determined, smart people are still very successful at bypassing them anyway.

      I still want to learn to play at a reasonable level though, I'd rather waste my time on bridge than chess. But it needs to be home games, and there's no way I'm going to find the partners when spades and bid whist are out there and easy to learn.

      1 reply →

    • Surely then you're just in a game of bluff with a Sicilian ... ie then you just feel your partner to do the opposite and make sure it's caught, resulting in them taking the action you intended?

      IANABridgePlayer, clearly.

      2 replies →

  • If people are just expected to be human state machines and are penalized for not doing the prescribed automata, then you might as well flip a coin for the trophy and skip the game.

    This is like saying a catcher can't signal to a pitcher.

    Information-passing is a human skill that adds a dimension to the game. Let the best win.

    • Yeah, I lost all interest in Bridge when I found out the people who play it hate 100% of the interesting parts and had outlawed them, and that every time someone comes up with another cool approach, they outlaw that, too.

      Initially learning the game it was like “oh wow, that feature of the game has some really cool implications! This is amazing!” but then reading about how real bridge tournaments run, yeah, they crafted the rules to remove every single one of those cool implications.

      [EDIT] to be fair, the basic rules would also result in a terrible game as soon as people got too good at exploiting them. I just think they’ve managed to find another way to ruin the game while keeping it technically playable.

      2 replies →

    • > Information-passing is a human skill that adds a dimension to the game.

      Nah. You choose the game that you prefer. You can play the game where you cheat all the time, but don't play it with people who like bridge without asking them first.

    • Bridge has a built in channel for communication that has very limited bandwidth. The bidding conventions are about maximizing how much you communicate with limited symbols and almost no attempt at secrecy. Effectively it'd turn the game into one where players play with their hands face up, because that's the most effective way to communicate. That doesn't sound very interesting to me.

    • If you want to invent a version of bridge where surreptitious information-passing is part of the game, more power to you, but it’s not the same game.

    • Quite untrue.

      There is this thing called "Bridge Judgement" that you are allowed to use.

      Just because your hand has 10 HCP... but has 13 spades... doesn't mean you will pass. You'll bid your 7 Spades and call it a day.

      Bridge has many shades of grey. It is learning how to dance them correctly that is hard.

  • Wow, looking at this with a red-team cap on, there is so much human "messiness" to exploit here. It shouldn't be too hard too be able to pass a bit or two of information.

    • It might be interesting for a security person to try to come up with ways to hypothetically assure a trustworthy bridge game, assuming no limits on costs or inconvenience (i.e. if a trustworthy bridge game takes three months to play, or requires launching a satellite into orbit, so be it.)

  • Bridge is a really weird game. It's all about secret communication with your partner, but it's not allowed to be secret. You can communicate, but no communication! Very odd.

    • Bridge tournament rules are crafted as if everyone involved wishes they were playing a different game, but are for some reason stuck with the basic rules of Bridge. There’s a pile of rules about how you aren’t allowed to do all kinds of things that the basic rules would enable.

      It’s like if baseball couldn’t change field size or mound height or whatever and just had to add lots of rules about how you aren’t allowed to throw too fast or hit too far et c., but kept the physical reality of the game the same.

    • Communication through bidding is fascinating. Any type of collusion during any kind of auction is fascinating.

  • It seems there is still a possibility for passing information. For example, you can shove the little table across the barrier, or slowly slide it to indicate something. That's how the guy in the upper right passed it the first and second time.

    • There are endless ways to pass information. Notice the sibling comment about "active ethics." It's the game sort of saying "there's really no fool-proof way to keep you from cheating, so please just be a good person. Even to the point that if you're put into a situation where you could accidentally cheat, you should intentionally play non-optimally."

Here's a 2008 article about a 2001(!) paper noting such timing attacks: https://lwn.net/Articles/298833/

>This weakness was outlined in a 2001 paper entitled Timing analysis of keystrokes and timing attacks on SSH" [PDF] which looked specifically at the timing-based attack:

>In this paper we study users' keyboard dynamics and show that the timing information of keystrokes does leak information about the key sequences typed. Through more detailed analysis we show that the timing information leaks about 1 bit of information about the content per keystroke pair. Because the entropy of passwords is only 4-8 bits per character, this 1 bit per keystroke pair information can reveal significant information about the content typed.

I thought this was fixed a long time ago and I thought there was a fix pushed around the 2012 time period. I'm totally shocked this has not been previously address.

  • > I'm totally shocked this has not been previously address.

    same same

    i rekon there is more going on

Some day we have to use packets which are pre-filled by random data to hide our keystrokes in. Not quite steganography, but close. Could also be used to make traffic-analysis harder/impossible even?

This makes me wonder about newer terminal emulators on maccOS like Warp[1], and if they're for example taking all input locally, and then sending it over the remote host in a single blob or not? I imagine doing so would possibly break any sort of raw-mode input being done on remote host but I'd also imagine that is a detectable situation in which you could switch into a raw keystroke feed as well.

[1]: https://warp.dev

  • In general once you’re connecting over SSH the connection itself is always in raw mode and then the remote host deals with its pty normally (which can be in line or raw mode). Terminals with special shell integrations usually need them installed on the remote host too (some have support that does that somewhat transparently though).

    This is why mosh can have better behaviour than pure SSH over high latency connections. However this feature isn’t going to apply to mosh.

    • I wonder if SSH can honor line-buffered mode. It should be able to detect it, but then if it incorrectly switches to line buffering then random stuff might deadlock.

  • It's really hard for me to imagine that an app that markets "AI for your terminal" is going to be "more secure and private" than some standard Unix tool.

    Perhaps some very specific example of a security feature (such as protecting against timing attacks) could be protected against in a new tool, and not in the older more standard one. But it seems far more likely that many other security features would get forgotten in the newer tool, and by adding "AI" so many more attack vectors would be added.

    It's honestly hard to even believe in the privacy claims of warp. Almost all NLP tools in today's age seem to fall towards cloud solutions, which almost immediately makes that likelihood of privacy close to nil.

  • If they’re designed to take in data at some baud rate, wouldn’t the blob feed in at that rate too?

What is the threat that this mitigates?

  • An eavesdropper cannot see the content of your keystrokes, but (previous to this feature) they could see when each keystroke is sent. If you know the target's typing patterns, you could use that data to recover their content. You could collect the target's typing patterns by getting them to type into a website you control with a Javascript enabled browsers, or from an audio recording of their typing. (Some online streamers have been hacked as of late using AI models trained to steal their passwords using the sounds of them typing on their keyboards).

    • I didnt find an article about actual hacks carried out with that technique, but here’s a HN discussion [1] from this month about a paper on the topic.

      From that discussion it sounds like you need to train on data captured from the actual target. Same physical keyboard in the same physical space with the same typer.

      Pretty wild despite those specific conditions. Very interested to know if people have actually been attacked in the wild with this and if the attackers were able to generalize it down to just make and model of a keyboard, or if they could gather enough data from a stream.

      [1]: https://news.ycombinator.com/item?id=37013704

  • IIRC there is at least one paper, maybe around 2005, where they were able to determine what was being typed in an encrypted ssh session, using packet timings correlated to collected human typing statistics. Looks like this adds noise to prevent that.

  • Basically you can analyze typing speed to make some assumptions

    For example, since users tend to type their passwords quicker than other things, you could see how many keystrokes were sent in a burst and guess the user's password length when they sudo something.

How much latency does this add? Latency, particularly unpredictable latency, is one of the greatest stressors in software development work.

  • It's right there.

    > ...by sending interactive traffic at fixed intervals (default: every 20ms) when there is only a small amount of data being sent...

  • > Latency, particularly unpredictable latency, is one of the greatest stressors in software development work.

    It took me a second, but I'm pretty sure the comment above is referring to latency in the user experience; namely, the delay between a keypress and perceived result. [1]

    FWIW, tools like Mosh [2] go a long way towards reducing perceived latency. Mosh displays the user keypress as soon as it is registered locally (which happens without a perceptable delay). To indicate that it has not round-tripped, the character is shown in a washed-out color, last I checked. (Or maybe underlined?) After the round-trip completes, the character is displayed normally.

    [1] If you greatest stressor in software development is the latency of your keypresses, you sound very lucky to me.

    [2]: https://mosh.org

I know some people do network monitoring for hands-on-keyboard shells (presumably) by measuring packet timing, I wonder if this will mess with those detections and if so by how much.

  • I hope that kind of thing goes the way of other corporate efforts to break/backdoor encryption for the sake of "security". IMO, it's really the wrong way to go about security. Sure it would be nice to know if some automated script is being used to log into a machine, but better design can mean that information isn't important.

    • This has nothing to do with breaking encryption and of all the sketchy corporate surveillance tooling that's deployed for security purposes (so say nothing of HR purposes) monitoring for shells on the network seems about as benign as it comes.

      2 replies →

  • What non-malicious use case is there for this?

    • Network monitoring for unauthorized/unusual access, reading more into how this works I don't think this would actually change anything, you can probably still discern scripted vs manual shells it would just be a bit harder.

Sending a packet every 20ms seems like a lot of extra traffic.

  • An empty IPv4 packet is 20 bytes, and an empty IPv6 packet is 40 bytes. An empty TCP header is 20 bytes. Therefore, if you want to send a single byte over TCP, you need 41 bytes over TCP/IPv4, or 61 bytes over TCP/IPv6.

    Let's call that 64 bytes/packet for a small packet.

    20ms/packet = 50 packets/sec = 3200 bytes/sec = 3.125KiB/s.

    For comparison, a copper-wire non-broadband modem in the early '90s ran at 33.6kbps (kilobits/sec) which worked out at 4.1KiB/s. So a packet every 20ms wouldn't even saturate 30-year old modem tech. And believe me, that was slooooooow!

    • But slow forced people to use their brains. Around 2002 I did WFH using some 30 kbit/s practical speed. My X11 desktop was shared pixel-accurate with decent response times over VNC.

      20 years later if someone shares their code over Google Meet I see some blurred stuff. And red font takes 3 seconds to become clear.

    • The US hasn't really improved infrastructure since the 90s though. So yeah, it's slow, but it's also still that slow for many people.

    • yeah, keeping the total bandwidth used to be less than a dialup modem connection was an explicit goal when choosing the 20ms default interval.

  • Only while you are typing and for a random time period after you stop. 50 packets per second, while I'm typing, doesn't seem like too many packets.

  • I didn't read the code, but as I understood it, it was more like a frame rule ("imagine a bus stop..."), where your keystrokes will be delayed/buffered for a few milliseconds and then sent in regular 20ms interval bursts

  • Fortge chaff packets, sure. But aggregating keystrokes for 20 ms into one packet may save data.

    • I can type pretty fast (100+ WPM) but I'm sure as hell not getting multiple keystrokes into a 20ms window

  • Guess that'd depend on how big the packet ends up being.

    • TCP has an overhead of 20 bytes. I'm not sure how much openssh adds, but if it's just a keystroke I can't imagine it'd be over 64 bytes. Add those together and multiply by 50 packets per second (20ms between each packet), and it works out to a whopping 4.2kB/s.

Does mosh do something similar? It seems like that'd be way more effective in a protocol that's already much more tolerant of random latency spikes already.

It's a nice security feature, true

Though I'm curious how does the project keep working with CVS in $year, I wonder if everybody just uses git cvsimport and just forgets about CVS most of the time

  • “No mention of openbsd on the internet is complete without a long thread about source control migration.” — tedu@

  • Here's an explanation: https://cvs.afresh1.com/~andrew/o/why-cvs.html

    • All of these reasons boil down to "if it ain't broke" and "that's what we're used to".

      Switching VCS for a project of this size is always complicated and OpenBSD devs are famously "old school" and conservative with their software choices.

      I used to use CVS before switching to SVN and later DCVS like Mercurial and Git. The claim that "it is unlikely that any alternative versioning system would improve the developer's productivity or quality" is absolutely laughable IMO.

      This is especially true nowadays where CVS support in modern tooling, if it even exists, is usually legacy and poorly maintained.

      3 replies →

    • The explanation all makes sense. But the key line of “we all know cvs” is effectively exclusionary to all the other developers in the world who don’t use cvs. At some point they will need new talent which will be harder to get.

      10 replies →

Makes you wonder: why do people still use password authentication with SSH?

  • Passwords are sent all at once from the client to the server. This feature is for obfuscating your keystroke timing within the encrypted connection.

  • Even if you don't use password authentication you may still type sensitive information in a SSH session. For example, password when using sudo.

  • For 40% of global use cases there’s probably probably little to no risk.

    The rest is probably a mix of good, bad, ‘just enough’.

  • A central aim of of SSH is confidentiality. There's a lot besides passwords that you can deduce with traffic analysis, especially if you can correlate with other observed events.

  • How else would I upload my public key?

    • A service may provision an account with a provided ssh public key, so that you never log in with a password, even once.

      It's sort of a chicken-egg problem though, presumably you do have a password somewhere along the line, such as in a portal where you created your account and uploaded your public key.

      5 replies →

    • The correct answer is using client certificates, but they're a great deal of pain to set up compared to "ssh-copy-id" (or using username/password!)

      1 reply →

  • This is irrelevant for SSH password authentication. The obfuscation is for the session _after_ authentication.

  • For real; you can even make sudo work with SSH_AGENT. Add hardware key and it's pretty nice setup.