Purpose of first-define is the rule: In placing configuration files higher than user-defined configuration but Only with SSH client, can want user to have control from their config files: Remove from config files Place a couple under Match/MatchGroup using deny/accept.
SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user,
system-wide,
specific user:
Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in:
With multiple config files overriding each other in an predictable order you can effectively allow users to change some settings while ignoring (overriding) whatever they set on others.
This is perhaps old sysadmin knowledge, but different tools have very different heuristics about how they parse configuration, and you have to check every time and not assume. Among the consequences to not checking are gaping security holes.
That's why I erase sshd_config and put what I really meant there. You may say "but isn't it better to patch it properly?". It is not. Yet another vps hoster –> yet another /etc/ssh directory template that may have all sorts of access issues in it. Better to replace it and make it do exactly what you have planned.
I've never liked the directory.d/* infrastructure. In so many cases, even with a properly configured sshd_config, the resulting configuration file is not so large that it benefits from being split up.
You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides.
I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
Config directories are there to solve change management problems like idempotency.
If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely.
This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly.
With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
The .d directories are important on Debian and Ubuntu where packaging needs to provide different snippets based on the set of installed packages, the VM environment, other configuration inputs like through cloud-init and so forth, and update them during upgrades, but also (as per policy) preserve user customisations on anything in /etc.
Since pretty much every file has different syntax, this is virtually impossible to do any other way.
The conf.d isn’t because the config file is large. It’s because it’s easier to disable or enable something with an “echo blah > conf.d/10-addin.conf” or an “rm conf.d/50-blah.conf” than it is to do sed -i or grep blah || echo blah >>
1. They add huge configuration files where 99% are commented out.
2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that.
I don't need all of that. I just need simple 5-line configuration file.
My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
Yeah, first-wins is definitely surprising. Off the top of my head, it feels like one would have to go out of one’s way to write a parser that does that (by storing an extra bit of state for each configuration item, and then checking it before setting the configuration item and toggling the state, rather than just applying the configuration item each time it is encountered).
Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
It probably makes a bit more sense when you think about the fact that SSH frequently does a "try to match this host against a list of configured host-patterns" operation. In that case, "first match" is the obvious thing to do.
The SSHD internally has a config structure. Initially, it's all initialized with -1 for flags/numbers/timeouts, and with NULL for strings. When an option is parsed, there is a check whether the config structure's corresponding field is -1/NULL (depending on the type) before the parsed value is saved.
Another program with first-wins I've seen used dict/map for its config so the check was even simpler: "if optname not in config: config[optname] = parsed_value".
It's actually the simplest scheme. Reparse from the top whenever you need to query a setting. When you see one, exit. No need to even bother to store an intermediate representation. No idea if this matches the actual ssh implementation, but that's the way many historical parsers worked. The idea of cooking your text file on disk (into precious RAM!) is fairly modern.
Nope, the actual ssh implementation parses all the config files once, at the startup, using buffered file I/O and getline(). That means that on systems with modern libc, the whole config file (if it's small enough, less than 4 KiB IIRC?) gets read into the RAM and then getline() results are served from that buffer.
The scheme you propose is insane and if it was ever used (can you actually back that up? The disk I/O would kill your performance for anything remotely loaded), it was rightfully abandoned for much faster and simpler scheme.
Loading up your parsing code and reopening the file every time a setting is queried sounds to me like it would increase the average memory use of most programs.
Yes. Run this as a validation step during base os image creation, if such image is intended to start system with sshd. That way you can verify that distro you use did not pull the carpet from under your feet by changing something with base sshd config that you implicitly rely on.
The only time I hear or see anything about cloudinit, it is always a problem.
Nobody ever said "we don't need worry about that, cloudinit takes care of it".
In this particular case cloudinit presence in the story is incidental, delivery mechanism of said config file could have been different.
It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
I initially read "the order of files in /etc/ssh/sshd_config.d/" to mean the order of files in the underlying directory inode, i.e. as returned by `ls -f` — and thought, "oh god"... But the lexicographical order, that's not too surprising.
but i guess learning is better late than never type of thing.
also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
I don't like these systems where configuration is built from a million separate files. They're unpleasant to work with.
The best reason to do it this way seems to be that files are the unit of package management. Perhaps we need a smarter package manager.
My nginx.conf life got better when I deleted sites-available and sites-enabled and defined my sites inline in nginx.conf.
The only thing worse is when the configuration is actually a program that generates the configuration, like in ALSA.
And the only thing worse than ALSA style is Xorg style, with a default configuration generated by C code and you can only apply changes to it without seeing it. Xorg also has this weird indirection thing where the options are like Option "Foo" "Bar" instead of Foo "Bar", but that's a nitpick in comparison.
Curious, since when is directory "/etc/ssh/sshd_config.d/" a thing ?
I checked on my OpenBSD (7.6) System and Slackware (15.0) and that directory does not exist. I checked the man page for sshd and there is no mention of that dir.
Is this a new thing/patch Linux people came up with ?
This is weird. I've been hitting funny problems while trying to get ssh to authenticate (using passwords) via. a keycloak instance. I've been trying to do using PAM script but have been pretty unsuccessful till now. Apparently, they don't play nice together.
This is interesting, usually it's the latter because the config is ran line by line
Also, if it's not too much trouble, would someone help me understand why such files are required to start with numbers? In this case it's 10-no-password.conf.
I have noticed similar structure for apt and many more packages
A lot of software which reads drop-in files will load them in numerical (or indeed alphabetical) order. Obviously this is important if the order your config files are loaded in matters, but otherwise it's just become a convention so people do it even if the load order doesn't actually matter.
Typically, config files are merged into one by loading them like ./conf.d/*, the order being determined alphabetically from their file names. You do not need to use numbers but they help to see that order.
I'm trying to brainstorm an answer. My best guess is that SSH is obsoleted by disposable instances. You can spin up a new instance for every version of your configuration, transition to it, and dispose of the original (or set it aside or whatever). That way, you could probably have a reasonably complete tech career and only ever use ssh as an implementation detail of git.
I have a feeling you were waiting to say that for a while. :D
Glad to make your day. You are welcome.
Ultimately we all end up reading the manual. I'd still prefer if I didn't have to remember how a certain C stdlib function works vs. what seems intuitive.
But that's a lost cause with a lot of people. They'll happily point out how "intuitive" differs among different groups of people and all that, merrily missing the point on purpose.
Oh well. At least I found out without locking my self out of my servers.
Every time I say RTFM (like in the case of strtok), I get down-voted. Some tools really cannot be dumbed down, and they should not. I do not know why people have an aversion to reading documentation. It is bad.
In the case of strtok, I am not going to implement my own if strtok does what I want it to do, and behaves how I know it does. Why would I?! Sometimes what I need is strtok, sometimes strsep, sometimes I may use strtok_r.
Purpose of first-define is the rule: In placing configuration files higher than user-defined configuration but Only with SSH client, can want user to have control from their config files: Remove from config files Place a couple under Match/MatchGroup using deny/accept.
SSHD (server/non-client) still support admin-defined by having system-wide settings done firstly. For those who have multi-file SSHD configurations, breakdown of the many config file locations and scopes here as it covers default user, system-wide, specific user:
https://egbert.net/blog/articles/ssh-openssh-options-ways.ht...
Also I broken out each and every SSHD and SSH options along with their ordering by execution by using file name and numbering as well as its various state machine, dispatch, CLI equivalence, network context, and function nesting, all in:
https://github.com/egberts/easy-admin/tree/main/490-net-ssh
https://github.com/egberts/easy-admin/blob/main/490-net-ssh/...
Disclaimer: I do regular code reviews of OpenSSH and my employer authorizes me to release them (per se contract and NDA)
Also this showed how to properly mix and match authentication types using OR and AND logic(s) in
https://serverfault.com/a/996992
It is my dump mess so wade 'em and enjoy.
The way you wrote the rule makes no sense to me. Maybe it's too early in the day for me?
"In placing configuration files higher than user-defined configuration but Only with SSH client, can want..."
With multiple config files overriding each other in an predictable order you can effectively allow users to change some settings while ignoring (overriding) whatever they set on others.
https://news.ycombinator.com/item?id=43605285
I find this SSHD snippet to be extremely useful in enterprise network, notably with OpenLDAP.
Also the most dangerous but flexible way to authenticate a user.
https://jpmens.net/2019/03/02/sshd-and-authorizedkeyscommand...
This is really good! Thanks!
For those that are exploring software-based public certificate and OpenSSH, Ive broken down the settings for most PKI handlers.
https://egbert.net/blog/articles/openssh-file-authorized_key...
Thanks for sharing this! I think I may now have what I need to set up a system with multi-user shared keys that only work for a given set of users.
1 reply →
We must have knocked your site offline
1 reply →
This comment seems to have a lot to say but it was word salad to me, quite confusing and hard to read :(
https://news.ycombinator.com/item?id=43605285
It has been translated from OpenSSH meta-spaghetti code logic. Break it down by parts of sentence.
2 replies →
This is perhaps old sysadmin knowledge, but different tools have very different heuristics about how they parse configuration, and you have to check every time and not assume. Among the consequences to not checking are gaping security holes.
You mean there are consequences to making assumptions? ;) (also old sysadmin)
That's why I erase sshd_config and put what I really meant there. You may say "but isn't it better to patch it properly?". It is not. Yet another vps hoster –> yet another /etc/ssh directory template that may have all sorts of access issues in it. Better to replace it and make it do exactly what you have planned.
I've never liked the directory.d/* infrastructure. In so many cases, even with a properly configured sshd_config, the resulting configuration file is not so large that it benefits from being split up.
You have to deal with ordering issues, symlink management in some cases, and unless the "namespace" of sorting number prefixes is strictly defined, it's never something that's convenient or durable to "patch" new files into. The proliferation of 99_* files shows the anti-utility this actually provides.
I much prefer configuration files with a basic "include" or "include directory" configuration item. Then I can scope and scale the configuration in ways that are useful to me and not some fragile distribution oriented mechanism. Aside from that with xz I don't think I want my configurations "patchable" in this way.
Config directories are there to solve change management problems like idempotency.
If you have one big file then different tools, or even the same tool but different points of that tools life cycle, can result in old config not correctly removed, new config applied multiple times, or even a corrupt file entirely.
This isnt an issue if you’re running a personal system which you hand edit those config files. But when you have fleets of servers, it becomes a big problem very quickly.
With config directories, you then only need to track the lifecycle of files themselves rather than the content of those files. Which solves all of the above problems.
6 replies →
The .d directories are important on Debian and Ubuntu where packaging needs to provide different snippets based on the set of installed packages, the VM environment, other configuration inputs like through cloud-init and so forth, and update them during upgrades, but also (as per policy) preserve user customisations on anything in /etc.
Since pretty much every file has different syntax, this is virtually impossible to do any other way.
The .d directories make management via tools such as ansible much much easier.
You don't have weird file patching going on with the potential to mess things up in super creative ways if someone has applied a hand edit.
With .d directories you have a file, you drop in that file, you manage that file, if that file changes then you change it back.
1 reply →
The conf.d isn’t because the config file is large. It’s because it’s easier to disable or enable something with an “echo blah > conf.d/10-addin.conf” or an “rm conf.d/50-blah.conf” than it is to do sed -i or grep blah || echo blah >>
6 replies →
That's one of my issues with most Linux distros.
1. They add huge configuration files where 99% are commented out.
2. Sometimes they invent whole new systems of configuration management. For example debian with apache httpd does that.
I don't need all of that. I just need simple 5-line configuration file.
My wish: ship absolutely minimal (yet secure) configuration. Do not comment out anything. Ask user to read manuals instead. Ship your configuration management systems as a separate packages for those who need it. Keep it simple by default.
Totally, don’t use .d for ssh. The configuration is not that complicated. If it is, you’re doing it wrong.
Yeah, first-wins is definitely surprising. Off the top of my head, it feels like one would have to go out of one’s way to write a parser that does that (by storing an extra bit of state for each configuration item, and then checking it before setting the configuration item and toggling the state, rather than just applying the configuration item each time it is encountered).
Is there a good reason for this design? I can’t think of one, again off the top of my head, but of course I could be missing something.
It probably makes a bit more sense when you think about the fact that SSH frequently does a "try to match this host against a list of configured host-patterns" operation. In that case, "first match" is the obvious thing to do.
The SSHD internally has a config structure. Initially, it's all initialized with -1 for flags/numbers/timeouts, and with NULL for strings. When an option is parsed, there is a check whether the config structure's corresponding field is -1/NULL (depending on the type) before the parsed value is saved.
Another program with first-wins I've seen used dict/map for its config so the check was even simpler: "if optname not in config: config[optname] = parsed_value".
iptables uses first-wins.
https://serverfault.com/questions/367085/iptables-first-matc...
It's actually the simplest scheme. Reparse from the top whenever you need to query a setting. When you see one, exit. No need to even bother to store an intermediate representation. No idea if this matches the actual ssh implementation, but that's the way many historical parsers worked. The idea of cooking your text file on disk (into precious RAM!) is fairly modern.
Nope, the actual ssh implementation parses all the config files once, at the startup, using buffered file I/O and getline(). That means that on systems with modern libc, the whole config file (if it's small enough, less than 4 KiB IIRC?) gets read into the RAM and then getline() results are served from that buffer.
The scheme you propose is insane and if it was ever used (can you actually back that up? The disk I/O would kill your performance for anything remotely loaded), it was rightfully abandoned for much faster and simpler scheme.
2 replies →
Loading up your parsing code and reopening the file every time a setting is queried sounds to me like it would increase the average memory use of most programs.
15 replies →
There is a nice sshd option (-T) that tells you what it's really doing. Just run
Except that doesn't tell you what it's doing, that tells you what it _might_ do, if you (re)start the server.
sshd -T reads the configuration file and prints information. It doesn't print what the server's currently-running configuration is: https://joshua.hu/sshd-backdoor-and-configuration-parsing
That's why I only use socket-activated per-connection instances of sshd.
Every configuration change immediately applies to every new connection - no need to restart the service!
2 replies →
Yes. Run this as a validation step during base os image creation, if such image is intended to start system with sshd. That way you can verify that distro you use did not pull the carpet from under your feet by changing something with base sshd config that you implicitly rely on.
The only time I hear or see anything about cloudinit, it is always a problem. Nobody ever said "we don't need worry about that, cloudinit takes care of it".
What good does cloudinit do really?
In this particular case cloudinit presence in the story is incidental, delivery mechanism of said config file could have been different.
It's useful for initializing state that could not have been initialized before booting in the target environment. Canonical example, I guess, being ssh server and client keys management, but the list of modules it implements is long.
Provides a moderately-configured starting point for new cloud VM deployments without requiring custom images
> Nobody ever said "we don't need worry about that, cloudinit takes care of it".
Well, why would it come up? You don't need to worry about things you don't need to worry about.
Of course the order matters, that’s why the file names have numbers in them.
I think their surprise comes from earlier config wins conflicts, rather than the other way around. That's not reflected in the title.
I initially read "the order of files in /etc/ssh/sshd_config.d/" to mean the order of files in the underlying directory inode, i.e. as returned by `ls -f` — and thought, "oh god"... But the lexicographical order, that's not too surprising.
yeah, this is confdir 101...
but i guess learning is better late than never type of thing.
also what confuses people more on this is that openssh is properly designed, so configs are first seen wins. exactly so that file 0_ wins from 99_... but most people see badly designed software where 99_ overrides 0_. openssh way is exactly so it works best with confir or ssh/options where it matches by hosts and you can place the more important stuff first without fear defaults will override it.
I've made a big stink about this last time: https://news.ycombinator.com/item?id=42133181
They've updated the documentation on /etc/ssh/sshd_config https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/20...
I don't like these systems where configuration is built from a million separate files. They're unpleasant to work with.
The best reason to do it this way seems to be that files are the unit of package management. Perhaps we need a smarter package manager.
My nginx.conf life got better when I deleted sites-available and sites-enabled and defined my sites inline in nginx.conf.
The only thing worse is when the configuration is actually a program that generates the configuration, like in ALSA.
And the only thing worse than ALSA style is Xorg style, with a default configuration generated by C code and you can only apply changes to it without seeing it. Xorg also has this weird indirection thing where the options are like Option "Foo" "Bar" instead of Foo "Bar", but that's a nitpick in comparison.
Most of my configuration files are in one file, but there are cases where it makes sense, such as /etc/modules-load.d, for one.
Curious, since when is directory "/etc/ssh/sshd_config.d/" a thing ?
I checked on my OpenBSD (7.6) System and Slackware (15.0) and that directory does not exist. I checked the man page for sshd and there is no mention of that dir.
Is this a new thing/patch Linux people came up with ?
It might not be an OpenBSD thing. It may be a Debian/ubuntu-ism.
OpenBSD is the author of OpenSSH and yes they support the Include directive in sshd_config but they do not use it in a default install.
1 reply →
Gentoo started doing this last year and I absolutely hate it.
This is weird. I've been hitting funny problems while trying to get ssh to authenticate (using passwords) via. a keycloak instance. I've been trying to do using PAM script but have been pretty unsuccessful till now. Apparently, they don't play nice together.
This is interesting, usually it's the latter because the config is ran line by line
Also, if it's not too much trouble, would someone help me understand why such files are required to start with numbers? In this case it's 10-no-password.conf.
I have noticed similar structure for apt and many more packages
A lot of software which reads drop-in files will load them in numerical (or indeed alphabetical) order. Obviously this is important if the order your config files are loaded in matters, but otherwise it's just become a convention so people do it even if the load order doesn't actually matter.
Typically, config files are merged into one by loading them like ./conf.d/*, the order being determined alphabetically from their file names. You do not need to use numbers but they help to see that order.
It's a pet gripe with cli apps that do first wins as well.
You should be able to set
And still write
espeak suffers this affliction
Same if you use ~/.ssh/conf.d/.
I have been bitten by this before. :(
Hence the tradition of numeric file naming in *.d directories.
this is true of all 'config.d' schemes, and why most such schemes suggest/use number-name.ext style filenames to deal with sorting.
title could use a clean up
[flagged]
In what world is ssh obsolete?
What made it obsolete?
I'm trying to brainstorm an answer. My best guess is that SSH is obsoleted by disposable instances. You can spin up a new instance for every version of your configuration, transition to it, and dispose of the original (or set it aside or whatever). That way, you could probably have a reasonably complete tech career and only ever use ssh as an implementation detail of git.
lol
Yikes. What a nasty surprise. These tools really should be phased out in their current form.
RTFM
No tool can protect you from your own assumptions about how said tool works.
In other words, prepare for maximum surprise? As a defensive posture in a hostile or random environment, that makes sense.
But as a design approach, most designs go for the “principle of least surprise.”
And that’s how I read the original comment: a well designed system wouldn’t do this. Joke is on them, though, because nobody designed this.
3 replies →
I have a feeling you were waiting to say that for a while. :D
Glad to make your day. You are welcome.
Ultimately we all end up reading the manual. I'd still prefer if I didn't have to remember how a certain C stdlib function works vs. what seems intuitive.
But that's a lost cause with a lot of people. They'll happily point out how "intuitive" differs among different groups of people and all that, merrily missing the point on purpose.
Oh well. At least I found out without locking my self out of my servers.
5 replies →
Every time I say RTFM (like in the case of strtok), I get down-voted. Some tools really cannot be dumbed down, and they should not. I do not know why people have an aversion to reading documentation. It is bad.
In the case of strtok, I am not going to implement my own if strtok does what I want it to do, and behaves how I know it does. Why would I?! Sometimes what I need is strtok, sometimes strsep, sometimes I may use strtok_r.
9 replies →