Comment by owyn

1 year ago

This looks neat. I have to look up the very fiddly and unintuitive systemd commands all the time. service start? service.foo start? start foo.service? Oh right, sudo systemctl start service.foo

And the feedback is so bad. It should know everything in its own config dir and tell me how to do what I want to do. Was it enabled? I forget. How do I look at logs? Oh right journalctl. Also the layout of things with lots of symlinks and weird directories in places that annoy my 90's linux sysadmin brain. Why am I looking at /lib/systemd/system

I am annoyed by the redundant "systemd/system" directory name every time I have to go there. At this point, just promote it to /etc/systemd and build a better CLI.

As a very occasional linux sysadmin just trying to make things work, the "typing at a console" systemd interfaces are not fun to work with. Maybe nobody should be doing that. In an enterprise, sure that's different. I think interfaces should be human, and linux should still be fun.

> I have to look up the very fiddly and unintuitive systemd commands all the time. service start? service.foo start? start foo.service? Oh right, sudo systemctl start service.foo

I don't get this complaint. It's the same order as almost every other command-line utility that has subcommands: <command> <subcommand> <thing to operate on>. To me, that kind of consistency is very intuitive.

    systemctl stop my-service
    systemctl status my-service
    git add my-file
    git remote remove upstream
    apt install my-package
    docker run my-container
    adb push local-file remote-file

  • I feel the same way. The big part for me is that it tells us that owyn doesn't use tab completion if they're forgetting about the ".service" part. Sure, I don't remember either, I don't have to.

    I'll add the abstraction for anyone confused

      program [command [subcommand]] [flags] [object]
      e.g.s
      systemctl status sshd.service
      systemctl enable --now sshd.service
      touch -c test.sh
      echo 'Hello World'
      echo "Hello ${USER}"
      

    Anything in brackets is optional and might not appear or be available. By command I mean a category of commands. Such as 'pip install' vs 'pip uninstall', which are sub-programs inside the main program. But this can have layers such as 'uv pip install'. Often flags can be used in any order because you'll just loop over all the arguments but this is still the standard order.

    There's also the two actor pattern

      program [command [subcommand]] [flags] source destination
      e.g.s
      cp /foo/bar/baz.txt "${HOME%/}/"
      scp -i "${HOME}/.ssh/foo" ${HOME}/to_upload.sh user@remote:~/
      dd if=/dev/urandom of=/dev/diskToDestroy
      rsync /mnt/hdd1/ /mnt/hdd2/

    • Also the `.service` part is optional anyway for most commands (including the start, stop, restart etc. ones they use in their examples). Only commands that can operate on services and other non-service units require it.

  • When it comes to starting and stopping services I want the verb to go last. Way easier to press up, backspace backspace backspace o p to change service ssh start to service ssh top. This is a frequent pattern I follow as I start/stop/restart/reload. Having to go back at least one word adds keystrokes that aren’t necessary.

  • I think the previous post mentions the `service` command, which has `service foo start` pattern.

    I'm used to `service` command, so I have muscle memory which `systemd` breaks. The way `systemd` commands are laid out is better, it just messes with `service` command muscle memory.

    • I disagree that the way systemd is laid out is better.

      With initV, issuing many command in sequence to the same service (enable, then start… stop then start… etc) is much quicker and easier as it’s “up-arrow, control-W, type new command. With the systemd option I either need to navigate one word to the left (or more depending on how the shell is configured wrt word separation) before I replace the command, or I need to delete both the command and the service name, and retype more.

      It’s not a huge difference but small downgrades in ergonomics add up over time.

      7 replies →

    • It's been a decade at this point since the last big distro (and surrounding drama), Debian and Ubuntu following shortly after, moved to systemd.

      1 reply →

  • I on the other hand DO get this complaint. It’s less about the “word order” and more remembering the words to type. As a software engineer I have to google stuff all the time. I haven’t touched C#’s PLINQ in five years, I doubt I could do it from memory. Likewise, I can count on one hand the number of times I will set up a systemd service in a given year.

    Many (most?) of us are all over the place, expected to “wear many hats” and aren’t in a single IDE or language all day every day. Certainly if I were a “Linux admin” I would have systemd pretty committed to memory. Anyone else probably wouldn’t.

  • For me:

    * /etc/init.d/my-service stop

    and Ubuntu’s:

    * service my-service stop

    both lurk in my brain.

    • Sure, it's different from the old way, but I don't think "unintuitive" is the right word for that. systemd forced people to change their habits so that it could be more intuitive. Of course, people are going to disagree about whether it was worth it - it's the age-old question about breaking backwards compatibility for the sake of minor improvement. Personally, I got used to it pretty quickly and I like it more than the old commands now.

      4 replies →

  • Plus, you could always write a quick shell script to swap the arguments if it detects this specific failure. Inability to remember or learn which commands expect which arguments is, at some point, not the responsibility of the software to fix. Unfortunately, this field does necessitate some amount of memorization and ability to problem-solve.

  • If I had to guess it's because of the 'service' command, which goes 'service foo start'.

    It took me ages to unlearn that pattern for using systemctl, even though as you say: it's far more consistent

My biggest annoyance is "systemctl status" gives you just enough of the service's log to make the output take up most of the terminal each time you run it but never enough of the service's log to get a useful picture of what's actually happened with the service lately.

Not to mention unless the problem with the service completely prevented it from running (it advises some commands to run in that case) you're supposed to just always remember "journalctl -xeu $SERVICE" was the incantation, less you want to go look up the flags again or manually parse the entire "journalctl" output.

Overall I generally like systemd though. The syntax can just be a burden sometimes.

  • it's the same mentality that brought us the git design - the easiest, least typing options are rarely, if ever the thing you want to do.

    Instead, these invocations give cryptic messages, throw errors, or sometimes, even break things.

    The most common and helpful things are hidden deep behind multiple flags and command line arguments in manuals that read like dictionaries more than guides.

    I'm always at a complete loss as to how such decisions are made. For instance, "git branch -vv" is the useful output you would like to see, every time, that should be "git branch". Why not make the current output, "git branch -qq"? Is a humane interface too much to ask for? Apparently...

    I know people defend this stuff, but as a senior engineer in programming pits for 30 years, they're wrong. Needless mistakes and confusions are the norm. We can do better.

    We need to stop conflating elitism with fucked up design.

    •   > Why not make the current output, "git branch -qq"? Is a humane interface too much to ask for? Apparently...
      

      Yes, it is too much.

      You have the wrong mentality, and I hope this can help make your life easier. Programs are made so that the simplest option is the base option. This is because there is a high expectation that things will be scripted AND understanding that there is a wide breadth of user preference. There's an important rule

        DON'T TRY TO MAKE A ONE SIZE FITS ALL PROGRAM
      

      Customization is at the root of everything. We have aliases that solve most of the problems and small functions for everything else. You default to an non-noisy output, showing only __essentials__ and nothing more unless asked. Similarly, you do no filtering other than hidden files. This way, everyone can get what they want. Btw, this is why so many people are upset with default options on things like fdfind and ripgrep.

      For your problem with git, there are 2 solutions you have.

        # alias git branch using git
        git config --global alias.branch 'branch -vv'
      
        # write a simple function and add to .bashrc or .zshrc or .*rc
        git() {
            case "$1" in
                branch)
                    shift
                    command git branch -vv "$@"
                    ;;
                *)
                    command git "$@"
                    ;;
            esac
        }
      
        > We need to stop conflating elitism with fucked up design.
      

      The design isn't fucked up, it is that you don't understand the model. This is okay. You aren't going to learn it unless you read docs or books on linux. If you learn the normal way, by usage, then it is really confusing at first. But there is a method to the madness. Things will start making more sense if you understand the reason for the design choices. (In a sibling comment I wrote the abstraction to command patterns that makes the gp's confusion odd. Because systemd follows the standard)

      Side note: if you try to design something that works for everyone or works for the average person, you end up designing something that is BAD for most people. This is because people's preference is not uniformly distributed, meaning the average person is not representative of any person in the distribution. This is because anything that is normally distributed has its density along the shell while a uniform distribution has a uniform density all throughout it.

      19 replies →

  • In case you don't know, you can use the `-n` argument to `systemctl status` to tweak the log output, e.g. `-n0` to disable the log output and `-n40` to get more than the default 10 lines.

    • That's a great option to tweak the behavior and I hadn't known about it (or if I ever had, I'd well forgotten). Thanks!

      From the man page it ?looks like? if you want reverse or full then it's still off to the journalctl command and arguments but at least "-n9999" is better than "always 10 lines".

      3 replies →

    • Nice! I didn't know that was an option. Definitely something I should make configurable in `isd` :+1:

  • In my experience, most of the times a service fails, it fails on startup (misconfiguration, missing resources). So the status is OK. Also, you can always request more lines, if you think that's helpful.

    This wouldn't be my problem with systemd. Not by a long shot.

  • > My biggest annoyance is "systemctl status" gives you just enough of the service's log to make the output take up most of the terminal each time you run it but never enough of the service's log to get a useful picture of what's actually happened with the service lately.

    How about

    systemctl status foo | tail

> I am annoyed by the redundant "systemd/system" directory name

It's not redundant, you also have /etc/systemd/user (and /lib/systemd/user) where units that run in the user context (as opposed to system-wide context) are stored.

  • It's also worth noting that this is a fairly standard pattern in /etc

      /
      | etc
      | | fail2ban
      | | | action.d
      | | | fail2band.d
      | | | filter.d
      | | | jail.d
      | | ssh
      | | | ssh_config
      | | | sshd_config
      | | systemd
      | | | network   # network wide context
      | | | nspawn    # containers
      | | | system    # system wide context
      | | | user      # user wide context
      

    Personally I like it more than

      /
      | etc
      | | cron.d
      | | cron.daily
      | | cron.hourly
      | | cron.monthly
      | | cron.weekly
      | | ...
      | | firewall
      | | firewalld
    

    Keeps things less cluttered. Hierarchical categorization is >> than lateral

This is why I have instated a policy of using systemd --user services whenever possible.

If you don't need elevated permissions this is ideal.

All you have to do is enable linger using loginctl if you want your service to auto start as the user on boot unattended.

Your user services live in ~/.config/systemd/user

  • I tried, but found it very confusing around the user session. Apparently an environment variable need to point to a started dbus. I can't get consistent results over local session vs ssh vs su vs sudo

  • I used to do this but frankly it's easier to run a system-level unit as whatever user you want and keep all the files in /etc instead of scattered around /home.

    The user-level units are most useful when running an actual multi-user system. If you trust your users to not abuse them, anyway.

I find that at least systemd means that it's consistent across distros. I spent way more time looking up this kind of thing when every distro rolled their own init system.

Author here: Yeah, I agree.

It is a bit weird. On the one hand, I understand that it makes sense to have [command] [verb] [object] on a "logical" level and that viewing logs should be a separate command (`journalctl`), but it is definitely not ergonomic. Especially if you frequently have to switch between start/stop/restart.

> As a very occasional linux sysadmin just trying to make things work, the "typing at a console" systemd interfaces are not fun to work with. Maybe nobody should be doing that. In an enterprise, sure that's different. I think interfaces should be human, and linux should still be fun.

This was precisely the case for me. I "enjoy" playing around with systemd and am super interested in better understanding it, but the feedback loop just felt sooo slow. So hopefully this TUI can make it "fun" again :)

  • When I'm really working with a service and get tired of typing sudo systemctl restart or status all the time, I'll just do a quick alias or two right then and there.

    alias s=sudo systemctl status alias r=sudo systemctl restart

    And if I'm only working with one service, I'll throw the service name in there too.

It is a bit crazy to me how everyone says “dont use cron systemd is in now” but cron just does what it says on the tin with no problems. I have lines that work fine ran in script or on my crontab but when wrapped in a launchd command no longer work (log says things work until the db is to be updated which tells me launchd ran processes lack sufficient permissions perhaps to update my db but its not clear why this is the case or how I can elevate launchd sufficiently.

  • > cron just does what it says on the tin with no problems.

    I can name a rather large problem with cron that systemd timers solve handily: long-running job duplication. When jobs take longer to run than the space between their triggering times, duplicates start piling up. I've had to rescue numerous systems from such states, which are difficult to detect until things have gotten quite bad. Sure, you can write a bunch of boilerplate to handle this yourself with cron, but with systemd timers it's all handled for you along with other niceties like capturing all output in journald and ensuring that the next run starts at soon as possible.

  • Launchd is not used in systemd - why would anything be wrapped in it? The config is simple - User for the user and Group for the group. You can print it the result of "id" if you're not sure what the result is.

    • Sorry I conflate the two since they share a lot of similarity. Launchd is what I use as its a macos system.

  • > ...cron just does what it says on the tin with no problems.

    Yeah, tell me about it.

    We had a production-down support ticket filed for one of the things that I work on at $DAYJOB. The customer's VM's disk was full. Why? Because the 'timer unit' that was supposed to run logrotate every day had never been run and was never scheduled to run. (The VM had been up for a month at this point.) No other customers had ever reported this issue, and we'd not changed anything about that timer unit or what it is supposed to run in ages.

    No amount of gyration or agitation with 'systemctl' and friends OR rebooting of the VM kicked the cron replacement into proper functioning. 'logrotate' was simply never being scheduled to run. This fucker was WEDGED, and the tools weren't helping us understand why.

    We read through all the docs on all the various kinds of units and don't see what we're doing wrong. We do a BUNCH of digging, and find an -IIRC- open Github issue from years back where someone was running into this problem. More or less the last word on the issue was Poettering saying something like "Well... actually, now that we've said that that particular cascade of options isn't actually supposed to work, now I'm not so sure that they're NOT supposed to work.". And that was -apparently- that.

    IMO, when your cron replacement can be easily configured in such a way as to never even try to run the thing that you scheduled it to run, you really need to go back and rethink how you've built your cron replacement.

    • Considering how many things of questionable veracity are said about systemd in general, and Lennart in particular, I would like a link to that issue before I take your characterization at face value.

Fiddly and unintuitive? A lack of exposure to something does not really allow room for valid criticism of the thing.

Just a tip, if you remember the service name, status will show the directory the unit file is in. That will hopefully get you over your issues with directories. Complaining about Linux directories seems weird though. Have you looked at... anything else at all?

I just wish systemd allowed abbreviations for the subcommands, like "ip" (and git, for long options).

I haven’t had much trouble with the cli, but it is kinda wordy. I made this alias:

    alias sc='sudo systemctl'

Now it matches the sc utility introduced in Windows many years back. Then remember the verb goes first.

  • Yeah, and you can omit the ".service" suffix, and you could use ".timer" suffix, too, if you have a timer for that service.

    As much as I do not like systemd, I do not think its commands are an issue, and I use "systemctl status" and "journalctl" as well with some flags at times.

I am going to be that guy, but I went full runit for similar reasons.

Give me files for my logs, give me a single place for my service definitions, be simple.

After years of systemd, I have never really "got it", while it took a weekend for runit.

It's not always our choice, corporate world is something else, I know...