Comment by jeroenhd
3 years ago
An important thing to know is that most automated exploitation techniques are only effective against old and outdated software. It's very rare someone will burn a 0day on the public internet.
My personal recommendations, ordered by importance, sort of, maybe:
- Minimize attack service: don't install it if you don't need it
- Assume you'll be compromised at some point. NSA servers get hacked. FSB servers get hacked. You're not better than them. Think of what you'll do when you get hacked: do you need to rotate keys, change passwords, delete tokens? It's better to have a plan, even if it's a vague idea, just in case.
- Think about who might hack you, and how. Are you an activist? You may have opponents that want to attack you directly. Are you just some guy with a blog? Focus on defending against your most probable attackers, like botnets and automated scripts. You can make grand plans for when the NSA tries to break into your secure server network, but in practice you should probably be fine if you just defend against botnets. You can find entire threat analysis guides on the internet but a quick think about "who will attack me why and how will they do it" will probably do.
- You probably won't get hacked. Don't become too paranoid; just taking the bare basic steps to prevent getting hacked will make you more secure than millions or even billions of hosts on the web.
- Install software in a way that automatic security updates will take care of common vulnerabilities (i.e. prefer apt-install over ./configure && make && sudo make install or curl | bash).
- Check how quickly your upstream sources (Termux, in this case) are issuing patches. I haven't checked Termux' security stats myself, but I'd personally try a Debian chroot over a Termux environment just because Debian has more funds available to provide timely updates.
- Principle of least privilege: barely anything needs root anymore. Ignore outdated guides and use separate users (or even random user IDs with systemd), it'll probably work. If your server provider gives you a root account, create a normal account, give it sudo permissions, and disable remote root login; root should only be for system maintenance and repair!
- Read the manual for suspicious config you copy/paste. Sometimes config is just data (hostnames etc.) but sometimes config disables or overrides default config; for those cases, think "why is this not on by default"
- When in doubt, update. Auto-update if you have to. I've run sudo apt update && sudo apt upgrade -y" in a cron job for years but there are better options for most server OS's. On Ubuntu (and probably Debian) run sudo dpkg-reconfigure unattended-upgrades to configure automatic security updates
- If you do install stuff manually (downloading a binary, curl2bash, etc.) then find out how to update such software and think of ways to automate the process
- Listen to warning messages. For example, many old Ubuntu guides will have you add PPAs in a manner that will give the security keys of a PPA the ability to sign packages from the main system repos. You'll get warnings if you use this method on a modern system and you should probably heed those.
- Reboot your server after important kernel updates. If you can afford the downtime, reboot after every update just to be sure; if you can't, check if any important patches were released during the past few days. You can often monitor the reboot status file (/var/run/reboot-required, for example) to see if a reboot is necessary.
- Make sure your services come back up after a reboot. One of my servers has a nasty firewall configuration bug somewhere that I haven't been able to track down that causes the firewall to refuse to start on boot. Better to find out that your server isn't reboot safe during a planned maintenance window than when you need the service and it suddenly stops working!
- An important file doesn't exist unless it's at three places, at least two which are in a different geographical location. Make backups!
- Test if you can recover from backups. An untested backup is a non-functional backup.
- Back up recovery/security keys to a safe location. Your fancy, unbreakable hard drive encryption is useless if you forget your password!
- When using Docker, make sure to update regularly. It's very easy to deploy a docker application and forget to update it for years!
- Even if nothing is going on, log in every now and then and check how your server is doing.
- Keep an eye on exploit/security lists for software that you expose publicly, especially if you're not updating them automatically. There are websites that will list all important new vulnerabilities for only certain projects (I use the RSS feed provided by the Dutch government, but others likely exist).
- When you see your server might be vulnerable, don't panic. Vulnerability descriptions often start with "an unauthenticated user may execute code and take over the system and turn the sky red and..." and end with "if the administrator has enabled the enable_apple_II_compatibility setting". For example, the Samba/SMB bug that allowed code execution on NAS devices only applied when the optional AppleTalk extension was enabled, which was commonly disabled by default.
Apparently there's a comment size limit, who knew! Anyway, continued:
- Did I mention updating? You'd better update! Did you update? Good, then you're probably fine!
- Install a firewall. The Internet may tell you that you need to learn nftables but in practice tools like `ufw` will protect you just fine. Work in whitelist mode for incoming traffic. Work in whitelist mode for outgoing traffic if you want to spend an hour every week debugging why your server went down again.
- Pick good passwords, or even better, don't use passwords at all. Use key based authentication for SSH, use WebAuthn/U2F/whatever for admin panels, make it as hard as possible to brute force your way in. Don't reuse passwords! You don't want to get hacked because your password showed up.
- Keep an eye on risky software. For example, WordPress itself is quite secure, but many of its plugins have been plagued with vulnerabilities.
- Make maintenance/updating your services easy. Sometimes that means you need to spend more time setting everything up. Sometimes that means configuring your system in a suboptimal way, or even disabling other security features. Locking everything down can make it impossible to properly update your software and software you don't update is just waiting to get exploited by a bot.
- Not a security measure per se, but useful to reduce log spam: change the SSH port from 22 to literally anything else.
- When serving anything UDP (DNS/SSDP/whatever) make sure to check if there's a DDOS potential and see if you can mitigate it. If you can't, see if you can whitelist destination/source IPs in your firewall.
- HTTPS is free and quite easy. Why not enable it? If you turn it on, you can use "outdated" security mechanisms like HTTP Basic Auth without hesitation!
- Make sure to migrate in time when a software package gets out of date. Ubuntu 18.04 will receive updates for years to come, but you should probably plan to migrate two years before the end date. The longer you wait, the more difficult migration will be, the longer you'll put it off and the more likely you'll run vulnerable, outdated software.
- Hack yourself. Run port scans against your own servers; you might just find that you forgot that Docker will bypass your firewall. That's how I found out! I've accidentally had stuff running publicly that should've been private for months because I didn't port scan after installing new software. Other tools include stuff like wordpress security scanners, OWASP ZAP, etc.
- Sometimes you'll find yourself in a tricky situation: one of your services is vulnerable but no update is out for your platform yet. Consider temporarily disabling the service if it's not _that_ important or restrict your firewall to accept only certain IP addresses if you can't live without it.
- If you're only going to use the service yourself, you might not need to expose it to the internet. Maybe you can use a VPN to only allow trusted devices to connect. Maybe you can use Tor to make a service available (from behind NAT, even!). Never fully disable authentication and such on trusted networks unless you don't care about getting hacked, though.
- If you feel like the software you're running may be a common target, read your web server logs. See if you're getting flooded with weird requests and Google the URLs to see if you might be vulnerable.
- Periodically check if software you've installed is no longer necessary. Removing unnecessary attack surface not only makes your server more secure, it'll also help you keep down the amount of maintenance you need to do!
- Install security updates. Really, that's maybe the most important part, after picking good passwords. An updated server is a happy server. In most cases, an updated server is an unhackable server!
- Don't fall for emails like "I found a vulnerability in your server, do you provide a bug bounty".
- If you write your own software, regularly update dependencies and run vulnerability scanners (preferably automated). Stuff like https://owasp.org/www-project-dependency-check/ isn't hard to integrate into your workflow and might warn you of vulnerable libraries you didn't even know you used!
- Terms to throw into Google if you're interested: WAF (Web Application Firewall), Fail2ban, Tarpit (networking), cgroups, systemd hardening, snort, suricata
Finally, for your personal devices: use a password manager with a Really Good master password and enable 2FA. For most services a good password is a password you don't know. Use software that's as convenient to use as possible without sacrificing security (i.e. use a password manager that integrates with your phone's/browser's autofill). Stuff like WebAuthn/U2F/FIDO2 can make 2FA very usable without having to copy a bunch of numbers every time you log in and they're available on more devices than you might think. You can go ultra secure (I, for one, enabled 2FA on SSH at some point) but the harder you make it for yourself, the less likely it is that you'll find yourself using your fancy security.
Some people will probably disagree with me on some points. Some of them are overkill, some of them are not strict enough. There's no one true answer for any of this, because every server is different and has different security requirements.
In case you feel overwhelmed: what I just typed out isn't followed by (way too) many companies. If you work outside a tech company I doubt the IT departments/people even know what company secrets are on what server. Companies will silently or unknowingly run vulnerable software for years without getting hacked! Your home server may be more secure than a critical application server that some Fortune 500 company relies on, simply because you can run updates and they're stuck with some outdated OS/software/hardware/network/configuration that they can't move away from without spending millions!
This is awesome! Thank you