← Back to context

Comment by JackFr

6 days ago

I have a strictly hobby web app that I work on. 6-7 years ago I inadvertently pushed AWS email service credentials to GitHub.

Half an hour after the push I got an email and text from GitHub that I had exposed credentials. I quickly logged in to my AWS to turn off the service, to see that AWS had suspended that service because the bounce rate on the 80000 emails sent in that 15 minute period was too high. It was crazy just how fast it was exploited.

>> It was crazy just how fast it was exploited.

People underestimate the speed, but also the number of pivots that advanced attackers will make. Sure, these kinds of problems are easy to exploit, but with major organizations that employ reasonable defenses, the attackers will pivot through 50+ exploits/machines/layers to get to the target(s). This can take place over weeks or months.

  • It still doesn't make sense that advanced attackers would go to those lengths in order to... refund the customers.

    • There are lots of smart kids who don't particularly need reasons for causing mayhem. Suppose it was somebody profit-motivated though. They might be:

      1. Distracting from a more important vulnerability

      2. Later contacting customers, advising them of the "accidental" refund and redirecting them to a more appropriate payment mechanism (one without the KYC Stripe does, were they to try to steal funds directly)

      3. Testing stolen credit cards before using them elsewhere

      Etc. Scamming people is a big industry, and not all of the plots are immediately obvious.

Here's an amusing thing to try on anything with SSH exposed. These log files go back a month.

  # zcat -f /var/log/auth.log* | awk '/sshd/ && /Invalid user/ && $6 != "from" {print $6}' | sort | uniq -c | sort -bnr | head -n 30
     5190 #redacted: my domain name
     3063 admin
     1891 #redacted: another domain name
      931 user
      724 ubuntu
      287 test
      268 solv
      206 odoo15
      200 solana
      197 sol
      184 ubnt
      173 wialon
      170 Antminer
      169 guest
      168 odoo17
      159 oracle
      157 postgres
      151 git
      150 support
      142 ftp
      135 ftpuser
      120 debian
      118 pi
       91 nginx
       85 baikal
       82 docker
       81 perl
       74 operator
       74 deploy
       72 dev

  • Interestingly, I have a server that only has IPv6 SSH open to the outside world, and it has exactly zero that aren't me fat-fingering a password. It does have an externally visible hostname, which says to me that the bots aren't looking at hostnames for SSH, just IP(v4) addresses.

  • Meanwhile on my publicly available Oracle VPS...

      22307 admin
      19668 user
      15396 ubuntu
       7038 user2
       6954 test
       6375 debian
       3938 ftpuser
       3433 postgres
       3416 oracle
       3076 deploy
       3003 steam
       2917 user1
       2830 dev
       2570 test1
       2352 es
       2187 server
       1957 hadoop
       1680 alex
       1676 guest
       1517 testuser
       1494 sammy
       1392 mysql
       1228 minecraft
       1218 pi
       1184 support
       1148 sysadmin
       1140 ubnt
       1120 from
       1090 123456
       1035 test2

  • I'm wondering what 'seekcy' is. Possibly a Chinese security product?

      $ journalctl | awk '/sshd/ && /Invalid user/ && $6 != "from" {print $8}' | sort | uniq -c | sort -bnr | head -n 30
         34 admin
         26 oracle
         21 postgres
         20 user
         18 test
         18 seekcy
         17 ftpuser
         15 ubuntu
         15 inspur
          9 zabbix
          8 nginx
          8 mysql
          8 jenkins
          8 hadoop
          7 server
          7 nagios
          6 teste
          6 amax
          5 support
          5 backup
          5 administrator
          4 git
          4 demo
          4 a
          3 zyfwp
          3 usuario
          3 tomcat
          3 testuser
          3 test1
          3 teamspeak

    • Googling it points to a Chinese IoT company, so I am thinking maybe they have some IoT software with known vulnerability where they have seekcy as the ssh username that is being actively scanned for.

> It was crazy just how fast it was exploited.

The internet is a wild place in any aspect of it. You should try spinning up a random virtual private server on any cloud provider, plug in a public IP address,and listen to traffic. Sometimes it takes seconds to get your first traffic from vulnerability scanners.

  • This 100%. I'm in a space with developers and customers deploying web servers for the first time. This traffic freaks them out.

    Basically a simple server listening on a port will experience hundreds of random script-probing attacks per day. And if any of them show the slightest hint of succeeding then that escalates quickly to thousands per minute.

    You don't need a DNS entry to "expose" the IP address (there are only 4 billion). Moving to another port reduces, but doesn't eliminate the traffic.

    Telling people this freaks them out. But as long as the server is done well its just noise.

    • Yesterday it was 4 seconds from a LE cert -> scans for .env and other low hanging info leak/vulnerabilities from at least 4 different scanners.

      There are groups out there just looking at the certificate transparency logs to get the newly added certs to scan.

      1 reply →

    • Short related story: some customer wanted an API with a basic firewall, they said they don't need filter rules as it won't be used or something. I put a dumb API online (doing nothing) and showed them the request logs after one day. They approved the filter rules immediately.

  • Years ago back in 2001, I had a /29 giving my 5 real IP addresses from my ISP.

    Back then, I mostly only ran Linux, but for some reason, I needed to run something on a Windows machine, so started installing Windows 98 SE onto a machine, and set it up with a public IP address without even thinking about it. It crashed before it'd even finished installing [0], and investigation showed that it'd been whacked by an automated RCE exploit scanner.

    I immediately decided it was never worth the risk of putting a Windows machine on a public IP, and set up a NAT even though I had 3 spare public IPs.

    [0] There was already a published update to protect against this exploit, but I was scanned and hacked between the base install exposing the bug and the automatic update patching it.

    • Yeah, I recall Windows sysadmins pulling out the LAN cable at bootup, installing updates via floppy disks and reconnecting the LAN cable.

Github repositories have statistics about access. Anyone can test this.

Create a new Github repo, within minutes there's someone (or something) poking around it.

Similarly if you post any credentials, they'll either get nabbed and used or a friendly scanner service will notify you about it and will tell you exactly where the credential is if you make an account in their service.

I think common people also underestimate how fast a computer really is nowadays, because they only know bloated MS Windows machines.