Comment by hibikir

1 day ago

And if you ingest patches in seconds, you now have an issue of checking that the patch is not poisonous. Many a modern outage comes from changes deployed too fast. Damned if you do, damned if you don't.

23 comments

hibikir

Reply
olmo23  1 day ago

if a CVE drops at 23pm, attackers have at least 8 hours before someone in the office has a possibility to take a look at it. nowadays, that seems to be plenty.

i don't have a solution.

  • saalweachter  21 hours ago

    I think "good security/engineering practices" still has a chance.

    Things like--

      * Minimizing attack surfaces by not running lots of software on the same server and not bloating feature sets beyond the minimum 
  * Limiting access so that most things can't be done by most users, there are *no* super-users, and escalating access is temporary ans requires multiple accounts to authorize
  * Encrypting data at rest

    All software has flaws and it's now easier than ever to find and exploit them, but this disproportionately hurts people who were only putting on a show of security and relying on a single layer or two of defense instead of developing a robust insider threat model (which helps with both insider threats and threats that get inside) and defense in depth.

    • grey-area  4 hours ago

      Completely agree there is lots you can do to shut down exposed surface area.

      Is encrypting data at rest useful when the app must have access to decrypt the data? What sort of scenarios do you use it for and where do the keys live?

    • lokar  21 hours ago

      Full zero trust internally. Both inbound and outbound proxies (different machines) to carefully limit access, and provide a place to mitigate attacks. Etc.

      If your plan is to quickly apply upstream patches, your plan is bad.

      4 replies →

  • wongarsu  21 hours ago

    Have on-call sysadmins or have them work in a three-shift schedule. That way there's always someone there to fix stuff as it happens, including deploying security patches or worst case taking affected systems offline until they can be patched

    This used to be common before devops and cloud took over. Looks like it might make a comeback. Maybe with a new fancier job title

  • avidiax  15 hours ago

    There's a really simple solution that people will hate: server goes offline at night.

    I've seen this a few times, and it's pretty annoying as a user, but ultimately it's not that bad usually. If you are ordering something at 3am, the physical shipment still occurs during daylight hours. Many banking operations are not actually instantaneous, they settle in a batch job overnight.

    And this doesn't have to be binary. Providing a read-only interface provides lots of value while having a reduced attack surface.

  • purplerabbit  21 hours ago

    People are gonna hate this but: we’ll need agents ready and waiting to autonomously patch and deploy instantly

    • Someone  21 hours ago

      We need to deploy essential services at least twice, apply CVE ‘fixes’ to a subset of them, and have tooling to diff their outputs.

      That tooling should be shut off from the internet [1], and if it uses a LLM that should be at least a few years old.

      [1] technically impossible if it also has the ability to veto responses, but the surface can be very small.