Comment by sgarland
1 day ago
> Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions
IME at many different SaaS companies, the only one that had serious reliability was the one that had “archaic grey beard architecture restrictions.” Devs want to use New Shiny X? Put a formal request before the architectural review committee; they’ll read it, then explain how what the team wants already exists in a different form.
I don’t know why so many developers - notably, not system design experts, nor having any background in infrastructure - think that they know better than the gray beards. They’ve seen some shit.
> and your lack of understanding what a pod is or how to get your logs from your cloud.
No one said the gray beards don’t know this. At the aforementioned company, we ran hybrid on-prem and AWS, and our product was hybrid K8s and traditional Linux services.
Re: cloud logs, every time I’ve needed logs, it has consistently been faster for me to ssh onto the instance (assuming it wasn’t ephemeral) and use ripgrep. If I don’t know where the logs were emitted from, I’ll find that first, then ssh. The only LaaS I’ve used that was worth a damn was Sumologic, but I have no idea how they are now, as that was years ago.
Splunk was (and is) the gold standard for centralized logging. The problem with it now is mainly that it's crazy expensive, though the operational engineering burden in order to run it well is non-zero and has to be accounted for. But being able to basically grep across all logs on the whole fleet, and then easily being able to visualize those results, made me never want to go back to having to ssh somewhere and run grep manually. I could write a script to ssh to all the app servers, grab the past 15 minutes of requests, extract their IPs, and plot them on a map to see which countries are hot, but that would be annoying enough that I'd really have to want to do that.
Meanwhile if you have Splunk, you specify the logfile name and how to extract the IP and then append "| iplocation clientip | geostats count by Country" to see which countries requests are coming from, for example. Or append "| stats count by http_version" and then click pie chart and get a visualization that breaks down how much traffic is still on HTTP 1.1, who's on 1.2, whos is on 2, and who's moved to QUIC/3.