Comment by rainforest
2 hours ago
I'm quite surprised to see the need to debug a live server here. I'm of the belief that the need to repro a problem locally and using a debugger lead to better understanding. SSHing into boxen feels like a cowboy behaviour on a modern stack - it shouldn't be necessary with competent observability and unit tests.
Sometimes you need to debug the observability stuff a little.
As a general rule, ssh'ing into prod is a terrible idea. Getting into a pre-prod box to figure out why metrics aren't getting pushed and trying something quickly before you go back to making the changes you need to push into the repo, less so.
I see that regularly at startups or other environments where the developers aren't necessarily professional software people (especially ML people!) The update-run-debug cycle is how they think and operate, at every level including on prod servers. Moving beyond that tends to require quite a bit of knowledge and infrastructure, and which infrastructure you need also requires knowledge.