← Back to context

Comment by anyfoo

4 years ago

Now imagine it didn't just say "test" or "asdf".

I still might sometimes put a funny/lighthearted twist on debugging logs that nobody outside the company would see, but I never put in swear words, condescending things, or anything else I wouldn't be okay with if it was accidentally logged in production.

I fortunately did not have to make that experience myself, but I've seen too many supposedly "internal only" messages turn uninternal.

I setup an apache webserver one time with a single index.html with the word "FOO" in it.

It was a placeholder until the software dev team responsible for the server could deploy software onto it (they asked it be setup with something in the index.html so they could confirm it was running, they didn't care what)

The networking team in advance of that wired it up to some load balancers.

They took an IP which had formerly been a decommissioned cluster of webservers serving the main website for the entire company (an internet retailer named after a river in some other country, you've probably never heard of them).

The DNS loadbalancers found the old IP (it had never been deleted from their configs which was the root cause) was now live and it was REALLY fast, so they shunted most of the traffic over to it.

Created a sev1 outage for "users reporting 'foo' on the website"

I'm happy I kept it professional that day.

  • future alien FOO technology

    > Let's say 99 of your 100 machines are taking 750 msec to handle a request (and actually do work), but this single "bad boy" [machine] is taking merely 15 msec to grab it and kill it. Is it any surprise that it's going to wind up getting the majority of incoming requests? Every time the load balancers check their list of servers, they'll see this one machine with nothing on the queue and a wonderfully low load value.

    > It's like this machine has some future alien technology which lets it run 50 times faster than its buddies... but of course, it doesn't. It's just punting on all of the work.

    -- https://rachelbythebay.com/w/2015/02/16/capture/

    • they were all returning 200s though for "GET /", so they were perfectly healthy and happy...

      optimized as shit for that one request...

I learned that lesson the hard way when I was a teenager.

I'd semi-automated the generation of emails to parents about how their kids were doing at summer camp. It was basically just a script that asked various questions and pulled various data about the camper from the database, then used that to generate a letter that could be used as a starting point and edited to be more personal (or, if in a rush, just sent as-is).

Long story short, due to user error, a value of 0 was set for a camper's behavior/politeness/helpfulness rating, which resulted in a joke sentence that I'd written as an Easter egg getting slipped into that particular report. Cue egg on my face when a parent calls in, baffled about the otherwise normal report containing a casual aside about how terrible their nice little girl was.

After that experience, I always assume that every string in the code will inevitably be seen by a real user/customer.

  • > assume that every string in the code will inevitably be seen by a real user/customer.

    That's a good rule, and the same principle can be applied to communications in general. In fact, I have the three rules printed on my office door:

        Dance like nobody's watching
        Encrypt like everybody is
        Email as if it were read out loud at a deposition

  • The first production outage of my sysadmin career was when I crashed a university email system at a summer camp (I was a kid attending, probably very early 1990s?) by circumventing the block on emailing the entire university, using the Mac keyboard shortcut Cmd-A to select all in the recipient field in the cc:Mail UI. I apologized for childishly causing harm with something I thought would be funny, and so they decided not to kick me out.

As a former HBO dev, can confirm that the majority of test cases or random debugging messages are Game of Thrones quotes.

This, a thousand times. I have a standard bit of lorem-ipsum-style content for this situation (whose origin I have forgotten and cannot credit, lo siento):

    This is a test.
    This is only a test.
    Had this been a real emergency, we would have fled
    and you would not have been informed.

  • To be honest things like that may sound funny to you, but it's easy to take things the wrong way. I'd leave out the last sentence.

This is a great point, but it's a good idea to be cautious about anything you write in a work context - like in IMs, commit messages or emails. I've been working by the principle that I won't write out things that I would be uncomfortable being confronted with out of context - say if a newspaper got hold of some leaks, or some court case caused a company to have to turn over emails etc. This doesn't mean I don't swear or joke ever, but it means that I don't send stupid one-off throway dumb things. So "The fucking $foo-service is down again, I've no idea what caused it" is fine, but "Well $foo-service is down, hope our idiot customers don't figure it out before we fix it lol" is obviously a no-no (not least because that's not actually how I joke or talk anyway).

> I still might sometimes put a funny/lighthearted twist on debugging logs that nobody outside the company would see, but I never put in swear words, condescending things, or anything else I wouldn't be okay with if it was accidentally logged in production.

A previous company I worked in once managed to send the following in patch notes:

"So sit back, smoke a spliff and stop worrying you nappy wearing fucktards"

... so, yeah, good advice.

Yep, a colleague of mine was running a quick test (this is a long time ago though) to see if some new feature, which sent email, was working and, because it was not supposed to actually sent email anywhere else than to him, he quickly put something with 'fucker' in it, pressed test and... yep all customers received that. Luckily we were in NL with only NL people receiving it and they mostly just found it funny. My colleague nearly got a coronary of course and never did that again.

Not using swear words was a big change I had to make when going from working alone as a freelancer to working with other engineers and testers. Where I work we delegated some frontend stuff to a team last year who would test with things like “God damned” and when I saw that I realized how much I’d matured as my first thought was, “unprofessional.”

  • One team I worked on put a profanity filter in the source code checkin filter so that we could put curse words in whatever code we were writing and make sure it didn't get merged until that code was refactored.

    • How tolerant was it against typos? For example: "fuk". Or did it contain every variation already?

I used to write silly or rude things when testing, however I once found out a client of ours was getting tagged onto all of my emails via bcc, From then on I always use "This is a test, Please disregard"

I worked with a guy that was convinced at the last second to change his test email from a swear word to just "test". When he accidentally sent that to the customer base the CEO tore him a new one; it could have been so much worse.

I learned my lesson from him and now I keep these things very clean, just in case.