Comment by orev
4 years ago
The bigger problem I see is that they’re clearly using production data on test systems (or using the production system for testing), including PII. This is a pretty big no-no and violates many security standards. I don’t blame the tester per se, but I do blame HBO for not having a process in place that prevents this kind of thing from happening.
Jump to conclusion and shame. Classic.
How about, just maybe, they use a mail list management system like millions of other companies and PII wasn’t available in tests at all?
Except they also sent emails to people who are not subscribers such as myself and there was no unsubscribe link in the email. There was also no header, footer or any branding at all in the email. The only content in the email was the single line of text that read:
"template is used by integration tests only."
None of that sounds like a mail list management system to me. Also nowhere does the OP appear to "shame" anyone. In fact the OP very clearly states they don't blame any person but that they felt fault lies in lack of process to prevent such incidents.
That sounds exactly like what you would send to a mail list management system. Since you’ve likely never used them, think of it like a black box you feed a template that has a bunch of vars you can reference like
“Dear {firstname},
Check out our new movie: Batman Undresses.
Thanks for being a subscriber for {accountlife}.
{termsfooter} {unsubscribefooter} {alternatelanguagesfooter} “
The whole point of a template is so you send to an entire distribution list with a single API call and the mail system handles rendering templates to per user emails, setting up the unsubscribe link, tracking pixels, etc.
> Except they also sent emails to people who are not subscribers
That has exactly zero relation ship to your name being in their mail distribution system.
2 replies →
No, no, that can't be. Don't be reasonable
This incident doesn't necessarily indicate that they were using prod data in a test system.
I can plausibly imagine that there's some separate system that takes an identifier for some list of customers, and some template, and blasts out emails. Such a system could exist to help manage compliance issues with e.g. unsubscribe requests.
If so, and with a few "shortcuts" taken in making test environments for integration testing, I could envision a scenario where this incident happens that don't involve the test having direct access to real user data.
HBO Max sounds like a big company (I've heard of HBO, and "Max" sounds big to me). But it is possible they are using something like Mailchimp for their mailing list and don't do it in house. I'm just guessing here - a quick look at the headers would reveal this.
And with a confusing and horrendous UI such as Mailchimp's, it's quite easy to send a test email to the "live list". VERY easy indeed.
We've done it twice now. Once to about 10,000 emails and another to almost the entire list of 800,000. Luckily the template we were testing was 95% complete and not many people noticed. It just looked like the email got truncated with gibberish at the end.
They use SendGrid. From the headers, abmail.mail.hbomax.com resolves to u6146175.wl176.sendgrid.net.
Or they ran a suite of integration tests against prod as a way to verify a release was working correctly.
but... why would you do that? production is not for that one
If you don’t test against prod your CD pipeline is missing a critical step.
1 reply →
+1. The first thought I had when I saw that email was that I felt bad for the tester/dev; it's not really their fault, but they're certainly going to get at least some backlash for this. Really, it's a director/VP level issue that this kind of mistake was even possible with my email address.
This tweet does at least give some some indication that they're not (outwardly) trying to throw their intern under the bus: https://twitter.com/HBOMaxHelp/status/1405712235108917249
It was IMO still an ill-considered tweet even if made in a jokey way. While no one was really hurt by this--OMG I got an extra email!? :-)--something more along the lines of Oops, sorry for sending out that test email by mistake. We'll be putting processes in place so it doesn't happen again.
Exactly this, though anonymizing production data properly so it can be used as test data is very hard.
Generating useful test data is much better, but it's hard to represent all the edge-cases you see in Production.