Comment by cortesoft
4 years ago
Similar story... about 10 years ago, I had written a really simple script to email all our customers. It worked great for a long time, but then suddenly we went over 1000 customers.
My script was supposed to try to grab batches of 1000 customers and keep looping until it ran out of customers (signaled by having retrieved less than 1000 customers in my last request for the next batch of 1000 customers).
My script was missing the offset part of the query, so after we hit 1000 users, it just kept looping, sending the same email over and over to our first 1000 users.
I felt so bad that day. From then on, sending out emails was this whole huge process that involved queuing them all and then having like 6 people review to make sure we didn't mess it up.
When I worked on email systems, my worst nightmare was a Sorcerers Apprentice sort of problem.
oh man I just wrote a piece of code that lets me write any markdown, push a button and it just sends it out to our ~1-2000 users.
An hour later I just commented it all out, and wrote a note to myself: "if you need this, uncomment and push back to stage". Just having that code even sitting around makes me nervous
We introduced an allowlist on all our testing and staging environments to ensure that only certain recipients can get email. We also make sure that no email address in these databases would work, unless we really want to send to it.
We did the same thing after that incident. It’s worked beautifully.
I always was super careful with that stuff. The final send function wasnt unlocked until knew for certain all worked as intended.