Comment by malikolivier

At a previous start-up, we did a similar mistake.

This was one of the worst day of my "engineer" life. I was very young and inexperienced at that time.

We were running a Ruby-On-Rails app with push notifications. We had an "on creating" life-cycle hook on a model recording an event, so that every time a new model was saved, a job sending a push notification (android or iPhone) to the appropriate user would the triggered. This would notify the users that the "event" was done.

The implementation was wrong. Instead of an "on creating" hook, the push notification logic was added in an "on saving" hook.

As the model was supposed to only ever be saved once (on creation), this worked for a while, and no-one ever realized. You now guess what's going to happen.

Then after some time, came the day when we needed to do some data migration. Easy right? The data-migration included updating some data in the the above model in question (all rows must be updated). We run a ruby script to update the model.

Everything was working in staging environment. So we release in production. The moment we release, we sent hundreds of push notifications to each of all our clients (including to our own phone, as we were using the service). Basically everyone was receiving as many push notifications as the number of "events" they completed until now.

Immediately, all the phones in the office (CEO's phone, sales rep, etc.) started to ring with complaints from customers asking what was going on. The CEO was as angry as you can imagine. But no time to be angry, as everyone needed to apologize and explain the situation to all our customers.

Basically the whole team was at fault for letting the problem slip through the code review. From then on, we improved code review process, and decided to include push-notifications in staging environment too.