Lessons From AppBrowzer Referral Program — 300k to 1.5 Million Users in 21 Days (Part 1)
Updated: 5 days ago
This is the first post in the series of three blog posts on the AppBrowzer Referral Program.
January 8th, 2019, as I walked into my office, one of the backend engineers told me that some of our servers are down and AppBrowzer App isn’t working. He wasn’t able to figure out the issue. I told him to check the logs in every server and see if he finds anything.
I sat on my desk thinking that this could be one of those crypto-jacking attacks that we had faced a few months ago. To rule that out, I logged into one of our main servers and checked.
Thankfully, it wasn’t the case.
In my inbox, there was a flood of emails from our users with the screenshot of our App showing “Unable to connect to server”.
On investigating further, I noticed that we were getting close to 500 requests per second on our web server — which was a lot. I checked if all those requests are coming from same the IP address or a set of IP addresses to rule out any other type of denial of service (DoS) attack. It wasn’t the case.
40,000 Signups in less than 24 hours
Then I opened up the Firebase console, and was surprised to see 7000 active users in the last 30 minutes!
I realized that our GPay like scratch cards-based referral program that we had released(silently) two weeks ago has gone viral. I was amazed and worried at the same time as we were still testing the feature out and many bugs had to be fixed!
That day we got more than 40,000 signups, which was 10x of our normal daily sign-ups, and then it exploded the next day giving us 100,000 signups! (the Playstore screenshot above is in UTC, check out the Facebook Analytics)
Sleepless Nights in our office
It took us four sleepless nights to make our backend capable to handle such massive traffic.
Hundreds of lines of code were rewritten, bugs were identified and fixed, servers were upgraded, and much more.
Below are a few pictures that were taken during that time. Everyone in the company contributed by going out of their way.
