Two weeks ago, our website became unstable and went down multiple times due to heavy load, particularly during lunch hours. And for some of our clients who have a lot of data, searching became very slow.
We do take this very seriously and have been working non-stop since then to implement certain optimization measures. Because we had to implement lots of system wide changes quickly to alleviate the pain, it caused the system to be buggy for the last 2 weeks. At this time, we have mostly finished with the site optimizations, and fixed the majority of the bugs. We apologize for the negative impact. We also thank you for being patient with us and helping us fix the bugs.
If you are curious, continue reading to find out what was the cause, what we have done to fix it, and what’s our long term strategy going forward.
What Caused The Slowness and System Downtime?
There are 3 main reasons that caused everything.
1. For the most part, it was caused by our own growth. We have been growing steadily since we started last February. Each premium client we bring on-board, not only add many agents/brokers who use the system actively throughout the day, they also bring on much more traffic from perspective renters through their ads and company website and personal websites.
2. PerfectUnits.com – We are now getting about 400 – 500 unique perspective renters a day searching for apartments. The site was not designed very well originally to handle the load. Not only was PerfectUnits.com became excruciatingly slow itself, it was also causing YGL to go down with it.
3. Data Complexity – As we introduced more listing sharing capabilities between accounts, our data structure has grown ever more complex. For example, for the same listing managed by a landlord, each agency / account has their own set of photos, descriptions, features and tags. Each agent can also have their own descriptions and ads. That’s a lot of copies of the same listing.
What Have We Done For The Last 2 Weeks To Solve The Problem?
We did 3 things to optimize the speed.
1. Server Upgrade – We upgraded our server from a quad core with 4 gigs of ram to a 8 core with 8 gigs of ram. A more powerful machine means we can handle more concurrent requests and processes. However, this was not nearly enough. We have already passed the stage that we can just throw more hardware at the problem.
2. Database Optimization – The YGL system is a very database intensive system. Each search, each listing view are all hits to the database. We were able to optimize our database queries to reduce load times. You should see the listing searches being faster.
3. Cache – Lastly and most importantly, we implemented a very complex caching system, which allows us to store commonly used items (like listings, comments, features, photos, leads) directly in memory, hence lightening the database load drastically. This was fairly challenging and caused the majority of the bugs you might have experienced.
What’s The Long Term Plan?
With the current measures in place, it should last us comfortably through this season. We’ll be continuously monitoring our system load. When necessary, we will employ multiple servers and load balancing.
Again, we apologize for the down time and the bugs that came with it, and thank you again for being patient with us.
