Things Are Not What They Seem
Astonishingly, the only reason my Ajax driven web app has been able to run consistently and successfully for the past two years is due to MySQL latency.
I know this because yesterday (with the help of Jonathon Hill) I optimized the database by moving it off the server and onto it’s own 4GB Rackspace instance. (Everything was running on a single 2GB Rackspace instance prior to this).
As soon as I got the new database up and running it was blazingly fast (using the Percona flavor of MySQL made it a breeze).
But today I noticed that the load average on the front end head server (now MySQLless) was running super high (100+) with customers being unable to login and generally use the service.
After some digging around I noticed that Ajax requests were being served wayyy faster allowing for each client to establish 5-10 concurrent ajax requests thus flooding the Apache child pool – amping up the load – and bringing down the site.
Semi-blocking Ajax Queue
To fix this I quickly implemented a semi-blocking queuing system for all Ajax requests.
asynchronousness (if that is a word!) concurrency.
I was expecting this to slow things down but it seems to have really speeded things up from both a user and backend perspective.
But perhaps even more importantly the server load average has gone from 100+ down to 0.2.
I can only assume the reason the system has been working for the past two years is because MySQL latency was forcing the Apache child processes to run slower, with a knock on effect of Ajax responses being slower, with a knock on effect of chained Ajax requests being delayed thus creating a less asynchronous environment.
In other words, the only reason my app worked for the past two years was due to a slow database!