Running Apache On A Memory-Constrained VPS

Yesterday about a hundred thousand people visited this blog due to my post on names, and the server it was on died several fiery deaths. This has been a persistent issue for me in dealing with Apache (the site dies nearly every time I get Reddited — with only about 10,000 visitors each time, which shouldn’t be a big number on the Internet), but no amount of enabling WordPress cache plugins, tweaking my Apache settings, upgrading the VPS’ RAM, or Googling lead me to a solution.

However, necessity is the mother of invention, and I finally figured out what was up yesterday. The culprit: KeepAlive.

Setting up and tearing down HTTP connections is expensive for both servers and clients, so Apache keeps connections open for a configurable amount of time after it has finished a request.  This is an extraordinarily sensible default, since the vast majority of HTTP requests will be followed by another HTTP request — fetch dynamically generated HTML, then start fetching linked static assets like stylesheets and images, etc.  Look, of 43 requests, 42 were not the last request in a 3 second interval.  It is a huge throughput win.  However, if you’re running a memory constrained VPS and get hit by a huge wave of traffic, KeepAlive will kill you.

When I started getting hit by the wave yesterday, I had 512MB of RAM and a cap (ServerLimit = MaxClients) of 20 worker processes to deal with them.  Each worker was capable of processing a request in a fifth of a second, because everything was cached.  This implies that my throughput should have been close to 20 * 60 * 5 = 60k satisfied clients a minute, enough to withstand even a mighty slashdotting.  (That is a bit of an overestimation, since there were also static assets being requested with each hit, but to fix an earlier Reddit attack I had manually hacked the heck out of my WordPress theme to load static assets from Bingo Card Creator’s Nginx, because there seems to be no power on Earth or under it that can take Nginx down.)

However, I had KeepAlive on, set to three seconds.  This meant that for every 250ms of a worker streaming cached content to a client, it spent 3 seconds sucking its thumb waiting for that client to come back and ask for something else.  In the meantime, other clients were stacking up like planes over O’Hare.  The first twenty clients get in and, from the perspective of every other client, the site totally dies for three seconds.  Then the next twenty clients get served, and the site continues to be dead for everybody else.  Cycle, rinse, repeat.  The worst part was people were joining the queue faster than their clients were either getting handled or timeouted, so it was essentially a denial of service attack caused by the default settings.  The throughput of the server went from about 60k requests per second to about 380 requests per second.  380 is, well, not quite enough.

Thus the solution: turning KeepAlive off.  This caused CPU usage to spike quite a bit, but since the caching plugin was working, it immediately alleviated all of the user-visible problems.  Bingo, done.

Since I tried about a dozen things prior to hitting on this, I thought I’d quick write them down in case you are an unlucky sod Googling for Apache settings for your VPS, possibly Ubuntu Apache settings, or that sort of thing:

  • Increase VPS RAM: Not really worth doing unless you’re on 256MB.  Apache should be able to handle the load with 20 processes.
  • Am I using pre-fork Apache or the worker MPM? If  you’re on Ubuntu, you’re probably using the pre-fork Apache.  MPM settings will be totally ignored.  You can check this by running apache2 -l .  (This is chosen at compile time and can’t be altered via the config files, so if — like me — you just apt-get your way around getting common programs installed, you’re likely stuck.)
  • What should my pre-fork settings be then?

Assuming 512 MB of RAM and you are only running Apache and MySQL on the box:

<IfModule mpm_prefork_module>
StartServers          2
MinSpareServers       2
MaxSpareServers      5
ServerLimit          20
MaxClients           20
MaxRequestsPerChild  10000
</IfModule>
You can bump ServerLimit and MaxClients to 48 or so if you have 1GB of RAM.  Note that this assumes you’re using a fairly typical WordPress installation, and you’ve tried to optimize Apache’s memory usage.  If you see your VPS swapping, move those numbers down (and restart Apache) until you see it stop swapping.  Apache being inaccessible is bad, swapping might slow your server down bad enough to kill even your SSH connection, and then you’ll have to reboot and pray you can get in fast enough to tweak settings before it happens again.
  • How do I tweak Apache’s memory usage? Turn off modules you don’t need.  Go to /etc/apache2/mods-enabled.  Take note of how many things there are that you’re not using.  Run sudo a2dismod (name of module) for them, then restart Apache.  This literally halved my per-process memory consumption last night, which let me run twice as many processes.  (That still won’t help you if KeepAlive is on, but it could majorly increase responsiveness if you’ve eliminated that bottleneck.)  Good choices for disabling are, probably, everything that starts with dav, everything that starts with auth (unless you’re securing wp-admin at the server layer — in that case, enable only the module you need for that), and userdir.
  • What cache to use? WordPress Super Cache.  Installs quickly (follow the directions to the letter, especially regarding permissions), works great.  Don’t try to survive a Slashdotting without it.
  • Any other tips?  Serve static files through Nginx.  Find a Rails developer to explain it to you if you haven’t done it before — it is easier than you’d think and will take major load off your server (Apache only serves like 3 requests of the 43 required to load a typical page on my site — and two of those are due to a plugin that I can’t be bothered to patch).
  • My server is slammed and I can’t get into the WordPress admin to enable the caching plugin I just installed:  Make sure Apache’s KeepAlive is off.   Change your permit directive in the Apache configuration to

<Directory /var/www/blog-directory-getting-slammed-goes-here>

Options FollowSymLinks

AllowOverride All

Order deny,allow

Deny from all

Allow from <your IP address goes here>

</Directory>

This will have Apache just deny requests from clients other than yourself (although Apache will keep the connection open if you’re using KeepAlive, which won’t due you a lick of good since it will still hold the line open so that it can deny their next request promptly — don’t use KeepAlive).  That should let you get into the WordPress admin to enable and test caching.  After doing so, you can switch to Allow from All and then test to see if your site is now surviving.

Sidenote: If you can possibly help it, I recommend Nginx over Apache.  I use Apache because a couple of years ago it was not simple to use Nginx with PHP.  This is no longer the case.  The default settings (or whatever  you’ve copied from the My First Rails Nginx Configuration you just Googled) are much more forgiving than Apache’s defaults.  It is extraordinarily difficult to kill Nginx unless you set out to do so.  Apache.conf, on the other hand, is a whole mess of black magic with subtle interactions that will kill you under plausible deployment scenarios, and the official documentation has copious explanations of What the settings do and almost nothing regarding Why or How you should configure them.

Hopefully, this will save you, brave Googling blog owner from the future, from having to figure this out by trial and error while your server is down.  Godspeed.

About Patrick

Patrick is the founder of Kalzumeus Software. Want to read more stuff by him? You should probably try this blog's Greatest Hits, which has a few dozen of his best articles categorized and ready to read. Or you could mosey on over to Hacker News and look for patio11 -- he spends an unhealthy amount of time there.

19 Responses to “Running Apache On A Memory-Constrained VPS”

  1. Philippp June 19, 2010 at 10:09 am #

    Have you considered moving away from Apache in favor of a coroutine-based webserver? Especially with the memory constraints you mentioned, it’d be good to move some of that effort over so you’re CPU bound instead.

  2. Saiyine June 19, 2010 at 10:23 am #

    Have you tried the Cherokee web server?

    http://www.cherokee-project.com/doc/cookbook_wordpress.html

  3. kle June 19, 2010 at 10:37 am #

    I’ve ditched Apache for lighttpd and it’s now invincible on 128MB RAM VPS, which has most of it used by MySQL.

  4. gees June 19, 2010 at 11:02 am #

    Or you could use an event-based httpd (like nginx … or lighttpd, cherokee). No poor mans threading like prefork. And you can handle that number on connection on 64mb RAM.

  5. crc5002 June 19, 2010 at 11:23 am #

    Just nitpicking here, but you are using the preforked Apache because some parts of PHP are not thread-safe. So, Ubuntu won’t let you install the threaded MPM package alongside with mod-php.

  6. nope June 19, 2010 at 11:29 am #

    you could just throw a squid in front and let it deal with all of your problems

  7. Chris June 19, 2010 at 12:47 pm #

    Since you mentioned you were using nginx, I would suggest just changing your configuration to have nginx proxy requests for Apache. Doing this will allow you to have have Keep Alive available for all URLs you are serving, which provides a better faster experience for your clients.

    To make this change, switch Apache to run on a different port and then point nginx to that port:

    server {
    listen 80;
    location / { proxy_pass http://127.0.0.1:8080/; }
    }

    Performance should be good since nginx can handle essentially an arbitrary number of connections, and they will get multiplexed onto a relatively small number of Apache children (I use 10 Apache process on a 1GB SliceHost machine)

  8. a web developer June 19, 2010 at 2:03 pm #

    As a note,

    Apache on Windows running under the thread based mpm_winnt, does not have this problem with KeepAlives causing requests to pile back on Apache’s ListenBacklog.

    With a thread base model, which Windows is very good at (opposed to a process based model), you can increase the default number of threads by a very large factor without having much of an impact on the server’s resources.

    Try that with a process based mpm and you’re dead.

    I would set the KeepAlive to 1 second, look into dropping any modules you are not using (such as mod_python/ruby) which can easily double the process size and leak memory, and setting the mpm setting appropriately (to what the system resources allow).

    You also have other options such as php op-code cache, mod_expires, mod_cache.

  9. Ross June 26, 2010 at 12:39 am #

    I am using the same slicehost VPS as you (but smaller – 256MB) and was able to run nginx with php, mysql and WP. Mostly using the instructions from here: http://www.mensk.com/webmaster-toolbox/perfect-ubuntu-hardy-nginx-mysql5-php5-wordpress/
    The big memory usage is from mysql and PHP, but I do not use custom compiled binaries and my settings are probably not optimal.
    My keep-alive settings are 5s.

  10. Gabor Vitez August 14, 2010 at 10:05 am #

    I’ve used apache’s event MPM with great success.

    Moving from the prefork MPM + mod_php to event MPM + mod_fcgid + php as fcgi saved ~300 MB of memory on a busy server.

  11. Javier November 2, 2010 at 3:40 pm #

    Hi Patrick,
    Great article, just one quick question, what about the TimeOut directive. Any special recommendations?

    Cheers.
    Javier.

  12. Web Hosting Shared Hosting Domain November 6, 2011 at 10:31 pm #

    You’re really a excellent webmaster. The website loading speed is incredible. It seems that you are doing any unique trick. Furthermore, The contents are masterpiece. you’ve done a wonderful activity on this matter!

Trackbacks/Pingbacks

  1. what do you think about these laptops ? | Best PC TV Software - June 19, 2010

    [...] Running Apache On A Memory-Constrained VPS: MicroISV on a Shoestring [...]

  2. Running Apache On A Memory-Constrained VPS: MicroISV on a Shoestring : Popular Links : eConsultant - June 19, 2010

    [...] reading here: Running Apache On A Memory-Constrained VPS: MicroISV on a Shoestring 19 June 2010 | Uncategorized | Trackback | del.icio.us | Stumble it! | View Count : 0 Next Post [...]

  3. Listen to 'Running Apache On A Memory-Constrained VPS' narrated by professionals, from 'MicroISV on a Shoestring' - Hear a Blog - June 20, 2010

    [...] Yesterday about a hundred thousand people visited this blog due to my post on names, and the server it was on died several fiery deaths. This has been a persistent issue for me in dealing with Apache (the site dies nearly every time I get Reddited — with only about 10,000 visitors each time, which shouldn’t be [...] Original post [...]

  4. Tweaking Wordpress | ShadowLANs - June 30, 2010

    [...] http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/ Read more from Site Updates Tweak blog comments powered by Disqus var disqus_url = 'http://shadowlans.net/2010/06/tweaking-wordpress/ '; var disqus_container_id = 'disqus_thread'; var facebookXdReceiverPath = 'http://shadowlans.net/wp-content/plugins/disqus-comment-system/xd_receiver.htm&#039;; var DsqLocal = { 'trackbacks': [ ], 'trackback_url': 'http://shadowlans.net/2010/06/tweaking-wordpress/trackback/&#039; }; (function() { var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; dsq.src = "http://shadowlans.disqus.com/disqus.js?v=2.0&slug=tweaking_wordpress&pname=wordpress&pver=2.33&quot;; (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); })(); [...]

  5. Optimizing Wordpress on my VPS « Code and Me - August 11, 2010

    [...] http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memory-constrained-vps/ [...]

  6. Fast WordPress/Nginx setup on a cheap VPS « Senko's Blog - February 28, 2011

    [...] First easy thing is to use Ningx instead of Apache for the web server. This may or may not be an easy thing to do – if you need (or prefer) to stay with Apache, Patrick McKenzie has some good advice on how to avoid Apache crashing and burning under load on a small VPS. [...]

  7. Setup nginx and wordpress on VPS | Nginx Lighttpd Tutorial - August 20, 2011

    [...] Here is a excellent tutorial show you how to Setup nginx and wordpress on VPS: First easy thing is to use Ningx instead of Apache for the web server. This may or may not be an easy thing to do – if you need (or prefer) to stay with Apache, Patrick McKenzie has some good advice on how to avoid Apache crashing and burning under load on a small VPS. [...]

Loading...
Grow your software business:
(1~2 emails a week.)