Re: recommended maxconn setting

Hi Guy,

On Tue, Jul 27, 2010 at 12:55:54AM -0700, Guy Knights wrote:
> We have a setup currently with haproxy in front of 5 webservers
> running nginx and apache. Nginx set up to proxy dynamic requests to
> PHP and serve out static content itself. It seems to work well, and
> over time we've adjusted the maxconn settings gradually to where it's
> now configured with 25000 maxconn, with each of the 5 web servers set
> to a maxconn of 5000. This is probably overkill as haproxy is usually
> serving no more than 500 current connections in total, but the web
> servers are running on some pretty powerful hardware so my thinking is
> that it doesn't hurt to have the option there in case load increases
> over time (which we forsee being the case).

Yes in fact it can hurt. The main goal of the server maxconn setting is precisely to prevent the server from crashing or swapping due to too many concurrent connections. While nginx will have no problem handling thousands of concurrent connections, it can become a different problem when those connections are dynamic because the amount of memory consumed by one connection it a lot higher than what it can be for a static file. When you see some apache processes running at 20 MB per process on PHP or Perl, you can easily imagine that even with 12 GB of RAM, they won't ever support more than 600 concurrent connections.

> The web servers are 16 core, with 12gb ram, while the haproxy server
> is quad core, also with 12gb ram. This being the case, can anyone
> offer any advice as to whether I should tweak the maxconn settings -
> whether to lower or raise it, either globally or at a server-specific
> level?

The correct way to proceed is the following :

1. determine your servers' maximum concurrent connections in the worst case (eg: all on PHP because some requests are slowing down). I mean the *real* limit, not an imaginary one. For instance, if you have 12 GB of RAM but you have an Apache MaxClients set to 256, no matter what you do, you'll never get above 256.
2. assing a slightly lower maxconn value to each server in the haproxy configuration. By "slightly lower" I mean that you want to save some resources on the server (eg: memory during backups or log rotation), and you also want to be able to directly connect to the servers from haproxy (health checks) or from your own browser to check that everything works.
3. count the sum of all the servers' maxconn in your backends, and you'll have the lower frontend's maxconn limit. Now you want haproxy to be able to manage queues and to support load variations, so you'll probably double or triple that value (or even more). Take into account the amount of available RAM on your haproxy machine. Count about 32kB per connection for haproxy plus about as much for the system when it's finely tuned. Check that haproxy plus the system don't eat more than 2/3 of the RAM. Note that on 32-bit systems, one haproxy process will never be able to allocate more than 2-3 GB depending on how the system was compiled. Then, assign the resulting value to the frontend's maxconn setting.
4. assign a value to the global maxconn setting that must be slightly above the frontend's maxconn. This will help with stats socket and will also allow you to support multiple concurrent frontends if required.

To give you an idea, I once got a report from a site running at 150000 concurrent connections in 4.5 GB of RAM for haproxy. They had done a very nice tuning, and this is the highest value I have ever observed in prod.

> Also, if there are any other config options I should consider
> I'd appreciate any advice as this is a new setup for us and we're
> learning as we go.

You can lower the buffer size in order to reduce memory usage. But in your case that does not seem really needed in my opinion. Other config tunables greatly depend on the workload. Long-lived connections are not tuned the same way as short-lived, and you don't tune equally for large and small objects.

However, what is certain is that if you blindly tune for too large a lot, you can get trouble before you reach your maximal load, and at this point you'll have to make tradeoffs between settings that have always been there without knowing if they're really required, and the ones you'd like to add to support a load increase. So you should really tune for something close to your current usage. If you peak at 500 concurrent connections, tune for 2000 and observe.

Regards,
Willy Received on 2010/07/27 21:27