Re: [haproxy] Help tracking down a possible HAProxy outage...

From: Willy Tarreau <w#1wt.eu>
Date: Tue, 2 Dec 2008 06:54:59 +0100


On Mon, Dec 01, 2008 at 09:14:56PM -0800, hernan wrote:
> > You likely have a global "maxconn" set to 2000, or a per-frontend maxconn
> > set to 2000.
>
> You're right, there was a maxconn=2000 declared in the defaults.
> Thanks for helping dissect the log calls, I have to assume that we did
> in fact have 2000 connections and we're investigating this. Since we
> started looking, the high water mark has been around 300 connections
> so whatever happened was an anomaly (in our code, in our usage
> patterns, or due to some malicious third party).

OK. Check the logs, you might have got an attack.

> >> HAProxy version info is "HA-Proxy version 1.3.14.6 2008/06/21" running on
> >> Linux (CentOS 5).
> >
> > You should upgrade to 1.3.14.10, as quite a few annoying bugs have been fixed
> > since 1.3.14.6, one of which is related to the server timeout which could be
> > ignored under some circumstances. This might cause dead connections to accumulate
> > in presence of server errors, up to the point maxconn is reached.
>
> The "recommended version" at http://haproxy.1wt.eu/ is 1.3.15.6.
> Shouldn't I just upgrade to that version or is it not considered
> production-ready?

Yes 1.3.15.6 is production-ready and in fact running on several big sites. It's just that I do not like to force people to upgrade, so if you wanted to stay on 1.3.14, you could. But of course I'd prefer it if everyone migrates to 1.3.15.

> >> Let me know if you can suggest additional things to look for in my HAProxy
> >> configuration. I've been looking at the host and I don't see anything that
> >> strikes me as out of the ordinary. I can share my haproxy.cfg if needed.
> >
> > it would obviously help.
> >
> > Please double-check your maxconn settings, and also consider upgrading. The
> > upgrade might fix one part of the problem and mask the config limit I suspect
> > though, so if you do, please keep a copy of your config so that we can study
> > it.
>
> Thanks for the response, Willy. I've been on this list for a while
> now and I'm always impressed at your level of engagement with your
> users.

Not that much in fact, since I'm really short in spare time. But I try to grant a bit of time to users who run into problems because it's also what helps fixing bugs in the product, and what makes users talk about it around them ;-)

Cheers,
Willy Received on 2008/12/02 06:54

This archive was generated by hypermail 2.2.0 : 2008/12/02 07:00 CET