Re: Odd problems with haproxy and sudden timeouts

Hi Gabriel,

On Fri, May 23, 2008 at 11:17:26PM +0300, Gabriel wrote:
> > s : the server-side time-out expired first.
> > C : see above
> > N : the client provided NO cookie. This is usually the case on new
> > connections.
> > N : see above
>
> Yep I looked these up but they did not make my life easier as they do
> not point at what could be wrong. They appear suddenly and in a great
> number so I guess there is something that goes wrong somewhere.

When you get server connection errors, it can mean three things :

• highly overloaded server which cannot even accept TCP connections (not likely the case here with 14 connections)
• there is a network problem between haproxy and the server. This is usually detected because you see very repetitive values in the connect times, generally multiples of 3 seconds.
• there is a poorly configured firewall between haproxy and the server which regularly goes out of connections. This sometimes happens on moderate traffic sites with firewalls left with default settings (unable to cope with more than 1000 hits/s or so), as well as server-based firewalls which generally are even worse and with very long session expiration timers. This means that after a few thousand connections have passed through them, the session table is full of terminated sessions waiting to expire and the firewall cannot accept any more sessions. This is quite common with Linux distros shipping with netfilter enabled by default.

I would bet you're in the third case. You can find if it is the proxy or a server which is causing the problem by checking if the proxy still accepts connections or not (telnet to the port while the problem appears). You can also regularly fire a "dmesg|tail" to check if you notice messages such as "conntrack state table full" or something like this.

> [WARNING] 143/230538 (3309) : SIGHUP received, dumping servers states.
> [...]
> [WARNING] 143/230538 (3309) : SIGHUP: Server sitex/siteyx1701 is UP.
> Conn: 13 act, 0 pend, 12651 tot.
> [WARNING] 143/230538 (3309) : SIGHUP: Server sitex/siteyx1701 is UP.
> Conn: 2 act, 0 pend, 1131 tot.
> [WARNING] 143/230538 (3309) : SIGHUP: Proxy sitex has 1 active servers
> and 1 backup servers available. Conn: act(FE+BE): 0+15, 0 pend (0
> unass), tot(FE+BE): 0+13782.
> [...]
> [WARNING] 143/230538 (3309) : SIGHUP: Proxy http-in has no servers.
> Conn: act(FE+BE): 22+0, 0 pend (0 unass), tot(FE+BE): 20063+0.

So http-in is down here.

> Another thing that I came to think of what that even though the
> statistics showed that the server have had retries (Retr) and downs I
> found nothing to indicate this in the logs. Is it not logged when a
> server goes down?

When a server goes down, it is logged with a different severity. It may be possible that your syslog sends it to another file (messages?). Idem when a server goes back up.

Regards,
Willy Received on 2008/05/23 22:31