Re: Odd problems with haproxy and sudden timeouts

From: Gabriel <tehviu#gmail.com>
Date: Mon, 2 Jun 2008 15:30:38 +0300

Hello again,

It's been a while since I last wrote because we haven't experienced the problem again running on another "identical" server. The next action we are going to do wipe clean the first and install it again assuming it was some hiccup somwhere that caused our problems. But I still have a few questions about the logging.

On Fri, May 23, 2008 at 11:31 PM, Willy Tarreau <w#1wt.eu> wrote:
> Hi Gabriel,
>
> When you get server connection errors, it can mean three things :
> - highly overloaded server which cannot even accept TCP connections
> (not likely the case here with 14 connections)
>
> - there is a network problem between haproxy and the server. This is
> usually detected because you see very repetitive values in the
> connect times, generally multiples of 3 seconds.
>
> - there is a poorly configured firewall between haproxy and the server
> which regularly goes out of connections. This sometimes happens on
> moderate traffic sites with firewalls left with default settings
> (unable to cope with more than 1000 hits/s or so), as well as
> server-based firewalls which generally are even worse and with very
> long session expiration timers. This means that after a few thousand
> connections have passed through them, the session table is full of
> terminated sessions waiting to expire and the firewall cannot
> accept any more sessions. This is quite common with Linux distros
> shipping with netfilter enabled by default.
>
> I would bet you're in the third case. You can find if it is the proxy
> or a server which is causing the problem by checking if the proxy still
> accepts connections or not (telnet to the port while the problem appears).
> You can also regularly fire a "dmesg|tail" to check if you notice messages
> such as "conntrack state table full" or something like this.
>

There is not firewall in between, just a switch.

>> [WARNING] 143/230538 (3309) : SIGHUP received, dumping servers states.
>> [...]
>> [WARNING] 143/230538 (3309) : SIGHUP: Server sitex/siteyx1701 is UP.
>> Conn: 13 act, 0 pend, 12651 tot.
>> [WARNING] 143/230538 (3309) : SIGHUP: Server sitex/siteyx1701 is UP.
>> Conn: 2 act, 0 pend, 1131 tot.
>> [WARNING] 143/230538 (3309) : SIGHUP: Proxy sitex has 1 active servers
>> and 1 backup servers available. Conn: act(FE+BE): 0+15, 0 pend (0
>> unass), tot(FE+BE): 0+13782.
>> [...]
>> [WARNING] 143/230538 (3309) : SIGHUP: Proxy http-in has no servers.
>> Conn: act(FE+BE): 22+0, 0 pend (0 unass), tot(FE+BE): 20063+0.
>
> So http-in is down here.

Is that a problem? The proxy still works.

>
>> Another thing that I came to think of what that even though the
>> statistics showed that the server have had retries (Retr) and downs I
>> found nothing to indicate this in the logs. Is it not logged when a
>> server goes down?
>
> When a server goes down, it is logged with a different severity. It
> may be possible that your syslog sends it to another file (messages?).
> Idem when a server goes back up.
>
> Regards,
> Willy

I found nothing in even messages, but might be a mistake by me but If so I cannot find what mistake could have caused it.I have

local4.*                                                /var/log/haproxy.log

in my syslog.conf so I assume it should write everything there? But the only thing I see is
Jun 2 13:50:44 localhost.localdomain haproxy[20745]: Proxy http-in stopped. Jun 2 13:50:44 localhost.localdomain haproxy[20771]: Proxy http-in started. Should it not print more with
log 127.0.0.1 local4 notice
in global ?

Sincerely,
Gabriel N. Received on 2008/06/02 14:30