Re: Too large of timeouts

Hi John,

On Wed, Jan 28, 2009 at 10:57:40AM -0500, John Lauro wrote:
> Hello,
>
>
>
> This is a relatively new setup (under a week), but had problems yesterday as
> load increased.
>
>
>
> This problem was reproducible in both the latest 1.2 (tried 1.2 after
> problems) and 1.3.15.7. With large values for timeout for clitimeout (also
> srvrtimeout, but to a lesser extent), I ran into some problems.
>
>
>
> The server would close the connection (and end up in CLOSE_WAIT on the
> haproxy machine I think), and still be counted as a connection for a long
> time. Would be fully closed on the actual servers.

In fact, there's no such thing has a "close" in TCP. Each side informs the other ones that it has nothing left to send (FIN). When both sides have said that, and acknowledged the other side's information, then the connection is considered closed.

When you see a CLOSE_WAIT on haproxy, that means that the other end of the connection has sent a FIN and the system is waiting for haproxy to shutdown the connection in turn. On a proxy, this is very frequent due to the 4-way close :

1. server says FIN to proxy => server-proxy connection goes CLOSE_WAIT
2. proxy says FIN to client => client-proxy connection goes FIN_WAIT1
3. client says FIN to proxy => client-proxy connection is closed
4. proxy says FIN to server => proxy-server connection is closed

If the client does not respond to 2, we have to wait for "clitimeout" before automatically going to state 4. Since you're speaking about extremely large timeouts (30 minutes), it's completely expected that some connection lie there doing nothing if the client suddenly got kicked off of the net without closing, thus keeping one connection alive to the server.

However, there's a workaround for this. You can tell haproxy that you want the connection to the server to be closed early, once the request has been sent. This is achieved by "option forceclose".

The following then happens just after the request is sent :

1. proxy says FIN to server => proxy-server connection goes FIN_WAIT1
2. server ACKs the FIN => proxy-server connection goes FIN_WAIT2
3. the server finally responds and closes => proxy-server connection is closed
4. proxy says FIN to client => client-proxy connection goes FIN_WAIT1
5. client says FIN to proxy => client-proxy connection is closed

Even in the case the client is dead, the server connection is closed before we wait for the client, so the remaining dead conns lie on the client side and not on the server side.

One thing that could be improved would be to support two timeouts per side : normal and fast. We would always use the normal timeout, unless we have sent a shutdown notification, in which case we would wait for the fast timeout. This can often be useful because it would speed up the wiping up of dead connections, but it's dangerous for a lot of protocols, including HTTP sometimes if servers take time to respond.

> That caused the number of tracked connections to run high, but that's not
> really the problem. the main problem is. after awhile under heave load,
> under errors, conn for the backend line it would increase but none of the
> servers would show any increase in errors or warnings, and none of the
> sessions for servers or backend were at the max. Connection failure rate
> would start slow and quickly speed up. Also noticed higher latency prior to
> failure, and the CPU seems to go to 100% at the same time instead of the
> normally only a few %.

If your CPU goes high, I suspect you're on a system which does not support a fast poller or that you have not enabled a scalable polling mechanism at haproxy build time. Could you please run "haproxy -vv" so that we try to find what is missing here ?

> It's as if I was reaching some limit, but from what I could tell no limit
> was being reached.

(...)

> I would expect if the server side closes, haproxy would close it's client
> side, and so wouldn't hurt to have an extra long timeout. Obviously that's
> not the case.

yes that's the case once you understand the explanation above ;-)

> I am working ok for now, but am a little concerned about the
> backend errors where it didn't try a server and didn't log anywhere as to
> why and AFAIK didn't reach any limits.

If the problem happens a lot, it is possible that you have some firewalls between the client and haproxy which expire the session before it expires on haproxy (this too is a common problem in multi-level architectures). If you can't increase your firewall timeouts to cover the client's or haproxy's, then you can enable "option tcpka" on haproxy so that it sends keepalives. But warning, it is the system which decides the keepalive interval. On linux for instance, it's two hours by default, so your firewall must not expire a live session before that delay.

Regards,
Willy Received on 2009/01/28 23:22