Re: tracking 502s

From: Willy Tarreau <w#1wt.eu>
Date: Wed, 25 Feb 2009 06:54:58 +0100


Hi,

On Sun, Feb 22, 2009 at 01:44:01PM -0800, Michael Fortson wrote:
> Given a log line like this:www haproxy[15344]: xx.x.xx.xx:33521
> [22/Feb/2009:20:27:13.580] webservers mongrels/webapp02-133 0/0/0/-1/5301
> 502 204 - - SH-- 250/246/9/0/0 0/0 "GET /info/heartbeat?1235334327821716000
> HTTP/1.1"
>
> Where I don't see the matching string in the mongrel logs, how might I go
> about finding the cause?

From what I see above, haproxy could connect to the remote host, it sent the request but observed an immediate close in response. This can happen when the system is accepting the connection, and the application (mongrel here) refuses it for whatever reason (basically does "fd=accept(); close(fd);"). It can also happen if the application crashes, for instance, while processing a previous request.

In such a case, I really suggest that you install tcpdump on one of the mongrels, wait for such an error to happen and check the trace. In parallel, it would be nice to also run "strace -tt" on the monitored process in order to find what it's doing when the problem happens. You should easily find an abusive close() or a process which dies.

It's the only valid solution.

Regards,
Willy Received on 2009/02/25 06:54

This archive was generated by hypermail 2.2.0 : 2009/02/25 08:00 CET