Re: HAPROXY in zLinux is presenting Segmentation fault

From: Willy Tarreau <w#1wt.eu>
Date: Tue, 27 Oct 2009 22:10:17 +0100

Hi Alexandre,

On Tue, Oct 27, 2009 at 03:41:55PM +0000, alexandre oliveira wrote:
>
> Willy, I have gdb but I dont know how to use it. Could you say me how to invoke haproxy command using it?

yes, basically you place yourself in the place where you have all your sources (and the binary) and use gdb as a wrapper to load the binary, then use "run" to start it with arguments. Example for the line below :

> holb001:~/haproxy-1.3.22 # haproxy -f /etc/haproxy/haproxy.cfg -db

It becomes :

holb001:~/haproxy-1.3.22 # gdb haproxy
(gdb) run -f /etc/haproxy/haproxy.cfg -db

When it crashes, it indicates where it was when it happened, almost everytime with the exact line number (unless some optimisations are causing issues, in which case you'll want to build with -O or -O0 instead of -O2). For instance (I've added an explicit crash to illustrate) :

Program received signal SIGSEGV, Segmentation fault. http_wait_for_request (s=0x80dda08, req=0x80ddee8, an_bit=2) at src/proto_http.c:1788 1788 *(int *)0=0;
(gdb)

Then you can inspect what other functios were above using "bt" (backtrace) :

(gdb) bt

#0  http_wait_for_request (s=0x80dda08, req=0x80ddee8, an_bit=2) at src/proto_http.c:1788
#1  0x08075688 in process_session (t=0x80dd488) at src/session.c:812
#2  0x0804e201 in process_runnable_tasks (next=0xffb4099c) at src/task.c:234
#3  0x0804ac3f in run_poll_loop () at src/haproxy.c:920
#4  0x0804afb8 in main (argc=4, argv=0xffb40a74) at src/haproxy.c:1195

(gdb)

You can also list the code around the place of the crash :

(gdb) list

1783                    req->analysers);
1784
1785            /* we're speaking HTTP here, so let's speak HTTP to the client */
1786            s->srv_error = http_return_srv_error;
1787
1788            *(int *)0=0;
1789
1790            if (likely(req->lr < req->r))
1791                    http_msg_analyzer(req, msg, &txn->hdr_idx);

1792
<gdb)

You can inspect some variables which were set before the crash :

(gdb) print s
$1 = (struct session *) 0x80dda08
(gdb) print req
$2 = (struct buffer *) 0x80ddee8
(gdb) print *s
$3 = {list = {n = 0x8098e58, p = 0x8098e58}, back_refs = {n = 0x80dda10, p = 0x80dda10}, task = 0x80dd488, listener = 0x80a2d38, fe = 0x809e7f8, be = 0x809e7f8, conn_retries = 0, (...)
(gdb) print *req
$4 = {flags = 9474050, rex = -1747828994, wex = 0, rto = 5000, wto = 0, cto = 0, l = 475, r = 0x80de11f "",   w = 0x80ddf44 "GET /stat HTTP/1.1\r\nHost: 127.0.0.1:8080\r\nUser-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.19) Gecko/20081226 SeaMonkey/1.1.14 Mnenhy/0.7.4.0\r\nAccept: text/xml,application/xml,app"..., (...)

I suspect you'll find a dereferenced null. Something like :

    ptr->member = something;

with ptr = NULL.

> I have used the SIGQUIT signal. The result is as follow:

ah thanks.

> holb001:~/haproxy-1.3.22 # haproxy -f /etc/haproxy/haproxy.cfg -db
> Available polling systems :
> sepoll : pref=400, test result OK
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 4 (4 usable), will use sepoll.
> Using sepoll() as the polling mechanism.
> Dumping pools usage.
> - Pool pipe (32 bytes) : 0 allocated (0 bytes), 0 used, 2 users [SHARED]
> - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool task (144 bytes) : 2 allocated (288 bytes), 2 used, 1 users [SHARED]
> - Pool hdr_idx (832 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool session (1040 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool buffer (16512 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> Total: 7 pools, 288 bytes allocated, 288 used.

At first glance, it seems pretty much common.

Thanks for your efforts in helping us fix this issue. Willy Received on 2009/10/27 22:10