strange behavior under benchmarking

From: Michael Fortson <mfortson#gmail.com>
Date: Thu, 15 Jan 2009 22:40:31 -0800


Our server (a recent 8-core xeon server) is able to handle a few rounds of benchmarks, but then connections just seem to start hanging somewhere -- they don't show up in haproxy_stats. At the same time, every server listed on the stats page starts going red (even though they are clearly fine -- you can hit them directly without problem). It does not seem to be memory-related (no bulging swap file at any rate).

My config file is as follows:



global
        log 127.0.0.1 local0 notice
        #log 127.0.0.1 local1 alert

        daemon
        nbproc 2
        maxconn 32000
        ulimit-n 65536

# and uid, gid, daemon, maxconn...
defaults
# setup logging
log global option tcplog option httplog option dontlognull
# setup statistics pages
stats enable stats uri /haproxy_stats stats refresh 10 stats auth xxxxxxxxxx:xxxxxxxxxxx mode http retries 3 option abortonclose option redispatch timeout connect 80s # time in queue timeout client 30s # data from client (browser) timeout server 80s # data from server (mongrel) frontend rails *:7000
# set connection limits
maxconn 700 default_backend mongrels backend mongrels fullconn 80 option httpchk option abortonclose contimeout 4000 balance roundrobin server webapp01-101 10.2.0.3:8101 minconn 1 maxconn 5 check inter 2s rise 1 fall 2 server webapp01-102 10.2.0.3:8102 minconn 1 maxconn 5 check inter 2s rise 1 fall 2 <snip> lots more of those over several machines... </snip>

## this is the service I'm testing against frontend webservers *:81

        mode tcp
        default_backend nginx

backend nginx
        mode tcp
        balance leastconn
        contimeout 1000
        server webapp01-http 10.2.0.3:8080 check
        server webapp02-http 10.2.0.4:8080 check
        server webapp04-http 10.2.0.17:8080 check
        server webapp05-http 10.2.0.18:8080 check

the test I'm running is:
ab -n 10000 -c 20 "nowhere.com:81/images/layout/blank.gif"

... from another machine on the LAN. It runs fine (and pretty fast) several times in a row, but then something happens; it'll hang somewhere in the test, and none of the requests will show up on the haproxy stats page. Sometimes Apache (which feeds into haproxy for its balancer for Rails requests) starts showing 503s as well.

I'm wondering if I have some bad combination of haproxy configuration, or if I should look elsewhere for the cause... the system is running FC7.

cheers Received on 2009/01/16 07:40

This archive was generated by hypermail 2.2.0 : 2009/01/16 07:45 CET