Re: Running haproxy on the cluster nodes

From: Martin Goldman <martin#mgoldman.com>
Date: Tue, 11 Dec 2007 08:28:49 -0500

Thanks Willy, I better understand what you're saying now. I checked and don't have the ip_conntrack module on my systems. I ran haproxy with -V and re-ran the same test, and the only output I got was what's below. I'm on 1.3.13.1 but can upgrade to the newest version if that would be helpful. Just in case there's something outrageously wrong with my config, I'm pasting that below too -- FYI, I removed a few things (such as logging) and the performance improved somewhat, but is still worse than that of the individual web servers. Please let me know if you have any other ideas. Thanks for all the help!

martin#kramer:~$ sudo /usr/sbin/haproxy -V -f /etc/haproxy.cfg Available polling systems :

       poll : pref=200,  test result OK
     select : pref=150,  test result OK

Total: 2 (2 usable), will use poll.
Using poll() as the polling mechanism.

listen kramer-web-svc kramer:80

        retries 3
        redispatch
        contimeout      4000
        clitimeout      40000
        srvtimeout      40000
       mode http
       option forwardfor
       option httpclose
       option httpchk HEAD /httpcheck.html HTTP/1.0
       option httplog
       option dontlognull
       balance roundrobin
       server jerry-web jerry.cp.priv:80 check inter 3000
       server george-web george.cp.priv:80 check inter 3000

On 12/11/07, Willy Tarreau <w#1wt.eu> wrote:
>
> Hi Martin,
>
> On Mon, Dec 10, 2007 at 11:17:44PM -0500, Martin Goldman wrote:
> > Hi Willy,
> >
> > So, I tried moving the proxy onto a separate server and ran the same
> test,
> > and found that although the CPU load wasn't quite as severe, the
> performance
> > wasn't really any better -- still about 9,000 requests/sec from the
> cluster,
> > compared to 15,000 from each of the individual web servers.
>
> Then there's a real problem. 9000 requests/s is approximately what I get
> on
> my notebook when I slow it down to 800 MHz in order to facilitate
> benchmarks.
>
> > I re-read your message, and I must admit I'm having a bit of trouble
> > following your math, but it sounds like you weren't particularly
> surprised
> > by my results.
>
> I was not surprized because it was an estimation of the performance you
> would
> get from associating two components on a single machine when you know the
> performance you achieve with only one component. Now, if haproxy alone is
> limited to 9000 hits/s, then there's a problem.
>
> > I'm having a bit of trouble wrapping my head around the idea
> > that haproxy should have trouble keeping up with the performance of
> apache
> > -- is that what you're saying you'd expect?
>
> No, quite the opposite. Till there, I've always observed haproxy being
> about
> 5 times faster than apache on the same machine. That's why I'm amazed.
>
> > With regards to ip_conntrack, I've done some googling around, but am
> having
> > trouble understanding figuring out just what it is. In any case, sysctl
> -a |
> > grep ip_conntrack doesn't get me anything.
>
> It is a linux kernel module, it assures the connection tracking for the
> netfilter firewall. You could see it by doing "lsmod". It has to
> internally
> create connections entries for each connection that enters the proxy and
> each
> connection that gets out of it. And since by defaults the hashes are very
> poor,
> it can take an awful lot of time for each connection.
>
> Also, could you send the output of "haproxy -V" (and haproxy -vv if it's
> as recent as 1.3.14) ?
>
> Please keep me updated, this is a problem which must be solved.
>
> Cheers,
> Willy
>
>
Received on 2007/12/11 14:28