Re: high cpu utilization

On Sun, Feb 17, 2008 at 12:04:03AM -0500, Marc Breslow wrote:
> I ran the commands you said and learned that I'm not using epoll :( The
> make file says it's automatic on Linux 2.6 and my kernel is 2.6.9-55. I
> just rebuilt actually specifying target=linux26 and now epoll is enabled. I
> think I just misunderstood what "automatic" meant. The Makefile clearly
> says "you should build with TARGET=os" but after reading further and seeing
> "enable epoll on Linux 2.6. Automatic" I didn't think I needed to. My bad.

Yes in fact it can be misinterpreted. "automatic" just means that the option is automatically selected when you you select the target OS. This is because you can also force the option on or off.

> I ran the -vv switch on the binary you have on the haproxy site but it looks
> like you compiled with some options that require kernel patches that I
> probably don't have (USE_TCPSPLICE and USE_TCPSPLICE) - are those key? What
> would that help with?

tcpsplice makes use of a kernel extension that Alexandre Cassen developped (see www.linux-l7sw.org). Its purpose is to forward the payload directly from the kernel without copying it to userland. While it mostly works, it is still very experimental and some strange corner cases happen. I enable it in my builds so that both Alex and I have something we can experiment with.

It will not change anything for you because you have to enable it using "option tcpsplice" in your configuration. And if you use it, haproxy will indeed tell you that your kernel is not patched and will refuse to start.

> So now I've rebuilt haproxy and ran your test config and see the same output
> that you saw where as with the build I had been running I saw:
> Available polling systems :
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 2 (2 usable), will use poll.
> Using poll() as the polling mechanism.

hehe indeed !

> Knowing how big an improvement epoll has over poll, I'm sure this will work
> wonders.

In fact, epoll itself is still costly because the epoll_ctl() syscall needs to be called very often (once per read and per write for flow control). That's why I developped "sepoll" which is "speculative epoll". With it, haproxy speculatively tries I/O (read, write, connect, ...) and only does epoll_ctl() when it has no choice. This has resulted in a noticeable speedup due to an important reduction of the use of this syscall (about -85%). That's why it is selected by default. Early versions were buggy, but having it enabled by default made all bugs quickly surface and since 1.3.14.2, it's rock solid.

> So I recompiled and did a soft-restart of haproxy using the new binary.
> Looks like the problem is solved! Thank you so much.
> ---Marc
>
> top - 00:01:07 up 12:34, 1 user, load average: 0.11, 0.64, 0.90
> Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.7% us, 2.4% sy, 0.0% ni, 96.2% id, 0.0% wa, 0.7% hi, 0.0% si
> Cpu1 : 0.3% us, 1.7% sy, 0.0% ni, 97.3% id, 0.0% wa, 0.7% hi, 0.0% si

Wow! it's impressive to see the load divided by 30!

I think I should write a troubleshooting guide for such situations, and reuse some of the tests you had to run (including strace -c).

Regards,
Willy Received on 2008/02/17 07:33