Re: high cpu utilization

From: Willy Tarreau <w#1wt.eu>
Date: Tue, 12 Feb 2008 21:51:08 +0100


On Tue, Feb 12, 2008 at 11:56:55AM -0700, Dan Zubey wrote:
> For logging to work, you need to enable syslog with the -r switch, and
> then restart syslog.

exactly :-)

> Check out debug mode, but Willy will have additional ideas in a bit here.

The early 2.6 kernels had *horrible* CPU usage patterns. You're lucky to just see 100% CPU, I've seen machines pausing several seconds, with vmstat freezing. The biggest of these troubles started to get fixed around 2.6.16, then slightly better with 2.6.18, and finally disappeared with the new CFS scheduler in 2.6.23.

What I've also observed was a perfectly regular process showing sine waves in CPU usage, from long periods at 0% to long periods at 100%, back and forth. I know the load was perfectly stable because it was regulated by an external injection system which itself was monitored, and all indicators on the network reported a 3-digit stability, while the 2.6 machine in the middle was oscillating between 0 and 100%. Obviously, even if 100% had been possible, 0 was not possible because the work it had to do made it run at about 25%, which must be noticeable.

Now, don't get me wrong. I'm not saying that there does not exist a problem with 1.2.17 which occasionally brings the CPU to 100%, what I'm saying is that *measuring* CPU on such an awfully crappy 2.6.9 means absolutely nothing.

I would propose something :

If the problem definitely disappears with either change, it means that the old version is the culprit. If the problem remains with 2.6.23.16, it means either that the load is really huge (gigabit, regex?) or that it's a yet undiscovered bug in haproxy. But judging by many people's usage, even high loads should not push your CPU to 100%.

Regards,
Willy Received on 2008/02/12 21:51

This archive was generated by hypermail 2.2.0 : 2008/02/12 22:00 CET