Re: Problems with load balancing on cloud servers

From: Jerry Champlin <jgc#absolute-performance.com>
Date: Mon, 12 Sep 2011 22:20:09 -0600


Liong:

Have you looked at average query response time statistics? Perhaps this could shed some light on your issue. A graph of response time vs. current requests for a given resource might help you narrow down the issue if it's not a load balancer issue or just requests processed / second compared between the two servers would give you more conclusive evidence that the load balancer is doing the wrong thing.

-Jerry

Jerry Champlin
Absolute Performance Inc.

--
Enabling businesses to deliver critical applications at lower cost and
higher value to their customers.


On Mon, Sep 12, 2011 at 9:38 PM, Liong Kok Foo <kokfoo.liong#innity.com>wrote:


> Hi Jerry,
>
> Appreciate the quick reply and pointing out this difference.
>
> However, we are aware of some server having swap and some not. We have
> explored turning swap on or off but it doesn't solve the issue.
>
> Thanks.
>
> Liong Kok Foo
>
>
> On 9/13/2011 11:15 AM, Jerry Champlin wrote:
>
> Liong:
>
> The only significant difference in the stats you posted is that the server
> without a load problem has no swap configured and the other one does. I
> would not expect that to cause a problem but that's the only difference that
> jumps off the page.
>
> -Jerry
>
> Jerry Champlin
> Absolute Performance Inc.
> --
> Enabling businesses to deliver critical applications at lower cost and
> higher value to their customers.
>
>
> On Mon, Sep 12, 2011 at 9:02 PM, Liong Kok Foo <kokfoo.liong#innity.com>wrote:
>
>> Hi,
>>
>> We have been using haproxy for many years now. Implemented it in few of
>> our systems. However, we have been facing some odd problem which we are not
>> sure if it is related to haproxy. The odd problem is that one of the server
>> is having higher load than the others. That is the last server in the LB but
>> we tried switching it to second last and still see only this server giving
>> high load. We also tried cloning the server from one of the existing server
>> that doesn't have this problem but it still is giving this problem.
>>
>> We have checked with the cloud provider to see if the odd server is hosted
>> in a different cloud segment in their datacenter. Doesn't seems so from
>> their reply.
>>
>> We have setup 5 instance of servers in the cloud computing hosted by
>> voxel. One is being used as haproxy server and the other 4 is running apache
>> serving website.
>>
>> Specs of the cloud servers:
>> 1 core CPU
>> 2GB RAM
>> Ubuntu 10.04 LTS
>> Haproxy 1.3.25
>>
>> Check the screen shot for the haproxy stats. We allocated weight of
>> 1:1:1:1. We just doesn't understand why the extra load on this server.
>>
>> Top for server 69 (no problem):
>> top - 10:51:08 up 145 days, 16:34, 1 user, load average: 1.06, 1.22,
>> 1.27
>> Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 28.8%us, 7.6%sy, 0.0%ni, 61.3%id, 0.0%wa, 0.0%hi, 2.1%si,
>> 0.2%st
>> Mem: 2050000k total, 1587608k used, 462392k free, 760100k buffers
>> Swap: 0k total, 0k used, 0k free, 432848k cached
>>
>> Top for server 70 (load problem)
>> top - 10:51:23 up 32 days, 22:21, 1 user, load average: 3.09, 2.99, 2.50
>> Tasks: 115 total, 3 running, 112 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 38.5%us, 11.0%sy, 0.0%ni, 48.2%id, 0.0%wa, 0.0%hi, 2.3%si,
>> 0.0%st
>> Mem: 2050000k total, 1049708k used, 1000292k free, 264264k buffers
>> Swap: 1052248k total, 876k used, 1051372k free, 418272k cached
>>
>> Sometimes server B's load will shoot up to 20 or more while server A (and
>> the rest remain at around 5).
>>
>> Would really appreciate any input on this matter.
>>
>> Thanks.
>>
>> --
>> Liong Kok Foo
>>
>>
>
Received on 2011/09/13 06:20

This archive was generated by hypermail 2.2.0 : 2011/09/13 06:30 CEST