Re: A (Hopefully Not too Generic) Question About HAProxy

Dnia 2010-05-17, pon o godzinie 14:45 -0700, Chih Yin pisze:
> Hi,
>
>
> Please excuse me if the information contained in this email is a bit
> generic. I'm not the regular administrator, but I've been given the
> task of troubleshooting some issues with my website. If more details
> are needed, I'll gladly go look for additional information.
>
>
> Currently, my website is experiencing a lot of errors and slowness.
> The server errors I see in the HAProxy log file are mainly 503
> errors. The slowness of page loads for my website can be as long as
> minutes. I'm trying to determine if anyone have had any similar
> issues with using HAProxy as a high availability load balancer.
>
>
> HAProxy 1.3.21
> CentOS running on Citrix XenServer
> HP blades
>
>
> There are actually almost 100 virtual servers running on the
> blades. A good many of the virtual servers are application servers
> running Glassfish. There are a few servers dedicated to CAS for
> authentication and access. I have three servers running Rsyslogd for
> writing HAProxy log data to file. A NetApp filer is used for storage.
>
>
> Currently, the website gets about:
>
>
> 73,000 pageviews a day
> 32,000 unique visitors a day
> 46,000 visit a day
>
>
> 3,000 pageviews a hr
> 1,300 unique visitors a hr
> 1,000 visit a hr
>
>
> I am using Akamai to help manage content delivery.
>
>
> One of the things Akamai is reporting to me is that they are having
> difficulty requesting content that needs to be refreshed. Akamai
> tries up to 4 times to get the content with a 2 second timeout to
> update content whose TTL has expired. After the 4th time, Akamai
> looks to their own cache before returning a 503 error to the user if
> the content is not available in the cache.
>
>
> Recently, I've noticed that Akamai is encountering an increasingly
> large number of 503 and 404 errors from my website. I've traced the
> 404 errors to missing images, but I'm not sure what the cause of the
> 503 errors could be. I had some external resources help me verify
> that they are able to retrieve the content from the Glassfish
> application servers even when HAProxy is reporting the 503 errors.
>
>
> One thing I did notice about the HAProxy configuration is that there
> are actually three servers running HAProxy with identical
> configurations. One serves as the primary high availability load
> balancer while the other two act as failovers. The keep-alive daemons
> are configured to accomodate that setup.
>
>
> From this generic description, is there something in the way this
> architecture is set up or in the configuration of HAProxy that may be
> causing the 503 errors to be reported to Akamai? As I mentioned, when
> an external resource makes a request for the same content directly
> from the application server, the same errors do not appear to occur.
>

503 would (usually) mean haproxy sees backends as DOWN, look for msg about server goin up/down in haproxy logs. Or, if thats not a case, grep haproxy logs for those 503 errors (make sure ure using http log mode), then go to section 8 in http://haproxy.1wt.eu/download/1.3/doc/configuration.txt and try to determine what was exact reason for error and/or post few examples here, your haproxy config (with "sensitive" information removed ofc ;) )

Having some kind of monitoring, or at least stats page active is also very helpful.

-- 
Mariusz Gronczewski (XANi) <xani666#gmail.com>
GnuPG: 0xEA8ACE64
http://devrandom.pl






application/pgp-signature attachment: To jest część wiadomości podpisana cyfrowo