Re: Haproxy 502 errors, all the time on specific sites or backend

From: Cyril Bonté <cyril.bonte#free.fr>
Date: Thu, 03 Nov 2011 15:32:06 +0100


Hi Benoit,

Le Jeudi 3 Novembre 2011 14:46:10 Benoit GEORGELIN a écrit :
> Hi !
>
> My name is Benoît and i'm in a associative project who provide web hosting.
> We are using Haproxy and we have a lot of problems with 502 errors :(
>
>
> So, i would like to know how to really debug this and find solutions :)
> There is some cases on mailling list archives but i will appreciate if
> someone can drive me with a real case on our infrastructure.

My first observations, it it can help someone to target the issue : In your servers responses, there is no Content-Length header, this can make some troubles.

502 errors occurs when asking for compressed data : - curl -si -H "Accept-Encoding: gzip,deflate" http://sandka.org/portfolio/ HTTP/1.0 502 Bad Gateway
- curl -si http://sandka.org/portfolio/
=> results in a truncated page without Content-Length Header

We'll have to find why your backends doesn't provide a Content-Length header (and what happens with compression, which should be sent in chunks).

> Details:
>
>
> Haproxy Stable 1.4.18
> OS: Debian Lenny
>
> Configuration File:
>
>
> ######################################################################
>
> global
>
>
> log 127.0.0.1 local0 notice #debug
> maxconn 20000 # count about 1 GB per 20000 connections
> ulimit-n 40046
>
>
> tune.bufsize 65536 # Necessary for lot of CMS page like Prestashop :(
> tune.maxrewrite 1024
>
>
> #chroot /usr/share/haproxy
> user haproxy
> group haproxy
> daemon
> #nbproc 4
> #debug
> #quiet
>
>
> defaults
> log global
> mode http
> retries 3 ##### 2 -> 3 le 06102011 #####
> maxconn 19500 # Should be slightly smaller than global.maxconn.
>
>
> ######## OPTIONS ##########
> option dontlognull
> option abortonclose
> #option redispatch ##### Désactive le 06102011 car balance en mode source et
> non RR ##### option tcpka
> #option log-separate-errors
> #option logasap
>
>
> ######## TIMeOUT ##########
> timeout client 30s #1m 40s Client and server timeout must match the longest
> timeout server 30s #1m 40s time we may wait for a response from the server.
> timeout queue 30s #1m 40s Don't queue requests too long if saturated.
> timeout connect 5s #10s 5s There's no reason to change this one.
> timeout http-request 5s #10s 5s A complete request may never take that long
> timeout http-keep-alive 10s
> timeout check 10s #10s
>
> #######################################################
> # F R O N T E N D P U B L I C B E G I N
> #
> frontend public
> bind 123.456.789.123:80
> default_backend webserver
>
>
> ######## OPTIONS ##########
> option dontlognull
> #option httpclose
> option httplog
> option http-server-close
> # option dontlog-normal
>
>
> ##### Gestion sur URL # Tout commenter le 21/10/2011
> # log the name of the virtual server
> capture request header Host len 60
>
>
>
>
> #
> # F R O N T E N D P U B L I C E N D
> #######################################################
>
> #######################################################
> # B A C K E N D W E B S E R V E R B E G I N
> #
> backend webserver
> balance source ##### Reactive le 06102011 #####
> #balance roundrobin ##### Désactive le 06102011 #####
>
>
> ######## OPTIONS ##########
> option httpchk
> option httplog
> option forwardfor
> #option httpclose ##### Désactive le 06102011 #####
> option http-server-close
> option http-pretend-keepalive
>
>
> retries 5
> cookie SERVERID insert indirect
>
>
> # Detect an ApacheKiller-like Attack
> acl killerapache hdr_cnt(Range) gt 10
> # Clean up the request
> reqidel ^Range if killerapache
>
>
>
> server http-A 192.168.0.1:80 cookie http-A check inter 5000
> server http-B 192.168.1.1:80 cookie http-B check inter 5000
> server http-C 192.168.2.1:80 cookie http-C check inter 5000
> server http-D 192.168.3.1:80 cookie http-D check inter 5000
> server http-E 192.168.4.1:80 cookie http-E check inter 5000
>
>
> # Every header should end with a colon followed by one space.
> reqideny ^[^:\ ]*[\ ]*$
>
>
> # block Apache chunk exploit
> reqideny ^Transfer-Encoding:[\ ]*chunked
> reqideny ^Host:\ apache-
>
>
> # block annoying worms that fill the logs...
> reqideny ^[^:\ ]*\ .*(\.|%2e)(\.|%2e)(%2f|%5c|/| \\\\ )
> reqideny ^[^:\ ]*\ ([^\ ]*\ [^\ ]*\ |.*%00)
> reqideny ^[^:\ ]*\ .*<script
> reqideny ^[^:\ ]*\ .*/(root\.exe\?|cmd\.exe\?|default\.ida\?)
>
>
> # allow other syntactically valid requests, and block any other method
> reqipass ^(GET|POST|HEAD|OPTIONS)\ /.*\ HTTP/1\.[01]$
> reqipass ^OPTIONS\ \\*\ HTTP/1\.[01]$
>
>
> errorfile 400 /etc/haproxy/errors/400.http
> errorfile 403 /etc/haproxy/errors/403.http
> errorfile 408 /etc/haproxy/errors/408.http
> errorfile 500 /etc/haproxy/errors/500.http
> errorfile 502 /etc/haproxy/errors/502.http
> errorfile 503 /etc/haproxy/errors/503.http
> errorfile 504 /etc/haproxy/errors/504.http
>
>
>
>
> Error 502 example (all the time)
>
>
>
> Log:
>
>
> Nov 3 13:52:15 127.0.0.1 haproxy[27813]: 216.46.4.5:49451
> [03/Nov/2011:13:52:14.584] public webserver/http-B 11/0/0/-1/985 502 280 -
> - SHVN 121/121/19/3/0 0/0 {website.com} "GET /portfolio/ HTTP/1.1" Nov 3
> 13:52:15 127.0.0.1 haproxy[27813]: 216.46.4.5:49452
> [03/Nov/2011:13:52:14.583] public webserver/http-B 1153/0/0/10/1163 404 849
> - - --VN 123/123/16/4/0 0/0 {website.com} "GET /favicon.ico HTTP/1.1"
>
>
> This is a ZenPhotos CMS ( http://sandka.org/portfolio/ )
> With more than one picture on it, 502 errors.
>
>
> I try to comment all reqideny and reqipass, but it still not working
> We use NFS protocol to share content on WebServer Side.Almost use php/mysql
>
>
> I can give you tcpdump but i'm not shure about te good syntaxe who can help
> in this case.
>
>
>
>
> Many thanks for your help and your time
>
>
> Best Regards
>
>
> Cordialement,
>
> Benoît Georgelin
> Web 4 all Hébergeur associatif
> Afin de contribuer au respect de l'environnement, merci de n'imprimer ce
> mail qu'en cas de nécessité

-- 
Cyril Bonté
Received on 2011/11/03 15:32

This archive was generated by hypermail 2.2.0 : 2011/11/03 15:45 CET