Re: haproxy & websockets

From: Dustin Moskovitz <moskov#asana.com>
Date: Wed, 28 Apr 2010 16:51:41 -0700


Actually, I should have mentioned at the beginning that we are using websockets to communicate with a *stateful* server, so we don't want to close the connection. However, explicitly declaring "mode http" in either the defaults section or the frontend & backend sections has made things happy.

Now, a new problem (please let me know if/when I should start new threads). If I introduce ssl into the mix, I cannot get it to work with http over a websockets connection (I have it working without websockets, so it is otherwise ok). Using the config at the bottom of this email, if I hit port 80 with a request or two and then move to ssl, haproxy will systematically *seg fault*, like so:

root#beta_lb001 ~# /var/asana-config/asana/3rdparty/haproxy/haproxy -f /var/asana-config/asana/admin/production/proxy/haproxy.cfg.beta -p /var/run/haproxy-private.pid -d -sf $(</var/run/haproxy-private.pid) Available polling systems :

       poll : pref=200,  test result OK
     select : pref=150,  test result OK

Total: 2 (2 usable), will use poll.
Using poll() as the polling mechanism.
00000000:ws_in.accept(0003)=0007 from [99.66.213.198:51011]
00000000:ws_in.clireq[0007:ffff]: GET /favicon.ico HTTP/1.1
00000000:ws_in.clihdr[0007:ffff]: Host: newproxy.asana.com
00000000:ws_in.clihdr[0007:ffff]: Connection: keep-alive
00000000:ws_in.clihdr[0007:ffff]: User-Agent: Mozilla/5.0 (Macintosh; U;
Intel Mac OS X 10_6_3; en-US) AppleWebKit/533.2 (KHTML, like Gecko)
Chrome/5.0.342.9 Safari/533.2
00000000:ws_in.clihdr[0007:ffff]: Accept: */*
00000000:ws_in.clihdr[0007:ffff]: Accept-Encoding: gzip,deflate,sdch
00000000:ws_in.clihdr[0007:ffff]: Accept-Language: en-US,en;q=0.8
00000000:ws_in.clihdr[0007:ffff]: Accept-Charset:
ISO-8859-1,utf-8;q=0.7,*;q=0.3
00000000:ws_in.clihdr[0007:ffff]: Cookie: __utmz=128306844.1272148864.12.4.utmcsr=ec2-204-236-249-67.compute-1.amazonaws.com:8080|utmccn=(referral)|utmcmd=referral|utmcct=/app/asana; __utma=128306844.96842382.1270269861.1272148864.1272331848.13; user_id=53; fbsetting_58eca25f34f388b0e5a53261df8b6b62=%7B%22connectState%22%3A1%2C%22oneLineStorySetting%22%3A3%2C%22shortStorySetting%22%3A3%2C%22inFacebook%22%3Afalse%7D; base_domain_58eca25f34f388b0e5a53261df8b6b62=asana.com; 58eca25f34f388b0e5a53261df8b6b62=78006d0ada0fbc51b2b2da7b59145631;
58eca25f34f388b0e5a53261df8b6b62_user=6;
58eca25f34f388b0e5a53261df8b6b62_ss=J1nPiTjQw_3fB2kfMf66EA__;
58eca25f34f388b0e5a53261df8b6b62_session_key=2.J1nPiTjQw_3fB2kfMf66EA__.3600.1272502800-6;
58eca25f34f388b0e5a53261df8b6b62_expires=1272502800
00000000:ws.srvrep[0007:0008]: HTTP/1.1 200 OK
00000000:ws.srvhdr[0007:0008]: Date: Wed, 28 Apr 2010 23:47:41 GMT
00000000:ws.srvhdr[0007:0008]: Content-Type: text/html
00000000:ws.srvhdr[0007:0008]: Content-Encoding: gzip
00000000:ws.srvhdr[0007:0008]: Transfer-Encoding: chunked
00000000:ws.srvhdr[0007:0008]: Server: Jetty(7.1.0-SNAPSHOT)
00000000:ws.srvcls[0007:0008]
00000000:ws.clicls[0007:0008]

00000000:ws.closed[0007:0008]
00000001:ws_ssl_in.accept(0004)=0007 from [99.66.213.198:51012] Segmentation fault

If I don't make any requests before hitting up ssl, it opens and closes without any information and the client sees "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): Unknown error." root#beta_lb001 ~# /var/asana-config/asana/3rdparty/haproxy/haproxy -f /var/asana-config/asana/admin/production/proxy/haproxy.cfg.beta -p /var/run/haproxy-private.pid -d -sf $(</var/run/haproxy-private.pid) Available polling systems :

       poll : pref=200,  test result OK
     select : pref=150,  test result OK

Total: 2 (2 usable), will use poll.
Using poll() as the polling mechanism.
00000000:ws_ssl_in.accept(0004)=0007 from [99.66.213.198:51041]
00000000:ws_ssl_in.clireq[0007:ffff]:
00000000:ws_ssl_in.clicls[0007:ffff]
00000000:ws_ssl_in.closed[0007:ffff]
00000001:ws_ssl_in.accept(0004)=0007 from [99.66.213.198:51042]
00000001:ws_ssl_in.clireq[0007:ffff]:
00000001:ws_ssl_in.clicls[0007:ffff]
00000001:ws_ssl_in.closed[0007:ffff]
00000002:ws_ssl_in.accept(0004)=0007 from [99.66.213.198:51093]
00000002:ws_ssl_in.clireq[0007:ffff]:
00000002:ws_ssl_in.clicls[0007:ffff]
00000002:ws_ssl_in.closed[0007:ffff]
00000003:ws_ssl_in.accept(0004)=0007 from [99.66.213.198:51095]
00000003:ws_ssl_in.clireq[0007:ffff]:
00000003:ws_ssl_in.clicls[0007:ffff]
00000003:ws_ssl_in.closed[0007:ffff]

Thoughts?

Current config:

global
  maxconn 4096
  user nobody
  group nobody

defaults
  retries 3
  maxconn 2000
  contimeout 5000
  clitimeout 50000
  srvtimeout 50000

frontend ws_in
  mode http
  bind *:80
  acl is_lists hdr_end(host) -i lists.asana.com   acl is_beta hdr_end(host) -i beta.asana.com

  use_backend beta_ws if is_beta
  use_backend lists_ws if is_lists
  default_backend ws

frontend ws_ssl_in
  mode http
  bind *:443
  default_backend ws_ssl

backend ws_ssl
  mode http
  option tcplog

  log /dev/log local0 debug

  balance leastconn
  server beta_ws_0 beta_ws0.ec2.:8443 check

backend ws
  mode http
  option tcplog

  log /dev/log local0 debug

  balance leastconn
  server ws_0 prod_ws0.ec2.:8080 check

backend beta_ws
  mode http
  option tcplog

  log /dev/log local0 debug

  balance leastconn
  server beta_ws_0 beta_ws0.ec2.:8080 check

backend lists_ws
  mode http
  option tcplog

  log /dev/log local0 debug

  balance leastconn
  server ws_2 prod_ws2.ec2.:8080 check

listen stats 0.0.0.0:8090
  mode http
  stats enable
  stats uri /haproxy
  stats refresh 5s

On Tue, Apr 27, 2010 at 10:09 PM, Willy Tarreau <w#1wt.eu> wrote:

> On Tue, Apr 27, 2010 at 05:36:12PM -0700, Dustin Moskovitz wrote:
> > Actually, I spoke too soon. When I create a config similar to Laurie's
> (see
> > below), I find that I still don't see headers and my default backend is
> > always utilized. I originally mistook this for working. Laurie, are you
> > positive you are actually routing traffic based on the hdr rule? This
> > implies to me the default is tcp? When I explicitly declare mode http in
> the
> > backends, the client never sees the response, as I mentioned in the first
> > email.
>
> You don't have any "mode http" statement, so haproxy does not wait for
> any HTTP contents to decide what backend to use.
>
> You have no httpclose/http-server-close option, so you're working in
> tunnel mode for backwards compatibility, where only the first request
> is processed and the rest is considered as data.
>
> Please just add "mode http" and "option http-server-close" to your
> defaults section and it should be OK.
>
> Regards,
> Willy
>
>
Received on 2010/04/29 01:51

This archive was generated by hypermail 2.2.0 : 2010/04/29 02:00 CEST