Re: haproxy and orbited (COMET server)

On Fri, Feb 01, 2008 at 12:04:36AM -0700, Christoph Dorn wrote:
> This exchange has been most beneficial. Thank You. I just want to touch
> on two things you mentioned to wrap things up for now.
>
> 1) "LB or FW silently timing out"
> - under which circumstances do you think this would occur?

after the inactivity timeout defined in the equipment. Every stateful equipment must create and maintain sessions in a table. After the session has been unused for some time, it will get purged without notification, because if it gets purged, this is because it is assumed that the client, the server, etc... are not there anymore. It is very comparable to an applicative session you would maintain on the server side. Once it expires, you remove it. You don't send a mail to the user to inform him that you removed his session :-)

> - after how long? Could this happen regularly within the 1 to 2 minute
> mark (I know there are lots of variables and different implementations,
> but from a protocol/conceptual and standard implementation point of view?)

It depends where the component is deployed. The outer the component, the slower the timeout because the higher risk of attacks. If a firewall keeps sessions for one hour, and supports 1 million sessions in its table, it means that you can saturate it with only 278 new sessions/s !

Also, more fragile firewalls which do not inspect sequence numbers, such as early netfilter in 2.4, or many commercial firewalls will have smaller timeouts because of the increased risk of getting new fake sessions created due to their lack of correct protocol inspection.

It is not uncommon to encounter firewall timeouts between 2 and 5 minutes when connected directly to the net. More robust firewalls such as recent netfilter, PF or cisco PIX may have larger timeouts counted in hours because of the low risk of getting fake sessions created.

A packet-based load balancer such as LVS or an Alteon will also generally have a small timeout. Most often, there are two types of timeouts, the fast ones and the slow ones. The slow ones are for completely established sessions, while the fast ones are for incomplete or half-closed sessions. This is often required because those load balancers may be deployed in "direct server return" (DSR) mode, in which they see only half of the stream. Obviously, having more than a few minutes timeout there is suicidal.

Generally, you could consider that such silent session-killers will not drop your sessions before 1 minute, and most commonly before 2-5 minutes. I also notice that timeouts are generally shorter when NAT is in use than when it is not, probably because of the higher cost for the equipment.

> - Whats the underlying difference between a "silent" timeout and a
> "clean" one where haproxy and the client would correctly close the
> connection?

The "silent" timeout is when the session is simply flushed from the component, which causes silent packet drops to further packets related to that session. Thus, both your client and server are still connected to each other but cannot communicate anymore because the packets they try to exchange are blocked by an intermediate equipment which has no known context for them.

The "clean" timeout is what is performed by any active equipment which relies on a real TCP stack. The session is then closed at an application level (the proxy, such as squid, haproxy, apache, ...) and the system sends a FIN to the other end indicating a wish to terminate the communication. Obviously, the other end had better respond quickly, before intermediate components flush the session which is now in half-closed state, but this is normally not a problem.

> - May the intention of a silent timeout from a FW or proxy be to trap
> your client as it matches criteria of typical malicious traffic?

Not at all. It is because it regularly checks its session table for obsolete sessions, and cleans them many at a time. The task responsible for doing this rarely has the ability to build a packet and send it. Also, poor firewall implementations which do not track sequence numbers will not even be able to send a valid packet because they won't know what sequence and ACK numbers to put in the packet. But some rare clean firewalls do this.

> - I guess what I am trying to get at is what timeout would you use that
> falls within the parameters of normal operation for most standard
> setups? The objective is to fall under the radar of any
> firewall/proxy/security appliance. Would 50 seconds be reasonable or too
> long?

50 seconds should pass most components. At worst, some anti-virii proxies would finally break the connection, but it will then be a clean timeout.

> 2) Port 443. Thats interesting. Never thought of that.
> - What do you think is the likelihood of intermediary agents detecting
> non-encrypted or non-standard traffic patterns and subsequently blocking
> your access?

The places where you would need an alternative port would most likely notice this, because creating tunnels through proxies is a well-known method to do remote SSH or even P2P. If a proxy or an IDS notices data which does not look like SSL (ie: not beginning with 0x16 x0x03 0x01 for instance), it will most likely break the connection.

> - So you believe if the client is unsuccessful in connecting to port 443
> it will return immediately rather than a lengthy timeout?

Yes, because when a client cannot connect to the net on a port different from 80, it is because it is firewalled/proxied. If it is firewalled, either the firewall is open on port 443, or it will immediately reject outgoing packets, otherwise web browsing would basically not work well for internals.

If the client is proxied, either port 443 will be open and forwarded, or it will be blocked by security/administrative policies. In both cases, the client will either be asked to authentify or simply blocked with an appropriate error message.

> - So in theory COMET should work as intended over port 443 without the
> limitations that HTTP has in terms of long-standing connections? So it
> could be used for streaming?

you would have the same limitations, but less chances of encountering some of the problematic components such as anti-virii. But proxies also have timeouts on tunnels of course, and silent connection drops will still exist on firewalls and load balancers. Using another port will just help you run two different services on the same IP without having to resort to content-switching on the server side.

> Please excuse my network terminology. My background is primarily in
> non-network related development.

no problem :-)

Cheers,
Willy Received on 2008/02/01 08:28