Kai Krueger reported a nasty problem en encountered and analyzed. When a server goes down, it requeues all of its connections waiting in the queue into the global queue. But when a session completes after that, haproxy checks if there are pending connections that this server can handle, without taking into account the fact that the server is dead. So the server can progressively suck all pending connections from the global queue just after it has been marked down. Yes, I know, this is stupid.
A check has been added so that it does not dequeue global connections when it's marked down. It's still required that it dequeues its own connections if any exists, if the "option persist" is in use, because this option precisely means that a server marked dead must be used anyway for persistent connections (I think almost nobody uses that anyway).
I don't think that many users have been hit by this issue, or just slightly, because it requires a failing server, low maxconns and long enough response times in order to still have unreplied requests when the server is marked down. Once the last connection ended, if both queues were empty, the server would stop processing requests. Anyway, it's possible that people who are currently experiencing too many 503 errors are concerned and will see the number of errors fall with the fix applied.
In order to fix this annoying issue, I've released 184.108.40.206 and 220.127.116.11, which are available as usual here :
I specially want to thank Kai for the remarkable analysis and tests he has done on this issue.
Willy Received on 2008/12/04 23:48
This archive was generated by hypermail 2.2.0 : 2008/12/05 00:00 CET