On 30-12-2009 14:04, Marcus Herou wrote:
> Hi Willy, thanks for your answer it got filtered, that's why I missed it
> for two weeks.
>
> Let's start with describing the service.
>
> We are hosting javascripts of the sizes up to 20K and serve flash and
> image banners as well which of course are larger. That is basically it..
> Ad Serving.
>
> On the LB's we have about 2MByte/s per LB = 2x2MByte/s = 4MByte/s
> ~30MBit/s at peak, that is not the issue.
>
> I've created a little script which parse the "active connections" from
> the HAProxy stat interface and plots it into Cacti, it peaks at 100
> (2x100) connections per machine which is very little in your world I guess.
>
> I've attached a plot of tcp-connections as well. Nothing fancy there
> either besides that the number of TIME_WAIT sockets are in the 10000
> range (log scale)
>
> Here's the problem:
>
> Everyother day I receive alarms from Pingdom that the service is not
> available and if I watch the syslog I get at about the same timings
> hints about possible SYN flood. At the same timings we receive emails
> from sites using us that our service is damn slow.
>
> What I feel is that we get "hickups" on the LB's somehow and that
> requests get queued. If I count the number of rows in the access logs on
> the machines behind the LB it decreases at the same timings and with the
> same factor on each machine (perhaps 10-20%) leading me to think that
> the narrow point is not on the backend side.
Maybe interesting, maybe not: I had some problems like this as well and in my case I think this was caused by the limited number of outgoing ports from my proxy machines..
I don't use connection keep-alives, and I think my balancers were reusing ports faster than the backend windows machines could handle or something like that.. Anyway, after I changed my FreeBSD's available outgoing ports range, all problems were solved again.
Here's my sysctl for my FreeBSD 7.2 machines, but as Willy said, this might not work for everyone.
kern.maxfiles=65535 kern.maxfilesperproc=32767 kern.ipc.maxsockbuf=16777216 kern.ipc.somaxconn=32768 net.inet.tcp.rfc1323=1 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.inflight.enable=0 net.inet.tcp.hostcache.expire=1 net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535
@Willy or someone else, feel free to comment on these settings if you see something strange.
-- With kind regards, Angelo Höngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 A.Hongens#netmatch.nl www.netmatch.nl ------------------------------------------Received on 2009/12/30 14:16
This archive was generated by hypermail 2.2.0 : 2009/12/30 14:30 CET