TCP server hanging

hello --

i am using haproxy 1.4.8 to load balance between four read-only postgresql databases. the databases are identical, the hardware is not. my configuration is included below. the connections can be short, like a few seconds, to long running (days or more.) the loads on the connections can also be varied.

it appears to work fine but after some random amount of time it starts taking a very long time to make connections through the proxy, even though connecting directly to one of the backends takes no time at all. if i look at the stats webpage it tells me that a small number of connections appear to be queued and not moving towards being fully connected, even though there are open connections available on the backends. i am at a loss for how to diagnose this further or fix this -- typically i do a restart

haproxy -f /usr/local/etc/pg_haproxy.cfg -sf $(cat /var/run/haproxy/pg_haproxy.pid)

and it usually fixes the problem.

i'm not sure if this is unrelated but i'm having trouble getting logging messages on my rsyslogd log. on startup i get something on /var/log/localmessages but nothing else.

help?

best regards, ben

configuration :

global

        log 127.0.0.1   local0 notice
        maxconn         4096
        maxpipes        4096
        chroot          /var/run/haproxy
        user            postgres
        group           postgres
        daemon
        pidfile         /var/run/haproxy/pg_haproxy.pid

defaults
        mode            tcp
        log             global
        option          tcplog
        option          dontlognull
        retries         5
        timeout connect 1s
        timeout client  1h
        timeout server  1h
        timeout queue   10m

listen pg :9999
        log             global
        fullconn        375
        balance         leastconn

        server          tickdb1 10.1.1.76:5432 minconn 10 maxconn 100 check
        server          tickdb3 10.1.1.78:5432 minconn 10 maxconn 100 check
        server          tickdb4 10.1.1.79:5432 minconn 10 maxconn 100 check
        server          tickdb5 10.1.1.80:5432 minconn 10 maxconn 75 check

listen private_monitoring :9998
        mode            http
        stats           enable
        stats uri       /admin?stats
        stats refresh   5s