Re: Preventing bots from starving other users?

Hi,

On Mon, Nov 16, 2009 at 04:33:34PM +0100, Wout Mertens wrote:
> Schweet! I'll give that a shot.

If you want to experiment a bit, with version 1.4 (development), you can even add a delay to all the requests from this boat. The idea is to identify the bot with an ACL and tell the TCP layer to wait for the full evaluation time before forwarding the request :

For instance, let's say that the bot does not set any user-agent. We then consider that any request with a user agent is a valid request :

frontend xxx

       ...
       acl valid_req hdr_cnt(user-agent) gt 0
       tcp-request inspect-delay 5s    # the time to wait for those which match
       tcp-request content accept if HTTP valid_req  # valid request passes
       tcp-request content accept if HTTP WAIT_END   # other ones wait
       tcp-request content reject                    # non-HTTP is rejected

You can already do that with 1.3.22 but only based on a layer 4 information (namely, the source IP address) :

       acl valid_src src 192.168.0.0/16
       tcp-request inspect-delay 5s    # the time to wait for those which match
       tcp-request content accept if valid_src  # valid request passes
       tcp-request content accept if WAIT_END   # other ones wait

Or if you know the bot :

       acl bot_src src 10.20.30.40
       tcp-request inspect-delay 5s    # the time to wait for those which match
       tcp-request content accept if bot_src WAIT_END  # bot waits
       tcp-request content accept      # other ones pass

With 1.4, it is even possible to combine that with cookies. Imagine that you add a small delay (eg: 1 second) for the first request of every user, then assign them a cookie and don't set the delay after that. If the bot does not learn the cookie (very likely), it will always suffer from the delay, for each request :

frontend xxx

       acl seen hdr_sub(cookie) SEEN=1
       tcp-request inspect-delay 1s             # the time to wait for new users
       tcp-request content accept if HTTP seen  # valid request passes
       tcp-request content accept if HTTP WAIT_END # other ones wait
       tcp-request content reject               # non-HTTP is rejected
       rspadd Set-Cookie: SEEN=1                # do not harm real browsers

Good luck !

Willy Received on 2009/11/16 20:58