Re: HA Proxy as Amazon S3 Content Switch

Hi Patrick,

Well, several other companies have used S3 to that end and served many magnitudes of traffic more than I intend to use it for. For example, Twitter, among others, use it. (Or at least they used to---no idea what they do now.) To that end I think it's a little harsh to judge the quality of the service based on the fact that the company who makes it doesn't eat its own dog food. Amazon and their infrastructure have been around for a lot longer than S3 so I'm not shocked them wouldn't immediately migrate to it. That, and Amazon is a *global* company and they need global content dissemination. S3 is not worldwide (unlike Akamai) so no nice edge stuff for them. In fact, for Amazon, that one fact is enough to disqualify it from usage.

I think that, for the price, and based on have a primarily US user base, S3 is a good option. Plus, when you use it in conjunction with EC2, the pricing is many orders of cheaper than any commercial CDN solution I know of.

Thanks for the read though... Maybe looking into a squid frontend is a better way to go, though I'd love to get this working.

Best,
JLS On Sep 27, 2008, at 5:12 PM, Patrick Viet wrote:

> Hey John,
>
> Sorry I haven't any insights about weird s3 behaviour.
>
> I'm just wondering : apart from the fact that s3 is so fast and
> reliable that amazon themselves don't even use it for their own
> website (they use akamai instead), why the hell would you want to use
> it as a backend ??
> That makes you pay twice the bandwidth ...
>
> Patrick
>
> On Sun, Sep 28, 2008 at 1:46 AM, John Singleton <jsinglet#gmail.com>
> wrote:
>> Hi All,
>> I've been using HA Proxy as a straight load balancer for some time
>> now and
>> decided to check out the Layer 7 features in version 1.3.
>> Specifically I am
>> trying to get content switching working with Amazon S3 as the
>> backend. I've
>> tried just about everything but I can't seem to get it going right
>> and I
>> suspect the reason has something to do with the way S3 handles http.
>> The setup:
>> global
>> log 127.0.0.1 local0
>> log 127.0.0.1 local1 notice
>> #log loghost local0 info
>> maxconn 4096
>> chroot /usr/share/haproxy
>> uid 99
>> gid 99
>> daemon
>> #debug
>> #quiet
>> defaults
>> log global
>> mode http
>> option httplog
>> option dontlognull
>> retries 3
>> redispatch
>> maxconn 2000
>> contimeout 5000
>> clitimeout 300000
>> srvtimeout 300000
>>
>> ##
>> ## HTTP Only Frontend
>> ##
>> frontend app X.X.X.X:80
>> mode http
>> acl image_content url_sub images #simple example to trigger
>> the
>> static cdn
>> use_backend static_content if image_content
>> default_backend dynamic_content
>> backend static_content
>> mode http
>> option httpclose
>> balance roundrobin
>> reqrep ^([^\ ]*)\ /(.*) \1\ /my-content-cache/web/\2
>> server CDN s3.amazonaws.com:80
>>
>> Here's what SHOULD happen.
>> 1) Url for /images/foo-image.jpg is requested
>> 2) HA Proxy forwards the request to the backend, static_content
>> 3) Rewrite rule rewrites request from /images/foo-image.jpg ->
>> /my-content-cache/web/images/foo-image.jpg
>> 4) File is served!
>> However, I don't get that result. I simply get a "could not find
>> bucket"
>> error from S3. S3 is very picky about bucket naming, (ie,
>> /web//images/image.jpg is not the same thing as /web/images/
>> image.jpg) but I
>> don't think that's the problem. Testing this setup against a
>> regular apache
>> server works fine---any idea what I am doing wrong?
>> Perhaps someone can shed some light on this?
>> All Best
>> JLS
>>
Received on 2008/09/28 02:56