Hi,

I have a few questions that I was hoping you could help me with please.

We run a social networking site heavy on images/photos etc. etc.
All static content though.

Our main servers are placed in Chicago, but our end users are actually all from Chile in South America.

Our idea is to serve all the static content and the images through a CDN, but due to the fact that Akamai are very expensive down here, and also that there are hardly any providers with nodes down here in South America (except the odd Brazilian node now and again), we are thinking of making our own CDN!

The idea is to set-up a dedicated server locally here in Chile and convert this into an edge node that would cache the static content from our servers in Chicago and ultimately serve it for the end users locally here in Chile.

1. First of all - Would this idea work?
2. And would we need to have a special software on the local server to make this work?

I came across this article (Search Google: how-to-build-your-own-cdn-using-bind-geoip-nginx-and-varnish) that talks about combining Nginx, Vanish, BIND and GeoIP - this sounds great, but almost too easy.... Any comments on this?

Also - we currently use memcached - what would you suggest between Vanish and Memcached, and why?

Thank you in advance for any input you could give me!

A CDN is really overkill for what you're looking to do, since your target customers are all in one geographic location. If you just want to host static content for your own users, then a single server or VPS connected to the ISPs that your users use would be best. If your bandwidth needs are not very high, than a server in Chile itself makes sense, perhaps with hosting the rest of the website in a location near to, but not necessarily in, Chile.

Most bandwidth in South America is prohibitively expensive, which is why most South American hosting is actually done in Miami or other locations in the southern United States. So it may make sense to host only part of the website in Chile (maybe the dynamic content only, or the static content only), and the rest of the website somewhere in the US, or it may make sense to host all of the website somewhere else in the US other than in Chicago where it is hosted now. Chicago is not an ideal location for reaching any part of South America, and will probably add around 100ms to your ping times compared to other locations in the US.

My understanding is, that most of the east half of South America connects to the internet via Miami, often even when connecting between different providers located in the same region. Since Chile is on the west half of the continent, I'm not sure what location would be best to reach the ISPs in that country. My understanding is the mountains in the middle of the continent make it difficult to lay fiber from the east to the west, so I would assume that a location that does a good job of reaching Brazil would not necessarily do a good job of reaching Chile, for example.

Because of this, I would try doing some pings or traceroutes to various providers in Los Angeles, Dallas, and Miami, and see which ones provide the best connectivity to the ISPs that your customers use. At that point, getting a VPS or a dedicated server to serve static content (or your whole website) from this more optimal location should provide the best tradeoff between cost and performance, without having to use a CDN.

You are right in your observations, a full on CDN provider would be overkill, and the mountains provide quite the barrier right through the whole continent complicating things a bit.

What we are think of doing now is as you suggest I think.

We will set-up a dedicated server placed here in Chile and then just have it serve the static content to the users.
I see your point with regards to Chicago not being optimal for serving the western part of South America (or any place in South America for that matter) but we have a killer deal for some powerful servers up there that no one have been able to beat yet ,"unfortunately", so at the end of the day I think it turns out ok if we can only get a machine set up on the spot down here to serve the static content.

We will try and set it up the server down here with the combination of NginX + Varnish + BIND + GeoIP.

I think this will do the trick, but I am off course open for suggestions and comments if anyone has technical experience in this matter?

Are you setting far future expires headers for the images?

If you do, then the users' ISPs' proxy servers may well cache the images for you.

Now there's an idea!
I wasn't aware of that trick actually.

I'll check that out - looks good so far from what I'm reading online.

What resource/site would you suggest to check out so as to implement it without any problems please?

Now there's an idea!
I wasn't aware of that trick actually.

I'll check that out - looks good so far from what I'm reading online.

What resource/site would you suggest to check out so as to implement it without any problems please?
askapache.com:

http://www.askapache.com/htaccess/apache-speed-expires.html

You are right in your observations, a full on CDN provider would be overkill, and the mountains provide quite the barrier right through the whole continent complicating things a bit.

What we are think of doing now is as you suggest I think.

We will set-up a dedicated server placed here in Chile and then just have it serve the static content to the users.
I see your point with regards to Chicago not being optimal for serving the western part of South America (or any place in South America for that matter) but we have a killer deal for some powerful servers up there that no one have been able to beat yet ,"unfortunately", so at the end of the day I think it turns out ok if we can only get a machine set up on the spot down here to serve the static content.

We will try and set it up the server down here with the combination of NginX + Varnish + BIND + GeoIP.

I think this will do the trick, but I am off course open for suggestions and comments if anyone has technical experience in this matter?

Nginx with it's caching module would work fine, instead of throwing varnish into the mix. Either way, you'll want the expire headers to be appropriate, otherwise the caching won't know what to cache.

As to the servers, I would be curious to know what kind of server power you're getting at what price, if you don't mind sharing, either here or via pm. If your site is cpu heavy, and you can't get reasonably priced cpu in Chile, it definitely makes sense to offload the static content in Chile and leave the dynamic stuff elsewhere.

Chile->Chicago?

Wouldn't providers in CA,TX,FL do much better for you?

Just take a look at the cable maps to decide.

Check out Aflexi for a do-it-yourself CDN platform.

NginX and GeoIP would do the trick for the locally placed server you say?
Ok, will check this out.

With regards to why we stay with our Chicago provider is that they gave us state of the art 12GM ram machines for around 275 USD/month. I haven't been able to find a better offer on the west coast unfortunately.

I talked to the Aflexi guys and they know what the are talking about without a doubt.
I think I'm just too cheap to pay for something that I hope we can set up ourselves. I might eat these words later on and come crawling back to them at a later time though ;-)

$275 / mo each? And by state of the art do you mean, dual cpu's (8 core), with fast hard drives, tons of bandwidth, etc etc?

Just the 12gb / state of the art part, is open to a lot of interpretation.

To do the geoip, you'll want to do it on your main webservers.

Basically, whenever you have say, an <img> tag, with src=http://chicago.static.oursite.com/images/whatever.jpg, if the user is in Chile, replace the src with img src=http://chile.static.oursite.com/images/whatever.jpg

So you'll need geoip installed on the main webserver, and it will have to change what links it shows to users who are in different geographies. Then you make sure that the "static" server in both geographies have synchronized / identical content, and away you go.

Ok, state of the art a few months back maybe :-)

Intel Quad Core i7 920 - 2.66GHz - 8MB Cache
12 GB DDR3 1333Mhz RAM
2 x 300GB SAS 15,000 RPM - 16MB Cache
100 Mbps Public
6,000 GB Public Bandwidth

I would off course be happy if someone on the west coast could beat that price?

Great info - thanks for the input regarding the GeoIP, makes better sense that way :-)

That's a good price but not amazing. You might want to consider colocation at pacificrack, since similar hardware runs less than 2k per server. The bandwidth, space, and power you'd need is pretty cheap, so you could make your money back on hardware pretty quickly. How many servers do you have of those specs?

Hi, we have 1 web server and 1 DB server with similar specs.

What west coast companies would you recommend with regards to price/power in the range of what we have now?

I haven't really looked into renting servers recently, mostly I've been colocating servers and working on setting up my VPS rental company. Just looking at how I figure what makes sense to me, in terms of how I price my VPS's, I try to price it out at a 10 month ROI on equipment, plus the costs of bandwidth and power and space, which, if I were offering dedicated servers, I'd do the same thing.

If you look at it that way, if I were to sell dedicated servers out of my space, I would look at the cost of the hardware (2.8ghz xeon 3460, 12gb ram, supermicro motherboard with SAS, hot-swap 4 drive bay 1u supermicro case, 2x300gb 15k SAS drives), that's probably $1900 worth of equipment, so the fair price on the equipment is $190 / mo. The power / space is probably around $25 / mo (based on 1/40th of a full rack with 40a costing $1000). The bandwidth, that really depends. If it were me, for this kind of setup, with 6000gb included b/w, would probably price that at $50 / mo.

So yeah I mean, the price is fair. If I added up all the costs and what I personally would charge to recoup those costs, given what I'm being charged at my current datacenter, I would probably also be charging the same rate you're paying now, just in LA instead of Chicago.

To directly answer your question, I colo with pacificrack.com / quadranet.com. They can probably offer dedicated servers with those specs at around the same price you pay now.

Well, if it's just dynamic content serving from other locations and latency is not the biggest concern, webnx can do very good deal in high end hardware though.

Aflexi will be in place when he needs to serve out of Chilean market. :agree: