I'm trying to gather data on the expenses that can be expected when implementing professional, high-grade hosting in-house for a high-traffic website. This is not so much because I am looking to make this choice, as I am looking to weigh pros and cons of having in-house servers vs. managed hosting.

So, let's take a nice, professional, well-established company such as Rackspace as an example. They use ten Tier one providers with Gigabit connections, and tons of CISCO hardware (routers, switches, firewalls, servers, etc.).

As an example, let's say someone is looking to host a website with tons of traffic and concurrent users, requiring numerous simultaneous servers setup to handle all the data.

What do you all think this would cost, between the connections themselves, the servers, the routers, and all the other items involved? As an estimate, assume 10 - 20 servers will be needed.

Thanks for your help, guys!

Well, I have somewhat experience in this. I run a free file hosting website, we currently have 8 servers, including administration etc it comes to around $6000/mo. That's on a 350mbit line capped at 45tb.

Thanks for the reply, Mituozo. Do you have any hardware implemented other than the servers themselves? How much did all of the hardware cost up-front?

In-house (assuming that you currently have no infrastructure setup) is going to cost you four or five times more than it would to colocate in a datacenter. Which estimate are you looking for?

My personal preference is to colocate the hosting, as I completely agree that it would be much more expensive to have it in-house at this time. My associate, though, is pushing for in-house hosting, and as a result I'm looking for an estimate of the high costs of in-house hosting to show him the disadvantage to this choice.

It depends where in-house is. If you're in a decent office building in manhattan, chances are that the building already has a good portion of the requirements needed to move forward -- good power, connectivity, etc., It obviously wouldn't be as good as a datacenter but i've seen racks in manhattan in normal office buildings, and those servers had a respectable uptime.

The main thing that'll distinguish rackspace from going in-house is their ability to provide technical support, remote hands, and repairs immediately. They will already have spare parts for immediate repair, and the ability to do it within an hour.Would you be able to say the same? Rackspace isn't cheap though and it might be comparable, in the long run, to making your own investment into all the hardware needed.

Just as a suggestion, you might try reverse engineering your question a bit.

Do you currently have the expertise in house to handle Cisco (or Juniper, or whatever) configurations, troubleshooting, or daily maintenance? If not, do you know what the going rates are for an infrastructure engineer in your area? Same goes for server hardware maintenance and server OS maintenance. And firewalls, and switches, and so forth.

Also, any major colo worth its salt has certain building enhancements such as environmentally controlled rack spaces, redundant power to the actual facility itself (generators with a minimum of 72 hours fuel as an example), dehumidifiers, UPS, off-site backup storage... The list goes on and on. Did I mention having to throw yourself on your own sword if things blow up?

Between initial equipment & service costs and ongoing maintenance, unless you plan on getting into that particular business on a full time basis, your money is probably better spent elsewhere. This ain't going steady, this is getting married! ;)

The answer really is predicated on the location like rohitj said as well as your expertise. Different locations, have different price tags. The price for power in California is different than that in Texas, and the price of Bandwidth is different in Los Angeles, California, than it is in Frankenstein, Missouri.

Also, when you are starting out, you won't have to worry about getting 10GigE lines from various providers, running your own BGP etc.. If you don't know how to configure switches, routers, servers, etc.. I would strongly advise against starting a colo, or even doing any colo that isn't fully managed by another provider who has the proper expertise. My suggestion to you, would be to colo with a DC that does that for you, and build out that way.

-Steven

My personal preference is to colocate the hosting, as I completely agree that it would be much more expensive to have it in-house at this time. My associate, though, is pushing for in-house hosting, and as a result I'm looking for an estimate of the high costs of in-house hosting to show him the disadvantage to this choice.


What if you colocated in a place nearby? That way you would still have physical access incase you needed to hard reboot a server or replace a broken item, but not have the high cost of in-house.

Everything you are saying definitely makes sense. Between the hardware itself, the network admin team to make sure it runs smoothly, and all the recurring expenses, it would be quite a task to undertake the hosting ourselves.

What if you colocated in a place nearby? That way you would still have physical access incase you needed to hard reboot a server or replace a broken item, but not have the high cost of in-house.
As rohitj and insiderhosting said, it would be easier to do in a place that at the very least has some of the necessary resources available. Unfortunately, fate has landed my company in a small town in Florida, and I doubt anyone has even heard of colocation. :-)

So, just to get back to my original question, if I were to have...
All the hardware that is needed to run a standard high-traffic operation (servers, routers, switches, etc.)
A fast ethernet connection (ten gigabit is probably pushing it; a one gigabit might be more realistic)
A standard person/team to administer the hardware and software (I have no clue how many people it would take to do administer my 10 - 20 server configuration)
A "small" backup generator of some sort (12 - 24 hours)...and if I were to leave out the additional perks of a real datacenter, such as dehumidifiers, outside backups, and whatnot...

How much should I expect to pay, as an estimate? I know that it's obviously going to vary, but I'm looking for some sort of ballpark. $1k/mo.? $10k/mo.? $100k/mo.?

Thanks again for putting up with my ignorance on the subject!

I'm currently renting from a dedicated provider, I should probably be looking into co-location, it's just the setup costs are rather daunting. We just run the servers on a VLAN, no special hardware other than the servers. Using a DNS RR (Roundrobin) setup to split traffic between the web servers.

The air conditioning/electricity payments are going to eat you alive. I know this from personal experience running our own in-house datacenter for a few years. Its just simply not worth it. Everyone underestimates the amount of heat that a few servers generates.

--Tina

You want to be careful being drawn into a price comparison chart as the basis for making a decision. Management (especially CFO and controllers) love to turn everything into a simple excel spreadsheet scenario with the lowest price getting the nod.

A colo or managed provider solution is most often a much different approach. Corporate or in-house decisions often overlook the following (because including it would immediately skyrocket the cost difference).

Full routing - BGP routing is a necessity. Without it, you can't use multiple bandwidth providers and have no contingency for when (not if) a link goes down. (As soon as you require BGP, you are talking expensive Cisco or Juniper routers and expertise to configure and run them.) Many data centers have proprietary enhancements or specialized hardware that customizes their routing further.

True power redundancy - Not just dual power supplies, but physical feeds that are at opposite corners of the building, coming from different utility feeder lines and/or different power grids. (Also, fiber should enter the facilities from more than one location.) Mundane and "low-tech", but construction workers outside digging up the street and cutting lines is a actually a much more likely failure scenario than router equipment failures.

Economy of scale - Many of the capabilities that you will need (firewalls, load balancing, network management) are very expensive to deploy and require special training skills to configure and operate. Even large players such as Google and Yahoo, have for the most post, purchased space in large Tier 1 providers rather than trying to do everything themselves. They could certainly save money, yet they would rather pay other people to do it for them.

(Of course, with the new "data center build-out" race by Google and Microsoft going on in The Dalles, Orgeon and other locations, the very big guys may be coming 'round to wanting to control everything themselves because they are building facilities 3 to 10 times larger than what currently exists.)

Yeah, the Cisco router alone was $13,000 for us to be able to house around 100 servers. You can buy a heck of a lot of colo for $13k. Now, throw in another $6k a month for bandwidth and a couple grand a month for electricity and another $2k in building rent...and you've got the bare minimum requirements to run a mediocre and non-redundant in-house datacenter.

So, for around $10k a month (and a relatively huge amount of start up costs)...you can begin to think about hosting around 100 servers on your own, without redundancy and giving each of them a mere 1 - 5 mbps dedicated bandwidth. This doesn't even include other expenses like switches, A/C units, backup power, staff, etc.

Further, if you only want to host around 20 servers...its not going to be a whole lot cheaper than above. You might save $1000 in electricity a month...but your bandwidth costs are probably going to be about the same. You get cheaper bandwidth the more you buy.

For $100 a month, you can get a pretty sweet deal on colo. ;)

Seriously, its NOT WORTH IT.

--Tina

Also, don't forget to factor in maintenance on your UPS, HVAC, ATS, genset etc. These will easily run you another $1000 - $2000/mo.

If you really want to look at going in-house and have absolutely no infrastucture, except perhaps an empty room, look at this stuff from APC:

http://www.apc.com/products/infrastruxure/index.cfm

Depending on the foot print you need, you can get racks, ups, cooling all inclusive from them in a simple, comprehensive package. On top of that you're also looking at $100K in start up costs for your servers, switches, routers, power system upgrades and a team of consultants to put all all together for you. Monthly recurring you're going to be paying through the nose for your bandwidth, if you can even get the bandwidth you require -- this is primarily because you would also have to pay for a local loop, whereas in a colo environment you don't have to pay any loop charges. It's really hard to say what your bandwidth costs will be, but they will likely be 200-300% more costly then if you colocated. Your other operational expenses - power, infrastructure maintenance, expert consultants, and remote environmental monitoring/alert services, additional insurance premium will easily add up to several thousand dollars a month.

Why bother for all of this when you can buy 20 servers, a single switch and colocate them for $700-$1200/mo base for the cabinet/power plus whatever you require for bandwidth?

Rackspace and the likes have invested tens of millions of dollars into their infrastructure. You really can't do it better for cheaper, particularly at the scale you describe.

Perhaps you should be considering where the datacentres are as well. Forgive my ignorance but isn't Florida in a less-than-stable part of the world for weather? Imagine you're on the news next year, and assuming the staff are not dead, they can go live in a hotel while the servers keep humming away in their DC in Texas, or whereever.

The part about hosting yourself v colocation is, as people have pointed out, is a no-brainer. If you buy hardware from a major supplier, you can get a support from them so if any hardware fails, the engineer will come out and replace it for you. As long as you can KVM in (or use a DataCentre employee get a base OS running), nobody would even need to be near the servers. If you're buying new servers, they can even be delivered to the DC, you wouldn't even need to see them until you sell them on ebay when they reach their end of life.

As long as you can get over the warm-and-fuzzy feeling of being able to watch the flashing lights in your in-house machine room, that is :)

Words of wisdom, everyone! Thanks for your advice. :-)