I have co-lo'd with a little local ISP in my town, but they really don't have it all together as a DC. They're primarily an ISP for dial ups. I have thought long and hard about it, and have convinced if I stick to my goal of fanatical customer support, I'll have to run my own DC. My current colo takes over a week just to allocate me a new IP address and get a server slot ready. I'm not sure what the bottleneck is, but it looks like getting new IP's is tough for them. Any way, the question is:

What is the minimum network gear (mid range grade with upgradeability to OC3) I need to provide redundant T3 connectivity to my customers?

I saw a CoLo solution (here: http://www.foundrynet.com/solutions/appNotes/CoLoApp.html) built on Foundry gear that uses 2 NetIron 400's connected to T3's from different carriers as the core layer. These are connected to each other via Gig-e copper to provide OSPF and/or BGP4 routing. The back end is connected to a pair of BigIron 8000 Layer3 switches, which serves ethernet connection to customer servers.

Is this the cheapest hardware available? I looked at cisco, and it looked to be roughly 50% more expensive.

Also, how do you keep spare IP's on standby without catching heat from ARIN (or whoever is going to be giving me ip's). Everything I've seen wants 80% utilization of IP addresses. What's a start-up DC to do? I don't want to be it the same situation as my current Co-Lo.

The DC location is in a medical office complex less than 1 block from our major hospital, so it has a good power grid to start with, but I'll be buying a 130 KW Generator, a UPS big enough to feed 12 42U racks + 1 Network gear rack (75Kw estimated). Any other must haves to get started?

Really, your best bet would be to get help from a company who builds/maintains DC for a living. That way they can help plan for everything.

We have used The Bick Group for several things, http://www.bickgroup.com/index_split.html

Hope that helps!

For two DS3s the Foundry boxes are overkill, get a couple of Csico 3640s off ebay and plug those into your DS3s. Run a pair of Foundry or Extreme switches behind the 3640s -- if configured right that will give you redundancy in your core. When you are ready to move to OC3s, then you can move to the Foundry gear.

there are all kinds of things you can do in regards to your IP question.. It also depends on the contracts you sign with your providers, but the simplest thing to do would start off with a /24 and then add IP's as you grow.
They should have no problem giving you more IP's if you can justify them.. if you go to them asking for a /22 most likely you will have to submit a network map including where ip blocks are allocated and such and so forth. This is simple enough to forge , however you may be asked upon to provide additional information such as domains hosted within your network, blahblahblah.

talk to your sales rep and see what they can swing. most are willing to work with you.. just dont ask for an obscene amount of IP's to begin with.

Originally posted by uuallan
For two DS3s the Foundry boxes are overkill, get a couple of Csico 3640s off ebay and plug those into your DS3s. Run a pair of Foundry or Extreme switches behind the 3640s -- if configured right that will give you redundancy in your core. When you are ready to move to OC3s, then you can move to the Foundry gear.

Maybe you want to take a few full views on those 3640s too. PM me when you post here about them not handling anywhere near line rate. Make sure to heavily ACL things up as well. That will speed things up.

</sarcasm>

Honestly though... if you're going to start this with 3640s, you may want to also read an article I saw in the news recently about converting your typical convenience store faux-flower tube into a crack pipe. If you're going to do ghettocolo, do it in style:

BigIrons are cheap enough on the used market and can handle a few bgp views and near line rate quite well. It may be a switch, but I'd take it any day over a 3640 for this application. Of course, you're not going to terminate a DS3 in those things. You can terminate an OC-3c though. You'll probably spend just as much on the cisco gear, judging by the auctions I've been to lately.

If you get some extremes, don't use them for anything other than L2. Better yet, sell them for scrap metal - with what they are selling for at auction, you can make a nice living this way.

But before we even decide what equipment to buy, the first problem is that running your own datacenter is a slippery slope. My advice is find a decent carrier neutral facility near you and run ethernet to your gear. You'll probably avoid gray hairs for a few more years and gain the same number of years on your life expectancy. You'll probably cut a profit too.

Originally posted by manmythlgnd


Maybe you want to take a few full views on those 3640s too. PM me when you post here about them not handling anywhere near line rate. Make sure to heavily ACL things up as well. That will


A 3640 is more than adequate to handle a DS3. I would not recommend it for anything faster than that, and certainly would not try and plug two DS3s into it, but I have tested 3640s and they can handle a DS3 connection no problem.


But before we even decide what equipment to buy, the first problem is that running your own datacenter is a slippery slope. My advice is find a decent carrier neutral facility near you and run ethernet to your gear. You'll probably avoid gray hairs for a few more years and gain the same number of years on your life expectancy. You'll probably cut a profit too.



I second that

But before we even decide what equipment to buy, the first problem is that running your own datacenter is a slippery slope. My advice is find a decent carrier neutral facility near you and run ethernet to your gear. You'll probably avoid gray hairs for a few more years and gain the same number of years on your life expectancy. You'll probably cut a profit too.


I would have to agree..

But before we even decide what equipment to buy, the first problem is that running your own datacenter is a slippery slope. My advice is find a decent carrier neutral facility near you and run ethernet to your gear. You'll probably avoid gray hairs for a few more years and gain the same number of years on your life expectancy. You'll probably cut a profit too.

I have thought about that, but the DC I'm currently in does not appear to offer me any advantages except for whatever seems to take a week to provision IP's. Sure, they have a Generator, but they don't provide UPS. They DO have 3 OC-3's in house, but That doesn't mean much to me. I can grow bandwidth starting a T-1's if I wanted to do this cheaply.

Maybe they are doing a bunch behind the scenes I'm not aware of, but it sure doesn't look like it.

If your only challenges are IPs and space, rent a whole rack from them and you can provision your space as needed. When you get 2/3 full, start the next rack.

As for the IPs, if this is the only real issue, order your own space from ARIN and have your ISP route your own IPs.

Depending upon the required reliability and redundancy needed, there is a lot more involved in building a data center than power and Internet connections. Mike suggested earlier contacting a contractor who builds DCs. This is a great way to learn what is actually involved.

What i ment, and what i think uuallan means is find a carrier neutral facility such as switch and data, or _insert here_ (www.switchanddata.com).

they provide a CNF in which you can lease rack/cabinet/or caged space. They provide the security, the power, the UPS, the generator..

You can then contact the backbone providers within that specific facility and pay for a cross connect between their equipment and yours. the nice thing about this is you can cut the costs of local loop and scale your bandwidth whenever you need.

it just depends on your business plan and financing I suppose.. but if you're looking to grow at a slower and at a more steady pace I would suggest going this route.. your overhead will be much less.

Originally posted by Brad@RackMy
If your only challenges are IPs and space, rent a whole rack from them and you can provision your space as needed. When you get 2/3 full, start the next rack.

As for the IPs, if this is the only real issue, order your own space from ARIN and have your ISP route your own IPs.


That's a good point. Thanks for the advice. My main goal is to be able to turn up a customer in < 48 hours. If my DC can give me a /24 block or will route one for me, I'm set. I'll put my own switch at the top of each rack, and get an ethernet drop for each rack from them.


Thanks

Originally posted by nikko


I have thought about that, but the DC I'm currently in does not appear to offer me any advantages except for whatever seems to take a week to provision IP's. Sure, they have a Generator, but they don't provide UPS. They DO have 3 OC-3's in house, but That doesn't mean much to me. I can grow bandwidth starting a T-1's if I wanted to do this cheaply.

Maybe they are doing a bunch behind the scenes I'm not aware of, but it sure doesn't look like it.

Another thing is... do you really want to deal with circuits like DS3's, OC3's and T1's? Ethernet is so cheap and you get a sweet port density. You don't have your telco randomly breaking your circuit either. You can get Big Iron and drop a bunch of ethernet handoffs in there, pull a few BGP tables and you are rocking - you won't need to upgrade for a while. You won't pay ridiculous prices for the port density either.

Originally posted by uuallan


A 3640 is more than adequate to handle a DS3. I would not recommend it for anything faster than that, and certainly would not try and plug two DS3s into it, but I have tested 3640s and they can handle a DS3 connection no problem.




I second that

In my experience it's going to top off at 30mbit and be a tight fit with a few full views - it's not as cost effective as the alternatives out there like a used Juniper M5 or M10. 7200vxr's are cheap off auction these days too.

Originally posted by manmythlgnd


In my experience it's going to top off at 30mbit and be a tight fit with a few full views - it's not as cost effective as the alternatives out there like a used Juniper M5 or M10. 7200vxr's are cheap off auction these days too.

In our tests we were able to max out at 60mbit (all Ethernet traffic, so YMMV with a HSSI). Of course you are right that a 7200/M5/M10 is going to be a much better choice.

You can find the 36XX series for about 1/2 of a Cisco 72XX and 1/3 that of a Juniper. May be a better solution, you can always upgrade when you get up there.

You're in the same place I am, and unless I missed a supplier around here, they're the best we've got, just give Hiwaay a call the other option is in KMC but the feeling I had when talking to them is that they're doing COLO in a frantic effort to get some cashflow going, my hunch is they got burned for a lotta losses thanks to some dot coms going bust on them with big bills.

I will say from shopping around, bandwidth ranges from Expensive to Ridiculous around here, if you want to do this I'd highly suggesting moving your ops to Atlanta, unless you have some really good connections maybe with KMC or Bellsouth.

Best quote I got was around $5500 for 10 mbits of a DS-3 from .. Bellsouth I believe, 2800 for local loop, 2700 for the 10 mbits. Full DS-3 was quoted near abouts $13K per month including local loop, that's not quite so hard to swallow. But if you want anything like redundancy you're gonna need another one from someone else. So call it $11K per month for 10 mbits of redundancy. Rent of course ain't much around here, I've seen 10x13 rooms renting for $200 a month, next to the bandwidth rent for a nice small DC ain't gonna be crap. You could likely go without generators for a while.

Keep me informed on what's happening I'm starting to think about having my own data room (wouldn't call it a center) as well but right now the figures ain't adding up for me.

Speaking of get with me sometime I need a good place to send folks who absolutely need Windows hosting and I'd rather send 'em local (we're all Linux/Apache no Windows here).

Edit:

Speaking of, I've got a rack nearly full and haven't had trouble getting IPs, I currently have 2 64 blocks and just a single Ethernet drop into my rack it took about 3 days after I requested it to get the 2nd block of 64.

Just fyi.

Originally posted by Dixiesys


Best quote I got was around $5500 for 10 mbits of a DS-3 from .. Bellsouth I believe, 2800 for local loop, 2700 for the 10 mbits. Full DS-3 was quoted near abouts $13K per month including local loop, that's not quite so hard to swallow. But if you want anything like redundancy you're gonna need another one from someone else. So call it $11K per month for 10 mbits of redundancy. Rent of course ain't much around here, I've seen 10x13 rooms renting for $200 a month, next to the bandwidth rent for a nice small DC ain't gonna be crap. You could likely go without generators for a while.


Wow. That's crazy... you guys must have to pull the circuit a decent distance to pay 2800 for the local loop. Of course, I'm used to doing business in large cities where zero mile loops are not a problem. Although, I'd much rather stick to my cheap ethernet cross connects and sub-$100/mbit bandwidth pricing (on very low commits mind you). I probably make up the price in space, but $500-750/rack isn't so bad when my bandwidth costs are an order of magnitude less.

Is that retail or quotes from a reseller?

http://www.newarchitectmag.com/documents/s=5512/new1013637498/1.htm

Originally posted by brian-WHT
http://www.newarchitectmag.com/documents/s=5512/new1013637498/1.htm

Brian, excellent article, but it is important to note that it was written by someone at AboveNet, a colocation company, so the article may be slightly biased :D.

slightly maybe, but there is truth to it ;>

Originally posted by Dixiesys
You're in the same place I am, and unless I missed a supplier around here, they're the best we've got...

Rent of course ain't much around here, I've seen 10x13 rooms renting for $200 a month, next to the bandwidth rent for a nice small DC ain't gonna be crap. You could likely go without generators for a while.

...

Keep me informed on what's happening I'm starting to think about having my own data room (wouldn't call it a center) as well but right now the figures ain't adding up for me.

...

Speaking of get with me sometime I need a good place to send folks who absolutely need Windows hosting and I'd rather send 'em local (we're all Linux/Apache no Windows here).

Edit:

Speaking of, I've got a rack nearly full and haven't had trouble getting IPs, I currently have 2 64 blocks and just a single Ethernet drop into my rack it took about 3 days after I requested it to get the 2nd block of 64.

Just fyi.

Thanks Dixiesys.

We're talking locally here, but others may get something out of it, so I'll keep it in the public forum.

I am (possibly) buying an office building in the medical district after the first of the year, and am asking all these questions because I'm considering turning one of the offices (it's a 4plex) into a DC. It's about 1200 sq feet, and being in the medical district, has power grid priority.

Also, I know where to get the $200 climate controlled office space, AND they already have local loop because of a PBX system, they said I could use it for free and get all the connectivity I can fit on it, but the last mile you'd lose all redundancy.

if it weren't for redundancy I'd probably already have my DS-3 ordered, it's getting that magical 2nd line that's keeping my costs prohibitive. If I could get a cogent drop for the main feed and then back it up with a good DS-3 from someone as a backup that'd probably work, but I don't reckon Cogent or Yipes is possible around here. A big/cheap main pipe with a good backup would be just fine, works for rackshack/dv2 it'd work for me too if I could get it.