Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : Need advice! Networking Equipment

[jarchee]

Hi everyone,

Brief: We migrated from being a single colo customer to a full rack. and surprisingly in less than 11 weeks we have 40+ servers. we have 100mbps connection which we are planning to double this month and by the end of the year it will be a full gigabit connection.

definition:
- With our suppliers advice, we relied on APC switched pdus for our electricity which is working great and cisco smb SFE2010 which is good for voip phones maybe!
- Our rack provider does not have fiber infrastructure so we need to rely on ethernet.
- We are using currently our full 100mbps and these servers run anything from vmware to mikrotik, centos and any other OS you can think of.

Problems:
- after some time due to a lot of broadcasts in the network the switch is flooded and the quality of our service is decreasing.
- as a solution to above problem we tried cisco 2950 and cisco 2960 but no luck. so the processing power was not an issue.

We worked with 3 different Cisco partners each prescribing different solutions as below:

- setting vlans on 2960 and having a router route the internet to vlans (we are upgrading to gigabit internet and I cannot afford to pay 25k for switch and gigabit router)

- getting layer 3 switches and sub-netting (I thought this might be a good idea)

- setting each server on a port on the router! (I have more than 40 servers and expecting to have much more during the next 9 month. How can this be possible?!)

Our purpose of business is web/vps and server hosting. Please help me to find the right choice for having my network optimized.
Thanks

[sct4a]

What is your expected growth as far as servers and bandwidth (utilization and capacity) over the next 6-12 months?

[voipcarrier]

Are you running everything on a single VLAN as one big subnet right now? If so, that's most definitely the first thing to fix.

It doesn't sound like you need a router since you're just using a single upstream. Just get a good layer 3 switch and split out your network into routed VLANs. That should fix your broadcast issue and give you plenty of room to grow.

If you need to, get a single "core" layer 3 switch and use cheaper layer 2 switches as you expand. You can use 802.1q trunking back into your main switch.

[skullbox]

You have the right idea as far as using more VLANs and routing the traffic. Now you need to find the cheapest way to do it. Try looking at HP Procurves. They have layer 3 gigabit switches and they MAY work for you. Check their specs in regard to throughput, VLANs, and their layer 3 capabilities. Last layer 3 one I used only allowed me to create 10 layere 3 interfaces, so watch out for that. I think it was the model and/or licensing on the switch I was using.

I suggested them because they are cheap, have a lifetime warranty, and perform well in data centers.

[media-hosts_com]

Ideally you will probably want a router and a switch. Especially if you are planning on growing over the next few years. There's no sense purchasing hardware over-and-over to simply outgrow it in a few months.

You can go with a layer 3 switch, but the main issue there is a single point of failure. If you're providing services for 40 servers, you should be making enough cash flow to invest in a decent setup. Even if you do some sort of leasing/financing to get you started now.

If I were in your position, I'd look at an M7i with some Juniper EX3200 switches. Enough to get you 2 gigabit uplinks (on copper or fiber) and plug in your two switches. That being said, if you have 40 servers and you're fitting everything under a 100mbit drop, then your customers must be pushing lower traffic levels. In which case this setup may be overkill. It will also use about 4U of rackspace. If you've got one rack with 40 servers you may be tight.

How much rack space do you have now? and are you thinking of bringing in a 2nd provider for redundancy eventually?

What DC are you at? Can you get bandwidth from other providers or only your colo provider?

These are all factors you should consider...

Your siggy says you're in Ottawa... That's where I'm at, so I'd be willing to have a coffee with you if you're looking for advice.

[jarchee]

Originally Posted by sct4a

What is your expected growth as far as servers and bandwidth (utilization and capacity) over the next 6-12 months?

Well, I'm looking at 180 servers for the first year.

[jarchee]

Originally Posted by voipcarrier

Are you running everything on a single VLAN as one big subnet right now? If so, that's most definitely the first thing to fix.

Yes, I'm running on defaults and thats the main weak point we got.

So far I've came across cisco 3560s

[jarchee]

Originally Posted by media-hosts_com

Ideally you will probably want a router and a switch. Especially if you are planning on growing over the next few years. There's no sense purchasing hardware over-and-over to simply outgrow it in a few months.

You can go with a layer 3 switch, but the main issue there is a single point of failure. If you're providing services for 40 servers, you should be making enough cash flow to invest in a decent setup. Even if you do some sort of leasing/financing to get you started now.

If I were in your position, I'd look at an M7i with some Juniper EX3200 switches. Enough to get you 2 gigabit uplinks (on copper or fiber) and plug in your two switches. That being said, if you have 40 servers and you're fitting everything under a 100mbit drop, then your customers must be pushing lower traffic levels. In which case this setup may be overkill. It will also use about 4U of rackspace. If you've got one rack with 40 servers you may be tight.

How much rack space do you have now? and are you thinking of bringing in a 2nd provider for redundancy eventually?

What DC are you at? Can you get bandwidth from other providers or only your colo provider?

These are all factors you should consider...

Your siggy says you're in Ottawa... That's where I'm at, so I'd be willing to have a coffee with you if you're looking for advice.

I try to have things pretty organized. Initially I was planning to have a switch per rack. but by looking at how much switches cost, I might go with one big switch yet not at this time.
I can get bandwidth from other providers but not at this point cause we don't use alot. I'll be connected to my DCs network for sometime to avoid the cost of building redundancy. I want to grow further but there are many things I and my team have to learn. we are good sellers (and buyers) but not network savvies.

[media-hosts_com]

Originally Posted by jarchee

I try to have things pretty organized. Initially I was planning to have a switch per rack. but by looking at how much switches cost, I might go with one big switch yet not at this time.
I can get bandwidth from other providers but not at this point cause we don't use alot. I'll be connected to my DCs network for sometime to avoid the cost of building redundancy. I want to grow further but there are many things I and my team have to learn. we are good sellers (and buyers) but not network savvies.

All things considered... Switches aren't that bad in-terms of price. Since you are considering to use them to their max potential here (layer 2 and 3) you should be investing in this for the long term.

Having one big switch for multiple racks of servers is a bad idea. Especially if you're looking at 200+ servers in the next 2 years.

Redundancy, availability and scalability should be the focus. At least make sure you have at one working spare of your "Big Switch" in-case something happens.

[jarchee]

Originally Posted by media-hosts_com

All things considered... Switches aren't that bad in-terms of price. Since you are considering to use them to their max potential here (layer 2 and 3) you should be investing in this for the long term.

Having one big switch for multiple racks of servers is a bad idea. Especially if you're looking at 200+ servers in the next 2 years.

Redundancy, availability and scalability should be the focus. At least make sure you have at one working spare of your "Big Switch" in-case something happens.

so you are suggesting to stick to a 3560 or 3570 for each rack. I'm fine with that. what models would you recommend?

[Jay Suds]

Originally Posted by jarchee

we are upgrading to gigabit internet and I cannot afford to pay 25k for switch and gigabit router

There's no reason a gigabit layer 3 switch should cost you so dearly, even if you want to take full BGP routes. Brocade CER2000 is under $10K for 24 ports of gigabit and can handle 512K routes in TCAM. These are 10GB capable as well. But that might be putting the horse way out in front of the cart, as it sounds like you do not have the expertise to manage a BGP speaking network. Thus, if you are just taking a default route to your provider, Cisco 3560G, Juniper EX3200, Brocade FCX are all great choices and can be at for a a few thousand each.

IMHO, there is no reason you should even be having broadcast issues if you only have 40 servers. We have customers who have well in excess of 40 servers in a single vlan and broadcast traffic is quite small.

Suggest that you look into hiring a network consultant ASAP to help you sort things as otherwise you will spend thousands of dollars and a countless number of hours making costly, service impacting mistakes.

[media-hosts_com]

We use Juniper switches on our network and I haven't followed Cisco closely enough to give any advice there...

AFAIK, both of those switches should do line rate forwarding. Maybe some other Cisco experts can give you a little more info in that.

[funkywizard]

Since you're not using tons of bandwidth, heck, a managed 10/100 switch with 2xgbit uplinks can be had for a couple hundred dollars these days. You should be able to use that to set every server on its own vlan, or at least, have a few vlans and split the servers between them. Then you'll want a layer 3 switch somewhere in your setup doing routing between the vlans, and routing to the upstream. The upstream connection should be in a separate vlan from everything else. The servers would use this switch as their default gateway, and the switch would use your upstream as its default gateway.

A decent layer 3 switch can be had for $1k or less. 48 port 10/100 with gig uplinks that supports vlans for as little as $100 depending where you look and what you get.

edit: as to the broadcast traffic, it's possible that your switch is seeing traffic from too many mac addresses. Switches can only learn so many mac addresses before their mac / port table gets full. Once that happens, it starts broadcasting what would normally be unicast traffic. It's possible that your upstream connection is chatting a lot of mac addresses across your line instead of sectioning off your uplink on a dedicated vlan. So you might want to look into that.

[jarchee]

I'm not married to Cisco but I know much less than other brands that I know from Cisco products.

I have a tech working on my issues. their deadline is tonight yet I have no idea what they are going to put in front of me as the result of their STUDIES!
I better register for a Net+ and CCNA course

[jarchee]

Originally Posted by funkywizard

A decent layer 3 switch can be had for $1k or less. 48 port 10/100 with gig uplinks that supports vlans for as little as $100 depending where you look and what you get.

edit: as to the broadcast traffic, it's possible that your switch is seeing traffic from too many mac addresses. Switches can only learn so many mac addresses before their mac / port table gets full. Once that happens, it starts broadcasting what would normally be unicast traffic. It's possible that your upstream connection is chatting a lot of mac addresses across your line instead of sectioning off your uplink on a dedicated vlan. So you might want to look into that.

Where do you get your switches from? introduce me to them here in Canada prices are a bit higher. we talk a bout thousands most of the time.

I'm on a dedicated vlan. all the broadcasts are from my IPs. there are 512 IP addresses in this lan. each assigned to a vps.