Web Hosting Talk : Web Hosting Main Forums : Colocation and Data Centers : The next step: Layer3 switch or router?

[Rijckholt]

Hi,

I am looking for some advice on how to expand my hosting set-up.

Currently we am running one cabinet with 32 servers generating almost 50 Mbit of traffic. All the servers are connected to 2 HP2524 Procurve switches that are stacked with a gigabit stacking kit. We have 4 subnets and all the routing is done by our uplink provider.

Since we are running out of space in our first cabinet we’ll start using a second cabinet very soon. One option would be to stack a third HP Procurve switch and continue on as before. Obviously this is not a very scalable solution down the road. Furthermore we would like to do some basic routing our selves and maybe some form of ACLing etc. It’s our expectation to grow our traffic to a 100 Mbit when we start filling up the 2nd cabinet.

With the 2nd cabinet we’ll also be getting a one gig fiber uplink

Someone suggested we use a Cisco WS-C2948G-L3 layer 3 switch combined with a PWR600-AC-RPS redundant power supply. Our current HP2524 Procurve switches would be connected to the cisco and used for switching and connecting the individual servers. The role of the Cisco would be to route our subnets and link-up with our colo provider.

Since this suggestion I have been reading up a bit on the Cisco 2948G-L3. It looks like we can get our hands on one via Ebay for a decent price. But one thing stays a bit unclear: performance. Some earlier posts lead me to believe that using this switch purely for layer 3 work is not a good idea because the switch won’t handle a big load (is 100mbit considered a big load?)

Who has hands one experience with the Cisco 2928G-L3 in a routing role? What kind of traffic can it handle?

Who has a better suggestion? I.e. should we buy a router? Bearing in mind that this machine will be a single point of failure so redundant power and spare parts need to be thought about as well.

Any advice will be greatly appreciated.

Thanks for your time and effort,

Onno Bakker

[jsw6]

Originally posted by Rijckholt
Someone suggested we use a Cisco WS-C2948G-L3 layer 3 switch combined with a PWR600-AC-RPS redundant power supply.

I can't imagine why anyone would suggest that. If the switch's internal power supply fails, a switch to the RPS feed will, at best, cause a switch reload. In addition, I think you'll find that access switches like these don't often experience power supply or other hardware failures. Most downtime is caused by configuration error, followed by software faults. An RPS unit is a foolish expense.

Some earlier posts lead me to believe that using this switch purely for layer 3 work is not a good idea because the switch won’t handle a big load (is 100mbit considered a big load?)

I don't think those posters know what they're talking about. That said, the 2948G-L3 is still probably not the platform for you. Doing both layer 2 and layer 3 functions on the 2948G-L3 is cumbersome at best. It's meant to be a layer 3 only product.

Who has a better suggestion?

Can you be more specific about your needs for multiple subnets? Are these customers who have their own, unmanaged or co-located, servers? Are they used for the same thing, but are assignments received from your upstream provider as you grew? Do you have any "customer managed" servers, or do you retain root/Administrator access on everything?

Bearing in mind that this machine will be a single point of failure so redundant power and spare parts need to be thought about as well.

I think you would be better served by eliminating this as a single point of failure than by throwing money (and rack space) at old gear to try to make it more fault-tolerant.

[Rijckholt]

Hi Jeff,

Thanks a lot for your reply. First of all let me tell you that I am not the technical person here. So please keep that in mind when reading some of my statements. I'll do my best to be as specific as possible:

I can't imagine why anyone would suggest that. If the switch's internal power supply fails, a switch to the RPS feed will, at best, cause a switch reload. In addition, I think you'll find that access switches like these don't often experience power supply or other hardware failures. Most downtime is caused by configuration error, followed by software faults. An RPS unit is a foolish expense.

Reading carefully on Cisco's website I concluded that the PWR600-AC-RPS can be used to get a fully redundant set-up. For this you must a so called two-to-one cable. The quasi redundant set-up requiring a reboot will be achieved when using a one-to-one cable.
Is this not correct?

I don't think those posters know what they're talking about. That said, the 2948G-L3 is still probably not the platform for you. Doing both layer 2 and layer 3 functions on the 2948G-L3 is cumbersome at best. It's meant to be a layer 3 only product.

Well, the whole idea is to use this switch only for layer3 stuff the way I understand it. We will use the switch to route subnets to specific ports. We will only connect HP procurve 2524 switches to the 2948. All the servers will (only) be connected to the HP switches.

Having read some more today, we would now also like to consider 2948's sister the 4908G-L3 switch with 8 gbic ports. We would then connect al the procurves via fiber.

Can you be more specific about your needs for multiple subnets? Are these customers who have their own, unmanaged or co-located, servers? Are they used for the same thing, but are assignments received from your upstream provider as you grew? Do you have any "customer managed" servers, or do you retain root/Administrator access on everything?

Right now we use tagged subnets assigned by our uplink provider. In that setup traffic that goes from one subnet to the other is billed twice because it has to go through the switch/router of the uplink provider. Something we want to move away from asap.
We use a subnet for our hosting platform a different subnet for server hosting for customers and another subnet for managed services customers. As we have been growing we now have 5 different subnets varying in size.

I think you would be better served by eliminating this as a single point of failure than by throwing money (and rack space) at old gear to try to make it more fault-tolerant.

Any suggestions?


Regards ,
Onno Bakker

[jsw6]

Originally posted by Rijckholt
Reading carefully on Cisco's website I concluded that the PWR600-AC-RPS can be used to get a fully redundant set-up. For this you must a so called two-to-one cable.

But now you're into two RPS units.

Well, the whole idea is to use this switch only for layer3 stuff the way I understand it. We will use the switch to route subnets to specific ports. We will only connect HP procurve 2524 switches to the 2948. All the servers will (only) be connected to the HP switches.

So you intend to attach a directed tree/graph of HP switches to the 2948G-L3, with an independent tree/graph per each layer 3 subnet, where the trees/graphs may be on the same hardware but seperate 802.1Q VLANs.

This is okay. You should remain aware that the 2948G-L3 or 4908G-L3 switches are not very flexible, and you may outgrow them in the future. Keep in mind that the tree/graph node uplinking to the 2948G-L3 in your current plan is also a single point of failure. I'm not sure you can make that go away on the 2948G platform.

Any suggestions?

I would recommend a pair of more modern, flexible layer 3 switches. This will be a bit expensive if you need GE density, e.g. Cisco 3550-12G / 3750G-12S; but you can do it pretty inexpensively if you instead use switches such as the 3550-24-EMI.

Once you've got these switches you can run HSRP and spanning-tree for redundancy in your own network, and you can make arrangements with your provider for a redundant uplink. It sounds like they are pretty flexible with you, so it shouldn't be hard to get them to run private AS eBGP.

[Rijckholt]

Hi Jeff,

Thanks again for your input! We had some more discussions here about our plans and have concluded that we want to change it a bit ;-)

I am now actively looking for two Cisco 3550-48-EMI switches. Our colo provider can outfit us with two fiber uplinks using HSRP easily. I must admit that our new network plan based on this set-up looks a lot better than the previous one. We are pretty excited about it. Thanks a lot for the trigger!

Another thing that came up during our discussions is our IP plan. Since it looks like we have to do some changes in our ip numbering I was thinking this would be a good opportunity to change our ranges. So when BGP comes within reach or when I want to change from up-link provider I am not stuck with IP ranges that cannot be moved. Any advice you can give me there. Can something like that be done without an AS number?


Thanks in advance

Onno Bakker

[jsw6]

Originally posted by Rijckholt
I am now actively looking for two Cisco 3550-48-EMI switches. Our colo provider can outfit us with two fiber uplinks using HSRP easily. I must admit that our new network plan based on this set-up looks a lot better than the previous one. We are pretty excited about it. Thanks a lot for the trigger!

You probably want to run HSRP facing your transit provider as well. This is still far less than ideal, and there are failure modes under which it will not protect you from an outage. I'd rather not go into too much detail on the board.

Another thing that came up during our discussions is our IP plan. Since it looks like we have to do some changes in our ip numbering I was thinking this would be a good opportunity to change our ranges. So when BGP comes within reach or when I want to change from up-link provider I am not stuck with IP ranges that cannot be moved. Any advice you can give me there. Can something like that be done without an AS number?

Yes, depending upon your IP space utilization. My guess is you won't meet the ARIN requirements for a single-homed organization. You'd need at least 4000 IPs utilized for this. The qualifications for multi-homed organizations, e.g. if you had an ASN and another transit provider, are much lower these days.

[appliedops]

2948G-L3's can handle a full gig+ of traffic, however they do not let you filter properly.

If you're really looking at doing any major routing, buy a big cisco router.

[sigma]

Originally posted by wotanworks
If you're really looking at doing any major routing, buy a big cisco router.

Or a Juniper, since nearly everything is wire-speed, there's one software train, and many things are simpler conceptually and more reliable in practice.

Kevin

[appliedops]

Originally posted by sigma
Or a Juniper, since nearly everything is wire-speed, there's one software train, and many things are simpler conceptually and more reliable in practice.

Indeed, however for the kind of stuff he was looking for, I'm thinking his operative goal was cheap