Hi

Does anyone know if the cisco PA-GE (1gb) will work with a Vip2 -50 in a Cisco 7507, as the cisco site says not.

Thanks Paul

Call it a hunch, But if the cisco site says no .. That would suggest to me it doesnt :)

yes that would normally be the case but I remember seeing some things on the net saying that it does but can't seem to find them anymore, I was just asking if anyone on here had tried or tested them

The backplane of a vip 2-50 would not be able to do a gig-e at line rate. You might have better luck with a vip 4-80. Even so, depending on what RSP you've got, the higher end VIPs may not be supported.

actually, if you have a good enough RSP, you will be able to use the GEIP+ line card. You wont need a VIP if I recall. I believe it is its independant.

Indeed, the GEIP+ connects directly to the cybus and does not require a VIP carrier.

Here's the knowledge that I know, considering I have a 7513 and a 7507 as border routers:

1) a GEIP is basically a PA-GE in a VIP2-50, it can handle about ~300-400Mbps worth of traffic (going on ebay for about $2k-$4k)

2) a GEIP+ is different type of PA-GE (A wide type bus to the VIP) in a VIP4-50? it can handle ~800Mbps worth of traffic and they go for more substancial amount more on ebay than a GEIP,
Think of this as a 64bit PCI Gigabit card, That's the only way to get the bandwidth on a PC, because a 32bit 33mhz PCI bus can only handle about 150mbps.

3) a PA-GE is ment to go into a 7200VXR router, it as the same cartaristics of a GEIP on a 7500.

4) Yes, PA-GE in a VIP2-50 will work, In all my vip's I use 8mb of sram and 128mb of running ram, mainly becasue I take on full bgp routes from my upstream, and those routes end up in the running ram on the VIP if you use dCEF. This configuration is not "offically supported" by cisco, but it does work.

Also to add to this, the 7513 and 7507 can only handle 2Gbps worth of traffic (which is alot), That's why cisco also says they can only handle 2 gigabit interfaces in a chassis, But currently I have 3 gigabit interfaces in my 7513 router and it's working just fine.

Yes that is what I was lead to beleive but wanted to check before I go and buy some.

what is dCEF as we have vip2-50's in all our 7507's

"ip cef distributed" is the command to enable dCEF
which his Distributed Cisco Express Forwarding,
basically the VIP cards are mini routers, when you turn on dCEF, they will fordward the packets through the backplane to the other interface, instead of passing the packets to the RSP to process.

7507/VIP2-50/PA-GE will work just fine just not at full line rate. Start looking at the 12xxx or a 65xxseries to handle that kind of traffic.

GSR is played out.

a Cisco 7500 is not going to be able to saturate gig-e at line rate, specifically with small packets (e.g. 64 bytes).

AFAIK (although I may be wrong here), the VIP's don't really have much support for GE PA. Your better option is to go with GEIP line card which then maxes out your RSP. Regardless, the 7500 is an old obsolete platform to do any kind of gigabit+ routing with serious level of traffic.

If you are looking at wirespeed capacity of GigE, recommendations as follows:

Juniper M5 or M7i
Cisco GSR (preferably with line card that has an ASIC to filter (e.g. E2))
Cisco 7300
or even a Layer3 Switch (note: layer3 switches have issues on their own. review your network needs first and match them accordingly to features supported)

Or at least a minimum of 7206vxr w/ NPE-G1 or 7301.

-J

Which one is better to do BGP for 300-500 Mbps of traffic

1. Cisco 7507, Dual RSP4, GEIP

2. Cisco 7507, Dual RSP4, PA-GE, VIP2-50

3. Cisco 7206vxr, NPE-300, PA-GE

4. Foundry BigIron B4000, B8GMR3

[5. Extreme Summit 48i (just kidding :D)]

Originally posted by FHDave
Which one is better to do BGP for 300-500 Mbps of traffic

1. Cisco 7507, Dual RSP4, GEIP


Probably not... They usually top out at 200Mbps w/ average sized packets in traffic. You can probably pull/push more than that though, but not very much.


2. Cisco 7507, Dual RSP4, PA-GE, VIP2-50


VIP2-50 is weak. RSP8 or RSP16 will probably do better I think? I am not too sure though since I never used RSP8 and up. Either way, 7500 architecture is just too old for any serious traffic at gigabit level.


3. Cisco 7206vxr, NPE-300, PA-GE


NPE-300 won't be able to cut 300-500mbps well and watch out for bandwidth points on 7206vxr's pci bus when you use PA's. Consider using NPE-400 at the minimum, or better yet NPE-G1. G1 has 3 GE's sitting right below the BCM1250 processor so you are not limited by the vxr bandwidth points, just the NPE's ability to route/forward packets (as long as you use the built-in GBICs on the NPE, instead of PA's). Cisco claims 1Mpps on the NPE-G1, reality shows otherwise, but close. I topped out at about 650Kpps or so using generally small 64 byte packets on top of regular traffic. 650Kpps is still good enough for 300-500Mbps traffic (given that most hosting networks generally generate big sized but smaller rate of packets *usually*. not always though.)


4. Foundry BigIron B4000, B8GMR3


No comment on that specific model since i don't have experience with bigiron b4000.


[5. Extreme Summit 48i (just kidding :D)]


with well behaved traffic, summit48i will probably perform better than 7500 i think.. but you know where we stand when we talk about distributed abuse traffic like increasing worms scanning the world...

-J

Originally posted by FHDave
4. Foundry BigIron B4000, B8GMR3
I'd stay away from using BigIron in a border/core role. It's good as a layer 2 switch, or in a layer 3 customer aggregation role; but you do not want to use a flow-based box like the BigIron at the edge. You have no oppertunity to protect it from DoS attacks with filtering, except by phoning your provider(s) to say your network is down and you're not sure why.

Now, aside from DoS traffic where the IronCore TCAM utterly fails, the BigIron with version 3 supervisors is a reasonably good box. If you aren't worried about DoS, it's an okay choice. Keep in mind it doesn't have the same degree of interface flexibility as the Cisco PA-based boxes you've mentioned.

Jeff,

Thank you for the input. Which of the first three routers will you recommend for a border router?

Originally posted by haesu
[B]Probably not... They usually top out at 200Mbps w/ average sized packets in traffic. You can probably pull/push more than that though, but not very much.



VIP2-50 is weak. RSP8 or RSP16 will probably do better I think? I am not too sure though since I never used RSP8 and up. Either way, 7500 architecture is just too old for any serious traffic at gigabit level.


That's quite dissapointing, considering the price difference between 7500 and 7200 :) But even with 7206 VXR, somebody told me that it won't do good for > 300 Mbps traffic.

7206 VXR with NPE-G1 is like $10-13K. At this point, the price is probably only 20-30% lower than Juniper M5. But at this moment, I don't need the capability of a Juniper M5. Surely there must be something in between that I can use for 300-500 Mbps of traffic?

Thank you for your inputs too!

Originally posted by FHDave
Thank you for the input. Which of the first three routers will you recommend for a border router?
I am not a proponent of any of the boxes you've listed. 7500 is big, old, and slow. 7200 is rapidly reaching the point where it is played out. If you have an immediate need for 300 - 500Mb/s in a border eBGP role, then consider how quickly you may outgrow it; and invest in a Juniper box.

If your immediate need is less, then buy some cheap POS box and shelf-upgrade to Juniper when the traffic need and CapEx budget are there. Think about using a PC from your dedicated server stock and add a PCI NIC or two.

My two cents.

Originally posted by FHDave
That's quite dissapointing, considering the price difference between 7500 and 7200 :) But even with 7206 VXR, somebody told me that it won't do good for > 300 Mbps traffic.

7206 VXR with NPE-G1 is like $10-13K. At this point, the price is probably only 20-30% lower than Juniper M5. But at this moment, I don't need the capability of a Juniper M5. Surely there must be something in between that I can use for 300-500 Mbps of traffic?

Thank you for your inputs too!

I dunno, I am pretty sure NPE-G1 can handle more than 300Mbps without a sweat when forwarding between built-in NPE gig-e interfaces (non-PA).

When you are looking for a router for a particular network, you must *know* your network and its traffic patterns. If you are not doing it already, I would advise that you not only monitor your network traffic in terms of bits/sec bandwidth, I woudl also monitor your pps rate as well.

When picking a router, there are two major topics to consider:
o bandwidth (i.e. how much bandwidth can it move thru its backplane/bus)
o packets/sec (router performs route lookup based on data encapsulated in packets. so even if your backplane can support 10Gbps, if you have a slow lookup algorithm, you will run into problems even before 1Gbps full of small packets).

You can also consider 7300 Series from Cisco as well. But with the similar price point, I'd personally find M5 more beneficial.

If you want to go really cheap and want to go for experience+learning mood, a powerful x86 hardware with 5.3-fbsd fastforwarding will probably come close to your requirement level.

If you are not looking for full bgp routes, and want simple router that can do gig-e upstream with FE's, a 3550 catalyst running EMI would run well for you, so long as you don't abuse its TCAM with many SVI's...

Originally posted by haesu
If you are not looking for full bgp routes, and want simple router that can do gig-e upstream with FE's, a 3550 catalyst running EMI would run well for you, so long as you don't abuse its TCAM with many SVI's...
I find the 3550 works okay with hundreds (!) of SVIs. Does your experience differ?

Originally posted by jsw6
I find the 3550 works okay with hundreds (!) of SVIs. Does your experience differ?

With straight shot SVI's, I've found no issues even when running with 70 of them on a 3550-24/EMI (even though SDM documentation on cisco.com recommends way less). However applying ACL to each and every one of them does eat TCAM space part by part. I should have been more clear when saying "SVI's" as obviously one could simply run it w/o any filtering or other tasks that take TCAM space.

-J

*nod* In most of our deployments, we have no need for ACLs on 3550s beyond QOS policers to rate-limit customer traffic for folks who buy sub-rate ports. I am priviledged enough to have Juniper upstream of nearly all my 3550s. ;)

As mentioned before, the Juniper is going to scale better for you. If you can afford the cost, it's a wise move.

In response to the BigIron question, go for the Jetcore instead of Ironcore. It has larger CAMS and can do hardware based (ASIC) rate limits and ACLS.

The larger TCAM on the JetCore modules has very little to do with why they perform better. Search my posts for "Foundry" and "FID" for a detailed walk-through of the differences between JetCore and IronCore in the real world.

While I agree that larger TCAMS don't solve everything, they do help a bunch. Also being able to do hardware ACLs is a BIG improvement. Denied traffic of a hardware ACL never hits the CPU which is a big bonus (only available in Jetcore) and there are some better CPU protection and threshold settings available in the later code, just to name a few of the big improvements.

Jetcore are way to expensive, and for that price perhaps Juniper M5 is already a viable alternative.

I would not compare Juniper with Foundry. Juniper makes great routers and Foundry makes great switches. Different platforms.

Originally posted by RackMy.com
While I agree that larger TCAMS don't solve everything, they do help a bunch.
The improvement you perceive to be a result of the 100% larger TCAM is actually a result of the improved TCAM miss behavior. The real reason Foundry increased the size of the TCAM is that they also doubled the size of some structures stored on the TCAM; and did not want customers upgrading from IronCore to JetCore to suddenly find themselves with fewer usable entries.

IronCore boxes forward the entire frame to the supervisor for process switching; while JetCore boxes copy only the first 64 bytes of the frame to the supervisor, buffer the frame on the ingress linecard, and wait for a FID update from the supervisor before forwarding the frame to the destination module(s) for egress.

Also being able to do hardware ACLs is a BIG improvement. Denied traffic of a hardware ACL never hits the CPU which is a big bonus
This is another case of increased sanity in when and how much data hits the supervisor; but keep in mind, the supervisor is still aware of many frames on a per-frame basis to implement QOS features.

there are some better CPU protection and threshold settings available in the later code, just to name a few of the big improvements.
You don't need JetCore modules to run the recent code. If Foundry would clue in to ISPs who are screaming "seperate control plane from data plane" these features would not even be necessary. Juniper did this right from the start. It's a shame they don't manufacture switches.

All this information is available in the Foundry JetCore architecture documentation. If you want to learn more about how their boxes operate "under the hood," there is plenty of information available.

"The real reason Foundry increased the size of the TCAM is that they also doubled the size of some structures stored on the TCAM; and did not want customers upgrading from IronCore to JetCore to suddenly find themselves with fewer usable entries." Can you elaborate?

Most of the CPU protection functions are only available with the Jetcore module.

Structures for many BigIron features, such as server load balancing and routing based on layer 4 header fields, have doubled in size. You can read the Foundry documentation for specifics.

The only true way to protect your Foundry is to only run controlled traffic through it. That said, I use them for customer aggregation, or core switching in a layer 2 role. Even with only layer 2 traffic, there are many caveats you must be aware of. IronCore does not handle unknown-unicast traffic gracefully (as should be obvious from my earlier post on TCAM miss behavior), for example; and gotchas like that can lead to serious problems.

Originally posted by RackMy.com
I would not compare Juniper with Foundry. Juniper makes great routers and Foundry makes great switches. Different platforms.

But I thought this is a discussion about routers?

It is a discussion about routers but someone brought up a BigIron.

Originally posted by RackMy.com
It is a discussion about routers but someone brought up a BigIron.

Okay guys, router is nothing more than a device that performs longest match lookup at layer3 level and forward the packet from one interface to another.

A Layer-3 switch is completely, and fully-reasonably, a router. It's just that in general industry trend, most low end layer3 switches lack options/flexibilities a full featured router would have. But there are also certain good higher end layer3 switches that could be used just as good as a backbone router (e.g. Cat6509 with parts that meet your needs).

Bottomline, layer-3 switch and router have no architectural difference. What differs however is each vendor/model's algorithm to perform the same task. That is why one needs to know the limits of each vendor's product and compare and contrast how it will play a role in *your* network's traffic. :)

-J

Originally posted by RackMy.com
It is a discussion about routers but someone brought up a BigIron.

Hey, I brought up the BigIron as one of the possible cheaper router solution :) But at for a router, once you talk about BigIron/Jetcore, then ... whoops, the price will be so expensive that Juniper M5 is a viable and perhaps better alternative already. Again, for a router that is.

Of course the BigIron gives additional benefits that it's also a switch and thus the price per port is much cheaper than Juniper. Let's try to get several PICs on Juniper with a total of 16 GE-SX ports, 48 FE ports, etc Whoops ....

Originally posted by FHDave
Hey, I brought up the BigIron as one of the possible cheaper router solution :) But at for a router, once you talk about BigIron/Jetcore, then ... whoops, the price will be so expensive that Juniper M5 is a viable and perhaps better alternative already. Again, for a router that is.


Of course the BigIron gives additional benefits that it's also a switch and thus the price per port is much cheaper than Juniper. Let's try to get several PICs on Juniper with a total of 16 GE-SX ports, 48 FE ports, etc Whoops ....

Which is why you know.. we have something called Teh 802.1Q VLAN trunking? :) Heheh.

Buy any cheapcake switch like extreme summit24e2 used from ebay or whatever... Trunk the gig-e to your M5 or POS pc router or cisco whatever router, and send vlan tags. :)

Indeed, if you need 16 GE-SX ports on your Juniper(s), you had better have traffic to run on them; and thus had better be able to afford the PICs anyway. Otherwise, read my post on layer 2 aggregation for transit circuits.
http://www.webhostingtalk.com/showthread.php?&postid=2045742#post2045742

If you think you need 48 ports of FastE on your Juniper(s), you need your head examined. ;)

Originally posted by jsw6
If you think you need 48 ports of FastE on your Juniper(s), you need your head examined. ;)

better have your parents get looked at too, it could be genetic ya know ;]

my 2 cents is that for a public network with a significant amount of traffic (a few hundred megs) and resiliency to abnormal traffic as a requirement, j is the only way to go. once you soup up the 7206 to a vxr to an npe-g1, you're spending approx the same amount of money you would on an m5 (and maybe m10), provided we are talking aftermarket. a corporate environment or an enduser network is an entirely different beast of course.

james,

how well does PacketOS deal with abnormal traffic, out of curiosity? ;]

paul

Originally posted by rusko
better have your parents get looked at too, it could be genetic ya know ;]

my 2 cents is that for a public network with a significant amount of traffic (a few hundred megs) and resiliency to abnormal traffic as a requirement, j is the only way to go. once you soup up the 7206 to a vxr to an npe-g1, you're spending approx the same amount of money you would on an m5 (and maybe m10), provided we are talking aftermarket. a corporate environment or an enduser network is an entirely different beast of course.

james,

how well does PacketOS deal with abnormal traffic, out of curiosity? ;]

paul

I assume by "abnormal" you mean random destination/floods/etc?

The key feature in that software is enabled when you turn on fastforwarding just like in regular FreeBSD. The exception however is that when you turn it on, it starts building a 4-level mtrie FIB and runs maintenance scheduler. All packet operations happen within the FIB, instead of regular kernel routing table (radix rib). Tests have shown that going over 650-700Kpps +/- (64byte pkts) is easy on any recent hardware w/ good NIC, or better yet with motherboard that has onboard controller configured in a way that less crosses accross the pci bus.

As far as random / abnormal destination traffic, in theory it should remain consistent or near consistent in performance, although we havent gotten to test that thoroughly as I'd like it to. As far as architecture goes, it does not use flow/traffic pattern based "fast cache" type used by older FreeBSD 4.x implementation and certain layer3 switching vendors. I suppose what would limit you primarily is how much your L2 cache can assist the CPU in differenciating searches for FIB entries..

-J

Originally posted by jsw6
If you think you need 48 ports of FastE on your Juniper(s), you need your head examined. ;)

Or cut off ...

That's why perhaps RackMy had suggested JetCore as a viable alternative to router, since you can connect your client's servers directly to the port and the cost to do so will be below the cost of doing it on Juniper (Juniper as router and additional switches, since as you said only crackhead will try to connect servers directly to Juniper ;)).

Juniper M5 with either one port GE-SX and 2-4FE TX ports _or_ two ports GE-SX runs about $19-$22K (ebay price). 7206vxr with NPE-G1, PA-GE, PA-4FE runs about $13-$14K (again, ebay price). Should definitely go with Juniper.

Juniper is cheaper than you think. I probably buy Juniper gear more often than 99% of WHT posters, though.

Can I buy Juniper gears through you Jeff?

Originally posted by FHDave
Juniper M5 with either one port GE-SX and 2-4FE TX ports _or_ two ports GE-SX runs about $19-$22K (ebay price). 7206vxr with NPE-G1, PA-GE, PA-4FE runs about $13-$14K (again, ebay price). Should definitely go with Juniper.

well, there's room to play with here. you could, for example, use oc12 interfaces instead of gig-e, since they're considerably cheaper.

<disclaimer> i'm not a proper netgeek </disclaimer>

p

Originally posted by rusko
well, there's room to play with here. you could, for example, use oc12 interfaces instead of gig-e, since they're considerably cheaper.
Sssshhh! Don't reveal our secrets for making real hardware cost effective! ;)

Originally posted by haesu
I assume by "abnormal" you mean random destination/floods/etc?


yes. even wimpy hardware can pass a lot if the packets are nice and fat, the major argument against the ciscos mentioned in this thread is the pretty flames you would witness if ddos/worm traffic happened across them.


The key feature in that software is enabled when you turn on fastforwarding just like in regular FreeBSD. The exception however is that when you turn it on, it starts building a 4-level mtrie FIB and runs maintenance scheduler. All packet operations happen within the FIB, instead of regular kernel routing table (radix rib). Tests have shown that going over 650-700Kpps +/- (64byte pkts) is easy on any recent hardware w/ good NIC, or better yet with motherboard that has onboard controller configured in a way that less crosses accross the pci bus.


i'm assuming this requires interrupt coalescing to be supported, yes?


As far as random / abnormal destination traffic, in theory it should remain consistent or near consistent in performance, although we havent gotten to test that thoroughly as I'd like it to. As far as architecture goes, it does not use flow/traffic pattern based "fast cache" type used by older FreeBSD 4.x implementation and certain layer3 switching vendors. I suppose what would limit you primarily is how much your L2 cache can assist the CPU in differenciating searches for FIB entries..


if you're prefix-based, the flows don't figure in the complexity, so we should be fairly certain that performance is independent of them. am i missing something?

p

Originally posted by FHDave
Can I buy Juniper gears through you Jeff?

hey, you two, get a room ;]

p

Originally posted by jsw6
Sssshhh! Don't reveal our secrets for making real hardware cost effective! ;)

eh, sorry ;] the only problem i see with this is the oc12 cards for bigirons (i was thinking m5->b4k via oc12) seem to be quite rare.

paul

Originally posted by rusko
yes. even wimpy hardware can pass a lot if the packets are nice and fat, the major argument against the ciscos mentioned in this thread is the pretty flames you would witness if ddos/worm traffic happened across them.


*nod*


i'm assuming this requires interrupt coalescing to be supported, yes?


(I previously said *nod* before reading your question clearly)
We use device polling to deal w/ hardware interrupts, and run all forwarding operations w/o switching context after upper layer ethernet driver stage. (e.g. when entering Layer 2 -> layer 3). With device polling, coalescing was of course not operating. So it is not technically needed.


if you're prefix-based, the flows don't figure in the complexity, so we should be fairly certain that performance is independent of them. am i missing something?

p

It is prefix based (aka mtrie), and thus far existing tests show performance remains constant. Also has a receive-acl.

-J

btw, since this is fbsd-based, i'm assuming you're not doing smp/smp doesn't help much?

p

Originally posted by rusko
btw, since this is fbsd-based, i'm assuming you're not doing smp/smp doesn't help much?

p

Nope not at this time. I don't think SMP helps a lot in routing for the time being, as most of the CPU work is done for tending to queue and looking up the prefix (and other stuff like checking through compiled firewall). The rest is stressing out your system bus.

-J

Thank you all ... I have finally (I think ;)) decided on Juniper m7i. It seems to be best for our needs and expected growth in the next 12-24 months.

Originally posted by FHDave
Thank you all ... I have finally (I think ;)) decided on Juniper m7i. It seems to be best for our needs and expected growth in the next 12-24 months.

good luck finding that on the aftermarket, dave ;] it's kind of like trying to buy a 2005 mustang gt used right now (for non-mustang fans, it's a brand new design, not being sold yet and with the first shipment completely reserved).

paul

hm, I did not mention used, did I? ;)

I did talk to Juniper sales rep, and m7i has a lead time of 4-6 weeks. It's amazing that Juniper does not even have routers on stock. They build their routers on needed basis (i.e. when there is a sale).

Right now, the M7is are much closer to 6 than 4 weeks.

Have fun with the new kit if you do go ahead!

Out of curiosity, how much was it?

Just give your local Juniper partner a call, they can give you pricing details at the drop of a hat :)

Originally posted by FHDave
hm, I did not mention used, did I? ;)


you did, when you mentioned numbers earlier in this thread. ;]

paul

Just give your local Juniper partner a call, they can give you pricing details at the drop of a hatI did last year, no return call. I always find it interesting on how so many hardware companies have such horrible sales departments.

Juniper doesn't sell direct to end-users, so they have no need for a user-facing sales department. If the partner you contacted was no good, just try another. Even in the UK there are some that are good and some that aren't.

Hi,

You posted this awhile back, but i'm curious if you ever went ahead with using pa-ge's with your vip2-50. How did it go?

Thanks

Juniper doesn't sell direct to end-users, so they have no need for a user-facing sales department. Sure they do, I just talked to one last week and am putting an order together :)

Glad to see you are on the Juniper wagon. Best of luck, as if you'll need it ;)

Thanks, pretty cool stuff!

Mike,

Sure they do, I just talked to one last week and am putting an order together

Oh right, well, have fun with it :) They may do things slightly differently here in the UK, but we were referred to one of their partners when we started asking about pricing etc.