Results 1 to 14 of 14
-
05-29-2011, 03:46 PM #1
Dell 6248 2.x firmware serious issue -- upgrade or be warned
Hi Everyone,
As you know, I'm a fan of the Dell 6248 powerconnect switch because you get 48 gig ports, up to 4 ten gig ports, and routing features for a really cheap price.
That said, I've seen a few odd issues here and there, many of which seem to go away with the new 3.x firmware. We had been putting off upgrading that firmware on our main switch as we didn't want to risk downtime for customers, but the time came last night to upgrade this during emergency maintenance due to a rather serious issue.
Specifically, in the 2.x firmware, you cannot disable ICMP destination unreachables and ICMP echo replies from being sent by the switch. Therefore, if you have an ip that should be directly reachable by your switch (it's an ip in an attached subnet), but it is not reachable, and someone pings that ip, the switch will reply back "destination unreachable".
No big deal right? Wrong! If you get more than a few of these, it will bog down the switch cpu, eventually causing network issues that start at making the switch web and telnet interfaces inaccessible, and, as the situation gets worse, the entire network may go offline. The end result is that something as simple as shutting down a server on your network, or someone sending 1mbps of pings to your network can take it offline.
How to solve this? Upgrade to a 3.x version firmware. Then under router -> ip -> interface configuration, click on the relevant vlan, and disable icmp unreachables. It also wouldn't hurt to tell the switch not to reply to icmp echo as well, another option in the 3.x firmware that's not in the 2.x
Hope this helps some people avoid some serious network issues like I had seen.IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-29-2011, 11:32 PM #2Web Hosting Master
- Join Date
- Apr 2009
- Posts
- 6,565
It seems like, from your description, this happened to you haha! I hope you got your stuff sorted out.
-
05-29-2011, 11:40 PM #3
Yeah, it's been sorted out now
The switch also has issues if you don't configure it right, that the switch cpu spends all of it's time messing around with arp tables, but that's solvable just by setting up your routing correctly. Apparently it has a well documented issue with spanning tree as well, where topology changes cause, you guessed it, the switch cpu to get overloaded.
The 2.x firmware also has a nasty habit of sometimes silently messing up some ethernet ports, where the only solution is to hard power off (at the pdu) the attached server, or, presumably, remove the ethernet cable by hand of the affected port. The affected port will seem to work fine for the most part, but silently discard arp traffic, causing the ips on the connected port to be unable to reply to arp requests, because they don't receive them. The connected server can work around this issue by sending unsolicited arp replies, but that's obviously hardly a solution.
Overall, if you're using the dell 6248 switch with the 2.x firmware and are using any of the layer 3 features, you need to get your head examined.IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-30-2011, 12:39 PM #4Web Hosting Master
- Join Date
- Nov 2009
- Location
- Cincinnati
- Posts
- 1,585
Didn't I tell you to upgrade a while ago? Pretty sure I did. Also enable flowcontrol on your switches if you have not. By default its off.
'Ripcord'ing is the only way!
-
05-30-2011, 01:18 PM #5
Already turned on flowcontrol a while ago. It's on by default in 3.x for what it's worth. And yes, I knew I had wanted to upgrade to 3.x for a while, but without any serious problem occuring, I wanted to make sure I had all my ducks in a row before upgrading. Unfortuantely, the need to upgrade came before I had everything tidy and ready. The ease of upgrading the switch was better than I thought (10 minutes of downtime, no configuration changes necessary afterwards, no weird suprises), and the consequences of not upgrading were higher than I had thought (you can get DoS'ed by simply having a high traffic server on your network crash)
IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-30-2011, 01:25 PM #6Master of the Truth
- Join Date
- Mar 2006
- Location
- Reston, VA
- Posts
- 3,131
Yellow Fiber Networks
http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net
-
05-30-2011, 01:40 PM #7
Yeah, the only problem I saw that didn't look like a strictly L3 issue was where a particular switchport would inexplicably refuse to send arp requests to the attached server, requiring you to hard power cycle the affected server. A TCP dump confirmed that the arp requests were being filtered by the switch for reasons completely unknown to me. All of the other issues manifest themselves in such a way that you wouldn't see any problem at all if you don't use any of the routing features of the switch.
IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-30-2011, 01:42 PM #8Web Hosting Master
- Join Date
- Nov 2009
- Location
- Cincinnati
- Posts
- 1,585
Your flowcontrol is on by default in 3.0???? I had to enable it on all our switches. I guess you had enabled it while running 2.x?
Nice, yeah the 6248s have been great for us. You can't beat it $ for performance.'Ripcord'ing is the only way!
-
05-30-2011, 01:44 PM #9
Running 3.2.1.3. All the documentation I read had said that flowcontrol on was the default after upgrading to this firmware, even if you had it turned off previously. But yes, we already had flowcontrol on before we upgraded the firmware, so I can't confirm that the documentation I read was accurate on this point.
edit: As to $ / performance, I couldn't agree more. It's pretty easy to find these on ebay with 4x10gbit XFP modules for $1000 total price, and then finding SR XFP optics for around $100 each is dead easy too. All told we can get a switch with 48xgig and 4x10gbit uplinks ready to rock for under $1500Last edited by funkywizard; 05-30-2011 at 01:57 PM.
IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-30-2011, 03:31 PM #10Web Hosting Master
- Join Date
- Nov 2009
- Location
- Cincinnati
- Posts
- 1,585
Yep, a comparable cisco is $2500 minimum without SmartNet.
The stacking ports work great on these to, we have a 2 switch stack and its never had an issue.'Ripcord'ing is the only way!
-
05-30-2011, 06:15 PM #11Web Hosting Master
- Join Date
- Aug 2007
- Location
- L.A., CA
- Posts
- 3,710
It doesnt seem there are any Dell 6248's with 10G on eBay right now, thats the problem with buying the used stuff is lack of reliable supply (or even predictable supply)
EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
Bandwidth Billing | Inventory & Asset Management | Server Control
Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management
-
05-30-2011, 06:26 PM #12IOFLOOD.com -- We Love Servers
Phoenix, AZ Dedicated Servers in under an hour
★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
Contact Us: sales@ioflood.com ★
-
05-31-2011, 07:59 AM #13Web Hosting Guru
- Join Date
- Oct 2004
- Location
- Houston, Tx
- Posts
- 307
Which Cisco model offers 48x gig ports and 4x 10gig ports for $2,500?
Can the Dell 6248 handle a few static routes?
-
05-31-2011, 08:06 AM #14Web Hosting Master
- Join Date
- Nov 2009
- Location
- Cincinnati
- Posts
- 1,585
Shop around for a used Cisco 3560E-48TD
'Ripcord'ing is the only way!
Similar Threads
-
dell 6248 compatible 1g optics?
By funkywizard in forum Colocation, Data Centers, IP Space and NetworksReplies: 12Last Post: 05-05-2011, 08:44 AM -
Anyone else having problems losing management interface to Dell 6248
By WebGuyz in forum Colocation, Data Centers, IP Space and NetworksReplies: 1Last Post: 04-18-2011, 03:54 PM -
layer 3 switch static routing PPS performance? (dell 6248 in particular)
By funkywizard in forum Colocation, Data Centers, IP Space and NetworksReplies: 15Last Post: 03-02-2011, 01:23 AM -
dell powerconnect 6248 sflow firmware?
By funkywizard in forum Colocation, Data Centers, IP Space and NetworksReplies: 1Last Post: 02-15-2010, 10:25 AM -
Dell 6248 Switch
By DaleF in forum Colocation, Data Centers, IP Space and NetworksReplies: 1Last Post: 10-04-2008, 05:09 PM