Results 1 to 14 of 14
  1. #1

    Dell 6248 2.x firmware serious issue -- upgrade or be warned

    Hi Everyone,

    As you know, I'm a fan of the Dell 6248 powerconnect switch because you get 48 gig ports, up to 4 ten gig ports, and routing features for a really cheap price.

    That said, I've seen a few odd issues here and there, many of which seem to go away with the new 3.x firmware. We had been putting off upgrading that firmware on our main switch as we didn't want to risk downtime for customers, but the time came last night to upgrade this during emergency maintenance due to a rather serious issue.

    Specifically, in the 2.x firmware, you cannot disable ICMP destination unreachables and ICMP echo replies from being sent by the switch. Therefore, if you have an ip that should be directly reachable by your switch (it's an ip in an attached subnet), but it is not reachable, and someone pings that ip, the switch will reply back "destination unreachable".

    No big deal right? Wrong! If you get more than a few of these, it will bog down the switch cpu, eventually causing network issues that start at making the switch web and telnet interfaces inaccessible, and, as the situation gets worse, the entire network may go offline. The end result is that something as simple as shutting down a server on your network, or someone sending 1mbps of pings to your network can take it offline.

    How to solve this? Upgrade to a 3.x version firmware. Then under router -> ip -> interface configuration, click on the relevant vlan, and disable icmp unreachables. It also wouldn't hurt to tell the switch not to reply to icmp echo as well, another option in the 3.x firmware that's not in the 2.x

    Hope this helps some people avoid some serious network issues like I had seen.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  2. #2
    Join Date
    Apr 2009
    Posts
    6,565
    It seems like, from your description, this happened to you haha! I hope you got your stuff sorted out.

  3. #3
    Quote Originally Posted by Zach Nelson View Post
    It seems like, from your description, this happened to you haha! I hope you got your stuff sorted out.
    Yeah, it's been sorted out now

    The switch also has issues if you don't configure it right, that the switch cpu spends all of it's time messing around with arp tables, but that's solvable just by setting up your routing correctly. Apparently it has a well documented issue with spanning tree as well, where topology changes cause, you guessed it, the switch cpu to get overloaded.

    The 2.x firmware also has a nasty habit of sometimes silently messing up some ethernet ports, where the only solution is to hard power off (at the pdu) the attached server, or, presumably, remove the ethernet cable by hand of the affected port. The affected port will seem to work fine for the most part, but silently discard arp traffic, causing the ips on the connected port to be unable to reply to arp requests, because they don't receive them. The connected server can work around this issue by sending unsolicited arp replies, but that's obviously hardly a solution.

    Overall, if you're using the dell 6248 switch with the 2.x firmware and are using any of the layer 3 features, you need to get your head examined.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  4. #4
    Join Date
    Nov 2009
    Location
    Cincinnati
    Posts
    1,585
    Didn't I tell you to upgrade a while ago? Pretty sure I did. Also enable flowcontrol on your switches if you have not. By default its off.
    'Ripcord'ing is the only way!

  5. #5
    Quote Originally Posted by Visbits View Post
    Didn't I tell you to upgrade a while ago? Pretty sure I did. Also enable flowcontrol on your switches if you have not. By default its off.
    Already turned on flowcontrol a while ago. It's on by default in 3.x for what it's worth. And yes, I knew I had wanted to upgrade to 3.x for a while, but without any serious problem occuring, I wanted to make sure I had all my ducks in a row before upgrading. Unfortuantely, the need to upgrade came before I had everything tidy and ready. The ease of upgrading the switch was better than I thought (10 minutes of downtime, no configuration changes necessary afterwards, no weird suprises), and the consequences of not upgrading were higher than I had thought (you can get DoS'ed by simply having a high traffic server on your network crash)
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  6. #6
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,131
    Quote Originally Posted by funkywizard View Post
    Already turned on flowcontrol a while ago. It's on by default in 3.x for what it's worth. And yes, I knew I had wanted to upgrade to 3.x for a while, but without any serious problem occuring, I wanted to make sure I had all my ducks in a row before upgrading. Unfortuantely, the need to upgrade came before I had everything tidy and ready. The ease of upgrading the switch was better than I thought (10 minutes of downtime, no configuration changes necessary afterwards, no weird suprises), and the consequences of not upgrading were higher than I had thought (you can get DoS'ed by simply having a high traffic server on your network crash)
    Must be a L3 only issue, we have 30 of these things deployed and never had one crash in L2 mode when a ddos hits.
    Yellow Fiber Networks
    http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
    Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net

  7. #7
    Quote Originally Posted by Spudstr View Post
    Must be a L3 only issue, we have 30 of these things deployed and never had one crash in L2 mode when a ddos hits.
    Yeah, the only problem I saw that didn't look like a strictly L3 issue was where a particular switchport would inexplicably refuse to send arp requests to the attached server, requiring you to hard power cycle the affected server. A TCP dump confirmed that the arp requests were being filtered by the switch for reasons completely unknown to me. All of the other issues manifest themselves in such a way that you wouldn't see any problem at all if you don't use any of the routing features of the switch.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  8. #8
    Join Date
    Nov 2009
    Location
    Cincinnati
    Posts
    1,585
    Your flowcontrol is on by default in 3.0???? I had to enable it on all our switches. I guess you had enabled it while running 2.x?

    Nice, yeah the 6248s have been great for us. You can't beat it $ for performance.
    'Ripcord'ing is the only way!

  9. #9
    Quote Originally Posted by Visbits View Post
    Your flowcontrol is on by default in 3.0???? I had to enable it on all our switches. I guess you had enabled it while running 2.x?

    Nice, yeah the 6248s have been great for us. You can't beat it $ for performance.
    Running 3.2.1.3. All the documentation I read had said that flowcontrol on was the default after upgrading to this firmware, even if you had it turned off previously. But yes, we already had flowcontrol on before we upgraded the firmware, so I can't confirm that the documentation I read was accurate on this point.

    edit: As to $ / performance, I couldn't agree more. It's pretty easy to find these on ebay with 4x10gbit XFP modules for $1000 total price, and then finding SR XFP optics for around $100 each is dead easy too. All told we can get a switch with 48xgig and 4x10gbit uplinks ready to rock for under $1500
    Last edited by funkywizard; 05-30-2011 at 01:57 PM.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  10. #10
    Join Date
    Nov 2009
    Location
    Cincinnati
    Posts
    1,585
    Yep, a comparable cisco is $2500 minimum without SmartNet.

    The stacking ports work great on these to, we have a 2 switch stack and its never had an issue.
    'Ripcord'ing is the only way!

  11. #11
    Join Date
    Aug 2007
    Location
    L.A., CA
    Posts
    3,710
    It doesnt seem there are any Dell 6248's with 10G on eBay right now, thats the problem with buying the used stuff is lack of reliable supply (or even predictable supply)
    EasyDCIM.com - DataCenter Infrastructure Management - HELLO DEDICATED SERVER & COLO PROVIDERS! - Reach Me: chris@easydcim.com
    Bandwidth Billing | Inventory & Asset Management | Server Control
    Order Forms | Reboots | IPMI Control | IP Management | Reverse&Forward DNS | Rack Management

  12. #12
    Quote Originally Posted by CGotzmann View Post
    It doesnt seem there are any Dell 6248's with 10G on eBay right now, thats the problem with buying the used stuff is lack of reliable supply (or even predictable supply)
    Yeah true. I haven't tried to buy any in a while, but there always were plenty when I was looking. But yeah, you need to plan ahead if you want to go this route.
    IOFLOOD.com -- We Love Servers
    Phoenix, AZ Dedicated Servers in under an hour
    ★ Ryzen 9: 7950x3D ★ Dual E5-2680v4 Xeon ★
    Contact Us: sales@ioflood.com

  13. #13
    Join Date
    Oct 2004
    Location
    Houston, Tx
    Posts
    307
    Which Cisco model offers 48x gig ports and 4x 10gig ports for $2,500?


    Can the Dell 6248 handle a few static routes?

  14. #14
    Join Date
    Nov 2009
    Location
    Cincinnati
    Posts
    1,585
    Shop around for a used Cisco 3560E-48TD
    'Ripcord'ing is the only way!

Similar Threads

  1. dell 6248 compatible 1g optics?
    By funkywizard in forum Colocation, Data Centers, IP Space and Networks
    Replies: 12
    Last Post: 05-05-2011, 08:44 AM
  2. Anyone else having problems losing management interface to Dell 6248
    By WebGuyz in forum Colocation, Data Centers, IP Space and Networks
    Replies: 1
    Last Post: 04-18-2011, 03:54 PM
  3. layer 3 switch static routing PPS performance? (dell 6248 in particular)
    By funkywizard in forum Colocation, Data Centers, IP Space and Networks
    Replies: 15
    Last Post: 03-02-2011, 01:23 AM
  4. dell powerconnect 6248 sflow firmware?
    By funkywizard in forum Colocation, Data Centers, IP Space and Networks
    Replies: 1
    Last Post: 02-15-2010, 10:25 AM
  5. Dell 6248 Switch
    By DaleF in forum Colocation, Data Centers, IP Space and Networks
    Replies: 1
    Last Post: 10-04-2008, 05:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •