Page 1 of 3 123 LastLast
Results 1 to 25 of 61
  1. #1

    NULL Routing 80Mbps of VoIP Traffic

    I have a server <through a reseller of> colocrossing in Chicago and they have null routed by box after my box pushed 80Mbps of UDP VoIP traffic. Does anyone have a contact there that can get this resolved? Support seems unable to escalate this to a senior level network engineer after having a ticket open since 4pm PST today (it's now 9pm). It seems strange that a provider that prides itself on 100% update does not have a network engineer that can undo a null route past 5pm EDT.

    Is it right me of to ask for an SLA credit?
    Last edited by Justin; 03-25-2011 at 10:42 PM. Reason: Factual cleanup
      0 Not allowed!

  2. #2
    Join Date
    Mar 2005
    Posts
    246
    Hello,

    Please supply me with your customer ID and a ticket number, I do not seem to be able to locate any of this based upon what you've posted here.
    ColoCrossing - Connecting Business
    Alex Vial | avial@colocrossing.com | 1.800.518.9716

    Enterprise-Class Colo & Dedicated Servers in BUF, CHI, DFW, NYC, SJC, ATL & SEA
      0 Not allowed!

  3. #3
    Join Date
    Jun 2006
    Location
    Calgary, Alberta
    Posts
    688
    Quote Originally Posted by avizzle View Post
    Hello,

    Please supply me with your customer ID and a ticket number, I do not seem to be able to locate any of this based upon what you've posted here.

    http://www.webhostingtalk.com/showpo...8&postcount=29

    I think one of the other employee's at ColoCrossing took care of it avizzle.

    (I'm not related to either, just finished reading the other ColoCrossing thread).
      0 Not allowed!

  4. #4
    13417 was the colocrossing ticket id--hopefully only so you guys can learn from the problems I have encountered with colocrossing.

    --matt
      0 Not allowed!

  5. #5
    Join Date
    Mar 2005
    Posts
    246
    You are not a customer of ColoCrossing.
    ColoCrossing - Connecting Business
    Alex Vial | avial@colocrossing.com | 1.800.518.9716

    Enterprise-Class Colo & Dedicated Servers in BUF, CHI, DFW, NYC, SJC, ATL & SEA
      0 Not allowed!

  6. #6
    If I am not a customer of ColoCrossing, then why is it that ColoCrossing was NULL routing my system?

    You are correct tho, I am not a direct customer of colocrossing, but that does not mean ColoCrossing is not responsible for incorrectly diagnosing my 80mbps of VoIP traffic as a DoS attack.

    It's far more honorable to admit ColoCrossing messed up, then to pass the buck. Good providers fess up
      0 Not allowed!

  7. #7
    Join Date
    Jun 2002
    Location
    PA, USA
    Posts
    5,143
    You do not have a server with ColoCrossing. And they are not under any obligation to communicate with you if you are not their direct customer. I think that's what he mean. Contact your provider and have them contact ColoCrossing.

    Good luck.
    Fluid Hosting, LLC - Enterprise Cloud Infrastructure: Cloud Shared and Reseller, Cloud VPS, and Cloud Hybrid Server
      0 Not allowed!

  8. #8
    For the record, I did go through the proper channels of escalating a ticket thru my provider (who is a colocrossing reseller).

    It took my own provider about 5 minutes to put in a ticket with colocrossing and my direct provider has been extremely transparent in updating me on it's status.

    From the ordeal, it was very clear that colocrossing has no problem placing an immediate 24 hour hold on my system, because they automatically assumed 60k pps of udp voip traffic was a DoS attack. Their techs where also unable to authorize it to be removed, leaving 5+ hours of downtime due to the null route.

    My take on this, is that colocrossing will not stand by their 100% SLA, their 24/7 techs are not capable of making network changes, their network is prone to DoS attacks (because they mistook my normal VoIP activity for a DoS attack), they have routers that are running at capacity that could not spare the additional 60,000pps, and they are not adapt at handling large scale VoIP traffic.

    I believe a full SLA credit to my reseller is in order, but instead they are trying to pass the buck. Would you want to deal with a company like this? I signed up for my server with my reseller because I knew they used colocrossing (they advertise it on their www site), and I thought I would give them a try, it turned out to be a bad deal for me, and just hope others learn from my mistake. YMMV
      0 Not allowed!

  9. #9
    Join Date
    Mar 2005
    Posts
    246
    Given that you are not a customer of ours, you do not understand our SLA. If you were, you would know that this situation does not apply in the least.

    I think it's very clear at this point that your scraping the barrel in an attempt to find some reason to slander ColoCrossing, perhaps for your own personal satisfaction. It's over, time to move on.

    Good luck finding a service provider to fit your needs.
    ColoCrossing - Connecting Business
    Alex Vial | avial@colocrossing.com | 1.800.518.9716

    Enterprise-Class Colo & Dedicated Servers in BUF, CHI, DFW, NYC, SJC, ATL & SEA
      0 Not allowed!

  10. #10
    Join Date
    Mar 2008
    Location
    Los Angeles, CA
    Posts
    555
    I know I would be pretty pissed if the colo company null-routed my IP just because it 'looked' like a DoS. This might be understandable if say it was all going to one IP but if its going to a bunch of IPs then its unlikely its a DDoS. Seems like more investigation of this could have been done...

    I will admit 60k pps is quite a lot. I do bonding of my home connection via a colo'd box + VPN. Its about 15k PPS and it would be 75-80 megabits all going to one IP address. Makes me wonder if that type of traffic would get me null-routed as well?
      0 Not allowed!

  11. #11
    Join Date
    Mar 2009
    Location
    Here Today - Gone to Maui
    Posts
    9,965
    Quote Originally Posted by MattFS218 View Post
    I have a server with colocrossing in Chicago and they have null routed by box after my box pushed 80Mbps of UDP VoIP traffic. Does anyone have a contact there that can get this resolved? Support seems unable to escalate this to a senior level network engineer after having a ticket open since 4pm PST today (it's now 9pm). It seems strange that a provider that prides itself on 100% update does not have a network engineer that can undo a null route past 5pm EDT.

    Is it right me of to ask for an SLA credit?
    You can certainly ask for one from your provider, but not from ColoCrossing.
    ProlimeHost - Dedicated Server Hosting & KVM SSD VPS
    Three Datacenter Locations: Los Angeles, Denver & Singapore
    SuperMicro Hardware | Multiple Bandwidth Providers | 24/7 On-site Engineers
      0 Not allowed!

  12. #12
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,525
    Many managed IP and colo providers will automatically null route IP's under attack, which usually is the right thing to do. In this case, it may be necessary to ask your provider to work with you to set more appropriate criteria for null routing.
      0 Not allowed!

  13. #13
    With SIP, g711 ulaw with 20ms udp payload intervals it's 50pps per voip channel. 1000 simultaneous g711 calls is 77mbps full duplex and is 50k pps. Unfortunately the only way to send fewer packets is to increase the interval of udp packets but that creates choppy voices and isn't supported by most SIP carriers.

    It's been my experience that many providers have automated systems, some that simply notify you and give you of an abnormal pps for your own safety and information. I think anyone with a sever would appreciate this warning...the problem with Colo Crossing is that there was no warning, and when they where notified the traffic pattern was typical of the box they where not able to lift the null route for 5+ hours. They could have also have noticed that by turning off outbound traffic to my box for 1 minute that the inbound UDP traffic would have immediately stopped and from that derived that it wasn't a DoS attack.

    I believe the best providers have in-dept tools in analyzing the cause of a DoS attack, but here's what I think colocrossing engineers did, which is an honest mistake, but could easily be avoided in the future (which is the purpose of me creating this thread--besides to get my box online).

    1) network engineers notice high load on a router or other reason to believe a DoS attack is occurring
    2) they check router logs for likely offending traffic to see which ports are acting abnormal (udp and high pps is reasonable to assume is the cause)
    3) if they find one, they null route the port/ip

    the assumption made is that the abnormal traffic pattern is the cause, of the attack and nothing is done to confirm the the abnormal patter is the real cause (except an irate customer calling), when in fact it could be many different factors including a lower pps large packet size attack on an adjacent port/server.

    Comments?
      0 Not allowed!

  14. #14
    Join Date
    Apr 2007
    Posts
    3,531
    Comments...

    If you have abnormal requirements (as you do) contact the network provider first to ensure they are happy to host you.

    Don't assume that they are going to allow the traffic, or that they would warn you before a Null route.

    Finally I think most of the blame here is actually on the client, I don't see the providers breaking their terms or SLA...
    BotWars.io - Code the AI of your Battle Bot!
      0 Not allowed!

  15. #15
    Join Date
    Mar 2009
    Posts
    568
    I have to side with ColoCrossing on the aspect that your original post is misleading potential clients to believe that ColoCrossing did this to you as a direct client, when that's not the case. Your traffic was blocked by them, probably by an automated system. You do not qualify for any relief from them. You should connect directly with the reseller to arrange your IPs to be unblocked and SLA credit applied.

    In the situation of why it took so long for ColoCrossing to reply to your vendor's request to have the connection unblocked, that's not your concern but your reseller's. The only person qualified to come in these forums and complain about that is the reseller themselves.

    --Chris
    The Object Zone - Your Windows Server Specialists for more than twenty years - http://www.object-zone.net/
    Services: Contract Server Management, Desktop Support Services, IT/VoIP Consulting, Cloud Migration, and Custom ASP.net and Mobile Application Development
      0 Not allowed!

  16. #16
    Join Date
    Aug 2001
    Posts
    4,028
    I just wanted to chime in and state we have 10+ servers through ColoCrossing spread throughout North America. Our clients are DEMANDING and they're happy becuase the ColoCrossing network is rock solid.

    I fully understand the OP is upset but as mentioned, the title is very misleading. ColoCrossing is a fantastic provider.

    Where is your reseller? What's their take on this?
      0 Not allowed!

  17. #17
    If my box does not have internet connectivity for 5+ hours, then I believe that is what the SLA is for. I don't blame colocrossing for null routing since it's the first time it has happened. I blame colo crossing for keeping the null route for 5+ hours after receiving justification for the high pps or not having a network engineer 24/7 that can remove a null route.

    The majority of hosting companies I work have routers that can handle the high pps (hence my comments about colocrossing having routers that are running over capacity) without breaking a sweat. I believe the high pps is more of a perceived issue rather than an actual issue.

    Unfortunately for most sales people it's very difficult to explain what high pps is, but I'll give it a shot next time I'm in the market for a new colo provider. Maybe tho, instead of marketing a 100mbps port they should market a 100mbps port with only a max of 20k pps, or at least bury it deep inside a terms of service agreement.
      0 Not allowed!

  18. #18
    The reseller is Vee from DediDirect. The delay in removing the NULL route is fully on ColoCrossing's side.

    It's ColoCrossings network, so I really don't see how it makes a difference if I am direct or not. I know I won't get an SLA credit, since I am not a direct customer, but my reseller should be entitled to one.
    Last edited by MattFS218; 03-25-2011 at 03:19 PM.
      0 Not allowed!

  19. #19
    Join Date
    Jan 2011
    Location
    Ohio
    Posts
    467
    I don't understand why ColoCrossing can't simply say.. We will credit our reseller, however it's up to our reseller to credit you. If the company you purchased from has already credited you then its none of your business whether they credit their reseller, or not. It's up to them to fight for their SLA. If the company you purchased from is saying they wont credit you, unless they receive credit from ColoCrossing then I honestly would say they shouldnt have said it, and they should make their own business decisions on deciding whethey they want to credit you because they want to keep you around, or your a great customer, or what have you.
      0 Not allowed!

  20. #20
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    Curious to know:

    1. Did you have similar traffic in the past?
    2. Did colocrossing notify your reseller of the null route?
    3. Did your reseller notify you of the null route?

    If you've had similar traffic in the past, they should have been able to obverse that your current traffic was within similar norms as historical traffic and made a better decision.

    Also, 80Mbps is hardly a lot of traffic. I don't know what their DDoS null routing policies are, but we very much take a wait and see approach to anything under 100Mbps for customers on 100Mbps ports and anything under 500Mbps for customers on gigabit ports. If there is any question about the nature of the traffic, we can then capture some packets to see if the traffic appears to be legitimate or not.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.
      0 Not allowed!

  21. #21
    Join Date
    Aug 2002
    Location
    Seattle
    Posts
    5,525
    Peer1, in my experience (just to cite an example), will set a null route at 60,000 PPS and then call the customer's emergency contact. I presume many managed providers do the same unless the customer establishes other expectations.
      0 Not allowed!

  22. #22
    the high pps and traffic pattern has been this way since early February. It probably increased about 10% tho this last week, but the low bandwidth high pps pattern has been for over a month. I think what happened was a "perfect storm", of a real DoS or other abnormality on the network caused colocrossing to look deeper at the router logs and come to the (incorrect) conclusion my traffic was the cause.

    I will check with my reseller if they where contacted, but I do know that when I initially submitted a ticket with my reseller they had no idea about the null route, so my presumption is that they where not told.
      0 Not allowed!

  23. #23
    We immediately contacted your supplier and began communicating with them. I do not understand why the poster (the non-customer) is suggesting that we weren't available to his supplier, because the ticket (which we opened) clearly shows quick and effective communication.

    We have a privacy policy in place which does not allow us to talk about specifics regarding your supplier. Any action taken by the engineering staff at ColoCrossing is done with the goal of protecting the network for everyone. We would, of course, recommend that you talk with your supplier about what policies can be put in place to make sure you don't have any similar issues in the future.
    ColoCrossing - Dedicated to Uptime
    Jon Biloh | jbiloh@colocrossing.com | 1.800.518.9716 | Skype: jbiloh
    DDOS Protected Colo & Dedicated Servers in Buffalo, Chicago, Dallas and Los Angeles
      0 Not allowed!

  24. #24
    I should also note that a packet capture would have showed the udp packets coming only from a handful of ip addresses, all of which map back to names/ip allocations of known voip providers.
      0 Not allowed!

  25. #25
    Join Date
    Feb 2002
    Location
    New York, NY
    Posts
    4,618
    If you're going to keep this discussion going, I think a mod should remove ColoCrossing's name from the title. Regardless of whether they had any fault, it's really not fair for a customer of a customer (i.e. a non-customer) to be dragging their name through the mud in a public forum. Privacy policies and general etiquette prevent them from really being able to properly defend themselves.
    Last edited by bqinternet; 03-25-2011 at 04:40 PM.
    Scott Burns, President
    BQ Internet Corporation
    Remote Rsync and FTP backup solutions
    *** http://www.bqbackup.com/ ***
      0 Not allowed!

Page 1 of 3 123 LastLast

Similar Threads

  1. high traffic = null route ?
    By coolnikin in forum Dedicated Server
    Replies: 41
    Last Post: 11-16-2010, 10:09 PM
  2. Replies: 2
    Last Post: 10-02-2010, 01:26 PM
  3. Problems with Ezzi - Null Routing IP
    By IWebSolution in forum Dedicated Server
    Replies: 23
    Last Post: 03-13-2006, 10:18 PM
  4. Routing of traffic?
    By Unknown_User in forum Hosting Security and Technology
    Replies: 15
    Last Post: 02-15-2004, 06:22 PM
  5. Sprint 'null routing'
    By magnafix in forum Running a Web Hosting Business
    Replies: 6
    Last Post: 12-24-2002, 02:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •