Page 1 of 2 12 LastLast
Results 1 to 40 of 50
  1. #1

    Major router down - Effecting East Coast & Europe

    Hello,

    I am not sure if anyone else knows yet, but there has been a mjor router problem in the East Coast & Europe. Its been going on for the last 10 hours or so. One of my site's is hosted on a CrystalTech server, and they are saying that it is not their problem, its the main router's problem which is handled by AOL & UUNET. Since this is my main site, I am going nuts here. I have heard nothing of this from any media source nor web site. The Internet Health Report & Internet Storm Watch have no record of this issue.

    Has anyone else experienced this?

    CompWorld

  2. #2
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    Yep, can't get to ev1servers.net. Few clients are reporting they can't get to sites on ev1 servers. No reports from clients with sites on NAC servers though.
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  3. #3
    Join Date
    Mar 2004
    Location
    Pennsylvania, U.S.A.
    Posts
    1,579
    I didn't hear about this but good to know, thanks.

  4. #4
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    We have servers with ServerMatrix and ECUI and they both went down about 10 minutes ago, so I guess it must have something to do with this - any more info appreciated if anyone has it :-)

  5. #5
    Nothing affected at ServInt that I can tell, no complaints from overseas either.

    I can also access EV1, NAC and SM from the Philly area without any problems..

  6. #6
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    Can someone check out the EV1 forum to see if someone has postyed any other info? I would do it but...
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  7. #7
    Just this that I see:

    Network Maintenance
    Network Maintenance
    03/22/04
    2:00 AM 4:00 AM CST (-6 UTC)
    Reason: Code Upgrade (router) upstream provider
    Benefit: Improved Network Performance


    Traffic will be re-routed; you may experience some latency as routes re-converge.

    Best regards,

    Val


    __________________

  8. #8
    Yea, it makes no sense. I had been told that it was the router that was maintained by UUNET or AOL TIME WARNER, but I've just been told that it is maintained by Level 3. I am trying to see if I can get any further information. But The host isn't really telling me much. I have also contacted other news media and Internet related companies, but it is really slow going. If other customers also contact their providers about this issue (if they are experiencing the same issue ofcourse), it may get to ball rolling a bit faster.

    CompWorld

  9. #9
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    I tracerouted a client's IP from the server his account is on. He's in Jordan:
    Code:
    [[email protected] root]# traceroute 80.90.162.38
    traceroute to 80.90.162.38 (80.90.162.38), 30 hops max, 38 byte packets
     1  rs-64-246-56-1.ev1.net (64.246.56.1)  0.353 ms  0.297 ms  0.190 ms
     2  ivhou-207-218-245-30.ev1.net (207.218.245.30)  1.598 ms  0.358 ms  0.386 ms
     3  ivhou-207-218-223-4.ev1.net (207.218.223.4)  0.389 ms  0.372 ms  0.591 ms
     4  216.54.253.1 (216.54.253.1)  1.034 ms  0.898 ms  0.978 ms
     5  dist-01-ge-2-0-0-0.hsto.twtelecom.net (168.215.172.44)  0.990 ms  0.935 ms  0.969 ms
     6  core-02-so-0-1-0-0.dlfw.twtelecom.net (168.215.53.61)  5.584 ms  5.542 ms  5.582 ms
     7  core-01-ge-0-3-0-1.nycl.twtelecom.net (66.192.255.0)  50.576 ms  50.587 ms  50.641 ms
     8  peer-01-so-0-3-1-0.nycl.twtelecom.net (66.192.240.62)  50.953 ms  51.100 ms  50.962 ms
     9  fe100-nyc-cb.emix.ae (198.32.160.93)  60.792 ms  233.729 ms  228.076 ms
    10  * * *
    11  * * *
    12  *
    198.32.160.93 is from California which is one weird route out of Texas via Colorado.
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  10. #10
    I know this effects customers that use Crystaltech as their host. I am not sure if effects any other service providers.

    CompWorld

  11. #11
    I can access CrystalTech right now from the Philly area...

    6 13 ms 14 ms 14 ms tbr2-p012301.phlpa.ip.att.net [12.123.137.66]
    7 20 ms 19 ms 19 ms tbr1-cl9.wswdc.ip.att.net [12.122.2.85]
    8 36 ms 35 ms 35 ms tbr1-cl4.sl9mo.ip.att.net [12.122.10.30]
    9 37 ms 37 ms 37 ms tbr2-cl2.sl9mo.ip.att.net [12.122.9.142]
    10 51 ms 51 ms 50 ms tbr2-cl6.dlstx.ip.att.net [12.122.10.90]
    11 70 ms 70 ms 70 ms gbr2-p40.phmaz.ip.att.net [12.122.10.86]
    12 78 ms 68 ms 69 ms gar2-p370.phmaz.ip.att.net [12.123.142.49]
    13 69 ms 68 ms 71 ms 12.125.99.94
    14 70 ms 70 ms 70 ms 216.119.107.1
    15 70 ms 69 ms 71 ms www.crystaltech.com [216.119.103.254]

  12. #12
    I can get to CrystalTech, but none of the sites hosted by CrystalTech. Strange... Makes no sense.

    CompWorld

  13. #13
    Whatever is going with these other sites it sure doesn't look like a major East Coast router issue to me...

  14. #14
    That was what I was told by Crystaltech. I do not know if it is just CT or anyone else. The rep said that it effected the major router that is supplied by L3. So, you would think that if it effected L3, then it would effect everything else down the line... No?

    CompWorld

  15. #15
    If the that router you're talking about was affected I would think that everything downstream would be affected as well, although stranger things have happened before. Do you have a domain or IP of an affected site I could check?

  16. #16
    Sure,

    Here's a few to check:

    216.119.97.54
    65.110.80.138
    216.197.126.154
    216.119.65.180


    CompWorld

  17. #17
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    There is definitely something weird going on because I can't access ev1servers.net but everybody else can.
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  18. #18
    Nope, I was able to access ev1servers.net w/o a problem.

    CompWorld

  19. #19
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    Exactly, tres bizarre....
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  20. #20
    They all seem to work for me....

    *****************************************************************
    6 12 ms 27 ms 12 ms tbr2-p012301.phlpa.ip.att.net [12.123.137.66]
    7 20 ms 19 ms 20 ms tbr1-cl9.wswdc.ip.att.net [12.122.2.85]
    8 37 ms 36 ms 37 ms tbr1-cl4.sl9mo.ip.att.net [12.122.10.30]
    9 37 ms 37 ms 40 ms tbr2-cl2.sl9mo.ip.att.net [12.122.9.142]
    10 50 ms 50 ms 50 ms tbr2-cl6.dlstx.ip.att.net [12.122.10.90]
    11 70 ms 67 ms 69 ms gbr2-p20.phmaz.ip.att.net [12.122.10.82]
    12 67 ms 69 ms 72 ms gar2-p370.phmaz.ip.att.net [12.123.142.49]
    13 69 ms 68 ms 68 ms 12.125.99.94
    14 197 ms 69 ms 68 ms 216.119.107.1
    15 * 84 ms 71 ms www.webengineer.nl [216.119.97.54]

    *****************************************************************
    6 14 ms 12 ms 17 ms tbr2-p012301.phlpa.ip.att.net [12.123.137.66]
    7 19 ms 19 ms 27 ms tbr1-cl9.wswdc.ip.att.net [12.122.2.85]
    8 35 ms 35 ms 36 ms tbr1-cl4.sl9mo.ip.att.net [12.122.10.30]
    9 37 ms 35 ms 35 ms tbr2-cl2.sl9mo.ip.att.net [12.122.9.142]
    10 52 ms 51 ms 50 ms tbr2-cl6.dlstx.ip.att.net [12.122.10.90]
    11 69 ms 69 ms 68 ms gbr2-p20.phmaz.ip.att.net [12.122.10.82]
    12 67 ms 68 ms 68 ms gar2-p370.phmaz.ip.att.net [12.123.142.49]
    13 68 ms 70 ms 68 ms 12.125.99.94
    14 124 ms 69 ms 68 ms 216.119.107.1
    15 69 ms 68 ms 69 ms mangom2.com [65.110.80.138]
    *****************************************************************
    6 13 ms 14 ms 15 ms tbr2-p012301.phlpa.ip.att.net [12.123.137.66]
    7 21 ms 19 ms 21 ms tbr1-cl9.wswdc.ip.att.net [12.122.2.85]
    8 35 ms 37 ms 36 ms tbr1-cl4.sl9mo.ip.att.net [12.122.10.30]
    9 36 ms 35 ms 36 ms tbr2-cl2.sl9mo.ip.att.net [12.122.9.142]
    10 50 ms 50 ms 50 ms tbr2-cl6.dlstx.ip.att.net [12.122.10.90]
    11 69 ms 71 ms 70 ms gbr2-p40.phmaz.ip.att.net [12.122.10.86]
    12 68 ms 68 ms 68 ms gar2-p370.phmaz.ip.att.net [12.123.142.49]
    13 68 ms 68 ms 69 ms 12.125.99.94
    14 84 ms 70 ms 69 ms 216.119.107.1
    15 70 ms 70 ms 70 ms villagex.net [216.197.126.154]
    *****************************************************************
    6 17 ms 13 ms 12 ms tbr2-p012301.phlpa.ip.att.net [12.123.137.66]
    7 19 ms 19 ms 20 ms tbr1-cl9.wswdc.ip.att.net [12.122.2.85]
    8 44 ms 35 ms 35 ms tbr1-cl4.sl9mo.ip.att.net [12.122.10.30]
    9 38 ms 35 ms 36 ms tbr2-cl2.sl9mo.ip.att.net [12.122.9.142]
    10 55 ms 50 ms 50 ms tbr2-cl6.dlstx.ip.att.net [12.122.10.90]
    11 69 ms 70 ms 70 ms gbr2-p40.phmaz.ip.att.net [12.122.10.86]
    12 69 ms 68 ms 68 ms gar2-p370.phmaz.ip.att.net [12.123.142.49]
    13 69 ms 70 ms 68 ms 12.125.99.94
    14 122 ms 70 ms 70 ms 216.119.107.1
    15 70 ms 69 ms 70 ms parkleigh.com [216.119.65.180]

  21. #21
    Are you located on the east coast of the US?

    CompWorld

  22. #22
    Right outside Philadelphia, 20 minutes from the Atlantic Ocean. I'd call that East Coast....

  23. #23
    FYI, EV1 and all sites hosted on my DC1 server are down from Sweden. SM box OK.

    Extremely annoying.

  24. #24
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  25. #25
    Yea.. I am loosing customers and sales to. I am seriously considering moving to a more reliable service. But, if it is affecting you also, then how can it be just Crystal Tech? You would think that since it affects serveral different hosts, one of the major media outlets would cover it. If they would, I bet it would be addressed and corrected much faster than this.

    CompWorld

  26. #26
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    Here's what I get....

    UK to ECUI:

    Tracing route to demontools.net [66.194.37.181]
    over a maximum of 30 hops:

    1 <1 ms <1 ms <1 ms 192.168.0.1
    2 7 ms 13 ms 15 ms 172.30.15.254
    3 14 ms 38 ms 10 ms cmbg-t2cam1-b-ge-wan42-109.inet.ntl.com [80.1.20
    2.137]
    4 11 ms 10 ms 12 ms cmbg-t2core-b-ge-wan61.inet.ntl.com [80.1.201.15
    3]
    5 12 ms 17 ms 11 ms nth-bb-b-so-210-0.inet.ntl.com [62.253.188.197]

    6 17 ms 16 ms 13 ms pop-bb-a-so-100-0.inet.ntl.com [213.105.172.14]

    7 17 ms 33 ms 11 ms pop-bb-b-ae0-0.inet.ntl.com [213.105.172.77]
    8 15 ms 17 ms 20 ms so-3-0-0.cr1.lhr3.uk.above.net [208.185.188.41]

    9 87 ms 88 ms 88 ms so-7-0-0.cr1.dca2.us.above.net [64.125.31.186]
    10 88 ms 87 ms 102 ms 64.125.29.122.available.above.net [64.125.29.122
    ]
    11 117 ms 117 ms 115 ms so-2-2-0.cr2.dfw2.us.above.net [64.125.29.9]
    12 136 ms 134 ms 116 ms so-0-0-0.cr1.dfw2.us.above.net [64.125.28.209]
    13 160 ms 161 ms 162 ms so-3-0-0.mpr4.sjc2.us.above.net [64.125.29.45]
    14 160 ms 160 ms 160 ms so-0-0-0.mpr3.sjc2.us.above.net [64.125.30.1]
    15 179 ms 179 ms 177 ms 64.125.28.22.available.above.net [64.125.28.22]

    16 203 ms 182 ms 177 ms pos2-0.mpr1.sea4.us.above.net [64.125.31.94]
    17 186 ms 178 ms 178 ms twtelecom.com [198.32.180.64]
    18 179 ms 178 ms 178 ms dist-02-so-2-0-0-0.sttl.twtelecom.net [168.215.5
    4.173]
    19 224 ms 222 ms 223 ms core-02-so-0-0-0-0.dlfw.twtelecom.net [66.192.24
    5.15]
    20 234 ms 235 ms 245 ms 66.192.246.17
    21 252 ms 420 ms 401 ms tagg-01-so-0-0-0.mmph.twtelecom.net [66.192.241.
    74]
    22 * * * Request timed out.
    23 * * * Request timed out.
    24 * * * Request timed out.
    25 * * * Request timed out.
    26 * * * Request timed out.
    27 * * * Request timed out.
    28 * * * Request timed out.
    29 * * * Request timed out.
    30 * * * Request timed out.

    Trace complete.

    The trace to Servermatrix actually makes it all the way through with no problems, but http requests just time out.
    Last edited by DTMark; 03-21-2004 at 01:55 PM.

  27. #27
    I can ping and tracert to all of those sites. Just cannot actually go to the web sites via a browser or check mail.

    CompWorld

  28. #28
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    This will be affecting anyone that is being routed through AboveNET. I'm sure most providers are dropping AboveNET and routing via another provider if they can. I hope ev1 yank them or do something soon. Been this way for 10 hours now.
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  29. #29
    Join Date
    May 2002
    Location
    Sunny California
    Posts
    1,679
    We're at AboveNet and haven't received any complaints. I logged into AboveNet's portal and there isn't a mention of network outages on their end, either.

    This looks like it may simply be an AboveNet to EV1 routing issue and not a larger outage.
    Erica Douglass, Founder, Simpli Hosting, Inc.
    I founded Simpli Hosting, and sold it in 2007 to Silicon Valley Web Hosting after over 6 years in the business.
    Now I'm blogging at erica.biz!

  30. #30
    Considering I am on Crystaltech, it is a bigger issue than that.

    CompWorld

  31. #31
    Well that would certainly explain why I had no problem accessing the sites. I don't go through aboveNET on any routes.
    Quote Originally Posted by compworld
    Yea.. I am loosing customers and sales to. I am seriously considering moving to a more reliable service. But, if it is affecting you also, then how can it be just Crystal Tech?
    I'm not sure why it would just be CrystalTech but possibly because CrystalTech can't re-route their traffic to a back up provider?

    Has EV1 re-routed around the issue now?

  32. #32
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    We had a call from a Customer this morning who could reach their site via FTP, but not HTTP. At the time all was working fine from here so I advised it was probably an issue with their ISP.

    However a couple of hours later we had Alertra reports for both our ECUI and SM servers simultaneously and we can't access them either now, so it would seem to be more widespread.

  33. #33
    Join Date
    May 2002
    Location
    Sunny California
    Posts
    1,679
    When I traceroute to demontools.net (actually, its DNS doesn't work, so I traced to 66.194.37.181 instead), I get the same problem as DTMark -- I can't go past twtelecom.net. AboveNet appears to be fine from everywhere. Since their portal doesn't mention anything about an outage, either, I am not sure this is an AboveNet problem.
    Erica Douglass, Founder, Simpli Hosting, Inc.
    I founded Simpli Hosting, and sold it in 2007 to Silicon Valley Web Hosting after over 6 years in the business.
    Now I'm blogging at erica.biz!

  34. #34
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    Our DNS is at ECUI which would explain that :-( The phone line seems to be red hot at the moment, and I'm sure they're doing everything possible to sort it; it isn't an ECUI issue by any means - it is a little frustrating as one of our Clients should be demo'ing their new site we helped them with, to their Client this evening :-(

  35. #35
    Seems to have been corrected now. Atleast at my host. Hopefully, everyone else who has the same problems have been corrected also.

    CompWorld

  36. #36
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    No joy yet, have been offline for 3 hours now :-(

  37. #37
    It's affecting Telewest customers in the UK - connections from there to SM servers are sporadic and connections timing out.
    Invectis - Windows 2000, 2003 and MS SQL Server web hosting

  38. #38
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    We're now up to 5 hours of hard downtime on our ECUI servers, traceroutes all dies at a TW Telecom router, this is getting quite worrying.

    I'm debating whether to move our secondary DNS server over to Servermatrix now, although the IP change would probably take a while to update in the face of the fact that I have no idea when TWTelecom will sort the problem out at least I could be "doing something", I'd hate this problem to go on for another 24 hours by which time I could have got the DNS moved and some of the key sites as well.

  39. #39
    Join Date
    Dec 2001
    Location
    Above The Clouds
    Posts
    6,999
    Got 4 hours sleep and woke up to find everything okay on our end.
    Laurence Flynn @ atOmicVPS LTD
    Linux & Windows Cloud Hosting Solutions Powered by OnApp
    Fully Managed [Shared][Reseller][Cloud VPS] [Dedicated]
    Featuring the atOmicSTACK ● Speed ● Performance ● Reliability

  40. #40
    Join Date
    Jun 2002
    Location
    Welwyn, UK
    Posts
    72
    All back up here as well. Now, what on earth happened? ;-)

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •