Results 1 to 26 of 26
  1. #1
    Join Date
    Mar 2009
    Posts
    389

    Hetzner packetloss - every single day!

    We run 60 Hetzner servers, almost all of them in DC20 facility.

    We are seeing packetloss (10%-20%) on a daily basis - lasting from 2-3 minutes up to 1 hour. When no one from our tech support is around to monitor the network, packetloss goes unnoticed by their network support for hours.

    Their network support used to acknowledge the mrt log reports and issue a short explanation, eg. server in your rack was under ddos, etc. , however the last couple of days they ignore the tickets and respond when they fix the problem - saying that no packetloss was detected, asking for another mrt log.

    Please name a couple of reputable online packetloss monitoring websites we can reference when submitting tickets to hetzner.

  2. #2
    Join Date
    Sep 2010
    Location
    Houston, TX
    Posts
    463
    I am assuming you run some sort of firewall? We have had some issues in the past with this. Make sure you are adding in the proper deny/allow. From my previous experience with them however, not a big fan of their network. I was stationed in Germany and would get 80-240 ms ping's 1.5 hours away.

  3. #3
    Join Date
    Mar 2009
    Posts
    389
    Quote Originally Posted by RBurns View Post
    I am assuming you run some sort of firewall? We have had some issues in the past with this. Make sure you are adding in the proper deny/allow. From my previous experience with them however, not a big fan of their network. I was stationed in Germany and would get 80-240 ms ping's 1.5 hours away.
    No ICMP firewall on any of our servers.

  4. #4
    I can confirm the packetloss, it is one of the reasons i have left hetzner some time ago (also left due to their arrogant support staff, customer is never right it seems... but some are happy with them, some are not)

    Be aware it is a budget provider and what you get is a budget product & support.

    You could try setting up observium or maybe paessler (google them)
    But even if you have proof it is Hetzner in the end that must be willing to help you and the packetloss must be in their part of the network.

  5. #5
    Join Date
    Jan 2013
    Location
    Texas
    Posts
    274
    Yeah, I have noticed some packetloss as well... I wouldn't say every day, but maybe 2 or 3 times a week, for 5m to an hour or so like you mentioned.

    I don't guess the datacenter really matters, since you can ping hetzner.de and see the same packetloss. A few times it's been level3, but mostly it just starts when you get to hetzner's local network. Maybe they are getting DDoS'ed a lot?...

    you can run a smokeping here: http://www.dslreports.com/smokeping

  6. #6
    Join Date
    Jun 2012
    Posts
    109
    Quote Originally Posted by softshop011 View Post
    ...
    Please name a couple of reputable online packetloss monitoring websites we can reference when submitting tickets to hetzner.
    https://www.pingdom.com
    http://cloudmonitor.ca.com/en/ping.php

  7. #7
    Join Date
    Jul 2011
    Posts
    40
    Same problem here!

    We are getting many complaints from our users. The problem has been appearing (at least) in the last 2 evenings (CET).

    Hetzner claims the problem is not on their side of the network. However, from what I see, in many occasions the problem occurs at their side since avg. ping time doubles within their network. I might be wrong, of course. In any case, we are experiencing 30% of data loss right now.

    I thought problems may be due to a faulty router that might be fixed but now I see that they have several other costumers with similar issues then the problem may be larger than what I imagined...

    I know they are a budget company but we have been happy with them and would not like to leave now after all this time. However we are getting complaints every one and then. We get a burst of reports from our users because videos are not properly streamed. Then the problem gets fixed automagically and then again after a couple of months the problem occurs again and will last for one or two weeks. Again, in these last 2/3 days then problem has reappeared and Hetzner support say the problem is not on their side. After examining MTR I believe it is indeed in their side. Way back MTR from hetzner is fine but it stalls within their network. But they say it's because of my ISP... How can I say my ISP that there is a problem when I look at these reports and I believe it is not?

    If Hetzner is right, I'd appreciate some more information who to contact and why so that I can call someone & fix the problem. We are stuck with this at the moment...

    Please update with your findings. Thanks
    Last edited by netlanbo; 02-05-2014 at 12:10 PM.

  8. #8
    Join Date
    Apr 2006
    Posts
    919
    Same problem, 10 servers with some packetloss. I hope more information from Hetzner.

  9. #9
    Join Date
    Jul 2011
    Posts
    40
    Did you get any information from Hetzner? They did not reply to the last message I sent them 6 hours ago and it's still failing...

  10. #10
    Join Date
    May 2012
    Location
    Germany
    Posts
    256
    Hello guys, thanks for all of your posts.

    We get sporadic DDoS attacks which can lead to temporary packet loss in certain network segments, but we don't have any major network problems. A few days ago there was an issue with a Russian ISP which caused problems for clients connecting to our network, but that was solved relatively quickly.

    We'd like to investigate the packet losses mentioned in this thread more closely. For anybody who wants to contact us please take a look at the following article in our wiki and please follow the instructions there:
    http://wiki.hetzner.de/index.php/Net..._an_Hetzner/en

    You are very welcome to send me the ticket numbers of the requests you open (so I can make sure they get thoroughly checked), or you can even send me a PM with the MTRs.

  11. #11
    Join Date
    Sep 2010
    Location
    Houston, TX
    Posts
    463
    Yeah, its not the greatest network I've seen.

    Quote Originally Posted by netlanbo View Post
    Same problem here!

    We are getting many complaints from our users. The problem has been appearing (at least) in the last 2 evenings (CET).

    Hetzner claims the problem is not on their side of the network. However, from what I see, in many occasions the problem occurs at their side since avg. ping time doubles within their network. I might be wrong, of course. In any case, we are experiencing 30% of data loss right now.

    I thought problems may be due to a faulty router that might be fixed but now I see that they have several other costumers with similar issues then the problem may be larger than what I imagined...

    I know they are a budget company but we have been happy with them and would not like to leave now after all this time. However we are getting complaints every one and then. We get a burst of reports from our users because videos are not properly streamed. Then the problem gets fixed automagically and then again after a couple of months the problem occurs again and will last for one or two weeks. Again, in these last 2/3 days then problem has reappeared and Hetzner support say the problem is not on their side. After examining MTR I believe it is indeed in their side. Way back MTR from hetzner is fine but it stalls within their network. But they say it's because of my ISP... How can I say my ISP that there is a problem when I look at these reports and I believe it is not?

    If Hetzner is right, I'd appreciate some more information who to contact and why so that I can call someone & fix the problem. We are stuck with this at the moment...

    Please update with your findings. Thanks

  12. #12
    Did someone have solved? I have a server in DC20 too and still facing problems...

  13. #13
    Join Date
    Feb 2006
    Location
    Greece
    Posts
    194
    Did you check this ?
    http://wiki.hetzner.de/index.php/Pin..._aren.27t_real

    Except the packetloss you are claiming, did you realize a loss? A real loss.

    Timeouts, disconnections, a live audio/video streaming with issues and cuts, double checked with pingplotter or mtr ?

  14. #14
    Join Date
    Jan 2013
    Location
    Texas
    Posts
    274
    Quote Originally Posted by Gino Ghini View Post
    Did someone have solved? I have a server in DC20 too and still facing problems...
    I should probably set pingplotter to just ping them 24/7 for a week or so and see what's going on...

    I haven't had much problem with ploss lately, but something is going on over there... as some days I'll have a 160ms ping & other days it'll be 195ms (with very little jitter). That doesn't occur on these oneprovider (via online.net) servers..

    (ed: some load balancing mechanism i guess?)

  15. #15
    Join Date
    Mar 2009
    Posts
    389
    packetloss + network disconnects today - at least 10 times in the last 2 hours..

  16. #16
    Join Date
    Nov 2013
    Posts
    103
    Quote Originally Posted by chrismfz View Post
    I don't see anything except core12.hetzner.de and core21.hetzner.de dropping my ICMP packets which is normal...

    And, uh.... why is this featured?

  17. #17
    Join Date
    Jul 2011
    Posts
    40
    Oh no! It's happening again! :-(

    Last time (2 weeks ago - Feb 4 -> Feb 6), they finally acknowledged they were under an attack.

    "Dear Client,
    unfortunately there was a incoming attack."

    However, first I was told that there was a problem within my ISP. I sent them several MTR traces which I believe were pointing to problems within their network. I was feeling a bit frustrated since I truly believed there was a problem in their network but they kept sending me tutorials how to analyze traffic and deriving the problem to my ISP. Then I checked this forum and found different reports from different users from different countries which was indicating pretty clearly they had a problem. I guess they finally investigated after receiving reports and fixed the issue. Too bad we spent 2 days dealing complaints from our users. At least Hetzner was finally kind enough to solve and acknowledge the real problem.

    And now, only 2 weeks after, it's happening again. I hope this time they can fix it quickly but this is happening more often which is worrisome.

    Are you also experiencing packet loss at this very same moment?
    Last edited by netlanbo; 02-21-2014 at 02:35 PM.

  18. #18
    Join Date
    Feb 2012
    Location
    New York, NY
    Posts
    568
    Quote Originally Posted by netlanbo View Post
    Oh no! It's happening again! :-(

    Last time (2 weeks ago - Feb 4 -> Feb 6), they finally acknowledged they were under an attack.

    "Dear Client,
    unfortunately there was a incoming attack."

    However, first I was told that there was a problem within my ISP. I sent them several MTR traces which I believe were pointing to problems within their network. I was feeling a bit frustrated since I truly believed there was a problem in their network but they kept sending me tutorials how to analyze traffic and deriving the problem to my ISP. Then I checked this forum and found different reports from different users from different countries which was indicating pretty clearly they had a problem. I guess they finally investigated after receiving reports and fixed the issue. Too bad we spent 2 days dealing complaints from our users. At least Hetzner was finally kind enough to solve and acknowledge the real problem.

    And now, only 2 weeks after, it's happening again. I hope this time they can fix it quickly but this is happening more often which is worrisome.

    Are you also experiencing packet loss at this very same moment?
    Which DC was that?

    I'm in DC19 and this week has been relatively trouble-free on the network wide other than a 6 minute period of packet loss Tuesday.

  19. #19
    Join Date
    Jul 2011
    Posts
    40
    Which DC was that?
    DC10. There is a problem in the last hop previous to Hetzner with IP 62.140.25.102.

    Hetzner has contacted Level 3 to solve the issue. I am grateful for that. This was reported yesterday night and it's still not fixed. There is 7/8% of packet loss right now which makes it impossible to stream video, for example. Our service is completely unusable.

    The strange thing is t it looks like that faulty router is only giving trouble to Hetzner clients.

    Try to do this simple search on Google "62.140.25.102" packet loss

    You'll find that EVERYBODY that complains is using Hetzner. I went through the first page of Google results and all results point to forums with Hetzner users complaining about packet loss. And it seems this is a recurring problem happening quite often.

    I feel frustrated. It is maybe a problem on Level 3 and not Hetzner but it is strange that all reports on packet loss for that IP cite Hetzner...

  20. #20
    DC20, 2 servers: packetloss since many months randomly, but in the last week the loss become HUGE from everywhere in europe to those servers, nagios from other webfarms keeps alerting.

  21. #21
    Join Date
    Jul 2011
    Posts
    40
    Quote Originally Posted by marcoparco View Post
    DC20, 2 servers: packetloss since many months randomly, but in the last week the loss become HUGE from everywhere in europe to those servers, nagios from other webfarms keeps alerting.
    Yes... I can imagine I'm not the only one affected. There are reports about this problem in many forums as I mentioned earlier.
    It's still failing. We cannot stream videos without cuts and buffering for the last 3 days. Normal web services are not affected but video streaming is impossible.

    I cannot put up with this situation any longer... I'm not sure whether the problem is with Hetzner of with the Level 3 access to Hetzner. In any case this directly affects Hetzner clients. We're dealing with complaints from our users every 2 weeks or so. It is impossible to use Hetzner for spanish users (Italy and south of Europe too, at least). And I don't care whose fault is. I also sent an e-mail to Level 3. Not sure if anyone received it or not. And Hetzner no longer replies to my messages. They only copy&paste their last reply with the same stuff "We've contacted Level 3 and it's up to them"... Whatever. Meanwhile that Level 3 router is only giving trouble to Hetzner clients. Other datacenters do not seem to be affected. Hetzner will not do/cannot do anything further to solve that problem because officially it is not their problem. So I'm stuck.

    Could you please recommend similar servers particularly targeted to users in Spain? I guess a datacenter in Spain will be better for our needs. Prices are more expensive but will have to go through that route I guess. Any recommendation?

    We don't need a beast machine. We basically only stream video and run a few php (not cpu intensive). Any recommendation will be welcome guys!

    Thanks!
    Last edited by netlanbo; 02-23-2014 at 06:39 AM.

  22. #22
    Join Date
    Oct 2004
    Posts
    294
    Problems seems to be again right now.

  23. #23
    Join Date
    Jul 2011
    Posts
    40
    Quote Originally Posted by bloodyman View Post
    Problems seems to be again right now.
    Yes. It's a complete mess... They won't reply to my messages anymore (they only copy & paste their last reply). I will change provider on Monday. Our service is not working right now which is a complete disaster for our business.

  24. #24
    I have servers with Hetzner in DC 13 and DC 14 and i have no problems at all.

  25. #25
    Join Date
    Jul 2011
    Posts
    40
    It's happening again. Right now...

    Look at this MTR:


    4.|-- 65.red-81-46-1.staticip.r 1.6% 1000 60.5 57.9 49.8 147.0 10.4
    5.|-- et7-0-0-0-grtbcnes1.red.t 6.2% 1000 64.5 57.9 49.4 149.6 12.3
    6.|-- xe4-0-6-0-grtpareq1.red.t 2.2% 1000 77.1 93.7 75.1 207.2 26.7
    7.|-- xe-11-2-0.edge3.paris1.le 2.0% 1000 93.3 90.5 82.8 207.6 12.8
    | `|-- 212.73.205.225
    8.|-- vlan90.csw4.paris1.level3 1.2% 1000 105.6 108.9 102.3 238.2 11.1
    9.|-- ae-92-92.ebr2.paris1.leve 1.3% 1000 104.7 109.3 102.3 237.1 12.0
    10.|-- ae-48-48.ebr1.frankfurt1. 0.8% 1000 105.5 108.1 101.6 218.8 10.8
    11.|-- ae-1-19.bar1.munich1.leve 0.8% 1000 143.4 112.1 102.6 220.0 12.3
    12.|-- ae-0-11.bar2.munich1.leve 1.5% 1000 105.0 111.9 102.1 237.4 16.5
    13.|-- 62.140.25.102 3.8% 1000 91.2 421.8 89.3 850.1 277.0
    14.|-- core12.hetzner.de 17.2% 1000 96.8 399.7 90.1 800.8 278.8
    15.|-- core21.hetzner.de 3.7% 1000 98.4 419.2 92.5 822.6 279.6
    16.|-- juniper1.rz10.hetzner.de 3.5% 1000 99.7 421.0 92.1 810.1 275.2
    17.|-- hos-tr1.ex3k10.rz10.hetzn 3.3% 1000 110.4 423.4 96.6 782.3 274.4


    Avg. packet time is multiplied by x4 at node 13 "62.140.25.102" and packet loss starts right at that node. Packet time gets to more than 400ms! And the loss right now > 3%. In a few minutes we'll get to 8% loss.

    Hetzner already told me node 62.140.25.102 belongs to Level 3 but, from what I found out, it looks like it's only giving trouble to Hetzner.

    Many people in the Internet with Hetzner servers complain about 62.140.25.102. Just take a look in Google ["62.140.25.102" packet loss]. All results are complaints from Hetzner users...

    Problems usually start at 20:00 CET and they are getting worse at 21:00 CET. Every single day (no exception) for the last week or more.

    In an hour or so the packet loss will be around 8%.

    Does your traffic go through 62.140.25.102 ?

    Are you noticing problems?

    Thanks
    Last edited by netlanbo; 02-25-2014 at 04:19 PM.

  26. #26
    Join Date
    Oct 2008
    Posts
    313
    My route doesn't pass through 62.140.25.102 (servers also in DC20) so I don't have this problem.

    I still don't understand why this thread is featured.

Similar Threads

  1. Hetzner packetloss
    By softshop011 in forum Providers and Network Outages and Updates
    Replies: 2
    Last Post: 12-29-2013, 10:10 PM
  2. Hetzner packetloss - whom to contact?
    By softshop011 in forum Dedicated Server
    Replies: 4
    Last Post: 09-10-2013, 12:48 PM
  3. Hetzner an unexpected single point of failure!
    By degrama in forum Dedicated Server
    Replies: 19
    Last Post: 03-15-2013, 03:01 PM
  4. [Hetzner] Problem with my server on first day use :(
    By okitainda in forum Dedicated Server
    Replies: 1
    Last Post: 01-23-2011, 04:42 AM
  5. Single EQ6 from Hetzner available for Sale
    By VBNHost Admin in forum Dedicated Hosting Offers
    Replies: 5
    Last Post: 01-14-2010, 12:40 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •