Before I say anything, they ARE working on it. But it's taking forever, and they seem completely disoriented about their own network.
We run a very above-ground business and we are well-known in our industry. We have several servers with 100tb.com because parts of our business is bandwidth-heavy. Our main website forwards all email to a CRM system running with Google Apps.
Yesterday, out of the blue, WestHost started blocking outbound port 25 on our server, resulting in us being unable to communicate with our customers. We conclusively tested that we could not connect to any mailserver from SSH on the server, and that we could make the same connections from any other command-line. So it wasn't a Google Apps issue -- our highly qualified server admins determined that this was purely a host issue.
As I said, we are very above-ground. We don't send bulk, we are on no blacklists. All our email is SPF and DomainKeys up the ying-yang. We routinely test email delivery, and we monitor all our logs. Our server is not a spam source, but a highly fortified and PCI compliant server.
Now, when we communicated with 100tb.com, they sent us to WestHost, and apparently, we have been migrated. Except that WestHost had no record of our server. Many people worked on the issue with admirable zest, but the 100tb.com WestHost system is broken. Before, we'd get lost somewhere in a SoftLayer datacenter. Now we're apparently with WestHost, and nobody can find our server.
Finally, a high-level admin arrived on the job and did some undescribed fix, and email started flowing again. For about 2 hours, then it froze up. We've now been down for 6 hours again, and we're waiting for this high-level admin to show up for work, even though we're told it's a weekend, so it's unknown when he might show up.
It's unnerving that WestHost/100tb.com/VPS.net/UK2 have absolutely no clue where our server is. It's frustrating that nobody knows what the heck is going on, including why some border-firewall is now blocking outbound email, with no notice to us, with no reason -- we are legitimate in every possible way -- and everybody is absolutely clueless.
Besides a sporadic flow of email this morning, we've soon had our entire company's email blocked for 24 hours. We have a heavy tour of seminars, and we're trying to send people their booking confirmations, as well as the God-darn address they have to go to for the seminar. People are paying $500 for these seats, and we can't communicate with them, and we're getting deer-in-the-headlights from the host.
The people are nice, and are trying hard. But the system is broken, and we're really in pain right now because of this.
If any admins are reading along, the 100tb.com Ticket# is 25376772
Hi Per, just saw this thread and it did seem odd to me. Talked to my IT Dir. and he said it was due to the mailchannels filtering server having issues. He's removed all policy routing so mail will flow normally until we can get this server issue resolved.
I will follow-up in the brand confusion and have the team work through you ticket. If you server was with 100TB it should still be there; I've never heard of anyone just switching brands like that... WestHost and 100TB are both part of the UK2 Group.
Last edited by jneeley | westhost; 06-11-2011 at 02:55 PM.
It is working again now, although I'm a bit rattled after this problem coming back again so soon after being fixed last time.
About brand confusion, we've always call the 800 number on 100tb.com, which sends us to Hosting Support in the UK. When we have filled out tickets, they've always been answered by Midphase.
With the servers with SoftLayer (previously), you've always had trouble finding them, and random routing problems have always cropped up after datacenter work, since you've never had a record of the IP addresses mapping to the machine, so after every major outage, the outage has always been extended by another 15 hours for us as we try to convince you that the IP addresses are not mapping to the machine.
Add to that that the 100tb.com control panel would routinely forget our servers. And we're still paying for nodes that aren't showing in the CP we have therefore been unable to upgrade our main server to the full number of nodes we're paying for.
And we have no idea if the server backups we're paying for are actually happening, since the control panel is completely out of touch with any back-end data. The daily backup seems to be current. The weekly and monthly backups shows as having last run in June 2010, but I've been assured that they're running -- not particularly reassuring.
Apparently now, our IP addresses belong to WestHost, so when I talked to 100tb.com support, they could no longer find any of our IP addresses with SoftLayer and had no record of us besides our customer number.
However, WestHost had never heard of us, except an old, old record from 2005 when we had a server directly with WestHost. That alone extended the outage by 8 hours. We were basically sitting with a server that had been firewalled out, a devastating outage for us in the middle of our busiest time, and nobody could find our server anywhere, and everyone kept passing it around. 100TB said "talk to WestHost" and WestHost said "talk to 100TB". Finally, Nicola at 100TB took matters into her own hands and started pushing for it.
All in all, when 100TB works, it works just fine. But when things go wrong, the primary problem is your absolutely stunning lack of contingency planning. You have absolutely no record of anything anywhere. You don't even know which IP addresses we're using. I've written previously to beg you to maintain just a modicrum of a customer record. But apparently the only information stored is the plan size and the credit card number. Everything else has to be supplied from scratch every single time. It would be the EASIEST thing in the world for you to store just a bit more information about each customer, and then the response time in a crisis could be reduced dramatically.
Can you email me your IP address? While I work specifically with VPS.NET, another UK2Group company, I can possibly help clear up the on-going confusion, or at least bring the right people into play to get things straightened out for you.
My email is terry(at)vps.net. Alternatively, if you'd like to work with Thomas, the MD of 100TB, he's always reachable at tom(at)uk2group.com.