Apologies in advance for the length, but I would appreciate it if folks read this in its entirety.

For the past 10 months I've had a hosting package with UK Web Solutions Direct (UKWSD). Up until now the service has been nothing short of excellent. However, an incident occurred last week that revealed a shockingly poor level of customer service. So much so that I must take this opportunity to bring it to the attention of anyone who may be considering their services.

Some time on March 3rd, I tried to access my web site to find it unavailable. My POP3 email access was also unavailable. Having ruled out a problem with my ISP, my next immediate thought was that UKWSD's servers were suffering some downtime. However, a friend's web site hosted by UKWSD on the same server was fine, so it looked like an issue specifically related to my domain.

I popped into the UKWSD client forums and found mention of a recent billing software upgrade, so I logged into my billing account to see if some sort of problem had occurred there. It transpired that my domain and hosting package were both flagged with a "suspended" status. I wasn't too alarmed, guessing that a billing software upgrade glitch may have caused this. I immediately opened up a support ticket with UKWSD to determine the source of the problem.

UKWSD's first reply was very short and abrupt. They simply informed me that copyrighted/illegal material had been found on my web site, that the subsequent data transfer was very large and that my account had therefore been terminated. They then provided me with a list of the files, but no evidence of the data transfer.

That was it. There was no offer to help me work out what happened. No billing invoice for the data transfer. No advice on how to go about resolving the issue. They didn't ask me for my side of the story to ascertain whether or not I'd been the victim of an exploit. In short, I felt like I was being treated as a criminal. Perhaps a melodramatic reaction on my part, but I would have expected something a bit more proactive from the get go.

I fully understand that it's the standard reaction of many host providers to terminate first and ask questions later, but UKWSD's initial response was to terminate and ask no questions. UKWSD had now placed the ball in my court. I explained that I had nothing to do with the files and offered a suggestion/educated guess that the files had found their way there as a result of an Invision exploit.

To their credit, UKWSD's second reply was a bit more helpful. Seeing as I was willing to pay the excess data transfer fee, they agreed to knock 30% off the bill. However, they were reluctant to agree that Invision was the source of the problem, telling me it could be anything. I was a bit perturbed by the fact that they didn't really seem to know what was going on, nor did they seem too bothered about investigating it. I don't know about anyone else, but if I were the administrator of a web hosting company and one of my users had suffered a damaging attack such as this, I'd be a bit concerned for the security of my servers and would be do everything possible to work out what happened. UKWSD didn't really seem too bothered by all of this.

Anyway, once I paid the bill, UKWSD sent a third message which was a bit more sympathetic, although they were still dithering about the precise cause of the problem, this time suggesting that Movabletype was responsible. This seemed unlikely. I'm pretty familiar with MT's security issues and this didn't sound like a known MT exploit. Besides, the malicious files appeared in a very specific directory of my Invision installation. There are documented reports of Invision exploits occurring within that subdirectory, so I was 99% sure that was what happened. UKWSD didn't really seem to have much idea of what was happening.

By this time, I was understandably annoyed and frustrated by the whole incident. Not so much with the exploit itself, which thankfully didn't cause any loss of data, but with UKWSD's initially stand off-ish attitude and their general lack of concern. Admittedly they did work quickly to restore my service and were kind enough to reduce the bill, but it was their very first, abrupt response to my support ticket that annoyed me and I'd lost all confidence in their ability to deal with this situation and assure me that something like this wouldn't happen again.

I decided to pop into the Something Awful forums, a regular haunt of mine, to post a message about my experience in their "Serious Hardware/Software Crap" forum. I also posted an apologetic message to the users of my Invision forums, explaining what happened, as well as writing an entry on my blog, which contained links to both these explanations.

UKWSD emailed me once again to express their disappointment that I'd decided to post comments that cast them in a negative light. I explained that while I appreciated the speedy resolution, I wasn't happy with the way I was initially treated and their general lack of concern about the whole incident, hence my decision to post details of my experience. I agreed that I would review the comments I made and make changes, if necessary, but would stand by what I said.

Now, this is where the whole incident should have ended. If things remained as they were at this stage, then I'd have happily continued to use UKWSD and would have removed the negative comments I made (I now felt that maybe I'd been a bit too quick to criticise UKWSD). The SA forum thread had only clocked up about a dozen readers at this point and had dropped off the first page, heading into obscurity. Give it another 24 hours and it would have been as if the incident never occurred.

However, one of UKWSD's support personnel decided to follow me into the SA forums in an attempt to defend his company's honour. While the forums are open to the public, there is a $10 fee for new accounts, so he certainly went out of his way to see what I had to say.

Within his very first message he fired off the legal guns, accusing me of deliberately lying, making libelous remarks and threatened to post the entire support ticket within the forums. I freely admitted that some of my initial remarks may have been inaccurate, but this was only because I was forced into a position by UKWSD where I had to try and guess what had happened, seeing as they weren't too bothered about investigating a breach of their servers themselves.

UKWSD's biggest bone of contention was that I was misleading everyone on the forums by telling them my Invision installation was up to date when it wasn't. It transpired this was true, but I then explained that as far as I was concerned, Invision was fully patched. The software has a built-in mechanism for informing the user if patches were available. In this case, the software was telling me I was up to date, when in fact I wasn't. This was the only reason I made a claim that my Invision installation was fully upgraded. I later retracted my initial claim in a subsequent post.

They then later went on to suggest that I was misleading people by telling them I'd requested certain things (such as ftp/http logs) when I hadn't. The truth was, I had requested these things. Admittedly not via the support ticket, but within the forums themselves. But this was due to the fact that the UKWSD support person was making numerous posts within the SA forums, responding to some of my messages in there. I assumed he was happy enough dealing with my issue in the forums in addition to the support ticket, so I pretty much lost track of what had been said via the ticket and what had been said in the forums. (Unfortunately I can't check because the support tickets I'm supposedly inaccurately reporting have been deleted outright and the minor ones have been closed.)

Anyway, to cut this portion of the story short, the UKWSD employee continued to make posts to the SA forum thread, inadvertently displaying an unprofessional attitude. Other SA readers, many of them server administrators themselves, started asking questions and noting the absurdity of a lot of what UKWSD's employee was saying in relation to various technical queries (such as why 20Gb worth of data was allowed to be transferred into a 300Mb hosting package -- UKWSD claimed there was no way to prevent it, while various SA readers sufficiently demonstrated that it was). By now, the thread was becoming extremely popular, clocking up approx. 10,000 page views.

I then submitted a support ticket to UKWSD asking them for some logs or records that showed the 59Gb worth of data transfer that they had been prepared to charge me for. UKWSD responded, but rather than offering to show me this evidence, they offered to refund the remainder of the fee I had paid. This second support ticket was then deleted. Naturally, I made a post in the SA forums informing everyone that UKWSD had waived the fee but that they were also unable to provide me with the logs showing the data transfer and that the support ticket had been deleted. UKWSD were now desperate to save face and responded by claiming I was continuing to lie and that no support ticket had been submitted. (I've got the receipt for the support ticket sitting in my inbox. How did I receive this if, according to UKWSD, I never submitted it?)

Once again, the issue began to die off. The UKWSD personnel stopped posting in the forums. I emailed UKWSD to thank them for waiving the entire fee and that I would consider removing my original post (although the size of the thread and the 10,000+ readers pretty much made this a moot point) and blog entries as a similar goodwill gesture. In the SA forum thread, I suggested I was willing to close the thread, but the regular readers suggested that it wasn't a good idea and that the standard policy was not to close threads in this particular forum. Rather than be subjected to a ban in the forums, I left the matter as it was, made a few changes to my original post, my blog entry and the message on my Invision subdomain and decided that that was the end of the matter.

Once again, things were beginning to die down and the thread was once again heading into the realms of obscurity.

A day or two later, a friend of mine happened to notice that the "Downtime" post I made in my blog about the incident (a pretty harmless article in itself that simply noted that a problem had occurred and contained a link to the SA thread in case people wanted to read up on it) was now missing. I thought this was a bit strange. I hadn't deleted the blog entry myself, so I found it odd that it would be missing.

After checking over my site, I discovered that the following things had happened.

1) The mySQL database from which my blog is built had been rolled back to a version prior to the "Downtime" entry I'd made.

2) The rendered HTML file that the Movabletype engine created from the database had been deleted. At first I suspected that the entire subdomain (or perhaps my entire web site) on which the blog was hosted had been rolled back too, but I found evidence to suggest otherwise.

3) The subdirectory in which the "Downtime" html file existed was still intact. If my subdomain had been rolled back or restored from a previous version, this subdirectory wouldn't exist.

4) In addition to the individual date-based archive entry (the one that was deleted), Movabletype had rendered a category index and various RSS files which still contained the entry.

There was only one logical conclusion to draw from what happened -- that someone had deliberately deleted the "Downtime" blog entry and rolled back the mySQL database to prevent it appearing when the blog was rebuilt. However, whoever deleted the file obviously overlooked the category index and the RSS files. Or maybe they didn't, because these files also contained lots of other data not related to the "Downtime" entry, which would have been quite damaging had they been deleted too.

I submitted a support ticket to UKWSD, explaining in detail the observations I made and asking for a possible explanation as to how the blog entry had been deleted and the database rolled back.

UKWSD immediately responded by informing me that they no longer wished to discuss any issues related to my domain and that they were issuing me 7 days notice to find an alternative hosting solution, after which they would terminate my account for breach of article 2.2 of their terms & conditions. (A follow-up message than added that they would also refund the subscription fee for remainder of my package.)

I couldn't believe it. I went back to the support ticket to enquire how the files were deleted, but the support ticket had been deleted.

Naturally, I once again posted an update of my experience in the SA thread, informing everyone of what happened. The UKWSD support personnel then appeared in the forums again to accuse me of lying, denied all knowledge of the file deletion/database rollback and said that they never received the support ticket. (Once again, I have a support ticket receipt in my inbox which suggests otherwise.)

That's the situation as is currently standards. I've not sorted out alternative hosting and I'm just waiting for the domain transfer.

I think it's safe to conclude that UKWSD deleted the file from my blog and rolled back the mySQL database. The file deletion was specific enough to be anything other than deliberate (unless UKWSD would try and have me believe that someone else hacked into my blog and decided to delete a single blog entry and roll back the database). The fact that they then had the sheer nerve to deny any knowledge of this incident and claim that I never submitted a support ticket is pretty absurd.

UKWSD's whole reaction to this incident has been nothing short of unprofessional, amateurish and despicable. While I can understand their need to defend their company's honour, going as far as to throw around unsupported legal threats, make unauthorised changes to their clients' web sites and accuse them of lying when there's a stack of evidence to the contrary is abominable.

I totally agree that things were blown out of all proportion, but UKWSD only have themselves to blame for doing that. An issue had been resolved and UKWSD decided to fuel the fires themselves by attempting to alter my web site without my authorisation (I can't see anything in their terms & conditions which gives them the right to do that -- they can certainly object to material and terminate accounts accordingly, but not directly tamper with their clients' data).

Their biggest mistake was resurrecting a dead thread with no more than a few dozen readers, throwing their weight around and displaying a wholly unprofessional attitude, which resulted in the thread growing into a gold-rated megabeast that has, at the time of writing, clocked up in excess of 15,000 readers and caused at least two other individuals to immediately abandon UKWSD and seek an alternative hosting solution.

Thank you for your time and patience in reading this message.

Hi,

I use UKWSD for one of my hosting account's and they have been nothing but excellent.

The fact you did not patch your forum software is not UKWSD fault it is in fact your own fault. These scripts you get for free off developers tend to have lots of security holes in them and it's your job to keep an eye on all the latest expoliots for that software.

If your a user of ukwsd as you say you are you'd of got a recent security news bulletin from them telling you what fantastico software needed updating because of security concerns. If you ignored this then you only have yourself to blame for the problem.

What UKWSD will of done with your database is restored a copy of it that was not compromised which they are well within there rights todo. The fact they waived the fee for excess space and traffic has to be applauded.

I'm save in saying i wont be abandoning my account with them because you failed to keep ontop of security on your forum. I've also recommended there service to my manager(s) and friends who have signed up with them.

One question for you, How come you blame another company who's bent over backwards to help you for your inability to patch a forum?

Did you actually read the message in its entirety? I suspect not.

For starters, you'll see that my complaint has nothing to do with Invision itself and the whether or not it was patched. I accepted all responsibility for this within the support tickets and the posts I made on the forums. My complaint is with the lies, deceit, duplicity and unprofessional attitude which followed.

The security bulletin UKWSD sent out about Invision patches came about as a direct consequence of the incident that happened to me.

I wouldn't really describe their decision to delete support tickets and claim I never sent them, their failure to provide evidence of the data transfer they were charging me for (even though I'd paid for it), as well as deliberately editing my web site without my authorisation and claiming not to have done so as "bending over backwards" to help me.

If you read my message properly, you would see that it was pretty obvious that UKWSD didn't merely "restore a copy of my database that wasn't compromised". (We're talking about my Movabletype database here, not my Invision installation in which the exploit occurred.)

Let me repeat:

The subdomain on which my Movabletype database resided was NOT restored from a previous back up. The database was deliberately rolled back to a version where a blog entry containing criticism of their service didn't exist. ONE VERY SPECIFIC FILE on my web site was DELETED by UKWSD without my authorisation. They have denied doing this. They are lying.

I understand your need to defend UKWSD's service. This time last week I would have done exactly the same thing as you.

Just to emphasise: I'm not blaming UKWSD for my inability to patch a forum. Nor am I denying their right to restore client web sites to a pre-exploit status. I'm criticising them for their unprofessional handling of the support ticket, their deliberate tampering with client data that isn't related to the exploit, simply because it contained a negative opinion of their service and their subsequent lies and denial regarding the issue.

For my next hosting option, I'll know better than to go for a two-man reseller operation run by a bunch of kids.

Ok,

If you say they deleted the ticket, You'd have proof to back this up in the shape of the autoresponse generated by the software they use, Whats the ticket number? that you say got deleted.

If you have not got an autoresponder with the ticket number then the ticket did not make it to the ukwsd mailservers. Has your ISP had any issues with out going mail?

Perhaps the backup of the database that you wanted was currpt? if your relying on the host todo all your backups and making sure that the integrity of them is good then you need to take a look at yourself and your website and see if they are important to you. If they are always make remote backups everyday specially if you have new content posted everyday.

Do you have proof they deleted the file? Can you look in your RAW log files and see a request to delete something made by an IP Address other than yours?

Before you accuse someone of lying bring proof not just what you believe to be the truth.

Hi Sonance,

First of all welcome to WHT ;)

We did take the account offline as first instance as I’m sure any responsible host would do given the situation. We did however restore the site quickly considering this was all happening at around 11pm (GMT).

At that point we thought it was his Invision forum that was exploited so we removed the files from the account and his account was re-enabled. Further investigation the following morning confirmed this, we found that Sonance failed to apply any of the last three security patches which left his forum wide open to being exploited, even though he insisted he did patch the forum. It was only after showing him the file names and dates proving they had not been patched that he held his hands up to this.

We did not delete/change anything from sonance’s website and can honestly say we did not receive the support ticket he claims he has sent regarding this. If we had of received the ticket we would have offered to restore any files missing from a backup as we perform daily backups. I double checked the email parser log to make sure the ticket had not been picked up as spam by the help desk. For those who use esupport you will know that it creates an entry in the parser log for an incoming ticket, i.e. Created reply from customer for <ticket-ref>(08 Mar 2004 07:53:39 AM) for a genuine ticket or Rejected Ticket, Email Banned (07 Mar 2004 03:32:34 PM) for spam.

There was no trace in the parser log so all I can say is we didn’t get the ticket. Do you really think we would delete files from his account knowing he would go running straight back to the somethingawful forum to post about it?

I’m guessing this post appeared here after being pushed by other members on the somethingawful forum who noticed I posted on here. Once they found I was a member over here this was the response of one of the posters on the forum:

“They've got four threads in the shared/reseller hosting offers forum. Let's troll them.”

In conclusion, we refunded the fee charged for data-transfer used in full as goodwill and the re-enabling the clients site was performed quickly. We did not roll back your DB or alter any of your files, like I said we would have been more than happy to restore a backup for you.

p.s. we are not resellers :)

Originally posted by Technics
If you say they deleted the ticket, You'd have proof to back this up in the shape of the autoresponse generated by the software they use, Whats the ticket number? that you say got deleted.

I can't post the full email here because of the "no urls until you've posted 5 messages restrictions", but you can find a link to it here:

h-t-t-p : : / / insert.xo.fxsonet.com/misc/ticket.txt

This ticket was present in their system, because I was briefly able to access it via their ticket web interface. However, it mysteriously "disappeared", UKWSD claimed to have never received it and instead issued me with my 7 days notice within minutes pretty soon after this ticket was submitted.

Perhaps the backup of the database that you wanted was currpt?

Nope. Following the initial exploit that occurred with Invision that forced UKWSD to suspend my domain, with my agreement they restored my entire web site back to full backup that was made before the exploit occurred, albeit with the Invision installation removed. That's fine. End of story. Everything working fine. Everyone happy.

HOWEVER, what happened 2-3 days later was that after I'd posted an entry regarding the incident in my blog, UKWSD then decided to roll back my mySQL blog (without telling/asking me) to a time AFTER the full post-exploit restore but BEFORE I posted the blog entry. Their intention was for the blog to automatically rebuild itself without the article in question. They also deleted the HTML file that's rendered from the database. However, they left evidence of the file's existence by failing to delete other files within the blog that pointed to it.

My support ticket asked them why they did this. This ticket disappeared and they denied ever receiving it.

if your relying on the host todo all your backups and making sure that the integrity of them is good then you need to take a look at yourself and your website and see if they are important to you.Thanks for the lecture, but I do make my own backups too.

Do you have proof they deleted the file? Can you look in your RAW log files and see a request to delete something made by an IP Address other than yours?The RAW logs only show http activity, not ftp activity. The way the files were deleted suggests they were done so via ftp rather than through my blog via http.

We can only roll back either 1 day or 7 days as anyone who knows how cpanel backups work will be able to confirm.

If we had rolled back one day the information you claim we had deleted would still be there, had we rollled back a week more would be missing. We did not alter or delete your site.

Originally posted by Paul-ukhost
We can only roll back either 1 day or 7 days as anyone who knows how cpanel backups work will be able to confirm.Okay, I'll accept the possibility that a cpanel-related rollback didn't occur, but there's no denying the fact that the mySQL database containing my blog entry "disappeared" and was "replaced" by a version of the database in which the entry was missing. I didn't do it, so who did? Are you suggesting that there's absolutel no way for either your or Neil to gain access to your clients' web sites other than through cpanel?

As for the missing support ticket, I'll also be prepared to accept the fact that there's some sort of error in the ticketing system that manages to receive tickets, display them on the web interface, sends me a confirmation email but then somehow manages to delete all traces of the ticket's existence.

Even so, you can probably understand why all of this would cause me to lose confidence in your service.

Sonance,

I do understand where you are coming from and fully appreciate your possition.

I can't explain how your file was edited/removed but can let you know 100% we had nothing to do with it. Like I said we would have restored any files for you from a backup, that would have been no problem.

Welcome to WHT Sonance. There seems to be several things going on here, reading it from the outside...

1) Ok, your forum was compromised and used as a dump - it happens. Why it didn't get caught by the quota is likely because of the way UKWSD runs their php (guessing) - if they run it as a apache module as cpanel comes by default, those files could have/would have been written by the user 'nobody', and thus not counted against your quota.

2) The host is trying to defend themselves, against a percieved slight by one of their clients. What would have been (more) appropriate would have been a single post to the forum, explaining the whys and hows that happened, and ended simply by "If anyone has any questions, I'll be happy to field them directly. Please feel free to contact me at...." and also an apology for any inconvienence.

3) That didn't happen, so now the threadstarter here (Sonance) feels slighted, and thus it carries onto WHT, and who knows where else. While I appreciate the articulate, informative posts that Sonance has made, that time might have better been spent finding a host more suitable to your needs.

4) Don't anger SA. ;)

So that brings us here... Off to read the SA thread, but I hope that lessons were learned by both sides in this equation, and I wish the best for you Sonance in finding a host more suitable to your requirements.

Originally posted by Paul-ukhost
I can't explain how your file was edited/removed but can let you know 100% we had nothing to do with it. Like I said we would have restored any files for you from a backup, that would have been no problem. Okay, we're going to have to agree to disagree on this one, otherwise we're going to be dancing around in circles for all eternity.

As far as I'm concerned, the issue's over and done with. Something's happened to my web site that neither of us can fully explain and we're parting company as a consequence.

When I'm up and running on my new hosting solution (Spenix, for anyone who's curious!), I'll be removing all mention of this incident from the blog and other pages. I won't be proactively continuing my complaint in here or in the SA forums. I've said what I need to say and I've pretty much exhausted all avenues of discussion regarding the incident. I'll just leave it up to other people to make their own judgement call.

If something's occurred that's outside both our means to control it, then I apologise for jumping to any conclusions and for any accusations which, in the goodness of time, prove to be inaccurate.

I agree Sonace, that this issue is closed.

I wish you luck with your new hosting provider.

Originally posted by Paul-ukhost
I agree Sonace, that this issue is closed.

I wish you luck with your new hosting provider.

I'd like to point out additionally you have never fully explained your quota errors.

Basically, the files uploaded through invision exceeded Sonace's quota limit. However, UKWSD's system is not set up to actually quota users in a useful way, therefore he went over his quota despite the fact that it should have been prevented by the methods in place.

Because of this, 51 gig of excess bandwidth was consumed, wheras it could and likely would have been less if quotas had fully restricted the upload. As you have refunded the costs, I'm not pressing this issue, I just want to know what steps (if any) you are taking to fix your quota support.

When the files were uploaded they were written as nobody as thedavid correctly pointed out in his post above.

Ok, read the thread - given the file sizes in question (dvdrips looks like) what could not have happened was a direct-through-http upload. That chokes at 8-10 megabytes, these looked lots larger.

Likely some other script was uploaded (phpshell/phpkonsole or similar) and those files were pulled from an external source using wget, lynx or something similar (and written as the user nobody, since that's who would be calling those applications)

So Sonance, make sure you double check to ensure that there's no strange php files floating around in your home directory when you move - else you'll likely get compromised again and go through the same stuff with the new host.

-David

^^ Which is another reason I advocate running php suexeced, I don't care of it makes it slower, it makes it secure.

Yawn.

Such a pity that they fell out.
A good host for a long time and a good customer for a long time, if I correctly understood.
Just one problem, although a major one, blows it up.
Ok, I hope that this makes both of you a better host and a better customer (there's always room for improvement), and all the readers too!

Originally posted by Paul-ukhost

I’m guessing this post appeared here after being pushed by other members on the somethingawful forum who noticed I posted on here. Once they found I was a member over here this was the response of one of the posters on the forum:

“They've got four threads in the shared/reseller hosting offers forum. Let's troll them.”


Belive it or not.. most the people that posted in that SA thread already knew about WHT. They did not go googling to find a place where you post at to mess with you.

I took the "Let's troll them" comment as a joke.. and so did most other readers I am sure. I give you my congrats on taking it out of context to make the posters of SA look bad.

That is all I have to say.

*Edit.. after following the thread I agree that it was the most unprofessional handling of a situation I have ever seen.

Grow up cybaix.
There was nothing unprofessional in the handling of this. There was poor communication perhaps.

Go back to your little forum and don't take yourself so seriously.

Originally posted by blue27
Grow up cybaix.
There was nothing unprofessional in the handling of this. There was poor communication perhaps.

Go back to your little forum and don't take yourself so seriously.

Wow.. that was rather immature. What an amazing place this must be if you cannot post your opinion without being flamed. So I will take your advice and go back to my "little forum" of 30,000 users where people are atleast allowed to have their own opinions.

I'd suggest you go read the thread in SA. Paul acted completly different in the thread and I think you have not read enough about this to say what you just said.

Have a nice day
:disagree: :disagree:

Originally posted by blue27
Grow up cybaix.
There was nothing unprofessional in the handling of this. There was poor communication perhaps.

Go back to your little forum and don't take yourself so seriously.

I'm afraid you'll find there was, firstly there was the charge of bandwidth overages not counted against a quota. I suppose this counts as 'professional' in your mind as you can charge the client more than is morally acceptable

There's the issue of deletion of tickets, files, and an email nobody will admit to

Stop trolling this forum and being an idiot, thanks.