I have several web sites hosted with KCDWorks. I have not had access to them or email since wednesday evening. Does anyone have any clues?

Just getting a little concerned...

http://www.dnsreport.com/tools/dnsreport.ch?domain=KCDWorks.com

Looks like the servers down.

Any idea how long they will be down or why?

We sent the following email too all affected customers (that I had personal addresses for) just a short while ago:

Dear Customers,

It is my hope that I reach you all with this message, as I do not have updated contact info for all of you.

As most of you are aware, I'm sure, our main production server that houses most of your accounts has been offline for close to 24
hours. This is due to a DDoS (Distributed Denial of Service) attack that was launched against our server.

At the present time, the server is being restored. There has been no loss of data as a result of this occurance. Rather than
transfer all sites to a new server (which would require that we wait 48-72 hours for DNS to resolve to the new server), we have
elected to bring this server back online as soon as possible.

At the present time, we are awaiting activation of a new CPanel license, as we have replaced the hard drive in the machine that was
attacked to ensure that the filesystem remains secure.

Once that license comes live, we will take a few minutes to transfer all customer data to the new hard drive and bring everyone's
site live once more.

I would like to thank everyone for their patience in this trying time, and I would also like to assure everyone that we are doing
all that we can to restore services as soon as accuracy allows. We must be very careful as we do so to ensure that your data and our
security is intact.

Anyone wishing to contact me may do so at <address edited, PM me if you need it. I will not be able to provide further details of the downtime due to
security issues, but I will keep you all appraised of the situation as it develops.

Regards,

Charlie Rhodes
KCDWorks.com

DISCLAIMER: The information contained in this E-mail is,
unless otherwise indicated, confidential and is intended
solely for the use of the named addressee. Access,
copying or re-use of the E-mail or any information
contained therein by any other person is not authorized.
If you are not the intended recipient please notify us
immediately by returning the E-mail to the originator.

Replacing a hard drive and Cpanel license because of a DDOS?

Wow, sounds intense. :eek:

Good luck

Thanks, hostpc

Thanks Charlie for letting us know what's going on.

Any ETA to bring back this server online?

Thanks.

Originally posted by mmastriker
Thanks Charlie for letting us know what's going on.

Any ETA to bring back this server online?

Thanks.



Anyone wishing to contact me may do so at <address edited, PM me if you need it. I will not be able to provide further details of the downtime due to
security issues

I guess I really don't understand why a DDOS requires a new license/hard drive.... would you be willing to talk about the specifics via e-mail?

Thanks Charlie for letting us know what's going on.

Any ETA to bring back this server online?

Thanks.

At this time there is no ETA. I opened a billing ticket with CPanel, as the license transfer is really all we're waiting on to get the server back online.

hostpc, you can send me a PM with your email address and I'll shoot you an email.

"I guess I really don't understand why a DDOS requires a new license/hard drive.... "

Same here. Charlie can you clarify this for us?

Originally posted by mmastriker
"I guess I really don't understand why a DDOS requires a new license/hard drive.... "

Same here. Charlie can you clarify this for us?

Sure. This box is at Rackshack. Their abuse department was worried that the box was compromised during the attack in some way, and are requiring us to have the system restored before it can be brought back online.

The system has been restored, all except for the CPanel license activation. Once that is complete, we will be restoring the customer files and databases, and then will begin the process of re-securing and re-compiling system software, along with bringing in an outside auditor to test our security.

As soon as Rackshack hears back from CPanel on the license, the box will be back up and we will begin data restoration.

At this point, Rackshack is saying that they are waiting on CPanel to respond about the license, and CPanel (according the billing ticket I opened) says that they haven't yet recieved the request from Rackshack. We're pushing it more with Rackshack at the moment to try and speed this up. The CPanel billing tech has promised to keep his eye on the queue for our IP address.

Wow. I was wondering what was going on with my website. Since December 2002 I have always been very pleased with their service. Quick to answer questions and good customer support. I am just kind of disappointed this time that we didn't get this email Wednesday or yesterday at least. There are some people business at stake. Maybe a little yelling at Rackshack might speed things up. I think it is kinda weird that the CPanel license is at fault.

Oh, well, I am sure that the guys at KCDWorks.Com are on it. Hope we get a little discount on our next billing. :)

<<<Signatures need to be set up in your profile per Forum rules.>>>

Originally posted by Albert Roux
Wow. I was wondering what was going on with my website. Since December 2002 I have always been very pleased with their service. Quick to answer questions and good customer support. I am just kind of disappointed this time that we didn't get this email Wednesday or yesterday at least. There are some people business at stake. Maybe a little yelling at Rackshack might speed things up. I think it is kinda weird that the CPanel license is at fault.

Oh, well, I am sure that the guys at KCDWorks.Com are on it. Hope we get a little discount on our next billing. :)

<<<Signatures need to be set up in your profile per Forum rules.>>>]

Always good to hear from satisfied customers.

As far as the email yesterday goes, we never expected things to be down this long. Today, when the downtime is seeming to be so lengthy, I decided to blast that email out to everyone. My apologies for not getting it to you guys sooner.

The downtime is not the fault of CPanel, but the *extended* downtime seems to be some fault of miscommunication between Rackshack and CPanel somewhere along the way.

You are right; there are some businesses at stake, ours included. Luckily, we have a great relationship with about 99% of our clients, and everyone has been very understanding about this unplanned outage, which we (and especially I) appreciate greatly.

I should say that the server went down about 10:30AM Central time yesterday, not Wednesday.

"Always good to hear from satisfied customers."

You are very welcome. I know how important customer feedback is for a business like yours. I am a very satisfied customer, and would recommend KCDWorks.Com to anyone. You provide me with an excellent service at an incredible price. I cannot complain.

And I was just joking for the billing discount. At $4.95/month I cannot complain.

I hope you nail this soon though.
Good luck,

<<<Signatures need to be set up in your profile per Forum rules.>>>

Good luck resolving this Charlie.

Out of interest, are Cpanel licences issues on a per IP basis? If not, is it possible to buy a spare to keep on hand for situations like this?

Gary

Originally posted by Editor
Good luck resolving this Charlie.

Out of interest, are Cpanel licences issues on a per IP basis? If not, is it possible to buy a spare to keep on hand for situations like this?

Gary

They are per IP. Unfortunately, that is also what makes it impossible to have a spare on hand. The CPanel database only allows one license to be logged for each IP.

Thanks for the well wishes. At this point, it's a "hurry up and wait" situation. Everyone is anxious to be back up, but all we can do is wait.

Being able to update the recorded IP per license in real-time would solve that. Might be worth suggesting.

Gary

Originally posted by Editor
Being able to update the recorded IP per license in real-time would solve that. Might be worth suggesting.

Gary

Probably wouldn't be a bad idea.

Thanks for ringing in here Charlie, was wondering what was going on. I didn't get that email you posted, so looks like I need to update that with you -- I'll PM that to you shortly.

Anyway, what about email? Since email is down w/ my sites and reseller client sites, is it getting bounced back to senders or will it go through and accumulate? Just wondering, as I was expecting some important emails today...

Mark
cimara.com

Hi Mark,

That depends on the mail server on the other side. If the sender's mail server is configured to try again every x hours over the course of a few days (most servers are) then you will recieve your mail hopefully without any issues.

If it is not set to retry, or if it is set to give up instantly on unroutable domains, then the mail will bounce back to the sender.

We haven't recieved any other updates at this point, and unfortunately aren't likely to until morning.

I was under the impression that Cpanel distributers/partner nocs had some sort of console access to the cpanel licensing server and could add/edit/etc in real time

Originally posted by Nicholas Brown
I was under the impression that Cpanel distributers/partner nocs had some sort of console access to the cpanel licensing server and could add/edit/etc in real time

I just found out that they can - as long as there isn't already a license listed for that IP. In that case, they have to email someone at CPanel to have it looked at, so everything can be verified by hand before a new license is granted.

In case there is anyone that still has not provided me with their contact email address, here is the latest notice I sent out to customers.

Dear customers,

At this time I unfortunately have no further updates.

The tech at CPanel that I have been speaking with (this person deserves a raise, let me tell you), has been most helpful. However,
our request for a CPanel license seems stuck in the endless queue at Rackshack. Below is a snippet that one of our administrators
got while speaking to Rackshack's live support about our ticket. This conversation happened after we were alerted that the Customer
Service department at Rackshack has our CPanel request and has not yet processed it:

[12:37:01] Keith N : Hello
[12:37:09] Thomas : Hello
[12:37:10] Keith N : Pulling up your account, one moment please
[12:38:49] Keith N : Right now we are waiting on our Customer Service department to get a license from cpanel--
[12:39:11] Thomas : You haven't requested it yet though as I just talked with cPanel
[12:39:16] Keith N : The ticket will be worked by the Customer Service department in the order in which it was recieved--
[12:39:38] Thomas : It was received more than 2 day ago which is why I wonder
[12:39:44] Thomas : 1 day I mean.
[12:40:23] Keith N : We will simply have to continue to wait, I am sorry, I cannot expedite the issue any further. It is in the
correct department, in the correct queue waiting ot be worked
[12:41:04] Thomas : Okay, thanks for your time.
[12:41:07] Thomas : Bye
[12:41:42] Keith N : Thank you for your patience with us, bye now

We have since requested that our case be taken to a supervisor, for which we were supplied an email address to contact that
supervisor. I am fairly certain that our request will not be seen until Monday in that event.

I am on my way back to the phones now, to see how loudly this "little fly" can buzz in the ear of Rackshack. Hopefully they will
press the issue through, as the tech at CPanel has assured me that he has an eye on the request queue there to snatch our license
request up so that it can be expedited on their side.

I would like to thank you all once again for your patience as we try to escalate this issue in order to restore services as quickly
as possible.

I'm halfway hoping that headsurfer or Mario will happen by this thread and assist us in escalating our issue here, since it is at the request of their Policy Enforcement Team that we are being asked to move to a new hard drive.

Was hoping to have things back up by the start of this work week. Got email update last night, but everything seems to still be down as of 2:30pm EST Monday. Just wondering if there was any new news or ETA to be back up/online?

Thx,

Mark

Originally posted by Cimara
Was hoping to have things back up by the start of this work week. Got email update last night, but everything seems to still be down as of 2:30pm EST Monday. Just wondering if there was any new news or ETA to be back up/online?

Thx,

Mark

Not at this time. RS is still telling us "The ticket is in the correct court to be worked. It is in the queue."

We are trying yet again for an escalation right now.

Charlie - did you try that support contact I gave you?

Originally posted by hostpc.com
Charlie - did you try that support contact I gave you?

We did, that's how I managed to get the license actually requested (finally)

Now, I'm not sure what the holdup is, but I'm trying that contact again.

Thanks very much for your help, by the way.

Ok, thanks for the update - hanging tight here and hoping for the best!

Mark

Originally posted by Cimara
Ok, thanks for the update - hanging tight here and hoping for the best!

Mark

The CPanel license was activated. Now we are awaiting the server to be reconnected so the transfer to the new hard drive can begin.

It took talking to a sys admin at Rackshack and the CFO of CPanel ... but we're on our way now.

I must also give a two thumbs up for KCDWorks. They are a great host. I've been with them since Oct of 2002 and they haven't given me much to complain about. :)

Originally posted by LucidParody
I must also give a two thumbs up for KCDWorks. They are a great host. I've been with them since Oct of 2002 and they haven't given me much to complain about. :)
Agreed. I've been with them since late November, and this is the only time I can remember that we've really had any downtime. Charlie is quick to respond and as seen in this thread, always puts his customers first.

All websites look like there back online, email seems to be still down...almost there! :)


Mark

Emails up! Looks like everything is back online!

Thanks for all the hard work and keeping us updated Charlie!

Thanks for the good words, guys. They are much appreciated.

3 hours of sleep in as many days. It's nap time.