OK I can now speak on the other end of having a DOWNED server. Ours crashed last night around midnight, all because of me thinking I knew how to fix a certian problem and I didn't, and it took me till 8AM just to have an up and running version with cPanel to accept clients.

-I never realised how easy it was to install cPanel
-It took us 8 hours max, but that is because I tried rebooting a couple of time, and took an hour nap when the techs were investigating the server
-It took them an hour to find out that they couldn't pivot root and had to do a reinstall of the OS *ouch*
-That took and hour, and from there I went on to install cPanel, and recreate user accounts

All in all it took 6-7 hours to get the server back to where it was able to accept clients. However, it is taking longer to tweak individual accounts and getting things to work the way they did before.

I openly admin for our service we do not do backups of the sites regurlary (we had a planned rsync of the server for today talk about Murphy's laws) in order to keep overhead down and give our clients the price that we do. I am very upfront about our backup because I tell them if it is a mission critical website they may want to look elsewhere.

But today was a really painful lesson that I wish I could've avoided, but instead of folding me and the guys worked through it. I think it makes us a better hosting company, but I can do with fewer lessons like today :)

It was painful and contacting our clients, sometimes waiting for them to contact us because of some of the lost client records. It doesn't do well to have this happen with mostly new clients. The good thing is that by being honest and telling them what really happened (I was an idiot) we didn't lose a single customer, and had several more people sign up today from customers our clients referred during the incident!

Here's to being up almost 24 hours! :D

Yep, a secondary backup drive is a good thing. :D I know what you're talking about... we experienced a server hardware failure on one of our servers last week, and it sure did hurt. Although we had backups, it was still a lot of work tweaking everything and getting it to run like it used to.

Anyway, good luck to you, and I imagine you have a couple more days work ahead of you as a result of this! :)

Actually I just have one last problem I need to solve, and everything is back to normal, and I hope for a long time. I have a client, we use cPanel, who has created a subdomain and redirected it to a URL on his site, but it is not working. If I can solve this I can take a break from the almost 24 hours of work, and time I have been awake.

Can't find anything on the cPanel forums about this problem, or I have and there is no resolution. I just don't know what to try now, and I don't want to start "tweaking" things again like I did last night and crash the entire server again. :(

Yep, from my experience for a cpanel restore, the following will give you trouble:

-subdomains interacting with parked domains
-SSL (ouch! don't even go there!)
-DNS entries sometimes don't get dedicated IP's

Other than those (which we were unaware of before we did the restore, and had to work through for about a week), everything went smoothly.

I mean, probably less than 5-10% even emailed us when the server was down, so I suppose everything went well, but when those 5-10% email you every 30 mins, it can be a nightmare! :)

You live and you learn, I guess! At least now we know what to expect next time this happens (hopefully not for a long time!)

Originally posted by mpope

...At least now we know what to expect next time this happens (hopefully not for a long time!)

*Knock on wood*

I hope it doesn't happen for a LONG LONG time :)

Doh! I forgot to knock on wood! (*knock knock*)

I hate to say it, but this really is the best way to learn what you're not doing right... we've got 10x better backup systems now :D. I just wish it didn't cost us the cash and potential reputation damage that it probably did.

Wait until you have 2 dozen servers. I am dying tonight (bad, bad flu) and one of our Cpanel servers is acting up. I swear, nothing ever happens until I'm one of the following:

a. sick
b. on vacation
c. just bragging about how great our uptime is. :P

--Tina

I hope it goes well for you Tina, and I can only hope of having 24 servers instead of 2 one day :) hoping...hoping....hoping...hoping

umm tina 10,000 customers on 2 dozen{24} servers ? i think i m lost can sumone slap me:D ehh!:eek:

:angry:

Originally posted by feather
umm tina 10,000 customers on 2 dozen{24} servers ? i think i m lost can sumone slap me:D ehh!:eek:

:angry:
Yes, we can arrange for someone to slap you. :D

We've only got teh 15 servers, and that's enough, especially with nice little cpanel surprises, every now and then. :eek2: :eek3:

Originally posted by feather
umm tina 10,000 customers on 2 dozen{24} servers ? i think i m lost can sumone slap me:D ehh!:eek:

:angry:

I never said we had 10,000 customers on 2 dozen servers.

--Tina

Tina,

I think he is referring to this (http://www.webhostingtalk.com/showthread.php?threadid=122275&highlight=inverseflow) thread where you mention you have 10,000+ clients. Along with this thread where you mention you have two dozen servers, that would lead one to think you have 10,000+ customers on two dozen servers.

Not to get the thread off-track but are you still liking Inverseflow? You can PM me...I did not want to revive that old thread. :)

Thanks

Back to the thread...

I never had a hard drive failure but it is something I pray doesn't happen. I am just starting to get organized with multiple servers. One thing this thread did point out to me is to hard copies of things, like client listings/domain listings, etc.

I'm glad everything worked out for you.

Originally posted by Chuggles
Tina,

I think he is referring to this (http://www.webhostingtalk.com/showthread.php?threadid=122275&highlight=inverseflow) thread where you mention you have 10,000+ clients. Along with this thread where you mention you have two dozen servers, that would lead one to think you have 10,000+ customers on two dozen servers.
How many domains/clients/customers a host claims to have, probably shouldn't be discussed in this thread.

We do more than just hosting. We also have local dialup customers, colo customers, web design clients, (I didn't really count these) domain registration customers and more.

Originally posted by AffordableHost
We do more than just hosting. We also have local dialup customers, colo customers, web design clients, (I didn't really count these) domain registration customers and more.

Oh, one more thing...there used to be a way to check all of a host's hosted domains. If you know how, run a check on ns1 through ns26.affordablehost.com. That will ONLY work for .com, .net and .org (I believe) AND it will not show our clients who are using their own nameservers. However, what it will show you, is that 10,000+ customers is a very realistic number for me to claim.

--Tina

Originally posted by feather
umm tina 10,000 customers on 2 dozen{24} servers ? i think i m lost can sumone slap me:D ehh!:eek:

:angry:

*slaps you and calls you a troll*

Originally posted by lightnin
*slaps you and calls you a troll*
Trolls at WHT??.....:eek2: :D

hmmm... 10000 customers.....

Originally posted by AffordableHost
Oh, one more thing...there used to be a way to check all of a host's hosted domains. If you know how, run a check on ns1 through ns26.affordablehost.com. That will ONLY work for .com, .net and .org (I believe) AND it will not show our clients who are using their own nameservers. However, what it will show you, is that 10,000+ customers is a very realistic number for me to claim.

--Tina

I beleive you used to be able to do this with Network Solutions, but it would only show domains registered with them that that were using that nameserver.

A domain and a client are 2 different things. The ratio is 10:1 for us - 700 clients and 7000 domains hosted on our servers.

Originally posted by NexDog
A domain and a client are 2 different things. The ratio is 10:1 for us - 700 clients and 7000 domains hosted on our servers.

Yes, I know. :D

--Tina

Originally posted by NexDog
A domain and a client are 2 different things. The ratio is 10:1 for us - 700 clients and 7000 domains hosted on our servers.
Yep. These days, number of domains hosted, doesn't mean a heck of a lot. It's the number of separately paying clients and at the end of the day, after tax profits. :)

Well, since the relationship between customers and sites has little to do with the title of this thread, perhaps we should continue that discussion eslewhere.