<warning>This is probably more of a rant than anything else.</warning>

For those of you who have been in business for quite some time, how do you stay calm, rational and sane during what I call freak weeks (or in my case, a freak month). For over a year business has, for the most part been 'smooth sailing'...hard work yes, and a few hiccups here and there, but no major problems. Then all of a sudden you find yourself going through a period where everything seems to go wrong. First, a cPanel bug all but cripples one of the servers, a bug which not even cPanel themselves are able to fix, days later when this is finally taken care of, the same server is then hacked (despite having a firewall configured and putting other security measures in place). You nevertheless get through those problems and everything seems to be returning to normal. Then comes the big daddy of all problems, the hard drive fails, it just simply stops spinning, but not to worry, I've been making daily backups right? Sure I have, so I'm safe. So I think, only to discover that the backup files on the secondary hard-drive seem to be corrupt. Leaving me with nothing but a 12 day old off-server backup.

How do you keep a cool head throughout all of this and resist the urge to run screaming in the opposite direction and go looking for a 9-5 job like other normal adults?

Sorry to hear about your issues that you had, they do happen to almost every host.

I was wondering what plans you had in place to deal with these issues if they were to arrise?

As they say... when it rains... it pours... sorry to hear about your recent troubles.

Maybe try helping out here on WHT more than usual. I feel it builds 'positive karma' against the evil ServerError Gods. :D

Originally posted by WebOnce
As they say... when it rains... it pours... sorry to hear about your recent troubles.

Maybe try helping out here on WHT more than usual. I feel it builds 'positive karma' against the evil ServerError Gods. :D

Lol, thanks. At least that made me smile :).

Originally posted by MH-Gary
Sorry to hear about your issues that you had, they do happen to almost every host.

I was wondering what plans you had in place to deal with these issues if they were to arrise?

Like all serious webhosts we have contigency plans in place, and of course, they vary according to the situation. Putting them in place worked out decently with the first couple of incidents that occured recently. However the hard drive issue threw us a curve ball, as the "backup plan" (no pun intended) didnt quite fall into place.

I know that this is sort of against "usual policy" with a web host (as in must stay going 24/7 with answers etc), but if you have other staff give this a shot. If they're on, ask them to take over for an hour and a half or so.

Here's what you do, or at least what I did (thanks to a friend for the advice).

Go on a walk. It doesn't matter to where, but just go for a walk outside. Look at the scenery, notice how nice it looks compared to a ticket window. Then in an hour and a half, come back and take a deep breath and handle the problems.

You'll find out that the serene walk is a good alternative to stressing out on other problems. It really does usually help! Now if the community outside is hectic, this may not be good of course, but usually it is calm and will be good for you :)

Hope this helps.

Greetings:

During those freak time periods -- hair pulling, gut wrenching, just want to vomit and pass out -- my saviour is remember that my Redeemer Lives (Job 19:25 and many more). The same God who saved me on May 8, 1990 still lives. He is still in control; and nothing freaks Him out.

God Bless.

--- Peter M. Abraham

Originally posted by Moni-Q
Lol, thanks. At least that made me smile :).

Yay! five more ponts of AntiCrash karma and one to SmartClientsWhoUpdateTheirphpBBInstalls!

I have it on good authority that you can cash in your karma points for frequent flier miles... but it seems like a bad idea, as then you would be out of karma and something could happen while on the plane!!! :D

Cheer up. If you have done the best you can to the best of your abilities, then your conscience should be clear. You may have some upset clients, but once you explain it, I have found in nine out of ten cases, they're still rotten but then you can a least laugh at them. :emlaugh: Kidding again, most will be happy to have you as a host provided you keep constant communications with them so they can relay to their customers what is happening.

I don't trust local backups for the most part anyway. They are merely nice to have 'if' they work. We do daily remote backups and random spot-checks for this reason, and also backup the DB data through a different system as well. Works nicely so far :D

edit: glaring spelling issue i had to squash.

Like Refresh said, go for a nice walk. Or if you have a bike, use that gets your heart pumping.

Then come back and get to work. Mind is refreshed, and make sure you are working with someone else. It helps a lot, just drawing on experience before Provincial exams and major test.

I agree with the "go for a walk" philosophy. Sometimes there are things that can get a bit overwhelming, and going for a nice long walk is the only way to take a necessary break.

Best of luck to you!

Go for a brisk jog or some aerobic exercises, believe me it does wonders.

You could also try to think more positive, I mean there is a positive side in the backup failure situation, you did have offsite backups that were only 12 days old (except for dynamic data for well-trafficked websites, that'd be good enough for most customers), meaning you were rather well-prepared. You mustn't ignore the good things or you'll be overwhelmed by the bad.

Originally posted by RambOrc
You could also try to think more positive, I mean there is a positive side in the backup failure situation, you did have offsite backups that were only 12 days old (except for dynamic data for well-trafficked websites, that'd be good enough for most customers), meaning you were rather well-prepared. You mustn't ignore the good things or you'll be overwhelmed by the bad.

That is so true. In the middle of moping to myself I had to stop and think of how fortunate I was to have any backups at all. What if I was relying soley on the secondary hard drive? :eek:
At that point I had to stop and say a prayer of gratitude that things did not go as badly as they could have gone.

We go to massage with "happy ending" ;) in this kind of situations

Just take a deep breath, realize that even in a 9-5 job, problems will happen to everyone. Then figure out the fastest way to get everything back to normal, and work towards it. I have found that if you are honest with your customers, a good customer will understand that things like that happen sometimes and will be understanding as long as you are up front with them.

Keeping out of the "Business Continuity Plan" arena (disaster recovery plan (because you already know about that by now))

Wake up early one morning and then purposely drive in rush-hour traffic. You'll remember why you started your hosting company real fast.

And....don't forget those compliments you save from customers. Review those too. :)

>Freak weeks. How do you deal with them.

I panic and freak out... :D but in the end this leads to the best results for me.

Seriously... you know that Hagakure quote from GhostDog movie ?:

"Among the maxims on Lord Naoshige's wall there was this one: Matters of great concern should be treated lightly. Master Ittei wrote: Matters of small concern should be treated seriously."

It means you need to be prepared so if something ****ed up happens you are prepared really well.

"Just keep swimming....."

Originally posted by Jimerson
"Just keep swimming....."

Yeah... thanks... i so needed that song stuck in my head again.

I would.

A) Call server admins and transfer all negative angry energy to them.

B) Go for a drive

C) Repeat steps A and B until problem is solved =]

>Call server admins and transfer all negative angry energy to
>them.

Not a good idea. You should try to maintain a good relationship to anyone you are working with. If you piss of business partners or employees it will come back to you in the end.

In weeks like that I sometimes get to the point that whilst everything is fscking or whatever there is 30 minutes where all I can do is watch a screen so as such is listen to a cd with new age music which just calms things down.

Also you have lots of angrey clients but I always try to put things into the perspective of that no one is going to die if their website is offline and if you can't get things to work any faster then just relax and let things wash over you

I built a pond.

It has turtles, flowers, bees, fish, birds, a working windmill, and running waterfalls.

I spend a lot of time sitting by the pond :)

I visit the pond multiple times a day...and spend extra moments out there during "THOSE DAYS!". If it's one of THOSE MONTHS then I find myself YANKING out weeds and shifting rocks or redigging the whole thing just to get energy out :)

The pond has helped me emotionally in more ways then I could ever express. It's not a joke ...it's truth. I strongly recommend anyone that's experiences high stress to do something like I have done with my pond. If you can't build a pond... (big or tiny...) try a garden box or anything that can be done 'close to work on a daily basis' so that it can work as an escape for a second here or a minute there.

Even a fishtank by the desk can be helpful. You're bound to obtain a fish that reminds you of "that client" or another that seems to be extra relaxing during "those moments" lol. Crazy geeky people can make up quite a bit of personality into a fish when needed for sanity's sake ;)

Most of all..... remember -- Breath by Breath. Step by Step. Nothing more, nothing less.

Deb

Thank you Deb. Just reading that I'm relaxed. :)

One final note, when the disaster is over....do yourself a favor and count the losses. You'll be surprised at customer loyalty, only a few will leave you and then you will feel all the stress anger an anxiety was worth it. That in itself is very calming (although not as calming as trickling water).

I also have a pond in my backyard.

It's great to go relax, except the work is a pain because it is slanted at about 45degree angle :( (the surrounding area)

If you can, build a minature one and put up a gazebo and put a chair in it. Really relaxing in summer.

Originally posted by ImageTutorials
I also have a pond in my backyard.

It's great to go relax, except the work is a pain because it is slanted at about 45degree angle :( (the surrounding area)

If you can, build a minature one and put up a gazebo and put a chair in it. Really relaxing in summer. There's always the 'upgraded version' :stickout: I recently put a nice webcam on my pond so even when sitting here working and/or too lazy to walk out there...I have full view of the critters :cool: Sure it's slacking... but it's worth it if I can smile and attend to the client on a rough day due to the sneak peaks of the turtles lol

A little bit of wildlife goes a long way..... most would be amazed at how helpful it can be to add a touch of a nature to this high tech industry.

Deb

Do you have any fish in yours? I have Koi fish most are valued at over $200 each....I started with 20 when I got the house.

Then a bastard of a Pelican came by and ate 5 in one day. $1000 down the drain just like that. Tried setting up a Scare Crow sprayer that when it detected motion it shot water in that direction...lost 2 more fish that way.

Finally I put 4 of them in surrounding areas, each in 1 corner of the pond, and blasted the heck outta the bird. Haven't seen it since ;)

Just remember: Always look on the bright side of life :)

Originally posted by ImageTutorials
Do you have any fish in yours? I have Koi fish most are valued at over $200 each....I started with 20 when I got the house.

Then a bastard of a Pelican came by and ate 5 in one day. $1000 down the drain just like that. Tried setting up a Scare Crow sprayer that when it detected motion it shot water in that direction...lost 2 more fish that way.

Finally I put 4 of them in surrounding areas, each in 1 corner of the pond, and blasted the heck outta the bird. Haven't seen it since ;) We've certainly been through the fish disasters :angry: That's when I decided not to spend so much on Koi and went with the $50 range.... Thankfully the current crew has faired well for quite some time. A redig that made the deep end just over 4 feet deep and the addition of more cave-like hiding places in the deeper areas of the pond seems to have resolved the problem. The fish are smart enough to hide when required. Another helpful tidbit was the addition of wild guppies to the pond...they breed like crazy and keep the fish, turtles, birds, frogs etc all fed ;)

Current count in the pond itself is 9 turtles, 10 koi, and an unknown number of guppies, frogs, toads etc.... it's about 40ftX40ft and 1 to 4ft deep depending on where you are in the pond...

At night there's a barred owl that hangs out on the windmill and during the day it's the hawks up there and the squirrels making a general menace of themselves lol

We are unable to keep lilies alive in the pond because the turtles love to eat them but we managed to put a 90 gallon preformed at the top of what we call "Spill Hill" which works as the waterfall and lily growing area. This is the first year with that setup so I'm not positive yet that the turtles wont navigate their way up there to eat the goods. I blew a ton of money on lilies the first year...wont make that mistake again lol.

Deb

Originally posted by thomas.smith
>Call server admins and transfer all negative angry energy to
>them.

Not a good idea. You should try to maintain a good relationship to anyone you are working with. If you piss of business partners or employees it will come back to you in the end.

i mainly meant it sarcastically. the server admin is a longtime friend so he enjoys the phone calls. ( its part of our business contract )

Wow, and I thought my pond was something...

Mine is about 15 x 20 with a total of about 30 fish. No turtles, frogs and especially Owls :S.

We never spent much money on the pond as it came with the house when we bought it, but we still have invested a few grand over the years.

The pond is at least 4 feet deep everywhere, and we dropped in a few of those drainage pipes (cut into pieces of about 2 feet-3 feet long. And put them in good spots for fish to hide.

The best part is that the Gazebo sticks about 3 feet over the pond, where fish can hide.

Wow did we ever get off topic...

Wow did we ever get off topic... Ummmmmmmm errrrrr welllllll..... we're just trying to let the stressed out into our destressed world. It's ummmm errrrr On-Topic Therapy that's all :)

My main pond was 15X20 until this year :) Had a few of THOSE MONTHS plus a few hurricanes that caused us to re-dig it... so bigger it got ;)

I'd love a gazebo!!!! That's one addition I've not been able to make yet that I hope to be able to add someday.

I do some turtle rescues so ponds were a natural progression... there's about 30 turtles here right now but you have to be pond capable and a permanent resident to graduate to our main pond lol

Thus the bottom line is...if you're stressed out.... step away from the computer (on topic hmph! :P)

I have better luck with servers than with fish it seems. Two of my precious arrowanas committed sucide by jumping out of the tank. I finally decided to stick to servers. If a server die, you can replace the parts, but living fishes are really a different matter.

Your fish must really, really hate you :P

Yep, you can replace parts but maybe not the data that was lost.

Well, your ponds do sound very nice. But I have some questions about them:

-Are you keeping backups of fish in a secondary pond?
-Do you have archival backups of those fish stored in a safe offsite location?
-Are you using RAIP (Redundant Array of Independent Ponds)?
-If so, are you using RAIP 1 or RAID 5 redundancy?
-Is your pond location secured with biometric scanners and restricted access areas?
-Do you have staff onsite to deal with the case of fish or pond failure?
-Are you well protected against single bird hackers with a water-gun-based firewall?
-Can you fend off DDOF (Distributed Denial of Fish) attacks?
-Have you got adequate redundancy in your water supply network?

You should think about these things to avoid any freak weeks with your ponds :)

-Are you keeping backups of fish in a secondary pond? I have the one main pond... but I also have two side ponds which are a lot smaller (2 and 300 gallon ponds) but life sustaining. I also have a 300 gallon aquarium in the home plus one 250 gallon aquarium plus two 180 gallon aquariums plus 2 55 gallon aquariums plus one 50 gallon aquarium plus one 40 gallon aquarium plus a few smaller aquariums plus four waterland breeding tubs (70 gallon water/200lb sand) in the porch. Ummmm.... Yes :P

-Do you have archival backups of those fish stored in a safe offsite location? If I have a location in another state I can buy from and have fish shipped in... does that count? Considering the three local locations (outdoors, indoor/outdoor porch, and indoors spread between various rooms) does that count as offsite? Hmmmmm

-Are you using RAIP (Redundant Array of Independent Ponds)? Absolutely... especially in the areas where the water is pumped through multiple ponds creating a virtual copy of the water throughout :P

-If so, are you using RAIP 1 or RAID 5 redundancy? Hmmmm that one really depends on the worth of the life within.... would rather not admit which clients errr I mean animal life holds what worth to me :P

-Is your pond location secured with biometric scanners and restricted access areas? If this were a small time setup they would be...but since our livestock is so important to us and their residence important to them we stepped up the security a notch. It's protected by triple fencing (6ft, then 4ft, then 3ft) and two large rottweilers plus a german shepherd/mix. Additionally, we have a security camera on the pond 24/7 that is not only watched by the staff but by the internet at large :D

-Do you have staff onsite to deal with the case of fish or pond failure? Yes. At all times. And we should point out...all of the fish made it through three hurricanes !!!!!

-Are you well protected against single bird hackers with a water-gun-based firewall? No. But I should refer you back to the dogs...they chase off those birds every day and to-date have had an excellent success rate. We always entrap the birds by strategically placing decoy food around the outer perimeters.

-Can you fend off DDOF (Distributed Denial of Fish) attacks? Those that choose to reside in our specialty setups indoors can in fact enjoy the safety net of this service. We carefully select which fish will reside in which tanks to avoid any chance of one particular fish adversely affecting any of the others. For our special fish, such as the 2lb Oscar, a 250 gallon aquarium is setup just for her in a room all her own ;)

-Have you got adequate redundancy in your water supply network? Absolutely! We are proud to be able to provide city water which is treated (far far too often with far far too much) chlorine and flouride to keep the fishy and turtle teeth healthy? *ahem* as well as a natural spring that flows right under our basement providing a steady supply of water to the basement (needing to be automatically pumped out to the bog in the yard... reference drowning puppy pic here (http://www.picolio.com/Albums/Trinity/TrinityMudPuddle070805e.jpg)) as well as a fully functional well for those that enjoy that sulfur smell that provides fresh untreated water to all of the tanks in addition to rain catchers for the acid lovers out there.

When you need a place to store your fins..... the choice is obvious!!! :rofl:

:crazy: :crazy: Deb :crazy: :crazy:

hahahaha.

Someone had too much time on his hands. I only have 4 of those! Oh no.

Originally posted by timdorr
Well, your ponds do sound very nice. But I have some questions about them:

-Are you keeping backups of fish in a secondary pond?
-Do you have archival backups of those fish stored in a safe offsite location?
-Are you using RAIP (Redundant Array of Independent Ponds)?
-If so, are you using RAIP 1 or RAID 5 redundancy?
-Is your pond location secured with biometric scanners and restricted access areas?
-Do you have staff onsite to deal with the case of fish or pond failure?
-Are you well protected against single bird hackers with a water-gun-based firewall?
-Can you fend off DDOF (Distributed Denial of Fish) attacks?
-Have you got adequate redundancy in your water supply network?

You should think about these things to avoid any freak weeks with your ponds :)

or one. Whatever. If we have issues with something we replace it right then. If we figure out what the problem is afterward and are able to fix it with a 100% sureness level, we sell it on ebay =P.

Sounds like 2 strikes would have been fine for that server. And by strike i mean ANYTHING. Hacked, you spill crap on it, it explodes, the color is a bit off from all the other [whatever]s of the same make/model we have, it looks at you the wrong way, etc...

I've found that somehow some crap gets cursed and no matter what you do, the spirit of the cursed thing will stick around to haunt you. By spirit i mean: if you have a f'd server and you take everything out of the chassis and put new stuff in, that new thing's still cursed4sure. get rid of everything and be done with it.

Also customers love to know that you're spending money to fix things for them. and don't just lie and say you are, 'cause they'll find out4sure.

In my few years running httpme, there were never really any freaky weeks, as such. It was all pretty constant. Each day was a challenge, no doubt, but things were generally pretty smooth. Sure we had our share of scammers, spammers etc, but that just comes with the territory. Other than that, no worries. :)
Originally posted by mrzippy
I agree with the "go for a walk" philosophy. Sometimes there are things that can get a bit overwhelming, and going for a nice long walk is the only way to take a necessary break.
Yeah, it's very good to pry ourselves away from the keyboard and get out there and breath some fresh air and get some good old fashioned excercise. My routine was that I would get up around lunch time, do a few hours work, and then get on the bike and ride to a local coffee shop, read the paper, and just chill out. Then I'm back in the office working until 6am. I slept from 6am to 12 midday.

Of course, working at home also makes things easier, imo. The hosting business is not such that you really have to be chained to your desk for 16 hours straight. I would have some DVDs on in the background, and take little breaks like that. You're never more than 5 meters away from your workstation, for when things need your immediate attention, so there's some flexibility there, and that keeps you sane. :)

I do back ups in mutiple ways. Try backing everthing up from the orginal server to th other. Try getting a seperate drive to the servers.

Hi.

I just dont know what the **** to do if my server's hard drive crashes. I understand the importance of doing local backup, but this will only backup the customer's files, and I am not even sure if that will backup my customer's MySQL database. I can easily perform a complete backup of the /home partition by loggin in to my server's SSH account and copying everything on that partition to a folder within my hard drive. BUT... In the event that my server's hard drive crashes, after I replace the HDD of my server and upload all of these files to the new /home partition, will THAT also restore my customer's database which appears at my WHM account? including my customer's cPanel passwords and User Name (I know my customers's User Name is the directory name that appears on the /home partition), will that also restore all of my customer's MySQL database? If not, then what is the exact linux location I would have to go in order to download the entire MySQL database? What EXACTLY needs to be backed up from the server in order to do a perfect restore by just re-uploading everything to the new HDD with a fresh new Linux installation and a fresh new cPanel/WHM installation?