Earlier today I went to update my site. To my dismay I was not able to access my site, or any site hosted on my server. I was, however, able to connect via FTP and SSH. traceroutes and pings to the server were fine.

The cpu usage was low and the load average was around 700-900

I figured a reboot may do me some good. It did not. the server did not come back online afterwards. A Traceroute ends one hop before my server, and pings time out.

what can I do?

Thanks

/etc/rc.d/init.d/httpd restart

Thanks webdude. :)

Becasue my server seems to be offline and I have no SSH access , there is a difference between what i should do and what I can do.:(

Ah..ok. I thought you still had ssh access. Do the only thing you can do, call the data center and sic one of their button pushers on it...

Call the NOC and ask them to reboot it.

I did get the NOC to reboot it, but it made no difference.

I'll see if I can get in contact with some of their button pushers

A hard reboot forces a scan of the drives. This can take from a few minutes to a few hours depending how big and how loaded your drives are. It could be hanging on something in that scan.

If the noc can log onto the server, have them issue this command::

/sbin/shutdown -Rfn now

That will by-pass the scan and bring it up right way. If it comes up, but services such as ssh, apache, etc, dont startup, then you are missing some rc files or have them turned off. They will need to at least start bind and ssh for you to be able to get into it..

thanks a lot. I'll mention that.

If i can get ssh access, at least, i'll be happy.

Originally posted by Webdude
/sbin/shutdown -Rfn now

Once the server comes back up, why would they shut it back down and have it skip the file system checks?

Sounds like a hardware problem to me. Either bad drive or bad memory.

In most data centers, you have button pushers...not actual techs. Rather than go thru all the headaches of explaining to them what to do over the phone, it's easier to have them issue that if nothing is coming online. Doing so removes any possibility that somehow the scan is stopping services service from loading up due to errors it may have found. Not having access to the machine myself and explaining over a forum, it's best to start at point A, and do a process of elimination.

Originally posted by Webdude
In most data centers, you have button pushers...not actual techs. Rather than go thru all the headaches of explaining to them what to do over the phone, it's easier to have them issue that if nothing is coming online. Doing so removes any possibility that somehow the scan is stopping services service from loading up due to errors it may have found. Not having access to the machine myself and explaining over a forum, it's best to start at point A, and do a process of elimination.

I think you meant:

shutdown -rfn now

The "-n" is deprecated, but should cause the sytem to kill all running processes instead of going through init. This would skip things when going down, not when coming up, which would be bad.

The "-f" just skips the filesystem checks on reboot and has nothing to do with services not starting. Anyway, if the machine wasn't making it past the file system check, then it wouldn't be up far enough for you to type "shutdown -rfn now". It would have to be already be up for you to type this. So shutting it down again and having it skip the file system check would be pointless.

What the user might try is actually forcing a check, in case it is a filesystem problem. The "button pushers" should even know this one:

shutdown -rF now

Another thing to keep in mind is that it might be looking for a keyboard. You did disable that in your bios before you shipped the box out, right? :)

More than likely it's hanging on a user input prompt on a fsck. Have the NOC hook up KVM to it and read whats on the screen to you.

You may also want to avoid rebooting servers to restart services. Try tinkering with individual daemons if they're not responding. It'll tell you much more to see the errors they produce, and will help you diagnose the bigger issues at hand. A reboot just makes diagnostic info go away or get buried deeper in log files.

George

That is very good advice, George. I'll keep that in mind.

My server is back up now after about 18 hours of downtime. I never did find out what the problem was or how it was fixed.

Originally posted by WoodShedd

My server is back up now after about 18 hours of downtime. I never did find out what the problem was or how it was fixed.

Ouch. Where is this server located?

Had a similar problem about 2 weeks ago - fortunately on a server that wasn't actually "live" for customers yet.

Problem turned out to be something that we really wouldn't have expected - fault power cable - just died on a reboot. I guess you never know what could go wrong. We always keep spare power supplies handy also as those go from time to time.

Hope you get it back up soon.

Cheers

I once had a problem where I changed the server name for example a.domain.com to b.domain.com in my main WHM and restarted apache, thinking all was well and the next thing I experience, I couldn't login to the server and upon checking using WHM which I could the server status most of the daemons was down.

Panic sets in and hurriedly contact support and fortunately the server came on back after 25 minutes with the new server name.

Hehe, Got a good lecture from support :)

Regards

Don't you love those lectures? I get reminded I am using an unmanaged box every time I ask a question (once a week at most)

Would still like to know who the provider is. Sounds like a good one. :rolleyes: