Or is this newbie overreacting?

All of a sudden, my forum is DOA. Get Nocster to reboot it, ok - took about 2 hours after opening a support ticket and pestering the person on the BURSTNET Aim handle.

Me: do you know what the problem was? (So I can prevent it from happening? )
BurstNET: actually, we had a power problem and the server was on its way up but got hung up to do an fsck

OK, I don't know what fsck is, so I think no big deal, as long as it works.

Now, the MYSQL db is really messed up, and I can't repair it using all the normal methods (repair table, myisamchk, etc). Says error 28: aka hard drive full.

Now, there are two hard drives, each 40 gigs. The files for the db are located in /var/lib/mysql/vbulletin

all 1.7 gigs of it. Problem is this:

/dev/sda3 2064208 2028512 0 100% /var

Burst/Nocster is no help. My brain is no help. My feeble attempt to learn about FSCK is no help. I'm PRAYING this forum can help?

I regularly look at the disk space using df -a, and NEVER were any of the partitions close to being full. My question is twofold:

1) Did nocster screw something up and put my vbulletin database in the wrong partition upon FSCKing it?
2) How can I fix this problem?

Thanks in advance, WHT Addicts!

Your title is a bit bad. Nocster would not purposely ruin your server. If anything, from what I have read, the server screwed itself up with this "fsck" thing. I did a search on Google for fsck, and it appears to be some kind of disk tool. So it is definetly related to your problem. Call up Nocster at their support number, its a lot more reliable than using the helpdesk or AIM.

fsck from my limited understanding of it is used to repair hard drive errors such as corrupted data and all that. Check your log files in the var partition. With all the activity going on they may have boomed and that might be why the partition is now full.

Yes fsck is like windows ScanDisk, just runs differently. I don't think anyone at Nocster would harm you server. Were you running the lastest patches? It is more likely that you were hacked or a disk error occured.

Thanks for the fast responses.

I have WHM and everything is up to date - apache, bind, all the latest stable releases, the whole nine yards.

I have called them and let the phone ring for at least 10 minutes twice so far, no help. A disk error did occur (I think) - because the burst tech said they lost power during reboot.

My DB is probably that big, has about 2 million posts. But, it wasn't supposed to be on a partition with only 2 gigs allotted. In fact, in the past, it never was (as far as I know, I hadn't checked it in a week).

At any rate, how would I go about moving the DB to another partition, where there's a lot more free room?

/var is home to /var/log ... Perhaps you could look there and clear out some old logs until you figure out where to move the db to.

Garrett: My best guess is this. Your hard drive was writing data at the exact time the power failure occured. If that is the case, your hard drive was at its most fragile state, and it seems data or the hard drive has become corrupted.

Now why a backup power didn't go on (or did it and then my theory means nothing) is beyond me.

Post a msg on their forums, submit a ticket with #1 priority and with a meaningful subject and someone will help you.

1) Did nocster screw something up and put my vbulletin database in the wrong partition upon FSCKing it?
2) How can I fix this problem?

1) Nope - by default, MySQL databases are placed in /var - most customers never have problems with /var filling up unless all the system logs get filled up in /var/log

And believe it or not, your mysql databases were always in the /var partition until you moved it today.

Like someone in here mentioned, fsck is nothing bad - it's is a utility that can fix a lot of logical drive errors like 'scandisk' for windows.

2) Took me about 6 hours to repair all your databases :eek4: Yikes- talk about an active board - just about 2k shy of two million posts.

Lastly, you have a separate backup drive - you should use it just in case ;) That's my advice for the day.

And congratulations for such a successful forum :)
Now, I'm going to go to bed :stickout:

Also look in /var/domlog

In my experience, sql stuff is the easiest to get corrupted. It's happen to me almost every time my server used to crash (a few servers ago ;)), although nothing major.

Your best bet, instead of clearing out logs, is to actually put your databases onto another partition. This will give them the freedom to grow to whatever size they need to be. You can either tweak your mysql to look in a different directory for the databases (on a different partition), or what I did was just move them and create a symbolic link to their new location at the old location.

If you are running a cPanel server, login to WHM and clear the mail queue out. That is the number 1 thing which causes /var to fill on cPanel based servers.

It is normally due to a cron job running very frequently (every 1 minute for example) and sending a confirmation message to the user. Before clearing the mail queue, check to see where it's all coming from, for example a cron deamon and then locate the user sending it all.

two hours? oops....quiet long for "managed" servers

Not necessarily. You have no idea how long the fsck took to run.

Thanks John, I really appreciate all the hard work. When I went to bed @ 4 am I ssaw the myisamchk faithfully running.

You are a real asset to Nocster/Burst.

Re: the auto-backup, whenever I have it running, the forum *seems* slow. I have to proof of this, other than "this sure is lagging." I'll turn it back on and see how it works.


Thanks again. Great service!


Originally posted by alpha
1) Did nocster screw something up and put my vbulletin database in the wrong partition upon FSCKing it?
2) How can I fix this problem?

1) Nope - by default, MySQL databases are placed in /var - most customers never have problems with /var filling up unless all the system logs get filled up in /var/log

And believe it or not, your mysql databases were always in the /var partition until you moved it today.

Like someone in here mentioned, fsck is nothing bad - it's is a utility that can fix a lot of logical drive errors like 'scandisk' for windows.

2) Took me about 6 hours to repair all your databases :eek4: Yikes- talk about an active board - just about 2k shy of two million posts.

Lastly, you have a separate backup drive - you should use it just in case ;) That's my advice for the day.

And congratulations for such a successful forum :)
Now, I'm going to go to bed :stickout:

When you run a backup the disk access probably causes a bottleneck... not sure if there's a way around that or not (or maybe a way to have mysql automatically replicate itself).

You could do it at off-peak times, only once a day or something... so even if there is slowdown it shouldn't last too long.