I had a user receiving the following error when using invision board:
mySQL query error: DELETE FROM ibf_sessions WHERE running_time < 1066569446 or ip_address='64.185.146.148'

mySQL error: Table 'ibf_sessions' is read only
mySQL error code: 1036
Date: Sunday 19th of October 2003 10:17:26 AM

I thought I would try restarting mysql. IT wouldn't stop via /etc/rc.d/init.d/mysql stop so I killed it's pid. Now when I try to start it I get:
root@h4 [/home/teamacid/www]# /etc/rc.d/init.d/mysql start
root@h4 [/home/teamacid/www]# touch: creating `/var/lib/mysql/h4.hostspace.us.err': Read-only file system
rm: cannot remove `/var/lib/mysql/h4.hostspace.us.pid': Read-only file system
Fatal error: Can't remove the pid file: /var/lib/mysql/h4.hostspace.us.pid
/usr/bin/mysqld_safe: line 269: /var/lib/mysql/h4.hostspace.us.err: Read-only file system
Please remove it manually and start /usr/bin/mysqld_safe again
mysqld daemon not started

I've tried to remove /var/lib/mysql/h4.hostspace.us.err manually and I get:
root@h4 [/home/teamacid/www]# rm /var/lib/mysql/h4.hostspace.us.err
rm: remove write-protected regular file `/var/lib/mysql/h4.hostspace.us.err'? y
rm: cannot remove `/var/lib/mysql/h4.hostspace.us.err': Read-only file system

How can it be that as root I can't remove this file?

And it seems to affect the whole filesystem:
root@h4 [~]# cp .bash_history test
cp: cannot create regular file `test': Read-only file system

root@h4 [~]# rm sysstat-4.0.7-3.i386.rpm
rm: remove write-protected regular file `sysstat-4.0.7-3.i386.rpm'? y
rm: cannot remove `sysstat-4.0.7-3.i386.rpm': Read-only file system

The server is with FDC... first I submitted a ticket asking about why mysql wouldn't start and they responded that the server is unmanaged...

Well, now I've rebooted the server and it has not come back online (pings, ssh, and http all fail). I've put in a high severity ticket with them 25 mins ago and still no response :(


Assuming they can get the system to boot up for me, does anyone have any ideas about the read only filesystem? The server was working perfect last night and nothing was changed.

Were you doing anything? Any commands in root? Sounds like an rm -r *. I did that and thats what happpened. Tell me what commands you were doing.

As I said... it started as a client complaining this morning of a forum not working. Last night it was working fine.

I didn't do anything to cause the problem... it was like it when I got there this morning to investigate.

...server is still offline and no response from FDC is 45 minutes.

If FDC has a number, call them. Even if it is a sales number because a tech will pick up.

They only have a ticket system, and a sales rep on icq (who is offline right now).

I'm kinda stuck... the server isn't being used in production _yet_, but I have several potentials already today which I would like to put on that system.

(312) 913-9304

Try that.

I've had a email response at 12:02pm (I was away from the computer). They're going to run fsck on my server.

Will that damage any data on there? Although there's no paying customers on there I do have some files.

I'm kinda stuck... the server isn't being used in production _yet_

I know this isn't going to help with getting your server online right now, but are you really happy putting paying customers on a server that you don't even have a contact phone number for the hosts?

Just a thought...

FDC is a large supplier of servers, and even on my small queries via tickets in the past I've had very prompt response.

That number is straight of of the Whois Info.

Well... they got the server to boot so I could ssh, but the whole filesystem is still read only... kindly they closed the ticket already.

I've reopened the ticket and I can see they're in there again and it's just been rebooted...

so far this is 2 hours downtime... I'm not happy.

I would really suggest you call them.

Read Only?!? Interesting.

Check `dmesg` for lines saying something along the lines of remounting your partitions as read-only due to <error>. Might give you some answers... YMMV.

OK, we've made some progress... they're going to replace the drive and copy the contents of the current one into a subdir on the new... I'll have to reconfigure cpanel again, but at least I'll have a working server :P

*crosses fingers* If this works I'll be happy :D, if not well... I won't be :angry:

You could ask them to dd the old drive to the new one then fsck it - saves re-installing everything...

unless there's physical damage. even dd with noerror can bail on surface failures.

True. You'd have a hard time just doing a `cp` in that case though ;)

My experience has been that nearly EVERY time you see something like:

Error in ext3 journal, remounting read-only

Its time to toss the drive in the can and start over. Even after you fsck, and remount the partition, it tends to happen again a few short days later.

Anyone know any technical details on how/why this happens? Bad sector on the drive? Who knows.

So far still no luck.

I had them put the faulty drive in another server of mine and I copied all the data to the secondary hard disk (/dev/hdb (which contains /home). Then this morning cpanel had SMART errors saying /dev/hda was dying.

I requested /dev/hda be replaced and redhat 9 and cpanel be reinstalled (this morning).

As for the machine the faulty drive was in, I requested _yesteday_ they replaced the hard disk, and put a clean install of freebsd 5.1 and cpanel on it, they did this and cpanel did not work. I requested they reinstall this machine completely also (yesterday).

Tonight I come home, and they have emailed me asking to confirm both jobs, instead of just doing them as I already requested. So right now, I have 2 servers offline, and this has been on going for almost 32 hours now. I am not happy as I am still paying for these servers with no discount, and I am losing business as the server that _was_ working (which now has the smart errors) was going to be the destination for new clients.

I'm off to read their TOS to see if there's a liability clause... if not they're in deep ****. :angry: :angry: :angry:

Well... no answer to my email, the phone number in their whois info rings and rings... no answer... and their icq sales rep is online, but also not answering.

Anyone want to make me a free first month offer to switch elsewhere? right now I'd seriously consider it.

P4 2.4ghz, 512mb ram, 120gb hdd, unmetered 5mbps conn sounds good for $150/mo (incl cpanel)... but not if the server isn't online!

OK, I've spoken with them on icq (about 7:30pm EST), they're fixing both servers now, and adjusting my billing period so I won't pay for the downtime.

Time will tell... sooner or later time will tell (quote from C&C Red Alert ;))