Results 1 to 12 of 12
  1. #1
    Join Date
    May 2001
    Location
    Dayton, Ohio
    Posts
    4,977

    WHT Outage Feb 9, 2007

    Today the post table on the WHT forum database became corrupted. I have restored from a live replicated backup and I am repairing the old post table so no posts will be lost. I am sorry for the inconvenience!

  2. #2
    Join Date
    Aug 2005
    Posts
    3,587
    live backups rule! Glad you could save this beast, good to be back. Good luck restoring the old posts etc.

  3. #3
    Join Date
    May 2001
    Location
    Dayton, Ohio
    Posts
    4,977
    Quote Originally Posted by amygdela
    live backups rule! Glad you could save this beast, good to be back. Good luck restoring the old posts etc.

    We have a couple slave MySQL servers, I backed up the current post table files to another location, moved the post table files from one of the slaves and turned the site back online. I have repaired the old post table and reintegrated any posts that were not written to the slave post table yet.

    Pretty simple process

  4. #4
    Join Date
    Aug 2001
    Location
    Boston
    Posts
    1,568
    I saw the outage this morning...and I was really crossing my fingers to see a new WHT layout when I came back since SWR made a vague announcement the other day!
    InterNich LLC
    Founder

    Bringing you PicResize.com (More than 95 million pictures resized since 2005)

  5. #5
    Join Date
    Aug 2005
    Posts
    3,587
    Pretty simple process
    With a sig like yours, I believe that

  6. #6
    Join Date
    Aug 2003
    Location
    Twin Cities Area
    Posts
    5,649
    that was a horrible experience!
    if you haven't considered chapter 7 bankruptcy, maybe you should.
    eliminate your debt, keep the property you want, most people qualify.
    contrary to popular belief - no attorney is necessary!

  7. #7
    Join Date
    Sep 2005
    Posts
    250
    Good Work
    -It's not about how much, it is about how to.

  8. #8
    Join Date
    Jun 2006
    Location
    Amex & Amex
    Posts
    1,291
    Quote Originally Posted by The Prohacker
    We have a couple slave MySQL servers, I backed up the current post table files to another location, moved the post table files from one of the slaves and turned the site back online. I have repaired the old post table and reintegrated any posts that were not written to the slave post table yet.

    Pretty simple process
    Wow, I'm really impressed.

    Seeing how I'm new to vbulletin management would you care to shed the light on how the database just came corrupted?

    Is it cause its just so large?

    How do you manage such a large database?

    When you say you have multiple "slave MySQL servers" does that mean that the main database is copied over to multiple other servers? Is that the "live replicated backup"?

    Sorry for so many questions but I'm really interested in this regard.

    Thanks

    ppc123

  9. #9
    Join Date
    May 2001
    Location
    Dayton, Ohio
    Posts
    4,977
    Quote Originally Posted by ppc123
    Wow, I'm really impressed.

    Seeing how I'm new to vbulletin management would you care to shed the light on how the database just came corrupted?

    Is it cause its just so large?
    From time to time the MyISAM index file can become corrupt, it happens for numerous reasons, sometimes it's a bug in MySQL and other times it's because a query improperly exited (or was killed). This time I'm not quite sure, the post table on dBforums became corrupt over the weekend as well, so I'm thinking it could be a MySQL bug in the version we are running. I am planning an upgrade this week.

    Quote Originally Posted by ppc123
    How do you manage such a large database?
    It takes a lot of patience, management tasks can take quite a long time on multi-gigabyte tables. You can't always use phpMyAdmin to do things, instead you have to do it by command line.

    Quote Originally Posted by ppc123
    When you say you have multiple "slave MySQL servers" does that mean that the main database is copied over to multiple other servers? Is that the "live replicated backup"?

    Sorry for so many questions but I'm really interested in this regard.

    Thanks

    ppc123
    We have two slave database servers, one for off loading searches and the other for backups. They are constantly replicating the changes that are done to the master database servers. It would take hours to run a repair on a 5gb+ table, so instead, I shut down the slave server and copy the replicated post table to the slave and restart the slave SQL instance. I then run the repair on the old post table and figure out if any posts were lost and reimport them into the new post table. It's complex but it gets the job done

  10. #10
    Join Date
    Sep 2006
    Location
    England, London
    Posts
    418
    Quote Originally Posted by The Prohacker
    From time to time the MyISAM index file can become corrupt, it happens for numerous reasons, sometimes it's a bug in MySQL and other times it's because a query improperly exited (or was killed). This time I'm not quite sure, the post table on dBforums became corrupt over the weekend as well, so I'm thinking it could be a MySQL bug in the version we are running. I am planning an upgrade this week.



    It takes a lot of patience, management tasks can take quite a long time on multi-gigabyte tables. You can't always use phpMyAdmin to do things, instead you have to do it by command line.



    We have two slave database servers, one for off loading searches and the other for backups. They are constantly replicating the changes that are done to the master database servers. It would take hours to run a repair on a 5gb+ table, so instead, I shut down the slave server and copy the replicated post table to the slave and restart the slave SQL instance. I then run the repair on the old post table and figure out if any posts were lost and reimport them into the new post table. It's complex but it gets the job done
    wow you really deserve your nick name ! Good job!
    Giving up does not always means you are weak, it Simply means you are strong to let go!

  11. #11
    Join Date
    May 2001
    Location
    Dayton, Ohio
    Posts
    4,977

  12. #12
    Join Date
    Nov 2006
    Posts
    35
    I assume you have all servers connected via internal interface? Are your clients authenticating?

    I had that problem recently where they weren't. What fixed it is:

    skip-name-resolve

    in the my.cnf. Although the server was listed as an IP Address, mysql was still trying to resolve hostnames and it delayed connections which can make it look like a lot of other things.

    Or maybe you have some bad hardware?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •