Results 1 to 5 of 5
  1. #1
    Join Date
    Dec 2001
    Location
    MO
    Posts
    629

    openvz system crash every 24 hours

    Hey guys,

    I'm normally pretty good at troubleshooting system issues, and I have a great system admin for the things I can't do, but this is really strange and neither of us can pinpoint the problem.

    Basically, we have a node running openvz on CentOS 5.3 64-bit and it has been solid forever. And then 2-3 days ago we upgraded the kernel from 028stab062.3 to 028stab064.4 (I think that's right, not near the system right now).

    About 24-48 hours after the upgrade, the node crashed. The datacenter rebooted it and we checked the logs... nothing. So exactly 24 hours after the first crash, it happened again. I asked the datacenter to give me all of the information they had and I was told:

    "Looks like a kernel panic related to a crond process, something about __posix_lock_file_conf".

    The problem is I just can't pinpoint where this is coming from. There are no oops messages, no conflicts, nothing starting right before the crash, nothing. There is just simply nothing being logged about this.

    Anyway, I know this is a long-shot, but has anyone ever experienced a problem related to "posix_lock_file_conf"? I've done the obvious and reverted to the old kernel for now, and I have another admin who's going to look into it tomorrow for me, but I figured I'd check here too...

  2. #2
    Join Date
    Feb 2008
    Location
    Houston, Texas, USA
    Posts
    2,955
    Hi there,

    We haven't hit this bug since we haven't upgrade to that kernel. But check out the change logs for 028stab064.4 here:

    http://kb.parallels.com/en/6564

    Especially this section:

    A logic error was found in the do_setlk() function of the Linux kernel Network File System (NFS) implementation. If a signal interrupted a lock request, the local POSIX lock was incorrectly created. This could cause a denial of service on the NFS server if a file descriptor was closed before its corresponding lock request returned. (CVE-2008-4307, Important)
    And CVE-2008-4307 here:

    https://bugzilla.redhat.com/show_bug.cgi?id=456282

    Relevant section is:
    This is a problem with how the RHEL4/5 kernels handle POSIX locks. It is
    possible for an fcntl call to race with a close on the same file descriptor. A similar problem was fixed recently but this appears to be a separate issue.
    There are other changes that seem to be dealing with locking in the changelog.

    Good luck
    UNIXy - Fully Managed Servers and Clusters - Established in 2006
    [ cPanel Varnish Nginx Plugin ] - Enhance LiteSpeed and Apache Performance
    www.unixy.net - Los Angeles | Houston | Atlanta | Rotterdam
    Love to help pro bono (time permitting). joe > unixy.net

  3. #3
    Join Date
    Jun 2006
    Location
    NYC
    Posts
    1,446
    Step the kernel back, would be my recommendation.
    FiberPeer.Com | | REAL DDoS Protection | Cloud Hosting | VPS | Dedicated Servers | High Bandwidth Hosting | 1Gbps-10Gbps Unmetered
    FiberPeer DDoS Mitigation | ethProxy Upgraded! | 14-Years Experience | Emergency 24/7 Support
    Visit us @ www.fiberpeer.com

  4. #4
    Join Date
    Dec 2001
    Location
    MO
    Posts
    629
    Quote Originally Posted by UNIXy View Post
    Hi there,

    We haven't hit this bug since we haven't upgrade to that kernel. But check out the change logs for 028stab064.4 here:

    http://kb.parallels.com/en/6564

    Especially this section:

    And CVE-2008-4307 here:

    https://bugzilla.redhat.com/show_bug.cgi?id=456282

    Relevant section is:

    There are other changes that seem to be dealing with locking in the changelog.

    Good luck
    Thanks for that, it's the first time I've read about that bug and it definitely seems to be in the same ballpark. I guess we'll just play the waiting game and see if it happens again with the old kernel back in place.

  5. #5
    Join Date
    Jul 2009
    Posts
    178
    Solution is to destroy and recreate the node which has been upgraded.

Similar Threads

  1. System Admin / OpenVZ Help
    By Russ Foster in forum Systems Management Offers
    Replies: 0
    Last Post: 12-29-2008, 12:35 PM
  2. UK News - HSBC - Major System Crash!
    By WirralNet Matt in forum Web Hosting Lounge
    Replies: 4
    Last Post: 01-03-2005, 08:13 PM
  3. Server crash every few hours
    By WWWhost in forum Hosting Security and Technology
    Replies: 21
    Last Post: 12-16-2004, 01:19 PM
  4. system crash / need help!
    By zappp in forum Hosting Security and Technology
    Replies: 7
    Last Post: 11-30-2002, 10:14 AM
  5. Replies: 6
    Last Post: 11-30-2001, 08:10 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •