Results 1 to 16 of 16
  1. #1
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687

    * Howto: Automatically verify crond is running and restart if not

    This one's pretty simple really, but given the fact that I'm constantly running into crond stops with CPanel servers (no clue why, or where, but it's happening), I'll post this here. If it helps someone, then great, if not, hey, no worries
    Firstly, login as root to your server through ssh, however you usually do .

    Next, the script (we'll call it /bin/croncheck.sh). Use whatever editor you choose to create the file.

    Code:
    #!/bin/bash
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    mail=/bin/mail 
    sysadmin=YOUREMAILHERE
    cronfile=/tmp/cron.txt
    logfile=/var/log/croncheck.log
    echo "$LOGDATE - Cron Check Service starting up " >> $logfile
    function stopcron
    {
    /sbin/service crond stop
    }
    function startcron
    {
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    stopcron
    echo "Cron Service Down, attempting to restart on $DATE" >> $cronfile
    cat $cronfile | $mail -s 'Cron Restart' $sysadmin
    /sbin/service crond start
    rm $cronfile
    }
    function checkpid
    {
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    cronfile=/var/run/crond.pid
    #get the pid, if it exists
    if [ ! -f $cronfile ];then
    echo "$LOGDATE - Dead PID" >> $logfile
    startcron
    fi
    thispid=`cat $cronfile`
    if [ ! -d /proc/$thispid ];then
    #startcron
    echo "$LOGDATE - Cron Stopped. Restarting" >> $logfile
    startcron
    else
    echo "$LOGDATE - Normal Cron Running" >> $logfile
    fi
    
    }
    function quit {
    exit
    }
    function hello {
    echo $1
    }
     COUNTER=0
    while [  $COUNTER -lt 10 ]; do
    checkpid
    sleep 30s
     done
    Simple, right? It actually is. Make sure the script is executable
    Code:
    chmod u+rxw /bin/croncheck.sh
    And add this to /etc/rc.local
    Code:
    /bin/sh /usr/bin/nohup /bin/croncheck.sh &
    To run without having to restart the server, simply type
    Code:
    /bin/sh /usr/bin/nohup /bin/croncheck.sh  >> /dev/null &
    and you're set.

    This code will email you upon failure, and attempt to start the crond service. Occasionally it'll not be able to do so, but at least you'll know when it's not able to do so and you can restart it yourself

    Enjoy
    Last edited by whmcsguru; 05-30-2005 at 06:24 PM.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  2. #2
    Join Date
    Jun 2003
    Posts
    961
    what about this scenario:
    between 2 runs of checkpid, cron gets stopped, some other process gets started, due to some randomization features (random pids), that new process gets the old pid from our cron process, the one in /var/run/crond.pid, i guess the script will fail then, right?
    maybe check if /proc/pid/exe points to /usr/sbin/cron or if cmdline does match "/usr/sbin/cron" ?

  3. #3
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    The likelyhood of a pid being reused in the (literal) milliseconds it takes from getting pid to getting link is about 0.000001%
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  4. #4
    Join Date
    Jun 2003
    Posts
    961
    i am talking about this part
    Code:
    while [  $COUNTER -lt 10 ]; do
    checkpid
    sleep 30s
    and not this one
    Code:
    thispid=`cat $cronfile`
    if [ ! -d /proc/$thispid ];then
    during that 30s sleep, crond might get killed, the pid file still has its old pid, cause it didnt get wiped, /var/run/crond.pid does still exist, 30s is plenty of time to start a few new processes, one of those might be running with the old crond pid

  5. #5
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Again, this isn't an issue.
    checkpid does everything in and of itself. There is no 30s delay between getting the pid and verifying the pid. It's milliseconds at best.
    checkpid is called every 30s, and checkpid verifies IN THAT INSTANCE that the pid is valid, not 30s later.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  6. #6
    Join Date
    Jun 2003
    Posts
    961
    so lets see,
    its 11:00:00 and we run the script, checkpid finds valid crond pid (1234) in /var/run/crond.pid and /proc/1234 does exist too, so we sleep 30sec now till next checkpid run
    11:00:05, crond gets killed, not a clean shutdown so /var/run/crond.pid does still exist and its content is 1234
    11:00:10 we start process XYZ, somehow it gets pid 1234
    11:00:30 checkpid runs again, checks still existing /var/run/crond.pid, reads crond pid from it (=1234) and checks if /proc/1234 does exist, since new process runs with pid 1234 the /proc dir does exist, checkpid assumes crond is still running while its not, just a new process running with old crond pid

    not an issue?

  7. #7
    Join Date
    Apr 2005
    Location
    silicon and earthquakes
    Posts
    255
    Originally posted by sehe
    not an issue?
    You can always add:
    Code:
    grep -q cron /proc/$thispid/cmdline
    if [ $? != 0 ]
    then
      echo "Other process"
    fi

  8. #8
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Ok, modified it up a bit (was sending out a TON of spam previously), new code is here
    Code:
    #automated Crond check script. Will restart Cron when it's down
    #script runs every 30 seconds, checks for pid and  pid file, then if not found
    #restarts cron
    #!/bin/bash
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    mail=/bin/mail
    [email protected]
    cronfile=/tmp/cron.txt
    logfile=/var/log/croncheck.log
    echo "$LOGDATE - Cron Check Service starting up " >> $logfile
    function stopcron
    {
    /sbin/service crond stop
    }
    function startcron
    {
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    stopcron
    echo "Cron Service Down, attempting to restart on $DATE" >> $cronfile
    cat $cronfile | $mail -s 'Cron Restart' $sysadmin
    /sbin/service crond stop
    /sbin/service crond start
    rm $cronfile
    }
    function checkpid
    {
    DATE=`date "+%m%d%y [%k:%M]"`
    LOGDATE=`date "+%m-%d-%y [%k:%M:%S]"`
    # ps xua | grep -q crond
    /sbin/service crond status |grep "is running..."
    if [ $? != 0 ]
    then
    echo "$LOGDATE - No Cron Running" >> $logfile
    startcron
    else
    echo "$LOGDATE - Normal Cron Running" >> $logfile
    fi
    }
    function quit {
     function quit {
    exit
    } 
     COUNTER=0
    while [  $COUNTER -lt 10 ]; do
    checkpid
    sleep 30s
    done
    This also solves the PID issue. So far this hasn't let me down, and it's restarted it a couple times, so I know it works
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  9. #9
    Join Date
    Mar 2002
    Location
    Melbourne
    Posts
    217
    so when the server is restarted, this script would start itself too?

  10. #10
    Join Date
    Mar 2006
    Posts
    418
    Quote Originally Posted by conanqtran
    so when the server is restarted, this script would start itself too?
    if you add the line in the original post to /etc/rc.local then yes, it'll get started on reboot

  11. #11
    Join Date
    May 2006
    Posts
    249
    so it works fine? or anyone have problem with that code?

  12. #12
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Quote Originally Posted by LowAsYou
    so it works fine? or anyone have problem with that code?
    This works just fine
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  13. #13
    Join Date
    Sep 2003
    Location
    India
    Posts
    127
    i got this error..

    ./croncheck.sh: line 3: found: command not found
    ./croncheck.sh: line 51: syntax error: unexpected end of file
    Freelance Web Designer & Web Host
    Dotcommakers- Animated banners, Website Templates, WordPress Themes - me [at] rakesh.in
    The Reliable Host - Shared & Reseller hosting
    End user support | CloudFlare | SEO Tools | Daily backups on 2 locations

  14. #14
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Use the original version. There appears to be something wrong with the second one (probably an updated system thing), but the original works like a charm.

    It's been like 5 years since I wrote this, so I may not remember the original context in which I did.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  15. #15
    Join Date
    Sep 2003
    Location
    India
    Posts
    127
    yes first version works

    but just one problem.. its not being finished.. when i tried check it manually

    it restarted crond service but it just stop by showing this line

    cat: /var/run/crond.pid: No such file or directory

    it just didnt come to prompt.. you know
    Freelance Web Designer & Web Host
    Dotcommakers- Animated banners, Website Templates, WordPress Themes - me [at] rakesh.in
    The Reliable Host - Shared & Reseller hosting
    End user support | CloudFlare | SEO Tools | Daily backups on 2 locations

  16. #16
    Wouldn't it be better to figure out why crond keeps stopping?
    Adam McMaster

    Valcato Hosting Over nine years of great service
    Shared, VPS and dedicated servers.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •