Web Hosting Talk : Web Hosting Main Forums : Programming Discussion : Help with Bash Monitoring Script...

[richardhay]

Good morning all.

While I know PHP, HTML, Java programming and can handle most linux tasks, I'm stuck on a bash script, and am hopeful for a little help.

Currently, we have two linked services (service1 and service2) running on one of our servers. Together, those provide access to an online conferencing system. Randomly and very infrequently, one of the services stop--usually service2--making the service completely unavailable. Since the services are linked, stopping and starting service1 (see below) restarts both.

Therefore, we set up a cron that runs a bash script every ten minutes to make sure that the service is running and, if it's not, attempts a restart. The script checks two things--the results of the "service status" command for each service and the availability and response of port 9000--the port that service1 uses to provide the conferencing access.

However, despite these checks, the service is still occasionally failing without being automatically restarted. So, my question to you is, based on the information above and the bash script below, can you think of anything that we can add to the bash script that would do a better job of monitoring the service and restarting it if necessary? Is there a third check that we could add that would further help catch outages? Finally, with the organization of the script, is the port check being run correctly?

The script is below... and thanks!


------
SERVICE1=`/sbin/service service1 status | grep -c "service1.*is running"`
SERVICE2=`/sbin/service service2 status | grep -c "service2.*is
running"`

(echo >/dev/tcp/localhost/9000) &>/dev/null
if [ $? -eq 0 ]; then
if [[ "$SERVICE2" == "0" || "$SERVICE2" == "0" ]]; then
/sbin/service service1 stop
sleep 10
/sbin/service service1 start
TO="XXX@xxxx.com"
echo "SERVICE `date`" | /bin/mail -s
"SERVER ALERT: RESTART" "$TO" -c "$CC"
fi
else
/sbin/service service1 stop
sleep 10
/sbin/service service1 start
TO="XXX@xxxx.com"
echo "PORT `date`" | /bin/mail -s "SERVER ALERT:
RESTART" "$TO" -c "$CC"

[foobic]

The logic seems a bit... strange, TBH. Indentation makes things a bit clearer:

SERVICE1=`/sbin/service service1 status | grep -c "service1.*is running"`
SERVICE2=`/sbin/service service2 status | grep -c "service2.*is running"`

(echo >/dev/tcp/localhost/9000) &>/dev/null
if [ $? -eq 0 ]; then
    if [[ "$SERVICE2" == "0" || "$SERVICE2" == "0" ]]; then
        /sbin/service service1 stop
        sleep 10
        /sbin/service service1 start
        TO="XXX@xxxx.com"
        echo "SERVICE `date`" | /bin/mail -s
        "SERVER ALERT: RESTART" "$TO" -c "$CC"
    fi
else
    /sbin/service service1 stop
    sleep 10
    /sbin/service service1 start
    TO="XXX@xxxx.com"
    echo "PORT `date`" | /bin/mail -s "SERVER ALERT:
    RESTART" "$TO" -c "$CC"

Looks like you're missing a "fi", and you only test service2 (or service2 ) if the echo test shows no error... What's the real aim here; to run all 3 tests and if any of them fails, run the restart procedure?

[richardhay]

Hi Chris.

Thanks so much for writing. The end goal is to basically test three things and, if any one of them fails, to stop the service, pause, and restart the service, and then send an email.

Would I be better off simplifying and setting it up like:

-----
SERVICE1=`/sbin/service service1 status | grep -c "service1.*is running"`
SERVICE2=`/sbin/service service2 status | grep -c "service2.*is running"`

(echo >/dev/tcp/localhost/9000) &>/dev/null

if [[ "$SERVICE2" == "0" || "$SERVICE2" == "0" || $? -eq 0 ]]; then
/sbin/service service1 stop
sleep 10
/sbin/service service1 start
TO="XXX@xxxx.com"
echo "PORT `date`" | /bin/mail -s "SERVER ALERT:
RESTART" "$TO" -c "$CC"
fi
-----

(By the way, the script had indentation, but I lost it in the paste. Sorry about that!)

Richard

[foobic]

That certainly looks better, but in bash the conditional operators are a bit weird. Not sure but I think you may need "-o" instead of "||", and watch out for string vs numeric comparisons. Personally I prefer the KISS principle, particularly for little scripts like this, eg.

RESULT="FAILED"
FAIL=0

SERVICE1=`/sbin/service service1 status | grep -c "service1.*is running"`
if [ $SERVICE1 -eq 0 ]; then
    RESULT="$RESULT 1 "
    FAIL=1
fi

SERVICE2=`/sbin/service service2 status | grep -c "service2.*is running"`
if [ $SERVICE2 -eq 0 ]; then
    RESULT="$RESULT 2 "
    FAIL=1
fi

(echo >/dev/tcp/localhost/9000) &>/dev/null
if [ $? -eq 0 ]; then
    RESULT="$RESULT 3 "
    FAIL=1
fi

if [ $FAIL -eq 1 ]; then
    /sbin/service service1 stop
    sleep 10
    /sbin/service service1 start
    TO="XXX@xxxx.com"
    echo "PORT `date`" | /bin/mail -s "SERVER ALERT: RESTART. $RESULT" "$TO" -c "$CC"
fi

[richardhay]

Wow Chris... thank you so much. I'm off to try this now.

Richard

[bdowne01]

Just wanted to toss in a word for "monit" here. It might be a little overkill for your precise need in this case, but it does exactly what you're trying to achieve with a shell script. I found it to be one of those things that I find many other uses for once I have it in place.