Results 1 to 24 of 24
  1. #1

    I need a linux script to control a runaway process...

    Do you know if this is possible, and how hard it would be to write?

    I need a script that does the following for linux:

    1. Checks for the existence of a specified process.
    2. If the process is found, has it been running for longer than 5 minutes at greater than 25% CPU usage?
    3. If so, then kill the process, write the event to a log file, and optionally send an email to admin.

    The script should accept input switches. For example:

    scriptname -p process_name -s 300 -u 25 -e admin@domain.com

    -p is the process name to monitor
    -s is the number of seconds the process must have been running before action is taken
    -u is the % of CPU that the process must be using before action is taken
    -e is the email address to receive event notification. Only send email if this switch is specified.

    The script should be executable via cron.

    Thanks!
    We are eNom PLATINUM PLUS resellers!
    Sign up today for an eNom.com reseller account with lowest possible pricing.
    * We provide support and service to over 4275 happy eNom domain name and SSL certificate resellers!

  2. #2
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,379
    i got a script that checks for processes and lists them like this

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 1 0.0 0.0 1416 452 ? S Aug25 0:07 init [3]
    root 2 0.0 0.0 0 0 ? SW Aug25 0:00 [keventd]
    root 3 0.0 0.0 0 0 ? SW Aug25 0:00 [kapm-idled]
    root 4 0.0 0.0 0 0 ? SWN Aug25 0:02 [ksoftirqd_CPU0]
    root 5 0.0 0.0 0 0 ? SW Aug25 1:33 [kswapd]
    root 6 0.0 0.0 0 0 ? SW Aug25 0:00 [kreclaimd]
    root 7 0.0 0.0 0 0 ? SW Aug25 0:00 [bdflush]
    root 8 0.0 0.0 0 0 ? SW Aug25 0:00 [kupdated]
    root 9 0.0 0.0 0 0 ? SW< Aug25 0:00 [mdrecoveryd]
    root 13 0.0 0.0 0 0 ? SW Aug25 0:58 [kjournald]
    root 89 0.0 0.0 0 0 ? SW Aug25 0:00 [khubd]
    root 184 0.0 0.0 0 0 ? SW Aug25 0:00 [kjournald]
    root 185 0.0 0.0 0 0 ? SW Aug25 1:04 [kjournald]
    root 829 0.0 0.1 1484 528 ? S Aug25 0:30 syslogd -m 0
    root 834 0.0 0.0 1992 432 ? S Aug25 0:00 klogd -2
    rpc 854 0.0 0.0 1564 440 ? S Aug25 0:00 portmap
    rpcuser 882 0.0 0.1 1608 596 ? S Aug25 0:00 rpc.statd
    root 979 0.0 0.0 0 0 ? SW Aug25 0:07 [rpciod]
    root 980 0.0 0.0 0 0 ? SW Aug25 0:00 [lockd]
    root 1037 0.0 0.1 2676 792 ? S Aug25 0:01 /usr/sbin/sshd
    root 1071 0.0 0.1 2276 808 ? S Aug25 0:21 xinetd -stayalive
    root 1146 0.0 0.2 5320 1488 ? S Aug25 0:07 sendmail: accepti
    root 1194 0.0 0.1 1596 600 ? S Aug25 0:00 crond
    root 2367 0.0 0.0 1396 360 tty1 S Aug25 0:00 /sbin/mingetty tt
    root 2368 0.0 0.0 1388 360 tty2 S Aug25 0:00 /sbin/mingetty tt
    root 2369 0.0 0.0 1388 360 tty3 S Aug25 0:00 /sbin/mingetty tt
    root 2373 0.0 0.0 1388 360 tty4 S Aug25 0:00 /sbin/mingetty tt
    root 2374 0.0 0.0 1388 360 tty5 S Aug25 0:00 /sbin/mingetty tt
    root 2375 0.0 0.0 1388 360 tty6 S Aug25 0:00 /sbin/mingetty tt
    root 16769 0.0 0.1 2264 628 ? S Aug25 0:00 /bin/sh /usr/bin/
    mysql 16794 0.0 2.9 38188 15064 ? S Aug25 0:11 /usr/sbin/mysqld
    mysql 16796 0.0 2.9 38188 15068 ? S Aug25 0:00 /usr/sbin/mysqld
    mysql 16797 0.0 2.9 38188 15068 ? S Aug25 0:24 /usr/sbin/mysqld
    root 637 0.0 0.3 5540 1824 ? S Aug26 0:01 /usr/bin/perl /us
    root 16720 0.0 0.7 77836 3760 ? S 13:46 0:00 /usr/sbin/httpd -
    mysql 24598 0.8 2.9 38188 15068 ? S 14:42 0:24 /usr/sbin/mysqld
    root 26042 0.0 0.2 3564 1228 ? S 14:52 0:00 /usr/sbin/sshd
    root 26057 0.0 0.2 2532 1144 pts/0 S 14:52 0:00 -bash
    mysql 27196 0.7 2.9 38188 15068 ? S 14:59 0:13 /usr/sbin/mysqld
    mysql 27199 0.8 2.9 38188 15068 ? S 14:59 0:14 /usr/sbin/mysqld
    mysql 27250 0.8 2.9 38188 15068 ? S 15:00 0:14 /usr/sbin/mysqld
    apache 27626 0.6 1.4 79940 7312 ? S 15:03 0:09 /usr/sbin/httpd -
    apache 27675 0.7 1.8 81988 9288 ? S 15:03 0:10 /usr/sbin/httpd -
    apache 27732 0.4 1.8 82188 9544 ? S 15:04 0:06 /usr/sbin/httpd -
    apache 27753 0.4 1.4 80176 7452 ? S 15:04 0:06 /usr/sbin/httpd -
    apache 27769 0.5 1.6 81284 8556 ? S 15:04 0:07 /usr/sbin/httpd -
    apache 27821 0.4 1.8 82420 9584 ? S 15:05 0:06 /usr/sbin/httpd -
    root 27846 0.0 0.4 6072 2508 ? S 15:05 0:00 sendmail: ./g7PJc
    apache 28000 0.6 1.3 79780 7020 ? S 15:06 0:07 /usr/sbin/httpd -
    apache 28007 0.8 1.6 81196 8588 ? S 15:06 0:10 /usr/sbin/httpd -
    apache 28012 0.5 1.7 81620 9008 ? S 15:06 0:06 /usr/sbin/httpd -
    apache 28013 0.5 1.3 79940 7056 ? S 15:06 0:07 /usr/sbin/httpd -
    apache 28065 0.5 1.6 81288 8440 ? S 15:07 0:06 /usr/sbin/httpd -
    apache 28091 0.4 1.3 79736 7032 ? S 15:07 0:05 /usr/sbin/httpd -
    apache 28118 0.7 1.6 81240 8360 ? S 15:07 0:08 /usr/sbin/httpd -
    mysql 28146 0.8 2.9 38188 15068 ? S 15:07 0:10 /usr/sbin/mysqld
    mysql 28147 0.8 2.9 38188 15068 ? S 15:07 0:10 /usr/sbin/mysqld
    apache 28312 0.5 1.6 81304 8600 ? S 15:09 0:06 /usr/sbin/httpd -
    apache 28404 0.6 1.7 81736 9076 ? S 15:09 0:07 /usr/sbin/httpd -
    apache 28409 0.6 1.6 81236 8580 ? S 15:09 0:06 /usr/sbin/httpd -
    apache 28729 0.3 1.3 79852 7004 ? S 15:11 0:03 /usr/sbin/httpd -
    apache 28841 0.5 1.3 80936 7116 ? S 15:12 0:04 /usr/sbin/httpd -
    apache 28910 0.9 1.8 82272 9284 ? S 15:12 0:08 /usr/sbin/httpd -
    apache 29022 0.6 1.3 79820 6948 ? S 15:13 0:05 /usr/sbin/httpd -
    apache 29193 0.2 1.5 80816 7904 ? S 15:15 0:02 /usr/sbin/httpd -
    apache 29318 0.4 1.3 79612 6852 ? S 15:16 0:03 /usr/sbin/httpd -
    apache 29447 0.7 1.8 82200 9552 ? S 15:17 0:04 /usr/sbin/httpd -
    apache 29458 0.3 1.2 79496 6524 ? S 15:17 0:02 /usr/sbin/httpd -
    apache 29684 0.6 1.7 81520 8732 ? S 15:18 0:03 /usr/sbin/httpd -
    apache 29857 0.7 1.4 79984 7200 ? S 15:20 0:03 /usr/sbin/httpd -
    apache 30034 0.2 1.1 78840 5972 ? S 15:21 0:01 /usr/sbin/httpd -
    apache 30103 0.4 1.3 79748 6752 ? S 15:22 0:01 /usr/sbin/httpd -
    2475 30126 0.0 0.2 2628 1448 ? SN 15:22 0:00 ftpd: 12.109.106.
    apache 30223 0.7 1.7 81744 8748 ? S 15:22 0:02 /usr/sbin/httpd -
    apache 30442 0.7 1.2 79600 6592 ? S 15:23 0:01 /usr/sbin/httpd -
    apache 30594 0.2 1.1 79344 6092 ? S 15:24 0:00 /usr/sbin/httpd -
    apache 30783 0.8 1.2 79384 6452 ? S 15:26 0:00 /usr/sbin/httpd -
    apache 30792 0.0 0.8 77992 4376 ? S 15:26 0:00 /usr/sbin/httpd -
    apache 30852 0.0 0.8 77940 4108 ? S 15:26 0:00 /usr/sbin/httpd -
    apache 30910 0.4 0.9 78296 4968 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30916 1.9 1.2 79520 6268 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30919 1.1 1.2 79236 6160 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30932 0.2 0.9 78172 4812 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30936 0.1 0.8 77976 4368 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30937 3.7 1.2 79756 6540 ? S 15:27 0:00 /usr/sbin/httpd -
    apache 30940 0.1 0.8 77952 4320 ? S 15:27 0:00 /usr/sbin/httpd -
    icompact 30962 8.4 0.0 0 0 ? Z 15:27 0:00 [ultimatebb.cgi <
    icompact 30980 0.0 0.9 5964 4764 ? R 15:27 0:00 /usr/bin/perl ult
    ksfba 30981 0.0 0.4 3436 2140 ? S 15:27 0:00 /usr/bin/perl ser
    ksfba 30983 0.0 0.1 2672 744 ? R 15:27 0:00 ps aux

    this what your lookin for?

  3. #3
    Join Date
    Mar 2002
    Location
    St. Louis, MO
    Posts
    1,379
    errr. though it doesnt control them, it just lists them, thats all. it lists other things besides that

  4. #4
    Originally posted by DigiCrime
    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    username 30983 0.0 0.1 2672 744 ? R 15:27 0:00 program_name

    this what your lookin for?
    I'm looking for something that will be able to check and see if the program_name process exists, and if it does.. kill it if it's been using more than XX% of the CPU for XX seconds.

    Can this be done?
    We are eNom PLATINUM PLUS resellers!
    Sign up today for an eNom.com reseller account with lowest possible pricing.
    * We provide support and service to over 4275 happy eNom domain name and SSL certificate resellers!

  5. #5
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,979
    Another idea would be to create a user just for running that process, then set limits for that user.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  6. #6
    Join Date
    Jul 2002
    Posts
    311
    Nice question, i too would like to know !!!

    any one has it ?

  7. #7
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    285
    Hey Guys,

    Try this script. Took me about 10 mins and I can't honestly say it will work perfectly but it may be of help.

    ---- START --

    #!/usr/bin/perl

    $process_name = $ARGV[1];
    $time = $ARGV[3];
    $cpu = $ARGV[5];
    $email = $ARGV[7];

    @process_list = `ps -aux -c`;
    foreach $line (@process_list) {
    chomp($line);
    @process_list_line = split(/\s+/, $line);
    $proc_id = $process_list_line[1];
    $time_run = $process_list_line[2];
    $cpu_run = $process_list_line[9];
    $process_run = $process_list_line[10];
    if($process_run =~ /$process_name/i) {
    ($hour,$min) = split(/\:/, $time_run);
    $time_run = ($min * 60) + ($hour * 60 * 60);
    if(($time_run > $time) and ($cpu_run > $cpu)) {
    # Send a -9 kill to the process
    system("kill -9 $proc_id");
    $killinfo = $line . "\n";
    }
    }
    }

    if(($killinfo) and ($email)) {
    open(MAIL, "|/usr/sbin/sendmail -t");
    print MAIL qq~To: $email
    From: root\@yourserver.com
    Subject: Killed a process
    I have killed the following processes.
    $killinfo

    See ya,
    Your Server
    ~;
    close(MAIL);
    }

    -- END ---

    chmod it with +x at least.

    Can write up a nice script with some proper code (ie. less messy) and test it for a small price if anyone is interested.

    Cheers,

    Stuart

  8. #8
    Join Date
    Jul 2002
    Posts
    311

    Thumbs up

    Thank You very much Stuart

    How many of you would put your faith in this script and say it is healthy enough to go on the servers.


    Regards/-

  9. #9
    Join Date
    Jul 2001
    Location
    UK
    Posts
    350
    Originally posted by Perlboy
    Can write up a nice script with some proper code (ie. less messy) and test it for a small price if anyone is interested.

    Cheers,

    Stuart
    I am interested.

  10. #10
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    285
    Ok,

    Anyone that's interested drop me an email in the next 24 hrs or so. Depending on how many people show interest will define the price for each. Basically, the more people willing to put in something the smaller the amount they'll have to put in (although, as it is you'd probably be looking at <=$20).

    processkill@perlboy.org

    Send the email there.

    Cheers,

    Stuart
    Biza•io - Digital Wealth Management
    The Blockchain Wallet - Multi Crypto Wallet

  11. #11
    I suspect if you made something like this available for sale for around $10 on the forums here you'd have some decent interest.

    Even better would be a script that looks for ANY process that's been running longer than XX seconds and is using XX CPU % and then kill it automatically (and email a notice to the server admin). Every host who has their own server could use such a "runaway killer" script like that.

    Of course, you'd want to include a config file that has a list of excluded processes...

    Cheers!

    Originally posted by Perlboy
    Ok,

    Anyone that's interested drop me an email in the next 24 hrs or so. Depending on how many people show interest will define the price for each. Basically, the more people willing to put in something the smaller the amount they'll have to put in (although, as it is you'd probably be looking at <=$20).

    processkill@perlboy.org

    Send the email there.

    Cheers,

    Stuart
    We are eNom PLATINUM PLUS resellers!
    Sign up today for an eNom.com reseller account with lowest possible pricing.
    * We provide support and service to over 4275 happy eNom domain name and SSL certificate resellers!

  12. #12
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    285
    Hey MrZippy,

    Sounds different to what you originally specified.

    You may want to check out

    http://www.webhostingtalk.com/showth...threadid=51590

    I have another tool similar to this one which handles processes based on CPU load with an excluded Process name list.

    Stuart
    Biza•io - Digital Wealth Management
    The Blockchain Wallet - Multi Crypto Wallet

  13. #13
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,979
    Perlboy, I would recommend:

    1. Using strict and maybe even do some validation on the input params.
    2. Try a kill -15 first, then sleep, if it's still running then send the terminator after it, kill -9.
    3. Use the absolute path to your ps command.
    4. Not offering to do it for money in this forum.
    5. Might use Getopt for params.

    My idea about pam limits wouldn't work for daemons, cgi, or forked procs probably.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  14. #14
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    285
    1. Using strict and maybe even do some validation on the input params.
    2. Try a kill -15 first, then sleep, if it's still running then send the terminator after it, kill -9.
    3. Use the absolute path to your ps command.
    5. Might use Getopt for params.
    All of these are applicable, if I wasn't just writing the script quickly because someone requested it from the forum. What's better than something for nothing?

    4. Not offering to do it for money in this forum.
    You're right and as such I asked people to contact me OFF forum keeping that info out of the forum itself. If anyone wants to make a formal request for such a script to be properly made I would advise them to do so in the "Job Offers & Requests" forum.

    Cheers,

    Stuart
    Biza•io - Digital Wealth Management
    The Blockchain Wallet - Multi Crypto Wallet

  15. #15
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,979
    Originally posted by Perlboy
    All of these are applicable, if I wasn't just writing the script quickly because someone requested it from the forum. What's better than something for nothing?
    I meant when you rewrite it of course.

    You're right and as such I asked people to contact me OFF forum keeping that info out of the forum itself. If anyone wants to make a formal request for such a script to be properly made I would advise them to do so in the "Job Offers & Requests" forum.
    Not any more proper than anyone else saying that they'll do web design, web hosting, or perl programming for a price. Just contact them off forum. Advertising is advertising.

    I'll rewrite any script to do the above that perlboy writes for a fee, for free.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  16. #16
    Join Date
    Jul 2001
    Location
    UK
    Posts
    350
    Originally posted by bitserve




    I'll rewrite any script to do the above that perlboy writes for a fee, for free.
    Why don't you do it? I need it and many others

  17. #17
    Join Date
    Dec 2001
    Location
    QLD, Australia
    Posts
    285
    I'll rewrite any script to do the above that perlboy writes for a fee, for free.
    If you wish I can return the favour and do a spell check on your FAQ located at http://www.bitserve.com/support.html ?

    That said, I'll take a pass on you rewriting my code.

    Thanks for the kind offer,

    Stuart
    Biza•io - Digital Wealth Management
    The Blockchain Wallet - Multi Crypto Wallet

  18. #18
    Originally posted by bitserve
    I'll rewrite any script to do the above that perlboy writes for a fee, for free.
    Go for it. I don't see why you would purposely try to take paying work from another member of the forums, though. That's kind of.. I don't know.. wrong.

    But if you really insist on doing this script for free.. go for it. I'm sure whatever you create will be useful and wanted by many.

    To anyone else.. buyer beware. You get what you pay for. If you want a script that's done right, I highly recommend you research who is writing the script first. Free doesn't always equal better.

    That being said, I look forward to seeing the free script.
    We are eNom PLATINUM PLUS resellers!
    Sign up today for an eNom.com reseller account with lowest possible pricing.
    * We provide support and service to over 4275 happy eNom domain name and SSL certificate resellers!

  19. #19
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,979
    Originally posted by Perlboy
    If you wish I can return the favour and do a spell check on your FAQ located at http://www.bitserve.com/support.html ?

    That said, I'll take a pass on you rewriting my code.

    Thanks for the kind offer,
    Code:
    $rewrite = 0;
    if ($rewrite == 1) {
      $spellcheck = 1;
    }
    (Code for illustrative purposes only.)

    How does that returning a favor work if you disallow the favor that is being returned?

    Anyway, whatever. I'm sorry that I hurt your ego enough for you to visit a site that I may maintain in order to find flaws so that you might feel better about your flaws.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  20. #20
    I bet I can piss further than BOTH of you!
    We are eNom PLATINUM PLUS resellers!
    Sign up today for an eNom.com reseller account with lowest possible pricing.
    * We provide support and service to over 4275 happy eNom domain name and SSL certificate resellers!

  21. #21
    Join Date
    Nov 2001
    Location
    Ann Arbor, MI
    Posts
    2,979
    The way it works is that someone pays perlboy money for a script to do the above. I rewrite it (write a script that does the same thing, only with at least one better feature) and give it to everyone else.

    That way you get to pay for something that everyone else gets a better version of for free.

    My reason was so that shameless advertising in this thread wouldn't prove to be profitable. If you feel that it's not a worthy cause, I'd be happy to discuss it off forum.
    -Mark Adams
    www.bitserve.com - Secure Michigan web hosting for your business.
    Only host still offering a full money back uptime guarantee and prorated refunds.
    Offering advanced server management and security incident response!

  22. #22
    Join Date
    Jul 2001
    Location
    UK
    Posts
    350
    Perlboy

    any news regarding the script.

  23. #23
    Join Date
    Jul 2002
    Posts
    311

    *

    No body seems to be answering
    common guys, what's wrong with your spirits ?

    i am also looking forward for the Script !!!!

    Cheers...

  24. #24
    Join Date
    Nov 2000
    Location
    Tasmania, Australia.
    Posts
    3,590
    Originally posted by oc3
    No body seems to be answering
    Stuart's away right now though, so I have left him a message. You might want to post something in the Related Offers and Requests forum though.......

    Cheers

    Gary

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •