Results 1 to 27 of 27
  1. #1

    Reboot System - we develop ,Please provide your opinion

    Dear WHTers ,

    We are working this days on finishing up of our new Reboot System - server monitore.
    This reboot system will allow server owners and Server Resellers to have there servers monitored 24x7 without any human touch .

    Here is the system information:
    "Server Monitor" monitors your server's ports response constantly. For each monitored port, you can choose how Server Monitor will act when the port is not responsive:
    1. Run a custom command on the server using SSH. For example, when the Apache HTTP demon is down (port 80 is not responsive), start it again.
    2. Send an E-Mail to the server's DC, asking to reboot the server. For example, when the SSH demon is down.


    "Server Monitor" has two types of consoles:
    · Reseller console - Displays information about the servers` reachability, their ports` responsiveness and enables the reseller to do the following actions:
    o Add\Remove\Edit servers
    o Add\Remove ports to each server
    o Add\Remove\Edit DCs
    · Client console - Displays information about the client’s servers` reachability, their ports` responsiveness and enables the reseller to do the following actions:
    o Edit client’s server details
    o Add\Remove ports to each of the client’s servers


    Please provide your opinions/suggestions.
    and if you think this kind of system will be usable and for what range of people.

    Thank you.

  2. #2
    Join Date
    Sep 2004
    Location
    Uk
    Posts
    422
    Sounds really good.
    Only concern i have is the use of login passwords.
    How will it store the root password for each server?
    Which would be required to restart apache etc.

  3. #3
    Originally posted by fusioncroc
    Sounds really good.
    Only concern i have is the use of login passwords.
    How will it store the root password for each server?
    Which would be required to restart apache etc.
    It will be stored using an encrypted format inside the system DB (server side).

  4. #4
    Join Date
    Sep 2004
    Location
    Uk
    Posts
    422
    Sounds even better then.

    I may even be interested in this.
    Contact me on MSN,AIM or YIM.

  5. #5
    Join Date
    Jan 2004
    Location
    North Yorkshire, UK
    Posts
    4,163

  6. #6
    Originally posted by fusioncroc
    Sounds even better then.

    I may even be interested in this.
    Contact me on MSN,AIM or YIM.
    We'll notify you when we'll get online.
    It'll run on our main server and we'll also offer bigger quantity of accounts in lower fee for server resselers like you.

  7. #7
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    A couple of things:

    Instead of passwords, use DSA keys to get in and out of the system, always a much more secure way of doing things.

    You'll need to develop some sort of hardware interface, otherwise you won't truly be able to call this "remote reboot", as most servers that need a "remote reboot" aren't online and can't be connected to, which means the only way to do this is hardware

    How will you be monitoring these ports?
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  8. #8
    Originally posted by RazorBlue - Dan
    A plugin with the APC API would be nice too for those companies who offer APC reboot ports with their machines ...

    We'll do that.

  9. #9
    Originally posted by linux-tech
    A couple of things:

    Instead of passwords, use DSA keys to get in and out of the system, always a much more secure way of doing things.

    You'll need to develop some sort of hardware interface, otherwise you won't truly be able to call this "remote reboot", as most servers that need a "remote reboot" aren't online and can't be connected to, which means the only way to do this is hardware

    How will you be monitoring these ports?
    Servers can be rebooted by sending a reboot request E-Mail to the DC.
    We are monitoring ports by trying to open a connection to the and looking at the response (using sockets).

  10. #10
    Join Date
    Sep 2004
    Location
    Uk
    Posts
    422
    XpServ i believe the points LinuxTech made were very important.

  11. #11
    Join Date
    Sep 2004
    Location
    Uk
    Posts
    422
    Well, especially about the DSA keys.

  12. #12
    Originally posted by fusioncroc
    XpServ i believe the points LinuxTech made were very important.

    Sure , Will do.

  13. #13
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Servers can be rebooted by sending a reboot request E-Mail to the DC.
    Kind of takes the APC out of things in that case, then, as in most cases (where this would truly be useful) the server will be nonresponsive, and you have to go through this. APC is especially good in this case, which is why some sort of HW interface is needed.

    We are monitoring ports by trying to open a connection to the and looking at the response (using sockets).
    That doesn't necessarily answer the question. You can use a variety of methods using "sockets", such as ICMP, UDP, TCP, etc. If you're going to design something like this, you need to do this properly.
    A> open a socket to the system (ANYTHING but ICMP should work. ICMP is bad for anything, especially since those of us who are rather paranoid tend to disable ICMP right out of the box)
    B> Get a response
    C> Analyze that response
    D> Time that response

    If C> isn't correct, then you need to send an alert to the admin
    If D> takes too long, then you need to send an alert to the admin

    Forget doing this automatically, this is why human intervention is SOOOOOOO important here. If you do this automatically, you're asking for problems. An example of this:

    Say the response takes too long to get, because the server is loaded down. A competent admin can tell this right off the bat. However, an "automated" system won't. So, what does it do? It restarts the server (httpd, etc). Ok, that's good, but that ADDS to the system load itself, and causes more problems.

    Say, as well, the server load is too high. Great, so, you need to do something about it. Rather than simply reboot (which is what a number of them do), the admin needs to be notified so that he/she/it can resolve the situation properly.

    Taking into account the last scenario, it's entirely possible to attack a server so that it goes into a loaded down state (trust me, it's not too hard to do), then the system freaks out, does it's "automatic" thing, and 5 minutes later the process starts again. See the point there? It's entirely possible to totally neutralize a server this way.

    Take into account, the fact that the end server is on a DC that's, well, the network is shoddy, and it goes down for an hour or two every once in a while. A smart admin can compensate for this, and will realize that this is the case. An "automated" system will not.

    Sometimes, network connections can be bad, which will cause an invariably bad response. You can't alert (or take action) on ALL failures, seriously, that'd be bad, bad, bad and would only result in poor performance.

    In addition, you can't run this from just one location. You need to run this from multiple locations, outside of said DC for best results. Otherwise, your results can't be trusted. The reasoning is quite simple really:

    DC networks go down, it's a fact of life. When these go down, the outside world can't connect, but the inside world CAN. So, you're reporting a false positive, instead of a negative which it should be.

    There's a LOT to take into consideration when designing something like this. You can't just say "hey, we're doing this", and not look at what's going on, and the responses. Hell, I've been in the process of writing the same type of application for 3 years, and, eventually it'll get released, but not until it's done and tested, and ready

    Good luck.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  14. #14
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    Sorry for the double.
    Another thing you may want to consider is allowing individuals to access their server stats remotely, on their page, customized. It's not that hard to do, and can make for happy people
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  15. #15
    linux-tech:

    What we are trying to make, is a system that will increase the server uptime by starting services that went down or rebooting servers that are not responsive.

    You are talking about a very different system that monitors the server closely, a behavieor which is very problematic to implement.

    I believe that many server owners and resellers really need a system that will send a reboot request E-Mail to the DC when their server requires a hard reboot and they are asleep. They need a system that will start thier httpd\mysql when it went down or even reboot the server using APC.

    What can we do ? we have to sleep at night when our servers keep running and might stop functioning propertly.

    What do you guys think ?

  16. #16
    Join Date
    Jul 2003
    Location
    Connecticut
    Posts
    3,038
    Alot of DC's do not use email as their method of support. Which would relegate this to a restart script for your services. Not a bad idea all in all though. Keep us posted.

  17. #17
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    What we are trying to make, is a system that will increase the server uptime by starting services that went down or rebooting servers that are not responsive.
    No, you're talking about the same thing here, an application that "monitors" the server, yet doesn't take proper response:
    For each monitored port, you can choose how Server Monitor will act when the port is not responsive:
    Automatically doing something upon failure is bad. That's the first way to open yourself up for attack. Then, all someone has to do is attack your server so well it's not responding, and voilla, you're screwed again.
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  18. #18
    linux-tech:

    On the contrary, it makes you more attack proof.
    What the system does is to get dead things alive. When a server was attacked and it's http demon went down, the system will run it again. When a server was attacked got dead, the system will get it restarted (by E-Mail to the DC or APC API).

    This system is not meant to monitoring every parameter of the server and fixing every problem. It's meant to keep your server and services up, the most important thing for most of us.

    If need want fancy graphs and a thousand parameters to set, you'll need to get some other system.

    I appriciate your reponse.
    10x,
    Matan

  19. #19
    Originally posted by RazorBlue - Dan
    A plugin with the APC API would be nice too for those companies who offer APC reboot ports with their machines ...
    Dan, do you have a link to such API documentation?

    Thanks
    ^_^

  20. #20
    Join Date
    Oct 2000
    Posts
    1,654
    Would also be interested in seeing a plugin for BayTech remote reboot units.
    [QuickPacket™] [AS46261]
    Located in Atlanta, GA and Los Angeles, CA
    Dedicated Servers, KVM, Xen & OpenVZ VPS, Co-location, R1Soft Data Backup, Shared & Reseller Hosting

  21. #21
    Join Date
    Sep 2002
    Location
    Top Secret
    Posts
    11,687
    On the contrary, it makes you more attack proof.
    What the system does is to get dead things alive. When a server was attacked and it's http demon went down, the system will run it again. When a server was attacked got dead, the system will get it restarted (by E-Mail to the DC or APC API).
    Again, not so. Think this through here. Please, understand, the last thing I'm after doing here is attacking what you're doing. It's a GOOD thing. However, there is a very serious flaw in your thinking.

    Automated systems are not humans, they're automated, the responses are limited to what you tell them to do. So, let's say I load a system down using a php script (which, btw is VERY easy to do), or attack it somehow, how is the system going to know this? It's not! It will do what it's trained to do, restart the http server, even though the server is running just fine. Now, what's that going to do? Solve the problem? No, it's going to add more of a problem to the problem.

    Let's take that one step further, shall we?
    Let's say your system is automatically configured to reboot the server every time the load is above X, or certain number of servers don't respond. Well, you just gave me a hole in which to exploit the server. Believe it or not, it IS possible to do this. Not, of course, that I would, but it is very possible to do.

    At the end of the day, automated responses for servers and services like this are useless. Usually, there's some underlying problem that stops the server from starting, or, the server is just under attack. Either way, by continually allowing services to be "automatically" restarted, you're posing a great risk to the security of the server itself, which, inevitably, will be exploited
    WHMCS Guru - WHMCS addons, management, support and more.
    WHMCS Notifications Extended - Add slack, hipchat, SMS, pushover to WHMCS !!
    Always looking for Linux, WHMCS, Support Desk work. PM for details

  22. #22
    linux-tech:

    It doesn't matter why the server is not responding for the monitoring system. The monitoring system restarts/starts a service or reboots the server, sends an E-Mail to the owner and logs everything. Then, the server's admin can check why it got unresponsive and fix it. If it was attacked, got unresponsive and the monitoring system rebooted it, E-Mailed the admin and logged the details while the admin was sleeping, It's a good thing. Because, it might get the server out of the attack. And even if it didn't, it logged it and notified the admin.

    The most important thing that the system does, is to get the server and its services alive when they are not, and you can't argue with that fact. Ofcourse that an attack can take down the server again and again, but the system is not ment to be a shiled against attacks. It notifies the admin and he handles such thing.

  23. #23
    Join Date
    Jul 2002
    Location
    London, United Kingdom
    Posts
    4,362
    What we are trying to make, is a system that will increase the server uptime by starting services that went down or rebooting servers that are not responsive.
    Actually it's likely to decrease uptime not increase it, if it's contantly restarting services, especially if you ignore the items linux-tech pointed out to you.

    How will you know the issue's not with *your* connectivity rather than the end server(s) ?

    Have you built in a system to allow for regular "maintenance" - like backup timings where the server may be less responsive, or yum updates, where ftp/smtp connections are slower or temporariliy stopped ?

    You can cause some *serious* damage to the setup and stability to a box but just deciding to restart services, what manual check for load/config will you be doing each time before issuing the restart ssh commands ?

    How will the DC authenticate your reboot request. No properly run datacentre would take the word of a 3rd party email to get a server rebooted.

    What can we do ? we have to sleep at night when our servers keep running and might stop functioning propertly.
    This is why business that are serious about their hosting go with organisations that have 24/7 access to trained server admins - people that would know how to investigate why email isn;t arriveing - and restarting exim is *rarely* the answer - more important would be knowing why it stopped !

    And if your server is misbehaving and others are relying on it for their business, and you dont have 24/7 cover then you dont get to sleep ! We've all been there ...

    Good luck with your endeavour. Maybe i'm getting old and cynical - but your target market are the hosts who don't have the necessary staff/skills to monitor and manage servers properly - and they're the ones that most need to be learn some admin skills and write a few cron jobs before graduating to nagios etc ...
    Rob Golding Astutium Ltd - UK based ICANN Accredited Domain Registrar - proud to accept BitCoins
    Buying Web Hosts and Domain Registrars Today @ hostacquisitions.co.uk
    UK Web Hosting | UK VPS | UK Dedicated Servers | ADSL/FTTC | Backup/DR | Cloud
    UK Colocation | Reseller Accounts | IPv6 Transit | Secondary MX | DNS | WHMCS Modules

  24. #24
    Join Date
    Apr 2005
    Location
    Atlanta, Georgia
    Posts
    520
    glad to see someone else pointed out n agios / netsaint... why not just use it?

    It already does all the things you are talking about, and does them very well.

  25. #25
    othellotech:

    How will you know the issue's not with *your* connectivity rather than the end server(s) ?
    It pings some major servers (Google, Yahoo...).

    Have you built in a system to allow for regular "maintenance" - like backup timings where the server may be less responsive, or yum updates, where ftp/smtp connections are slower or temporariliy stopped ?
    Ofcourse.

    what manual check for load/config will you be doing each time before issuing the restart ssh commands ?
    We don't do manual checks. The whole point is to make something automatic. We restart stuff that are not responding, which means that they are useless, so it will cause no harm and increase the uptime.

    How will the DC authenticate your reboot request. No properly run datacentre would take the word of a 3rd party email to get a server rebooted.
    We'll have each user's SMTP details, so we'll send it from his E-Mail address.

    This is why business that are serious about their hosting go with organisations that have 24/7 access to trained server admins
    This system is ment for those who have smaller server and smaller budget. Those who can't pay anyone to keep their server alive while they are sleeping.

    I apriciate your response.

  26. #26
    Any other suggestions ?

  27. #27
    I bet you guys have more to say about it.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •