Results 1 to 11 of 11
  1. #1

    Tektonic VPS down 17 hours and counting

    207.210.77.190 has been down since Sat Apr 12 20:36:55 EDT 2008. That is 17 hours and counting. Virtuoso control panel does not work at https://207.210.77.190:4643

    Support finally came back with this:
    The following was found on your VPS:
    fl.jpg getcp index.php installed-packages.txt jj.pdf
    jj.pdf.1 john-1.7.2 m mysqlaccess.log sites w.jpg webscan
    webscan.tgz

    In addition, binaries have been replaced causing segmentation faults, indicative of a rootkit being installed. The binaries are required by Virtuozzo to start your VPS environment, so your VPS will not start unless you reinstall.

    We suggest doing this from your control panel, if you need any assistance, please let us know.

    As mentioned, I can not do a restore because I have no access to Virtuoso. I told them that 7 hours ago and got no response. If this were the first incident that would be fine, but look at the record:

    Previous event:
    Recovery Time: Thu Mar 27 22:03:33 EDT 2008
    Total Time on Error: 23 hours, 1 minute, 6 seconds

    Previous event:
    Recovery Time: Wed Mar 26 18:53:19 EDT 2008
    Total Time on Error: 3 hours, 56 minutes, 18 seconds

    Previous:
    Recovery Time: Fri Mar 21 01:19:59 EDT 2008
    Total Time on Error: 4 days, 4 hours, 49 minutes, 33 seconds

    Again:
    Recovery Time: Thu Feb 28 23:24:12 EST 2008
    Total Time on Error: 10 hours, 53 minutes, 21 seconds

    Again:
    Recovery Time: Tue Feb 05 22:32:22 EST 2008
    Total Time on Error: 3 hours, 39 minutes, 28 seconds

    Look at those numbers: 23 hours, 3 hours, 4 days, 4 hours, 10 hrs, 3 hrs.

    I'm sure that thewiseone (Matt Ayres) knows what he is doing but I'm not so sure about the other support staff.

  2. #2
    Join Date
    Aug 2003
    Location
    Chicago, IL USA
    Posts
    2,781
    Is this a managed VPS? Also, what were the reason for your VPS being down the other times.

    Also, I'm not sure how your VPS being down (your fault), entitles you to post a thread claiming Tektonic is down.

  3. #3

    Tektonic VPS Down

    Quote Originally Posted by daejuanj View Post
    Is this a managed VPS? Also, what were the reason for your VPS being down the other times.

    Also, I'm not sure how your VPS being down (your fault), entitles you to post a thread claiming Tektonic is down.
    1. The concern is the amount of time down.
    2. We have automatic software updating in place.
    3. The reason given the other times down was file system corruption, run fsck.

  4. #4
    Join Date
    Aug 2003
    Location
    Chicago, IL USA
    Posts
    2,781
    Quote Originally Posted by chiefwebofficer View Post
    1. The concern is the amount of time down.
    2. We have automatic software updating in place.
    3. The reason given the other times down was file system corruption, run fsck.
    I don't think you understand my post. I'm aware you've had downtime, but I'm unaware whose fault it was.

    1.) Yes, I know
    2.) Irrelevant, I asked if it was managed by Tektonic.
    3.) File-system of the root OS or your VPS?

  5. #5
    Join Date
    Apr 2008
    Location
    USA & Germany
    Posts
    193
    Virtuozzo uses a kind of chroot-filesystem. This can be access via host. Ask your vps-provider to make a backup of your vps-filesyslem.

  6. #6
    Not managed by Tektonic. However, quoting from their website: "We also ensure the host server maintains security, stability and excellent performance." The only software we have added to the base server they provide is Gallery2.

    I could not tell you whether the file system of the root OS was affected, or just my VPS. I can offer these messages from tech support if you can see something relevant that might answer your question:

    Thu, 27 Mar 2008 18:54:43 -0400 "The VPS appears to be corrupt. AT this point i would recommend a recover or a reinstall."

    Sun, 30 Mar 2008 11:09:23 -0500 "You are being sent this automated message to let you know the hardware node one of your VPS's are on is performing a filesystem check. Filesystem checks are required if the filesystem has errors or if a fsck has not been performed in 6 months."

  7. #7

    Virtuozzo uses a kind of chroot-filesystem

    Quote Originally Posted by nice-price View Post
    Virtuozzo uses a kind of chroot-filesystem. This can be access via host. Ask your vps-provider to make a backup of your vps-filesyslem.
    Thanks for your reply. I find it informative and helpful. We had performed a backup of the file system. They moved it to a VPS under HyperVM now.

  8. #8
    Join Date
    Jan 2004
    Location
    Pennsylvania
    Posts
    939
    I have performed a case report for this ticket and below are my findings.

    1. The ticket for this issue was opened on Sun Apr 13 2008 01:13AM. We cannot be aware of issues until they are reported.
    2. Support responded at Sun Apr 13 2008 01:49AM to let you know the VPS appeared compromised and research was being performed (I like this as I have beat it into supports heads to let customers know their issue is being worked on).
    3. At Sun Apr 13 2008 02:46AM support responded with the reply you quoted in your original post.
    4. At Sun Apr 13 2008 10:39AM you replied back stating you could not access VZPP.
    5. At Sun Apr 13 2008 02:10PM support responded that you had been moved to a new system that does not have VZPP, sent you the login details to the new control panel, and copied your Virtuozzo backup to your VPS's root directory.

    Notice was sent out regarding the migrations to the new servers. VZ21 (the host you were on) has 1kb inode/block sizes which made for a very long fsck. The notice informing of the move stated the reason for the migration was to be moved to a node with 4kb inode/block sizes for a 4x faster fsck. This is why the host node you were on was a good candidate for migration.

    What I imagine happened is that your VPS has been hacked for a very long time and once a VPS is hacked you have no real control over it, especially if it has a rootkit installed. I'd bet most of the outages were due to the hacker tools eating up your bandwidth and resources. What made the hack apparent was the migration was forced to stop your VPS on the old host node and start it up on the new. Since it wouldn't start our support was able to figure out you were hacked.

    All in all I'm personally happy with how the ticket was handled.

    daejuanj - Yes, he is on our lowest unmanaged (unmetered) plan.

  9. #9
    Join Date
    Aug 2003
    Location
    Chicago, IL USA
    Posts
    2,781
    Quote Originally Posted by TheWiseOne View Post
    I have performed a case report for this ticket and below are my findings.

    1. The ticket for this issue was opened on Sun Apr 13 2008 01:13AM. We cannot be aware of issues until they are reported.
    2. Support responded at Sun Apr 13 2008 01:49AM to let you know the VPS appeared compromised and research was being performed (I like this as I have beat it into supports heads to let customers know their issue is being worked on).
    3. At Sun Apr 13 2008 02:46AM support responded with the reply you quoted in your original post.
    4. At Sun Apr 13 2008 10:39AM you replied back stating you could not access VZPP.
    5. At Sun Apr 13 2008 02:10PM support responded that you had been moved to a new system that does not have VZPP, sent you the login details to the new control panel, and copied your Virtuozzo backup to your VPS's root directory.

    Notice was sent out regarding the migrations to the new servers. VZ21 (the host you were on) has 1kb inode/block sizes which made for a very long fsck. The notice informing of the move stated the reason for the migration was to be moved to a node with 4kb inode/block sizes for a 4x faster fsck. This is why the host node you were on was a good candidate for migration.

    What I imagine happened is that your VPS has been hacked for a very long time and once a VPS is hacked you have no real control over it, especially if it has a rootkit installed. I'd bet most of the outages were due to the hacker tools eating up your bandwidth and resources. What made the hack apparent was the migration was forced to stop your VPS on the old host node and start it up on the new. Since it wouldn't start our support was able to figure out you were hacked.

    All in all I'm personally happy with how the ticket was handled.

    daejuanj - Yes, he is on our lowest unmanaged (unmetered) plan.
    I knew there was something missing from his post.

  10. #10
    Join Date
    Jan 2007
    Location
    Miami Beach, FL, USA
    Posts
    764
    I wanted to ask that yesterday but got sidetracked. If it is unmanaged like you said, then the OP is responsible and this thread should not even exist.

  11. #11
    Join Date
    Jan 2004
    Location
    Pennsylvania
    Posts
    939
    Please, there is no need to say anything bad about Mark. I'm sure he was just frustrated at the time. He had a talk with one of our staff on the phone today and we are working with him to get him up and running again on an OS that will be more secure (he was still running Fedora Core 1, ouch!).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •