Results 1 to 30 of 30
  1. #1
    Join Date
    Nov 2006
    Posts
    68

    My rackspace experience

    We are a small hosting company and decided to collaborate with rackspace for 2 new servers and 1 dedicated firewall. We have quite a good server configuration:

    DELL PowerEdge 2850 Linux
    Dual 3.2 GHz Intel Xeon
    4 GB RAM
    RAID 1
    2 x 146GB SCA Ultra 320 10K RPM SCSI Drive

    We are statisfied of the deal we got with them even if a bit expensive but everbody knows rackspace... So far we have few complains about the setup of those servers:

    - They knew (while negotiating with the sales) the number of virtual hosts we would have on those servers but the server setup has been done in their standard way not taking into consideration that one of the host would get about 500 domains... so they had to recompile apache (+ other software) multiple times because of different parameters that change regarding the number of domains you host (example: open files...). They did it without any discussion but just adding: "normally this is not a free service"... But they still did it free of charge. Of course we wont get the automatic patching because of the software recompilation.

    - Those last days we had some troubles with apache on one of the boxes. We have been told that we should think migrating about 200 domains to another box, which means that a server could handle about 300 domain names while the server was idle 80% of the time... Kind of amazing when the crashes happened during night time (05 to 07 am) and considering the fact that during day time all was working smoothly. We simply could not accept to host so less domains as 300 on such expensive boxes (even if we can afford it lets be realistic). They finally found out an apache configuration issue... They eventually fixed it and we were satisfied (what else could we ask) ?

    - Few days later (maybe 1 or 2), apache crashed again and it was due - this time - to an exploit of a vulnerable script hosted by one of our clients... You know this kind of software they setup through the application vault...? And here rackspace really did not react the way they should (just let me know if you think that I am wrong):

    - "You must update your web sites". Correct but those applications are setup through the latest release of the application vault so updated already.

    - "There is no way to find the vulnerable scripts". On this point I have my doubts: we host some other boxes at another datacenter (alabanza.com) and they usually report us by email and eventualy lock down (for security purposes) vulnerable scripts hosted on our servers.

    - We have been asking to configure the firewall properly, after all they are the experts but it seems that they wait for us to know what to do and no proposition are coming up.

    - For the scripts why not proposing us mod_security ? Isnt it a layer of security that will help prevent those kind of incidents ?

    Until now we are satisfied of rackspace but as far as security is concerned I am not sure that they are top notch. I am still negotiating with our account manager for them to come to us with solutions, propositions, scenarios but until now I did not see anything...

    I ll keep you updated.

  2. #2
    Keep us informed about your further actions.

  3. #3
    Join Date
    Nov 2006
    Posts
    68
    I talked to my account manager for about 1 hour on the phone today and he really shows that he s willing to help the situation. He ll be asking a senior engineer to review our configuration...

    Lets see...

  4. #4
    Join Date
    Nov 2006
    Posts
    68
    Regarding mod_security, this is the answer of rackspace: "As this is a Plesk server, updates are done using packages provided by SW-Soft, the vendor of Plesk. Rackspace is not responsible for any vulnerabilities in the bundled PHP applications. Furthermore, mod_security requires a high degree of customization to ensure that it does not break your existing sites. Being an add-on module, mod_security falls outside the realm of support by Rackspace."

  5. #5
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,571
    Quote Originally Posted by sunray69
    Regarding mod_security, this is the answer of rackspace: "[I]mod_security falls outside the realm of support by Rackspace."I]"
    Outside the realm of free support, or both free and paid support? Just curious. I'm also a little dismayed that they will not accept responsibility (at least security-wise) for applications they directly sell (plesk, ect). Bundled or not, it's something they sell you, and if it's vulnerable out of the box, well..there you go.
    Last edited by FastServ; 11-09-2006 at 08:47 AM.
    Fast Serv Networks, LLC | AS29889 | Fully Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  6. #6
    Join Date
    Nov 2006
    Posts
    68
    Outside the free support for sure... Now if we request to pay them for this service I guess it could be possible.

    Thats kind of funny because I specified them on the phone that I did not need a company like rackspace for restarting apache but for some pro-active tasks which I am afraid they wont provide in any case. They were kind of suprised when I asked them for those pro-active jobs, like I was the first one to ask them... Am I really the first client to ask them this kind of things ??

    As well I have been told that we have too many virtual hosts on the server but what the hell is the point? You can have vulnerable scripts with 10 virtual hosts as well as with 600. Yes I understand that their task is harder when the client reaches 500 to 600 virtual hosts / server. Every datacenter would dream to manage servers with only 10 static pages !

  7. #7
    Join Date
    Mar 2003
    Location
    WebHostingTalk
    Posts
    16,947
    Not a good answer there from Rackspace.

    I thought they are managed? This is not good... Better decide now...
    Specially 4 You
    .
    JoneSolutions.Com ( Jones.Solutions ) is on the net 24/7 providing stable and reliable web hosting solutions and services since 2001

  8. #8
    Join Date
    Nov 2006
    Posts
    68
    I guess that we just have to wait that apache crashes again for them to simply restart it and wait until it crashes again... Is it what we have to do ? I recognize that it does not crash often (until now) but our job as shared hosting provider isnt it to prevent those kind of events ? I personnally thought it was also rackspace job... I might be mistaken.

    We already think to hire a management company regarding hardning the servers. When they learned we were at rackspace they clearly told us that it is within their scope and it should be included inside their services.

    Now I simply wonder to know if they dont wanna do it or they simply do not know the "how to" do it...

  9. #9
    Join Date
    Nov 2003
    Location
    Newport Beach, CA
    Posts
    2,920
    I do understand their position on the scripts not being their problem.

    Plesk comes with scripts to install that are current as of the time of distribution. But updates are made and security patches released virtually daily on one or more of those scripts. It would be nearly impossible to keep up with that. Plesk does not have a way to just go update all the installed scripts ( i could be wrong but I'd be very surprised).

    So the fact of the matter is, if you allow clients to use the application vault, you have to assume responsibility to maintain it.

    It falls outside the scope of work because it's not supported by Plesk either. you are required to update your site's scripts.
    Show your reciprocal links on your website. eReferrer

  10. #10
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,290
    Looks like rackspace does not support much at all after all.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  11. #11
    Join Date
    Nov 2006
    Posts
    68
    They just suggested the alert logic scanning and IDS system.... They ll be talking to us tomorrow about the cost of that... Any idea is this can help ? I have been looking into www.alertlogic.com and it looks interesting... Now I am still convinced that rackspace failed performing some basic security actions on our servers.

    We ll have this sorted out tomorrow during a conference on the phone. I will keep you updated on that.

    If anyone has any idea about cost of IDS and aler logic scanning, more than welcome so we ll have an idea about what we are talking tomorrow...

  12. #12
    Join Date
    Oct 2006
    Location
    Torbay - UK
    Posts
    23
    I would look into installing mod_security with a custom ruleset from Gotroot

    If Rackspace won't even do the basics what is the point of being with them.

    As for the application Vault packages disable them. Make your customer's use the most up to date versions from the creators websites.

    Can I ask what services RackSpace do provide for the very large fees they charge...

    Alan

  13. #13
    Maybe I read the original post wrong, but you're not expecting Rackspace to go through scripts you have written or installed yourself and find potential areas for exploit are you?

    Hosts will often inform you of an exploited script, because it will perhaps cause a high load on the server and affect other customers. If you have a dedicated box though it won't be affecting anyone other than you, so Rackspace are unlikely to know about it.

  14. #14
    Rackspace is just not really what everyone put them up to be. We have used them for many years, and while network is excellent in terms of uptime, their support lags behind that of cheaper datacenters and seems to be on the downswing recently. When you do complain, your account manager at rackspace will call and that touch is pleasant although it doesn't necessarily solve anything. But other than that, we tend to find their support lacking to be frank.

    We had an issue once where a site was being ddos on the apache level. While we do run a variety of scripts to block it, and even when we were ready to consider purchasing their firewall solution (which they claim can block such attacks), they told us that our server is unable to add this service as it is in a "different section" of the datacenter.

    Things got really bad the last 2 weeks. The server crashed suddenly. We opened a reboot ticket, and other than the useless auto-reply, the first reply came 9 hours later. No updates to the tickets, nothing. As this wasn't a critical server, we didn't think much about it. A few days later, the server crash again. This time the reboot took 4 hours. Apparently each time they mentioned that it needed to be checked by the DC techs i.e . they have frontline techs who just reboot it via remote ports and once it fails, good luck as their DC techs can take a while.

    But clearly at this point, there is some serious issues with the server. But at no point in either ticket was this mentioned. A few days later, it crashed again and this time it died for good.

    When I called they said that the server used for installations is down. So 4 1/2 hours later, they replied in the ticket and said it has been reinstalled with Redhat 7.2 (yes you read it right, Redhat 7.2 which hasn't been patched even by Progeny or Fedora-Legacy for over a couple of years). So when I asked they said that they can rekick it to RHE but there's a charge of $75. While they did waive it later, it did leave a sour taste. On top of that, they didn't give me the root password. Took 30 minutes after I asked. Then it appears they disabled root access but didn't give me the admin username / password which led to a delay of about an hour. 1 hour later, i decided to reinstall it again to RHE. Took them 6 hours this time to reinstall it.

    So all in all, it took about 12 hours in total. Naturally we have remote backups and already restored the sites to another server.

    Throughout the episode, updates were sparse, response slow, or not forthcoming etc.

    The account manager did call later and frankly, that to us means nothing. The crunch and critical tests are when you have a technical issue. They have never really delivered support wise when it matters. As such, frankly, it best to save your monthly and go elsewhere and hire external support techs. The only plus has been their network uptime. But datacenters like ev1 or theplanet (other issues aside) has provided similar uptime for a lower premium.

    This is probably going to be the nail in the coffin and we are probably not renewing the server.
    Like us on Facebook to qualify for discounts!
    http://www.sprintserve.net
    Offering: | Internap FCP Bandwidth! | Rebootless Kernel Updates! | Magento Optimized Hosting | Wordpress Hosting |
    Services: | Managed Multiple Cores 64bit Servers | Server Management |

  15. #15
    Join Date
    Jun 2005
    Location
    Canada
    Posts
    2,493
    Quote Originally Posted by Steven
    Looks like rackspace does not support much at all after all.
    Nail on the head. One of the most lauded companies out there and they are next to useless. I can pay half of the cost for a service many, many times over.
    GeeksGather - Undergoing redevelopment. Stand by.

  16. #16
    Quote Originally Posted by PixelManual
    Nail on the head. One of the most lauded companies out there and they are next to useless. I can pay half of the cost for a service many, many times over.
    This is not our experience, although I admit we're hosting with Rackspace UK and not the US counterpart.

    I've called support in the daytime, in the middle of the night and on a Sunday and they've resolved our issues there and then on the phone. Usually tickets are responded to withint an hour - when they're not the ticket usually requires more attention and is not urgent anyway.

    Although we're hosting with Rackspace UK I know their support switches to the US at night and sometimes during weekends.

    I cannot rate this company highly enough so far. They also allow you to rate their support ticket responses, and they publish the results each month.

  17. #17
    Quote Originally Posted by Celly
    This is not our experience, although I admit we're hosting with Rackspace UK and not the US counterpart.

    I've called support in the daytime, in the middle of the night and on a Sunday and they've resolved our issues there and then on the phone. Usually tickets are responded to withint an hour - when they're not the ticket usually requires more attention and is not urgent anyway.

    Although we're hosting with Rackspace UK I know their support switches to the US at night and sometimes during weekends.

    I cannot rate this company highly enough so far. They also allow you to rate their support ticket responses, and they publish the results each month.
    I don't doubt that their support works for some of the clients if not most. Otherwise I doubt they can become the company they are today. But it didn't work for me when we most needed it and that's how it is.
    Like us on Facebook to qualify for discounts!
    http://www.sprintserve.net
    Offering: | Internap FCP Bandwidth! | Rebootless Kernel Updates! | Magento Optimized Hosting | Wordpress Hosting |
    Services: | Managed Multiple Cores 64bit Servers | Server Management |

  18. #18
    Join Date
    Nov 2006
    Posts
    68
    About mod_security, rackspace wont provide the setup either the configuration. They dont support it.

  19. #19
    Join Date
    Nov 2006
    Posts
    68
    We are also hosted in Rackspace UK.

    We have an issue with a "managed" dedicated CISCO PIX 506 firewall at rackspace. In fact I understand that this firewall is not managed at all since the technician are waiting for us to tell them the exact rules, IN EXACT FORMAT, to apply to the PIX 506. We have been giving them so iptables firewall rules set that are applied on some other servers outside rackspace network. At first they were telling that they could not "convert" the rules into the PIX 506. After discussion with our account manager and threats not to pay the bill for an unmanaged firewall it seems that suddenly the technician was able to convert the iptables rules into PIX 506 format...

    Strange...

  20. #20
    Join Date
    Nov 2006
    Posts
    68
    Anothe incident, yes i know that s a lot but this is why we chosed Rackspace for their ability to provide hosting for our critical mission.

    We realized we had a problem with qmail and the queue was counting about 11000 emails. No doubts that we had some spam inside. We asked rackspace support for that. The answer was that 1) we have too many domains on the server (he wrote 560) 2) it was surely sent through one of he scripts hosted by the clients.

    Wat this technician does not know is that WE TESTED THEM because the PSA offer some tools to determine which script was used for sending those spam emails.

    Of course when we replied to the ticket in a very hard way, suddenly it was possible the find out how the spam was being sent.

    Fanatical Support ??? Mmmmm noooo.... Fanatical Marketing yes !

    Once again RACKSPACE FAILED in their critical mission which is to provide Fanatical Support, they dont search the cause of the problem in most of the case. An example, if apache crashes, what they would do in most of the case is to restart apache and they wont go further IF YOU DO NOT PUSH THEM TO INVESTIGATE THE "WHY APACHE CRASHED"??

    Now I dont say that all rackspace is bad... Not at all and we might even consider taking a new server over there because their London DC is very convenient to us BUT if you think you can go to Rackspace and can rely 100% on them, this is wrong (it was our idea, maybe or surely a mistake from us...). Rackspace does not provide PRO-ACTIVE management.

    Now we decided the hire ServerWizards for:

    1) Securing the servers (rackspace wont do it)
    2) Upgrade / updates (rackspace wont do it as our configuration has been recompiled already due to the number of vhosts)
    3) Provide us a pro-active approach of our servers
    4) Things that Rackspace wont do...

  21. #21
    Only a few complaints? I think the experience with RackSpace varies with who you talk to, when you buy it, and what you buy...

  22. #22
    Quote Originally Posted by joshcrick
    Only a few complaints? I think the experience with RackSpace varies with who you talk to, when you buy it, and what you buy...
    And probably how you manage it yourself too. At the end of the day, you're partly responsible for your own server. I'm not suggesting anyone here has done crazy things with their server, but if they have then Rackspace can't always be expected to clean up the whole mess.

    Of course I might be singing a different tune if we have problems and Rackspace won't help us - however we manage all aspects of our server very carefully from rotating logs, dealing with spam and monitoring CPU and memory loads - so that we rely on RS support as little as possible.

    Regarding managed firewall configurations, isn't it correct that the customer should supply Rackspace with the rules? For example, if they want to lock SSH to your IP address, you've got to supply your IP address for them to do this. Furthermore, you might not want SSH limiting in this way (if you don't have a fixed IP). I imagine a default config without any input from the customer could cause real problems.

  23. #23
    Join Date
    Nov 2006
    Posts
    68
    They should not supply the rules of course not but when the client comes with a set of rules that in used in iptables they should be able to set the PIX 506 properly, at least without threatening not paying the bill for this firewall...

    Rackspace is NOT a pro-active management company and will wait the problems occur to react. That s OK for me since we decided to hire another organization for the pro-active side of the management BUT I think it was necessary to clarify it into this forum because some clients may think (after receiving thousands of promises during the Fanatical Sale Negotiations) that they will do all the job... That s not the case...

  24. #24

    Are your expectations realistic?

    Guys,

    I am very familiar with Rackspace's setup and truly have not experienced a better hoster.

    Surely maintaining your firewalls config is your responsibility? How are they expected to know what your security requirements are.

    The way Rackspace has communicated their Support to me is as follows:

    They will fully support all software components I purchase from them, so that would include the OS and any applications, such as MSSQL, Urchin, IMail ...

    Any components installed by myself, would not fall in the realms of supported products and would at best be supported on a best effort basis (i cannot call up chasing them along) or on a billable basis (in which case I can call and chase).

    Surely we are not expecting a limited number of tech to expertly know each and every application or RPM out there?
    Furthermore, they have explicitly said they do not troubleshoot coding. They are not developers.

    M experience has shown that they are awesome at what they support and try their utmost with that which they do not.

    They need to draw a line somewhere however.

  25. #25

    ok

    Quote Originally Posted by sunray69
    Now we decided the hire ServerWizards for:

    1) Securing the servers (rackspace wont do it)
    2) Upgrade / updates (rackspace wont do it as our configuration has been recompiled already due to the number of vhosts)
    3) Provide us a pro-active approach of our servers
    4) Things that Rackspace wont do...
    I'm a current rackspace customer and I have nothing but praise for them. Sure I agree that they should secure the servers but isnt that what a firewall is for? Also how are they meant to know every single application that is in the open source community and the possible security flaws with each?

    I have their platinum Rackwatch and while I agree that some underling issues have been hard to identify and resolve, I'm always aware of any incident and my server is always back up online with priority.

  26. #26
    Quote Originally Posted by superman_returns
    Guys,

    I am very familiar with Rackspace's setup and truly have not experienced a better hoster.

    Surely maintaining your firewalls config is your responsibility? How are they expected to know what your security requirements are.

    The way Rackspace has communicated their Support to me is as follows:

    They will fully support all software components I purchase from them, so that would include the OS and any applications, such as MSSQL, Urchin, IMail ...

    Any components installed by myself, would not fall in the realms of supported products and would at best be supported on a best effort basis (i cannot call up chasing them along) or on a billable basis (in which case I can call and chase).

    Surely we are not expecting a limited number of tech to expertly know each and every application or RPM out there?
    Furthermore, they have explicitly said they do not troubleshoot coding. They are not developers.

    M experience has shown that they are awesome at what they support and try their utmost with that which they do not.

    They need to draw a line somewhere however.
    Could not agree with you more. I think some people think "management" means they will install all your 3rd party apps, trouble shoot them and fix bugs in scripts that you've written.

    If you rent serviced offices, you get the windows cleaned and any major defects to the property repaired in with the deal. This does not mean the management/service company will make the tea for you and answer your phone!

  27. #27
    Join Date
    Feb 2004
    Posts
    634
    Quote Originally Posted by superman_returns
    Surely maintaining your firewalls config is your responsibility? How are they expected to know what your security requirements are.
    In my experience, having worked on projects using vendors such as Verisign, ISS, Sun, Sungard, FishNet, etc., the definition of "managed firewall service" certainly includes some rule changes/updates, this after doing an initial configuration. Now they may put a limit on the number of rule changes they will do per month, but I've never heard of a vendor not doing any rule changes under something labeled as "managed firewall services." Aside from patch updates (and a device as ancient and underpowered as a PIX 506 isn't going to have many updates; none of the version 7 code base will even run on it), what else would they be doing otherwise?

  28. #28

    rule changes

    Quote Originally Posted by lockbull
    In my experience, having worked on projects using vendors such as Verisign, ISS, Sun, Sungard, FishNet, etc., the definition of "managed firewall service" certainly includes some rule changes/updates, this after doing an initial configuration. Now they may put a limit on the number of rule changes they will do per month, but I've never heard of a vendor not doing any rule changes under something labeled as "managed firewall services." Aside from patch updates (and a device as ancient and underpowered as a PIX 506 isn't going to have many updates; none of the version 7 code base will even run on it), what else would they be doing otherwise?
    I thnk the point was that while it is a "managed firewall" how are Rackspace to know what services should be running and what white and black list ips need to be added to the firewall? I think its like you tell them what to set and they'll set it. They are not mind readers.

  29. #29
    Join Date
    Jun 2004
    Posts
    137
    Quote Originally Posted by superman_returns
    Guys,

    I am very familiar with Rackspace's setup and truly have not experienced a better hoster.

    Surely maintaining your firewalls config is your responsibility? How are they expected to know what your security requirements are.

    ...[/COLOR]
    [/COLOR]
    I think the number of new users who signed up just to defend Rackspace is laughable. This guy only has one post at present.

  30. #30
    Join Date
    Nov 2006
    Posts
    68
    I think the number of new users who signed up just to defend Rackspace is laughable. This guy only has one post at present.
    I did not notice it but now that you mention it you are right. Well, I still think that our choice will go towards rackspace for the third very new server but this time we ll know on what we can count on and on what we cannot. Of course we do not ask suppot for setting up third party softwares, this would be ridiculous from our side and of course we do not ask them to support them.

    As far as for third party software support when they have time, my own experience tells me that I do no believe it. This is not real, we got a very rude answer concerning mod_security and their billing fees to support that. Even worse, I played a bit with the iptables on 1 server and it was conflicting with the physical firewall, what was my reaction when I got an answer from a technician claiming 75$ / 30 minuts for fixing that. What I have to say is when I called them, I had a wonderful guy on the phone who said nothing and fixed the issue immediately, this was support !

    So I dont surely say that all is bad but sometimes they dont have the level they advertise.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •