Results 1 to 17 of 17
  1. #1
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348

    Latest CL update breaks some binaries

    Just a heads up for providers running CloudLinux - their latest update pushes numerous things however it appears their bsocks-* packages (from release 1 -> release 2) breaks some things including sendmail.

    To fix it immediately downgrade that package and add bsocks* to your yum.conf excludes.

    Code:
    yum downgrade bsock-libs-0.09-1.el6.x86_64 bsock-0.09-1.el5
    The broken version is

    Code:
    bsock-libs-0.09-2.el5h
    bsock-0.09-2.el5
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  2. #2
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Quote Originally Posted by CodyRo View Post
    Just a heads up for providers running CloudLinux - their latest update pushes numerous things however it appears their bsocks-* packages (from release 1 -> release 2) breaks some things including sendmail.

    To fix it immediately downgrade that package and add bsocks* to your yum.conf excludes.

    Code:
    yum downgrade bsock-libs-0.09-1.el6.x86_64 bsock-0.09-1.el5
    The broken version is

    Code:
    bsock-libs-0.09-2
    bsock-0.09-2
    This affects both EL5/EL6.
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  3. #3
    Join Date
    Mar 2012
    Posts
    34
    ^ This happened to us! -- proxyexec was broken :/ had to restart it, happened across 2 of my servers.

  4. #4
    Yep, we've come across issues this morning too. Looks like it also breaks cpanel / webmail redirects. This is the second time in as many weeks that CL has broken systems with updates that they've pushed out.

  5. #5
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    This is another example of why I recommend people turn off auto updates for both cPanel and OS packages. Nothing worse than ending up with every single server being broken due to a botched update in the middle of the night. (If it ain't broken, don't fix it?)
    Patrick William | RACK911 Labs | Software Security Auditing
    400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com

    www.HostingSecList.com - Security notices for the hosting community.

  6. #6
    Join Date
    Mar 2006
    Location
    Servers
    Posts
    1,588
    The have done it again after their MySQL mess before. How to fix:

    Cpanel redirection' issue you can fix by running:
    #service proxyexecd restart

    Also fixed bsock* packages are out -- please try to update:
    # yum clean all --enablerepo=cloudlinux-updates-testing
    # yum update bsock bsock-libs --enablerepo=cloudlinux-updates-testing
    QHoster.com - Web Hosting with DDoS Protection | Shared & Reseller in Europe/North America
    Linux/Windows RDP VPS 13 Locations : UK, US (5 states), Mexico, Canada, Bulgaria, Lithuania,
    Italy, France, Germany,Netherlands, Switzerland, Rissia, Singapore | OpenVPN/PPTP Enabled
    INSTANT | PayPal, Skrill, Payza, Bitcoin, WebMoney, Perfect Money, Ukash, CashU, paysafecard

  7. #7
    Join Date
    Dec 2009
    Posts
    140
    Was this package pushed to stable or beta ?

  8. #8
    Join Date
    Jun 2011
    Posts
    2,286
    Looks like they made an official post: http://cloudlinux.com/blog/clnews/378.php

  9. #9
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Quote Originally Posted by bdx33 View Post
    Was this package pushed to stable or beta ?
    Stable repositories.

    Quote Originally Posted by Patrick View Post
    This is another example of why I recommend people turn off auto updates for both cPanel and OS packages. Nothing worse than ending up with every single server being broken due to a botched update in the middle of the night. (If it ain't broken, don't fix it?)
    What if it's a critical exploit / bug being fixed?
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  10. #10
    Join Date
    Mar 2003
    Location
    Canada
    Posts
    8,910
    Quote Originally Posted by CodyRo View Post
    What if it's a critical exploit / bug being fixed?
    People should have various mailing lists for security related notices / important updates go to their 24/7 help desk for immediate resolution then - or their phones. (CloudLinux should setup two mailing lists... one for security and one for normal updates, like what a lot of other vendors do.)

    I certainly see the benefits of auto updates but I'm sure you can remember a few years back when a cPanel update broke a ton of things and it was a mess for a lot of people. It's not just cPanel too, several other vendors have pushed out updates that ended up breaking things and this will keep happening in the future.

    Most security flaws that are patched fortunately don't have public exploits available removing that "MUST PATCH THIS VERY SECOND!" approach allowing the admin a few hours or longer to manually apply the patch and see if anything is broken. Personally, when I apply updates it's to one server and then a little bit later pushed out to the rest just to see if anything breaks or if any tickets get opened for something obscure that I didn't catch during my own testing.

    Edit:

    Another theoretical scenario... imagine an auto update server being compromised and then thousands of random servers updating in the middle of the night downloading a backdoor. I mean, there's a lot of arguments for/against auto updates, I guess it depends on the person and what other safe guards they have in place.
    Patrick William | RACK911 Labs | Software Security Auditing
    400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com

    www.HostingSecList.com - Security notices for the hosting community.

  11. #11
    Join Date
    Feb 2006
    Location
    Buffalo NY
    Posts
    1,348
    Quote Originally Posted by Patrick View Post
    People should have various mailing lists for security related notices / important updates go to their 24/7 help desk for immediate resolution then - or their phones. (CloudLinux should setup two mailing lists... one for security and one for normal updates, like what a lot of other vendors do.)

    I certainly see the benefits of auto updates but I'm sure you can remember a few years back when a cPanel update broke a ton of things and it was a mess for a lot of people. It's not just cPanel too, several other vendors have pushed out updates that ended up breaking things and this will keep happening in the future.

    Most security flaws that are patched fortunately don't have public exploits available removing that "MUST PATCH THIS VERY SECOND!" approach allowing the admin a few hours or longer to manually apply the patch and see if anything is broken. Personally, when I apply updates it's to one server and then a little bit later pushed out to the rest just to see if anything breaks or if any tickets get opened for something obscure that I didn't catch during my own testing.

    Edit:

    Another theoretical scenario... imagine an auto update server being compromised and then thousands of random servers updating in the middle of the night downloading a backdoor. I mean, there's a lot of arguments for/against auto updates, I guess it depends on the person and what other safe guards they have in place.
    Every large company I've been with has an auto update policy - it's not feasible to manually monitor every aspect of your environment for packages and manually apply updates. It's simply not a feasible policy.

    This is why organizations like RedHat exist - updates don't get pushed until it's been tested and is stable. This is why they backport patches instead of running bleeding edge. Another great example of this is Debian - if you've followed their release cycles you'll see what I mean.

    Being aware of issues is absolutely necessarily - relying on manually doing things is simply dumb and naive if you think that scales out beyond a few machines. Auto updating will save you more times than it'll hurt you - unfortunately some vendors have a not-so-savory build environment / testing environment.
    Cody R.
    Hawk Host Inc. Proudly Serving websites since 2004.
    Let's Encrypt Sponsor.

  12. #12
    Join Date
    Dec 2009
    Posts
    140
    Quote Originally Posted by CodyRo View Post
    Every large company I've been with has an auto update policy - it's not feasible to manually monitor every aspect of your environment for packages and manually apply updates. It's simply not a feasible policy.

    This is why organizations like RedHat exist - updates don't get pushed until it's been tested and is stable. This is why they backport patches instead of running bleeding edge. Another great example of this is Debian - if you've followed their release cycles you'll see what I mean.

    Being aware of issues is absolutely necessarily - relying on manually doing things is simply dumb and naive if you think that scales out beyond a few machines. Auto updating will save you more times than it'll hurt you - unfortunately some vendors have a not-so-savory build environment / testing environment.
    I believe Patrick meant vendor auto update should not be activated by default.
    You should first test released patches and updates on a test machine before pushing it to your own park. You must decide whether to mass apply a change to your systems and not the software vendor.

    This can be done by running your own RPM repository from which your servers pull updates, or any other kind of automation.

  13. #13
    Join Date
    Mar 2012
    Posts
    34
    Agreed with Cody, however I also agree with Patrick - it's a win-loose situation where unfortunately security is one of those things, if we had 2 or 3 servers which could be manually managed then manual updates then the process will be a lot easier to manage, however when you start managing more then 40+ servers, things get that little bit difficult. We rely on automatic updates, we rely on our vendors.

  14. #14
    Join Date
    Mar 2006
    Location
    Servers
    Posts
    1,588
    CloudLinux should test or have some testers before release such an update causing troubles and tickets and reputation to the customers.
    QHoster.com - Web Hosting with DDoS Protection | Shared & Reseller in Europe/North America
    Linux/Windows RDP VPS 13 Locations : UK, US (5 states), Mexico, Canada, Bulgaria, Lithuania,
    Italy, France, Germany,Netherlands, Switzerland, Rissia, Singapore | OpenVPN/PPTP Enabled
    INSTANT | PayPal, Skrill, Payza, Bitcoin, WebMoney, Perfect Money, Ukash, CashU, paysafecard

  15. #15
    Join Date
    Mar 2003
    Location
    California USA
    Posts
    13,294
    Quote Originally Posted by CodyRo View Post
    Every large company I've been with has an auto update policy - it's not feasible to manually monitor every aspect of your environment for packages and manually apply updates. It's simply not a feasible policy.

    This is why organizations like RedHat exist - updates don't get pushed until it's been tested and is stable. This is why they backport patches instead of running bleeding edge. Another great example of this is Debian - if you've followed their release cycles you'll see what I mean.

    Being aware of issues is absolutely necessarily - relying on manually doing things is simply dumb and naive if you think that scales out beyond a few machines. Auto updating will save you more times than it'll hurt you - unfortunately some vendors have a not-so-savory build environment / testing environment.
    I deal with some companies that are enterprisey type of companies and auto updates == no no. We have to schedule a change control, get on a conference call, and file paper work.

    Even some hosting companies I work with run LTS versions of cpanel and its weeks of 'testing' on their side before they will update to anything unless there is security benefit.

    Its really not uncommon, its a huge battle for me to get updates applied with some our customers.
    Steven Ciaburri | Industry's Best Server Management - Rack911.com
    Software Auditing - 400+ Vulnerabilities Found - Quote @ https://www.RACK911Labs.com
    Fully Managed Dedicated Servers (Las Vegas, New York City, & Amsterdam) (AS62710)
    FreeBSD & Linux Server Management, Security Auditing, Server Optimization, PCI Compliance

  16. #16
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    836
    This was our fault 100% -- without any valid excuse. Very similar issue as with MySQL governor, but from another team. I started the work to structure the way we do all releases right after MySQL Governor. The changes were not propagated fast enough through all the teams (we have 4 at this moment).
    We will stop any new non-critical releases at all, until we finalize our release procedures, and create fail safes to make sure everyone follows it. The new procedures should go in effect by the end of this week.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  17. #17
    Join Date
    Mar 2006
    Location
    Servers
    Posts
    1,588
    Quote Originally Posted by iseletsk View Post
    This was our fault 100% -- without any valid excuse. Very similar issue as with MySQL governor, but from another team. I started the work to structure the way we do all releases right after MySQL Governor. The changes were not propagated fast enough through all the teams (we have 4 at this moment).
    We will stop any new non-critical releases at all, until we finalize our release procedures, and create fail safes to make sure everyone follows it. The new procedures should go in effect by the end of this week.
    Really nice for CloudLinux to admit the mistake. Great software and support.
    QHoster.com - Web Hosting with DDoS Protection | Shared & Reseller in Europe/North America
    Linux/Windows RDP VPS 13 Locations : UK, US (5 states), Mexico, Canada, Bulgaria, Lithuania,
    Italy, France, Germany,Netherlands, Switzerland, Rissia, Singapore | OpenVPN/PPTP Enabled
    INSTANT | PayPal, Skrill, Payza, Bitcoin, WebMoney, Perfect Money, Ukash, CashU, paysafecard

Similar Threads

  1. Beware !!! - Windows update KB951748 breaks ZoneAlarm
    By The Dude in forum Computers and Peripherals
    Replies: 1
    Last Post: 07-09-2008, 12:43 PM
  2. cPanel automatic update breaks "service status" page again?
    By GeorgeC in forum Hosting Security and Technology
    Replies: 6
    Last Post: 10-04-2005, 09:28 PM
  3. Cpanel update breaks Movable Type
    By erick_p in forum Dedicated Server
    Replies: 5
    Last Post: 07-03-2005, 04:28 AM
  4. cPanel Update, Breaks again
    By slice16 in forum Hosting Security and Technology
    Replies: 7
    Last Post: 05-01-2005, 01:49 AM
  5. cPanel Update breaks server once more :(
    By slice16 in forum Web Hosting
    Replies: 12
    Last Post: 02-14-2005, 11:47 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •