Results 1 to 13 of 13
  1. #1
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174

    Service Affecting increase in disk io after installing Cloud Linux

    Hey Folks,

    Looking for some advice, thoughts or opinions. I'm running a WHM/cPanel server with about a 180GB of data in /home I installed Cloud Linux a few days ago. My backups used to run in about 6hrs now they are taking 12-15hrs. I have determined the cause to be a significant increase in disk io (see the attached image) You can see that starting on the 24th the disk io demand changed significantly which is when Cloud Linux was installed.

    No changes were made to the backup configuration so its clearly a side effect of CL but does anyone have an idea's why?

    I tried increasing the KB chunk to 512 last night thinking if it did compression in memory and it was to compress larger chunks it would have less frequent io, but I think that it had the opposite effect by the looks of today's stats. I plan to try reducing it down to the minimum 128 tonight and maybe reduce the number of pigz processes to 6 and see if that alleviates the demand, but I'd still like opinions and comments.

    TECHNICAL HARDWARE DETAILS:
    sda is a 1TB RAID10 array on a 3ware 9650SE-4LP
    sdc is a 2TB drive on the motherboard
    Server is a Dual E5410 8GB RAM

    BACKUP CONFIGURATION:
    Compressed Backups using the new cPanel implementation (not legacy)

    COMPRESSIONS SETTINGS:
    use pigz - yes
    gzip compression level - minimum (1)
    Number of pigz processes - 8
    Number of kilobyte chunks per compression work unit - 512

    Thanks
    Attached Thumbnails Attached Thumbnails diskio.png  

  2. #2
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    Could you check that raid still in tact? We have seen few cases where raid would start re-syncing after reboot.
    Also, is it KVM or Xen or real hardware?
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  3. #3
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    Hi Igor, thanks for jumping in

    I just checked and can confirmed that the RAID is verified and fully in-tact. This is 100% real hardware nothing virtualized.

    Furthermore I've been looking at stats and thinking, at first I thought maybe CL was placing a considerably higher demand on sda for general activity which left less disk resource available for the backup, BUT when the backups not running the disk io is the same as before the switch so it doesn't appear to be a base increase in disk-io. So that leaves the question what about the backups under CL is different then not.

    Thanks

  4. #4
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    This makes more sense, as single threads get lower IO priority under CL --> maybe that is what causing the slower back ups. What do you use for backup?
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  5. #5
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    Im using the native whm backup. (the new one they implemented a few months ago, the legacy backup system is disabled.)
    Zenon Kazienko
    President,
    Budget Host Canada
    Your Windows & Linux Solution for Shared, Reseller & Dedicated Hosting

  6. #6
    Join Date
    Jun 2001
    Location
    Princeton
    Posts
    1,029
    My guess it has to do with process IO priorities. Ping our support if it causes a problem. It shouldn't create extra load on IO --> it should just take longer.
    Igor Seletskiy
    CEO @ Cloud Linux Inc
    http://www.cloudlinux.com
    CloudLinux -- The OS that can make your Shared Hosting stable

  7. #7
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    I assume by pinging you mean submit a ticket? Its definitely causing a problem LOL just look at the 6hrs of 100% utilization.

  8. #8
    Join Date
    Jul 2007
    Posts
    2,051
    Can you confirm if you have kept the backup schedule and the stats updation schedule to different time? In the recent cPanel updates, the Analog stats is enabled by default and it takes a very long time to update the stats and also the i/o increases. Make sure the backup and stats updation does not run at the same time and also disable Analog stats.
    Prashant T.

    Don't run after Success. Run after Excellence and Success will soon follow.

  9. #9
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    I'm processing log every 12 hrs and bandwidth every 6hrs. I have checked the box to "Prevent cpanellogd (Log Processing) and cpbackup (Backups) from running at same time"

    I have NOT selected times from "Statistics Processing Times" grid where it tells me "Check all hours that you would like log analysis NOT to be performed"

    Am I right that so long as the "Prevent cpanellogd (Log Processing) and cpbackup (Backups) from running at same time" box is selected that also prevents stats from running during backups or is cpanellogd separate from that?

    Analog stats is disabled by default, AWstats and Webalizer is active.

    :::NOTE:::
    I appreciate your suggesting and am very welcoming to configuration tweaks to alleviate the symptoms, however I'd like to underscore the other primary focus of this thread is to discuss why with all things being equal the disk activity during backups is so much greater under CL vs a native Centos environment. yet there is little to no change in the disk activity during normal server functions.

  10. #10
    Join Date
    Jun 2011
    Location
    USA/UK/SG
    Posts
    3,636
    In this case, the word 'ping' does indeed mean to contact them.

    http://www.audioenglish.org/dictionary/ping.htm

    For reference, they can be contacted here: https://helpdesk.cloudlinux.com/inde...Tickets/Submit

    Quote Originally Posted by Zenon View Post
    I assume by pinging you mean submit a ticket? Its definitely causing a problem LOL just look at the 6hrs of 100% utilization.
    ~]# Ethernet Servers Ltd - Est. 2014! - sales @ ethernetservers.com
    ~]# Try out our WordPress speed tests for yourself!
    ~]# NVMe Web Hosting | Unmanaged VPS | Fully Managed VPS | Dedicated Servers | Domain Names
    ~]# Don't settle for any less than the very best - come & join our family today!

  11. #11
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    I went ahead and submitted a ticket last night so we shall see what we find out. Coincidentally I lowered the KB per compression chunk down to 128 and the backup completed in about 12 hours with only 6hrs spent at 100% disk utilization. So that did improve it, but its still significantly more disk activity then previously to the CL.

    PS So it doens't get lost in the ticket, I would still like someone to confirm/clarify my understanding of the stats processing scheduling options I described above.

  12. #12
    Join Date
    Oct 2005
    Posts
    2
    Any updates on this? I was looking into cloud linux. As for the WHM backups, mine were taking too long also and I disabled compression. My content is 95% jpgs and gz compression has no effect on them anyway. But it does take a lot of cpu to create the compressed file.

  13. #13
    Join Date
    Oct 2007
    Location
    Hamilton, Ontario Canada
    Posts
    174
    I believe we solved the issue yesterday, but one backup run is not enough data for me to be sure, I will reply back either tomorrow or Monday after I can re-introduce some of my tweaks and evaluate the results thereof.

    Again I'll post more details of an eval of cloud linux later, but in short so far I am very, very happy with the results and support assistance I have received. (hopefully my satisfaction is not pre-mature)
    Zenon Kazienko
    President,
    Budget Host Canada
    Your Windows & Linux Solution for Shared, Reseller & Dedicated Hosting

Similar Threads

  1. Amazon Web Service Expert Cloud Engineer Linux Admin
    By cloudwow in forum Employment / Job Requests
    Replies: 0
    Last Post: 09-12-2013, 02:04 PM
  2. Any Way to increase Disk I/O ??
    By MetroVPS_Mahfuz in forum VPS Hosting
    Replies: 21
    Last Post: 01-09-2013, 05:07 AM
  3. Linux permission issues affecting lighttpd, cron and qmail
    By omega36 in forum Hosting Security and Technology
    Replies: 7
    Last Post: 12-22-2007, 10:36 AM
  4. Replies: 2
    Last Post: 11-13-2003, 08:23 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •