Results 1 to 25 of 25

Thread: NAS, or DAS ?

  1. #1

    NAS, or DAS ?

    Hi,

    I run a small website online, which a portion of does file hosting.

    I have an MD1000 which I use for storage, but I'm about to run out of room, and wondering to get next. (instead of another MD)

    there is how I tested out the MD, from first to last

    6x 500GB 7.2k RAID 5...HORRIBLE
    6x 500GB 7.2k RAID 10...HORRIBLE
    6x 450GB 15k RAID 5...HORRIBLE
    6x 450GB 15k RAID 10...FANTASTIC!

    stripe size: 128k
    block size: 4k

    ...now the problem is the SAS drives get expensive and the MD takes up alot of room and uses alot of power.

    I can get used netapp, emc, dell, and products like that for cheap. so am open to ideas/products. basically looking for something fairly prices and scalable, with a high throughput.

    Note that this is not a database array, simply storage that's gonna be used for file downloads. so no 2k, 4k reads, mainly files from 5megs to 10megs.

    ...i still don't understand the difference between random i/o and sequential but the files are stored like: (can someone explain? ...google didn't help)
    /mount-point/1
    /mount-point/2
    /mount-point/3
    /mount-point/4

    (all the files are numerical, the actual file name is stored in a database)


    lastly, is there anyway I can get away with 1TB sata 7.2k drives? or do I need spindles for what I'm doing?

  2. #2
    Join Date
    Apr 2009
    Posts
    604
    Since you already have a MD1000 I would recommend getting the MD3000i which is a IP NAS system and you can daisy chain that to your MD1000 as well as connect multiple servers to it now, instead of just 1. You can also scale up to another MD1000 enclosure if required which would bring the total to 45TB (45 x 1TB drives).

  3. #3
    Sequential io is requesting sequential blocks on the disk (head doesn't need to reposition), random io are non-adjacent sectors that require the head to reposition. Are you running linux? run this:

    iostat -x 1

    while under normal and peak load and determine how many iops you're actually doing (r/s & w/s, and then average op size (in sectors, can run it with -k if you prefer KB)), this will give you some idea of exactly what kind of performance you need. Also will be helpful to know what filesystem you're currently running.

    Serving 5-10 meg files over the web doesn't tell you much about your actual io pattern, that depends on kernel/tcp window scaling + webserver + way files are written to disk, also if you're writing data to the array (logs, etc) consistently that's a factor too (mixed r/w iops).

  4. #4
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132

  5. #5
    Quote Originally Posted by cacheflymatt View Post
    Sequential io is requesting sequential blocks on the disk (head doesn't need to reposition), random io are non-adjacent sectors that require the head to reposition. Are you running linux? run this:

    iostat -x 1

    while under normal and peak load and determine how many iops you're actually doing (r/s & w/s, and then average op size (in sectors, can run it with -k if you prefer KB)), this will give you some idea of exactly what kind of performance you need. Also will be helpful to know what filesystem you're currently running.

    Serving 5-10 meg files over the web doesn't tell you much about your actual io pattern, that depends on kernel/tcp window scaling + webserver + way files are written to disk, also if you're writing data to the array (logs, etc) consistently that's a factor too (mixed r/w iops).
    Code:
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.43     0.97  0.96  0.83    28.04    14.45    23.72     0.01    3.85   0.63   0.11
    sda1              0.01     0.00  0.00  0.00     0.03     0.00    14.92     0.00   28.61  24.94   0.00
    sda2              0.42     0.97  0.96  0.83    28.01    14.45    23.73     0.01    3.82   0.63   0.11
    dm-0              0.00     0.00  1.37  1.80    28.00    14.44    13.35     0.03   10.69   0.36   0.11
    dm-1              0.00     0.00  0.00  0.00     0.01     0.01     8.00     0.00    2.74   0.23   0.00
    sdc              43.14    23.30 420.28  2.02 27492.50   202.55    65.58     2.15    5.08   1.21  50.90
    sdc1             43.14    23.30 420.28  2.02 27492.48   202.55    65.58     2.15    5.08   1.21  50.90
    sdd               2.48   153.54  0.17  2.77    21.09   156.31    60.48     0.11   37.55   0.35   0.10
    sdd1              2.47   153.54  0.16  2.77    21.07   156.31    60.50     0.11   37.57   0.35   0.10
    sde              13.89   303.96  0.43  5.42    35.00   309.38    58.92     0.10   17.56   0.22   0.13
    sde1             13.89   303.96  0.42  5.42    34.97   309.38    58.93     0.10   17.56   0.22   0.13
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00    2.99   36.82    0.00   60.20
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00     0.00  2.00  0.00    24.00     0.00    12.00     0.01    5.50   5.50   1.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda2              0.00     0.00  2.00  0.00    24.00     0.00    12.00     0.01    5.50   5.50   1.10
    dm-0              0.00     0.00  2.00  0.00    24.00     0.00    12.00     0.01    5.50   5.50   1.10
    dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdc              46.00     0.00 589.00  0.00 34792.00     0.00    59.07     2.77    4.68   1.27  74.80
    sdc1             46.00     0.00 589.00  0.00 34792.00     0.00    59.07     2.77    4.68   1.27  74.80
    sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdd1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00    2.99   33.17    0.00   63.84
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdc              80.00     0.00 558.00  0.00 37224.00     0.00    66.71     2.76    4.90   1.30  72.80
    sdc1             80.00     0.00 558.00  0.00 37224.00     0.00    66.71     2.76    4.90   1.30  72.80
    sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdd1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00    3.25   34.75    0.00   62.00
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00     2.97  0.00 12.87     0.00   126.73     9.85     0.00    0.31   0.08   0.10
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda2              0.00     2.97  0.00 12.87     0.00   126.73     9.85     0.00    0.31   0.08   0.10
    dm-0              0.00     0.00  0.00 15.84     0.00   126.73     8.00     0.00    0.25   0.06   0.10
    dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdc              44.55     0.00 610.89  0.00 36855.45     0.00    60.33     2.81    4.66   1.15  70.10
    sdc1             44.55     0.00 611.88  0.00 36855.45     0.00    60.23     2.81    4.65   1.15  70.10
    sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdd1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00    2.75   33.50    0.00   63.75
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    dm-0              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdc              21.00   190.00 607.00  4.00 36192.00  1552.00    61.77     2.61    4.28   1.21  74.20
    sdc1             21.00   190.00 606.00  4.00 36192.00  1552.00    61.88     2.61    4.29   1.22  74.20
    sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sdd1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sde1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    Thank you for the speedy reply and info.

    Can you interpret that for me?

    ...If im correct I'm around 650 IOPs/sec?

    Quote Originally Posted by Spudstr View Post
    get a sas 5/e card and another MD1000 daisy them together and install solaris+zfs. Scale on the fly/on demand as you need to.
    I already have the sas 5/e, and I don't think I need another to daisy chain, just a cable, maybe 2. (for split-mode)

    Can you elaborate on your "Scale on the fly/on demand" comment? sounds fantastic.

  6. #6
    ...anyone?

  7. #7
    Join Date
    Nov 2003
    Location
    Lynnwood, WA
    Posts
    438
    Avg read requests/s over 5 seconds: 557 reads/s
    Avg sectors read/s over 5 seconds: 34511 sectors/s

    Avg write requests/s over 5 seconds: 1 write/s
    Avg sectors written/s over 5 seconds: 350 sectors/s

    Assuming you're using a > 2.4 kernel, a sector should be 512-byte in size, so you're doing:

    Avg read/s: 17,255.5 KB
    Avg writ/s: 175 KB

    If all my assumptions are right, that is.

    Interestingly, going on that, you're doing roughly 32 KB per read request. You say this is all large file reads..?

  8. #8
    Quote Originally Posted by Nex7 View Post
    Avg read requests/s over 5 seconds: 557 reads/s
    Avg sectors read/s over 5 seconds: 34511 sectors/s

    Avg write requests/s over 5 seconds: 1 write/s
    Avg sectors written/s over 5 seconds: 350 sectors/s

    Assuming you're using a > 2.4 kernel, a sector should be 512-byte in size, so you're doing:

    Avg read/s: 17,255.5 KB
    Avg writ/s: 175 KB

    If all my assumptions are right, that is.

    Interestingly, going on that, you're doing roughly 32 KB per read request. You say this is all large file reads..?
    Linux 2.6.18-128.1.6.el5PAE

    and yes, it's all large files, maybe several small ones but on average I would say about 90-100mb/file.

    is there possibly any other reason on why the 32k number came up?

  9. #9
    wait

    took some research but i think I know why the 32kb number came up.

    I have NFS running on the file server with these specs:

    192.168.1.5:/A0 /A0 nfs ro,noatime,rsize=32768,wsize=32768 0 0

    ...and if i'm not wrong 32kb is max for NFS, so i'm limited to that read size, in other words screwed?

  10. #10
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by MikeyT View Post
    wait

    took some research but i think I know why the 32kb number came up.

    I have NFS running on the file server with these specs:

    192.168.1.5:/A0 /A0 nfs ro,noatime,rsize=32768,wsize=32768 0 0

    ...and if i'm not wrong 32kb is max for NFS, so i'm limited to that read size, in other words screwed?

    that read size is pointless imho if your running a 1500MTU

  11. #11
    Quote Originally Posted by Spudstr View Post
    that read size is pointless imho if your running a 1500MTU
    MTU is actually 9,000 from NFS to webserver, then 1,500 from webserver to the public/world.

    Code:
    eth3      Link encap:Ethernet  HWaddr 00:1B:21:3C:02:F5  
              inet addr:192.168.1.6  Bcast:192.168.123.255  Mask:255.255.255.0
              inet6 addr: fe80::21b:21ff:fe3c:2f5/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
              RX packets:412818454 errors:42 dropped:0 overruns:0 frame:21
              TX packets:169 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:2884324549 (2.6 GiB)  TX bytes:20215 (19.7 KiB)
              Memory:df6a0000-df6c0000
    going by what you said, would decreasing the NFS read size to 9,000 affect(help) performance at all?

  12. #12

    k

    ........anyone?

  13. #13
    Join Date
    Jun 2006
    Location
    NYC
    Posts
    1,446
    OpenSolaris (ZFS + RAIDZ) - absolutely incredible performance with an unbeatable price. NAS in theoretically, 20 minutes or less.

    http://blogs.sun.com/icedawn/entry/bondin
    http://developers.sun.com/openstorag...ge_server.html
    http://blogs.sun.com/bonwick/entry/raid_z

    Even better...never do another fsck again.

    I'd also mention NexentaStor since it's now Enterprise ready:
    http://www.nexenta.com/corp/ including support. Big plus is unlimited file sizes and unlimited snapshots. Less than $1100 per 4TB
    Last edited by FiberPeer; 07-08-2009 at 02:42 AM.
    FiberPeer.Com | | REAL DDoS Protection | Cloud Hosting | VPS | Dedicated Servers | High Bandwidth Hosting | 1Gbps-10Gbps Unmetered
    FiberPeer DDoS Mitigation | ethProxy Upgraded! | 14-Years Experience | Emergency 24/7 Support
    Visit us @ www.fiberpeer.com

  14. #14
    Quote Originally Posted by serverorigin View Post
    OpenSolaris (ZFS + RAIDZ) - absolutely incredible performance with an unbeatable price. NAS in theoretically, 20 minutes or less.

    http://blogs.sun.com/icedawn/entry/bondin
    http://developers.sun.com/openstorag...ge_server.html
    http://blogs.sun.com/bonwick/entry/raid_z

    Even better...never do another fsck again.

    I'd also mention NexentaStor since it's now Enterprise ready:
    http://www.nexenta.com/corp/ including support. Big plus is unlimited file sizes and unlimited snapshots. Less than $1100 per 4TB
    ya...I've been doing alot of research on this lately, and let me say that you're the 3rd person to tell me about ZFS. so I think that's the way we might go...but then again, i gotta choose the hardware too.

    I've pretty much made my decision and it's going to be one of the following systems:

    1. NS2000 - http://www.scalecomputing.com/products_architecture.html

    2. Dell EqualLogic PS5000XV http://www.equallogic.com/products/view.aspx?id=2509

    pretty much the same cost/terabyte on both machines...tho the dell supports SAS drives as well verses the scalebox which only has sata drives.

    I attached another iostat output...this one is for a few hours ran with "iostat -dkxt" and 60 second intervals...Can anyone tell me if sata disks will work for me according to this data?
    Attached Files Attached Files

  15. #15
    Join Date
    Jun 2006
    Location
    NYC
    Posts
    1,446
    Wish I was more of a disk guy so I can't say from that log. I'm sure someone will know.

    As for ZFS, I can speak to it and say that it is a gift from the open source gods. I've never managed a simpler filesystem with so much flexibility, stability, and overall speed. The ease of adding/removing physical disks in a RAID config, adding disks to mirrors, running cleanups, etc. It's insane. It cuts the time down to 10% of what it used to take to perform those tasks. Time spent doing maintenance or recovery when a failure does occur, becomes less than 10 minutes of actual work in most cases. Instant snapshots...Sweet... Not to mention, I really meant it when I said no more fsck's. SIMPLY not having to run a fsck -f on a 1TB+ disk ever again made me an instant follower.

    Even if you don't choose it, I'd recommend trying it out. Read up on it and do some research. I am simply in awe every time I have to do something and it takes me 1 command to add a disk to an array or remove a disk, etc.

    Good luck
    FiberPeer.Com | | REAL DDoS Protection | Cloud Hosting | VPS | Dedicated Servers | High Bandwidth Hosting | 1Gbps-10Gbps Unmetered
    FiberPeer DDoS Mitigation | ethProxy Upgraded! | 14-Years Experience | Emergency 24/7 Support
    Visit us @ www.fiberpeer.com

  16. #16
    Quote Originally Posted by serverorigin View Post
    Wish I was more of a disk guy so I can't say from that log. I'm sure someone will know.

    As for ZFS, I can speak to it and say that it is a gift from the open source gods. I've never managed a simpler filesystem with so much flexibility, stability, and overall speed. The ease of adding/removing physical disks in a RAID config, adding disks to mirrors, running cleanups, etc. It's insane. It cuts the time down to 10% of what it used to take to perform those tasks. Time spent doing maintenance or recovery when a failure does occur, becomes less than 10 minutes of actual work in most cases. Instant snapshots...Sweet... Not to mention, I really meant it when I said no more fsck's. SIMPLY not having to run a fsck -f on a 1TB+ disk ever again made me an instant follower.

    Even if you don't choose it, I'd recommend trying it out. Read up on it and do some research. I am simply in awe every time I have to do something and it takes me 1 command to add a disk to an array or remove a disk, etc.

    Good luck
    wow, I'm sure ZFS like's the pedestal you just put her on.

    thank you for your input, I'll now wait and someone will hopefully reply in regards to the iostat data.

  17. #17
    Join Date
    Jun 2006
    Location
    NYC
    Posts
    1,446
    eh well I am kind of Sun fanboy anyway
    FiberPeer.Com | | REAL DDoS Protection | Cloud Hosting | VPS | Dedicated Servers | High Bandwidth Hosting | 1Gbps-10Gbps Unmetered
    FiberPeer DDoS Mitigation | ethProxy Upgraded! | 14-Years Experience | Emergency 24/7 Support
    Visit us @ www.fiberpeer.com

  18. #18
    Join Date
    Nov 2003
    Location
    Lynnwood, WA
    Posts
    438
    +1 for ZFS as well. The Nexenta stuff just makes it easy and wraps it up with a bow. ZFS is the nirvana of filesystems, right now, as far as I'm concerned, and I'll deploy it anywhere I can get away with it.

  19. #19
    Coolio.

    Just to update anyone else reading the 2nd page of this thread:

    I attached a iostat log 2 posts ago, wondering if someone can go through and interpret it for me.

    thanks in advance.

  20. #20
    ...so I did some calculations myself.

    and I'm doing about 120MB/sec throughput and around 3,200 IOPS

    so need something that can support that if not more.

    any suggestions?

    comments?

  21. #21
    Join Date
    Mar 2006
    Location
    Reston, VA
    Posts
    3,132
    Quote Originally Posted by MikeyT View Post
    ...so I did some calculations myself.

    and I'm doing about 120MB/sec throughput and around 3,200 IOPS

    so need something that can support that if not more.

    any suggestions?

    comments?
    how big is your actual working data set?

  22. #22
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,301
    Not physically possible to get 3200 IOPs out of 6 15K SAS drives. Assuming you have 3.5ms seek time (don't know the exact drive you have), you are only looking at 285 IOPs per drive. Read is 285*6 (1714), write is 285*3 (855).

    Why don't you just drop some more drives into the MD1000 and expand your array?
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  23. #23
    Quote Originally Posted by Spudstr View Post
    how big is your actual working data set?
    What do you mean by a "data set"?

    Quote Originally Posted by Jay Suds View Post
    Not physically possible to get 3200 IOPs out of 6 15K SAS drives. Assuming you have 3.5ms seek time (don't know the exact drive you have), you are only looking at 285 IOPs per drive. Read is 285*6 (1714), write is 285*3 (855).

    Why don't you just drop some more drives into the MD1000 and expand your array?
    Well, the numbers I posted are from combines benchmarks from 3 different MDs with 3 different hosts. so essentially, 45 drives.

  24. #24

    hmmm

    so I've decided which box i'm gonna get.

    it supports both NFS and ISCSI.

    how do I choose which one to go with?

    ...from what I understand iscsi has a bit 25% higher throughput as it's a lower level(block) mount?

    as for my architure...

    The SAN/NAS is going to be a big raid-10 array (5-10tb)
    there is going to be 6 total servers connecting to it. 1 server doing only writes and another 5 read-only serving files over the web.

    Any suggestions?

  25. #25
    ...anyone?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •