Results 1 to 25 of 25
Thread: NAS, or DAS ?
-
07-02-2009, 07:41 AM #1Newbie
- Join Date
- Jul 2009
- Posts
- 15
NAS, or DAS ?
Hi,
I run a small website online, which a portion of does file hosting.
I have an MD1000 which I use for storage, but I'm about to run out of room, and wondering to get next. (instead of another MD)
there is how I tested out the MD, from first to last
6x 500GB 7.2k RAID 5...HORRIBLE
6x 500GB 7.2k RAID 10...HORRIBLE
6x 450GB 15k RAID 5...HORRIBLE
6x 450GB 15k RAID 10...FANTASTIC!
stripe size: 128k
block size: 4k
...now the problem is the SAS drives get expensive and the MD takes up alot of room and uses alot of power.
I can get used netapp, emc, dell, and products like that for cheap. so am open to ideas/products. basically looking for something fairly prices and scalable, with a high throughput.
Note that this is not a database array, simply storage that's gonna be used for file downloads. so no 2k, 4k reads, mainly files from 5megs to 10megs.
...i still don't understand the difference between random i/o and sequential but the files are stored like: (can someone explain? ...google didn't help)
/mount-point/1
/mount-point/2
/mount-point/3
/mount-point/4
(all the files are numerical, the actual file name is stored in a database)
lastly, is there anyway I can get away with 1TB sata 7.2k drives? or do I need spindles for what I'm doing?
-
07-02-2009, 10:37 AM #2Web Hosting Master
- Join Date
- Apr 2009
- Posts
- 604
Since you already have a MD1000 I would recommend getting the MD3000i which is a IP NAS system and you can daisy chain that to your MD1000 as well as connect multiple servers to it now, instead of just 1. You can also scale up to another MD1000 enclosure if required which would bring the total to 45TB (45 x 1TB drives).
-
07-02-2009, 10:38 AM #3WHT Addict
- Join Date
- Jun 2007
- Posts
- 110
Sequential io is requesting sequential blocks on the disk (head doesn't need to reposition), random io are non-adjacent sectors that require the head to reposition. Are you running linux? run this:
iostat -x 1
while under normal and peak load and determine how many iops you're actually doing (r/s & w/s, and then average op size (in sectors, can run it with -k if you prefer KB)), this will give you some idea of exactly what kind of performance you need. Also will be helpful to know what filesystem you're currently running.
Serving 5-10 meg files over the web doesn't tell you much about your actual io pattern, that depends on kernel/tcp window scaling + webserver + way files are written to disk, also if you're writing data to the array (logs, etc) consistently that's a factor too (mixed r/w iops).
-
07-02-2009, 10:52 AM #4Master of the Truth
- Join Date
- Mar 2006
- Location
- Reston, VA
- Posts
- 3,131
get a sas 5/e card and another MD1000 daisy them together and install solaris+zfs. Scale on the fly/on demand as you need to.
Yellow Fiber Networks
http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net
-
07-02-2009, 07:53 PM #5Newbie
- Join Date
- Jul 2009
- Posts
- 15
Code:Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.43 0.97 0.96 0.83 28.04 14.45 23.72 0.01 3.85 0.63 0.11 sda1 0.01 0.00 0.00 0.00 0.03 0.00 14.92 0.00 28.61 24.94 0.00 sda2 0.42 0.97 0.96 0.83 28.01 14.45 23.73 0.01 3.82 0.63 0.11 dm-0 0.00 0.00 1.37 1.80 28.00 14.44 13.35 0.03 10.69 0.36 0.11 dm-1 0.00 0.00 0.00 0.00 0.01 0.01 8.00 0.00 2.74 0.23 0.00 sdc 43.14 23.30 420.28 2.02 27492.50 202.55 65.58 2.15 5.08 1.21 50.90 sdc1 43.14 23.30 420.28 2.02 27492.48 202.55 65.58 2.15 5.08 1.21 50.90 sdd 2.48 153.54 0.17 2.77 21.09 156.31 60.48 0.11 37.55 0.35 0.10 sdd1 2.47 153.54 0.16 2.77 21.07 156.31 60.50 0.11 37.57 0.35 0.10 sde 13.89 303.96 0.43 5.42 35.00 309.38 58.92 0.10 17.56 0.22 0.13 sde1 13.89 303.96 0.42 5.42 34.97 309.38 58.93 0.10 17.56 0.22 0.13 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.99 36.82 0.00 60.20 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 2.00 0.00 24.00 0.00 12.00 0.01 5.50 5.50 1.10 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 2.00 0.00 24.00 0.00 12.00 0.01 5.50 5.50 1.10 dm-0 0.00 0.00 2.00 0.00 24.00 0.00 12.00 0.01 5.50 5.50 1.10 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 46.00 0.00 589.00 0.00 34792.00 0.00 59.07 2.77 4.68 1.27 74.80 sdc1 46.00 0.00 589.00 0.00 34792.00 0.00 59.07 2.77 4.68 1.27 74.80 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.99 33.17 0.00 63.84 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 80.00 0.00 558.00 0.00 37224.00 0.00 66.71 2.76 4.90 1.30 72.80 sdc1 80.00 0.00 558.00 0.00 37224.00 0.00 66.71 2.76 4.90 1.30 72.80 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 3.25 34.75 0.00 62.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 2.97 0.00 12.87 0.00 126.73 9.85 0.00 0.31 0.08 0.10 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 2.97 0.00 12.87 0.00 126.73 9.85 0.00 0.31 0.08 0.10 dm-0 0.00 0.00 0.00 15.84 0.00 126.73 8.00 0.00 0.25 0.06 0.10 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 44.55 0.00 610.89 0.00 36855.45 0.00 60.33 2.81 4.66 1.15 70.10 sdc1 44.55 0.00 611.88 0.00 36855.45 0.00 60.23 2.81 4.65 1.15 70.10 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 2.75 33.50 0.00 63.75 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 21.00 190.00 607.00 4.00 36192.00 1552.00 61.77 2.61 4.28 1.21 74.20 sdc1 21.00 190.00 606.00 4.00 36192.00 1552.00 61.88 2.61 4.29 1.22 74.20 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Can you interpret that for me?
...If im correct I'm around 650 IOPs/sec?
I already have the sas 5/e, and I don't think I need another to daisy chain, just a cable, maybe 2. (for split-mode)
Can you elaborate on your "Scale on the fly/on demand" comment? sounds fantastic.
-
07-03-2009, 08:48 PM #6Newbie
- Join Date
- Jul 2009
- Posts
- 15
...anyone?
-
07-04-2009, 03:36 PM #7Aspiring Evangelist
- Join Date
- Nov 2003
- Location
- Lynnwood, WA
- Posts
- 438
Avg read requests/s over 5 seconds: 557 reads/s
Avg sectors read/s over 5 seconds: 34511 sectors/s
Avg write requests/s over 5 seconds: 1 write/s
Avg sectors written/s over 5 seconds: 350 sectors/s
Assuming you're using a > 2.4 kernel, a sector should be 512-byte in size, so you're doing:
Avg read/s: 17,255.5 KB
Avg writ/s: 175 KB
If all my assumptions are right, that is.
Interestingly, going on that, you're doing roughly 32 KB per read request. You say this is all large file reads..?
-
07-05-2009, 06:48 AM #8Newbie
- Join Date
- Jul 2009
- Posts
- 15
-
07-05-2009, 09:36 PM #9Newbie
- Join Date
- Jul 2009
- Posts
- 15
wait
took some research but i think I know why the 32kb number came up.
I have NFS running on the file server with these specs:
192.168.1.5:/A0 /A0 nfs ro,noatime,rsize=32768,wsize=32768 0 0
...and if i'm not wrong 32kb is max for NFS, so i'm limited to that read size, in other words screwed?
-
07-05-2009, 09:42 PM #10Master of the Truth
- Join Date
- Mar 2006
- Location
- Reston, VA
- Posts
- 3,131
Yellow Fiber Networks
http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net
-
07-06-2009, 05:00 AM #11Newbie
- Join Date
- Jul 2009
- Posts
- 15
MTU is actually 9,000 from NFS to webserver, then 1,500 from webserver to the public/world.
Code:eth3 Link encap:Ethernet HWaddr 00:1B:21:3C:02:F5 inet addr:192.168.1.6 Bcast:192.168.123.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fe3c:2f5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets:412818454 errors:42 dropped:0 overruns:0 frame:21 TX packets:169 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2884324549 (2.6 GiB) TX bytes:20215 (19.7 KiB) Memory:df6a0000-df6c0000
-
07-08-2009, 02:20 AM #12Newbie
- Join Date
- Jul 2009
- Posts
- 15
k
........anyone?
-
07-08-2009, 02:30 AM #13Web Hosting Master
- Join Date
- Jun 2006
- Location
- NYC / Memphis, TN
- Posts
- 1,454
OpenSolaris (ZFS + RAIDZ) - absolutely incredible performance with an unbeatable price. NAS in theoretically, 20 minutes or less.
http://blogs.sun.com/icedawn/entry/bondin
http://developers.sun.com/openstorag...ge_server.html
http://blogs.sun.com/bonwick/entry/raid_z
Even better...never do another fsck again.
I'd also mention NexentaStor since it's now Enterprise ready:
http://www.nexenta.com/corp/ including support. Big plus is unlimited file sizes and unlimited snapshots. Less than $1100 per 4TBLast edited by PeakVPN-KH; 07-08-2009 at 02:42 AM.
≈ PeakVPN.Com | Complete Privacy VPN | Cloud Hosting | Guaranteed Security | 1Gbps-10Gbps Unmetered
≈ PeakVPN | 31 VPN Servers | 17-Years Experience | Emergency 24/7 Support
≈ Visit us @ PeakVPN.Com (Coming SOON) | ASN: 3915
-
07-08-2009, 03:17 AM #14Newbie
- Join Date
- Jul 2009
- Posts
- 15
ya...I've been doing alot of research on this lately, and let me say that you're the 3rd person to tell me about ZFS. so I think that's the way we might go...but then again, i gotta choose the hardware too.
I've pretty much made my decision and it's going to be one of the following systems:
1. NS2000 - http://www.scalecomputing.com/products_architecture.html
2. Dell EqualLogic PS5000XV http://www.equallogic.com/products/view.aspx?id=2509
pretty much the same cost/terabyte on both machines...tho the dell supports SAS drives as well verses the scalebox which only has sata drives.
I attached another iostat output...this one is for a few hours ran with "iostat -dkxt" and 60 second intervals...Can anyone tell me if sata disks will work for me according to this data?
-
07-08-2009, 03:30 AM #15Web Hosting Master
- Join Date
- Jun 2006
- Location
- NYC / Memphis, TN
- Posts
- 1,454
Wish I was more of a disk guy so I can't say from that log. I'm sure someone will know.
As for ZFS, I can speak to it and say that it is a gift from the open source gods. I've never managed a simpler filesystem with so much flexibility, stability, and overall speed. The ease of adding/removing physical disks in a RAID config, adding disks to mirrors, running cleanups, etc. It's insane. It cuts the time down to 10% of what it used to take to perform those tasks. Time spent doing maintenance or recovery when a failure does occur, becomes less than 10 minutes of actual work in most cases. Instant snapshots...Sweet... Not to mention, I really meant it when I said no more fsck's. SIMPLY not having to run a fsck -f on a 1TB+ disk ever again made me an instant follower.
Even if you don't choose it, I'd recommend trying it out. Read up on it and do some research. I am simply in awe every time I have to do something and it takes me 1 command to add a disk to an array or remove a disk, etc.
Good luck≈ PeakVPN.Com | Complete Privacy VPN | Cloud Hosting | Guaranteed Security | 1Gbps-10Gbps Unmetered
≈ PeakVPN | 31 VPN Servers | 17-Years Experience | Emergency 24/7 Support
≈ Visit us @ PeakVPN.Com (Coming SOON) | ASN: 3915
-
07-08-2009, 03:38 AM #16Newbie
- Join Date
- Jul 2009
- Posts
- 15
-
07-08-2009, 04:09 AM #17Web Hosting Master
- Join Date
- Jun 2006
- Location
- NYC / Memphis, TN
- Posts
- 1,454
eh well I am kind of Sun fanboy anyway
≈ PeakVPN.Com | Complete Privacy VPN | Cloud Hosting | Guaranteed Security | 1Gbps-10Gbps Unmetered
≈ PeakVPN | 31 VPN Servers | 17-Years Experience | Emergency 24/7 Support
≈ Visit us @ PeakVPN.Com (Coming SOON) | ASN: 3915
-
07-08-2009, 08:43 AM #18Aspiring Evangelist
- Join Date
- Nov 2003
- Location
- Lynnwood, WA
- Posts
- 438
+1 for ZFS as well. The Nexenta stuff just makes it easy and wraps it up with a bow. ZFS is the nirvana of filesystems, right now, as far as I'm concerned, and I'll deploy it anywhere I can get away with it.
-
07-08-2009, 09:26 AM #19Newbie
- Join Date
- Jul 2009
- Posts
- 15
Coolio.
Just to update anyone else reading the 2nd page of this thread:
I attached a iostat log 2 posts ago, wondering if someone can go through and interpret it for me.
thanks in advance.
-
07-08-2009, 07:24 PM #20Newbie
- Join Date
- Jul 2009
- Posts
- 15
...so I did some calculations myself.
and I'm doing about 120MB/sec throughput and around 3,200 IOPS
so need something that can support that if not more.
any suggestions?
comments?
-
07-08-2009, 09:03 PM #21Master of the Truth
- Join Date
- Mar 2006
- Location
- Reston, VA
- Posts
- 3,131
Yellow Fiber Networks
http://www.yellowfiber.net : Managed Solutions - Colocation - Network Services IPv4/IPv6
Ashburn/Denver/NYC/Dallas/Chicago Markets Served zak@yellowfiber.net
-
07-09-2009, 12:02 AM #22Web Hosting Master
- Join Date
- Jun 2001
- Location
- Denver, CO
- Posts
- 3,302
Not physically possible to get 3200 IOPs out of 6 15K SAS drives. Assuming you have 3.5ms seek time (don't know the exact drive you have), you are only looking at 285 IOPs per drive. Read is 285*6 (1714), write is 285*3 (855).
Why don't you just drop some more drives into the MD1000 and expand your array?Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
Current specials here. Check them out.
-
07-09-2009, 12:17 AM #23Newbie
- Join Date
- Jul 2009
- Posts
- 15
-
07-09-2009, 03:21 AM #24Newbie
- Join Date
- Jul 2009
- Posts
- 15
hmmm
so I've decided which box i'm gonna get.
it supports both NFS and ISCSI.
how do I choose which one to go with?
...from what I understand iscsi has a bit 25% higher throughput as it's a lower level(block) mount?
as for my architure...
The SAN/NAS is going to be a big raid-10 array (5-10tb)
there is going to be 6 total servers connecting to it. 1 server doing only writes and another 5 read-only serving files over the web.
Any suggestions?
-
07-12-2009, 10:11 AM #25Newbie
- Join Date
- Jul 2009
- Posts
- 15
...anyone?