Web Hosting Talk : Web Hosting Main Forums : Hosting Security and Technology : Nginx NAS serving large files

[central10]

Hello Ladies and Gentlemen,

We have a very frustrating problem with a NAS configuration that we recently set-up. After running a filesharing website on 8 servers for 1+ years we decided to switch to a NAS configuration. We set-up our NAS (24x1TB Raid 5) to serve files to 9 front end servers over a private switch - doing 9 servers on 1 disk(logical)... not sure if nginx can handle this. Since we've moved to the NAS BW usage has dropped significantly even though our traffic has stayed the same. Download speeds are pretty bad aswell but this is due to IO issues that are being created by NGINX. so here's the problem we are facing...

In theory nginx is creating an io bottleneck because it's opening to many times without closing. We're using nginx to serve files to our 9 front end servers - each file will stay alive though for 45+ minutes. Instead of opening it for 10 seconds, reading into memory, and closing - it's opening it for 10 seconds, reading into memory, streaming it for 45 minutes, THEN closing. We ran a test with nginx and we were only able to download at 30kbps on port 182, how-ever over apache we were able to reach speeds of 20mb/s.

We've re-compiled nignx and installed the most recent stable version but that didn't fix anything.

Can someone please suggest a fix here? I've ran out of ideas...

Thank you!

[tanfwc]

How big is each file size? Do you plan to enable caching on each nginx node so it does not reach your NAS for each file access?

[central10]

Hello Tan,

Each file is anywhere from 40mb up to 1GB. We have nginx configured to cache files aswell as the diskcache in linux.

This is a very frustrating problem. Were paying big bucks for our servers but aren't getting much use out of them.

[tanfwc]

You might want to check your nginx configuration because nginx has been performing real good on my setup.

[central10]

We've made many changes to the nginx config but there are no performance changes.

What type of configuration are you operating on?

[twikamltd]

You're using nginx on the NAS server to serve the frontends? Wouldn't it make more sense to use Nginx on the frontends only and use something like NFS to connect to the NAS and also have some degree of caching on the frontends?

[central10]

You're using nginx on the NAS server to serve the frontends? Wouldn't it make more sense to use Nginx on the frontends only and use something like NFS to connect to the NAS and also have some degree of caching on the frontends?

Sorry for my poor explanation. We are using the box with NFS and we have the fronends using nginx. We've re-compiled nginx with the developers version how-ever that too didn't solve the problem.

Are there any other possible configurations because as of now we are out of ideas. Maybe we could do something else either then using a NFS...???

Any help is much appreciated.

[UnixCabin]

Wait..... I'm confused... What communication protocol are you using? NFS?

[central10]

Originally Posted by UnixCabin

Wait..... I'm confused... What communication protocol are you using? NFS?

We have a Network File System set-up between our front end servers and our NAS server. Nignx is on on all of our front end servers and is used to serve files to users.

Were open any ideas...

[silasmoeckel]

Are you using http://labs.frickle.com/nginx_ngx_slowfs_cache ?

[central10]

Hello,

That caching plugin isn't going to be applicable to us, we couldn't cache ~5tb files in ram. What we need is a change in the underlying way nginx operates with opening files.

[mistwang]

NFS latency will kill the performance of event-driven web server when streaming large static file from NFS backend, that's for sure.
cache wont help, only one possible tweak is to enable AIO, but I am not sure how robust nginx AIO implementation is, or its AIO have any effect on NFS.

[Steven]

Originally Posted by mistwang

NFS latency will kill the performance of event-driven web server when streaming large static file from NFS backend, that's for sure.
cache wont help, only one possible tweak is to enable AIO, but I am not sure how robust nginx AIO implementation is, or its AIO have any effect on NFS.

How well would litespeed work in this instance.
Would you meet the 20mb/s he did with apache?

[mistwang]

Does not like AIO implementation in other web servers, which only work with direct I/O access mode, Seems linux NFS AIO implementation is still a work in progress.

LiteSpeed AIO works with buffered I/O, can take advantage of kernel cache memory.
I have not done much testing with AIO under NFS, so I cannot be very sure. It is also depends on Linux NFS client implementation. I will give it a try when I get a chance, the result will be interesting.

[File1eu]

You can try opening the file in smaller chunks instead of opening it at once. Maybe NFS+nginx creates a bottleneck when it tries to open a large amount of data at once?