I'm not sure if server location matters, but if it does I seriously doubt its at an any more focused level than by country.
One thing you might want to do also is look at using different TLDs to target different countries.
★ Sam Barrow - CEO @ SQUIDIX(1-855-SQUIDIX)
★ Ask Us About Sponsoring Your Web Site (High Traffic Sites Only)
★ Squidix - Shared, Reseller, Semi-Dedicated, Managed VPS and Managed Dedicated Hosting
★ Midwestern Web - Web Design & Development Services
Yes, local servers will improve your website speed and also google will give you little importance on the country where the servers are located. There is a blog entry regarding this on their webmasters blog.