Does the location of a server affect speed much? I have several download servers all located in the US, and speeds are fine for me (I'm in the US as well), but some people in France, Russia, China, etc., report that speeds are often kind of slow. Would the location of the server be causing this issue with international visitors? Would there be a large enough increase in speed to make it worth it for me to get a few servers in various countries around the world to cater to my international visitors?
If your server is located in the USA or EU, you should expect that visitors from across the pond will be able to access your server with great speed (for the web that is, not recommended for game servers). In this case you're limited by your provider and not limited by location.
However, data transfer in and out of China (and other countries) can be very saturated causing a big slowdown. You can normally see this slow down if you tried accessing a Chinese or Russian website, the pages usually load rather slow.
If you shove a test file on your server, and then post it here, i'm sure people will download it and report speeds from a variety of locations, to help you determine where the slowdowns are. You can grab a test file from here: http://speedtest.layeredtech.com/
If you want to serve alot of data to Europe, consider a location in NY, to eliminate the path from NY to WI to get to your data. This will help but will not ultimately eliminate all issues with European providers. Some of the local Euro ISPs just do not have fast enough access to main lines, so some users speeds will still be limited.
Let me know if you need EU but since there are many EU pep in here, I may not need post them. I do not live in none of these place, just happen to have server there. Considering all the server located in datacenter with good oversee network, this does not look good.