Many hosting companies (especially unlimited as mentioned above) have an "inode" limit, which increases when you add more files and folders. If you go over that limit, some hosts stop including your account files in backups, and often people have their accounts suspended after reaching a few hundred thousand. 200,000 would definitely be close to the limits.
However, the files simply being present on the machine won't make any difference to the speed of the actual server (page loads etc).
Yes, the number of uploaded files, make slow down the server. It also depends on weight of files if you uploads heavy files then server will slow down more while in light weight files not disturbs server more.
Well it also depends on the servers configuration and hardware. Lightspeed for example will as every other OS will have to load all the files first but it processes PHP MUCH faster then RedHat for example so it loads faster.
█ Garbott Ltd - Professional web development & consultancy services
█ Oxford.Hosting - Prestigious shared, cloud, dedicated and gaming hosting solutions.