Our shopping cart (CS-Cart) doesn't allow S3 integration and we need to host downloadable products off-site. CS-Cart allows this via URL access only... I was thinking of using Rackspace's Cloud and their Public Containers, but I just got done chatting with them and they say that robots.txt files don't work in the containers and there's no way to prevent search engines from cataloging the files.

Since this is our digital assets, the question is important, but I tend to think that even if this data is catalogued, it won't appear anywhere useful in the rankings.

Do search engines catalog the metadata within the files (say mp3s or whatever) at all or do they just go by filenames?

Thanks for your insight!