DNS makes very little difference to most sites. When somebody first accesses your site, a DNS lookup is generally done. This is then cached for whatever the TTL for the records is — at least hours, if not days or weeks. On top of this, most ISPs cache DNS records. This ultimately leads to DNS being an efficient and fast service in virtually all cases, so any investment into it is going to make little if any appreciable difference. Put this at the bottom of any optimisation efforts.
Obviously your server makes a big difference if you're serving dynamic pages. The faster the server, the faster server-side stuff (e.g. PHP, MySQL) can be executed and delivered to the user. The server's performance can also impact how much traffic you can handle, although it should be noted this comes down to software configuration as much as hardware. It's perfectly possible for a 256MB non-SSD VPS to outperform a 4GB SSD VPS if the former has a superior set-up (Nginx instead of Apache, MariaDB instead of MySQL, PHP-APC, etc).
A caching plug-in can make a difference for dynamic pages, however this has the obvious penalty of your pages needing to be static. If your server is efficient than such a plug-in probably isn't necessary unless you have very high levels of traffic. Mine for example outputs WordPress pages in ~0.065 seconds so the pages being static makes relatively little difference.
A CDN will definitely make a difference as long as your site has a consistent throughput of traffic. Generally, CDNs fetch content from your server and then cache it for hours or days (depends on your cache headers). However, it's only cached in any given location (e.g. Amsterdam) when someone from that area visits. If you have very little traffic, your CDN will end up fetching it from your server more often than it already has it cached, which results in slower performance for users than if the CDN weren't used at all (I've verified this with EdgeCast, CDNify, CDN.net, CloudFlare, and CDN77). However, if you have enough traffic for the content to be cached most of the time then people will definitely benefit from the content being served from a node ~50ms away rather than 150–400ms away (US to EU or vice versa).
Last edited by Ryan Williams; 12-24-2013 at 09:25 AM.