>> Can anyone give some guidence at the right time to upgrade your web server in regards to CPU Usage?
A: According to the details you have furnished, it is clear that you have quite high amount of CPU usage and it should be upgraded as soon as possible.
>> If you see the CPU spiking to 100% time to time, is this a good sign to think about upgrading?
>> Heres a screenshot of my current webserver CPU usage, does this look really bad?
A: Unfortunately, yes!
It does look like both CPUs are working hard at the same time, so extra cores might help.
But I would try to find out what is causing the spikes. It could be something on the webserver you can optimize. How many hits are you getting? If millions an hour, extra CPU is nice. Otherwise find out what is running slow and then optimize it. If your process is truly multi-threaded then more cores will help. But faster CPU only gets you so far nowadays. CPUs are improving performance by core count not so much by clock speed, so you must confirm that your process will take advantage of many threads -- and typically these issues are related to a single query doing something the slow way causing CPU spikes.
I suggest you to monitor the high CPU usage processes. Sometimes, this happens if you have heavy traffic web sites or web sites have bad written scripts (PHP, ASP etc.). I would suggest to optimize your server as much as possible and monitor the usage of the websites. If this doesn't fix the problem, you may upgrade your server.