I run a 50 slot server and when the slots are all filled we run about 25% of a quad-core w/ hyper-threading.
The processor I'm running my server on is Intel Xeon X3450 at 2.67GHz. So as you can see I've allocated more than plenty to operate the server even when the slots expand, our ram usage at 50/50 is around 3.5-4.0GB.
Can you provide a list of plugins, you should be able to access the list as OP by typing /plugins in game. You may have some unnessacary plugins operating and using more CPU to cover Ram costs.
I recommend you look into Java Process Monitoring, either VisualVM or YourKit Java Profiler, which would allow you to narrow down any CPU/Ram issues.
Also if you're utilizing anything that is supposed to "increase performance" such as NoLagg or PTweaks, please make sure they are configured properly, NoLagg will actually store every chunk loaded into the Ram Cache which will end up eating everything you have allocated to your Java process, you need proper heap timing for the cache cleanup and NoLagg can definitely improve server performance. PTweaks on the other hand utilizes more CPU to same Ram.
Hope this helps,
Calvin
Linear Mode
