How much disk swapping is acceptable before adding new memory?

One of my servers has the following;
Memory Usage
Type Percent Capacity Free Used Size
Physical Memory 32% 168.12 MB 80.25 MB 248.37 MB
Disk Swap 5% 489.89 MB 27.79 MB 517.68 MB

I wonder where the swapping comes from since we didn't change anything and have the same visits/hits. It was 0% all the time since a couple of days.