Absolutely, in fact I would say that it is very common for a dedicated server to share a port. How much to be concerned about it will really depend on what you are doing, and how your host shares ports. If the bandwidth isn't heavily oversold, and they do some work to make sure that no single server can monopolize the bandwidth you should be fine. However if you have a very high bandwidth requirement you might want to get your own dedicated bandwidth.
These days the vast majority of 100mbps ports are uncontended, in fact I'm struggling to think of any providers in Western Europe/N America/Canada who offer anything less.
When it comes to 1gb ports the situation is quite different and only a few big players like leaseweb and softlayer explicitly advertised dedicated ports on low (Less than 100TB) commitments. This seems quite reasonable to me - why would you need a truly dedicated 1GE port for a 5TB commitment, which is like 1.5% utilisation?
Anything like 10GE on a dedicated server (OVH) is highly likely to be contended as no provider in their right mind is going to dedicated a full 10GE for a few hundred meg cdr.