Most, but definitely not all - merchant accounts have annual contracts, early termination fees and monthly minimums.
If you shop around, there are definitely options available that have none of the above drawbacks and there are many free setup programs available as well for a merchant account & gateway combination.
If you are completely just testing the waters and have an audience that would be completely receptive to PayPal, that's fine... and having PayPal as an option regardless is never a bad thing.
But if you are trying to market a new online store, product or service you will probably want to offer a direct means by which a customer can pay via credit card without using PayPal in order to accomodate as many customers as possible... and some consumers get concerned if the only option a merchant offers is PayPal from a perception standpoint.
You will definitely want to minimize your costs, of course, but I think you can get the total solution desired within your budget from the sound of your post.
Best of luck with your search!