First, this is not infeasible for a one-man show, but you need to know what you're doing. You need to understand the key concepts of VoIP, understand how SIP works, how it differs from RTP and what codecs are used, what the limitations are, etc.
You will need:
1) Inbound SIP trunks
2) IP interconnect
3) Server hardware
If you want to provide a toll-free dial-in number, then it will cost you per each connected minute. Some providers charge based on origin so thus a call from No Where, OH will cost you 4-5 cents/min, while call from Los Angeles, CA < 1 cent/min. Other providers bill using a blended rate, and that is usually 1.5 - 2.5 cents/min. These are just sample numbers; the actual rates depend on your commitment volume and negotiation power. In this economy many providers operate on pre-paid basis, so you will need to shell out the $$$ first before getting service, so be wise choosing a provider.
Your calls can be delivered via TDM or IP. IP will be cheaper (and more flexible), so you will need an Internet connection with decent latency to your provider (<30 ms would be good), low jitter and virtually no packet loss. Jitter is important because VoIP is a real-time protocol so late packets are useless. They need to arrive on time. Your initial numbers of 20 participants x 10-20 conferences, you will need 32+ Mbit bandwidth (symmetric) as each call is ~80 kbps with overhead using G711. That also translates to 40,000 packets/sec if you really have 400 channels in use. The actual monthly transfer amount depends on the actual usage, but don't expect to have all channels full at all times -- not even close.
Next you need a decent hardware capable of handling the call volume. Your 400 channels can fit on a single server higher-end server (say a Sandy Bridge Xeon). Mixing 20 conferences with 20 participants in G711 isn't going to be very CPU intensive, but your box needs to be handle all the packet flow.
Then you need software to run it all. There are commercial and open source products to choose from. Open source options include FreeSwitch and Asterisk, both have a learning curve -- especially if you want to deal with hundreds of channels with media.
Finally, you need some kind of way to manage it all. Bill your customers, set up conference rooms and route calls there, etc. If you are in for the long run, expect to be able to offer the service at competitive rates and have the demand for your services, then the investment will be worth it. Otherwise, go with an established conferencing service. Either way, good luck.
Pings <1 ms, Unlimited Transfer, Lowest Price: http://localhost/