at the end of the day, control panels are more to make the server easier to admin. I think it really depends on the person controlling, as some people find doing everything via command line better. Personally, I'd rather whm/cpanel too as i've yet to find any problems with it
I've been doing like you -- I did my first Unix admin work in 1990 or so (actually VMS and HP-UX, but you get the idea) and I'd just worked from the command line. It's a good way to manage a solid, reliable server.
But then I started to host sites for clients, and it really starts to become an issue. When you're the go-to guy for every e-mail change or small bit of tinkering, what's the minimum amount per month that you're willing to accept for that level of hassle?
For me, it was too much to host for more than a few clients, so I was referring them to systems like SME Server (see contribs.org for details) which made everything easy for them with web based controls and what-not. The thing is, I started to use the same tools in-house -- it's just easier to handle e-mail accounts, redirects, new domains and the rest in a good interface. Yes, I can manage the .conf files by hand and restart services as required, but it sure was nice to not have to.
So, as I'm now moving to some servers hosted off-site, I'm definitely moving to a panel. I want the flexibility to say "Yes, Maam -- all your controls are right here, so you can manage your e-mail accounts yourself and make your own backups whenever you want." If it works well I'll end up offering hosting to a wider range of people, as it really seems like a good control panel (I'm looking hardest at H-Sphere) and a well-built and well-managed server can lead to a minimum of time dealing with hosting clients.