What would be the best forum software to use for setting up a forum? (I'm familiar with forums but have never set one up as an admin)
The main points that I am aware of is that PHPBB is open source and allows you to mod it according to your needs (plus it's free). However, I've seen concerns over security (high hack risk) as well as poor SEO.
vBulletin seems to offer you more out of the box and more user friendly for people like me who have limited coding experience. one of the main drawbacks seems to be the inability to do much (if any) modding and of course the cost for the software.
If you have the extra cash , go for vBulletin, no questions asked even!
every web application has its security risks, it is up to you - to make sure your software is secure and patched.
another thing you can do - start with phpBB get some members see how things work for you - its free and worth the try.
after a few months, review if phpBB seems to meet your needs, stick with it.
if you feel its not working out for you, you will be able to migrate over to vBulletin.
as for modding / SEO , its more up to your coder / webmaster to take care of that part for you.
I recommend vbulletin. it's easy to use and i believe it's more secure. with phpbb, you will have to install mods to get same features of vbulletin. also vbulletin has excellent paid mods such as vbseo that can come in handy big time if you have a big board.