Your problem lacks more description.
You may try to leave out front page's directories out of the rewrite (by using RewriteRule ^(_vti_bin|_vti_adm|_vti_aut) - [L]).
I might have missed out some directories above so be sure to check that out too.
Note that using mod_rewrite with .htaccess is different than using it with httpd.conf
Well - FPSE wants to control everything and when you start to use htaccess - it starts to control everything. They are basically fighting each other.
I do not recommend FPSE on a *NIX machine, only windows since it seems to be so much easier to control. You might check out Frontpage Forums and talk to some of the people there who specialize more in Frontpage