Let's say you have a bunch of links up on high PR sites, and your PR is around 5 or so. After a while, if the links on those other high PR sites are taken down, will your site's PR also go away next update?

If where talking about max. 5 links, then I don't think it affect you but if more than 5 links....it will affect your pr.

Always keep in mind that your pagerank dose not only depend on link popularity.

if all or most of your PR is coming from those sites, YES, it will affect PR. If you have other site linking to youas well, the difference will be less noticable.