does any one care if their servers are on the other side of the world or do you look for servers that are as close to home as possible.

The closer the better. But, it doesn't really matter if there are good techs on site.

Most of the visitors to my site are the local college students that I teach. It would be great to have my site hosted right here in town, so access and publishing times would be faster (theoretically).

Unfortunately, that is not an option where I live. To eliminate branding and ownership issues, I've never put my site on the college's Web server. So, instead, I found a fast, reliable hosting company that is halfway across the country, but has really fast servers, Tier 1 bandwidth, and great support. It's fast, I pay for it myself, and I have total control of it.