From my understanding (not sure if they've changed this), the site performance feature in Google Webmaster is based on data aggregated from individuals with the Google Toolbar installed on their browsers with the PageRank feature on. This means that should someone on a dialup connection visits your site, his load speed data would be included in this performance data -- something I wouldn't consider to be an accurate measure of your host's/server's actual connection speed. Also, should your visitors come from the other side of the globe, it's obviously going to be slow, yet this will be calculated in Google's data. I recall Google also mentioning that this feature is not meant to be accurate and is aggregated from fewer than 100 data points (something like that).
There are a few tools out there like ones from Pingdom which will pull up your site from their servers and show you an item by item cascade, where certain areas are taking longer than normal to load. This could help with debugging any one area that's hanging and slowing down your site. But regarding Google's performance data, I wouldn't rely on them as the final say.