Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : Google Sitemap. XML, good for spiders but not for visitors?

[taltos1]

Hello,
I am trying to get my bearings with some SEO. I am using Google Webmaster tools and I am working on a sitemap. I am going to use http://www.xml-sitemaps.com/ to create the XML file and then submit it.

Now my question is, the XML file is not that great for actual human users. It looks just like a bunch of text (unless I am doing something wrong). So is there a way to create a Google compliant XML sitemap that is usable by spiders AND users?

Also, is it best practice to create a new sitemap each time I add pages/content and then resubmit?

Thanks a lot

[RatePointGuy]

XML Sitemaps help spiders know what pages exist. This is especially helpful if you don't have pages that are organically accessible to the crawler via links. These XML based sitemaps were never created with the end-user in mind.

You'll have to have dueling strategies. One is your XML file which no user will ever see, the other is an HTML file that for example could also serve as your 404 not found page, which would contain links to popular pages on your website. If you do EVERY page in this version of the sitemap, you'll be forced to update this constantly, and you may have too many pages for it to be useful to an end-user. So just watch out for that.

You SHOULD update your XML sitemap periodically and use the optional tags to let Google know if pages have been updated / modified or if pages have been added or removed. Google automatically downloads your XML sitemap regularly and expects changes.

Originally Posted by taltos1

Hello,
I am trying to get my bearings with some SEO. I am using Google Webmaster tools and I am working on a sitemap. I am going to use http://www.xml-sitemaps.com/ to create the XML file and then submit it.

Now my question is, the XML file is not that great for actual human users. It looks just like a bunch of text (unless I am doing something wrong). So is there a way to create a Google compliant XML sitemap that is usable by spiders AND users?

Also, is it best practice to create a new sitemap each time I add pages/content and then resubmit?

Thanks a lot

[JRSEOMarketing]

I would recommend a "human" sitemap and an "XML" site map.

Create the human sitemap to look nice for your visitors, and the XML just for google. I would suggest resubmitting the XML site map every week or so. But in that week.. add 2 content pages.

[1boss1]

Sure make a HTML sitemap for your visitors but the XML is pretty useless. It doesnt help you rank and if you need it for spiders to crawl your site then you have bigger problems.

[taltos1]

Originally Posted by 1boss1

Sure make a HTML sitemap for your visitors but the XML is pretty useless. It doesnt help you rank and if you need it for spiders to crawl your site then you have bigger problems.

So why does Google's webmaster tool request an XML sitemap?

[1boss1]

They dont "request" you add one, they have an option to let Google know if your site does have one.

It does nothing to help you rank, Google themselves have stated this. If you actually "need" an XML sitemap so your site gets crawled properly it indicates your sites structure needs addressing.

A HTML sitemap on the other hand assists visitors, provides an internal backlink with anchor text to your inner pages which gives ranking gains plus provides an alternate route to your content to all search engines not just Google.

Implemented in to your 404 it provides numerous options for all engines to keep crawling, as well as acts as a navigational aid to help your visitors find what they were looking for instead of bouncing.

There are more technical reasons but thats the basics of it.

[Melnel]

Originally Posted by 1boss1

... but the XML is pretty useless. It doesnt help you rank and if you need it for spiders to crawl your site then you have bigger problems.

That may be true if you have a site of 25 pages with a good Pagerank, but if you have a new site of 10,000 or so pages an XML sitemap submitted to all the major search engines is a great help and it costs little to implement so why pass up one more useful tool?

You will still need other strategies to get all your pages indexed, and more still to get all those pages ranking, but don't discard ideas before you try them.

Example:
A new site with 12,000+ pages was created on Jan 8 this year and up until one week ago it had only 1264 pages indexed in Google, a Sitemap was submitted five days ago and today the site has 4490 pages indexed in Google.

I like xml sitemaps.

[1boss1]

Originally Posted by Melnel

That may be true if you have a site of 25 pages with a good Pagerank, but if you have a new site of 10,000 or so pages an XML sitemap submitted to all the major search engines is a great help and it costs little to implement so why pass up one more useful tool?

You will still need other strategies to get all your pages indexed, and more still to get all those pages ranking, but don't discard ideas before you try them.

Example:
A new site with 12,000+ pages was created on Jan 8 this year and up until one week ago it had only 1264 pages indexed in Google, a Sitemap was submitted five days ago and today the site has 4490 pages indexed in Google.

That's indication you're doing something wrong with your internal link architecture, page copy and site promotion. I kicked off a site at the beginning of January, and by the end of the month it had 76,800 pages indexed. So to only have 4,490 indexed all year is rather worrying.

I see on your own SEO Company you have inherent problems, like canonical URL's, poor filename selection such as dash, underscore, upper & lower case characters all in the one URL, your XML change frequency as Weekly when your pages last modified headers show months ago, 2 instances of Analytics code on the one page, displaying a W3C compliant badge when your page has errors, no text link to your homepage with your primary term as anchor on your subpages etc etc.

So i'm starting to see an indication why your other site is struggling to get indexed.

XML being a "great tool" as you say is a hindrance for me, you see the major engines are link based crawlers. By adding an XML sitemap it destroys valuable optimization data by obscuring how crawlers navigate your site. You may have a whole subsection that Google can't internally access due to a Javascript issue, incorrect link structure or many other reasons.

Would you rather know this data and be able to internally correct and optimize it, or throw a band-aid over a gaping wound and lead Google in by the hand to find these pages with an XML sitemap?

Of course you don't, you want authority/pagerank to flow naturally through your sites to all the right places complete with targeted anchor text.

As for don't knock something until i've tried it, i have most certainly done A/B split testing on XML/Non-XML sitemap versions and i own some big sites. Microsoft.com has just over a million pages, i have a site 3 times bigger than the Microsoft.com domain and no it doesn't have an XML sitemap and never will.

[Melnel]

Originally Posted by 1boss1

That's indication you're doing something wrong with your internal link architecture, page copy and site promotion. I kicked off a site at the beginning of January, and by the end of the month it had 76,800 pages indexed. So to only have 4,490 indexed all year is rather worrying.

I see on your own SEO Company you have inherent problems, like canonical URL's, poor filename selection such as dash, underscore, upper & lower case characters all in the one URL, your XML change frequency as Weekly when your pages last modified headers show months ago, 2 instances of Analytics code on the one page, displaying a W3C compliant badge when your page has errors, no text link to your homepage with your primary term as anchor on your subpages etc etc.

Not sure what pages you are looking at, but it sure as hell is not http://theseoshop.com. But even if it were there is absolutely no problem with dashes in URLs, Upper and lower case letters in URLs etc.

This may be news to you but it is no longer necessary to update your content every week in order to get indexed or to get rankings

So i'm starting to see an indication why your other site is struggling to get indexed.

XML being a "great tool" as you say is a hindrance for me, you see the major engines are link based crawlers. By adding an XML sitemap it destroys valuable optimization data by obscuring how crawlers navigate your site. You may have a whole subsection that Google can't internally access due to a Javascript issue, incorrect link structure or many other reasons.

I did not say its struggling to get indexed, nor do you have any knowledge whatever about that site, its topic, how many links are pointing at it etc etc, so IMO its very unprofessional to make judgements with no information.

Would you rather know this data and be able to internally correct and optimize it, or throw a band-aid over a gaping wound and lead Google in by the hand to find these pages with an XML sitemap?

Of course you don't, you want authority/pagerank to flow naturally through your sites to all the right places complete with targeted anchor text.

There you go again assuming things out of thin air. Submitting a sitemap is in no way an admission of site structures that need correction and I could give a damn about PageRank. IF you had any information besides your suppositions to guide you it might be that you would find that the site in question does just what its supposed to do in the way its supposed to do it.

As for don't knock something until i've tried it, i have most certainly done A/B split testing on XML/Non-XML sitemap versions and i own some big sites. Microsoft.com has just over a million pages, i have a site 3 times bigger than the Microsoft.com domain and no it doesn't have an XML sitemap and never will.

Pray enlighten us poor unwashed SEOs ( who just may have more years experience than you and may well have more sites ranking well than you have had hot dinners) about the results of your testing.

I really don't care how big you site it except to say that I have never seen any site that had a logical reason to have over a million pages.

[Melnel]

Originally Posted by 1boss1

...

Would you rather know this data and be able to internally correct and optimize it, or throw a band-aid over a gaping wound and lead Google in by the hand to find these pages with an XML sitemap?

....

Let me correct one more of your erroneous assumptions if I may.

I don't know how you create large XML sitemaps, but I do it by spidering all the pages and correcting any mistake found before the sitemap is submitted to Google.

I also find that when I submit xml sitemaps to Google through webmaster tools I get back information from them on any errors that they find, once again giving me additional information which is not available elsewhere.

I realize that you seem to think that your own opinions are god given, but I have to struggle a bit to gain the information that I post, but since I have been gaining this information in the process of making a living doing SEO for 13 years, I do have a bit stored up, and I gain more every day as the moderator of a well known SEO forum.

[1boss1]

You really don't understand do you?

Ok carry on, i can see your going great guns with your "Expert SEO" term with just 469k results.. Your about to break in to the top 30.

[Melnel]

Well someone doesn't seem to understand.

One of the first things you need to understand Josh, is that I really don't need to optimize my own site, I have plenty of business so your barbs seem kind of silly to me.

[hekwu]

Originally Posted by Melnel

Well someone doesn't seem to understand.

One of the first things you need to understand Josh, is that I really don't need to optimize my own site, I have plenty of business so your barbs seem kind of silly to me.

Actually, I think the bigger point is if your site is not setup correctly, why would (SHOULD) anyone else trust you with their site. In addition, if you start struggling with a customer's site and they get a second opinion.... I'm sure the second opinion will go to your site... and we know how that will turn out. lol

I know you don't like it, but 1boss1 makes some good points.

It was kinda funny reading you two go at it... I won't pick at the scab in fear of getting blood on my shoes... but it was funny.

[Melnel]

What on earth makes you think my "site is not set up correctly"?

The reason people trust me with their site is because I make it rank well not because of what my site does or does not do? Well perhaps if someday I have to "struggle" with a customers site that will happen but so far I do not have to struggle much and so far (13 years) no one has had to go for a second opinion.

[1boss1]

Originally Posted by Melnel

Well someone doesn't seem to understand.

One of the first things you need to understand Josh, is that I really don't need to optimize my own site, I have plenty of business so your barbs seem kind of silly to me.

I don't know who Josh is but anyway... So you are saying as an "Expert SEO" you don't need to optimize your site, and having it contain fundamental optimization errors is a great testament to your skill?

Fair enough, i don't "need" to optimize my sites or pages either because it occurs naturally with the creation of my documents due to it being second nature.

Originally Posted by Melnel

What on earth makes you think my "site is not set up correctly"?

See just some of the examples i listed before, everything from URL canonization to to your great XML sitemaps configured with a "weekly" change frequency when your page hasn't been modified since 2005.

As i said and you just don't understand, by adding an XML sitemap you are destroying valuable data about the natural indexing of your site. What's the point in getting a poorly linked page in to the index via an XML that doesn't have a hope in ranking? Wouldn't you rather be able to identify areas of a site that are having issues with the natural crawling process, and be able to optimize this area via deeplinks and structural alterations to achieve peak performance?

XML Sitemaps do NOT have any bearings on rankings, Google themselves have even said this.

Read some articles by people such as Rand from SEOmoz, Dave Naylor, Joost DeValk etc on XML sitemaps that reiterates exactly what i have said.

BTW when you hit front page for the term "SEO" i might listen to your "13 years" experience.