Web Hosting Talk : Web Hosting Main Forums : SEO / SEM Discussions : Sitemap Configure & Install

[jlsrpierce]

I am a newbie at setting up a website. My website has static pages, a forum which I think is called dynamic content, and javascript for the menu.

I registered my website with Google, MSN, Yahoo but they don’t index my site very well so I went to http://www.google.com/webmasters/ and logged in and Google recommends that I install an industry standard “Sitemap” to help web crawlers locate all the pages on my website and presumably the posts in my forum. Google’s directions on setting up a “Sitemap” are located at https://www.google.com/webmasters/to...or.html#config

I downloaded the latest Sitemap version 1.4 from Google’s link to SourceForge.net, and am working my way through Google’s directions on creating a configuration .xml file, (Step 2).

Question 1) There are several methods of generating a “Sitemap”, which do you recommend for my website?

Question 2) I think Access Logs is the correct methos, but I can’t find my access.log file(s) on my Hostgator server for my website, where do I look?

Question 3) Once I have the Sitemap working will it index both my static pages and my Forum?

Thanks for your support, John.

[headout]

Maybe you can use xml-sitemaps.com/, which is very easy to use. Just upload your sitemap.xml; log into your google account and point to your newly uploaded xml file.

[jlsrpierce]

Thanks headout, so why doesn't Google recommend something that simple?

[zionchild]

Your forum can be indexed quickly after you have several links pointing to your site. I understand that you start with content and then build your links. Google will find you through back links.

[jlsrpierce]

OK, back to this topic:

1) I went to https://www.google.com/webmasters/tools and looked at google's crawl results and all looks OK except Not Found = 20, so looks like my old sitemap_config.xml is out of date as the 20 pages that google was trying to crawl no longer exist.

2) So I resubmitted my sitemap_config.xml file after which google said error.

3) So I just went to http://www.xml-sitemaps.com and generated a sitemap.xml file for my website, FTP'd it to my root directory and told google to use this new file, google says "Status Pending".

Questions:

Q1) Is http://www.xml-sitemaps.com the best way to go?

Q2) That website while generating said for example:

Links depth: 8
Current page: forum/index.php?action=profile;u=2;sa=showPosts;attach;sort=a.filename;start=150
Pages scanned: 1400 (22,634.0 KB)
Added in sitemap: 418
Pages left: 52
Time passed: 14:53
Time left: 0:33

So it looks like it was indexing my SMF forum as well as my website correct?

Q3) That website at end of my sitemap.xml generation said "Maximum 500 pages Limit Exceeded", suggested that I try their $20 standalone sitemap generator, and that:

Request date: 24 May 2008, 11:48
Processing time: 950.93s
Pages indexed: 500
Pages size: 23.68Mb

I don't have anywhere near 500 webpages, it must have indexed my forum until it hit it's max 500 page mark, correct?

Q4) Do I really need spend the $20 and index over 500 pages?

Q5) Given that the forum section changes daily, how often do I need to rerun and re-FTP and re-tell google that new sitemap.xml file is ready for it to check?

As you can tell by my questions, I am quite lost here, thanks for advice . . .

[jpozadzides]

John,

Seriously, I have to recommend that you scrap the hand built Web site in lieu of a widly supported open source application like WordPress (with BBPress for the forums), Joomla, or some other.

These applications have plugins for them that take care of the indexing issue, and they will allow you to concentrate on being a Webmaster, not just writing code all the time. :-)

John

[eCommercePro]

Hi,

It's very like lyyour forum was indexed. However, you can stop google crawling it by adding an entry in the robots.txt file of your root directory.

There is no fixed rule as to how often to resubmit the new sitemap to google, but the frequency of the crawler visiting your site will still depend on google and it's own algorithm.

However, if your application supports RSS generation, you can also just submit the RSS feed to google for indexing, it's much less hassle than generating the sitemap manually.

If you still wish to generate the sitemap yourself without the use of an open source platform and its plugins, have a look at phpSitemapNG.

Hope this helps.

[xworker]

you can find free software for making sitemap.

[mrphp]

Google has also a sitemap example. Go to google.com and signup at webmaster area.

Reards,
Mark

[Xaurora]

Thanks for this post... very helpful information here!