Web Hosting Talk : Other Forums : Web Hosting Lounge : Google crawl very fast

[hellind2]

I made a post on a forum and within the minute my post was on Google. How does Google do that?

Possibly Google have classified all forum sites as real-time sites, and frequent these sites every minute, but only search for threads with recent creation date.

[InfiniteTech]

You do realise that they have 1000s (10K?) of bots crawling the web simultaneously, right?

You might also be interested in reading these pages:
http://en.wikipedia.org/wiki/Google_Panda
http://www.google.com/about/company/tech.html

[SPINIKR-RO]

This forum alone probably has a Goog bot or two that never even leaves. They have a ton of crawlers, forums are so dynamic that you can almost always find multiple bots crawling all the time.

The more impressive thing in my opinion is how fast Google Alerts works.

[SWH-Ritchie]

god, how many servers they might have to run all these Bots?

[loremar]

Yes, they're very fast and they eat your resources very fast as well.

[sempaiserver]

yes i reconned it fast but needd good score or some sort of listing to get it listed to the right keyword

[JayNL]

All content we post on our website is in the SERPs within a minute. It's just how much Google likes your website

[IRCCo Jeff]

Originally Posted by hellind2

I made a post on a forum and within the minute my post was on Google. How does Google do that?

Possibly Google have classified all forum sites as real-time sites, and frequent these sites every minute, but only search for threads with recent creation date.

Google will scan your sitemap .xml for changes and then crawl thereafter.

[Patrick]

Google knows what you're thinking.

[hellind2]

Originally Posted by IRCCo Jeff

Google will scan your sitemap .xml for changes and then crawl thereafter.

Posts on WHT, for example, are updated to Google near-instant. Does WHT, or any other vbulletin forums, have 'recent posts' accessible to search engines.

One method Google could be using is to incrementally look up this link:
http://www.webhostingtalk.com/showpost.php?p=8042067

[InfiniteTech]

Originally Posted by hellind2

Posts on WHT, for example, are updated to Google near-instant. Does WHT, or any other vbulletin forums, have 'recent posts' accessible to search engines.

One method Google could be using is to incrementally look up this link:
http://www.webhostingtalk.com/showpost.php?p=8042067

https://www.google.com/webmasters/tools/home?hl=en

Feed it in there.

[hellind2]

Originally Posted by InfiniteTech

https://www.google.com/webmasters/tools/home?hl=en

Feed it in there.

Does WHT webmasters feed it every few minutes?

[InfiniteTech]

Originally Posted by hellind2

Does WHT webmasters feed it every few minutes?

I can't speak for them. I don't know.

But you could, for your own website.

[SPINIKR-RO]

You can set your desired crawl rates but it by no means that they will obey them. If you set a fast crawl rate with a site that does not have anything unique or is not updated that fast it will ignore it.

[himanuzo]

Profiles below:

Ted
1FC
G2A
About
MPP
Learnem
Ocmoto
Smadaver
3GN
Xenagia
Com2us
Chirs
FGB
VTF
PSO
PSA
LP
EP
NMF
Conceiva
WF
SD
Nrmrwib
GCOJ
Jub
UPM
Approva
OS
WNC

Google usually crawls them slowly or doesn't crawl them.... you need to ping them in order to force Google to crawl them faster.