hosted by liquidweb


Go Back   Web Hosting Talk : Other Forums : Web Hosting Lounge : Google crawl very fast
Reply

Forum Jump

Google crawl very fast

Reply Post New Thread In Web Hosting Lounge Subscription
 
Send news tip View All Posts Thread Tools Search this Thread Display Modes
  #1  
Old 03-25-2012, 10:34 PM
hellind2 hellind2 is offline
Web Hosting Master
 
Join Date: Jul 2004
Posts: 2,360

Google crawl very fast


I made a post on a forum and within the minute my post was on Google. How does Google do that?

Possibly Google have classified all forum sites as real-time sites, and frequent these sites every minute, but only search for threads with recent creation date.



Sponsored Links
  #2  
Old 03-25-2012, 10:40 PM
InfiniteTech InfiniteTech is offline
Corporate Member
 
Join Date: Dec 2006
Location: Netherlands
Posts: 1,403
You do realise that they have 1000s (10K?) of bots crawling the web simultaneously, right?

You might also be interested in reading these pages:
http://en.wikipedia.org/wiki/Google_Panda
http://www.google.com/about/company/tech.html

__________________

Infinite Technologies
- Dedicated Servers with 32GB RAM and 64 IPs at EvoSwitch/ Netherlands



  #3  
Old 03-25-2012, 11:07 PM
SPINIKR-RO SPINIKR-RO is offline
Web Hosting Master
 
Join Date: Feb 2012
Location: Memphis, TN
Posts: 2,907
This forum alone probably has a Goog bot or two that never even leaves. They have a ton of crawlers, forums are so dynamic that you can almost always find multiple bots crawling all the time.

The more impressive thing in my opinion is how fast Google Alerts works.

__________________
hostingcove.com | Tennessee Based Hosting Provider.
cPanel Shared & Reseller Hosting - Domain Names
Join thousands of happy customers. Secure & Stable
HeroicVPS Premium KVM VPS. Ashburn / Phoenix


Sponsored Links
  #4  
Old 03-30-2012, 12:20 PM
SWH-Ritchie SWH-Ritchie is offline
VPS Hosting Guru
 
Join Date: Nov 2010
Location: /etc/profile
Posts: 1,471
god, how many servers they might have to run all these Bots?

__________________
Samrixe Web Hosting
Fully Managed VPS Hosting Powerful Dedicated Servers
www.samrixe.com sales@samrixe.com
1 Gbps Network Daily Backups Follow Us on Twitter

  #5  
Old 03-31-2012, 06:53 PM
loremar loremar is offline
Junior Guru Wannabe
 
Join Date: Jun 2011
Location: Cebu, Philippines
Posts: 58
Yes, they're very fast and they eat your resources very fast as well.

__________________
game development for noobs
Learn how to astral travel

  #6  
Old 04-01-2012, 06:30 AM
sempaiserver sempaiserver is offline
New Member
 
Join Date: Apr 2012
Posts: 2
yes i reconned it fast but needd good score or some sort of listing to get it listed to the right keyword

  #7  
Old 04-01-2012, 06:31 AM
JayNL JayNL is offline
Web Hosting Master
 
Join Date: Aug 2005
Location: The Netherlands
Posts: 3,279
All content we post on our website is in the SERPs within a minute. It's just how much Google likes your website

__________________
The three rules for a good hosting experience:
1) pay for it. 2) don't ask for gold when you pay nickles. 3) get real.

Don't PM me, we have nothing to discuss

  #8  
Old 04-01-2012, 07:49 AM
ddosguru ddosguru is offline
CISSP-ISSMP, CISA
 
Join Date: Aug 2002
Location: Los Angeles, CA
Posts: 5,427
Quote:
Originally Posted by hellind2 View Post
I made a post on a forum and within the minute my post was on Google. How does Google do that?

Possibly Google have classified all forum sites as real-time sites, and frequent these sites every minute, but only search for threads with recent creation date.
Google will scan your sitemap .xml for changes and then crawl thereafter.

__________________
.._(_)_.. Black Lotus Communications - AS32421
(_)@(_) >> Carrier grade DDoS mitigation for service providers and enterprises
....(_)..... >> 480 Gbps active DDoS filtering capacity available in Los Angeles and Ashburn facilities

  #9  
Old 04-01-2012, 08:30 AM
Patrick Patrick is offline
Security Ninja
 
Join Date: Mar 2003
Location: Canada
Posts: 8,614
Google knows what you're thinking.

__________________
Patrick William | Rack911 Research Labs | Software Security Auditing
250+ Vulnerabilities Found - Get a quote on a professional audit @ Rack911.com

www.HostingSecList.com - Security notices for the hosting community.

  #10  
Old 04-01-2012, 08:44 AM
hellind2 hellind2 is offline
Web Hosting Master
 
Join Date: Jul 2004
Posts: 2,360
Quote:
Originally Posted by IRCCo Jeff View Post
Google will scan your sitemap .xml for changes and then crawl thereafter.
Posts on WHT, for example, are updated to Google near-instant. Does WHT, or any other vbulletin forums, have 'recent posts' accessible to search engines.

One method Google could be using is to incrementally look up this link:
http://www.webhostingtalk.com/showpost.php?p=8042067

  #11  
Old 04-01-2012, 08:54 AM
InfiniteTech InfiniteTech is offline
Corporate Member
 
Join Date: Dec 2006
Location: Netherlands
Posts: 1,403
Quote:
Originally Posted by hellind2 View Post
Posts on WHT, for example, are updated to Google near-instant. Does WHT, or any other vbulletin forums, have 'recent posts' accessible to search engines.

One method Google could be using is to incrementally look up this link:
http://www.webhostingtalk.com/showpost.php?p=8042067
https://www.google.com/webmasters/tools/home?hl=en

Feed it in there.

__________________

Infinite Technologies
- Dedicated Servers with 32GB RAM and 64 IPs at EvoSwitch/ Netherlands



  #12  
Old 04-01-2012, 09:28 AM
hellind2 hellind2 is offline
Web Hosting Master
 
Join Date: Jul 2004
Posts: 2,360
Quote:
Originally Posted by InfiniteTech View Post
Does WHT webmasters feed it every few minutes?

  #13  
Old 04-01-2012, 11:50 AM
InfiniteTech InfiniteTech is offline
Corporate Member
 
Join Date: Dec 2006
Location: Netherlands
Posts: 1,403
Quote:
Originally Posted by hellind2 View Post
Does WHT webmasters feed it every few minutes?
I can't speak for them. I don't know.

But you could, for your own website.

__________________

Infinite Technologies
- Dedicated Servers with 32GB RAM and 64 IPs at EvoSwitch/ Netherlands



  #14  
Old 04-01-2012, 11:53 AM
SPINIKR-RO SPINIKR-RO is offline
Web Hosting Master
 
Join Date: Feb 2012
Location: Memphis, TN
Posts: 2,907
You can set your desired crawl rates but it by no means that they will obey them. If you set a fast crawl rate with a site that does not have anything unique or is not updated that fast it will ignore it.

__________________
hostingcove.com | Tennessee Based Hosting Provider.
cPanel Shared & Reseller Hosting - Domain Names
Join thousands of happy customers. Secure & Stable
HeroicVPS Premium KVM VPS. Ashburn / Phoenix


  #15  
Old 04-02-2012, 01:42 PM
himanuzo himanuzo is offline
Newbie
 
Join Date: May 2008
Location: Asia
Posts: 21
Profiles below:

Google usually crawls them slowly or doesn't crawl them.... you need to ping them in order to force Google to crawl them faster.

__________________
<<Please see rules for signature setup>>

Reply

Similar Threads
Thread Thread Starter Forum Replies Last Post
How Does Google Crawl To the Website? dotsquares SEO / SEM Discussions 12 11-09-2011 04:30 AM
90%+ google crawl error??? guruproj SEO / SEM Discussions 5 07-03-2011 09:15 AM
let google crawl your contents twice Peter-SexyWing SEO / SEM Discussions 6 04-14-2010 03:10 AM
Google Crawl HOD-Jardin SEO / SEM Discussions 17 09-17-2009 08:01 AM
how to speed up google crawl vynx_1999 SEO / SEM Discussions 8 03-07-2008 05:06 PM

Related posts from TheWhir.com
Title Type Date Posted
Mistakes To Avoid When Launching A New Website Design Blog 2014-01-10 10:53:50
Google Cloud Provides Support For Native MySQL Connections Web Hosting News 2013-11-01 14:36:06
Google Offers 15GB of Free Storage Across Cloud Services Web Hosting News 2013-05-13 15:00:03
Why You Should Embrace Google Plus Blog 2013-03-08 09:18:17
Google Expected to Launch IaaS at I/O Developers Conference this Week Web Hosting News 2012-06-28 14:38:03


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes
Postbit Selector

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump
Login:
Log in with your username and password
Username:
Password:



Forgot Password?
Advertisement:
Web Hosting News:



 

X

Welcome to WebHostingTalk.com

Create your username to jump into the discussion!

WebHostingTalk.com is the largest, most influentual web hosting community on the Internet. Join us by filling in the form below.


(4 digit year)

Already a member?