Results 1 to 18 of 18
  1. #1

    IPSec and googlebot problem

    Hi,
    3 months ago I've moved about 30 sites from old server to new server with different IP and then all the pages in all sites are listed with url only on Google search results! No title, no desc, just url! Also I've checked log files and there was no googlebot traffic at all! What hapened??? (But I can see other spiders from other search engines come and index).

    The new server is Windows 2003 server with IPSec enabled. The following ports are opened as I think it a must for hosting service:

    -TCP 53 (DNS IN & OUT)
    -UDP 53 (DNS IN & OUT)
    -TCP 80 (IN)
    -TCP 443 (https IN)
    -TCP 3389 (Terminal service IN)
    -TCP 21 (FTP service IN)
    -TCP 110 (POP3 Out)
    -TCP 25 (STMP IN)
    -TCP 1433 (SQL IN)

    I blocked all ICMP traffic (in & out), so the server does not reponse from ping requests.

    I guess that not my server configuration problem (because other spider can visit and index!) But why no goolebot!

    The 30 sites were very good ranking on google (usually from page 1 to page 3 on google result) before moving to new server. I also checked blacked list to see if my server IP was banned from google (thanks god there no my IPs on that list!).

    So that's not server configuration problem, not IP banned problem, not search engine cloaking problem, and then ...what is really the problem?

    The only thing I can guess is blocking ICMP traffic may cause trouble with googlebot! (But I dout about it!)

    I've waiting for googlebot 3 months! Nothing hapened! Does google forget my sites?

    Anybody has the same problem with me? Please help me!

    Thanks a lot

  2. #2
    This is just a stab in the dark but might the google bot be ping'ing each server before it connects? Which you just drop and so it moves on to the next link it's scraped. Does disabling icmp really buy you anything anyway? Might be worth re enabling it.

    Pete

  3. #3
    Thanks jesta. Actually I think do we really need to block ICMP traffic or not?

  4. #4
    Join Date
    Apr 2002
    Posts
    321
    Interesting. It's the first time I've heard of an IP change causing a site to get yanked from Google. Have you tried writing them?

    And no, it has nothing to do with ICMP. We block ICMP except on the mail servers IP address (some mail server software requires echo requests to verify a remote server), and have no problems with Google ranking.

  5. #5
    yes, I've sent email to tell Google at [email protected] about my problem, and they suggested I should read guideline!

    I know there's no Google cloaking here (because these site were good ranking on Google before moving to new server!), and there no blocking for standard web traffic (because other search engine can find and index my sites! MSN bot, Yahoo bot ...etc are indexing like crazy on my new server every day!)

    So what is the problem? My customers continue asking me why their sites dont have good result (actually no result at all) on Google!

    Any idea? I'm going crazy now!

    Thanks for replying

  6. #6
    Join Date
    Apr 2002
    Posts
    321
    Are you sure the IP's are not blacklisted somewhere? Been used by someone else before, who might have done bad things?

  7. #7
    Hi JDMundo, I'm pretty sure there no my IPs on Google's blacklisted. Why? Because when I search on google like this:
    site:mysite.com, it will return all pages for that site! But with url only instead of title, desc like other search results.

    Same for all my sites in the new server. According to Google, if we search like this: "site:mysite.com", and if it says: there no result at all, then I'm sure that my site was banned. But as I said, there were results with my sites, but just url! People can not read url to understand what site is about :-)

    I don't know what should I do to get googlebot come and index again!

    Life is not the same without Googlebot!!!

  8. #8
    Join Date
    Nov 2004
    Location
    UK
    Posts
    774
    It could be coincidence.

    The Googlebot is now very, very slow. Unfortunately it's possibly slower than most other search engines on the planet. It now takes on average around 3 months for a new site to be listed! Google is having quite a few problems with their search technology at the moment. I think it's something to do with when they doubled the index to 8billion pages (although I'm not sure).

    The Googlebot should not require any other ports than port80. It functions like an "automatic web browser" that clicks any and all links it sees. If a browser can see the page, Googlebot should also be able to see it (offtopic: apart from frames and iframes).

    My suggestion: try contacting Google again. They've been really helpfull to us when we've had similar problems being indexed by Google.

  9. #9
    Hi shockuk,

    Yes!!!!!!!!!! "If a browser can see the page, Googlebot should also be able to see it " You are completely right! As I think, if the browser can see, so does the google! But why too long! Someone said because it uses IP cached for about 1 month, and I've waiting for 3 months! Anyway, I will try to contact Google again for my problem

    Thansk shockuk

    (Actually I always think we don't have to open unecessary port like ICMP for googlebot!)

  10. #10
    Chances are it has nothing to do with your ICMP or other server configs. Google saw the URL change to a new IP and knows there *may* have been a transfer of ownership for the site, and that it needs to be re-indexed. URLs that Googlebot knows about but have not been fully indexed (read, parsed, cached, etc) are listed without a description and such. Eventually Googlebot will catch up in its queue of sites to visit and fil in the details.
    Dan Grossman - dan @ awio.com
    My Blog | Affiliate Program for Web Hosts

  11. #11
    Thanks Dan Grossman, just because i've waiting too long so i thought maybe google forgot me!

    For your opinion, how long will google come and index again in this situation?

  12. #12
    Join Date
    Apr 2005
    Location
    silicon and earthquakes
    Posts
    255
    I have blocked ICMP recently and changed the IP and Google sees my sites just fine.

  13. #13
    Join Date
    Nov 2004
    Location
    UK
    Posts
    774
    According to Google, they update the index everytime the Googlebot has crawled 8,000,000,000 web pages. Iv'e no idea how long on average this takes

  14. #14
    Hi rotoiti, i'm glad to hear that , because this is my first server so I don't have experience for this problem.

    Thanks

  15. #15
    Hello conkien2004

    i have the same problem with my site. it's only 1 week, and I'm going crazy when i think it will continue more than 3 months :-(
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  16. #16
    as i told the same history. site which is hosted in one VPS account (first hoster) and had problems with overloading. [3-5% uptime]

    i try to move to the second hoster, and after 1 day hosting i remove my site back (=> to the first hoster).

    after this, my site continue overloading, and 5 days ago i check that site start work without any problem.

    check stats and understand , that there ins't any googlebot visits and any link from google (in latest visitors section).

    if before this i got ~ 1500-2000 visitors from google. now 0 visitors (or 3-5 visitors).

    google drop all my keywords from SERP.

    u don't tell about ur keywords which work before this remove.
    do they work now?
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

  17. #17

    * Googlebot is comming in!

    Just one hit today but a big hit for me

    Thanks all, you really help me not do stupid things to get Googlebot visit my server!

  18. #18
    Just one hit today but a big hit for me
    hmm :-) it very interesting ... because i get few googlebot fisits too ... (with a little traffic - 20-30mb)
    Your Health Encyclopedia
    Medical and health consumer information resources containing comprehensive and unbiased information in patient-friendly language

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •