phatweb Posted January 10, 2004 Share Posted January 10, 2004 Google is hammering my site... does this mean they are indexing my site or what? looks like 60+ of them on my site.. all ips resolve to crawlerxx.googlebot.com Ideas? Link to comment Share on other sites More sharing options...
burt Posted January 10, 2004 Share Posted January 10, 2004 You are being spidered. Link to comment Share on other sites More sharing options...
♥Vger Posted January 10, 2004 Share Posted January 10, 2004 Either you are being spidered, or those are your own entries showing up - as you work on the site online. The IP addresses shown on the image do not conform to google ip addresses. To prevent the whole of the site being spidered (by responsible search engines) set up a robots.txt file and disallow entry to all but the top level pages. e.g User-agent: * Disallow: /cgi-bin/ Disallow: /private/ Disallow: /images/ Just put the above, plus other directories you don't want spidered, into a simple text file, name it 'robots.txt' and load it into the html area of your site. Hope this helps - Vger Link to comment Share on other sites More sharing options...
Tarquin3 Posted January 10, 2004 Share Posted January 10, 2004 How do you manage to post up the screen shot like that?? Link to comment Share on other sites More sharing options...
phatweb Posted January 10, 2004 Author Share Posted January 10, 2004 I did printscreen then pasted into paint and saved as a .jpg. Uploaded to webserver and attached using the "IMG" function on the forum. Link to comment Share on other sites More sharing options...
dumb_question Posted January 11, 2004 Share Posted January 11, 2004 Is there any reason we shouldnt allow spiders to see our images. I have alt text on my images . Should they be looking at that? Link to comment Share on other sites More sharing options...
dyland Posted January 11, 2004 Share Posted January 11, 2004 You want the spiders to index your images so you can show up in image only searchs currently available on Google, Yahoo, Ask Jeeves, etc. Only block spidering images if you're worried about bandwidth - it shouldn't be an issue. If you have small and large images (i.e. zoom in images), you could experiment with blocking just the larger image. Dylan Link to comment Share on other sites More sharing options...
dumb_question Posted January 11, 2004 Share Posted January 11, 2004 thanks dylan Link to comment Share on other sites More sharing options...
New 
Recommended Posts
Archived
This topic is now archived and is closed to further replies.