How to include Yahoo in my robot.txt file

[Trentide]

Hi thank you for looking at my post. Right now I'm blocking Google from indexing certain pages and want to do the same with Yahoo. I've been getting a lot of bots indexing my cookie_usage page. What a pain!

 

Does anyone know how to include Yahoo in my robot.txt file?

 

Thanks so much for your help!~

 

Jason

[Guest]

In your Admin sessions section it is not set up properly

 

Force Cookie Use False

Check SSL Session ID False

Check User Agent False

Check IP Address False

Prevent Spider Sessions True

Recreate Session False

 

Your robots.txt file blocks pages such as your SSL pages but if the bots are hitting cookie useage page try the Sessions setting above. The bots you are referring to are in includes/spiders.txt file and can be updated from the contributions section.

[Trentide]

Thanks. I did have to change force cookie use to false. Let's see if that works.

 

Also, right now, I have:

 

User-agent: Googlebot-Image

 

at the top of my robot.txt file.

 

Shouldn't I just include a Yahoo user agent in the file to keep Yahoo bots from accessing certain files?

 

Thank you so much,

 

Jason

 

In your Admin sessions section it is not set up properly

 

Force Cookie Use False

Check SSL Session ID False

Check User Agent False

Check IP Address False

Prevent Spider Sessions True

Recreate Session False

 

Your robots.txt file blocks pages such as your SSL pages but if the bots are hitting cookie useage page try the Sessions setting above. The bots you are referring to are in includes/spiders.txt file and can be updated from the contributions section.

[Guest]

User-agent: Googlebot-Image

 

That is just for Google images. Bots like Yahoo are controlled with your spiders.txt file

[Trentide]

OK, thank you. I made the changes. I hope it works!

 

Jason

 

That is just for Google images. Bots like Yahoo are controlled with your spiders.txt file

[Trentide]

I think now I'm getting a lot of visitors that go to pages with this type of URL"

 

html?filter_id=23&sort=2a&osCsid=40cc55ab0a021100ce0196ecd4ca02e8

 

Is that normal?

 

Jason

 

 

 

OK, thank you. I made the changes. I hope it works!

 

Jason