Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Robot visit more than customers?


noob

Recommended Posts

i have a visitor contribution installed that records all the visits,their ip,and other info

 

since late august till now i had 5032 visits out of which 1557 were robots

 

they use to not come as often but lately thers atleast 1 everyday

 

is it normal?

 

what should i do

Checkout my osc contrib here!

You can also checkout the site im working on by clicking the card button!

Link to comment
Share on other sites

its set to true and has been set to true since the store started

but the session id's are still created <_<

Checkout my osc contrib here!

You can also checkout the site im working on by clicking the card button!

Link to comment
Share on other sites

Make sure that part of the USER AGENT of the spider is in your sipders.txt file.

 

You can find the user agent of the spider by looking in your raw logs, or in your 'latest visitors' link in your website cpanel.

-------------------------------------------------------------------------------------------------------------------------

NOTE: As of Oct 2006, I'm not as active in this forum as I used to be, but I still work with osC quite a bit.

If you have a question about any of my posts here, your best bet is to contact me though either Email or PM in my profile, and I'll be happy to help.

Link to comment
Share on other sites

i looked thru the logs and these r some of i gotthat were getting id's:

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Mediapartners-Google/2.1

msnbot/0.3 (+http://search.msn.com/msnbot.htm)

 

is that what i add to spiders.txt?

Checkout my osc contrib here!

You can also checkout the site im working on by clicking the card button!

Link to comment
Share on other sites

:angry:

I have Admin>Sessions>Prevent Spider Sessions=true and I have the latest spiders.txt and robots.txt but I am still being plagued by the M$ bot spidering my store. M$ is every day, sometimes all day long. My site is being killed by the M$ spider.

 

What can I do? I don't have this problem with Google. Google seems to be behaving pretty well.

 

Help.

 

Rick Knight

Link to comment
Share on other sites

Can you tell me what you are talking about and explain to me how to check my site for such problems? Thanks,

-g.b.

 

 

:angry:

I have Admin>Sessions>Prevent Spider Sessions=true and I have the latest spiders.txt and robots.txt but I am still being plagued by the M$ bot spidering my store. M$ is every day, sometimes all day long. My site is being killed by the M$ spider.

 

What can I do? I don't have this problem with Google. Google seems to be behaving pretty well.

 

Help.

 

Rick Knight

Link to comment
Share on other sites

Can you tell me what you are talking about and explain to me how to check my site for such problems?  Thanks,

-g.b.

 

If you have User Tracking installed, take a look at it. It gives some stats about page hits for your site. Look for googlebot.com and msnbot.com listed in the stats. Look at the connect and disconnect times and whether a session is created either of them.

 

On my site this morning msnbot was connected and indexing my site, it's still connected. It will disconnect at some point tonight, but then reconnect tomorrow. It's almost allways connected and indexing my site and it's having a major (negative) impact.

 

I think I found my problem though, I had the robots.txt in the wrong place. I just moved it and restarted my server, so tomorrow I'll know if it is fixed or not.

 

Rick Knight

Link to comment
Share on other sites

  • 3 months later...

You said about spider.txt and robots.txt, i cant seem to find it in my files. Could you tell me where I can find this my my file manager? I also have set the spider session to true, but I dont know how to check it if its really functioning well.

 

Thanks.

 

:angry:

I have Admin>Sessions>Prevent Spider Sessions=true and I have the latest spiders.txt and robots.txt but I am still being plagued by the M$ bot spidering my store. M$ is every day, sometimes all day long. My site is being killed by the M$ spider.

 

What can I do? I don't have this problem with Google. Google seems to be behaving pretty well.

 

Help.

 

Rick Knight

Link to comment
Share on other sites

You said about spider.txt and robots.txt, i cant seem to find it in my files.....

Thanks.

 

robots.txt sits in your root directory. you must create your own robots file to allow or disallow which bots you want and where.

 

everything you wanted to know about robots.txt but were afraid to look !

 

spiders.txt is for osCommerce. look inside your osCommerce files for it. look inside the folders inside your osCommerce files.

 

get the latest and greatest spiders.txt here !

 

if you somehow didn't get a spiders.txt with your oscommerce cart just get the one at the contribution page.

Link to comment
Share on other sites

Thanks for the information. I have seen the spider.txt and is now looking and reading for the robot.txt.

 

Though I have another issue.

 

I dont know how to do the file. If there is a someone who can give me a sample one that I refer to.

 

Thanks

 

 

robots.txt  sits in your root directory. you must create your own robots file to allow or disallow which bots you want and where.

 

everything you wanted to know about robots.txt but were afraid to look !

 

spiders.txt is for osCommerce. look inside your osCommerce files for it. look inside the folders inside your osCommerce files.

 

get the latest and greatest spiders.txt here !

 

if you somehow didn't get a spiders.txt with your oscommerce cart just get the one at the contribution page.

Link to comment
Share on other sites

robots.txt  sits in your root directory. you must create your own robots file to allow or disallow which bots you want and where.

I have oscommerce in a sub domain. I only need a robots text file for the sub as there are no sensitive files in my domain. Do I put it in the root of my subdomain or the root of the domain?

Link to comment
Share on other sites

I have oscommerce in a sub domain. I only need a robots text file for the sub as there are no sensitive files in my domain. Do I put it in the root of my subdomain or the root of the domain?

Put it in the root of the domain and set accordingly if you can view your server log you will find the files that the robots spiders are scannings.

If you see the robots spiders are scanning where they should not be scanning add the url address in to the robot text file to deny there access you can down load robot text from.

 

http://www.oscommerce.com/community/contri...l/search,robots

( WARNING )

I think I know what Im talking about.

BACK UP BACK UP BACK UP BACK UP

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...