Robots & Indexing - Question on what you would do?

[networkdad]

I recently upgraded a website from using frontpage & htm (gasp, choke!) to a full blown e-com site using oscommerce/php.

 

However, the search engines still have the old links and many people are still clicking on them when doing searches. Obviously, the pages won't come up anymore because they are not there.

 

Currently, i have them going to a custom 404 error page, and from there, there is a link to the main page, as seen in my .htaccess file here:

DirectoryIndex default.php

ErrorDocument 400 /error_docs/400request.htm

 

What i want to know is, would it just be better to redirect all 404's to the main default.php page instead ???

 

Would this be more helpful for the search engines, or is it going to screw them up even more ?

 

Any suggestions would be helpful!

 

Thanks

~Jason

[mattice]

Why not direct them to the advanced_search.php while feeding it whatever they asked for. (Only works if you had common naming scheme on your html pages).

 

You take the REQUEST_URI and strip the last page off / any path that might be usefull to search on.

 

Mattice

[Mark Russell]

I just dealt with a similar issue. I moved my file structure up a level to root and renamed default.php to index.php.

 

My problem: I have SE listings that look like this:

 

www.mysite.com/catalog/default.php?cPath=22 (category page)

 

I needed requests to those links to result in:

 

www.mysite.com/index.php?cPath=xxx

 

I used mod_rewrite in /httpdocs/.htaccess as follows:

 

Options -Indexes +FollowSymlinks

RewriteEngine on

RewriteRule ^catalog/default.php$ /index.php [R=301,L]

ErrorDocument 404 /custom.php

 

The Rewrite rule does all the work. The the left hand side is a regex that states for links of the format catalog/... rewrite the URL to /index.php. The cPath=xxx is stripped off at the start of processing and reapplied to the rewritten index URL, so you don't really need to worry about that piece of the URL. R=301 is a redirect code and L signifies Last Rule to process.

 

Note: ErrorDocument is independent of mod_rewrite. It's my custom page for handling page not found errors.

 

This works really nice, but only on Apache servers with mod_rewrite installed/enabled.

 

Next steps - use mod_rewrite to simplify all my category and product page URLs to make my URLs more search engine friendly.

 

M

[mattice]

I've seen some discussion on that on Sitepoint (I believe)...

it basically stated that mod_rewrite and osCommerce for 'more simplefied' URI's

beyond the stock implemented SE friendly URLs is somewhat impossible...

had to do with all the GET / POST vars not being exactly 'schematic' throughout the design of osCommerce.

Or did you write that ? :)

[Mark Russell]

Naw, I didn't write that. Nor have I tried anything beyond the default to index rewrite yet. Not sure I follow the GET/POST issue, though. Then again, I'm really quite the noob at mod_rewrite.

 

M

[jon_l]

I have the same problem at the moment. I'm trying to get the courage up to move the shop to the root from the shop directory, but don't want to lose all the search engine traffic, or get 404's.

 

I was wondering if it would be possible to keep the current shop directory fully functioning alongside a root install, but changing all the links, so that as soon as someone clicks on the link it takes them to the root version. I haven't looked in to it yet, but wondered if it might be possible just by changing the paths in the config file.

 

Then to exclude the shop directory in the robots file to stop search engines indexing it in future. They should pick up on the root install from other links. Once the traffic on the old install is minimal, I'd remove it and replace with a simple 'this page has moved, click here'.

 

As regards the original question....I'd just put a page up as above saying that 'this page has moved, click here'. If you put a auto redirect in some search engines won't like it (e.g. Google).

 

Jon.

[Mark Russell]

The mod_rewrite solution is much more elegant than redirects or mirrored directories. It is definitely worth the look and effort. Unless of course, you're not on Apache or your host doesn't offer mod_rewrite.

 

M

[mattice]

Naw, I didn't write that. Nor have I tried anything beyond the default to index rewrite yet. Not sure I follow the GET/POST issue, though. Then again, I'm really quite the noob at mod_rewrite.

 

M

 

had something to do with the product_id being on different locations in the URI each time:

 

/product_info.php/pID/33

 

/product_info.php/bla/true/pID/33

 

etc.

 

The poster wanted to use mod_rewrite to chop out all the pages so he'd end up with yoursite.com/pID/33 on ALL pages (reviews, new etc).

As regards the original question....I'd just put a page up as above saying that 'this page has moved, click here'. If you put a auto redirect in some search engines won't like it (e.g. Google).

 

You are redirecting to a 404 doc along with the http header that comes with it, the searchengine will not follow that up anyway AFAIK.

You'd rather be serving a client that wanted a particular page then some spider that will not index the 404 anyway I guess.

[networkdad]

Thanks for all the helpful info - I'll probably test a few different ideas, and see which works best, as im guessing theres not going to be any perfect answer.