Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

.htaccess redirect and robots.txt confilct


Recommended Posts

Hi all.

I discovered that my new shop website is not working and it seems I have an .htaccess and robots.txt conflict that I am unable to sort out.

First I noticed that Google Sitemap XML was not able to access my robots.txt, after some help from Jack_mce and my host technical support, it was found that the ,htaccess redirect was causing the problem.

Obviously I need both of these files for my site to work properly, can anyone suggest a fix for this .htaccess file.

My shop is in the public_html folder along with the robots.txt and .htaccess.

The .htaccess file is

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.retro-antiques-curios.co.uk/$1 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]
RewriteCond %{REQUEST_URI} !^/RAC_Shop [NC]
RewriteRule ^(.*)$ /RAC_Shop/$1

 

My robots.txt file is

# Robots.txt

 

User-agent: *

Disallow: /includes/

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_newsletters.php

Disallow: /account_notifications.php

Disallow: /account_password.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_process.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /logoff.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /redirect.php

Disallow: /shopping_cart.php

Disallow: /user.css

sitemap: https://www.retro-antiques-curios.co.uk/RAC_Shop/sitemapindex.xml

 

Thanks in advance Shaun

Running BS Edge.

Link to comment
Share on other sites

I'm not totally sure I understand your site structure but assuming the site is actually in the subdirectory /RAC_Shop/ I can see a couple of issues

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]

redirects any file or directory name that doesn't exist sending it as a parameter to /index.php

I guess this is url rewriting but if your shop is in a subdirectory then the rewrite should point to the subdirectory:

RewriteRule ^(.*)$ /RAC_Shop/index.php?/$1 [L]

The other issue is with the last redirect - when anyone looks for robots.txt they get redirected to the subdirectory so you need an extra rule to exclude the file:

RewriteCond %{REQUEST_URI} !^/robots.txt$ [NC]

put before the last rewrite rule

Contact me for work on updating existing stores - whether to Phoenix or the new osC when it's released.

Looking for a payment or shipping module? Maybe I've already done it.

Working on generalising bespoke solutions for Quickbooks integration, Easify integration and pay4later (DEKO) integration at 2.3.x

Link to comment
Share on other sites

Hello John

Thank you for your code, that worked perfectly, you are a star, I can now call my robots.txt file.

However in my Google webmaster account when I attempt the robots.txt tester I get an error Fetch Failed!

Could this be anything to do with my original problem, or will I need to start a new thread about Google Webmaster?

Kind regards Shaun

 

Link to comment
Share on other sites

My guess is it's related.

So when you go to Crawl / robots.txt Tester you probably see the results of a previous fetch attempt. If you click on See live robots.txt does that work ok?

If so, try clicking submit at the bottom so it attempts another fetch.

Contact me for work on updating existing stores - whether to Phoenix or the new osC when it's released.

Looking for a payment or shipping module? Maybe I've already done it.

Working on generalising bespoke solutions for Quickbooks integration, Easify integration and pay4later (DEKO) integration at 2.3.x

Link to comment
Share on other sites

Again your instructions worked perfectly, thank you John, the robots.txt tester is now working.

Hopefully this matter is now resolved.

I will get back if I discover any more errors or problems.

Shaun

 

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...