Search Facility - Improvements Needed

[talbot649]

I'm hoping someone can help improve the OSCommerce search facility, as I'm frustrated that when a customer enters a search term, if the operator is set to 'and' it will not return a result if they add anything that isn't explicitly in the description. Alternatively, if it is set to 'or' it will return hundreds of results. The latter would be fine, except that there is no relevance to the order.

 

Is there a way to display search results in order of relevance? ie results which contain more than one of the search terms would be at the top.

 

 

Any advice is really gratefully received

[arietis]

I'm hoping someone can help improve the OSCommerce search facility, as I'm frustrated that when a customer enters a search term, if the operator is set to 'and' it will not return a result if they add anything that isn't explicitly in the description. Alternatively, if it is set to 'or' it will return hundreds of results. The latter would be fine, except that there is no relevance to the order.

 

Is there a way to display search results in order of relevance? ie results which contain more than one of the search terms would be at the top.

 

my opinion is that this would require a major re-write. the reason relevance is difficult to obtain is that each item found by the search would need to be assigned some weighting, based on applying the search terms to that item. then, the items need to be sorted based on that weighting and then displayed in that order.

 

this is why search engines are pretty specific pieces of software. that weighting is a difficult algorithm. for example, if you sell clothes, and your search terms are 'cotton and orange' because you're looking for an orange t-shirt. which word is more important? is it the order of the words, so in this case 'cotton' has more weight than 'orange'?

 

as far as searching using the 'and' operator goes, you're telling it that you want both words to be found in the product description. if you have no orange cotton t-shirts, it shouldn't return anything. i believe that is correct behavior. if that's not what you want then you should search for 'cotton orange' instead of 'cotton and orange'.

 

some sort of a weighting algorithm would help when it returns a bunch of items using 'cotton or orange'. but weighting based on the number of times the word is used in the description would be more difficult. i think the current algorithm just looks for the existence of the word. finding the word and continuing to see if it's used more times in the description would make the search take even longer.

 

in order for that to work, you'd have to parse the descriptions and build a table of words from the description, with occurrence counts, per product, per language. then you'd search the word count table instead of the products description table. you'd also have to update the word count table each time you edit a product...removing any html in the product description while doing that. (after all, orange text in the description should not indicate that 'orange' was found as part of the the product description...)

 

yeah, it'd be nice. but this is going to take a few days of work to get it to go.

[talbot649]

this is why search engines are pretty specific pieces of software. that weighting is a difficult algorithm. for example, if you sell clothes, and your search terms are 'cotton and orange' because you're looking for an orange t-shirt. which word is more important? is it the order of the words, so in this case 'cotton' has more weight than 'orange'?

 

You've raised some really good points, but what I'm thinking of would hopefully be relatively simple.

 

I wouldn't necessarily want to go through the masses of work involved in assigning weightings to keywords, or calculating the number of keyword occurances in a description. What I was thinking of would be that search results which contain both 'cotton' and 'orange' would appear above results which contain only either of the keywords.

 

For example, say my store had 1000 products made of cotton, and 500 products coloured orange (but made of pvc, polyester etc), but only 10 products which were made of cotton and the colour orange. I'd be happy for the search results to display all 1500 results, but would like the 10 orange and cotton products to be the first 10 results.

 

I hope that makes sense! In my (relatively simple, non-skilled programming) mind that shouldn't be too difficult? Essentially, if you entered 3 search terms, the results which contained all 3 would be top of the results, results which contained 2 of the terms would be next, followed by the masses of results which contained just one.

 

If anyone can think of a way to implement this I'd be so grateful I would most likely explode :)

[talbot649]

Anbody fixed theirs in this way?

[spooks]

I think your main issue is that 'and' is the default search operator, so if 'and' is included in the keyword list it is used as a keyword, ie product must include 'and' to be listed.

 

Answer, remove 'and' from keyword list.

[♥kymation]

There is a way to do this, or something very like it, but it's a lot of work: Get a real search engine. You can use a service like Google, or add a PHP-based seqrch engine to your site. I took the latter approach by adding the open-source Sphider code. This also has the advantage that it will return useful results when your customer searches for something that is not on a product page, such as your Shipping terms. The disadvantages are in the time spent to set up the software, and in setting the search spider up to run periodically.

 

Regards

Jim

[talbot649]

There is a way to do this, or something very like it, but it's a lot of work: Get a real search engine. You can use a service like Google, or add a PHP-based seqrch engine to your site. I took the latter approach by adding the open-source Sphider code. This also has the advantage that it will return useful results when your customer searches for something that is not on a product page, such as your Shipping terms. The disadvantages are in the time spent to set up the software, and in setting the search spider up to run periodically.

 

Regards

Jim

 

 

That sounds like a plan. I'm happy to spend the time setting it up - I spend all my time playing with the site anyway! What would you recommend for a novice? Google?

 

In response to the post before ('removing the 'and' from the keyword lis') - I don't really understand what you mean...? I wouldn't want to set the default operator to 'and' as it would rely on a customer entering character perfect search terms for my products.

[♥kymation]

That sounds like a plan. I'm happy to spend the time setting it up - I spend all my time playing with the site anyway! What would you recommend for a novice? Google?

 

In response to the post before ('removing the 'and' from the keyword lis') - I don't really understand what you mean...? I wouldn't want to set the default operator to 'and' as it would rely on a customer entering character perfect search terms for my products.

Google seems to be fairly easy to set up. I haven't actually used it on a live site, just tested it. The search page doesn't end up looking much like the rest of your site. You can add your header image and that's about it. You might be able to put the Google results page in an Iframe and add the rest of your normal site navigation outside the frame. I haven't actually done this, so I can't say for certain that it will work, but it should.

 

The Sphider code is more complex, but you can customize it to look just like the rest of your site. You also have control over the way it works, so you can add additional emphasis to words in the title, for example. You can also block areas of the page that you don't want in the results, such as the site navigation. This gave me better results than Google, and quality results were the reason for doing this in the first place, so I went with Sphider.

 

Most modern search engines default to "and" for multiple words. People are used to this behavior, and may become frustrated if your search engine works differently. That won't affect correct spelling of terms anyway; the osCommerce search form relies on exact spelling. Google will suggest an alternative for words that it thinks are misspelled (sometimes incorrectly) and the Sphider engine can be modified to do custom suggestions. Again, I prefer to control this myself, rather than depend on an outside service getting it right.

 

Regards

Jim