Jump to content
  • Checkout
  • Login
  • Get in touch


The e-commerce.

Forced Labour for Spiders


Recommended Posts

I am a heavy user of page-cache as it reduces my server load dramatically and "server" is a big word for what I have. Yet, using page-cache has one inherent problem, it takes away much of the dynamics of the site.


I have many "unique" products which, when sold, need to show "sold" without the buy button. But that is not


possible on the cached pages. So I wrote a php program which deletes all cache files and execute that whenever


a product sell makes it go from available to sold. But, depending on the number of cached files, that process


may take anything from 2 seconds to 60 seconds and that is simply added to the response time from confirmation


to success, basically punishing the person who buys.


So I thought, why not let one of those spiders, who are always present, take that burden?


They will also receive that additional response time but they have no emotional reaction to that and they are not buying anything anyway.


So now I use a job control table with switches which get set realtime and which trigger incoming spiders to take the associated task rather than the real customer.


I use it for most tasks which need direct execution (rather than time scheduled) but would add substantial to online response times. For now, cache clearing and also-purchased pre-selection processes.


This is how:



in application_top right after the spider identification:


// spider only executed jobs

if ($spider_flag) {

include(DIR_WS_INCLUDES . 'spider_jobs_switchboard.php');




file spider_jobs_switchboard.php:



// get the switchboard data

$spider_jobs_query = "select cache_clear,


from spider_jobs ";

$jobs_query = tep_db_query($spider_jobs_query);

$jobs_switches = tep_db_fetch_array($jobs_query);

if ($jobs_switches['cache_clear']) {

// reset the switch

tep_db_query("update spider_jobs set cache_clear = 0");

// perform the clearing of the cache files job

include(DIR_WS_INCLUDES . 'clear_cache.php');

} elseif ($jobs_switches['ap_preselect']) {

// reset the switch

tep_db_query("update spider_jobs set ap_preselect = 0");

// perform the also purchased preselection job

include(DIR_WS_INCLUDES . 'ap_preselect.php');




The Job Control table:



CREATE TABLE spider_jobs (

cache_clear tinyint(1) unsigned default '0',

ap_preselect tinyint(1) unsigned default '0'


INSERT INTO spider_jobs (cache_clear) VALUES("0");

INSERT INTO spider_jobs (ap_preselect) VALUES("0");



in checkout_process when a unique product goes from available to sold:


// set the spider job switch for cache clearing

tep_db_query("update spider_jobs set cache_clear = 1");


and whenever we have a sale by a registered customer:


// set the spider job switch for ap_preselection refresh

tep_db_query("update spider_jobs set ap_preselect = 1");

// set the spider job switch for cache clearing

tep_db_query("update spider_jobs set cache_clear = 1");

Treasurer MFC

Link to comment
Share on other sites


This topic is now archived and is closed to further replies.

  • Create New...