My first project here. However I've used other freelance sites before.
I have a gallery directory script that diplays a listings of links on the output pages. Right now the output pages work like this; I create a list of keywords in the admin area. For each keyword, the script will create a [url removed, login to view] script. When a surfer visits [url removed, login to view] page, it will query the database, find gallery links with description that matches the keyword string and displays them. It also creates spanning pages i.e. [url removed, login to view], [url removed, login to view], etc depending on how many gallery records in the database that matches the keyword string. There are templates and settings in the admin to customize how the output pages looks and max spanning pages. Also to prevent the [url removed, login to view] pages from querying the database everytime a surfer visits the page, all [url removed, login to view] pages caches itself in a text file for a certain amount of time. So when someone visits any [url removed, login to view] page, it first checks to see if there is a current cache, if there is, then it will just display the cache version and doesn't need to query the database. If there is no cache or if the cache has already expired, it will query the database, display the output results, and cache itself for a specific amount of time (Currently 6 hours).
Here is the problem: Although the cache works, when there are lots of records in the database, I mean 100,000 or above, it takes a long time to query the database, cache itself, and display results. Of course if a [url removed, login to view] page has already been cached it displays the output pages pretty fast because it doesn't have to query the database. I need to have the cache expire because that way output pages and results are constantly updated and fresh when I add new records to the database.
Here is what I need: I need a cron script that I can run daily (or any amount of time) to do one of the following depending on what would be the best solution.
1. Query the database, use the template files and settings, and build all [url removed, login to view] including all spanning pages as static pages. I would run the cron daily and everytime it runs it will overwrite any existing [url removed, login to view] output pages and build new static pages. That way I can have fresh output pages daily, and it won't load slow to the surfers since it doesn't have to do any database queries.
2. A cron script that will visit all [url removed, login to view] pages and have them cache itself. I would have to run this cron script every 6 hours when caches expires. That way the [url removed, login to view] pages doesn't have to wait for a surfer to visit it to cache itself.
I prefer to have the first option. The cron script has to be coded in a way so that it doesn't overload the server when building the output pages, because I will probably have over 1000 keywords and 25 spanning pages for each keyword. So it will be building over 25,000 pages daily.
Private Message me if you have any questions before bidding. Thanks!