HTML Static Page creator
What we’re trying to do is to let ecommerce or other big site owners, make their dynamic pages viewable for the search engine spiders by creating a static html page for each dynamic page they have. The way this would work is that, this will be a spider which will go to the user’s site, then visit each link on the site, copy the html of each page into local file and save it as html. Again, this should visit every single page on any site, as long as there is a link to a page, it would be seen. The link could also be from drop down menu choices and any other possible way, again this should work for those situations too.
if interested, please read the doc. There are many freeware code to use for this but I need complet spider which won't miss any links and would also copy html in its format.
Here is an article that talks about the problems and solutions, you have to take care of all the problems.