I'm building a website that needs to crawl another website and find all of the web pages within that site. Much like a xml sitemap generator, such as [url removed, login to view]
However, I don't want it to output to a sitemap. Instead I'd like the function to simply write out each URL in a new line on a web page, such as [url removed, login to view]("URL....<br>"). I can then take the code from there.
So the basic premise is, I define a website URL and then the code you write will need to crawl that website and find all of the URL s within that website. You will need to code the app to read and follow HTML links (both followed and nofollowed) that point internally.
I don't want sub domain URLs or third party domain URLs, only URLs within the domain I detailed at the beginning.
The app needs to be reasonably quick, as fast as it can be but I prefer it to be reliable and thorough over fast. I need to ensure I get all of the URLs in the site.
I'd also like this code pretty quick if possible. Within 5 days would be ideal.
It needs to be VB and in ASP.NET 4.0 and in a website project, i.e. like the one attached.
I'll pay on completion of the code and when it has been tested.