The structure of the site map is the following:
- about 50 cat1 categories
- about 50 cat2 categories of each cat1
- about 50 cat3 categories of each cat2
- about 50 pages for each cat 3
These aren't fixed numbers. Just so you get an idea of the size of the site (+6 million pages).
I can tell you the url of the site via PM.
I need the output in CSV format, with the following fields: name, type (cat1, cat2, cat3 or page), url.
You can split it in several CSV files if needed.