Hello Coders!
?
I am looking for an application that will crawl and scrape [login to view URL] and [login to view URL] to retrieve Related Searches (detailed steps to follow).
**Both engines have API but I am not sure if you can get related searches. If this is the case and you can use the API, then this would be a much better solution!!!**
API information:?
[login to view URL]
[login to view URL]
?
?
## Deliverables
If you will be scraping, here is what I would like the program to do:
?
1) User enters:
- A list of search terms
- Dig level
- Checks if he wants Ask and/or MSN results (checkbox)
?
2) Application will go to [login to view URL] and make a query with the first term
3) Application will get the link text of all the "Related Searches" on the right part of the page
4) Application will also click on each of the "Related Searches" to get more results. If the dig level is 3, then the application will "dig" 3 times
5) Do this for all search terms
6) Repeat steps 2-5 from the US version of MSN if user selected MSN.
?
7) After the process is finished, list all the terms scraped in a txt file or txt box for easy copy/pasting:
- Remove duplicates
- List in alphabetical order (under Ask and MSN boxes)
- List in alphabetical order (both engines)
?
The application can be either desktop-based or web-based (php, etc.).