Need help with hadoop crawler? Hire a freelancer today! Do you specialise in hadoop crawler? Use your hadoop crawler skills and start making money online today! Freelancer is the largest marketplace for jobs in the world. There are currently 17,764 jobs waiting for you to start work on!
I am looking for a guy who can realize the following project:
In Google, the following search string is inserted:
"[search keyword] site:*/impressum"
What then shall be done is the following:
- The urls of all findings are extracted
- The urls shall be saved in a DB
- The urls are cut that way that just the domain name with the tld (with http://www. in front) is saved in the DB...
- Develop crawler/scraper that is easy to customize to new webshop
- Customize crawler/scraper for...every week
I suggest we setup system to run crawler from AWS. Store feeds in S3.
Maybe you have a
...an intelligent spider/web crawler. Basically the web crawler will crawler through a list of 10 to 30...in translating them.
I want to use a web crawler to minimise data bandwidth and time (ie. I don't