We are looking to Scrapy expert to do the following;
We will provide you login access to a machine.
Your tasks are following using Scrapy
We want to get data from a specific site and the following things need to happen
1) We go to the site then we need to search (obvious but....)
- This can be partial search, a wild card or combination (like increment a number, letter combinations like abc etc.)
- This is to get all the data on the site, we call this as a sitemap.
2) Then the same script should have an API class, which means if we want to update a specific record we should use reg number or something to get the latest data of that record.
Since we have done the scraping in option 1. We know this unique reg number and we can pass that to the class.
3) Scrapy capture deployment dashboard, where we can see the running scripts.
4) Then use the Scrapy pipeline to load the data to MySQL.
5) You should have incorporated Selenium & Beautifulsoup with Scrapy.
In short, we are looking for someone who has done all the above. We need you to give the framework, We want you to build one script.
You need to transfer the knowledge to us, we have junior/mid level Python programmers, who understand all the above.
We will then build individual scrapers from the framework.
If you have standard libraries like extracting data from a grid, error handling. Please include.
We have multiple developers in different places, so if this is built in a docker or something it would be great for deployment and replicate.
We have a GitHub you can upload code to that as well.
We are looking at a very capable person. Price can be negotiated. Before we award, you need to demo a system like that you have developed before.
You can quote a fixed amount as well.