An existing recursive python crawler using beautiful soup and sqlite needs the following fixed:
The crawler is quite brutal during parsing and recursive threads are sometimes grabbing the same links. Each running thread should instead grab a diffrent link, not the same one.
The crawler needs to be made less "brutal" and needs to be tested on a limited site-only crawl and on open internet-wide crawl to see if both work correctly.
**IMPORTANT:**
Please quote the full name of the inventor of Python for your bid to be considered.
The 3 days deadline WILL be increased on the request of the worker IF there is sufficient progress observed during the first 3 days (and each successive deadline extension) of the project.
The employer will not act as a tester for the worker. Testing of the code to be up-to-spec is a requirement of this project. Employer will test the final deliverables once the worker is truly confident the deliverables work up-to-spec.