Closed

Scrapy scraper

This project was awarded to kamosion for $22 USD / hour.

Get free quotes for a project like this
Employer working
Awarded to:
Project Budget
$15 - $25 USD / hour
Total Bids
15
Project Description

We're looking for a developer who's experienced in using the Scrapy framework to develop spiders to scrape airline websites. For this project, we are specifically going to scrape the flight details from 3 sites namely:

[url removed, login to view]
[url removed, login to view]
[url removed, login to view]

The scraped data will be standardized and loaded to a separate MongoDB datastore. We already have a framework that utilize an API (also coded in Python) that interfaces with a distributed architecture for spiders using IronWorker. Javascript knowledge is a must since most of these sites utilize JS to load their content.

We have developed other spiders already thus we expect the time spent on the project actually writing the spiders versus learning the architecture thus we will need proof of capabilities.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online