Real time scraper / browser automation middleware

This project received 12 bids from talented freelancers with an average bid price of $732 USD.

Get free quotes for a project like this
Employer working
Project Budget
$250 - $750 USD
Total Bids
Project Description

I'm looking to create a scraper that will act as middleware/API to a front end app. Thus the data will need to be returned via JSON. The site to be scraped is only 1 airline site with AJAX/Javascript all over so it is essential to have the skills needed to:
1. parse & manipulate Javascript
2. circumvent anti-scraping techniques
3. Scalable - since every user who logs in the front end will create an instance of the scraper

Frontend controls -> Scraper Middleware -> Airline site
- Thus the middleware should scrape/parse the necessary data and send it to the front end in real time.

Flight data to be provided - airport code, domestic/international, # passengers, date of departure, date of arrival

The use case for the scraper middleware is:
1. User interacts with front end to search for a flight
2. Front end triggers scraper middleware to run (a headless browser/selenium/etc).
3. Scraper middleware will then wait for the results to load and pass back the data to the front end.
4. User in front end selects flights to book and triggers scraper to load these info to the airline site.
5. Airline site provides confirmation and PDF tickets.
6. Scraper middleware passes back these data to the front end.

Before you submit a bid, please send me a proposed system architecture/framework for the scraper. Please include services/database tools etc.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online