Closed

Need a script to scrape craiglist ads with my keywords in every city

This project received 13 bids from talented freelancers with an average bid price of $257 USD.

Get free quotes for a project like this
Employer working
Project Budget
$30 - $250 USD
Total Bids
13
Project Description

Hello,

I'm looking for software that will scrape a craiglist category of my choice, looking for my specific keywords. THe search would work exactly how their search on their site works, to find related posts. For example, I pick "sporting" category. I would then enter keywords, such as, "basketball" "baseball" "soccerball".
It would have up to 10 separate lines for search So for the example I used, it would find all posts for basketball, then find all posts that contain baseball, then all that contain soccerball. And in the end it will make a long list of all the posts it finds. I need it to remove duplicates. And I need it to be able to be date specific. So if I want June 1 - June 15, I can do that. And on the output file, it will show the dates I chose. I would also like to be able to save this file, so I can open it up when I need. I would like to be able to mark it as "replied" if and when I reply to the ad. I also want to have a box for "does not contain" beside the search box. So if I'm searching for "soccerball", and for "Does not contain" I put "red", it will filter out any posts that are selling a red soccerball.

When you do a search it usually also lists "here is what we found nearby", which lists results from cities that are close. Since we are doing each city, it would probably be best if we could remove that from scraping, as we'll end up with dozens of duplicate posts, that will eventually have to be filtered out anyways. If there's no easy way to do that, than whatever. I just wanted to give a heads up that the search results do that.

I also think it might be a good idea to set timers for the search, as I'm sure they don't what 1000s of search queries in minute. Either that or I'll need an option to use proxy servers for search.

So for the output, I want to have a long list, with the post titles separated in groups by their search term, ie "baseball" "soccerball", etc.
Then when I click the ad, it will bring me to the online ad. It will also have the date beside it.

This project may be a Windows desktop application, or an online script, whichever is easier and cheaper. This is for personal use, so I don't care too much how the program actually looks.

Let me know if you have any questions.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online