I'm looking for software that will scrape a craiglist category of my choice, looking for my specific keywords. THe search would work exactly how their search on their site works, to find related posts. For example, I pick "sporting" category. I would then enter keywords, such as, "basketball" "baseball" "soccerball".
It would have up to 10 separate lines for search So for the example I used, it would find all posts for basketball, then find all posts that contain baseball, then all that contain soccerball. And in the end it will make a long list of all the posts it finds. I need it to remove duplicates. And I need it to be able to be date specific. So if I want June 1 - June 15, I can do that. And on the output file, it will show the dates I chose. I would also like to be able to save this file, so I can open it up when I need. I would like to be able to mark it as "replied" if and when I reply to the ad. I also want to have a box for "does not contain" beside the search box. So if I'm searching for "soccerball", and for "Does not contain" I put "red", it will filter out any posts that are selling a red soccerball.
When you do a search it usually also lists "here is what we found nearby", which lists results from cities that are close. Since we are doing each city, it would probably be best if we could remove that from scraping, as we'll end up with dozens of duplicate posts, that will eventually have to be filtered out anyways. If there's no easy way to do that, than whatever. I just wanted to give a heads up that the search results do that.
I also think it might be a good idea to set timers for the search, as I'm sure they don't what 1000s of search queries in minute. Either that or I'll need an option to use proxy servers for search.
So for the output, I want to have a long list, with the post titles separated in groups by their search term, ie "baseball" "soccerball", etc.
Then when I click the ad, it will bring me to the online ad. It will also have the date beside it.
This project may be a Windows desktop application, or an online script, whichever is easier and cheaper. This is for personal use, so I don't care too much how the program actually looks.
Let me know if you have any questions.
13 freelancers are bidding on average $257 for this job
I specialize in web scraping. I can meet you requirements on this project. Be wary of low bids from inexperienced programmers, I have scraped craigslist before and I can complete this job properly.
Hi sir, I am ready to start I will provide you a PHP application which can be use as a website or as a desktop application . Please let me know on which site you want to search. Thanks Proweb solutionz