Need a script to scrape craiglist ads with my keywords in every city
$30-250 USD
Closed
Posted almost 11 years ago
$30-250 USD
Paid on delivery
Hello,
I'm looking for software that will scrape a craiglist category of my choice, looking for my specific keywords. THe search would work exactly how their search on their site works, to find related posts. For example, I pick "sporting" category. I would then enter keywords, such as, "basketball" "baseball" "soccerball".
It would have up to 10 separate lines for search So for the example I used, it would find all posts for basketball, then find all posts that contain baseball, then all that contain soccerball. And in the end it will make a long list of all the posts it finds. I need it to remove duplicates. And I need it to be able to be date specific. So if I want June 1 - June 15, I can do that. And on the output file, it will show the dates I chose. I would also like to be able to save this file, so I can open it up when I need. I would like to be able to mark it as "replied" if and when I reply to the ad. I also want to have a box for "does not contain" beside the search box. So if I'm searching for "soccerball", and for "Does not contain" I put "red", it will filter out any posts that are selling a red soccerball.
When you do a search it usually also lists "here is what we found nearby", which lists results from cities that are close. Since we are doing each city, it would probably be best if we could remove that from scraping, as we'll end up with dozens of duplicate posts, that will eventually have to be filtered out anyways. If there's no easy way to do that, than whatever. I just wanted to give a heads up that the search results do that.
I also think it might be a good idea to set timers for the search, as I'm sure they don't what 1000s of search queries in minute. Either that or I'll need an option to use proxy servers for search.
So for the output, I want to have a long list, with the post titles separated in groups by their search term, ie "baseball" "soccerball", etc.
Then when I click the ad, it will bring me to the online ad. It will also have the date beside it.
This project may be a Windows desktop application, or an online script, whichever is easier and cheaper. This is for personal use, so I don't care too much how the program actually looks.
Let me know if you have any questions.
We have wide experience in web scrapping, we specialize in data-mining software. VB 6, VB.Net desktop applications. Yellow pages, forums, classified sites, directories and others are some of the areas we've developed crawlers for. Our software runs fine against common protections, uses anti banning features, it's user customizable, allows direct URL scraping, amongst other features. Please check PMB for samples of our work
Hi sir,
I am ready to start I will provide you a PHP application which can be use as a website or as a desktop application .
Please let me know on which site you want to search.
Thanks
Proweb solutionz
Dear anon989,
I can quickly and cost effectively develop a Windows client based automated system to quickly and accurately scrape craiglist ads with your specified keywords in every city for you.
An automated solution to your project will most likely be much faster than anyone else doing this all manually, which will therefore require fewer hours to complete and therefore ultimately Save You time and Money!, you know?
Additionally, as an automated system can run 24 hours per day by 7 days per work, whereas someone cannot work continuously doing this manually, it is likely that an automated system will complete your project much faster than someone doing this manually, you know?
Please find/see demo videos of and sample results from some of the similar automated systems I have developed in my Portfolio.
Please Hire me for your project(s) and I can quickly and cost effectively develop and run a similar automated system for you now!
I look forward to hearing from you and working together with you towards the successful completion of your project soon!
- Michael S. DeVries
Hi there, I recently wrote software which scrapes information from OLX, so getting it to work with Craigslist is no problem. My software inserts the scraped data into an online database, and I can export this to excel if needed. I also developed a text messaging module which enables you to send promotional text messages to these people. The software marks off which people have received a text message, so in your case this would just mark off which ads you have dealt with. Best Regards, Curt