Website Crawler


We want to build a crawling software which will be able to extract products details from 10 sites in the USA and store it in an SQL database.

Software should have start, pause, stop functionality. Once fully run, the software should monitor for any changes in Pricing and Stock. If new URL or product is added it should add to the database.

We will start with one site and later extend the architecture to other 9 sites. I need the quote only for one site. Software can be in [url removed, login to view] and MySQL database.

Please contact me only if you can prove you have done similar work before.

Some important features

Block Category for Extracting

In each site, I will create list of categories or sub categories from which the software should not extract any data

Fields will vary according to category. You need to come out with the fields which you will be extracting

Software should also store the category structure and URL of the product page

Images should be extracted from the site and stored in local folder. Corresponding path and file name should be stored in the database. If the product has got multiple images, the same needs to be stored. No more than 1000 images should be stored in single folder. Each single folder should not store more than 1000 sub folders. If the main folder reaches 1000 folders, then you should start creating folders inside another subfolder.

Folder Name Should start with ProductImage + Time example ProductImages181230

Each product should have a auto generated product code.

Export of Product

1. Software should extract the newly added data into a folder every 1 Hour

2. Software should extract the modified data into a folder every 1 Hour. Example Price and Stock

3. Same needs to be uploaded to an FTP location

We also need a separate Exe to do all settings. Example schedule settings, Block category, Image folder directory

Software should be able to connect to a proxy server to get the data's.

You can start the software buy building a proto type with screens and once I approve the proto type, you can start the database design. Once database design is approved you can start the coding work.

Skills: .NET, Data Mining, MySQL, Software Architecture, Web Scraping

See more: Data crawling, website database design software, web proxy directory, web new start creating, web design pricing usa, web design price hour, web design folder structure, web design exe, web crawler features, web crawler architecture, usa net, sql server price list, proxy directory list, features of a web crawler, directory web scraping, directory scraping software, data structure coding, data scraping from website software, creating website url, architecture of web crawler, website crawler, scraping crawler, proto, price crawler

Project ID: #4286370

Awarded to:


Hi. I have a proven scraping solution that can be used as a base for your project. Please, see PMB.

$400 USD in 14 days
(2 Reviews)

21 freelancers are bidding on average $569 for this job


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$250 USD in 5 days
(120 Reviews)

Hi, Experts team of SEO/Wordpress/PHP/joomla/Drupal developers and designers. Thanks Gaurav

$600 USD in 20 days
(103 Reviews)

HI, Kindly see details in PMB Thank you

$250 USD in 4 days
(75 Reviews)

##### YOUR SEARCH ENDS HERE! ##### GET IT RIGHT THE FIRST TIME!. Check the message and contact us. SI TEAM - SI Team.

$750 USD in 21 days
(19 Reviews)

I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$350 USD in 5 days
(21 Reviews)

Hello, We understand your requirement . Please check your Private Message Box for more details. Thank you, Gourav

$750 USD in 15 days
(9 Reviews)

Hi, My offer is a windows desktop app for each website separately. More details and samples in my message. Thanks

$500 USD in 7 days
(30 Reviews)

Automation expert here. I can easily write such a bot/scraper.Please check your PMB.

$700 USD in 10 days
(43 Reviews)

hi sd2013. Everything is clear from your project description and I can do it with no problem. I'll produce one main app (or a windows service) and one configuration app, both written in .net framework. Scheduled checks More

$550 USD in 7 days
(10 Reviews)

//////////////SELL ITEMS////////////////

$250 USD in 1 day
(37 Reviews)

Hi, Experienced team here.We can deliver this crawler for you.And ready for a mockup also.Can you please respond to us.So that we can work upon it. Thanks and Regards Karan

$600 USD in 15 days
(14 Reviews)

hi, i can do this, have done many scraped and auto bots, last one i did was for with multithreads proxies lists, so letsme do this lets talk in skype fabiofelgueiras82

$700 USD in 15 days
(7 Reviews)

Hi, I have long experience in handling web scraping projects and can definitely take up this for you. Thanks

$650 USD in 10 days
(8 Reviews)

Check your Inbox. Thanks

$750 USD in 19 days
(10 Reviews)

Hi, please check your inbox for more details. Thanks!

$600 USD in 20 days
(9 Reviews)

Ready to start work please check your PMB.

$550 USD in 25 days
(2 Reviews)

Hi I am a very experienced web scrapper, and is very much capable of doing this job. I have recently just finished a web scrapping job, and have done these kinds of stuff since I was in college. Please see PMB for more More

$500 USD in 5 days
(1 Review)

Dear Employer, I did project like this for my master thesis, I will glad to work this project, anyone placing low bids I think they can't do this as well as you want, Note: my bid is for non-flash reference sites :)

$1000 USD in 45 days
(0 Reviews)

i can do it.

$850 USD in 20 days
(0 Reviews)

Hi, Im new here, but this is a project i can do no problem, i just wrote a program that monitior yahoo stocks via YQL. Questions... Do you need UI? Can I do it in c#? SQL 2008? What Will Be The Tables? If i unde More

$600 USD in 10 days
(0 Reviews)