Closed

Website Crawler - repost

We want to build a crawling software which will be able to extract products details from 10 sites in the USA and store it in an SQL database.

Software should have start, pause, stop functionality. Once fully run, the software should monitor for any changes in Pricing and Stock. If new URL or product is added it should add to the database.

We will start with one site and later extend the architecture to other 9 sites. I need the quote only for one site. Software can be in vb.net and MySQL database.

Please contact me only if you can prove you have done similar work before.

Some important features

Block Category for Extracting

In each site, I will create list of categories or sub categories from which the software should not extract any data

Fields will vary according to category. You need to come out with the fields which you will be extracting

Software should also store the category structure and URL of the product page

Images should be extracted from the site and stored in local folder. Corresponding path and file name should be stored in the database. If the product has got multiple images, the same needs to be stored. No more than 1000 images should be stored in single folder. Each single folder should not store more than 1000 sub folders. If the main folder reaches 1000 folders, then you should start creating folders inside another subfolder.

Folder Name Should start with ProductImage + Time example ProductImages181230

Each product should have a auto generated product code.

Export of Product

1. Software should extract the newly added data into a folder every 1 Hour

2. Software should extract the modified data into a folder every 1 Hour. Example Price and Stock

3. Same needs to be uploaded to an FTP location

We also need a separate Exe to do all settings. Example schedule settings, Block category, Image folder directory

Software should be able to connect to a proxy server to get the data's.

You can start the software buy building a proto type with screens and once I approve the proto type, you can start the database design. Once database design is approved you can start the coding work.

Skills: .NET, Data Mining, MySQL, Software Architecture, Web Scraping

See more: website database design software, web proxy directory, web new start creating, web design pricing usa, web design price hour, web design folder structure, web design exe, web crawler features, web crawler architecture, vb.net features, usa net, sql server price list, proxy directory list, features of a web crawler, directory web scraping, directory scraping software, data structure coding, data scraping from website software, creating website url, architecture of web crawler, Data crawling, website crawler, scraping crawler, proto, price crawler

About the Employer:
( 0 reviews ) Rijswijk, Netherlands

Project ID: #4287123

12 freelancers are bidding on average $481 for this job

SigmaVisual

I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$500 USD in 10 days
(125 Reviews)
7.5
mantislin

Hi sir, please check PM, thx Kimi.

$250 USD in 5 days
(135 Reviews)
6.9
proteamspb

Hi, we specialize in web crawlers. Our team is ready to start.

$420 USD in 6 days
(34 Reviews)
6.4
phpXpertbd

I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$690 USD in 10 days
(21 Reviews)
5.8
igorfreelancer

hi marcplatvoet. Everything is clear from your project description and I can do it with no problem. I'll produce one main app (or a windows service) and one configuration app, both written in .net framework. Scheduled More

$550 USD in 7 days
(12 Reviews)
5.7
truongngocthanh

Hello, I can do this task. Please see your PM for more detail. Thank you

$300 USD in 5 days
(38 Reviews)
5.6
raul27868

Hello, I think I can help you with your project. I'm specializing in website data scraping. I have developed software that performs data collection on the Web and automate processes. If you send me details More

$450 USD in 30 days
(13 Reviews)
4.9
SEOSEARCHTEAM

For more details please check PMB... Thanks

$388 USD in 8 days
(7 Reviews)
3.8
solutino

Hi. I am a php, mysql expert. I have done many such crawling algorithms . I will show you the sample links in pmb. Also if you provide me a sample link to get crawled, I will show you the crawled data with program onl More

$600 USD in 10 days
(2 Reviews)
2.9
agfjesus

Senior .NET Professional Developer in Web Scraping

$700 USD in 30 days
(0 Reviews)
0.0
mikenew16

Hi, I'm keen to do this work for you ... I'm happy to do it in VB or Python - you choose. Thanks Mike

$500 USD in 15 days
(0 Reviews)
0.0
ayazzaidi

Hi, I'm an experienced developer with multiple Microsoft certifications (MCP, MCTS). I have developed number of scrappers/crawlers in past and owns a deep experience on .NET development and Microsoft database systems More

$425 USD in 5 days
(0 Reviews)
0.0