Website Crawler

Budget $250 - $750 USD
Bids 23
Average Bid $585

We want to build a crawling software which will be able to extract products details from 10 sites in the USA and store it in an SQL database.

Software should have start, pause, stop functionality. Once fully run, the software should monitor for any changes in Pricing and Stock. If new URL or product is added it should add to the database.

We will start with one site and later extend the architecture to other 9 sites. I need the quote only for one site. Software can be in [url removed, login to view] and MySQL database.

Please contact me only if you can prove you have done similar work before.

Some important features

Block Category for Extracting

In each site, I will create list of categories or sub categories from which the software should not extract any data

Fields will vary according to category. You need to come out with the fields which you will be extracting

Software should also store the category structure and URL of the product page

Images should be extracted from the site and stored in local folder. Corresponding path and file name should be stored in the database. If the product has got multiple images, the same needs to be stored. No more than 1000 images should be stored in single folder. Each single folder should not store more than 1000 sub folders. If the main folder reaches 1000 folders, then you should start creating folders inside another subfolder.

Folder Name Should start with ProductImage + Time example ProductImages181230

Each product should have a auto generated product code.

Export of Product

1. Software should extract the newly added data into a folder every 1 Hour
2. Software should extract the modified data into a folder every 1 Hour. Example Price and Stock
3. Same needs to be uploaded to an FTP location

We also need a separate Exe to do all settings. Example schedule settings, Block category, Image folder directory

Software should be able to connect to a proxy server to get the data's.

You can start the software buy building a proto type with screens and once I approve the proto type, you can start the database design. Once database design is approved you can start the coding work.

Post a Project Like This

This project was awarded to

Mikhail19

Ultimate professional. Smart, intelligent. Careful to think about maintenance of programs.
About the Freelancer
Mikhail19 Profile Picture

C#, .NET, Web Scraping

Looking to make some money?

  • Set your budget and the time frame
  • Outline your proposal
  • Get paid for your work

Bids on this Project

  • igorfreelancer Profile Picture

    igorfreelancer

    Zagreb,  Croatia

    I am a coder of 25 years with extensive experience in the following: - C - C++ - .NET - and SQL. I am a database administrator and system developer. I specialize in data mining, algorithm design, optimizations and CPU intensive tasks.

    ASP, C Programming, Java, and Visual Basic

  • mesuttalebi Profile Picture

    mesuttalebi

    istanbul,  Turkey

    My name is Mesut, I am 33 years old, living in ─░stanbul / Turkey. I got my Master Degree in Karadeniz Technical University, Compter Engineering. I'll glad to work with employers from all around the world.

    ASP, .NET, Website Design, and Data Processing

  • SigmaVisual Profile Picture

    SigmaVisual

    Taxila Cantt,  Pakistan

    Web, Bots, Crawlers, and Scrapers Development. I have expertise in automation services and I can automate any manual process.

    PHP, Python, SEO, and Data Entry

  • bodmin Profile Picture

    bodmin

    Ashqelon,  Israel

    12 years experience in VBA, VB, VB.NET, ASP.NET

    Javascript, XML, Visual Basic, and .NET

  • jitendraparmar07 Profile Picture

    jitendraparmar07

    indore,  India

    we provide highly professional and skilled solution in 1.) Financial software solutions specialized in forex market 1.1) Expert Advisers , Indicators and Scripts ( MQL 4/5 / C# / C++ ) 1.2) Algorithmic Trading Software ( Desktop and Web Apps ) 1.3) Forex Strategy Development ( MQL 4/5 / C# / C++ ) 1.4) APIs Development ( Advanced algorithmic calculations) 1.5) Platform - Meta Trader 4 / 5 , Ninja Trader 2.) Automated web scraping and data mining solutions 2.1) Using Java , C# and PHP . 2.2) HTML Unit , Selenium and all leading technologies 2.3) Manage CMS - Magento,WordPress ,Joomla .

    PHP, C Programming, Java, and .NET

  • NiceDeveloper Profile Picture

    NiceDeveloper

    Yazd,  China

    I'll stay with you until you're completely satisfied...even after your project is finalized!

    PHP, Javascript, iPhone, and Android

  • innovese Profile Picture

    innovese

    Ghaziabad,  India

    algoscale technologies was started keeping in mind the tremendous potential in the big-data field and being a buzzword of the industry. It is about data which has its presence in every sphere of technological domain. Having expertise in the arena, we are dedicated to help businesses in solving their big-data problem, which includes (not limited to) setting up sophisticated infrastructure to meet their scale needs, building out a cost-effective technology platform using advanced open-source technologies along with our proprietary tools. Apart from setting and deploying these systems, we help businesses in aggregating the data flowing within their system and using it to derive insights & thus helping in taking business decisions based on actionables and measurable metrics.

    PHP, Java, Javascript, and Python

  • ehsankayani Profile Picture

    ehsankayani

    Islamabad,  Pakistan

    I am a C#/ Asp(.)net (Website and Software Developer). And have strong knowledge in the skills listed below: 1- C#, Javascript, Asp(.)net, jQuery , Html,Css 2- Windows Services 3- Web Services 4- Windows Form Application,Windows Console App ( Desktop Application For Windows) 5- Automated Softwares (Ex . Linkedin Bot to Post on Groups, Send Messages etc) 6- Scraping Tools, Crawlers , Web Spiders. 7- Sql server (05,08,12) , Visual studio (10,12) , 8- Access / Excel VBA Tools 9- And at the moment learning Xamarian , for Android / IOS & windows phone development. You will see reviews on my profile from all the skills listed above and many more and every review marks my services as 100 % perfect. Also i always provide quality work in less price and on time. You can see that from my project reviews listed on this profile. Also i am almost online 17 + hours a day so my response time is very quick . Thank you

    PHP, .NET, Internet Marketing, and Banner Design

  • ValueCoders Profile Picture

    ValueCoders

    Noida,  India

    Skysoftworld is a NCR (INDIA) based Software development & IT Services provider. (Established in 2007) We are ISO 9001:2008 certified for our quality and management processes. More than 500 projects successfully delivered till date (150+ here on freelancer itself). 60+ Team size (Dedicated Teams for design, QA and development, backed by Team leads and project manager) Dedicated customer support department. Technology: .NET Stack : asp.net, Dotnetnuke (evoq), MVC, kendo UI, kentico CMS, Umbarco CMS, Sharepoint PHP Stack : Joomla, Drupal, Codeignitor, Wordpress, MVC Mobile Stack : Android, IOS, Windows Phone eCommerce : Magento, Prestashop, NBStore, VirtueMart, WooCommerce Client Side : Node.js, canvas etc Verticals: Real estate, Finance, eCommerce, Retail, Virtual Currencies (bitcoin etc), Single Sign ons (SSO) etc

    PHP, ASP, Javascript, and Adobe Flash

  • akhila27 Profile Picture

    akhila27

    Eldeniya,,  Sri Lanka

    Hi, My name is Akhila Praveen. You can call me Akhil. I'm the Managing Director of 'Solution Infinity'. What may you ask ? Solution Infinity is a place where along with my dedicated team members we undertake your software and application needs and develop them for you, from the nut to the shell,even better than you imagined it would be. I know not all are talented programmers, and yes it is hard to do your programming alone, or sometimes you discover your knowledge is not enough and cant wait till you gain it, or even worse your a non-IT/programming person looking for a Soft ware/application solution for your business/PC. That is where we step in, we will take your burden from those shoulders and provide you what you will admit is simply the best IT solution that can be catered. Our charges you ask ? No worries, All this doesn't include a bill like you dined at Hilton hotels, We will be charging fair, fitting to your budget. So why don't you take a peak around ? there are some project's we've done, and my crew at work, plus there is also my contact info you to contact me for your IT needs. Remember - Solution Infinity, the Solutions are Infinite!!!

    PHP, C Programming, Java, and Javascript