Website Crawler - repost

Budget $250 - $750 USD
Bids 12
Average Bid $481

We want to build a crawling software which will be able to extract products details from 10 sites in the USA and store it in an SQL database.

Software should have start, pause, stop functionality. Once fully run, the software should monitor for any changes in Pricing and Stock. If new URL or product is added it should add to the database.

We will start with one site and later extend the architecture to other 9 sites. I need the quote only for one site. Software can be in [url removed, login to view] and MySQL database.

Please contact me only if you can prove you have done similar work before.

Some important features

Block Category for Extracting

In each site, I will create list of categories or sub categories from which the software should not extract any data

Fields will vary according to category. You need to come out with the fields which you will be extracting

Software should also store the category structure and URL of the product page

Images should be extracted from the site and stored in local folder. Corresponding path and file name should be stored in the database. If the product has got multiple images, the same needs to be stored. No more than 1000 images should be stored in single folder. Each single folder should not store more than 1000 sub folders. If the main folder reaches 1000 folders, then you should start creating folders inside another subfolder.

Folder Name Should start with ProductImage + Time example ProductImages181230

Each product should have a auto generated product code.

Export of Product

1. Software should extract the newly added data into a folder every 1 Hour
2. Software should extract the modified data into a folder every 1 Hour. Example Price and Stock
3. Same needs to be uploaded to an FTP location

We also need a separate Exe to do all settings. Example schedule settings, Block category, Image folder directory

Software should be able to connect to a proxy server to get the data's.

You can start the software buy building a proto type with screens and once I approve the proto type, you can start the database design. Once database design is approved you can start the coding work.

Post a Project Like This

Looking to make some money?

  • Set your budget and the time frame
  • Outline your proposal
  • Get paid for your work

Bids on this Project

  • igorfreelancer Profile Picture

    igorfreelancer

    Zagreb,  Croatia

    I am a coder of 25 years with extensive experience in the following: - C - C++ - .NET - and SQL. I am a database administrator and system developer. I specialize in data mining, algorithm design, optimizations and CPU intensive tasks.

    ASP, C Programming, Java, and Visual Basic

  • ayazzaidi Profile Picture

    ayazzaidi

    Melbourne,  Australia

    Over the last 7 years, I have developed wide range of websites and desktop applications using ASP.NET, PHP, VB, C#, XHTML, JS, MYSQL, MSSQL etc. My projects include sites for non-profit organizations to large businesses applications. I also have good experience in the following areas: Ajax, JQuery, LINQ, XML / XSLT and software design and testing. I have joined Freelancer to provide quality service to benefit clients while at the same time enhancing my experience and establishing myself as a good service provider in Freelancer.com

    PHP, ASP, C Programming, and Javascript

  • SEOSEARCHTEAM Profile Picture

    SEOSEARCHTEAM

    JAIPUR,  India

    OUTSTRIP Info Tech Private Limited have a dedicated team of 100+ professional’s designers, developers for Mobile Application development, website design & development, software development,, e-commerce development, Social Networking websites and portal development; we thrive on the idea that design makes a difference.At OUTSTRIP Info Tech, we have accumulated a brilliant team of digital strategists, designers, developers, marketers, communications specialists, project managers and business developers to work in a combined manner on our client projects. We build on each other's expertise, working together in teams to generate great outcomes for projects. What makes our company great? The team members at Outstrip are some of the most innovative, interesting and devoted people you'll ever meet, exactly the kind of people you'd love to work with. We Highlights: - More than 4 Year, Best Mobile and Web development Company. - 500+ Satisfied Client Around the Globe. - 900 + Web & Mobile project delivered. - 35% Repeat Client - We Build Brands with 360° Solutions - Creative Ideas That Perform, Our achievement is measured by outcomes Our Services - Web Development - Mobile Apps Development - Web Identity Development - Rich Internet Application - Open Source Development - Open Source Customization - Community Portal Development - Custom E-commerce Development Specialized In 1. Web Development - Ecommerce Development - Marketplace - Dating Sites - Job Sites - Community & Networking Sites - Social Networking Sites - Classified Sites - CMS based Sites - Corporate Site for IT Services Company Web Technologies - PHP - AJAX - JQuery - JavaScript - XML/XSL Open Sources - Drupal - WordPress - Joomla - Magento - Prestashop - OsCommerce - ZenCart Graphics & Multimedia - Adobe Photoshop - Adobe Flash - Adobe Illustrator - Adobe After-Effects Programming Language Support - C - C++ - Objective-C - JAVA - VB - PHP - Android - iOS - Ruby on Rails - Python Database Tools - MS SQL Server - MySQL - MS Access - Oracle - NOSQL Frameworks - Codeigniter - Yii - CakePHP 2. Mobile Apps Development - Ecommerce Apps - Dating Apps - Job Searching Apps - Social Networking Apps - Classified Apps - CMS based Backend System Language - Java - C - C++ - Objective C - jQuery - Java Script - HTML 5.0 - CSS 3 Environment and Platform - Mac OS, - Android SDK - iPhone SDK 5 and later - iOS - Windows - Phone Gap - Titanium - JSON Parsing - Angular.JS - Node.JS - Android Networking - Restful APIs Development Frameworks - Cocoa - OpenGL ES Database - MySQL - SQL Lite - Core Data - Rest Kit We are a Team of Web Designers, PHP Programmers, HTML Developers, iPhone - iPad and Android Apps Developers, specialized open source Developers, Team Leaders, Project Managers, Quality Assurance engineers, SEO Experts, and Business Analysts. The philosophy of our Outstrip infotech is centered in the satisfaction of our clients and maintain long term beneficial business relationship. We insisted on exceeding its expectations at any moment, working close to them to obtain a solution according to its tastes and necessities. We have defined development processes that assure quality in end products. Our main competitive advantage consists in offering products with an excellent relation quality and price.

    PHP, Java, Data Entry, and Link Building

  • SigmaVisual Profile Picture

    SigmaVisual

    Taxila Cantt,  Pakistan

    Web, Bots, Crawlers, and Scrapers Development. I have expertise in automation services and I can automate any manual process.

    PHP, ASP, C Programming, and Javascript

  • raul27868 Profile Picture

    raul27868

    Madrid,  Spain

    I'm specializing in website data scraping. I have developed software that performs data collection. I'm specializing in website data scraping. I can develop software in languages​​: Perl, PHP and Javascript I can perform installations and settings for Joomla and WordPress

    PHP, Perl, C Programming, and Javascript

  • phpXpertbd Profile Picture

    phpXpertbd

    Dhaka,  Bangladesh

    Custom Website Development E-commerce website development with Opencart Website Data Scraping and saving into XLSX, XLS, CSV, TSV, MySQL, MSSQL, XML, PDF Making another website with the scraped data Website Cloning Issues or bug fixes in the website Ongoing maintenance, administration or supports Etc...

    Perl, Javascript, XML, and Python

  • proteamspb Profile Picture

    proteamspb

    St.Petersburg,  Russian Federation

    Tools: C/C++, VC++/MFC, dotNet, Delphi, Pascal, VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 Databases: MS SQL Server, MySql, BDE, Paradox, DB2, SYBASE, Informix. ProTeam SPb specializes in the area of software development and commercial realization of own S/W products - Professional team (MCP, MCSD, etc.) - More than 15 years of experience of a software development - More than 20 successfully completed large projects of a software development - More than 60 successfully completed moderate outsourcing projects of a software development - Clients in: USA, Great Britain, Spain, Germany, Russia - Platforms in use: Windows since 95 till Win7, SUN Solaris, MS-DOS - Development tools: Net (C#, C++.Net, ASP.Net, VB.Net, ADO.Net), Delphi; Pascal; C, C++, VC++/MFC; VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 - Databases: MS SQL Server, My Sql, BDE, Paradox, DB2, SYBASE, Informix Area of Expertise - Platforms in use: Windows 95/98/NT/2000/XP/2003/7/8, Android, SUN Solaris, MS-DOS - Development tools: Net (C#, C++.Net, ASP.Net, VB.Net, ADO.Net), Delphi; Pascal; C, C++, VC++/MFC; VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 - Databases: MS SQL Server, MySql, BDE, Paradox, DB2, SYBASE, Informix

    Project Management, SEO, Data Entry, and Engineering

  • solutino Profile Picture

    solutino

    Trivandrum,  India

    A team of IT professionals providing a spectrum of services mainly in Web, Mobile and Social Media. We expertise in PHP, AJAX, JQUERY, DRUPAL, WORDPRESS, Payment Gateways, FACEBOOK APPS, iOS,iPhone Apps, Android Apps etc

    PHP, Javascript, XML, and Adobe Flash

  • truongngocthanh Profile Picture

    truongngocthanh

    Ho Chi Minh city,  Vietnam

    I do good work as a database developer, data processing as well as developing applications using the web. Areas of expertise include Coldfusion, .NETand web full-stack. I am also expert in HTML, JavaScript, CSS (frontend developing)

    PHP, ASP, Java, and Javascript

  • mikenew16 Profile Picture

    mikenew16

    Myocum,  Australia

    Technology and business consultant with extensive experience as a project director and project manager, over 13 years international IT experience in investment banking, mining and not-for-profit experience. Expertise includes project and program management with significant sales and general consulting experience. 4 years as a quantitative analyst designing and writing automated trading systems for a major investment bank, including data mining and big data analysis Industry Experience o Investment Banking o Mining o Professional services management o Not for Profit organisations o Technology consulting Specialist Knowledge o COTS customisation and deployment o Big Data and data mining o Trading and risk management o Business risk management o Workers Compensation o Data warehouse and reporting o Internet Portal o Application development Relevant Capabilities o Project management o Sales consultant o Professional services management o Life Coaching

    Python, Project Management, Data Entry, and Excel