We want to build a crawling software which will be able to extract products details from 10 sites in the USA and store it in an SQL database.
Software should have start, pause, stop functionality. Once fully run, the software should monitor for any changes in Pricing and Stock. If new URL or product is added it should add to the database.
We will start with one site and later extend the architecture to other 9 sites. I need the quote only for one site. Software can be in [url removed, login to view] and MySQL database.
Please contact me only if you can prove you have done similar work before.
Some important features
Block Category for Extracting
In each site, I will create list of categories or sub categories from which the software should not extract any data
Fields will vary according to category. You need to come out with the fields which you will be extracting
Software should also store the category structure and URL of the product page
Images should be extracted from the site and stored in local folder. Corresponding path and file name should be stored in the database. If the product has got multiple images, the same needs to be stored. No more than 1000 images should be stored in single folder. Each single folder should not store more than 1000 sub folders. If the main folder reaches 1000 folders, then you should start creating folders inside another subfolder.
Folder Name Should start with ProductImage + Time example ProductImages181230
Each product should have a auto generated product code.
Export of Product
1. Software should extract the newly added data into a folder every 1 Hour
2. Software should extract the modified data into a folder every 1 Hour. Example Price and Stock
3. Same needs to be uploaded to an FTP location
We also need a separate Exe to do all settings. Example schedule settings, Block category, Image folder directory
Software should be able to connect to a proxy server to get the data's.
You can start the software buy building a proto type with screens and once I approve the proto type, you can start the database design. Once database design is approved you can start the coding work.
Looking to make some money?
- Set your budget and the time frame
- Outline your proposal
- Get paid for your work
Bids on this Project
I am a coder of 25 years with extensive experience in the following: - C - C++ - .NET - and SQL. I am a database administrator and system developer. I specialize in data mining, algorithm design, optimizations and CPU intensive tasks.
Over the last 7 years, I have developed wide range of websites and desktop applications using ASP.NET, PHP, VB, C#, XHTML, JS, MYSQL, MSSQL etc. My projects include sites for non-profit organizations to large businesses applications. I also have good experience in the following areas: Ajax, JQuery, LINQ, XML / XSLT and software design and testing. I have joined Freelancer to provide quality service to benefit clients while at the same time enhancing my experience and establishing myself as a good service provider in Freelancer.com
Taxila Cantt, Pakistan
Web, Bots, Crawlers, and Scrapers Development. I have expertise in automation services and I can automate any manual process.
Custom Website Development E-commerce website development with Opencart Website Data Scraping and saving into XLSX, XLS, CSV, TSV, MySQL, MSSQL, XML, PDF Making another website with the scraped data Website Cloning Issues or bug fixes in the website Ongoing maintenance, administration or supports Etc...
St.Petersburg, Russian Federation
Tools: C/C++, VC++/MFC, dotNet, Delphi, Pascal, VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 Databases: MS SQL Server, MySql, BDE, Paradox, DB2, SYBASE, Informix. ProTeam SPb specializes in the area of software development and commercial realization of own S/W products - Professional team (MCP, MCSD, etc.) - More than 15 years of experience of a software development - More than 20 successfully completed large projects of a software development - More than 60 successfully completed moderate outsourcing projects of a software development - Clients in: USA, Great Britain, Spain, Germany, Russia - Platforms in use: Windows since 95 till Win7, SUN Solaris, MS-DOS - Development tools: Net (C#, C++.Net, ASP.Net, VB.Net, ADO.Net), Delphi; Pascal; C, C++, VC++/MFC; VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 - Databases: MS SQL Server, My Sql, BDE, Paradox, DB2, SYBASE, Informix Area of Expertise - Platforms in use: Windows 95/98/NT/2000/XP/2003/7/8, Android, SUN Solaris, MS-DOS - Development tools: Net (C#, C++.Net, ASP.Net, VB.Net, ADO.Net), Delphi; Pascal; C, C++, VC++/MFC; VB; Java; Java Script; VB Script; Modula-2; Fortran; PL/1 - Databases: MS SQL Server, MySql, BDE, Paradox, DB2, SYBASE, Informix
A team of IT professionals providing a spectrum of services mainly in Web, Mobile and Social Media. We expertise in PHP, AJAX, JQUERY, DRUPAL, WORDPRESS, Payment Gateways, FACEBOOK APPS, iOS,iPhone Apps, Android Apps etc
Ho Chi Minh city, Vietnam
Technology and business consultant with extensive experience as a project director and project manager, over 13 years international IT experience in investment banking, mining and not-for-profit experience. Expertise includes project and program management with significant sales and general consulting experience. 4 years as a quantitative analyst designing and writing automated trading systems for a major investment bank, including data mining and big data analysis Industry Experience o Investment Banking o Mining o Professional services management o Not for Profit organisations o Technology consulting Specialist Knowledge o COTS customisation and deployment o Big Data and data mining o Trading and risk management o Business risk management o Workers Compensation o Data warehouse and reporting o Internet Portal o Application development Relevant Capabilities o Project management o Sales consultant o Professional services management o Life Coaching