In Progress

Store Crawler - Spider

Subject of this project is to develop a program – Store Crawler – which will search the web for retail stores and record next variables:

-Crawl date/time

-URL of the product

-name of the store and its logo

-SKU - product code (if available and can be parsed out of the product page)

-product EAN (if available and can be parsed out of the product page)

-product name

-category and subcategories




Every crawled product must be grouped by SKU, EAN or part of the product name. Grouping algorithm must be very intelligent. It must know which product is similar to other products, so they can be grouped together by product name.

There must be also an option, which tells the crawler which country will be crawled.

The identification whether crawled URL is a retail store or not, must be decided by the robot.

The goal is to build product comparison site.

Your task would be to fill the database with every product searchable through internet with method described below.

For each store custom spider would be made.

Url's of that stores would be scraped from directories or manually entered. Number of retail stores, approx. 3000 or more

Then we need to get all URL's from specific retail store: like sitemap or get it through google index with api search method, using "site:[url removed, login to view]" parameter.

Custom spider GUI would have start and stop field for each needed attribute, which would instruct the spider what to search on specific retail store.

After that, spider/bot would go through all url's of particular store and search for pre entered terms.


go at [url removed, login to view]

We need the price.

Source code tells us, that the price value begins after

td><b>Cena z DDV:</b></td><td>

and ends before


so crawler would search for that plain text.

Won't work with any type of broker or middleman. You must be the actual coder. We work with individual coders only. We work direct with our vendors.

Details of this small, starter project will be shared inside the PMB with qualified vendors.

Post an offer of $666 and 6 days on this project so I know you've read this and understand English. Price will be set afterwards, when all details will be discussed. Place an offer of anything other than $666 and 6 days and you'll be ignored, I promise.

If you're a real provider, with real experience, post an offer, message and then we'll chat in the PMB.

Skills: Data Mining, Data Processing, PHP, Software Architecture, Web Scraping

See more: crawler web, spider store, what z data processing, what's an algorithm, what's algorithm, us algorithm, text search algorithm, text comparison algorithm, search algorithm example, search algorithm comparison, robot's name, robot name, robot algorithm, program algorithm example, need bot coder, name the robot, name that robot, name of the robot, name for a robot, logo robot

About the Employer:
( 11 reviews ) Nova Gorica, Slovenia

Project ID: #2519305

Awarded to:


Hello, please check PMB, let's talk about details. thank you.

$900 USD in 6 days
(6 Reviews)

24 freelancers are bidding on average $1069 for this job


Hi, We have designed and built websites for various types of businesses very effectively. We work with all of our clients individually to easily coordinate and to keep track of the requirements and scope. We fulfill More

$1500 USD in 20 days
(220 Reviews)

We can help in your project, please check PMB and our ratings/reviews to get idea of our experience.

$750 USD in 15 days
(235 Reviews)

Hi My names Mark from The iDevelopment Team based in Jersey. We are experts in high end web design, ecommerce, internet marketing, social media optimisation and SEO, we would be pleased to help. There are a lo More

$1000 USD in 35 days
(13 Reviews)

$666 and 6 days (Can't be posted in BID due to restriction of minimum amount $750). Hi, Bid is to provided exactly what its stated in post, I can show you my recent similar jobs. Many Thanks

$1500 USD in 6 days
(20 Reviews)

$666 and 6 days, I can do it

$1000 USD in 6 days
(48 Reviews)

Hello There I would like to help you in this task.i will charge $666 and 6 days on this project Thankyou

$800 USD in 10 days
(18 Reviews)

Hi, :-) I can develop this site pls. contact me for details.

$766 USD in 6 days
(3 Reviews)

Hello, I can do this work for you and I'm ready to start. Please see pmb for details. Regards Raul

$750 USD in 7 days
(13 Reviews)

I had a crawler script written by myself that may able to handle what you need.

$1666 USD in 6 days
(3 Reviews)

I can do this. Thanks.

$750 USD in 7 days
(4 Reviews)

I already done like this project. I will collect all products details in CSV. More details sent to you, Thank you

$750 USD in 5 days
(5 Reviews)

Check your PMB for more detailed information.

$1666 USD in 6 days
(1 Review)

Hello Sir, i made the bid as you requested but the mini is 750 so no one can make 666 :!! I'm expert in web scrapping , crawling .

$750 USD in 6 days
(1 Review)

We are a Web design and development company that focuses on regular communication with our clients and timely delivery for all our projects. Some of the technologies/area that we focus include:

$750 USD in 15 days
(2 Reviews)

Dear Sir, Veltrod Technologies is a global software consulting company specialized in providing Mobile applications, Social media frameworks and eCommerce solutions. Leveraging best-in-class people, processes, and More

$1000 USD in 28 days
(5 Reviews)

Hello Respected Client, 666 for 6day I have Read your requirements and we are very experience in this concept. please check Message Board for more details. Thanks

$750 USD in 6 days
(1 Review)

Custom software development (<b><i>Removed by Admin</i></b>)

$1500 USD in 1 day
(0 Reviews)

$666 - 6 days Hi, I know your requirements and I can do this project for you.

$750 USD in 6 days
(0 Reviews)

We have looked at your project, and are confident we can provide quality service in a timely manner. Kindly see our portfolio attached.

$1500 USD in 27 days
(1 Review)

Hi, I tried bidding at $666 as per your description/requirement but it is not letting me place a bid lesser than 750. Please check your PM.

$750 USD in 6 days
(0 Reviews)