Autonomous Amazon & Flipkart Crawler

@AZTURK1

Hi there, I’ll build two independent, non-blocking crawlers (Amazon + Flipkart) that run per-seller-account and pull supplied ASINs into MySQL every 24h; I’ll use headless Playwright for JS-rendered pages and Scrapy for lightweight pages to minimise blocking and scale. - Deliverable 1: two runnable crawler services (amazon_crawler, flipkart_crawler) with per-account session handling, ASIN queue, and direct MySQL write (product name, seller, star rating, ratings count, MRP, price). - Deliverable 2: scheduling config (systemd timers/cron), migration SQL, SMTP email alert module, README and commands to add ASINs without code changes. - Risk/quality-control: built-in retry & self-healing with exponential backoff, captcha detection, staged deployment + rollback plan and post-fix validation. Skills: ✅ Python (Playwright / Scrapy) ✅ MySQL ✅ Queueing & scheduler (cron / systemd timer) ✅ Self-healing & retry logic (exponential backoff, resume-from-failure) ✅ Deployment/hosting on VPS or cloud ✅ Monitoring/logging (per-crawler logs, lightweight dashboard) Certificates: ✅ Microsoft® Certified: MCSA | MCSE | MCT ✅ cPanel® & WHM Certified CWSA-2 I’m available to start immediately; Is this already running on live production seller accounts? Price: $5000 , Delivery: 1 day. Best regards,

@ashinnd84

Hi! I'm excited to discuss your project. Could you share more details about your specific requirements? Thanks Ashish Kumar.

@sharmin283

With over a decade of experience in software development, I've handled projects similar to your Autonomous Amazon & Flipkart Crawler requirement. I'm certain that I have the skills necessary to build reliable and efficient crawlers for both Amazon and Flipkart, whilst integrating seamlessly with your MySQL database. My experience extends beyond data gathering with critical emphasis on building robust self-healing and retry mechanisms, qualities you mentioned are crucial for this project. In terms of stack, while I'm comfortable with Python's Scrapy or Playwright and Node.js' Puppeteer, I'd like to propose a more lightweight and efficient solution in Laravel/PHP along with MySQL. Aside from offering headless navigation possibilities, the PHP ecosystem boasts a wide range of libraries, like Guzzle, which can efficiently handle simultaneous asynchronous requests without blocking. In line with your need for separate queues, logs and threads/processes but sharing a single MySQL instance, I can ensure an architecture that allows for independent yet collaborative functionalities.

@Hrolgar

Two crawlers, shared nothing except the schema. Amazon and Flipkart each get their own queue, log file, and retry budget so one platform stalling doesn't drag the other. Seller account sessions are the real complexity here. Public product pages won't give you the full seller view, so both crawlers authenticate via Playwright in stealth mode, checkpoint session state, and rotate headers to stay below detection thresholds. CAPTCHA or rate limit? Exponential backoff handles it automatically. If retries exhaust, SMTP fires with which ASINs failed and why. Schema via Alembic migrations so adding a field later is a one-liner, not a manual ALTER. Systemd timer (or cron, your call) fires every 24h, picks up from the last checkpointed ASIN cursor, writes to MySQL, outputs a per-run summary with fetched/failed/skipped counts. Deliverables: - Amazon + Flipkart crawlers, fully independent - MySQL schema with Alembic migrations - Systemd units and config file (DB, SMTP, ASIN lists) - SMTP alerts on retry exhaustion - Per-run log summary - 5 days INR 8,000. Three quick questions: roughly how many ASINs per platform? Is seller login needed on both, or are public product pages enough for one? And preferred hosting, your VPS or a container image you deploy?

@Mayanks010

Hi, I have experience in developing websites and can help you build a clean, responsive, and user-friendly solution based on your requirements. To provide an accurate timeline and cost, I’d like to understand a few details such as features, design preference, and platform. Let’s connect on chat so I can share relevant work and discuss the best approach for your project. Best regards, Mayank

@sjohn117

Hi Client, I’m Sean, Senior Full-Stack Engineer with 8 years’ experience specializing in Python, Scrapy/Playwright, and MySQL. I build resilient, scalable crawlers and backend systems. I previously delivered a fault-tolerant product crawler that ingested 1M+ SKUs daily into a centralized MySQL store with automated recovery and alerting. My skills map directly to your requirements: I will implement two independent, non-blocking crawlers (Amazon and Flipkart) with separate queues, logs, and processes, robust anti-blocking strategies, and durable checkpoints so I can do this project perfectly and ensure each run resumes from the exact failure point. I will include exponential backoff, captcha handling strategies, proxy pools, and retry/self-healing logic. I typically deliver this scope in 14 days, including tests, deployment scripts, and scheduler configuration. I will provide unit/integration tests, structured logging/monitoring, OWASP basics, clean code, docs, and an SMTP alert module; data privacy and retry-eval guardrails are included. What is the expected daily ASIN volume and how many concurrent seller accounts/connections should each crawler support initially? Sincerely, Sean

@nksharma420

I’ll build two independent, non-blocking crawlers for Amazon and Flipkart that securely use your seller access to fetch product data every 24 hours and store it in a shared MySQL database. The system will be fully automated, resilient, and designed to recover from failures while continuing from the last processed item. Separate worker processes with their own queues, logs, and retry logic (self-healing with checkpoints) Automated scheduling, structured logging, and a lightweight dashboard/log summary for monitoring runs and errors Email alert system for critical failures, plus clean, documented code with easy configuration and ASIN management The solution will be scalable, headless, and easy to maintain, ensuring consistent data updates without manual intervention.

@ELGAMLDev

Hi, Two independent Python/Playwright processes, each with its own queue and logs, sharing one MySQL instance. Playwright with stealth patching handles the JavaScript-rendered variant dropdowns on both platforms. For Amazon the crawler walks the variant selector to discover all child ASINs from the parent. For Flipkart it iterates through size/color dropdowns and builds the SKU combinations. Both stored with parent reference in the normalized schema. Self-healing follows your spec: exponential backoff per error type (network, captcha, and 429 are handled separately), proxy rotation on captcha, max 5 retries before the product gets flagged and an SMTP alert fires. Queue continues rather than halting. Scheduling via systemd timers Persistent=true handles missed runs on recovery, which cron can't do. Status log writes last run, next run, ASINs processed, and error history to file and MySQL. One thing to confirm: your Flipkart description mentions stock status per variant but it's not in the schema. Should I include it?

@usamaansari786

Hi, With over 15 years of professional experience and more than 1500 projects completed, I offer the deep expertise you need to successfully execute your Amazon and Flipkart crawlers. I have a strong background in web-related technologies such as Node.js, MySQL and web scraping, all fundamental to the delivery of this project. To ensure reliability, my approach features self-healing capabilities and comprehensive error-reporting mechanisms to promptly address any interruptions, with immediate email alerts informing you of the same. My technical proficiency has been honed through specialized projects like this- automating complex systems, precisely fetching data and integrating varied APIs. The automation workflows built by me are resilient and require minimal manual intervention. Comfortable working with Python using Scrapy as well as other proven tool sets like with Puppeteer. I'd strongly recommend using the headless Python/Scrapy stack for maximum scalability and efficiency. Best, Usama K

@arshkumar213

I will build two independent, self-healing crawlers (Python + Playwright/Scrapy) for Amazon & Flipkart with retry logic, captcha handling, and cron scheduling. Data stored in MySQL with logs, alerts, and dashboard. Result: reliable 24/7 scraping with zero manual effort and easy ASIN updates.

@Sadiyasyeda24

Hello, I have read your project details and I get what you need. I am a skilled freelancer with 4 years of experience in PHP, Java, Python, Software Architecture, MySQL, Node.js. Check my profile for portfolio and reviews. Please connect in chat so we can discuss. Warm regards, Syeda Tahreem

@Smarty613

Hi! This is a well-structured crawling task and I can build exactly what you've described. My approach: - Two independent Python scrapers (Playwright for JavaScript-rendered pages on Amazon/Flipkart) running in separate processes - Self-healing logic: exponential backoff on rate limits, captcha detection with automatic session rotation - MySQL schema for clean storage of the fields you need (ASIN, product name, seller, rating, ratings count, MRP, current price, crawl timestamp) - Cron/systemd scheduling for 24-hour cycles - SMTP alert module triggered when retry budget exhausted - Lightweight HTML dashboard (or plain log file, your choice) showing last run, next run, error history The code will be clean, well-commented, and structured so you can add new ASINs via a config file or DB entry—no code changes needed. I've built similar scrapers for e-commerce monitoring projects. A few technical questions: Do you need proxy rotation built in, or do you have proxies ready? And are the seller accounts India-based? That affects which Amazon region I target. — Jacques A.

@antonp02

Hello [Client], I’ll deliver your dual Amazon and Flipkart crawlers with precision and efficiency, ensuring reliable, error-free data extraction and seamless integration with your MySQL database. Each crawler will run independently with self-healing logic to handle captchas, rate limits, and network issues, automatically recovering and alerting you via email if a fault persists. You’ll get well-commented source code, a robust database schema, scheduling configuration, and a lightweight dashboard for easy monitoring. My focus is on practical, scalable solutions using the best headless technology—Python with Playwright or Scrapy, depending on your preference—and clear documentation so you can manage ASINs effortlessly. I’m ready to start immediately. Would you like me to outline the first milestone so you can see how it would look? Regards, Anton Prinsloo

@webscraping143

Hi, I can build two independent, non-blocking crawlers for Amazon and Flipkart that reliably fetch your ASIN data every 24 hours and store it directly into MySQL. I’ll design each crawler as a separate service with its own queue, logs, and retry logic, so failures in one won’t affect the other. The system will include self-healing mechanisms to handle captchas, rate limits, and network issues—resuming exactly where it left off. If an issue persists, you’ll receive an instant email alert with a clear error summary. For stability and scale, I recommend Python with Playwright (stealth + headless) combined with a task queue (Celery/Redis) and cron/systemd scheduling. Data pipeline → validate → store in MySQL → log → schedule next run. I’ll also include a lightweight dashboard/log view to monitor last run, next run, and error history. Deliverables will include clean, well-structured code, MySQL schema, scheduler setup, retry/self-healing logic, SMTP email module, and a clear README for setup and adding new ASINs without code changes. Estimated timeline: 4–6 days Happy to discuss architecture before starting. Thanks.

@mihajlodjokic

Let me get it done, I’ll develop two independent, non-blocking crawlers using Python with Scrapy to extract ASIN data from Amazon and Flipkart, storing it in a MySQL database. Your project will be finished in 7-10 days, I’ve built similar data extraction systems for e-commerce monitoring. Here is how I would approach it: 1. I can provide a demo. We’ll define ASINs and database schema. 2. Implement Scrapy spiders for each platform with robust error handling. 3. Develop a MySQL schema and migration scripts. 4. Integrate self-healing retry logic and email alerts. 5. Create a lightweight dashboard summarizing run status and errors. I recommend Python with Scrapy for headless, non-blocking operation and scalability. Let’s talk as soon as possible and get it done. Best Regards, Mihajlo

@DeadKira666

Hi, I have experience building reliable web crawlers and automation systems using Python and Node.js, including long-running scraping tasks with scheduling, database integration, and failure recovery. I can develop two independent crawlers (Amazon and Flipkart), each running automatically every 24 hours to fetch data for the ASINs you provide: product name, seller, rating, number of reviews, MRP, and current price. All data will be stored directly in a MySQL database with a clean and scalable schema. The system will be designed for reliability and autonomy, including: Separate crawlers with independent execution and logs Non-blocking, asynchronous architecture Automatic retry and self-healing logic for captchas, rate limits, and network failures Resume-from-failure capability Email alerts via SMTP in case of persistent errors Scheduled execution using cron or systemd Simple monitoring logs showing last run, next run, and error history The code will be modular, well-commented, and easy to maintain. You’ll also be able to add new ASINs without modifying the core logic. I recommend Python with Playwright or Scrapy for better stability in headless scraping, but I can adapt to Node.js if required. I can start immediately and deliver a production-ready solution within your timeline.

@ammark477

I’d be glad to build an autonomous crawler for Amazon and Flipkart that reliably collects and structures product data for your use case. The goal will be to create a system that runs efficiently, handles updates automatically, and delivers clean, usable data. I can develop a solution with smart scraping logic, scheduling/automation, data extraction (pricing, ratings, availability, etc.), and structured storage (DB/CSV/API). I’ll also ensure the crawler is robust, rate-aware, and fault-tolerant so it performs consistently over time. My approach is to design a scalable pipeline—from data collection to processing and storage—while keeping the system maintainable and easy to extend. If you’re looking for a dependable, automated data collection solution, I’d be glad to get started. Best Regards, Ammar Sajid

@edwardscool2

°•. ★.•° Expert Web Automation Engineer for Reliable E-commerce Crawlers °•. ★.•° ✨ Hi there ✨ Scraping Amazon and Flipkart at scale isn’t just about fetching data—it’s about staying undetected, recovering from failures, and running 24/7 without breaking. Most basic crawlers fail here. I build systems that keep running. I’ve developed resilient crawlers with proxy rotation, retry queues, and self-healing logic—ensuring consistent data collection even under rate limits and interruptions. My Approach: ✅ Architecture: Two independent services (Amazon + Flipkart) with separate queues, logs, and workers ✅ Tech Stack: Python + Playwright (stealth mode) for dynamic rendering + MySQL storage ✅ Data Flow: ASIN input → crawl → parse → validate → store (with dedup + timestamping) ✅ Reliability: – Retry queues with exponential backoff – Session/proxy rotation to handle captchas & limits – Checkpointing to resume from failure point ✅ Scheduling: Cron/systemd for 24h cycles with watchdog monitoring ✅ Alerts: Email notifications (SMTP) for persistent failures with clear logs ✅ Monitoring: Lightweight dashboard/log summary (last run, next run, errors) Deliverables include clean code, DB schema, setup scripts, and a README showing how to add ASINs without touching logic. Timeline: 5–7 days for a stable, production-ready system I focus on building crawlers that don’t just run—but keep running reliably under real-world conditions. Let’s get this automated properly. ✨ Best regards ✨

@ttobio69

Hi there, I have read your requirements carefully, and I understand you need two independent Amazon and Flipkart crawlers that can fetch fresh product data every 24 hours for your supplied ASINs/SKUs, store it in MySQL, and continue running reliably with separate logs, queues, retry logic, and email alerts. I'm a new freelancer, but I have rich experience in this field. I can build this in Python with a stable architecture where: -Amazon and Flipkart run as separate workers --both write to the same MySQL database failures are logged clearly -retries/self-recovery resume from the last unfinished item -email alerts are sent if an issue persists -scheduling runs automatically every 24 hours I will focus on: -product name, seller, rating, ratings count, MRP, selling price -clean MySQL schema + migration scripts -independent crawler health monitoring -setup docs and easy addition of new ASINs without code edits -simple run-status summary/dashboard or log view The best approach is to keep the system modular and reliable first, so it can run unattended and be maintained easily later. If account-based access or allowed data endpoints exist, I can structure the solution around the safest and most stable method available. If I work with you, I feel happy. Best regards, Oluwatobi Okedairo