~/projects/ecommerce-scraper
E-commerce Scraper
Anti-detection extraction pipeline across three marketplaces.
- Year
- 2024-25
- Role
- Backend Developer
- Status
- Private Client
- Type
- Data Engineering
stack — Python · Scrapy · Playwright · MongoDB · Docker
A private data operation needed continuous product intelligence — prices, stock, ratings and seller shifts — across Amazon, eBay and AliExpress, refreshed daily and queryable on demand.
The deliverable is a Dockerized pipeline that crawls all three marketplaces on a schedule, survives their bot defenses, and lands clean, deduplicated data in MongoDB.
The problem
Each marketplace fights scrapers differently: fingerprinting, aggressive rate limits, IP bans and shifting markup. Naive scripts die within hours. The client needed consistent coverage across three very different sites with predictable cost and zero manual babysitting.
The build
Scrapy forms the crawling core; Playwright renders pages where bot checks demand a real browser fingerprint.
Rotating residential proxies with per-domain pacing, back-off and ban detection keep request success rates high.
Fingerprint hardening — headers, viewport, timing jitter — tuned per marketplace.
Field-level parsers normalized into one schema so Amazon, eBay and AliExpress data is directly comparable.
MongoDB stores raw + normalized documents with upsert-on-change so price history accumulates naturally.
Everything ships as Docker containers driven by scheduled runs, with run reports on output volumes and failures.
What it does
Three marketplaces, one schema
Normalized product records make cross-site comparison a query, not a project.
Anti-detection
Proxy rotation, fingerprint tuning and adaptive pacing designed against real ban behavior.
Price & stock history
Upsert-on-change storage accumulates a time series without duplicating documents.
Hands-off operations
Dockerized scheduled runs with monitoring reports — no babysitting required.
Outcome
- 3
- Marketplaces
- Daily
- Refresh cycle
- MongoDB
- Storage
- Docker
- Deployment