About

I need an experienced Python/Scrapy developer to build a price-scraping microservice that will run on my existing DigitalOcean droplet (Ubuntu). My main backend is built with Next.js, so your role is to create a separate Python service that exposes scraping endpoints through an API (FastAPI or Flask). In the API i will send multiple URLs and i need the following in return: - Product price - URL name - Promotion available, if yes then details of it - In Stock or Out of Stock. Your tasks: 1. Build a Scrapy project • Create spiders for multiple ecommerce websites • Scrape: product title, price, stock status, currency, and URL • Accept dynamic URLs passed into spiders (not hardcoded) • Handle JavaScript-rendered pages (Playwright/Splash if needed) • Robust error handling and anti-blocking techniques 2. Build a REST API wrapper Using FastAPI or Flask, create endpoints: • POST /scrape → scrape a single URL, return JSON • POST /batch-scrape → scrape multiple URLs • GET /health → service status The API will run locally on a different port (e.g., 5001) and will be called by my Next.js backend. 3. Deploy on my current DigitalOcean droplet • Set up Python virtual environment • Deploy Scrapy + API • Configure systemd service so API runs 24/7 • (Optional) Reverse proxy with nginx • Make sure the API is accessible ONLY from localhost 4. Logging + Monitoring • Implement structured logging • Save logs to /var/log/scrapy-service/ • Write clear error messages if price not found. Example of URLs i use are attached in the file. See attached. These are not the only ones, there will be more. But the idea is to be able to scrape the following data: - Product Price - Promotion available, if yes then details of it - In Stock or Out of Stock.