Pricify

Overview

Pricify is a self-hosted price-tracking application which periodically scrapes e-commerce websites, stores historical data, and notifies users when prices go below configurable thresholds. The system is composed of several services rather than a single application: a web frontend backed by a REST API, a standalone scheduler for dispatching scrape jobs, a headless browser scraping service for JavaScript-heavy sites, and a relational database for product data, e-commerce site configuration and user data.

Architecture

The web application is built with Next.js, which serves both the frontend and REST API from the same application. It is organized by domain (products, stores, jobs, settings, admin), backed by an ORM over Postgres. The data model covers products, stores, price history, scrape jobs, notifications, and a permissions model consisting of users, roles, and audit logs.

Scraping is handled by an engine that iterates a product’s tracked URLs, extracting a price from each page using a CSS selector, regular expression, or structured path depending on store configuration. The lowest successful price across a product’s URLs is taken as its current price. System-wide scraping defaults are merged with per-store overrides at request time rather than being fixed at configuration time, allowing global settings to change without editing every store individually.

Scheduling runs as its own service rather than as a background job within the main application. It reloads store schedules from the database on an interval, converts each store’s configured interval or cron expression into an actual schedule, and calls back into the main application over HTTP to trigger a scrape. This keeps the scheduler independent of the scraping logic itself.

Authentication uses session-based auth with a custom permissions layer on top, supporting granular resource-level permissions, including separate variants for a user’s own resources versus all resources. This is enforced through a shared wrapper applied to most API handlers, which also handles audit logging and rate limiting.

Design Decisions

Store scraping rules may include user-supplied regular expressions, which introduces a denial-of-service risk. Patterns are validated against known catastrophic-backtracking shapes, length-limited, and executed within a timeout guard.

The plain-fetch scraping path checks specifically for a Cloudflare challenge page and returns an error directing the user to switch to a proxy or the headless-browser scraping mode, rather than failing with a generic error.

The permissions model was designed with future multi-user deployments in mind, including resource sharing and auditability, despite these features not being required for the initial single-user deployment.

Challenges

Pricify initially used basic cURL requests to retrieve page contents. This worked well for simple sites but failed on websites requiring JavaScript execution, requiring a move to Playwright-based scraping.

Some websites returned different pricing data to automated clients compared with normal browser sessions. This meant successful requests did not always produce reliable results, requiring browser-based scraping, stealth configuration, and more controlled request patterns.

Lessons Learned

The move to an ORM was a net improvement: replacing raw SQL scattered across large service files with typed entities made the codebase considerably easier to work with and simplified large-scale refactors.

Keeping the scheduler unaware of how scraping works, and having it call back into the main application over HTTP to trigger a run, kept the separation between scheduling and execution clean. The tradeoff was some duplicated model definitions that needed consolidation.

When making large architectural changes (such as the ORM migration), it’s important that the documentation is updated in the same change rather than afterward to prevent drift.

What’s Next

• Consolidate the duplicated data model between the main application and the scheduler into a single shared definition

• Add row-level validation on import instead of relying on row count alone

• Add per-route ownership checks to the permissions layer, beyond the current all-or-nothing model