How the viewer, Next.js proxy, and reddit-scraper-express-server work together — for self-hosters and developers on the open-source repos.
Free Reddit Scraper is a viewer-only Next.js app. Paste a public reddit.com post URL, read post and comments, and reopen scrapes from Threads — no account, no cloud save.
This page explains how Free Reddit Scraper is built: what runs in the browser, what the Next.js server proxies, and which open-source repos you clone. It is not a Reddit URL tutorial — paste a thread on the home page to scrape.
| Reader | What you are trying to learn | Where to start on this page |
|---|---|---|
| Researchers or moderators | Understand where thread data goes and that history stays on this device. | Trust and Data flow, then scrape on /. |
| Self-hosters | Run Express + web locally with the right env vars and ports. | Repositories, Viewer-only, HTTP, and Stack. |
| Developers extending the OSS repos | See folder layout and ADRs before changing code. | Web code, OSS Express, and OSS patterns. |
| Coding agents | Onboard to ADR 019 and repo boundaries quickly. | Agent callout below, then .cursor/rules/AGENTS.md. |
For coding agents
Reference: Web ADR 019 — Viewer-only Reddit scraper web
| Capability | In the UI | Backend |
|---|---|---|
| Landing scrape form | /#scraper | scrapeThreadThunk → POST /api/scrape |
| Thread detail view | /thread-detail-page | Redux currentScrape + static route (ADR 008) |
| Session history | /threads | Redux scrapes dump + localStorage |
| Express health check | Operator / setup | GET /api/health on port 3037 (default) |
Public OSS pair on GitHub org Luckee-Core. No hosted CMS repo is listed here.
| Repository | Runs on | Owns | Does not own |
|---|---|---|---|
| free-reddit-scraper-web | Vercel + local dev | Landing, scrape form, Threads, detail UI, POST /api/scrape proxy | Apify runs or SCRAPER_API_KEY on the client |
| reddit-scraper-express-server | Self-host / local (default port 3037) | POST /api/services/get-reddit-thread-data, /docs, api-docs.json | Browser UI or long-lived scrape storage |
Full env and route tables: docs/oss/wire-contract.md in the web repo.
Browser UI, Next.js proxy, and self-hosted Express + Apify — three zones with different secrets.
Redux holds scrape results and session history in localStorage on this device. No account row is created in a hosted CMS.
POST /api/scrape validates the body and forwards to Express with x-scraper-api-key from server env. The key never appears in NEXT_PUBLIC_* or client bundles.
Express holds APIFY_API_TOKEN and SCRAPER_API_KEY. POST /api/services/get-reddit-thread-data runs a paid Apify actor — protect the port and rotate keys if exposed.
| Secret / data | Allowed | Never |
|---|---|---|
SCRAPER_API_KEY | Next .env.local + Express .env (server only) | NEXT_PUBLIC_* or browser JavaScript |
APIFY_API_TOKEN | Express .env only | Next.js web repo or browser |
Scrape payloads | Redux + localStorage on the user device | Automatic cloud backup in viewer-only v1 |
No Luckee sign-in gate on the landing page. Legacy CMS routes redirect; scrapes stay in this browser session.
| Path | Behavior |
|---|---|
/ | Marketing landing + live scrape form (#scraper) |
/threads | Session scrape list (Redux dump) |
/thread-detail-page | Detail for currentScrape (no URL id segment) |
/learn | Static system overview (this page) |
/login, /dashboard, … | Legacy CMS paths redirect to / or /threads (ADR 019) |
| Variable | Purpose |
|---|---|
REDDIT_SCRAPER_EXPRESS_URL | Next server — Express base URL (default http://localhost:3037) |
SCRAPER_API_KEY | Next server → Express header; must match Express .env |
APIFY_API_TOKEN | Express only — Apify actor runs |
From paste on / to rows in Threads on this device.
Does data leave my machine?
Thread content goes to Apify via your self-hosted Express instance when you scrape. The viewer does not send SCRAPER_API_KEY or APIFY tokens to Luckee cloud.
Can I use only the web repo?
You need a reachable reddit-scraper-express-server for real scrapes — local dev is the default setup.
Next proxy and Express both use { success: true, data } or { success: false, error } JSON for scrape routes.
Express documents routes at GET /docs and GET /api-docs.json on the scraper API repo.
| Method | Path | Purpose |
|---|---|---|
POST | /api/scrape | Proxy to Express scrape (body: url, optional mode) |
| Method | Path | Purpose |
|---|---|---|
GET | /api/health | Liveness |
POST | /api/services/get-reddit-thread-data | Apify sync scrape (header x-scraper-api-key) |
GET | /api-docs.json | API catalog |
GET | /docs | HTML API reference |
Full env and route tables: docs/oss/wire-contract.md in the web repo.
Versions follow each repo package.json at release time.
Luckee Next.js template conventions. ADRs live in .cursor/architecture/.
| ADR | Summary | This product |
|---|---|---|
| 001 | Redux: flat layers; manual thunks; no selector functions | Yes |
| 002 | Thin app routes; features in src/packages/ | Yes |
| 004 | Thunks call /api/scrape only for scrapes | Yes |
| 008 | Static /thread-detail-page + currentScrape | Yes |
| 019 | Viewer-only — no auth/CMS dependency | Yes — product |
| 020 | Landing + gray app chrome | Yes |
| 021 | This /learn page | Yes |
Stateless PNG generation for integrators. Not wired into the manageqrcodes.com browser generator.
| ADR | Summary | This product |
|---|---|---|
| 001–002 | Services + router factory; handlers → processX() | Yes |
| 006 | Emoji logging; success/error JSON | Yes |
| 012 | API docs catalog + /docs HTML | Yes |
Same Next.js and Express template families as other Luckee apps; this product adds browser QR generation and optional hosted save APIs.
| Term | Meaning |
|---|---|
| currentScrape | Redux slice pointing at the row shown on /thread-detail-page (ADR 008). |
| Scrape dump | Redux scrapes collection persisted in localStorage for this browser profile. |
| Mode | thread (default) fetches a post + comments; listing fetches recent posts without comments. |
| Viewer-only | No Supabase session or CMS persistence in free-reddit-scraper-web (ADR 019). |
Agents: after this page, open .cursor/rules/AGENTS.md in the repo you are editing.