Learn the system

How the viewer, Next.js proxy, and reddit-scraper-express-server work together — for self-hosters and developers on the open-source repos.

Overview

Free Reddit Scraper is a viewer-only Next.js app. Paste a public reddit.com post URL, read post and comments, and reopen scrapes from Threads — no account, no cloud save.

This page explains how Free Reddit Scraper is built: what runs in the browser, what the Next.js server proxies, and which open-source repos you clone. It is not a Reddit URL tutorial — paste a thread on the home page to scrape.

Who should read this page

ReaderWhat you are trying to learnWhere to start on this page
Researchers or moderatorsUnderstand where thread data goes and that history stays on this device.Trust and Data flow, then scrape on /.
Self-hostersRun Express + web locally with the right env vars and ports.Repositories, Viewer-only, HTTP, and Stack.
Developers extending the OSS reposSee folder layout and ADRs before changing code.Web code, OSS Express, and OSS patterns.
Coding agentsOnboard to ADR 019 and repo boundaries quickly.Agent callout below, then .cursor/rules/AGENTS.md.

You might use Free Reddit Scraper if…

  • You want post + comments from a public thread without a Reddit API key in the browser.
  • You self-host reddit-scraper-express-server and keep SCRAPER_API_KEY server-side.
  • You need session-local history in Threads, not a synced account.

You should not expect…

  • Signed-in sync, credits, Stripe, or CMS communities from the old product — see ADR 019.
  • Instant scrapes — Apify runs often take about a minute.
  • Private or logged-in Reddit content the actor cannot reach.

For coding agents

  • Read free-reddit-scraper-web/.cursor/rules/AGENTS.md and ADR 019 (viewer-only) before adding auth or CMS calls.
  • Product split: web UI + POST /api/scrape proxy | reddit-scraper-express-server (Apify + x-scraper-api-key).

Reference: Web ADR 019 — Viewer-only Reddit scraper web

What you get

CapabilityIn the UIBackend
Landing scrape form/#scraperscrapeThreadThunk → POST /api/scrape
Thread detail view/thread-detail-pageRedux currentScrape + static route (ADR 008)
Session history/threadsRedux scrapes dump + localStorage
Express health checkOperator / setupGET /api/health on port 3037 (default)

Out of scope for this app

  • Supabase auth, saved accounts, or credit checkout in this web repo
  • Calling Apify or Express directly from the browser bundle
  • Cross-device scrape sync without your own backend

Repositories

Public OSS pair on GitHub org Luckee-Core. No hosted CMS repo is listed here.

RepositoryRuns onOwnsDoes not own
free-reddit-scraper-webVercel + local devLanding, scrape form, Threads, detail UI, POST /api/scrape proxyApify runs or SCRAPER_API_KEY on the client
reddit-scraper-express-serverSelf-host / local (default port 3037)POST /api/services/get-reddit-thread-data, /docs, api-docs.jsonBrowser UI or long-lived scrape storage

Deploy flow

  1. Clone free-reddit-scraper-web and reddit-scraper-express-server from Luckee-Core.
  2. Set APIFY_API_TOKEN and SCRAPER_API_KEY in Express .env; start the API (default http://localhost:3037).
  3. Set REDDIT_SCRAPER_EXPRESS_URL and SCRAPER_API_KEY in Next .env.local (server-only).
  4. npm run dev in the web repo; scrape from / or reopen rows under Threads.

Full env and route tables: docs/oss/wire-contract.md in the web repo.

Trust and security

Browser UI, Next.js proxy, and self-hosted Express + Apify — three zones with different secrets.

Zone 1 — Browser

Redux holds scrape results and session history in localStorage on this device. No account row is created in a hosted CMS.

Zone 2 — Next.js server

POST /api/scrape validates the body and forwards to Express with x-scraper-api-key from server env. The key never appears in NEXT_PUBLIC_* or client bundles.

Zone 3 — Express + Apify

Express holds APIFY_API_TOKEN and SCRAPER_API_KEY. POST /api/services/get-reddit-thread-data runs a paid Apify actor — protect the port and rotate keys if exposed.

Where secrets live

Secret / dataAllowedNever
SCRAPER_API_KEYNext .env.local + Express .env (server only)NEXT_PUBLIC_* or browser JavaScript
APIFY_API_TOKENExpress .env onlyNext.js web repo or browser
Scrape payloadsRedux + localStorage on the user deviceAutomatic cloud backup in viewer-only v1

Further reading

  • ADR 019 — .cursor/architecture/019-viewer-only-reddit-scraper-web.md
  • ADR 004 — .cursor/architecture/004-api-integration.md
  • Wire contract — docs/oss/wire-contract.md

Viewer-only product

No Luckee sign-in gate on the landing page. Legacy CMS routes redirect; scrapes stay in this browser session.

PathBehavior
/Marketing landing + live scrape form (#scraper)
/threadsSession scrape list (Redux dump)
/thread-detail-pageDetail for currentScrape (no URL id segment)
/learnStatic system overview (this page)
/login, /dashboard, …Legacy CMS paths redirect to / or /threads (ADR 019)

Web environment (server + operator)

VariablePurpose
REDDIT_SCRAPER_EXPRESS_URLNext server — Express base URL (default http://localhost:3037)
SCRAPER_API_KEYNext server → Express header; must match Express .env
APIFY_API_TOKENExpress only — Apify actor runs

Data flow

From paste on / to rows in Threads on this device.

Steps

  1. User pastes a public reddit.com thread URL in the landing form.
  2. scrapeThreadThunk POSTs { url, mode } to /api/scrape on the Next server.
  3. Next forwards to Express POST /api/services/get-reddit-thread-data with x-scraper-api-key.
  4. Express runs Apify, returns JSON; thunk stores the scrape in Redux and localStorage.
  5. User opens /thread-detail-page (currentScrape) or finds the row again under /threads.

When something goes wrong

  • SCRAPER_API_KEY missing in Next — /api/scrape returns 500 Service unavailable.
  • Express down or wrong REDDIT_SCRAPER_EXPRESS_URL — proxy errors; check GET /api/health.
  • Invalid or non-public URL — upstream 400 from Express validation or empty Apify dataset.
  • Timeout — scrape can run up to ~300s; wait for the request to finish.

Common questions

Does data leave my machine?

Thread content goes to Apify via your self-hosted Express instance when you scrape. The viewer does not send SCRAPER_API_KEY or APIFY tokens to Luckee cloud.

Can I use only the web repo?

You need a reachable reddit-scraper-express-server for real scrapes — local dev is the default setup.

HTTP contract

Next proxy and Express both use { success: true, data } or { success: false, error } JSON for scrape routes.

Express documents routes at GET /docs and GET /api-docs.json on the scraper API repo.

Next.js (free-reddit-scraper-web)

MethodPathPurpose
POST/api/scrapeProxy to Express scrape (body: url, optional mode)

OSS Express API (self-hosted)

MethodPathPurpose
GET/api/healthLiveness
POST/api/services/get-reddit-thread-dataApify sync scrape (header x-scraper-api-key)
GET/api-docs.jsonAPI catalog
GET/docsHTML API reference

Full env and route tables: docs/oss/wire-contract.md in the web repo.

Stack

Versions follow each repo package.json at release time.

Web (qr-code-generator-web)

  • Next.js App Router (next ^16)
  • React 19, Redux Toolkit, manual thunks
  • Tailwind CSS 4
  • Session scrapes in Redux dumps + localStorage
  • Node 22.x

OSS Express (qr-code-generator-open-source-express-server)

  • Express 5, TypeScript
  • Apify Reddit Scraper Lite via sync actor run
  • GET /docs + api-docs catalog
  • Default port 3037
  • Node 22.x

How the web app is built

Luckee Next.js template conventions. ADRs live in .cursor/architecture/.

Folder map

  • src/app/**/page.tsx — thin routes
  • src/app/api/scrape/route.ts — Express proxy
  • src/packages/landing — marketing + scrape form
  • src/packages/threads — session history list
  • src/store/thunks — scrapeThreadThunk and navigation
  • src/model — scrape row types

ADR quick reference

ADRSummaryThis product
001Redux: flat layers; manual thunks; no selector functionsYes
002Thin app routes; features in src/packages/Yes
004Thunks call /api/scrape only for scrapesYes
008Static /thread-detail-page + currentScrapeYes
019Viewer-only — no auth/CMS dependencyYes — product
020Landing + gray app chromeYes
021This /learn pageYes

How the OSS Express API is built

Stateless PNG generation for integrators. Not wired into the manageqrcodes.com browser generator.

Folder map

  • src/services/get-reddit-thread-data/ — router, handler, processGetRedditThreadData
  • src/services/api-docs/ — catalog + HTML docs
  • src/services/health/

ADR quick reference

ADRSummaryThis product
001–002Services + router factory; handlers → processX()Yes
006Emoji logging; success/error JSONYes
012API docs catalog + /docs HTMLYes

Luckee OSS patterns

Same Next.js and Express template families as other Luckee apps; this product adds browser QR generation and optional hosted save APIs.

  • Luckee-Core OSS pair — web viewer + Express Apify proxy
  • oss-api-docs-pattern — Express /docs + api-docs.json
  • reddit-scraper-express-server SECURITY.md — rate limits and key exposure

Glossary

TermMeaning
currentScrapeRedux slice pointing at the row shown on /thread-detail-page (ADR 008).
Scrape dumpRedux scrapes collection persisted in localStorage for this browser profile.
Modethread (default) fetches a post + comments; listing fetches recent posts without comments.
Viewer-onlyNo Supabase session or CMS persistence in free-reddit-scraper-web (ADR 019).

Next steps

Agents: after this page, open .cursor/rules/AGENTS.md in the repo you are editing.