Job Scrapper — Low-Frequency LinkedIn Job Collection (Personal Use)

This is a Node.js service that (1) scrapes LinkedIn Jobs public pages (no login), (2) stores only new jobs in MongoDB, (3) optionally analyzes jobs with Gemini against your resume and shortlists good fits, and (4) emails a report.

It’s intentionally designed to be safe and low-frequency: it runs only in a time window, uses randomized intervals, applies strict daily caps, and stops immediately if a block/CAPTCHA is detected.

---

What you get

• Up to ~50 new jobs/day (configurable cap) with full job details (title/company/location/experience/description/URL)
• Runs only 7:00 AM–7:00 PM IST (configurable via env)
• Randomized schedule (min/max interval + jitter) to avoid predictable patterns
• No duplicates (enforced by DB unique indexes on (platform, jobId) and (platform, jobUrl))
• Two modes:
  • Testing / verification (POST /scraper/test): one random keyword from JOB_KEYWORDS, fetch jobs for that keyword only, store the first job that satisfies all conditions, then stop immediately.
  • Normal scheduled (autonomous cron): one random keyword per run, fetch jobs, evaluate all against filters, store every job that passes (no early stop).
• Efficient scraping:
  • Single random keyword per run (scheduled or verification)
  • Light pagination (start=0, 25, 50...) or single-page scroll (see JOBS_FROM_SINGLE_PAGE) with early-stop when results are already old
  • Batch “exists” check to avoid opening detail pages for jobs already stored
  • List-page “posted X ago” prefilter to avoid wasting detail-page visits
• Optional pipeline:
  • Gemini analysis (shortlist) cron
  • Gmail email reports cron
  • Daily cleanup cron

---

Architecture (high level)

Scheduler (7am–7pm, randomized) ──▶ Scraper (Playwright) ──▶ MongoDB
                                         │
                                         ├──▶ Gemini analyze cron (optional) ──▶ ShortlistedJob
                                         └──▶ Email cron (optional) ──────────▶ Gmail SMTP

---

Prerequisites

• Node.js 20+
• MongoDB (local or managed)
• Playwright Chromium: npm run setup (installs chromium + chromium-headless-shell for Linux/CI)
• Optional:
  • Gemini API key (for analysis)
  • Gmail App Password (for email reports)

---

Quick start

cp .env.example .env
# edit .env

npm install
npm run setup
npm start

Useful endpoints:

• GET /jobs — list jobs (pagination: ?page=, ?limit=, optional ?from=...)
• POST /scraper/test — verification mode: one random keyword, store first job that passes, then stop
• GET /health — health (in production also checks MongoDB)
• GET /ping — lightweight keepalive

---

Configuration

All configuration is via environment variables. Use .env.example as the source of truth.

Core knobs (recommended defaults)

• Run window: SCRAPE_WINDOW_START_IST=7, SCRAPE_WINDOW_END_IST=19
• Daily caps: MAX_TOTAL_JOBS_PER_DAY=50, MAX_DAILY_RUNS=12
• Per-run cap: SCHEDULED_JOBS_PER_PLATFORM=10
• Interval randomness:
  • MIN_SCRAPE_INTERVAL_MINUTES=60
  • MAX_SCRAPE_INTERVAL_MINUTES=150
  • SCRAPE_INTERVAL_JITTER_PERCENT=15
• “New job” definition: JOB_POSTED_WITHIN_MINUTES (e.g. 1440 = last 24 hours)

LinkedIn efficiency controls (keep small for safety)

• MAX_LIST_PAGES_PER_RUN=2 — number of list pages fetched per run (start, start+25, ...)
• MAX_JOB_LINKS_TO_CONSIDER=150 — hard cap on candidate links per run
• STOP_AFTER_CONSECUTIVE_OLD_DETAILS=8 — stop early when detail pages are consistently older than threshold

---

API

Method	Path	Notes
GET	/jobs	Query: ?page=, ?limit=, optional ?from=
POST	/scraper/test	Verification: 1 random keyword, store first job that passes, then stop; 202 (409 if already running)
POST	/gemini/test	Verification: analyze jobs, stop after first 2 shortlisted; 202 (requires GEMINI_API_KEY, RESUME)
POST	/email/test	Verification: send up to 2 emails (2 jobs per email), then stop; 202
POST	/test/all	Phase 1 = verification scrape; then analyze → email; returns 202
GET	/health	In production, returns 503 if MongoDB is down
GET	/ping	Always 200 keepalive

Testing / verification vs normal scheduled

• POST /scraper/test (manual trigger)
  Verification mode: the scraper picks one random keyword from JOB_KEYWORDS, fetches jobs for that keyword only, iterates through them, and stops as soon as one job is successfully stored. Used to confirm that scraping, filtering, and storage work.

• Scheduled job (autonomous)
  Normal mode: each run picks one random keyword from JOB_KEYWORDS, fetches jobs for that keyword, evaluates every job against the defined filters, and stores all that pass. No early stop after the first insert.

• POST /gemini/test (manual trigger)
  Verification mode: analyzes unanalyzed jobs and stops as soon as 2 are shortlisted. Used to confirm analysis and shortlisting work.

• Gemini cron (10 AM / 3 PM IST)
  Normal mode: analyzes all unanalyzed jobs, shortlists every good match, and runs to the end with no early stop.

• POST /email/test (manual trigger)
  Verification mode: sends at most 2 emails (2 jobs per email), then stops.

• Email cron (11 AM / 4 PM IST)
  Normal mode: sends all unemailed shortlisted jobs in batches of 2 per email (e.g. 10 jobs → 5 emails).

---

Safety & production behavior

• Hard limits: daily run cap, daily job cap, minimum interval gate
• Randomization: next run time is randomized (min/max + jitter)
• Stop-on-block: if a CAPTCHA/block page is detected, the run stops immediately
• Failure circuit breaker: repeated failures disable scraping until manual reset (see ScrapeState)
• No secrets in logs: logging is intentionally minimal and safe
• DB consistency: duplicates are prevented by MongoDB unique indexes and atomic inserts

---

Project structure

src/
├── api/routes/       # HTTP endpoints
├── config/           # env-based config
├── cron/             # analyze, email, delete
├── lib/              # logger, safety-gate, time helpers, utils
├── models/           # Job, ScrapeState, ShortlistedJob
├── scheduler/        # internal randomized scheduler
├── scrapers/         # LinkedIn scraper + orchestrator
└── services/         # Gemini, email

---

Deployment notes

• Set NODE_ENV=production
• Provide secrets via your platform’s secret manager (do not commit .env)
• Ensure Playwright Chromium is installed (npm run setup) on the target environment
• For structured logs in production, set LOG_FORMAT=json
• To force UTC timestamps in logs, set LOG_TIMEZONE=UTC (default is IST)

---

Disclaimer

Personal use only. Use responsibly and comply with the terms of service of any websites you access.