Skip to content

feat(twitter): full-archive likes/bookmarks sync with JSONL streaming#2143

Open
naughtyGitCat wants to merge 3 commits into
jackwener:mainfrom
naughtyGitCat:codex/twitter-full-sync-on-1.8.6
Open

feat(twitter): full-archive likes/bookmarks sync with JSONL streaming#2143
naughtyGitCat wants to merge 3 commits into
jackwener:mainfrom
naughtyGitCat:codex/twitter-full-sync-on-1.8.6

Conversation

@naughtyGitCat

@naughtyGitCat naughtyGitCat commented Jul 17, 2026

Copy link
Copy Markdown

Summary

Adds full-archive support for Twitter/X likes and bookmarks, so large personal archives can be synced without loading every post into memory.

  • twitter likes / twitter bookmarks:
    • --all to paginate until exhausted
    • --resume-file for crash-safe continuation
    • --output-file to stream results as JSONL instead of one large in-memory array
    • --max-pages safety cap (raised automatically with --all)
  • Escape U+2028 / U+2029 when writing JSONL so tweet text cannot split logical records across physical lines
  • Preserve existing 1.8.x behavior for limited fetches, media metadata (has_media / media_urls / media_posters), and top-by-engagement
  • Harden browser lease close: propagate closeWindow failures and retry once with optional OPENCLI_VERBOSE diagnostics
  • Add opencli browser tab current-window to inspect the focused real Chrome window when leftover sessions are suspected
  • Fix: hitting --max-pages no longer pretends the archive is complete. Resume state is kept, and JSONL mode returns complete: false plus cursor / resumeFile so the next run can continue.

This is used by a local archive tool that needs to pull full likes/bookmarks histories (tens of thousands of posts) reliably.

Test plan

  • node --check clis/twitter/likes.js
  • node --check clis/twitter/bookmarks.js
  • npm run build
  • npx vitest run src/execution.test.ts clis/twitter/likes.test.js clis/twitter/bookmarks.test.js (48 passed)
  • opencli twitter likes --help / opencli twitter bookmarks --help show the new flags
  • Manual: limited likes/bookmarks fetch still works unchanged
  • Manual: --all --output-file ... --resume-file ... --max-pages 1 streams partial results, keeps resume, and continues without duplicates on the next run
  • Manual: browser command closes the automation lease; with OPENCLI_VERBOSE=1 close is logged

Manual smoke notes (local 1.8.6 build)

  • limited likes --limit 5 / bookmarks --limit 5 OK
  • likes resume: 100 → 198 rows across two --max-pages 1 runs, 0 duplicates
  • bookmarks resume: 95 → 167 rows across two --max-pages 1 runs, 0 duplicates
  • OPENCLI_VERBOSE=1 logs closed browser window for twitter/{likes,bookmarks}
  • note: browser extension currently reports v1.7.2 while daemon is v1.8.6; tab current-window needs a matching extension rebuild/reload to expose the new op

Rebase our xfetch-oriented OpenCLI mods onto upstream main organically:
keep the 1.8.x likes/bookmarks media metadata and auth hardening, then
add --all/--resume-file/--output-file JSONL streaming with U+2028/U+2029
escaping, raise the full-archive page budget, retry browser lease close
failures, and expose browser tab current-window diagnostics.
--max-pages is a safety budget, not archive exhaustion. Keep the resume
file and report complete=false so full-sync can continue instead of
restarting from the top.
@naughtyGitCat

Copy link
Copy Markdown
Author

Follow-up (manual smoke + fix)

Pushed 09531e43 after live smoke testing against a real X session:

Bug found in smoke

--max-pages was treated as successful archive completion:

  • resume file was deleted
  • JSONL mode returned complete: true

That made safety-budget stops unresumable. Fixed so only true timeline exhaustion clears resume state; partial runs now return complete: false + cursor/resumeFile.

Verification

  • unit/adapter: 48 passed (execution + likes + bookmarks)
  • limited likes/bookmarks: OK
  • likes resume across two --max-pages 1 runs: 100 → 198, 0 dupes
  • bookmarks resume: 95 → 167, 0 dupes
  • verbose close path logs closed browser window for twitter/{likes,bookmarks}

Note

Local Browser Bridge extension was still on v1.7.2 during smoke (daemon 1.8.6). browser tab current-window needs the rebuilt extension service worker reloaded; core likes/bookmarks full-sync path does not depend on that op.

The full-sync branch adds `browser tab current-window`, so the nested
tab help snapshot must count 5 commands instead of 4.
@naughtyGitCat

Copy link
Copy Markdown
Author

Request: please approve workflows

This is a first-time fork PR, so GitHub is holding CI behind the “Approve and run workflows” gate. All check suites currently report action_required with 0 jobs executed (latest head 488b2b3b).

Once approved, these should run:

  • CI
  • Doc Check
  • Build Chrome Extension
  • Security Audit

Pre-approval signal

While upstream PR workflows were blocked, the same change set was exercised on the fork and locally:

  • Local focused tests for likes/bookmarks/execution: passed
  • Manual smoke against a real X session:
    • limited likes/bookmarks OK
    • --all --output-file --resume-file --max-pages 1 resume continuity OK (no dupes)
    • browser lease close logged under OPENCLI_VERBOSE=1
    • rebuilt extension exposes browser tab current-window (v1.0.22)

CI failure already fixed

Fork CI previously failed one unit test after adding browser tab current-window:

src/cli.test.ts > renders nested browser parent structured help for a subtree
expected command_count: 4
received command_count: 5

Fixed in 488b2b3b (test(cli): expect browser tab current-window in structured help).

Thanks for approving the workflows when you get a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant