feat(twitter): full-archive likes/bookmarks sync with JSONL streaming#2143
feat(twitter): full-archive likes/bookmarks sync with JSONL streaming#2143naughtyGitCat wants to merge 3 commits into
Conversation
Rebase our xfetch-oriented OpenCLI mods onto upstream main organically: keep the 1.8.x likes/bookmarks media metadata and auth hardening, then add --all/--resume-file/--output-file JSONL streaming with U+2028/U+2029 escaping, raise the full-archive page budget, retry browser lease close failures, and expose browser tab current-window diagnostics.
--max-pages is a safety budget, not archive exhaustion. Keep the resume file and report complete=false so full-sync can continue instead of restarting from the top.
Follow-up (manual smoke + fix)Pushed Bug found in smoke
That made safety-budget stops unresumable. Fixed so only true timeline exhaustion clears resume state; partial runs now return Verification
NoteLocal Browser Bridge extension was still on v1.7.2 during smoke (daemon 1.8.6). |
The full-sync branch adds `browser tab current-window`, so the nested tab help snapshot must count 5 commands instead of 4.
Request: please approve workflowsThis is a first-time fork PR, so GitHub is holding CI behind the “Approve and run workflows” gate. All check suites currently report Once approved, these should run:
Pre-approval signalWhile upstream PR workflows were blocked, the same change set was exercised on the fork and locally:
CI failure already fixedFork CI previously failed one unit test after adding Fixed in Thanks for approving the workflows when you get a chance. |
Summary
Adds full-archive support for Twitter/X likes and bookmarks, so large personal archives can be synced without loading every post into memory.
twitter likes/twitter bookmarks:--allto paginate until exhausted--resume-filefor crash-safe continuation--output-fileto stream results as JSONL instead of one large in-memory array--max-pagessafety cap (raised automatically with--all)has_media/media_urls/media_posters), andtop-by-engagementcloseWindowfailures and retry once with optionalOPENCLI_VERBOSEdiagnosticsopencli browser tab current-windowto inspect the focused real Chrome window when leftover sessions are suspected--max-pagesno longer pretends the archive is complete. Resume state is kept, and JSONL mode returnscomplete: falsepluscursor/resumeFileso the next run can continue.This is used by a local archive tool that needs to pull full likes/bookmarks histories (tens of thousands of posts) reliably.
Test plan
node --check clis/twitter/likes.jsnode --check clis/twitter/bookmarks.jsnpm run buildnpx vitest run src/execution.test.ts clis/twitter/likes.test.js clis/twitter/bookmarks.test.js(48 passed)opencli twitter likes --help/opencli twitter bookmarks --helpshow the new flags--all --output-file ... --resume-file ... --max-pages 1streams partial results, keeps resume, and continues without duplicates on the next runOPENCLI_VERBOSE=1close is loggedManual smoke notes (local 1.8.6 build)
likes --limit 5/bookmarks --limit 5OK--max-pages 1runs, 0 duplicates--max-pages 1runs, 0 duplicatesOPENCLI_VERBOSE=1logsclosed browser window for twitter/{likes,bookmarks}tab current-windowneeds a matching extension rebuild/reload to expose the new op