Perf/collection mount#1
Open
chirag-bruno wants to merge 17 commits into
Open
Conversation
Introduces a sqlite-backed Index class under bruno-electron's mount pipeline that provides git-like status/stage operations against a collection on disk. Status walks the collection, compares mtime then content hash against stored entries, and reports added/updated/removed. Stage upserts or deletes individual entries; transaction wraps batches. Uses node:sqlite with an in-memory option for tests, path.matchesGlob for caller-supplied denylist filtering, and SHA-256 of file bytes for change detection.
Adds a Parser class under bruno-electron's mount pipeline that runs
file parsing across a piscina worker pool. Each worker reads the file,
captures mtime and content hash, and dispatches to the @usebruno/filestore
parser by (format, type): bru/yml requests, collections, folders,
environments, dotenv, and json config.
Public API:
- parse(collectionPath, relativePath, format, type) -> Promise<result>
- parseAll(collectionPath, entries[]) -> Promise<Map>
- results getter, clear(), close()
Result shape: { relativePath, mtime, hash, data, format, type }.
Errors are surfaced both as promise rejections and as entries in the
results Map ({ relativePath, error }), so parseAll does not fail the
batch when a single file is malformed.
Pins piscina at 5.1.4.
Adds buildTree(collectionPath, parserResults) under bruno-electron's
mount pipeline. Consumes the Map produced by the worker-pool Parser
and emits a tree shaped to match what the renderer already expects:
{ pathname, brunoConfig, root, items, environments }
Folders are synthesized from nested request paths; folder.bru data
attaches name/seq/root to its folder (and creates the folder even
when it has no descendants). collection.bru/yml and bruno.json are
extracted into tree.root and tree.brunoConfig. environments/*.bru
land in tree.environments instead of tree.items.
Sort fidelity matches the current renderer: folders before requests
at every level, requests sorted by seq, folders sorted via
sortByNameThenSequence (alpha baseline + splice by seq + group on
collision) ported verbatim.
Uids are deterministic: sha256(toPosix(absolutePath)). Hashing the
absolute path (not the relative path) ensures two collections with
identical internal layouts get distinct uids. Updates git-lite's
Index.id to the same formula so snapshot and renderer share one
identifier scheme.
Introduces a callback-based Watcher around chokidar that surfaces
'add'/'change'/'unlink'/'addDir'/'unlinkDir' events with the same
ignored semantics as the existing collection watcher (DENY_DIRS plus
caller-supplied denylist globs).
The Mount orchestrator at ipc/mount/index.js plugs the four mount
modules together. On start, it hydrates an in-memory state Map from
the snapshot DB, runs an initial reconcile against disk, emits the
first tree, and awaits chokidar's ready before returning. Subsequent
FS events trigger a debounced reconcile:
Index.status() -> Parser.parseAll(added+updated) ->
Index.transaction(stage all) -> buildTree -> onTree(tree)
Path-based classify() dispatches each relative path to the right
(format, type) pair for the parser; files that don't match any
known type are skipped.
Adds Index.entries(collectionPath) to read back all stored rows as a
Map<relPath, { data }>, used by Mount to repopulate state across
restarts.
Adds a 'mount-v2' beta preference that, when enabled, routes renderer:mount-collection through the new sqlite-backed Mount pipeline (Index + Parser + Watcher + buildTree) instead of the existing chokidar-driven per-file event stream. Main: - New src/ipc/mount-v2.js owns singleton Index and Parser, manages per-collection Mount instances, and sends main:collection-tree-loaded plus main:collection-loading-state-updated. - renderer:mount-collection branches on the flag. - renderer:remove-collection unmounts the Mount. - before-quit shuts down singletons. Renderer: - BETA_FEATURES.MOUNT_V2 registered, surfaced in Preferences > Beta. - New main:collection-tree-loaded listener in useIpcEvents. - New collectionLoadedFromTree reducer that replaces collection.items, environments, root, and brunoConfig from the tree blob and runs addDepth(items). The renderer's existing collection actions are unchanged; the flag check happens server-side so all entry points pick up the new path.
Moves the mount pipeline v2 toggle from Preferences > Beta into a File cache entry under Preferences > Cache, alongside the existing SSL session cache. The flag itself moves from beta['mount-v2'] to cache.file.enabled with a matching isFileCacheEnabled() helper. Adds renderer:get-file-cache-size and renderer:clear-file-cache IPCs backed by the snapshot sqlite file (Index.clear() drops all rows, Index.dbPath exposes the location). The Cache UI shows the current cache size and a Clear cache button per row. Cache section redesigned to a card layout: gray header with title, beta badge where applicable, and toggle; white body with description, size readout (for File cache), and a bordered Clear cache button. Fixes ToggleSwitch using a hardcoded id so multiple instances on the same page now get unique label-input pairs via React.useId(), which was making the SSL toggle redirect clicks to the File cache toggle.
Removes the cache.file.enabled flag branch from renderer:mount-collection in ipc/collection.js, restoring the old handler to its pre-v2 state. The v2 pipeline now has its own dedicated IPC channels: - renderer:mount-collection-v2 - renderer:unmount-collection-v2 - main:collection-loading-state-updated-v2 Both handlers create their own transient directory and use BrowserWindow.fromWebContents(event.sender) so the v2 path no longer shares any code path with the old chokidar-based watcher. Strict separation: the frontend chooses which channel to call based on the flag; main has no flag-branching.
Renderer: - mountCollection picks renderer:mount-collection-v2 when cache.file.enabled is on, else renderer:mount-collection. - removeCollection invokes renderer:unmount-collection-v2 alongside renderer:remove-collection so cleanup runs for either path (both IPCs are idempotent). - New listeners for main:collection-loading-state-updated-v2 and main:bruno-config-update-v2; both dispatch the existing shared reducer actions (updateCollectionLoadingState, brunoConfigUpdateEvent) since the state shape is identical and the trigger paths are separate. Main: - mount-v2 sendTree now runs transformBrunoConfigAfterRead on tree.brunoConfig, stores the result via setBrunoConfig, emits main:bruno-config-update-v2, and only then dispatches the tree. - tree-builder splits yml collection root: when the parsed payload carries both collectionRoot and brunoConfig, hoist brunoConfig to the tree level so the v2 path matches the legacy yml flow.
tree-builder: - Hydrate per-item uuids on request data (params, headers, vars, assertions, body lists) and on examples (deterministic example uid via idFor(absolutePath#example#index) plus parent itemUid). Matches the legacy hydrateRequestWithUuid behavior. - Apply the same hydration to tree.root and each folder.root. mount-v2: - Hydrate environments after build: per-variable uuid plus secret value decryption via EnvironmentSecretsStore + decryptStringSafe. - After mount.start() resolves, look up snapshotManager.getCollection and emit main:hydrate-app-with-ui-state-snapshot-v2 so the renderer can restore tabs and active environment, mirroring the legacy onWatcherSetupComplete path. useIpcEvents: add a v2 snapshot-hydration listener that dispatches the existing hydrateCollectionWithUiStateSnapshot action.
watcher: - On chokidar error with code ENOSPC or EMFILE, close the failing watcher and re-attach with usePolling: true. One-shot fallback; subsequent unrelated errors propagate to onError. Matches the fallback behavior in collection-watcher.js. mount: - Detect renames during reconcile: when an added entry's hash matches a removed entry's hash, treat the pair as a rename and migrate the request uid for the new absolute path. Maintains an in-memory uidOverrides Map so per-session renames preserve the same uid the renderer was already keyed on. tree-builder: - buildTree takes options.uidOverrides; buildRequestNode consults it before falling back to the deterministic sha256(absPath) uid. git-lite: - Index.status includes hash on removed entries so rename detection can match by content.
Adds a TransientWatcher (mount/transient-watcher/index.js) that
watches a collection's temp directory at depth 1, ignores
metadata.json, parses each transient file via the shared Parser,
and maintains an in-memory state Map. Fires onUpdate whenever the
state changes.
mount-v2 now creates one TransientWatcher per collection alongside
the Mount. Both update channels feed a shared emitMergedTree path:
mount.getTree({ transientEntries }) is called with the watcher's
current entries, and the result is dispatched via the existing
v2 tree IPC. Transient items appear as flat top-level leaves with
isTransient: true and participate in the same root-level sort.
tree-builder:
- Splits buildRequestNode into a collectionPath-relative helper and a
buildRequestNodeForAbsolutePath core that accepts an absolute path
directly, so transient nodes (which live outside collectionPath)
reuse the same hydration and uid logic.
- buildTree accepts options.transientEntries; appends them to root
items, flags them isTransient: true, sorts the resulting items.
mount:
- Exposes getTree(options) so callers can pass transient entries
through to buildTree without touching private state. The existing
tree getter delegates to getTree().
mount-v2: - Attach dotenv-watcher (shared with v1) on mountCollectionV2 so the v2 path gets .env/.env.* shell-environment propagation. Detach on unmountCollectionV2. - Subsequent mountCollectionV2 calls for an already-mounted collection refresh the stored win reference and re-emit the current tree plus UI state snapshot. This is how the v2 path recovers after a renderer reload: the renderer re-invokes renderer:mount-collection-v2 on startup; the still-running Mount serves the existing state without re-walking disk. - Move the win reference onto the mount entry so callbacks read it through the entry instead of capturing it in a closure.
Replaces crypto.randomUUID() with sha256-derived ids for every child node generated by hydration. Stable IDs are required for any snapshot/persistence layer that pins to uids across app restarts (otherwise a clean reload reissues random ids every parse). tree-builder: - Per-item uids in request.params/headers/vars.req/vars.res/assertions/ body.formUrlEncoded/body.multipartForm/body.file are now sha256(toPosix(absolutePath)#section#index). - Example child lists follow the same scheme seeded by the example uid. mount-v2 environment hydration: - Variable uid is sha256(envUid#var#name) (falls back to index when name is missing). Stable across restarts; only changes when a variable is renamed.
Bruno's shared uidSchema enforces exactly 21 alphanumeric chars (packages/bruno-schema/src/common/index.js). Full sha256 hex was 64 chars, which made the renderer reject every v2 collection item with 'uid must be 21 characters in length'. Every uid generator in the v2 pipeline now slices the sha256 hex to its first 21 chars: - tree-builder idFor, idForChild, and example uid (toUid helper) - mount Mount.defaultUidFor (rename uid migration) - mount-v2 hydrateEnvironments variable uid (toUid helper) - git-lite Index.idForAbsolutePath (DB id, kept consistent for parity with the renderer-side uid) 21 hex chars = ~84 bits of entropy; collision probability stays negligible for any realistic collection size. Updates the tree-builder spec helper to truncate too so existing deterministic-uid expectations still match.
findCollectionPathByItemPath in ipc/collection.js previously queried only collectionWatcher.getAllWatcherPaths(), which is empty for v2 collections. Renderer-driven actions like renderer:new-request that resolve the owning collection from an item pathname were failing with 'Collection not found for the given pathname' whenever the user had the file cache enabled. mount-v2 now exports getMountedCollectionPaths() which returns the list of currently-mounted v2 collection paths. The path-resolver concatenates both sources before matching. No flow mixing: each pipeline still owns its own registry; the resolver is a read-only directory utility that needs to know about all mounted collections regardless of which pipeline put them there.
collectionLoadedFromTree was replacing collection.items wholesale on every tree push, which wiped every renderer-only field on a request item (response, requestSent, cancelTokenUid, run timeline, ...). Saving a request and then re-receiving the tree would erase the visible response. The reducer now merges by uid (stable in v2): - Matched request items keep all existing fields; only the file-derived fields (name, type, seq, tags, request, settings, examples, filename, pathname, partial, loading, size, error, isTransient) are overwritten. - Seq-only updates take a fast path: only seq (and draft.seq if a draft exists) are updated, preserving the rest. Matches the legacy collectionChangeFileEvent behavior for sort/re-order operations. - Drafts are cleared only when the file content matches the draft (autosave caught up). Otherwise the draft survives, preserving characters typed after the most recent save. - Matched folders keep runtime state (collapsed, etc.); only the file-derived fields are overwritten. Children are merged recursively. - New items (no existing uid) are inserted as-is. - Existing items missing from the new tree are dropped (deleted on disk).
Three coupled fixes that together let v2 own the scratch-collection flow end-to-end: 1. workspaces/actions.js mountScratchCollection branches on cache.file.enabled. When the flag is on it calls renderer:mount-collection-v2 instead of the v1 renderer:add-collection-watcher. Eliminates the v1 chokidar watcher that was firing for scratch collections regardless of the flag. 2. tasks/middleware.js extracts the open-request logic into tryOpenRequestTasks and adds a parallel listener on collectionLoadedFromTree. The v2 reducer never dispatched collectionAddFileEvent, so the OPEN_REQUEST task for a freshly created transient request stayed queued and the tab never opened. 3. collections/index.js collectionLoadedFromTree annotates items as isTransient when their pathname lives under state.tempDirectories[collectionUid]. Mirrors how the v1 per-event reducer computed isTransientFile, so scratch collections (where the temp dir IS the collection path) get every item flagged transient. Regular collections are unaffected since their items live under collection.pathname, not the temp dir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Contribution Checklist:
Note: Keeping the PR small and focused helps make it easier to review and merge. If you have multiple changes you want to make, please consider submitting them as separate pull requests.
Publishing to New Package Managers
Please see here for more information.