feat: Integrate StorageService for large controller data #22943
+628
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add StorageService integration to enable controllers to offload large data
from Redux state to persistent storage.
**Problem**:
- 10.79 MB Engine state with 92% in 2 controllers
- Slow app startup parsing large state
- High memory usage from rarely-accessed data
**Solution**:
StorageService integration with FilesystemStorage adapter:
- Platform-agnostic service via messenger actions
- Event system for reactive patterns
- FilesystemStorage adapter for mobile persistence
- Namespace isolation (storageService:{namespace}:{key})
**Implementation**:
- StorageService registered in Engine
- FilesystemStorage adapter with namespace filtering
- Messenger delegation configured
- Service available for all controllers
- Uses STORAGE_KEY_PREFIX constant
**Impact** (when controllers migrate):
- 92% state reduction potential (10.79 MB → 0.85 MB)
- SnapController: 6.09 MB sourceCode candidate
- TokenListController: 4.09 MB cache candidate
**Files**:
- storage-service-init.ts: FilesystemStorage adapter
- storage-service-messenger.ts: Messenger factory
- Engine types/messengers: Registration
- Tests: Integration verified
4 tasks
Adapter methods now receive namespace and key separately: - getItem(namespace, key) - Builds storageService:namespace:key - setItem(namespace, key, value) - Builds full key internally - removeItem(namespace, key) - Builds full key internally Mobile adapter builds keys as: storageService:namespace:key Using STORAGE_KEY_PREFIX constant from core package. Matches new StorageAdapter interface from @metamask/storage-service.
…serialization - Define StoredDataWrapper locally (no longer exported from core) - Adapter now fully responsible for: - Building full storage keys (storageService:namespace:key) - Wrapping data with metadata (timestamp) - Serialization/deserialization - Aligns with updated StorageAdapter interface from core
…/metamask-mobile into feature/storage-service
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Required for messenger parent type resolution - the Messenger constructor validates that both Actions and Events are in the parent's types.
…NAMES StorageService is a stateless service, not a controller with state. Excluding it from ControllerPersistedState fixes the TS2536 error.
- Add jest.clearAllMocks() in beforeEach to fix mock accumulation bug - Add comprehensive tests for mobileStorageAdapter methods: - getItem: happy path, null, parse error, storage error - setItem: happy path, Android device, storage error - removeItem: happy path, storage error - getAllKeys: happy path, empty, null keys, storage error - clear: happy path, null keys, empty namespace, storage error - Update existing tests to verify getAllKeys and clear methods
…rage mock FilesystemStorage.getItem returns string | undefined, not string | null
Change 'handles empty namespace gracefully' to 'removes zero keys and logs count when namespace has no matching entries'
- getItem now throws on error instead of returning null - getAllKeys now throws on error instead of returning [] - Consistent with setItem, removeItem, and clear which already throw - Allows callers to distinguish 'not found' from 'error'
- Use Json type from @metamask/utils for type safety - Remove StoredDataWrapper - just stringify/parse values directly - Update tests to reflect simplified storage format
Contributor
🔍 Smart E2E Test Selection
click to see 🤖 AI reasoning detailsThis PR adds a new StorageService infrastructure to the Engine core module. Here's my analysis: Changes Summary:
Impact Analysis:
Risk Assessment: MEDIUM
Test Selection: SmokeCore
I recommend running SmokeCore to verify:
Why not other tags:
Confidence: 85%
|
|
github-merge-queue bot
pushed a commit
to MetaMask/core
that referenced
this pull request
Nov 28, 2025
# Add StorageService for Large Controller Data
## Explanation
### What is the current state and why does it need to change?
**Current state**: MetaMask Mobile Engine state is 10.79 MB, with 92%
(9.94 MB) concentrated in just 2 controllers:
- **SnapController**: 5.95 MB of snap source code (55% of total state)
- **TokenListController**: 3.99 MB of token metadata cache (37% of
state)
This data is **rarely accessed** after initial load but stays in Redux
state, causing:
- Slow app startup (parsing 10.79 MB on every launch)
- High memory usage (all data loaded even if not needed)
- Slow persist operations (up to 6.26 MB written per controller change)
**Why change**: Controllers need a way to store large,
infrequently-accessed data outside of Redux state while maintaining
platform portability and testability.
### What is the solution and how does it work?
**New package**: `@metamask/storage-service`
A platform-agnostic service that allows controllers to offload large
data from state to persistent storage via messenger actions.
**How it works**:
1. Controllers call `StorageService:setItem` via messenger to store
large data
2. StorageService saves to platform-specific storage (FilesystemStorage
on mobile, IndexedDB on extension)
3. StorageService publishes events
(`StorageService:itemSet:{namespace}`) so other controllers can react
4. Controllers call `StorageService:getItem` to load data lazily (only
when needed)
**Storage adapter pattern**:
```typescript
// Service accepts platform-specific adapter (like ErrorReportingService)
const service = new StorageService({
messenger,
storage: filesystemStorageAdapter, // Mobile provides this
});
```
**Example controller usage**:
```typescript
// Store data (out of state)
await this.messenger.call(
'StorageService:setItem',
'MyController',
'dataKey',
largeData,
);
// Load on demand (lazy loading)
const data = await this.messenger.call(
'StorageService:getItem',
'MyController',
'dataKey',
);
// Subscribe to changes (optional)
this.messenger.subscribe(
'StorageService:itemSet:MyController',
(key, value) => {
// React to storage changes
},
);
```
### Why this architecture?
**Platform-agnostic**: Service defines `StorageAdapter` interface;
clients provide implementation (mobile: FilesystemStorage, extension:
IndexedDB, tests: in-memory)
**Messenger-integrated**: Controllers access storage via messenger
actions, no direct dependencies
**Event-driven**: Controllers can subscribe to storage changes without
coupling
**Testable**: InMemoryAdapter provides zero-config testing (no mocking
needed)
**Proven pattern**: Follows ErrorReportingService design (accepts
platform-specific function)
### Expected impact?
**With both controllers optimized**:
- State: 10.79 MB → 0.85 MB (**92% reduction**)
- App startup: 92% faster state parsing
- Memory: 9.94 MB freed
- Disk I/O: Up to 9.94 MB less per persist
**This PR adds infrastructure** - Controllers can now use
StorageService. Separate PRs will integrate with SnapController and
TokenListController.
## References
- **ADR**:
[0017-storage-service-large-data.md](https://github.com/MetaMask/decisions/blob/core/0017-storage-service-large-data/decisions/core/0017-storage-service-large-data.md)
- **Mobile PR**: MetaMask/metamask-mobile#22943
## Checklist
- [x] I've updated the test suite for new or updated code as appropriate
- 100% test coverage (44 tests)
- Tests for all storage operations, namespace isolation, events, error
handling
- Real-world usage test (simulates 6 MB snap source code)
- [x] I've updated documentation (JSDoc, Markdown, etc.) for new or
updated code as appropriate
- Complete README with examples
- JSDoc for all public APIs
- Architecture documentation in ADR
- [x] I've communicated my changes to consumers by updating changelogs
for packages I've changed, highlighting breaking changes as necessary
- CHANGELOG.md created for initial release
- No breaking changes (new package)
- [x] I've prepared draft pull requests for clients and consumer
packages to resolve any breaking changes
- N/A - No breaking changes
- Consumer PRs will be created after this is merged and released
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces `@metamask/storage-service` with messenger-based APIs and
an in-memory adapter to offload large controller data, plus repo wiring
and ownership updates.
>
> - **New Package**: `packages/storage-service`
> - `StorageService`: Messenger-exposed methods `setItem`, `getItem`,
`removeItem`, `getAllKeys`, `clear`; publishes
`StorageService:itemSet:{namespace}` events.
> - `InMemoryStorageAdapter`: Default non-persistent adapter
implementing `StorageAdapter` with key prefix `storageService:`.
> - Types/exports: `StorageAdapter`, `StorageGetResult`, messenger
types, action types file, constants `SERVICE_NAME`,
`STORAGE_KEY_PREFIX`.
> - **Tests & Docs**:
> - Comprehensive unit tests for service and adapter (namespace
isolation, events, errors); Jest config with 100% coverage thresholds.
> - `README.md` with usage/examples; `CHANGELOG.md` initialized;
dual-license files.
> - **Repo Wiring**:
> - Add package to `tsconfig.build.json`, `yarn.lock`, exports/index,
and TypeDoc config.
> - Update `.github/CODEOWNERS` and `teams.json` to include
`storage-service` ownership.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
6bf384a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Mark Stacey <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integrate StorageService for Large Controller Data
Description
Integrates
@metamask/storage-serviceto enable controllers to offload large, infrequently-accessed data from Redux state to persistent storage.CHANGELOG entry: null
Problem
State bloat impacting mobile performance:
Root cause: Controllers store large data (snap source code, token caches) in Redux state instead of on disk.
Solution
Add StorageService integration - Controllers can now store large data via messenger:
What's Included
StorageService Integration:
storage-service-init.ts- FilesystemStorage adapter for mobilestorage-service-messenger.ts- Messenger factoryFilesystemStorage Adapter (mobile-specific):
StorageAdapterinterfacegetAllKeys,clear)STORAGE_KEY_PREFIXconstant (storageService:)Impact
Immediate (infrastructure ready):
When controllers migrate (separate PRs):
Architecture
Platform-agnostic service:
Key format:
storageService:{namespace}:{key}Example:
storageService:SnapController:snap-id:sourceCodeTesting
Test coverage:
Follow-up Work
Controller migrations (separate PRs):
Each migration requires:
Related
@metamask/storage-service@^1.0.0(from core)Checklist
Notes
This PR adds infrastructure only - No immediate user-facing changes.
Performance improvements come when controllers are migrated to use StorageService (separate PRs for each controller).
Why FilesystemStorage? Optimized for large files (multi-MB). MMKV is better for small, frequently-accessed data (< 1 MB).
Note
Introduces
StorageService(with FilesystemStorage adapter) and wires it into Engine, messengers, and types, with tests.StorageServiceviastorage-service-initinEngine.tsand add toSTATELESS_NON_CONTROLLER_NAMES.CONTROLLER_MESSENGERSusinggetStorageServiceMessenger.mobileStorageAdapterincontrollers/storage-service-init.tsusingredux-persist-filesystem-storagewith namespaced keys (STORAGE_KEY_PREFIX), timestamped wrapping, iOS flag, and error logging.messengers/storage-service-messenger.tsfactory and expose through index mapping.types.tsto includeStorageServiceactions/events and controller references; update required/optional controller typings and controller name unions.controllers/storage-service-init.test.ts) and messenger (messengers/storage-service-messenger.test.ts).@metamask-previews/storage-servicetopackage.json.Written by Cursor Bugbot for commit 08c5d98. This will update automatically on new commits. Configure here.