Skip to content

🔁 feat: Add a Migration to Stored File Links and Back - #16916

Open
Vidminas wants to merge 4 commits into
LibreChat-AI:devfrom
Vidminas:feat/stored-file-links-migration
Open

Vidminas wants to merge 4 commits into
LibreChat-AI:devfrom
Vidminas:feat/stored-file-links-migration

Conversation

@Vidminas

@Vidminas Vidminas commented Oct 9, 2026 •

Copy link
Copy Markdown

Pull Request

Before submitting, please review the Contributing Guide.

Documentation changes belong in the LibreChat documentation repository.

👍

Summary

Records written before STORAGE_PROXY_FILES was turned on hold presigned S3 URLs or Azure blob URLs, in file records and in every copy made of them. The file list and S3 avatars convert as they load. Messages, tool calls, agent avatars and Azure user avatars never do, so their images keep expiring (S3) or keep pointing at a container that may no longer be public (Azure).

npm run migrate:stored-file-links rewrites them once:

  • files: filepath, plus storageKey for S3 records that lack it, which the stored-file route uses to find a record;
  • messages: files, attachments, steer-part files and image_file parts;
  • tool calls: attachments;
  • agents and users: avatars.

How it runs:

  • It works in batches, under system context like the other cross-tenant migrations, and it's idempotent.
  • :dry-run reports counts without writing.
  • :revert writes storage links back, for turning the proxy off.
  • It refuses a direction the current setting wouldn't serve.

The read path is untouched.

Covers the conversation-history part of #16628: stored Azure blob URLs in messages and attachments become stored-file links.

Related to #16912. Depends on #16915

How it works

config/migrate-stored-file-links.js     # queries each collection, batches bulkWrite updates
packages/api/src/storage/proxy/relink.ts
  Relinker { source, toStored(link, key?), toStorage(link) }
  relinkLink / relinkFileRef / relinkMessage
packages/api/src/storage/s3/source.ts     s3Relinker              # presigned ⇄ stored; revert re-signs
packages/api/src/storage/azure/source.ts  createAzureRelinker(url) # blob URL ⇄ stored; keeps ?manual=

Recognising a link without a source. A user's avatar is a bare string with no source field, so each relinker identifies its own links from the URL. S3 links are always presigned. Azure links start with the configured container's URL.

Type of change

  • Bug fix
  • Feature
  • Refactor
  • Performance improvement
  • Breaking change
  • Documentation
  • Translation
  • Tests / tooling / CI

Testing

  1. With old S3 or Azure records present, set STORAGE_PROXY_FILES=true and run npm run migrate:stored-file-links:dry-run. It reports counts and writes nothing.
  2. Run npm run migrate:stored-file-links. Old conversations' images, agent avatars and user avatars load through /api/stored-files/.... A second run reports 0 changes.
  3. Run STORAGE_PROXY_FILES=false npm run migrate:stored-file-links:revert, then restart with the setting off. The links are storage URLs again.

Tested environments/configuration:

Automated tests:

  • npm run static-checks -- --against upstream/dev --full passes.

New tests:

  • proxy/__tests__/relink.test.ts:
    • Azure links in both directions, with queries kept;
    • matching by a record's source, and for bare links;
    • messages, with only the changed fields returned.
  • s3/__tests__/source.test.ts: s3Relinker, in both directions.
  • config/__tests__/migrate-stored-file-links.spec.js (in-memory MongoDB):
    • a dry run;
    • S3 and Azure records across files, messages, tool calls, agents and users;
    • idempotence;
    • revert;
    • the setting guard.

Screenshots / recordings

No user-facing change.

Risk / compatibility

  • It writes to messages and file records. Run :dry-run first. :revert restores storage links.
  • Revert signs new S3 links, which expire as presigned links always have.
  • Large message collections: the walk reads each matching message's files, attachments and content. --batch-size controls the write batches.

Checklist

  • I reviewed my own changes
  • Relevant tests have been added or updated
  • Existing relevant tests pass
  • The change does not introduce new warnings or errors
  • User-facing or complex behavior is documented where necessary
  • Required dependency changes have been merged/published
  • Required documentation PR: 🔁 docs: Document Migrating to Stored File Links and Back docs#819

Vidminas and others added 4 commits October 9, 2026 09:50
The fileAccess middleware decided inline whether a user may access a
file: same tenant for tenant-scoped files, then ownership, then access
through the agents the file is attached to. The decision now lives in
canAccessFile(user, file), which the middleware calls and exports, so a
route that already holds a file record can apply the same rules without
going through req.params.file_id.

Behaviour is unchanged; the cross-tenant denial is still logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With fileStrategy s3, browsers load images, previews and downloads
straight from presigned URLs. Those links expire after
S3_URL_EXPIRY_SECONDS and are stored in messages and file records, so
images break once they expire, and the bucket has to answer requests
from the internet, so it cannot be kept private to the deployment's
network (for instance with a bucket policy on aws:SourceVpce).

STORAGE_PROXY_FILES=true makes storage strategies link files to
LibreChat instead: /api/stored-files/<source>/<key>, which never expires.
The route reads the object through the strategy's StoredFileSource
adapter and streams it to viewers who may read it. This commit adds the
storage-agnostic route and links, and the S3 adapter. With the proxy on,
getS3URL returns stored file links, extractKeyFromS3Url reads them back
to keys, presigned links are replaced where the file list and avatars
already renew links, and getS3DownloadURL returns no direct link, so the
download and shared-link routes stream through their own checks. With it
off, stored links go back to presigned ones the same way. Off by
default.

The route signs the viewer in from the session cookie, as /images does,
since an image tag sends no bearer token; authenticateViewer carries the
image route's account checks (retired tokens, required 2FA enrollment).
Within the viewer's tenant it serves the owner named in the key, avatars
to any signed-in user (as CloudFront avatar cookies do), and anyone the
stored file's own rules admit through canAccessFile, such as a file on
an agent shared with the viewer; everyone else gets a 404. The file
lookup is scoped to the key's owner and source. Uploads are now served
from the app's own origin, so only raster images are sent inline; every
response carries a sandboxing CSP and nosniff.

Related to LibreChat-AI#16912

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With fileStrategy azure_blob, LibreChat stores each blob's plain URL, so
browsers can only display images from a container with public blob
access. A private container (AZURE_STORAGE_PUBLIC_ACCESS=false, or an
account that disallows public access) breaks every image and preview.

This adds the Azure adapter for the stored-file route. With
STORAGE_PROXY_FILES=true, uploads, saved buffers and avatars in the
configured container store /api/stored-files/azure_blob/<blob path>
instead of the blob URL, getAzureFileStream and deleteFileFromAzure read
those links back to blob paths, and the route streams blobs to viewers
who may read them under the same rules as S3. Azure keeps avatars beside
other images, so the adapter marks avatar-<ts> and agent-<id>-avatar-<ts>
files as avatars. Blobs in any other container keep their URLs, since a
stored link names no container. Off by default.

Related to LibreChat-AI#16912

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records written before STORAGE_PROXY_FILES was turned on hold presigned
S3 URLs or Azure blob URLs, in file records and in every copy made of
them. The file list and S3 avatars convert as they load, but messages,
tool calls, agent avatars and Azure user avatars never do, so their
images keep expiring (S3) or keep pointing at a container that may no
longer be public (Azure).

npm run migrate:stored-file-links rewrites them once, in batches, under
system context like the other cross-tenant migrations:

- files: filepath, and storageKey for S3 records that lack it, which the
  stored-file route uses to find a record;
- messages: files, attachments, steer-part files and image_file parts;
- tool calls: attachments;
- agents and users: avatars.

Each storage source converts its own links through a Relinker (S3 and
Azure), so the walk is storage-agnostic. It is idempotent, has a
:dry-run, and :revert writes the storage service's links back for
turning the proxy off. It refuses a direction the current
STORAGE_PROXY_FILES setting would not serve. The read path is untouched.

Related to LibreChat-AI#16912

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 10:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Vidminas
Vidminas marked this pull request as ready for review October 9, 2026 11:00
@codegraph-librechat codegraph-librechat Bot added 🗺️ Backend Infra codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) 🛡️ security review 🗺️ Platform Security codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) 🗺️ File Storage codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) labels Oct 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🗺️ Backend Infra codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) 🗺️ File Storage codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) 🗺️ Platform Security codegraph: the taxonomy area this belongs to (classifier, confidence ≥ 0.9) 🛡️ security review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants