Replace per-file HEAD calls with a single ListBlobs metadata cache

[tomlm]

Previously, every FileExists(string), FileLength(string), and OpenInput(string, IOContext) call issued an individual GetProperties (HEAD) request to Azure Blob Storage. For an index with N files, opening or refreshing a reader triggered N round-trips. This change introduces a BlobMetadata cache populated by a single GetBlobsByHierarchy call (which returns properties for all blobs in one page). The cache is:

• refreshed by ListAll(), which Lucene always calls first when opening or re-checking an index
• lazily populated on first access if ListAll() has not yet been called
• updated in-place after AzureIndexOutput.Dispose() completes an upload, using the already-known local length — preventing an unnecessary re-download when Lucene reads a file it just wrote
• invalidated per-entry on DeleteFile()

AzureIndexInput now reads ContentLength from the cache instead of calling GetProperties on the blob before deciding whether to re-download. Net result: N HEAD calls per reader open/refresh replaced by 1 ListBlobs call.

Net result is that it's ~2X to 2.5X faster than old AzureDirctory code.

[Copilot]

Pull request overview

This PR reduces Azure Blob Storage round-trips by replacing per-file property (HEAD) requests with a container listing–backed in-memory metadata cache, improving Lucene index open/refresh performance.

Changes:

• Add a blob metadata cache to AzureDirectory populated via GetBlobsByHierarchy() and used by ListAll(), FileExists(), FileLength(), and OpenInput().
• Update AzureIndexInput to use cached ContentLength instead of calling GetProperties().
• Update AzureIndexOutput to update the metadata cache after successful upload; bump package/version numbers.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

File	Description
source/Lucene.Net.Store.Azure/Lucene.Net.Store.Azure.csproj	Version bump for the new release containing the metadata-cache optimization.
source/Lucene.Net.Store.Azure/AzureIndexOutput.cs	Updates the metadata cache after upload to avoid immediate re-downloads.
source/Lucene.Net.Store.Azure/AzureIndexInput.cs	Uses cached blob length when deciding whether to download into the local cache.
source/Lucene.Net.Store.Azure/AzureDirectory.cs	Introduces and wires up the metadata cache; switches FileExists/FileLength to cache-backed lookups.

Comments suppressed due to low confidence (1)

source/Lucene.Net.Store.Azure/AzureIndexInput.cs:61

• The catch (RequestFailedException err) block in this length-comparison path no longer makes sense because the code inside the try no longer performs any Azure SDK call that could throw RequestFailedException. This looks like leftover logic from the previous GetProperties() approach; consider removing the catch or replacing it with handling that matches the new cache-based flow.

                {
                    try
                    {
                        long cachedLength = CacheDirectory.FileLength(name);
                        long blobLength = _azureDirectory.GetCachedMetadata(name)?.ContentLength ?? 0;
                        if (cachedLength != blobLength)
                            fileNeeded = true;
                    }
                    catch (RequestFailedException err)
                    {
                        // if blob not found
                        if (err.Status == 404)
                        {
                            // then we should remove from cache directory.
                            CacheDirectory.DeleteFile(name);
                            Debug.WriteLine($"{_azureDirectory.Name} {name} Does not exist");
                            throw new FileNotFoundException(name, err);
                        }
                    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.