Skip to content

Commit 454e446

Browse files
KSemenenkoKSemenenko
committed
Fix VFS immutable metadata reads and release 10.0.15
1 parent 78e93b4 commit 454e446

6 files changed

Lines changed: 157 additions & 15 deletions

File tree

‎Directory.Build.props‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@
2929
<RepositoryUrl>https://github.com/managedcode/Storage</RepositoryUrl>
3030
<PackageProjectUrl>https://github.com/managedcode/Storage</PackageProjectUrl>
3131
<Product>Managed Code - Storage</Product>
32-
<Version>10.0.14</Version>
33-
<PackageVersion>10.0.14</PackageVersion>
32+
<Version>10.0.15</Version>
33+
<PackageVersion>10.0.15</PackageVersion>
3434

3535
</PropertyGroup>
3636

‎ManagedCode.Storage.VirtualFileSystem/Metadata/IMetadataManager.cs‎

Lines changed: 11 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -275,22 +275,20 @@ protected Dictionary<string, string> ExtractCustomMetadata(IDictionary<string, s
275275

276276
foreach (var kvp in storageMetadata)
277277
{
278-
if (kvp.Key.StartsWith(MetadataPrefix, StringComparison.Ordinal))
278+
if (!kvp.Key.StartsWith(MetadataPrefix, StringComparison.Ordinal))
279279
{
280-
// Skip VFS system metadata
281-
if (kvp.Key.EndsWith(VfsVersionKey, StringComparison.Ordinal) ||
282-
kvp.Key.EndsWith(VfsCreatedKey, StringComparison.Ordinal) ||
283-
kvp.Key.EndsWith(VfsModifiedKey, StringComparison.Ordinal) ||
284-
kvp.Key.EndsWith(VfsAttributesKey, StringComparison.Ordinal) ||
285-
kvp.Key.Contains(VfsCustomPrefix, StringComparison.Ordinal))
286-
{
287-
continue;
288-
}
280+
result[kvp.Key] = kvp.Value;
281+
continue;
282+
}
289283

290-
// Include other custom metadata
291-
var key = kvp.Key[MetadataPrefix.Length..];
292-
result[key] = kvp.Value;
284+
var key = kvp.Key[MetadataPrefix.Length..];
285+
if (key is VfsVersionKey or VfsCreatedKey or VfsModifiedKey or VfsAttributesKey ||
286+
key.StartsWith(VfsCustomPrefix, StringComparison.Ordinal))
287+
{
288+
continue;
293289
}
290+
291+
result[key] = kvp.Value;
294292
}
295293

296294
return result;

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,7 @@ Cloud storage vendors expose distinct SDKs, option models, and authentication pa
108108
- Azure metadata preserves Unicode filenames and other logical string values through a provider-owned ASCII transport envelope; ordinary native ASCII metadata remains interoperable. See [metadata transport](https://github.com/managed-code-hub/Storage/blob/main/docs/Features/provider-azure-blob.md#metadata-transport).
109109
- Virtual File System package provides a file/directory API (`IVirtualFileSystem`) on top of the configured `IStorage` and can cache metadata for faster repeated operations, including browser storage verified through real Playwright flows in both Blazor WebAssembly and Interactive Server hosts.
110110
- For decisions requiring current storage state, `IVirtualFileSystem.StorageFileExistsAsync` bypasses that cache and propagates provider errors. `WriteBytesIfAbsentOrSameAsync` uses an atomic provider capability and accepts only an exact immutable retry. `ManagedCode.Storage.Core.Primitives.VerifiedContentSnapshot` verifies bounded reads by length and SHA-256.
111+
- Metadata supplied to a VFS immutable write is returned by `IVirtualFile.GetMetadataAsync`, including Unicode values and legacy VFS-prefixed user fields. Reserved VFS bookkeeping fields remain private to the overlay.
111112
- Comprehensive automated test suite with cross-provider sync fixtures, multi-gigabyte streaming simulations (4 MB units per "GB"), ASP.NET controller harnesses, SFTP/local filesystem coverage, and Playwright browser verification for browser storage small-file overwrites, concurrent tabs, VFS flows, a fast `128 MiB` browser large-file lane, and a separate `256 MiB` browser stress lane in both Interactive Server and Blazor WebAssembly hosts.
112113
- ManagedCode.Storage.TestFakes package plus Testcontainers-based fixtures make it easy to run offline or CI tests without touching real cloud accounts.
113114

‎Tests/ManagedCode.Storage.Tests/VirtualFileSystem/VfsImmutableMetadataTests.cs‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
using System;
2+
using System.Collections.Generic;
3+
using System.IO;
4+
using System.Text;
5+
using System.Threading.Tasks;
6+
using ManagedCode.Storage.Azure.Extensions;
7+
using ManagedCode.Storage.Core;
8+
using ManagedCode.Storage.Core.Primitives;
9+
using ManagedCode.Storage.Tests.Common;
10+
using ManagedCode.Storage.VirtualFileSystem.Core;
11+
using ManagedCode.Storage.VirtualFileSystem.Extensions;
12+
using Microsoft.Extensions.DependencyInjection;
13+
using Shouldly;
14+
using Testcontainers.Azurite;
15+
using Xunit;
16+
17+
namespace ManagedCode.Storage.Tests.VirtualFileSystem;
18+
19+
public sealed class VfsImmutableMetadataTests : IAsyncLifetime
20+
{
21+
private readonly AzuriteContainer _container = new AzuriteBuilder(ContainerImages.Azurite)
22+
.WithCommand("--skipApiVersionCheck").Build();
23+
24+
public Task InitializeAsync() => _container.StartAsync();
25+
public Task DisposeAsync() => _container.DisposeAsync().AsTask();
26+
27+
[Theory]
28+
[InlineData(false, "material.txt")]
29+
[InlineData(true, "material.txt")]
30+
[InlineData(false, "матеріал.txt")]
31+
[InlineData(true, "матеріал.txt")]
32+
public async Task ImmutableWrite_PreservesMetadataThroughFreshAndPreviouslyEmptyHandles(bool cache, string name)
33+
{
34+
await using var provider = CreateProvider(cache);
35+
await using var scope = provider.CreateAsyncScope();
36+
var vfs = scope.ServiceProvider.GetRequiredService<IVirtualFileSystem>();
37+
var path = new VfsPath("/learning/material/content");
38+
var previous = await vfs.GetFileAsync(path);
39+
(await previous.GetMetadataAsync()).ShouldBeEmpty();
40+
(await vfs.FileExistsAsync(path)).ShouldBeFalse();
41+
var bytes = Encoding.UTF8.GetBytes("exact immutable author bytes");
42+
var metadata = new Dictionary<string, string>
43+
{
44+
["fileName"] = name,
45+
["ownerUserId"] = "exact-owner",
46+
["contentType"] = "text/plain"
47+
};
48+
var options = new StorageWriteOptions { ContentType = "text/plain", Metadata = metadata };
49+
var written = await vfs.WriteBytesIfAbsentOrSameAsync(path, bytes, options);
50+
written.ReusedExisting.ShouldBeFalse();
51+
AssertMetadata(await previous.GetMetadataAsync(), metadata);
52+
(await vfs.FileExistsAsync(path)).ShouldBeTrue();
53+
var fresh = await vfs.GetFileAsync(path);
54+
await fresh.RefreshAsync();
55+
fresh.Size.ShouldBe(bytes.LongLength);
56+
AssertMetadata(await fresh.GetMetadataAsync(), metadata);
57+
await using var stream = await fresh.OpenReadAsync();
58+
using var content = new MemoryStream();
59+
await stream.CopyToAsync(content);
60+
content.ToArray().ShouldBe(bytes);
61+
var replay = await vfs.WriteBytesIfAbsentOrSameAsync(path, bytes, options);
62+
replay.ReusedExisting.ShouldBeTrue();
63+
replay.Info.ETag.ShouldBe(written.Info.ETag);
64+
AssertMetadata(await fresh.GetMetadataAsync(), metadata);
65+
}
66+
67+
[Fact]
68+
public async Task MetadataRead_PreservesLegacyCustomFieldsAndFiltersOnlyReservedVfsFields()
69+
{
70+
await using var provider = CreateProvider(false);
71+
await using var scope = provider.CreateAsyncScope();
72+
var storage = scope.ServiceProvider.GetRequiredService<IStorage>();
73+
var vfs = scope.ServiceProvider.GetRequiredService<IVirtualFileSystem>();
74+
var metadata = new Dictionary<string, string>
75+
{
76+
["ownerUserId"] = "exact-owner",
77+
["x-vfs-reviewer"] = "legacy-custom",
78+
["x-vfs-review-vfs-created"] = "user-field",
79+
["x-vfs-vfs-version"] = "1",
80+
["x-vfs-vfs-created"] = DateTimeOffset.UtcNow.ToString("O"),
81+
["x-vfs-vfs-modified"] = DateTimeOffset.UtcNow.ToString("O"),
82+
["x-vfs-vfs-attributes"] = "0",
83+
["x-vfs-vfs-internal"] = "private-vfs-state"
84+
};
85+
await storage.RequireObjectStorage().CreatePrivateContainerAsync();
86+
using var bytes = new MemoryStream(Encoding.UTF8.GetBytes("legacy bytes"));
87+
await storage.RequireObjectStorage().WriteObjectAsync("legacy/content", bytes,
88+
new StorageWriteOptions { Metadata = metadata });
89+
var file = await vfs.GetFileAsync("/legacy/content");
90+
AssertMetadata(await file.GetMetadataAsync(), new Dictionary<string, string>
91+
{
92+
["ownerUserId"] = "exact-owner",
93+
["reviewer"] = "legacy-custom",
94+
["review-vfs-created"] = "user-field"
95+
});
96+
}
97+
98+
private ServiceProvider CreateProvider(bool cache)
99+
{
100+
var services = new ServiceCollection();
101+
services.AddLogging();
102+
var containerName = $"vfs-metadata-{Guid.NewGuid():N}";
103+
services.AddAzureStorageAsDefault(options =>
104+
{
105+
options.ConnectionString = _container.GetConnectionString();
106+
options.Container = containerName;
107+
options.CreateContainerIfNotExists = true;
108+
});
109+
services.AddVirtualFileSystem(options =>
110+
{
111+
options.DefaultContainer = containerName;
112+
options.EnableCache = cache;
113+
});
114+
return services.BuildServiceProvider(new ServiceProviderOptions { ValidateScopes = true });
115+
}
116+
117+
private static void AssertMetadata(IReadOnlyDictionary<string, string> actual, Dictionary<string, string> expected)
118+
{
119+
actual.Count.ShouldBe(expected.Count);
120+
foreach (var item in expected)
121+
{
122+
actual[item.Key].ShouldBe(item.Value);
123+
}
124+
}
125+
}

‎docs/Architecture.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,13 @@ flowchart LR
104104

105105
### 2.2 Interfaces / contracts map
106106

107+
VFS immutable writes carry the caller's portable object metadata without rewriting
108+
its keys. A fresh `IVirtualFile.GetMetadataAsync` must return those fields as well
109+
as legacy VFS-prefixed custom fields. VFS version, timestamps, attributes and
110+
internal custom-state entries remain reserved and are excluded from this public
111+
custom-metadata view. The same contract applies with caching enabled or disabled;
112+
an immutable write invalidates previous absence and metadata cache entries.
113+
107114
```mermaid
108115
flowchart LR
109116
Providers["Provider Packages"]

‎docs/Features/virtual-file-system.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,17 @@ flowchart TD
3030

3131
- Directory listing is implemented by prefix-listing blob metadata via `IStorage.GetBlobMetadataListAsync(...)`.
3232

33+
### Immutable content and metadata
34+
35+
`IVirtualFileSystem.WriteBytesIfAbsentOrSameAsync` stores the provided portable
36+
object metadata alongside the exact immutable bytes. `IVirtualFile.GetMetadataAsync`
37+
returns those user fields from both fresh handles and handles which previously
38+
cached a missing file. This includes ASCII and Unicode values. Legacy user fields
39+
stored under the metadata manager's VFS prefix are exposed with that prefix
40+
removed. Only the overlay's reserved version, timestamp, attribute and internal
41+
custom-state fields are filtered; a user field merely containing those names is
42+
preserved. A matching retry retains the existing object and ETag.
43+
3344
## Components
3445

3546
Key files:

0 commit comments

Comments
 (0)