Use EncryptedFilesSyncJob #400

abuabraham-ttd · 2025-03-25T21:30:39Z

Update to use EncryptedFilesSyncJob and not PrivateSiteRefresh for syncing encrypted files.
Handle metadata missing first level hash

UID2-5060

vishalegbert-ttd · 2025-03-26T14:18:33Z

src/main/java/com/uid2/admin/store/writer/EncryptedSaltStoreWriter.java

@@ -27,6 +28,7 @@ public class EncryptedSaltStoreWriter extends SaltStoreWriter implements StoreWr
    private StoreScope scope;
    private RotatingCloudEncryptionKeyProvider cloudEncryptionKeyProvider;
    private Integer siteId;
+    private JsonObject metadata;


Maybe plaintextMetadata instead

vishalegbert-ttd · 2025-03-26T14:22:21Z

src/main/java/com/uid2/admin/store/writer/EncryptedSaltStoreWriter.java

@@ -105,8 +107,19 @@ protected List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvi
        return this.previousSeenSnapshots;


These two lines caused me a lot of confusion

In upload(Object data, JsonObject extraMeta), we already get a collection of saltSnapshots, presumably one old and one new. There is no point uploading twice

From the collection received in upload(), find the old snapshots, set it as a property, something like this.oldSnapshots

Call super.upload on the newest snapshot only

In getSnapshots, it should return the concat of this.oldSnapshots and data (which is the newest snapshot we just uploaded)

Unfortunately, no, in current implementation.

The execution flow is;

For snapshot in snapshots:
call super.upload()

So second snapshot writes metadata without including earlier snapshot. Hence we store previousSeenSnapshots.

@vishalegbert-ttd "Call super.upload on the newest snapshot only"

Will mean; metadata.json will only have newest entry

@Override protected List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvider.SaltSnapshot data){ if (this.allSnapshots == null) { throw new IllegalStateException("Snapshots are not initialized. Please call upload() first."); } return this.allSnapshots; } @Override protected JsonObject getMetadata() throws Exception { if (this.allSnapshots == null) { throw new IllegalStateException("Extra metadata is null. Please call upload() first."); } return this.extraMetadata; } @Override public void upload(Object data, JsonObject extraMeta) throws Exception { List<RotatingSaltProvider.SaltSnapshot> snapshots = new ArrayList<>((Collection<RotatingSaltProvider.SaltSnapshot>) data); int size = snapshots.size(); // Process first n-1 snapshots for (int i = 0; i < size - 1; i++) { RotatingSaltProvider.SaltSnapshot snapshot = snapshots.get(i); String location = getSaltSnapshotLocation(snapshot); super.uploadSaltsSnapshot(snapshot, location); } // Upload the last snapshot this.allSnapshots = snapshots; this.extraMetadata = extraMeta; super.upload(snapshots.get(size - 1)); }```

Does this make sense?

Yes, as discussed on call yesterday. I updated to what I had in mind, that includes modification to SaltStoreWriter . We can have a chat today and use either of this.

abuabraham-ttd · 2025-03-27T16:54:19Z

src/main/java/com/uid2/admin/job/EncryptionJob/SaltEncryptionJob.java

@@ -14,14 +14,14 @@

 public class SaltEncryptionJob extends Job {
    private final Collection<OperatorKey> globalOperators;
-    private final Collection<RotatingSaltProvider.SaltSnapshot> saltEntries;


Changes to accept saltProvider and not just saltEntries.

We can then pass unencrypted saltEntries along with metadata to encryption process.

abuabraham-ttd · 2025-03-27T16:56:17Z

src/main/java/com/uid2/admin/job/EncryptionJob/SaltEncryptionJob.java

        List<Integer> desiredPublicState = PublicSiteUtil.getPublicSaltSites(globalOperators);
-        multiScopeStoreWriter.uploadPublicWithEncryption(desiredPublicState, saltEntries, null);
+        multiScopeStoreWriter.uploadPublicWithEncryption(desiredPublicState, saltProvider.getSnapshots(), saltProvider.getMetadata());
    }


passing metadata in extraMeta argument to use this info while encrypting per site.

abuabraham-ttd · 2025-03-27T16:57:15Z

src/main/java/com/uid2/admin/store/writer/EncryptedSaltStoreWriter.java

-    protected void refreshProvider() {
-       // we do not need to refresh the provider on encrypted writers
-    }
-


re-organised the method to not have this noop

abuabraham-ttd · 2025-03-27T16:57:48Z

src/main/java/com/uid2/admin/store/writer/EncryptedSaltStoreWriter.java

-    }
-
-    @Override
-    protected List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvider.SaltSnapshot data){


dont need this as well, as getSnapshots is handled in upload(Object data, JsonObject extraMeta)

abuabraham-ttd · 2025-03-27T16:58:33Z

src/main/java/com/uid2/admin/store/writer/EncryptedSaltStoreWriter.java

-        }
+        @SuppressWarnings("unchecked")
+        List<RotatingSaltProvider.SaltSnapshot> snapshots = new ArrayList<>((Collection<RotatingSaltProvider.SaltSnapshot>) data);
+        this.buildAndUploadMetadata(extraMeta, this.uploadASnapshotsAndGetMetadata(snapshots));
    }


gets all snapshots, calls buildAndUploadMetadata with extraMeta and snapshotmeta.

buildAndUploadMetadata works same way for both unencrypted and encrypted

abuabraham-ttd · 2025-03-27T16:58:43Z

src/main/java/com/uid2/admin/store/writer/SaltStoreWriter.java

@@ -43,7 +43,7 @@ public SaltStoreWriter(JsonObject config, RotatingSaltProvider provider, FileMan
        this.versionGenerator = versionGenerator;
    }

-    protected List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvider.SaltSnapshot data){
+    private List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvider.SaltSnapshot data){
        if (provider.getSnapshots() == null) {


removed override

abuabraham-ttd · 2025-03-27T17:00:05Z

src/main/java/com/uid2/admin/store/writer/SaltStoreWriter.java

+    }
+
+    protected void buildAndUploadMetadata(JsonObject metadata, JsonArray snapshotsMetadata ) throws Exception{
+        final Instant now = Instant.now();


constructs metadata and upload based on init metadata.

For unencrypted, it is provider.getMetadata();
For encrypted, it is extraMeta

abuabraham-ttd · 2025-03-27T17:00:22Z

src/main/java/com/uid2/admin/store/writer/SaltStoreWriter.java

-        List<RotatingSaltProvider.SaltSnapshot> snapshots = this.getSnapshots(data);
-
+    protected JsonArray uploadASnapshotsAndGetMetadata(List<RotatingSaltProvider.SaltSnapshot> snapshots) throws Exception {
+        final JsonArray snapshotsMetadata = new JsonArray();


,split it to a function, no change in implementation , except making it return snapshotsMetadata

abuabraham-ttd · 2025-03-27T17:01:18Z

src/main/java/com/uid2/admin/store/writer/SaltStoreWriter.java

        // refresh manually
        refreshProvider();
    }

-    protected void refreshProvider() throws Exception {
+    private void refreshProvider() throws Exception {
        provider.loadContent();


removed the noop override. It can be private

vishalegbert-ttd · 2025-03-27T22:55:58Z

src/main/java/com/uid2/admin/store/writer/SaltStoreWriter.java

+    public void upload(RotatingSaltProvider.SaltSnapshot data) throws Exception {
+        JsonObject metadata = this.getMetadata();
+        List<RotatingSaltProvider.SaltSnapshot> snapshots = this.getSnapshots(data);
+        this.buildAndUploadMetadata(metadata, this.uploadSnapshotsAndGetMetadata(snapshots));


The original SaltStoreWrite code only uploaded the latest snapshot, now it will re-upload all of them..

`
public void upload(RotatingSaltProvider.SaltSnapshot data) throws Exception {
final Instant now = Instant.now();
final long generated = now.getEpochSecond();

JsonObject metadata = null; try { metadata = provider.getMetadata(); } catch (CloudStorageException e) { if (e.getMessage().contains("The specified key does not exist")) { metadata = new JsonObject(); } else { throw e; } } // bump up metadata version metadata.put("version", versionGenerator.getVersion()); metadata.put("generated", generated); final JsonArray snapshotsMetadata = new JsonArray(); metadata.put("salts", snapshotsMetadata); List<RotatingSaltProvider.SaltSnapshot> snapshots = this.getSnapshots(data); for (RotatingSaltProvider.SaltSnapshot snapshot : snapshots) { final String location = getSaltSnapshotLocation(snapshot); final JsonObject snapshotMetadata = new JsonObject(); snapshotMetadata.put("effective", snapshot.getEffective().toEpochMilli()); snapshotMetadata.put("expires", snapshot.getExpires().toEpochMilli()); snapshotMetadata.put("location", location); snapshotMetadata.put("size", snapshot.getAllRotatingSalts().length); snapshotsMetadata.add(snapshotMetadata); uploadSaltsSnapshot(snapshot, location); } fileManager.uploadMetadata(metadata, "salts", new CloudPath(provider.getMetadataPath())); // refresh manually refreshProvider(); }` how? Isn't the behavior the same? loop through snapshots and upload. Or did I miss anything?

snapshots = this.getSnapshots(data)

controls what snapshots are returned, and it only returns newestEffective. It isn't changed (if I understand it correctly)

Ah no sorry you're right

Use EncryptedFilesSyncJob

3e2d8a1

abuabraham-ttd requested review from aulme and cody-constine-ttd March 25, 2025 21:30

abuabraham-ttd added 5 commits March 25, 2025 14:37

Use blocking handler

dee6448

Add metadata to encrypted fodlers

4e7d595

Fix spacing

433bce0

Add/update tests

17e9d56

Add/update tests

4389717

vishalegbert-ttd reviewed Mar 26, 2025

View reviewed changes

Rename metadata

651cb78

abuabraham-ttd force-pushed the abu-add-encryption-urls branch from 340cf0d to 651cb78 Compare March 26, 2025 20:08

Split upload to smaller functions and reuse and override

1019944

abuabraham-ttd commented Mar 27, 2025

View reviewed changes

abuabraham-ttd added 2 commits March 27, 2025 10:28

Split upload to smaller functions and reuse and override

a60c81b

Fix typo

11da9d2

vishalegbert-ttd reviewed Mar 27, 2025

View reviewed changes

Release Workflow and others added 2 commits March 27, 2025 23:37

[CI Pipeline] Released Snapshot version: 5.18.8-alpha-125-SNAPSHOT

a14e301

Merge remote-tracking branch 'origin' into abu-add-encryption-urls

cec55dc

vishalegbert-ttd approved these changes Mar 28, 2025

View reviewed changes

Remove check to see if file exist

e5e0095

abuabraham-ttd force-pushed the abu-add-encryption-urls branch from 78700c1 to e5e0095 Compare March 31, 2025 19:10

Release Workflow and others added 2 commits March 31, 2025 19:12

[CI Pipeline] Released Snapshot version: 5.18.9-alpha-128-SNAPSHOT

d5cb5c2

merge

b7a7bbc

abuabraham-ttd merged commit 098c800 into main Mar 31, 2025
2 of 3 checks passed

abuabraham-ttd deleted the abu-add-encryption-urls branch March 31, 2025 23:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use EncryptedFilesSyncJob #400

Use EncryptedFilesSyncJob #400

abuabraham-ttd commented Mar 25, 2025 •

edited

Loading

vishalegbert-ttd Mar 26, 2025

vishalegbert-ttd Mar 26, 2025 •

edited

Loading

abuabraham-ttd Mar 26, 2025

abuabraham-ttd Mar 26, 2025 •

edited

Loading

vishalegbert-ttd Mar 26, 2025

vishalegbert-ttd Mar 26, 2025

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

vishalegbert-ttd Mar 27, 2025

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025

vishalegbert-ttd Mar 27, 2025

		@@ -105,8 +107,19 @@ protected List<RotatingSaltProvider.SaltSnapshot> getSnapshots(RotatingSaltProvi
		return this.previousSeenSnapshots;

Use EncryptedFilesSyncJob #400

Use EncryptedFilesSyncJob #400

Conversation

abuabraham-ttd commented Mar 25, 2025 • edited Loading

Choose a reason for hiding this comment

vishalegbert-ttd Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuabraham-ttd Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuabraham-ttd Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

abuabraham-ttd Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuabraham-ttd Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

abuabraham-ttd Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuabraham-ttd Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abuabraham-ttd commented Mar 25, 2025 •

edited

Loading

vishalegbert-ttd Mar 26, 2025 •

edited

Loading

abuabraham-ttd Mar 26, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading

abuabraham-ttd Mar 27, 2025 •

edited

Loading