Skip to content

Add an action to sync the remote Valkey cache#3032

Merged
gbrodman merged 1 commit into
google:masterfrom
gbrodman:valkeyCursor
May 5, 2026
Merged

Add an action to sync the remote Valkey cache#3032
gbrodman merged 1 commit into
google:masterfrom
gbrodman:valkeyCursor

Conversation

@gbrodman

@gbrodman gbrodman commented May 1, 2026

Copy link
Copy Markdown
Collaborator

This uses two cursors (one for hosts and one for domains) to track our progress in "catching up", or syncing recent changes to the database, using the update-timestamp field. When the cache is in use and fully caught up, these
cursors should be kept relatively up-to-date to the actual time, i.e. less than one hour behind

note: while the generated schema changes due to the cursor enum additions, the golden file doesn't change because our SQL scripts server-side don't enforce the enum restriction


This change is Reviewable

@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026

@ptkach ptkach left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How frequently do you intent to run it and how many domains do you expect it to sync (based on historic data)?

@ptkach reviewed 8 files and made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on gbrodman).


core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):

            .getResultList();
    if (hosts.isEmpty()) {
      logger.atInfo().log("No domains to process");

hosts?

@gbrodman gbrodman left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we run this every 15-20 minutes (we want data to be at most one hour out of date).

The batch size (the amount that we read from Postgres) is at most 10,000. Looking back at a few recent random days, the number of domains changed (select count(*) from "Domain" where update_timestamp < X and update_timestamp > Y ranges between like 1500 and 15,000. There are significantly fewer host updates.

@gbrodman made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on ptkach).


core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):

Previously, ptkach (Pavlo Tkach) wrote…

hosts?

whoops

@ptkach ptkach left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ptkach reviewed 1 file and all commit messages, and resolved 1 discussion.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on gbrodman).

This uses two cursors (one for hosts and one for domains) to track our
progress in "catching up", or syncing recent changes to the database,
using the update-timestamp field. When the cache is in use and fully
caught up, these
cursors should be kept relatively up-to-date to the actual time, i.e.
less than one hour behind

@ptkach ptkach left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ptkach reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on gbrodman).

@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 5, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 5, 2026
@gbrodman gbrodman added this pull request to the merge queue May 5, 2026
Merged via the queue into google:master with commit 81b3a2f May 5, 2026
14 checks passed
@gbrodman gbrodman deleted the valkeyCursor branch May 5, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants