Skip to content

Add loadAllOfSorted to TransactionManager#3136

Merged
CydeWeys merged 1 commit into
google:masterfrom
CydeWeys:load-all-of-sorted
Jul 7, 2026
Merged

Add loadAllOfSorted to TransactionManager#3136
CydeWeys merged 1 commit into
google:masterfrom
CydeWeys:load-all-of-sorted

Conversation

@CydeWeys

@CydeWeys CydeWeys commented Jul 7, 2026

Copy link
Copy Markdown
Member

During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.

To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.

Specifically, this commit:

  • Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
    JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
  • Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
    names in JpaTransactionManagerImpl before appending dynamic ORDER BY
    clauses, preventing JPQL/SQL injection since bind parameters cannot be
    used for schema identifiers.
  • Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
    sort registrars alphabetically by registrarName and registrarId.
  • Updates golden screenshot
    ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
    reflect the newly enforced deterministic alphabetical registrar ordering.
  • Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
    verifying exact sorted entity retrieval across multiple fields and regex
    rejection of invalid field names.

TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856


This change is Reviewable


private static final FluentLogger logger = FluentLogger.forEnclosingClass();
private static final Retrier retrier = new Retrier(new SystemSleeper(), 6);
private static final Pattern VALID_SORT_FIELD_PATTERN = Pattern.compile("^[a-zA-Z0-9_.]+$");

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment explaining why this exists / what it does.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added Javadoc explaining why regex validation is required to prevent JPQL/SQL injection in dynamic ORDER BY clauses.

* <p>This creates an inner wrapping transaction for convenience, so you don't need to wrap it in
* a transaction at the call site.
*/
public static <T> ImmutableList<T> loadAllOfSorted(Class<T> clazz, String... sortFields) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary. You only have a few callsites in tests; it's fine to wrap them there with tm().

Also I don't think you even use this helper method anyway?!

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed this unused helper method from DatabaseHelper.

@CydeWeys CydeWeys force-pushed the load-all-of-sorted branch 3 times, most recently from 51702fb to 6ed2ea7 Compare July 7, 2026 02:26
@CydeWeys CydeWeys requested a review from weiminyu July 7, 2026 02:26
@CydeWeys CydeWeys force-pushed the load-all-of-sorted branch from 6ed2ea7 to 3fad7b5 Compare July 7, 2026 02:42
@CydeWeys CydeWeys enabled auto-merge July 7, 2026 02:43
CydeWeys added a commit to CydeWeys/nomulus that referenced this pull request Jul 7, 2026
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.

Specifically, it:
- Adds Flyway migration
  V224__add_registrar_expiry_access_period_enabled.sql, which adds the
  expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
  the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
  schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
  consult db/README.md, follow the mandatory Two-PR deployment split when
  altering database schemas, and run both :db:test and
  :core:sqlIntegrationTest whenever database schema or ORM mappings are
  touched.
- Updates golden screenshot
  ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
  the new default database ordering after adding the column. (Note: A
  separate pull request, PR google#3136 / http://go/r3pr/3136, permanently fixes
  this underlying sort non-determinism across our persistence layer by
  enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).

TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
CydeWeys added a commit to CydeWeys/nomulus that referenced this pull request Jul 7, 2026
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.

Specifically, it:
- Adds Flyway migration
  V224__add_registrar_expiry_access_period_enabled.sql, which adds the
  expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
  the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
  schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
  consult db/README.md, follow the mandatory Two-PR deployment split when
  altering database schemas, and run both :db:test and
  :core:sqlIntegrationTest whenever database schema or ORM mappings are
  touched.
- Updates golden screenshot
  ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
  the new default database ordering after adding the column. (Note: A
  separate pull request, PR google#3136 / http://go/r3pr/3136, permanently fixes
  this underlying sort non-determinism across our persistence layer by
  enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).

TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
pull Bot pushed a commit to admariner/nomulus that referenced this pull request Jul 7, 2026
This commit implements the first stage of the Two-PR database schema
deployment for registrar opt-in to the Expiry Access Period (XAP), along
with updating GEMINI.md to mandate strict adherence to db/README.md for all
future database schema modifications.

Specifically, it:
- Adds Flyway migration
  V224__add_registrar_expiry_access_period_enabled.sql, which adds the
  expiry_access_period_enabled boolean column (DEFAULT false NOT NULL) to
  the Registrar table.
- Updates flyway.txt, nomulus.golden.sql, and ER diagrams to reflect the
  schema changes.
- Updates GEMINI.md with explicit instructions for AI agents to always
  consult db/README.md, follow the mandatory Two-PR deployment split when
  altering database schemas, and run both :db:test and
  :core:sqlIntegrationTest whenever database schema or ORM mappings are
  touched.
- Updates golden screenshot
  ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to reflect
  the new default database ordering after adding the column. (Note: A
  separate pull request, PR google#3136 / http://go/r3pr/3136, permanently fixes
  this underlying sort non-determinism across our persistence layer by
  enforcing deterministic alphabetical ordering in Registrar.loadAllSorted).

TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/437398822
@weiminyu

weiminyu commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

core/src/main/java/google/registry/persistence/transaction/JpaTransactionManagerImpl.java line 556 at r2 (raw file):

  @Override
  public <T> Stream<T> loadAllOfSortedStream(Class<T> clazz, String... sortFields) {

We may want to have an allow/forbidden list for clazz here, just in case someone uses it on domain or host.

Code quote:

Class<T> clazz

@weiminyu weiminyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weiminyu reviewed 9 files and all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on CydeWeys).

@CydeWeys CydeWeys left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CydeWeys made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on weiminyu).


core/src/main/java/google/registry/persistence/transaction/JpaTransactionManagerImpl.java line 556 at r2 (raw file):

Previously, weiminyu (Weimin Yu) wrote…

We may want to have an allow/forbidden list for clazz here, just in case someone uses it on domain or host.

loadAllOf() already exists though, and this particular footgun kind of falls into the umbrella category of "You can't really prevent people from writing bad code" (other than the normal code review process that would hopefully catch it). Someone could just as easily still call any of a number of other ways to load all entities from the database that would bypass this restriction on this one helper method, or they could even do something much more dangerous like deleting all entities of a given type.

@CydeWeys CydeWeys left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL

@CydeWeys made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on weiminyu).

@weiminyu weiminyu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weiminyu resolved 1 discussion.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on CydeWeys).

During our implementation of Expiry Access Period (XAP), adding a field to the
Registrar entity altered database heap scan order when loadAllOf() was
called without an explicit ORDER BY clause. This caused non-deterministic
row shifting in the Angular console registrar table (/console/registrars),
breaking golden image comparisons in ConsoleScreenshotTest.

To permanently fix this root cause and guarantee deterministic UI rendering
and test stability across schema evolutions, this commit introduces sorted
database loading across our persistence layer.

Specifically, this commit:
- Adds loadAllOfSorted and loadAllOfSortedStream to TransactionManager,
  JpaTransactionManagerImpl, and DelegatingReplicaJpaTransactionManager.
- Enforces strict allowlist regex validation (^[a-zA-Z0-9_.]+$) on property
  names in JpaTransactionManagerImpl before appending dynamic ORDER BY
  clauses, preventing JPQL/SQL injection since bind parameters cannot be
  used for schema identifiers.
- Adds Registrar.loadAllSorted and updates RegistrarsAction to explicitly
  sort registrars alphabetically by registrarName and registrarId.
- Updates golden screenshot
  ConsoleScreenshotTest_globalRole_registrars_registrarsPage.png to
  reflect the newly enforced deterministic alphabetical registrar ordering.
- Adds comprehensive unit test coverage in JpaTransactionManagerImplTest
  verifying exact sorted entity retrieval across multiple fields and regex
  rejection of invalid field names.

TAG=agy
CONV=f2488c74-8b4a-43f1-9c22-d1dddbdbb4e0
BUG=http://b/531889856
@CydeWeys CydeWeys force-pushed the load-all-of-sorted branch from 3fad7b5 to 948fe7d Compare July 7, 2026 18:53
@CydeWeys

CydeWeys commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

PTAL, had to sync to HEAD.

@CydeWeys CydeWeys added this pull request to the merge queue Jul 7, 2026
Merged via the queue into google:master with commit 2ce91e3 Jul 7, 2026
15 of 16 checks passed
@CydeWeys CydeWeys deleted the load-all-of-sorted branch July 7, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants