feat(spanner): Support dynamic TLS certificate and key rotation for Spanner Omni

[sagnghos]

Overview

In Spanner Omni deployments (e.g., on-premises, AWS EKS, GKE), client mTLS certificates/keys and server root CA certificates can be rotated on disk periodically by automated systems (such as Kubernetes cert-manager or HashiCorp Vault).

Previously, Java TLS components cached credentials in memory at initialization. When certificates were rotated on disk, long-running applications and proxies (such as PGAdapter) would eventually fail handshakes until the process or connection pool was restarted.

This PR adds zero-downtime dynamic TLS certificate and private key rotation for Spanner Omni, allowing clients to detect and pick up rotated credentials on disk with zero application restarts, zero downtime, and zero background thread leaks.

---

How It Works

1. On-Demand & Zero Thread Leaks: Instead of background polling threads, file stat checks (lastModified and length) are piggybacked synchronously onto connection attempts (< 5µs check), throttled to a default 5-second interval.
2. Race-Condition Free: Uses ReentrantLock with double-checked locking so concurrent handshakes during rotation wait for the reload rather than failing with stale or mismatched credentials.
3. Cryptographic Validation & Fault Tolerance: Rotated private keys are verified against certificate public keys before adoption. If a file is incomplete or corrupt during an in-flight write, the manager safely retains the previous working credentials.
4. In-Flight Handshake Safety: Retains a short history of versioned key aliases so connections initialized right at the moment of rotation complete uninterrupted.

---

Key Changes

• DynamicKeyManager (com.google.cloud.spanner.omni):
  • Custom X509ExtendedKeyManager that dynamically reloads client certificate chains and RSA/EC PKCS#8 private keys (PEM, Base64, and binary DER formats).
• DynamicTrustManager (com.google.cloud.spanner.omni):
  • Custom X509ExtendedTrustManager that dynamically reloads server root CA certificate bundles into an in-memory keystore.
• SpannerOptions:
  • Added setCaCertificate(String) / getCaCertificate().
  • Updated useClientCert(String, String) to configure dynamic key management.
  • Added unwrappable OmniSslChannelConfigurator to avoid nested lambda wrapping across .toBuilder().build() calls.
  • Preserved instanceType, usePlainText, and host across .toBuilder().build().
• SpannerPool:
  • Added support for caCertificate URI parameter and surfaced validation for partial certificate/key configurations.

---

Issue Link

Fixes b/562755231

[gemini-code-assist]

Code Review

This pull request introduces DynamicKeyManager and DynamicTrustManager to support dynamic loading and automatic reloading of client certificates, private keys, and server root CA certificates for Spanner Omni. It also updates SpannerOptions, ConnectionOptions, and related classes to support these new configurations. The review feedback highlights a potential performance issue where blocking disk I/O and CPU-heavy cryptographic operations are performed on the calling thread (typically a Netty EventLoop thread) during TLS handshakes. Additionally, the feedback points out a potential mismatch bug in DynamicKeyManager's fallback logic when an alias is evicted, suggesting safer alias resolution and returning empty arrays instead of nulls to minimize null pointer risks.

[sagnghos]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces dynamic certificate and key reloading for Spanner Omni instances by adding DynamicKeyManager and DynamicTrustManager to automatically reload client certificates, private keys, and server root CA certificates from disk when modified. It also updates configuration classes, connection properties, and connection pools to support these new options, alongside comprehensive unit tests. The review feedback highlights a critical issue in both new managers where using ThreadPoolExecutor.DiscardPolicy silently discards rejected tasks, permanently blocking future reloads because the isReloading flag is never reset. To resolve this, it is recommended to use ThreadPoolExecutor.AbortPolicy so that RejectedExecutionException is correctly thrown and handled.

[styee]

Code Review & Assessment for b/562755231

Thank you for working on this! This PR directly targets the issue described in b/562755231 ("PGAdapter: Rotated certificates are not picked up until restart"). Dynamically reloading mTLS client key material and server root CA certificates in Spanner Omni without requiring application or process restarts is a significant improvement.

The overall approach of using custom X509ExtendedKeyManager and X509ExtendedTrustManager implementations integrated into Netty's SslContext is the right design.

There are two critical issues in the current implementation, along with a few minor observations, that should be addressed before merging:

---

Critical Issues

1. Asynchronous reload causes the first connection after rotation to fail (Race Condition)

In both DynamicKeyManager.java and DynamicTrustManager.java:

void checkAndReload() {
  ...
  if (checkIntervalNs == 0) {
    try {
      doReloadCheck(now);
    } finally {
      isReloading.set(false);
    }
  } else {
    try {
      ASYNC_RELOAD_EXECUTOR.execute(() -> {
        try {
          doReloadCheck(System.nanoTime());
        } finally {
          isReloading.set(false);
        }
      });
    } catch (RejectedExecutionException e) {
      isReloading.set(false);
    }
  }
}

In production, checkIntervalMs defaults to 5000L (checkIntervalNs > 0). When a handshake occurs:

1. checkAndReload() is called from chooseEngineClientAlias() or checkServerTrusted().
2. It sees now - lastCheckedNs >= checkIntervalNs and queues doReloadCheck() onto ASYNC_RELOAD_EXECUTOR.
3. checkAndReload() returns immediately, before the background thread has read or parsed the updated files on disk.
4. The calling thread immediately proceeds with the handshake:
   • In DynamicKeyManager: it returns currentMaterial.alias (the old certificate/key). If the previous certificate is already expired, the client presents expired credentials and the server rejects the handshake.
   • In DynamicTrustManager: Netty calls checkServerTrusted(), which validates the server's certificate against currentMaterial.delegate (the old CA). If the server is presenting a certificate signed by the newly rotated CA, this throws a CertificateException and immediately terminates the TLS handshake.

Note: Unit tests did not catch this because all rotation tests instantiate the managers with checkIntervalMs = 0L, which takes the synchronous execution path.

Suggestions:

• Calling File.lastModified() and File.length() takes microseconds (< 5 µs). If timestamps and lengths have not changed, it returns immediately without any disk read or cryptographic parsing.
• If a modification is detected, reload synchronously on that handshake so that the incoming connection uses the updated material.
• Alternatively: Run a proactive background timer (ScheduledExecutorService) that checks and reloads the files periodically every 5–30s independently of handshakes. That way, currentMaterial is already fresh before any handshake arrives, avoiding both event-loop I/O and handshake failures.

---

2. JSSE Specification Violation in DynamicKeyManager.getCertificateChain(String alias)

In DynamicKeyManager.java:

@Override
public X509Certificate[] getCertificateChain(String alias) {
  KeyMaterial mat = (alias != null) ? materials.get(alias) : this.currentMaterial;
  return mat != null ? mat.certificateChain.clone() : new X509Certificate[0];
}

The standard specification for X509KeyManager.getCertificateChain(String alias) states:

"Returns: the certificate chain associated with the given alias, or null if the alias can't be found."

Returning new X509Certificate[0] instead of null causes standard JSSE and TLS engine implementations that do:

X509Certificate[] certs = keyManager.getCertificateChain(alias);
if (certs != null) {
  PublicKey pubKey = certs[0].getPublicKey();
}

to fail with an ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0.

Suggestion:
Return null when mat == null (or when mat.certificateChain is empty):

@Override
public X509Certificate[] getCertificateChain(String alias) {
  KeyMaterial mat = (alias != null) ? materials.get(alias) : this.currentMaterial;
  return (mat != null && mat.certificateChain != null && mat.certificateChain.length > 0)
      ? mat.certificateChain.clone()
      : null;
}

---

Observations & Suggestions

1. SpannerOptions.toBuilder().build() configurator chaining:
   In SpannerOptions.java, each call to toBuilder().build() wraps the existing channelConfigurator with an additional lambda layer that re-applies sslContext. Consider ensuring that repeated toBuilder().build() calls do not accumulate redundant configurator layers.

2. Documentation on CA certificate rotation:
   In SpannerOptions.Builder, DynamicTrustManager is initialized only when hasCaCert is true (builder.caCertificate is set). If a user only sets useClientCert(...) without setCaCertificate(...), the server root still relies on the static JVM trust store. Please highlight in the PGAdapter documentation that to enable dynamic CA rotation, users must explicitly supply the CA certificate path (e.g., via -r "caCertificate=/path/to/ca.crt;..." or setCaCertificate(...)).

3. Test coverage for non-zero check interval:
   Adding a test for DynamicKeyManager and DynamicTrustManager where checkIntervalMs > 0 and verifying that rotation works seamlessly would prevent regressions.