[Bug] Canal source client is incompatible with modern protobuf-java (NoSuchMethodError on connect)

[david-streamlio]

Description

The Canal source connector's client library (com.alibaba.otter:canal.client:1.1.7 / canal.protocol:1.1.7) is incompatible with the protobuf-java version the build resolves. Its generated CanalPacket code calls com.google.protobuf.GeneratedMessageV3.makeExtensionsImmutable(), a method removed in modern protobuf-java (the 3.21+/4.x line). So CanalConnector.connect() throws NoSuchMethodError and the source can never complete the canal-server handshake.

Found while adding the first integration test for the module (#48 / #98) — the module had zero tests, so this was never exercised.

Evidence

On the canal module's test classpath, protobuf-java resolves to 4.31.1 (pulled by pulsar-client/pulsar-functions-instance), and GeneratedMessageV3 in that version has no makeExtensionsImmutable. The integration test reproduces the NoSuchMethodError on connect() without a version pin.

Nuance worth a maintainer's eye — is production affected?

• The module's runtimeClasspath (what the NAR bundles) resolves no external protobuf-java at all, and canal.protocol-1.1.7.jar does not bundle protobuf classes. So the NAR appears to rely on whatever protobuf the Pulsar runtime provides — which is modern — meaning the connector would very likely hit the same NoSuchMethodError in production, not just in the test. This should be confirmed against a real deployment.

The fix in #98, and why it needs review

#98 makes the integration test pass by forcing protobuf-java to 3.6.1 (canal's declared version) via configurations.all { resolutionStrategy { force(...) } }. That works, but:

1. 3.6.1 is from 2018 and carries known CVEs (e.g. CVE-2021-22569, CVE-2022-3171). Pinning the whole module — including the shipped NAR — to it is heavy.
2. The demonstrated breakage is on the test classpath; whether the NAR needs the pin depends on the production question above. If production is unaffected, the force should be scoped to test configurations only.
3. canal.client 1.1.8 exists (latest). If it is built against a newer protobuf, upgrading is a cleaner fix than downgrading protobuf. Worth checking before committing to the 3.6.1 pin.

Suggested resolution

Decide among: (a) upgrade to canal.client/canal.protocol 1.1.8 if protobuf-4-compatible; (b) shade/relocate protobuf for the canal NAR; or (c) keep the 3.6.1 pin but scope it correctly (test-only if production is unaffected) and accept the CVE tradeoff. #98 currently takes (c) module-wide; flagging for a decision.

[david-streamlio]

Root-caused this on a Linux x86-64 host while verifying #98. The findings narrow the issue considerably:

1. The protobuf 4.x on the canal test classpath came from mysql-connector-j, not the platform. gradle :canal:dependencyInsight shows protobuf-java:4.31.1 selected "by conflict resolution: between versions 4.31.1, 3.6.1 and 3.25.5" — the 4.31.1 is a transitive of the test-only com.mysql:mysql-connector-j:9.4.0 (X DevAPI support, unused by plain JDBC). The platform itself pins 3.25.5.

2. makeExtensionsImmutable() exists in protobuf-java 3.25.5 — it was removed in the 4.x line. Verified empirically, not just by reading release notes: a standalone canal 1.1.x client run against exactly protobuf-java-3.25.5.jar completes the canal-server handshake and delivers CDC entries (GOT MESSAGE id=1 ... table=products). So the incompatibility is canal ↔ protobuf 4.x, not "modern protobuf" generally.

3. Production is not affected today. The NAR bundles no protobuf (the com.google.protobuf group is excluded from NARs by convention; the Pulsar runtime's java-instance provides it), and the Pulsar runtime ships the 3.25.x line, which retains the method. The connector works against the runtime-provided protobuf.

4. Upgrading to canal 1.1.8 does not fix the 4.x incompatibility. Its regenerated CanalPacket$Packet still calls makeExtensionsImmutable() (verified via javap -c on canal.protocol-1.1.8.jar — note the call sites are in the inner Packet classes, easy to miss when inspecting the outer class). An earlier attempt to run the integration test with client+server 1.1.8 and no pin failed with the same NoSuchMethodError.

Resolution taken in #98: drop the module-wide 3.6.1 force entirely and instead exclude protobuf-java from the test-only mysql-connector-j dependency. The test classpath then resolves the platform's 3.25.5 — the same protobuf line production uses — and the integration test passes with no pin.

What remains of this issue: a forward-looking constraint, not a current bug — if/when the Pulsar runtime moves to protobuf 4.x, the canal client (any released version, 1.1.8 included) breaks on connect(). Options then are shading protobuf into the canal NAR or an upstream canal fix. Suggest either re-titling this issue to track that constraint or closing it in favor of a note in the module.