[CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand by xuzifu666 · Pull Request #5295 · apache/calcite

xuzifu666 · 2026-09-26T11:12:51Z

Jira Link

Changes Proposed

RexSimplify replaced a comparison of an expression x with a BOOLEAN literal by x or NOT(x) without checking that x was BOOLEAN, so AND(=(?0.varchar0, true), =(?0.varchar1, true)) became AND(?0.varchar0, ?0.varchar1), an AND of VARCHAR operands. Clients that build Rex trees directly produce such comparisons: Flink hit it with WHERE b = true AND c = true on a VARCHAR column c (FLINK-27402).

Two places did the rewrite. The EQUALS loop in simplifyAnd2ForUnknownAsFalse replaced =(x, TRUE) and =(TRUE, x) inside an AND, under unknown-as-FALSE only. simplifyComparison checked only that the first operand was BOOLEAN, so it rewrote the forms with the literal first in every RexUnknownAs mode: =(TRUE, x) to x, =(FALSE, x) to NOT(x), and <>(FALSE, x) to x, because Comparison.of swapped the operands.

Both places now rewrite only when both operands have type BOOLEAN (isBooleanComparison). The check compares SqlTypeName, so =(x, TRUE) with a nullable BOOLEAN x is still simplified. It cannot move into isAlwaysTrue, which already returns false for a non-BOOLEAN literal, because it guards the other operand, the one that replaces the comparison. Throwing on a VARCHAR compared with a BOOLEAN would break clients that build such calls today.

One result changes for an input that is not malformed: under unknown-as-FALSE, AND(=($0, true), ?0.bool1) with $0 of type ANY used to become AND($0, ?0.bool1) and is now left unchanged, since a value of type ANY need not be BOOLEAN. ITEM on a MAP<VARCHAR, ANY> column, such as MongoDB's _MAP, has type ANY.

RexProgramTest.testSimplifyAndEqualityTrue (new) fails without the change. It covers VARCHAR and INTEGER operands, nullable and NOT NULL, inside an AND with the literal second, the = with TRUE and FALSE and <> with FALSE forms with the literal first, and the ANY case, and checks that BOOLEAN operands are still simplified. Paranoid mode is off for the non-BOOLEAN inputs so that a regression prints the simplified expression instead of failing inside RexSimplify.verify.

🤖 Generated with Claude Code

mihaibudiu · 2026-09-26T16:11:29Z

        RexCall call = (RexCall) term;
-        if (call.getOperands().get(0).isAlwaysTrue()) {
+        if (call.getOperands().get(0).isAlwaysTrue()
+            && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {


I hope you have checked that this works for any nullability

This is bullshit.

CALCITE-5907 is invalid, because you should never have "varchar-column = bool-literal" in a Rex node. If we see non-boolean terms in a boolean expression, we should throw.

Let me share my understanding of this part—though I might be mistaken: I agree that a well-typed Rex tree should never contain =(varchar, bool), and that SqlValidator is the right place to enforce comparability. However, that invariant is not enforced at the Rex layer itself: RexBuilder.makeCall(EQUALS, ...) performs no operand-family check (EQUALS' return-type inference is unconditionally BOOLEAN), so downstream systems that construct expressions programmatically (e.g. Flink, the reporter of https://issues.apache.org/jira/browse/FLINK-27402) can and do produce such nodes.

The bug here is not the input's existence, but that the simplifier turned a valid BOOLEAN condition into a non-BOOLEAN one. A rewrite rule must never produce output less valid than its input; leaving the equality untouched is the minimal, safe behavior. Throwing from a simplification hot path would turn a silent robustness gap into planner crashes for rels that currently plan and execute fine, and conflates rewriting with validation, Calcite keeps those separate (cf. RexChecker/RexUtil.verify).

Ah, it's a malformed =, not a malformed boolean term in AND or OR. That's not OK, but it's understandable. I think it would break a lot of clients it Calcite started to reject such = calls.

So I think you should make isAlwaysTrue more defensive - push the 'type is boolean' check into that method. Thus for the string literal 'true', isAlwaysTrue will return false.

Ditto isAlwaysFalse and any analogous methods.

Thanks for the suggestion @julianhyde . isAlwaysTrue / isAlwaysFalse are already defensive: both RexLiteral and RexCall return false unless the expression has BOOLEAN type, so the string literal 'true' already returns false.The call-site check guards the other operand (the one that would replace the =` term), so it can't be pushed intoisAlwaysTrue. Without it,AND(=(x, TRUE), ...)with VARCHAR x would simplify toAND(x, ...) which is the CALCITE-5907 bug. Please correct me if I am wrong～

I hope you have checked that this works for any nullability

Good point @mihaibudiu . The check only looks at the type name, not nullability, and it is safe for both cases: =(x, TRUE) → x with nullable x (if x is NULL, =(NULL, TRUE) and bare NULL are both UNKNOWN, i.e. FALSE under UNKNOWN_AS_FALSE) and with NOT NULL x (trivially equivalent).
I had added NOT NULL variants of VARCHAR, INTEGER and BOOLEAN to testSimplifyAndEqualityTrue to cover this.

Maybe just check that the operands have the same type? Your change is making RexSimplify more complicated, because we are accommodating a client that is sending us garbage. That is tech debt right there.

Or call a 'isWellFormed' method before entering the loop. It sends a strong message that "this code doesn't run on garbage".

Good idea, done @julianhyde . The loop now starts with a well-formedness guard: unless both operands of = have the same type name (e.g.=(x, TRUE) with VARCHAR or INTEGER x is malformed), it breaks without simplifying. This is equivalent to the previous check, sinceisAlwaysTrue() can only be true for a BOOLEAN expression, but it states the precondition up front. I compare getSqlTypeName() rather than RelDataType.equals, because TRUE is NOT NULL and equals would compare nullability, blocking the valid simplification of=(nullable Bool X, TRUE).

xuzifu666 · 2026-09-30T02:40:19Z

Since this pr is approved and should had addressed comments, if no other objections I would merge it in few days. cc @julianhyde @mihaibudiu

vlsi

The same defect remains when the operands are swapped. simplifyComparison checks only that o0 is BOOLEAN (RexSimplify.java#L751), and Comparison.of then swaps the operands so that cmp.ref is the VARCHAR one. With this PR applied, in every RexUnknownAs mode:

Input	Result
`=(true, ?0.varchar0)`	`?0.varchar0`
`AND(=(true, ?0.varchar0), =(true, ?0.varchar1))`	`AND(?0.varchar0, ?0.varchar1)`, of type VARCHAR
`AND(=(false, ?0.varchar0), =(false, ?0.varchar1))`	`AND(NOT(?0.varchar0), NOT(?0.varchar1))`
`AND(<>(false, ?0.varchar0), <>(false, ?0.varchar1))`	`AND(?0.varchar0, ?0.varchar1)`

A front end that builds =(TRUE, c) without a cast, as Flink does in the JIRA, still gets the plan CALCITE-5907 reports for WHERE TRUE = c AND TRUE = d. testSimplifyAndEqualityTrue needs these four inputs too.

The JIRA summary, and with it the PR title and the squashed commit, does not name the symptom, and the bug is wider than AND. Something like RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand covers the swapped and FALSE forms as well.

vlsi · 2026-10-05T22:51:28Z

+        if (call.getOperands().get(0).getType().getSqlTypeName()
+            != call.getOperands().get(1).getType().getSqlTypeName()) {
+          break;
+        }


Julian suggested an isWellFormed check (comment). A helper of that shape, called here and from simplifyComparison before L751, would also stop the swapped-operand case from the review body, and its name would say why the term is left alone. As written, the guard exists only in this loop and carries no comment.

This also changes plans for ANY: under UNKNOWN_AS_FALSE, AND(=($0, true), ?0.bool1) with $0 of type ANY used to become AND($0, ?0.bool1) and now stays unchanged. It needs a mention in the JIRA and a test, since ITEM on a MAP<VARCHAR, ANY> column, such as MongoDB's _MAP, produces ANY.

vlsi · 2026-10-05T22:51:28Z

+    // "=(x, true)" can be simplified to "x" only if x has BOOLEAN type;
+    // there is no implicit cast from VARCHAR or INTEGER, so the equality
+    // must be retained


The stated reason is not accurate: Calcite does coerce VARCHAR and numeric types to BOOLEAN (SqlTypeCoercionRule). The equality has to stay because replacing it with x would put a non-BOOLEAN operand into the AND. Suggest: "=(x, true)" simplifies to "x" only if x is BOOLEAN; otherwise the AND would get a non-BOOLEAN operand.

vlsi · 2026-10-05T22:51:28Z

+    checkSimplifyUnchanged(
+        and(eq(vVarchar(0), trueLiteral),
+        eq(vVarchar(1), trueLiteral)));


With paranoid mode on (RexProgramBuilderBase turns it on for every test), a regression here throws ClassCastException from RexInterpreter inside RexSimplify.verify and never shows the simplified expression. Turning paranoid mode off for the non-BOOLEAN inputs, as testSimplifyItemRangeTerms does with simplify = simplify.withParanoid(false), makes checkSimplifyUnchanged print the expression it got instead.

Nit: the second argument of and( is indented at the same level as and( itself, so it reads like a second argument of checkSimplifyUnchanged; the same applies at L2487, L2494, L2497, and L2500.

xuzifu666 · 2026-10-06T10:18:18Z

The same defect remains when the operands are swapped. simplifyComparison checks only that o0 is BOOLEAN (RexSimplify.java#L751), and Comparison.of then swaps the operands so that cmp.ref is the VARCHAR one. With this PR applied, in every RexUnknownAs mode:

Input Result
=(true, ?0.varchar0) ?0.varchar0
AND(=(true, ?0.varchar0), =(true, ?0.varchar1)) AND(?0.varchar0, ?0.varchar1), of type VARCHAR
AND(=(false, ?0.varchar0), =(false, ?0.varchar1)) AND(NOT(?0.varchar0), NOT(?0.varchar1))
AND(<>(false, ?0.varchar0), <>(false, ?0.varchar1)) AND(?0.varchar0, ?0.varchar1)
A front end that builds =(TRUE, c) without a cast, as Flink does in the JIRA, still gets the plan CALCITE-5907 reports for WHERE TRUE = c AND TRUE = d. testSimplifyAndEqualityTrue needs these four inputs too.

The JIRA summary, and with it the PR title and the squashed commit, does not name the symptom, and the bug is wider than AND. Something like RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand covers the swapped and FALSE forms as well.

Good catch, thank you @vlsi !
The swapped case indeed slipped through: simplifyComparison only checked o0 and Comparison.of happily swapped the operands, so =(TRUE, c) still collapsed to c. Fixed per your suggestion — there is now a private isWellFormed(left, right) helper (same type name on both sides), called both from the AND loop and from simplifyComparison before the BOOLEAN branch, so it now guards both directions and the = false / <> false forms. testSimplifyAndEqualityTrue covers the four inputs from your table (as checkSimplifyUnchanged), plus AND(=($0, true), ?0.bool1) with $0 of type ANY; the non-BOOLEAN inputs run with withParanoid(false) so a regression reports the simplified expression rather than a ClassCastException.

Regarding the summary for the Jira issue and PR, I agree that your suggestion is reasonable. I have updated the wording to "RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand," and this change will be reflected during the subsequent merge.（I look forward to your review～）.

xuzifu666 · 2026-10-08T12:57:39Z

@vlsi Are you happy with current changes? if there no other objections I would merge it in few days.

vlsi · 2026-10-08T13:45:32Z

+  /** Returns whether a comparison between {@code left} and {@code right}
+   * is well-formed, that is, its operands have the same type.
+   *
+   * <p>Comparisons whose operands have different types (e.g. a VARCHAR
+   * column compared with the BOOLEAN literal TRUE) never result from
+   * Calcite's own SQL translation, but clients that build Rex trees
+   * programmatically sometimes create them; such comparisons are left
+   * unsimplified. */
+  private static boolean isWellFormed(RexNode left, RexNode right) {


The javadoc says that comparisons whose operands have different types "never result from Calcite's own SQL translation". Nothing in RexSimplify or its tests backs this, and no change to the converter would update it. Both call sites need only "both operands are BOOLEAN": in simplifyComparison o0 is already BOOLEAN, and in the AND loop only a BOOLEAN node can be isAlwaysTrue(). A helper named for that check, such as isBooleanComparison(left, right) with the javadoc Returns whether both operands of a comparison have type BOOLEAN., behaves the same at both sites and needs no claim about where such trees come from. isWellFormed names a general property, so a later caller could use it to skip valid simplifications such as INTEGER compared with BIGINT. The first sentence also says "the same type" where the check compares SqlTypeName.

Thanks for your suggestion: I renamed the helper to isBooleanComparison(left, right) with your javadoc (the unverified provenance claim is gone; at simplifyComparison the call collapsed to a single condition, behavior unchanged).

vlsi · 2026-10-08T13:45:32Z

+  /** Test case for
+   * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+   * Unexpected boolean expression simplification for And expression</a>. */


The link text should match the JIRA summary, which is now RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand.

vlsi · 2026-10-08T13:45:32Z

+    // Paranoid mode throws from RexSimplify.verify for these non-BOOLEAN
+    // inputs, so turn it off to let checkSimplifyUnchanged report the
+    // simplified expression instead


Paranoid mode throws only when the simplified expression evaluates differently from the original, not for these inputs as such. Suggest: // With paranoid mode on, a wrong simplification of these non-BOOLEAN inputs fails inside RexSimplify.verify with an exception that hides the result; turn it off so that checkSimplifyUnchanged prints the simplified expression.

vlsi · 2026-10-08T13:45:32Z

+    // "=$0, true" where $0 has type ANY is no longer simplified to "$0";
+    // ITEM on a MAP<VARCHAR, ANY> column (e.g. MongoDB's _MAP) produces ANY,
+    // which may be BOOLEAN at runtime


"=$0, true" is missing its parentheses. "is no longer simplified" describes the change rather than the rule, and "which may be BOOLEAN at runtime" gives the opposite reason: the term stays because a value of type ANY need not be BOOLEAN. Suggest: // "=($0, true)" stays when $0 has type ANY, since its value need not be BOOLEAN; ITEM on a MAP<VARCHAR, ANY> column, such as MongoDB's _MAP, has type ANY.

sonarqubecloud · 2026-10-08T14:52:10Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

vlsi · 2026-10-08T21:06:38Z

@xuzifu666 I squashed the branch into one commit, 2ff3fb9 and updated the PR description. The code is the same as in 939e7c1.

I guess we can merge this once CI succeeds.

Thank you.

…al by a non-BOOLEAN operand

mihaibudiu approved these changes Sep 26, 2026

View reviewed changes

mihaibudiu approved these changes Sep 28, 2026

View reviewed changes

mihaibudiu added the LGTM-will-merge-soon Overall PR looks OK. Only minor things left. label Oct 1, 2026

vlsi requested changes Oct 5, 2026

View reviewed changes

xuzifu666 requested a review from vlsi October 6, 2026 10:32

xuzifu666 changed the title ~~[CALCITE-5907] Unexpected boolean expression simplification for And expression~~ [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand Oct 6, 2026

vlsi approved these changes Oct 8, 2026

View reviewed changes

vlsi force-pushed the 5907 branch from 939e7c1 to 2ff3fb9 Compare October 8, 2026 21:02

[CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN liter…

f5e9f01

…al by a non-BOOLEAN operand

xuzifu666 force-pushed the 5907 branch from 2ff3fb9 to f5e9f01 Compare October 9, 2026 12:52

xuzifu666 merged commit 2e5c158 into apache:main Oct 9, 2026
16 checks passed

Conversation

xuzifu666 commented Sep 26, 2026 • edited by vlsi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Link

Changes Proposed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vlsi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuzifu666 commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xuzifu666 commented Oct 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Oct 8, 2026

Quality Gate passed

Uh oh!

vlsi commented Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xuzifu666 commented Sep 26, 2026 •

edited by vlsi

Loading

xuzifu666 Sep 28, 2026 •

edited

Loading

xuzifu666 Sep 29, 2026 •

edited

Loading

xuzifu666 commented Sep 30, 2026 •

edited

Loading

xuzifu666 commented Oct 6, 2026 •

edited

Loading