Skip to content

[FLINK-41568][docs] Add pages for the comparison, boolean, hash, JSON and VARIANT functions - #30033

Open
raminqaf wants to merge 8 commits into
apache:masterfrom
raminqaf:FLINK-41568
Open

raminqaf wants to merge 8 commits into
apache:masterfrom
raminqaf:FLINK-41568

Conversation

@raminqaf

@raminqaf raminqaf commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Step 2 of FLINK-23804, continued after #29864: every function and operator of five more categories gets a page of its own, generated from YAML. Each page was rechecked against the implementation, and every example result comes from running its SQL in batch and streaming mode. The recheck changed what the docs say, for example:

Function Old docs Master
= "Returns TRUE if value1 is equal to value2" TRUE = 'true' is FALSE: the string isn't cast
SIMILAR TO "The escape character has not been supported yet." ESCAPE works. A . matches any character but a newline, as in Java
JSON_OBJECT "keys must be non-NULL string literals" In SQL, a NOT NULL string column works too. The Table API needs literals
JSON_QUOTE Escapes ", \, / and the control characters \b, \f, \n, \r, \t An emoji comes out garbled, and CHR(1) isn't escaped
JSON_UNQUOTE A value that isn't a valid JSON string literal "is passed through unmodified" '"a"b"' returns a"b

Brief change log

  • The first commit applies two rules to the merged pages, and the new pages follow them too. docs/README.md states them.
    • An argument that takes any expression is expr, or expr1 and expr2, as Databricks, Snowflake and DuckDB name it. Other names are snake_case, such as json_string, and no argument or alias is a word the syntax highlighter colors like SQL, such as type in CAST(value AS type) or offset in LAG. AGENTS.md states this one too.
    • An example selects its inputs first and names each result column after its expression, such as qty > 10 AND price > 10 AS `qty > 10 AND price > 10` , so a row reads from input to result without going back to the query.
  • The second commit shows example results as the query returns them. The cells went through Markdown, so quotes came out curly, backslash escapes disappeared and tags rendered as HTML: on REGEXP_REPLACE, the masked value ****-**** showed as a bold dash, and on the new XML_STRING page <a><b>1</b></a> would have shown as a bold 1.
  • Add 40 pages: Comparison 9, Boolean 4, Hash 7, JSON 12, Variant 8. Every row of these categories now points at a page.
  • Comparison and Boolean are operators. A page covers an operator and its NOT form, such as IS NULL and IS NOT NULL. The operators =, <>, <, <=, > and >= share a "Comparison operators" page, and AND, OR and NOT a "Logical operators" page. The two VARIANT subscript rows share one page. The split follows Snowflake for the grouped operators and Databricks for the IS tests.
  • The Table API column shows the Java calls instead of the Scala ones, such as isEqual(...) for === and jsonQuote() for JsonQuote(). The Chinese table gets the JSON(value) row it lacked.
  • The XML mapping of VARIANT, about 220 lines with tables, moves from the category page to a page "XML and VARIANT", and the path syntax to a page "VARIANT Paths". The Variant category page keeps its intro and table, like the other categories. Links and the anchors of the old single functions page follow. Two sentences there are fixed: TRY_VARIANT_GET returns NULL for a malformed path that isn't a literal, and reading XML back with PARSE_XML needs the same force_array.
  • Chinese pages show the English sections. Where a row had a Chinese description, its first sentence becomes the summary, except on three pages where it described only one operator or mistranslated it. This drops about 1,600 Chinese characters across 49 rows. FLINK-41590 translates these pages, as FLINK-41529 does for the previous five categories.

Verifying this change

This change is documentation only.

  • Every SQL example ran in batch and streaming mode on master with a local runner, which isn't part of this PR. The results on the pages are pasted from those runs.
  • A second review per category checked each statement against the type strategies, the runtime code and the tests. A third pass removed what the reviews of [FLINK-41433][docs] Add pages for the regular expression, cast, conditional, window and utility functions #29864 asked to leave out.
  • Hugo 0.126.3 builds both languages without warnings. A link check finds no new broken internal links. Every commit builds on its own.

The recheck found bugs in the code, which this PR doesn't fix. The pages describe the current behavior, also where it's a bug that users would plausibly hit, such as JSON_STRING writing a FLOAT with the digits of a DOUBLE. They name no tickets. I'll file the bugs separately. FLINK-41591 is the first, for the nanosecond range of PARSE_XML.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf
raminqaf marked this pull request as ready for review November 20, 2026 08:00

@MartijnVisser MartijnVisser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I ran all 146 examples in batch and streaming mode and they match. A few statements don't hold, see the comments.

type: VARIANT
required: true
description: An array or an object.
- name: n

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YAML reads an unquoted n as a boolean, so the Arguments table renders false here. LAG and LEAD show the same since the rename, and NTILE already did. Quoting it as "n" fixes it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. Quoted as "n" here and on LAG, LEAD and NTILE.

Comment thread docs/data/functions/json/json_array.yml Outdated
syntax:
sql: JSON_ARRAY([expr [, ...]] [{NULL | ABSENT} ON NULL])
java: jsonArray(JsonOnNull.ABSENT, expr1, expr2, ...)
python: json_array(JsonOnNull.ABSENT, expr1, expr2, ...)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call fails in PyFlink: py4j doesn't pack Java varargs, so it reports Method jsonArray(...) does not exist. Same for json_object. I think it's better to leave it out, like DESCRIPTOR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed here and on JSON_OBJECT, as on DESCRIPTOR. Every function built on _varargs_op has this problem, while and_ and or_ work because they build a Java array first. That needs a separate fix in PyFlink.

- name: pattern
type: CHAR, VARCHAR, STRING
required: true
description: The pattern. `%` matches any number of characters, and `_` exactly one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_ doesn't match a newline, while % does: 'a' || CHR(10) || 'b' LIKE 'a_b' is FALSE, because SqlLikeUtils turns _ into . without DOTALL. That's a bug for Jira, and a note here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. SIMILAR TO does the same for _ and ., so its ". matches any character" was wrong too. Both pages have a note now, backed by a regression probe. I'll file the Jira together with the other bugs found while writing these pages later. I want to keep the scope of the errors small now and re-evaluate them later


returns:
description: The number of fields of an object or of elements of an array, at `path` or of the whole value. A scalar has length 1, also a JSON `null`.
nulls: Returns NULL if `jsonValue` is NULL or isn't valid JSON, or if `path` finds no value or more than one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Content after the first value is ignored here, as on the IS JSON page: JSON_LENGTH('[1,2] x') returns 2, and JSON_TYPE('{"a":1} x') returns object.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, and it holds for all five: JSON_VALUE, JSON_QUERY, JSON_EXISTS, JSON_LENGTH and JSON_TYPE. Each page now has the same note as IS JSON, with an example.

`TIME` keeps microseconds, so `12:30:45.1234567` is stored as `12:30:45.123456`. `TIMESTAMP` and
`TIMESTAMP_LTZ` keep nanoseconds if the value has digits below a microsecond and is in the
nanosecond range. A 64-bit count of nanoseconds since `1970-01-01 00:00:00` UTC covers
`1677-09-21 00:12:43.145224192` to `2262-04-11 23:47:16.854775807` UTC. A `dateTime` without an

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lower bound loses its nanoseconds: this value reads back as 1677-09-21 00:12:43.145224, and so does anything before 00:12:44. I think ChronoUnit.NANOS.between in BinaryVariantUtil overflows there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created a bug ticket and assigned it to @manner https://issues.apache.org/jira/browse/FLINK-41591

Comment thread docs/README.md Outdated

NULL handling has no section of its own. It goes into Returns, and the examples show it.

Name an argument that takes any expression `expr`, or `expr1`, `expr2` when there are several, as other engines do: `expr1 = expr2`, `LAG(expr [, n])`. Give an argument with a fixed type or role a name that says so, such as `str`, `pattern` or `lower_bound`. The Java tab writes the same names in camelCase. Don't name an argument or an alias in an example after an SQL keyword, such as `type`, `key` or `default`. The syntax highlighter colors those like SQL, so write `data_type`, `field_name` or `default_value` instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SQL syntax of eight new pages uses jsonValue, hashLength and returnType, while this asks for names like lower_bound there. Should those become json_value, hash_length and return_type?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are now json_string, hash_length and return_type, and extract_index on REGEXP_EXTRACT and REGEXP_EXTRACT_ALL too.

- sql: value IS NOT NULL
table: value.isNotNull
description: 如果值不为 `NULL` 返回 `TRUE`。
- sql: expr BETWEEN [ ASYMMETRIC | SYMMETRIC ] lower_bound AND upper_bound

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops about 1,600 characters of Chinese text from these five categories, and FLINK-41529 only covers the previous five. Could you extend it or open a translation ticket for these?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened FLINK-41590 for these five categories

syntax:
sql: |
CASE value WHEN value1 [, value2]* THEN result1 [WHEN ... THEN ...]* [ELSE result_else] END
CASE expr WHEN expr1 [, expr2]* THEN result1 [WHEN ... THEN ...]* [ELSE result_else] END

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if we should specify anywhere whether it is lazy or not, same for others
what about other vendors?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good questions. BigQuery, Databricks and PostgreSQL document this per function, PostgreSQL with the same caveat, while Snowflake and DuckDB don't. Each of these pages now has a note, and a regression probe keeps each note true.

Comment thread docs/data/functions/json/json_quote.yml Outdated
- title: Characters above U+FFFF
description: An emoji is escaped wrongly and doesn't come back from `JSON_UNQUOTE`.
sql: |
SELECT JSON_QUOTE('😀') AS emoji, JSON_UNQUOTE(JSON_QUOTE('😀')) = '😀' AS round_trips;

@snuyanzin snuyanzin Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't it be considered as a bug which might be fixed at some point?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's a bug, and it's on our list together with CHR(1) not being escaped. We document the current behavior and say that it's wrong, as agreed with Martijn on #29864, since users who try it would otherwise think the docs are wrong. The fix will update this example in the same PR. I'm collecting the bugs found while writing these pages and will file them together once the last categories are done.

…r expressions

Two rules for the function pages, applied here to the pages of the cast, conditional, regular expression, utility and window functions. docs/README.md states both, and AGENTS.md the first.

An argument that takes any expression is expr, or expr1 and expr2, as Databricks, Snowflake and DuckDB name it. No argument or example alias is a word that the syntax highlighter colors like SQL: CAST(value AS type) becomes CAST(expr AS data_type), and LAG's offset and default become n and default_value. The window pages used expression, and IFNULL and TYPEOF input.

An example selects its inputs first, from a VALUES table, and names each result column after its expression, so that a row reads from input to result without going back to the query. An example of literals only becomes one row per case, and a window function names its window, as in RANK() OVER w.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
The result of an example is a Markdown table, and the page rendered its cells as Markdown. On the REGEXP_REPLACE page, the masked value ****-**** showed as a bold dash. Quotes came out curly, backslash escapes disappeared, and a tag such as <b> in a value rendered as HTML.

Render the result table as HTML with escaped cells, so that each cell shows the text the query returns.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. A page covers an operator and its NOT form, and the six operators =, <>, <, <=, > and >= share one page. Compared with the old table rows, the pages add:

- <, <=, > and >= cast a string to a number, while = and <> reject a string and a number. TRUE = 'true' is FALSE.
- SIMILAR TO supports ESCAPE, and a . matches any character.
- NOT IN with a NULL in the list or the sub-query keeps no row. Sub-queries run in streaming mode too.
- The Java and Python calls, such as isEqual(...) instead of the Scala ===.

The IS_NAN page documents the name IS_NAN, which SQL doesn't resolve yet: built-in functions are registered under their name, here isNan, not their SQL name. Its example results come from running ISNAN, and the fix gets its own ticket.

The Chinese pages keep the first sentence of the existing Chinese description as their summary. The comparison operators and IS DISTINCT FROM pages show the English summary, since their Chinese sentence described only = or mistranslated the operator.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against Calcite's operators, the generated code and the tests, and every example result comes from running its SQL. AND, OR and NOT share one page, and IS TRUE, IS FALSE and IS UNKNOWN each get a page with their NOT form. Compared with the old table rows, the pages add:

- FALSE AND NULL is FALSE and TRUE OR NULL is TRUE, while the other combinations with NULL are NULL.
- The IS tests never return NULL. IS UNKNOWN only takes a BOOLEAN and has no Table API method.
- The Java and Python calls. The old rows showed only the Scala ones.

The IS NULL page links to IS UNKNOWN from this commit on.

The Chinese pages keep the first sentence of the existing Chinese description as their summary. The logical operators page shows the English summary, since its Chinese sentence described only OR.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, the generated code and the tests, and every example result comes from running its SQL and matches the digest of another tool. Compared with the old table rows, the pages add:

- The digest is lower-case hexadecimal, and a string is hashed as its UTF-8 bytes. In SQL, a binary string works too.
- The result types: CHAR(32) to CHAR(128) for the fixed-length functions, VARCHAR(128) for SHA2.
- SHA2 fails for a length other than 224, 256, 384 or 512.
- The Python calls. md5 to sha512 are properties there.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, Calcite's JSON functions, the generated code and the tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add:

- Strict mode is the default, so a missing key goes to ON ERROR, and ON EMPTY applies only in lax mode. JSON_LENGTH and JSON_TYPE take no mode at all.
- How JSON_STRING writes each type. A FLOAT gets the digits of a DOUBLE.
- JSON_QUOTE escapes a character above U+FFFF wrongly and leaves most control characters as they are.
- Whether JSON_OBJECT and JSON_ARRAY insert a nested call as JSON depends on the query. Wrapping a value in JSON always works.
- The Table API calls, such as jsonQuote() instead of JsonQuote() and jsonLength(...) as a method.

The Chinese table gets the JSON(value) row it lacked. The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. The two subscript rows, variant[INT] and variant[STRING], share one page. The pages link the sections of the category page on XML and on paths instead of repeating them. Compared with the old table rows, the pages add:

- The result types and NULL cases of each function. A JSON null is a variant null, not SQL NULL.
- A subscript on the wrong kind of value fails the job, also for a JSON null in a chain, while VARIANT_GET returns NULL.
- XML_STRING fails for a VARIANT that isn't an object with one field, or that has no XML form.

The category page now says that TRY_VARIANT_GET returns NULL for a malformed path that isn't a literal, and that reading XML back with PARSE_XML needs the same force_array. The JSON pages link to the VARIANT pages from this commit on.

The Chinese pages keep the first sentence of the existing Chinese description as their summary.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
…their own pages

The XML mapping, about 220 lines with tables, filled most of the Variant category page. It moves to a page "XML and VARIANT" with both directions, since they share the conventions for attributes, text, order and xsi:type. The paths that VARIANT_GET and TRY_VARIANT_GET take move to a page "VARIANT Paths". The category page keeps its intro and the function table, like the other categories, and links both pages.

The function pages, the VARIANT section of the data types page and the anchors of the old single function page point at the new pages. Apart from one intro sentence, the moved text doesn't change.

Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants