Repository navigation
Conversation
53946e3 to
c22cef1
Compare
MartijnVisser
left a comment
There was a problem hiding this comment.
Thanks, I ran all 146 examples in batch and streaming mode and they match. A few statements don't hold, see the comments.
| type: VARIANT | ||
| required: true | ||
| description: An array or an object. | ||
| - name: n |
There was a problem hiding this comment.
YAML reads an unquoted n as a boolean, so the Arguments table renders false here. LAG and LEAD show the same since the rename, and NTILE already did. Quoting it as "n" fixes it.
There was a problem hiding this comment.
Thanks, good catch. Quoted as "n" here and on LAG, LEAD and NTILE.
| syntax: | ||
| sql: JSON_ARRAY([expr [, ...]] [{NULL | ABSENT} ON NULL]) | ||
| java: jsonArray(JsonOnNull.ABSENT, expr1, expr2, ...) | ||
| python: json_array(JsonOnNull.ABSENT, expr1, expr2, ...) |
There was a problem hiding this comment.
This call fails in PyFlink: py4j doesn't pack Java varargs, so it reports Method jsonArray(...) does not exist. Same for json_object. I think it's better to leave it out, like DESCRIPTOR.
There was a problem hiding this comment.
Agreed, removed here and on JSON_OBJECT, as on DESCRIPTOR. Every function built on _varargs_op has this problem, while and_ and or_ work because they build a Java array first. That needs a separate fix in PyFlink.
| - name: pattern | ||
| type: CHAR, VARCHAR, STRING | ||
| required: true | ||
| description: The pattern. `%` matches any number of characters, and `_` exactly one. |
There was a problem hiding this comment.
_ doesn't match a newline, while % does: 'a' || CHR(10) || 'b' LIKE 'a_b' is FALSE, because SqlLikeUtils turns _ into . without DOTALL. That's a bug for Jira, and a note here.
There was a problem hiding this comment.
Confirmed. SIMILAR TO does the same for _ and ., so its ". matches any character" was wrong too. Both pages have a note now, backed by a regression probe. I'll file the Jira together with the other bugs found while writing these pages later. I want to keep the scope of the errors small now and re-evaluate them later
|
|
||
| returns: | ||
| description: The number of fields of an object or of elements of an array, at `path` or of the whole value. A scalar has length 1, also a JSON `null`. | ||
| nulls: Returns NULL if `jsonValue` is NULL or isn't valid JSON, or if `path` finds no value or more than one. |
There was a problem hiding this comment.
Content after the first value is ignored here, as on the IS JSON page: JSON_LENGTH('[1,2] x') returns 2, and JSON_TYPE('{"a":1} x') returns object.
There was a problem hiding this comment.
yes, and it holds for all five: JSON_VALUE, JSON_QUERY, JSON_EXISTS, JSON_LENGTH and JSON_TYPE. Each page now has the same note as IS JSON, with an example.
| `TIME` keeps microseconds, so `12:30:45.1234567` is stored as `12:30:45.123456`. `TIMESTAMP` and | ||
| `TIMESTAMP_LTZ` keep nanoseconds if the value has digits below a microsecond and is in the | ||
| nanosecond range. A 64-bit count of nanoseconds since `1970-01-01 00:00:00` UTC covers | ||
| `1677-09-21 00:12:43.145224192` to `2262-04-11 23:47:16.854775807` UTC. A `dateTime` without an |
There was a problem hiding this comment.
The lower bound loses its nanoseconds: this value reads back as 1677-09-21 00:12:43.145224, and so does anything before 00:12:44. I think ChronoUnit.NANOS.between in BinaryVariantUtil overflows there.
There was a problem hiding this comment.
Created a bug ticket and assigned it to @manner https://issues.apache.org/jira/browse/FLINK-41591
|
|
||
| NULL handling has no section of its own. It goes into Returns, and the examples show it. | ||
|
|
||
| Name an argument that takes any expression `expr`, or `expr1`, `expr2` when there are several, as other engines do: `expr1 = expr2`, `LAG(expr [, n])`. Give an argument with a fixed type or role a name that says so, such as `str`, `pattern` or `lower_bound`. The Java tab writes the same names in camelCase. Don't name an argument or an alias in an example after an SQL keyword, such as `type`, `key` or `default`. The syntax highlighter colors those like SQL, so write `data_type`, `field_name` or `default_value` instead. |
There was a problem hiding this comment.
The SQL syntax of eight new pages uses jsonValue, hashLength and returnType, while this asks for names like lower_bound there. Should those become json_value, hash_length and return_type?
There was a problem hiding this comment.
Yes, they are now json_string, hash_length and return_type, and extract_index on REGEXP_EXTRACT and REGEXP_EXTRACT_ALL too.
| - sql: value IS NOT NULL | ||
| table: value.isNotNull | ||
| description: 如果值不为 `NULL` 返回 `TRUE`。 | ||
| - sql: expr BETWEEN [ ASYMMETRIC | SYMMETRIC ] lower_bound AND upper_bound |
There was a problem hiding this comment.
This drops about 1,600 characters of Chinese text from these five categories, and FLINK-41529 only covers the previous five. Could you extend it or open a translation ticket for these?
There was a problem hiding this comment.
I opened FLINK-41590 for these five categories
| syntax: | ||
| sql: | | ||
| CASE value WHEN value1 [, value2]* THEN result1 [WHEN ... THEN ...]* [ELSE result_else] END | ||
| CASE expr WHEN expr1 [, expr2]* THEN result1 [WHEN ... THEN ...]* [ELSE result_else] END |
There was a problem hiding this comment.
i wonder if we should specify anywhere whether it is lazy or not, same for others
what about other vendors?
There was a problem hiding this comment.
Good questions. BigQuery, Databricks and PostgreSQL document this per function, PostgreSQL with the same caveat, while Snowflake and DuckDB don't. Each of these pages now has a note, and a regression probe keeps each note true.
| - title: Characters above U+FFFF | ||
| description: An emoji is escaped wrongly and doesn't come back from `JSON_UNQUOTE`. | ||
| sql: | | ||
| SELECT JSON_QUOTE('😀') AS emoji, JSON_UNQUOTE(JSON_QUOTE('😀')) = '😀' AS round_trips; |
There was a problem hiding this comment.
shouldn't it be considered as a bug which might be fixed at some point?
There was a problem hiding this comment.
Yes, it's a bug, and it's on our list together with CHR(1) not being escaped. We document the current behavior and say that it's wrong, as agreed with Martijn on #29864, since users who try it would otherwise think the docs are wrong. The fix will update this example in the same PR. I'm collecting the bugs found while writing these pages and will file them together once the last categories are done.
…r expressions Two rules for the function pages, applied here to the pages of the cast, conditional, regular expression, utility and window functions. docs/README.md states both, and AGENTS.md the first. An argument that takes any expression is expr, or expr1 and expr2, as Databricks, Snowflake and DuckDB name it. No argument or example alias is a word that the syntax highlighter colors like SQL: CAST(value AS type) becomes CAST(expr AS data_type), and LAG's offset and default become n and default_value. The window pages used expression, and IFNULL and TYPEOF input. An example selects its inputs first, from a VALUES table, and names each result column after its expression, so that a row reads from input to result without going back to the query. An example of literals only becomes one row per case, and a window function names its window, as in RANK() OVER w. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
The result of an example is a Markdown table, and the page rendered its cells as Markdown. On the REGEXP_REPLACE page, the masked value ****-**** showed as a bold dash. Quotes came out curly, backslash escapes disappeared, and a tag such as <b> in a value rendered as HTML. Render the result table as HTML with escaped cells, so that each cell shows the text the query returns. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. A page covers an operator and its NOT form, and the six operators =, <>, <, <=, > and >= share one page. Compared with the old table rows, the pages add: - <, <=, > and >= cast a string to a number, while = and <> reject a string and a number. TRUE = 'true' is FALSE. - SIMILAR TO supports ESCAPE, and a . matches any character. - NOT IN with a NULL in the list or the sub-query keeps no row. Sub-queries run in streaming mode too. - The Java and Python calls, such as isEqual(...) instead of the Scala ===. The IS_NAN page documents the name IS_NAN, which SQL doesn't resolve yet: built-in functions are registered under their name, here isNan, not their SQL name. Its example results come from running ISNAN, and the fix gets its own ticket. The Chinese pages keep the first sentence of the existing Chinese description as their summary. The comparison operators and IS DISTINCT FROM pages show the English summary, since their Chinese sentence described only = or mistranslated the operator. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each operator was rechecked against Calcite's operators, the generated code and the tests, and every example result comes from running its SQL. AND, OR and NOT share one page, and IS TRUE, IS FALSE and IS UNKNOWN each get a page with their NOT form. Compared with the old table rows, the pages add: - FALSE AND NULL is FALSE and TRUE OR NULL is TRUE, while the other combinations with NULL are NULL. - The IS tests never return NULL. IS UNKNOWN only takes a BOOLEAN and has no Table API method. - The Java and Python calls. The old rows showed only the Scala ones. The IS NULL page links to IS UNKNOWN from this commit on. The Chinese pages keep the first sentence of the existing Chinese description as their summary. The logical operators page shows the English summary, since its Chinese sentence described only OR. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, the generated code and the tests, and every example result comes from running its SQL and matches the digest of another tool. Compared with the old table rows, the pages add: - The digest is lower-case hexadecimal, and a string is hashed as its UTF-8 bytes. In SQL, a binary string works too. - The result types: CHAR(32) to CHAR(128) for the fixed-length functions, VARCHAR(128) for SHA2. - SHA2 fails for a length other than 224, 256, 384 or 512. - The Python calls. md5 to sha512 are properties there. The Chinese pages keep the first sentence of the existing Chinese description as their summary. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, Calcite's JSON functions, the generated code and the tests, and every example result comes from running its SQL. Compared with the old table rows, the pages add: - Strict mode is the default, so a missing key goes to ON ERROR, and ON EMPTY applies only in lax mode. JSON_LENGTH and JSON_TYPE take no mode at all. - How JSON_STRING writes each type. A FLOAT gets the digits of a DOUBLE. - JSON_QUOTE escapes a character above U+FFFF wrongly and leaves most control characters as they are. - Whether JSON_OBJECT and JSON_ARRAY insert a nested call as JSON depends on the query. Wrapping a value in JSON always works. - The Table API calls, such as jsonQuote() instead of JsonQuote() and jsonLength(...) as a method. The Chinese table gets the JSON(value) row it lacked. The Chinese pages keep the first sentence of the existing Chinese description as their summary. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
Each function was rechecked against its type strategies, runtime code and tests, and every example result comes from running its SQL. The two subscript rows, variant[INT] and variant[STRING], share one page. The pages link the sections of the category page on XML and on paths instead of repeating them. Compared with the old table rows, the pages add: - The result types and NULL cases of each function. A JSON null is a variant null, not SQL NULL. - A subscript on the wrong kind of value fails the job, also for a JSON null in a chain, while VARIANT_GET returns NULL. - XML_STRING fails for a VARIANT that isn't an object with one field, or that has no XML form. The category page now says that TRY_VARIANT_GET returns NULL for a malformed path that isn't a literal, and that reading XML back with PARSE_XML needs the same force_array. The JSON pages link to the VARIANT pages from this commit on. The Chinese pages keep the first sentence of the existing Chinese description as their summary. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
…their own pages The XML mapping, about 220 lines with tables, filled most of the Variant category page. It moves to a page "XML and VARIANT" with both directions, since they share the conventions for attributes, text, order and xsi:type. The paths that VARIANT_GET and TRY_VARIANT_GET take move to a page "VARIANT Paths". The category page keeps its intro and the function table, like the other categories, and links both pages. The function pages, the VARIANT section of the data types page and the anchors of the old single function page point at the new pages. Apart from one intro sentence, the moved text doesn't change. Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)
c22cef1 to
c4cdd0c
Compare
What is the purpose of the change
Step 2 of FLINK-23804, continued after #29864: every function and operator of five more categories gets a page of its own, generated from YAML. Each page was rechecked against the implementation, and every example result comes from running its SQL in batch and streaming mode. The recheck changed what the docs say, for example:
=TRUE = 'true'is FALSE: the string isn't castSIMILAR TOESCAPEworks. A.matches any character but a newline, as in JavaJSON_OBJECTJSON_QUOTE",\,/and the control characters\b,\f,\n,\r,\tCHR(1)isn't escapedJSON_UNQUOTE'"a"b"'returnsa"bBrief change log
docs/README.mdstates them.expr, orexpr1andexpr2, as Databricks, Snowflake and DuckDB name it. Other names are snake_case, such asjson_string, and no argument or alias is a word the syntax highlighter colors like SQL, such astypeinCAST(value AS type)oroffsetinLAG.AGENTS.mdstates this one too.qty > 10 AND price > 10 AS `qty > 10 AND price > 10`, so a row reads from input to result without going back to the query.****-****showed as a bold dash, and on the new XML_STRING page<a><b>1</b></a>would have shown as a bold 1.IS NULLandIS NOT NULL. The operators=,<>,<,<=,>and>=share a "Comparison operators" page, andAND,ORandNOTa "Logical operators" page. The two VARIANT subscript rows share one page. The split follows Snowflake for the grouped operators and Databricks for theIStests.isEqual(...)for===andjsonQuote()forJsonQuote(). The Chinese table gets theJSON(value)row it lacked.TRY_VARIANT_GETreturns NULL for a malformed path that isn't a literal, and reading XML back withPARSE_XMLneeds the sameforce_array.Verifying this change
This change is documentation only.
The recheck found bugs in the code, which this PR doesn't fix. The pages describe the current behavior, also where it's a bug that users would plausibly hit, such as
JSON_STRINGwriting aFLOATwith the digits of aDOUBLE. They name no tickets. I'll file the bugs separately. FLINK-41591 is the first, for the nanosecond range ofPARSE_XML.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.328 (Claude Opus 5.5)