Adopt GB18030-2022 #336
Open
Adopt GB18030-2022 #336
+129
−58
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030. In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following: 1. Merges the first set of 18 mappings, which are bidirectional, directly into index gb18030, replacing existing PUA entries. This ends up impacting GBK and gb18030. 2. The second set of 18 mappings (from PUA to bytes) are encoded as an encoder only table, for both GBK and gb18030. 3. The third set of 18 mappings (from bytes to code points) are ignored, as they are already covered by index gb18030 ranges. (Presumably they are included because the recommendation covers the transition from "Previous Mappings" to "Current Mappings" to "Recommended Mappings", whereas we are going directly from "Previous Mappings" to "Recommended Mappings".) The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely. Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240. This supersedes #335. This fixes #27 and fixes #312.
This was referenced Sep 18, 2024
Closed
annevk
added a commit
to annevk/WebKit
that referenced
this pull request
Sep 18, 2024
https://bugs.webkit.org/show_bug.cgi?id=279903 Reviewed by NOBODY (OOPS!). For GBK and gb18030 we have used the same backing table for quite a while now. This backing table was updated to account for GB18030-2022 at some point and this impacted GBK as well. However, the encoder side table was kept disabled for GBK, despite it actually allowing GBK to be more compatible with its former self. whatwg/encoding#336 now standardizes the behavior that GBK and gb18030 are to remain aligned in these matters and this change implements that. The corresponding tests are from this PR: web-platform-tests/wpt#48240 * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-decoder.any.js: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder.html: * Source/WebCore/PAL/pal/text/TextCodecCJK.cpp: (PAL::gb18030AsymmetricEncode): (PAL::gbEncodeShared):
This was referenced Sep 18, 2024
hsivonen
reviewed
Sep 18, 2024
hsivonen
approved these changes
Sep 18, 2024
xfq
added
the
i18n-tracker
hsivonen
approved these changes
Sep 19, 2024
webkit-commit-queue
pushed a commit
to annevk/WebKit
that referenced
this pull request
Sep 20, 2024
https://bugs.webkit.org/show_bug.cgi?id=279903 Reviewed by Alex Christensen. For GBK and gb18030 we have used the same backing table for quite a while now. This backing table was updated to account for GB18030-2022 at some point and this impacted GBK as well. However, the encoder side table was kept disabled for GBK, despite it actually allowing GBK to be more compatible with its former self. whatwg/encoding#336 now standardizes the behavior that GBK and gb18030 are to remain aligned in these matters and this change implements that. The corresponding tests are from this PR: web-platform-tests/wpt#48240 * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-decoder.any.js: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder.html: * Source/WebCore/PAL/pal/text/TextCodecCJK.cpp: (PAL::gb18030AsymmetricEncode): (PAL::gbEncodeShared): Canonical link: https://commits.webkit.org/283987@main
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030.
In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following:
The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely.
Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240.
This supersedes #335. This fixes #27 and fixes #312.
(See WHATWG Working Mode: Changes for more details.)
Preview | Diff