feat: encoding/utf16

[peter-jerry-ye]

This PR adds UTF16 encoding/decoding, and deprecate any other String to Bytes conversions as they are misleading.

[coveralls]

Pull Request Test Coverage Report for Build 1636

Details

• 65 of 83 (78.31%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.09%) to 89.775%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
encoding/utf16/decode.mbt	46	64	71.88%

Totals
Change from base Build 1635:	-0.09%
Covered Lines:	9842
Relevant Lines:	10963

---

💛 - Coveralls

[bobzhang]

fn encode(StringView, bom? : Bool, endianness : Endian = Little) -> Bytes

[peter-jerry-ye]

This will likely double the code size, introduce unnecessary branching.

[bobzhang]

in most cases, this is not what we need, encode is mostly what we need

[peter-jerry-ye-code-review]

Potential buffer overflow in big-endian decode when converting surrogate pairs

Category
Correctness
Code Snippet
string_bytes[i] = (higher & 0xFF).to_byte()
string_bytes[i + 1] = (higher >> 8).to_byte()
string_bytes[i + 2] = (lower & 0xFF).to_byte()
string_bytes[i + 3] = (lower >> 8).to_byte()
i += 4
Recommendation
Add bounds checking before writing to string_bytes array, or pre-calculate the required buffer size considering surrogate pairs
Reasoning
The code assumes each surrogate pair needs 4 bytes but the initial buffer is allocated as bytes.length() which may not account for the expansion from UTF-16 surrogate pairs to the internal string representation

Inefficient byte-by-byte copying in big-endian encoding

Category
Performance
Code Snippet
for i in 0..<str.length() {
let code_unit = str[i]
arr[i * 2] = (code_unit >> 8).to_byte()
arr[i * 2 + 1] = (code_unit & 0xFF).to_byte()
}
Recommendation
Use bulk memory operations or SIMD instructions when available, similar to how little-endian uses blit_from_string
Reasoning
The big-endian path manually loops through each character and performs bit operations, while little-endian uses optimized blit_from_string. This creates a performance disparity between endianness modes

Duplicated BOM handling and validation logic between decode functions

Category
Maintainability
Code Snippet
let bytes = if ignore_bom {
if endianness is Little && bytes is [.. "\xff\xfe", .. rest] {
rest
} else if endianness is Big && bytes is [.. "\xfe\xff", .. rest] {
rest
} else {
bytes
}
} else {
bytes
}
Recommendation
Extract BOM handling into a separate helper function: fn strip_bom(bytes: BytesView, endianness: Endian) -> BytesView
Reasoning
The exact same BOM stripping logic is duplicated in both decode and decode_lossy functions, violating DRY principle and making maintenance harder