What version of regex are you using?
1.11.1
Describe the bug at a high level.
When using regex::bytes with unicode mode enabled (https://docs.rs/regex/latest/regex/bytes/struct.RegexBuilder.html#method.unicode), iterating over matches does not respect unicode character boundaries, but instead iterates over the raw bytes.
What are the steps to reproduce the behavior?
let re = regex::bytes::RegexBuilder::new(r"").unicode(true).build().unwrap();
let subject = "😃".as_bytes(); // I.e. U+1F603
assert_eq!(subject, b"\xF0\x9F\x98\x83"); // 4 UTF-8 bytes
for m in re.find_iter(subject ) {
println!("{:?}", m);
}
let res = re.replace_all(subject, b"<$0>");
println!("{}", String::from_utf8_lossy(&res));What is the actual behavior?
The above prints
Match { start: 0, end: 0, bytes: "" }
Match { start: 1, end: 1, bytes: "" }
Match { start: 2, end: 2, bytes: "" }
Match { start: 3, end: 3, bytes: "" }
Match { start: 4, end: 4, bytes: "" }
<>�<>�<>�<>�<>
In other words, both find_iter and replace_all are operating on the individual byte level and not the UTF-8 character level.
What is the expected behavior?
I expected the above to print:
Match { start: 0, end: 0, string: "" }
Match { start: 4, end: 4, string: "" }
<>😃<>
which is exactly what happens when I use a regex::RegexBuilder instead of a regex::bytes::RegexBuilder.
If I change the regex from "" to ".', both work properly:
Match { start: 0, end: 4, string: "😃" }
<😃>
You may just tell me "don't use regex::bytes", but that is not a solution if what i'm matching over has mixed valid and invalid UTF-8, whereas '.' works correctly there.
What version of regex are you using?
1.11.1
Describe the bug at a high level.
When using
regex::byteswith unicode mode enabled (https://docs.rs/regex/latest/regex/bytes/struct.RegexBuilder.html#method.unicode), iterating over matches does not respect unicode character boundaries, but instead iterates over the raw bytes.What are the steps to reproduce the behavior?
What is the actual behavior?
The above prints
In other words, both
find_iterandreplace_allare operating on the individual byte level and not the UTF-8 character level.What is the expected behavior?
I expected the above to print:
which is exactly what happens when I use a
regex::RegexBuilderinstead of aregex::bytes::RegexBuilder.If I change the regex from
""to".', both work properly:You may just tell me "don't use
regex::bytes", but that is not a solution if what i'm matching over has mixed valid and invalid UTF-8, whereas '.' works correctly there.