fix(url): remove dot segments and fold special-scheme separators

changelog.d/13195-url-dot-segments.md

@@ -0,0 +1,20 @@
+Remove `.` and `..` path segments while parsing a URL, per the WHATWG "path state" (#12864). `new URL("https://x/a/b/../c").pathname` was `/a/b/../c` instead of `/a/c`, so `href` carried the unnormalized text too — the case routers and path-based access checks depend on (`/public/../admin` must read as `/admin`).
+
+Root cause was structural, not a dropped branch: `parse_url` never had the path-state pass at all. It split the authority off the input and stored the remaining text as `pathname` verbatim; every consumer (`create_url_object`, `new URL(str, base)`, `URL.parse`, the `href`/`pathname` setters, `fileURLToPath`) then built on that raw string. WHATWG keeps a URL's path as a *list* of segments and drops dot segments as it reads them, which is why the standard's rules are asymmetric in a way a plain `pop` loop cannot reproduce: a dot segment is dropped, but when it is the LAST thing in the path the parser appends an empty segment instead of nothing, so `/a/..` is `/` while `/a/b/..` is `/a/`.
+
+`crates/perry-runtime/src/url/parse.rs` gains `normalize_hierarchical_path`, the path-shape rules applied to a path string: `.`/`%2e` are single-dot segments, `..`/`.%2e`/`%2e.`/`%2e%2e` are double-dot (ASCII case-insensitive, per the standard), empty segments from a `//` run survive verbatim (`//a/../b` → `//b`), a segment that merely contains a dot is untouched (`/a/.../b`), and the `file:`-only rules apply — a Windows drive letter canonicalizes to `:` (`file:///C|/x` → `/C:/x`) and a path that is only a drive letter is never shortened (`file:///C:/..` stays `/C:/`). `parse_url` applies it to the hierarchical branches only, so an opaque path (`mailto:a/../b`, `data:`) keeps its text.
+
+The same parse pass also had the rest of the special-scheme shape missing, and it is one rule family, so it lands here rather than as a second defect:
+
+- **`\` is a separator for special schemes.** `http://x\a\b` is `http://x/a/b` and `\` terminates the authority, so folding happens before the host is read. Query and fragment keep theirs — their encode sets do not encode `\` (`?q=\a`, `#f\b`) — and a non-special path keeps them too (`foo://a/b\c` is `/b\c`). A backslash in a non-special *host* is a forbidden host code point, so `foo://a\b` is now invalid, as in Node.
+- **The `//` after a special scheme is optional and its separator run is skipped.** `https:x`, `https:/x` and `https:\x` host `x`; `https:///x` and `https:////x` do too. Special schemes are therefore never classified as opaque, while `file:` keeps its own separator rules: extra separators are path (`file:////x` is an empty host with the path `//x`), its host may be empty (`file:`, `file:x`, `file:/x`), and `file:\\host` carries an authority marker written with backslashes.
+- **`file:` hosts canonicalize like any other special scheme's** (`file://SERVER/x` → `server`, `file://123/x` → `0.0.0.123`, `file://LOCALHOST/x` → empty host). They previously skipped the WHATWG host parser entirely.
+- **`is_valid_absolute_url` mirrors all of the above** instead of requiring a literal `//` for a special scheme, and no longer rejects an opaque URL with nothing after the colon (`mailto:` is valid, as in Node).
+
+`js_url_set_pathname` runs the same states, because assignment clears the path and re-parses the value: a special scheme folds `\`, the value gains a leading separator when it has none (`\a\b` and `b\..\c` both root), dot segments and a `file:` drive letter are canonicalized, and a non-hierarchical URL is left alone.
+
+Note the fast-path guard in the dot-segment pass: it must skip work only when the path has neither `.` nor `%`. `%2e` contains no literal dot, so a `.`-only check silently returned the unnormalized path for every percent-encoded dot segment — caught by the new unit test rather than by reasoning.
+
+Tests: `test-parity/node-suite/url/url/dot-segments.ts` covers the issue's own reproduction plus the trailing-slash asymmetry, empty segments, contained dots, percent-encoded spellings, non-special and opaque schemes, `file:` and Windows-drive paths, base resolution, the `pathname` setter, and `URL.parse`. `test-parity/node-suite/url/url/backslash-and-separators.ts` covers the special-scheme shape: backslash folding and authority termination, the skipped separator run, the optional `//`, the `file:` forms (including its host canonicalization), the `|` drive spelling, invalid non-special hosts, `URL.canParse`, and the setter for all three shapes. Unit coverage in `crates/perry-runtime/src/url/mod.rs` pins the path rules, the parser's classification, and the validator table directly.
+
+Validated byte-for-byte against the pinned Node oracle: the issue's command prints `0 /a/c /c`, both new cases PASS through `run_parity_tests.sh`, and the whole `test-parity/node-suite/url` module is 69/69 PASS with no previously-passing case changed. No version or dependency changes.

crates/perry-runtime/src/url/parse.rs

@@ -4,6 +4,24 @@ use super::*;
 
 use super::search_params::{create_url_search_params_object, parse_query_string};
 
+/// WHATWG "special scheme": has a host, and its parser treats `\` as a path
+/// separator (`foo:`, `mailto:`, `data:` paths keep it).
+pub(crate) fn is_special_url_scheme(protocol: &str) -> bool {
+    is_special_scheme_name(protocol.strip_suffix(':').unwrap_or(protocol))
+}
+
+fn is_special_scheme_name(scheme: &str) -> bool {
+    ["ftp", "file", "http", "https", "ws", "wss"]
+        .iter()
+        .any(|name| scheme.eq_ignore_ascii_case(name))
+}
+
+fn special_scheme_prefix(input: &str) -> bool {
+    input
+        .find(':')
+        .is_some_and(|colon| is_special_scheme_name(&input[..colon]))
+}
+
 /// Simple URL parser
 /// Returns (protocol, host, hostname, port, pathname, search, hash)
 pub(crate) fn parse_url(url_str: &str) -> (String, String, String, String, String, String, String) {
@@ -35,6 +53,17 @@ pub(crate) fn parse_url(url_str: &str) -> (String, String, String, String, Strin
         remaining = &remaining[..query_idx];
     }
 
+    // A special scheme's `\` is a separator, including where the authority
+    // marker sits. Query and fragment keep theirs (their encode sets don't
+    // encode it) and were split off above, so the fold happens here.
+    let folded;
+    let mut remaining = if special_scheme_prefix(remaining) && remaining.contains('\\') {
+        folded = remaining.replace('\\', "/");
+        folded.as_str()
+    } else {
+        remaining
+    };
+
     // Track whether the `file:` URL carried a `//` authority marker so the
     // file branch below can split out a UNC-style host (`file://host/path`)
     // from a hostless path (`file:///path`, `file:/path`). #2975
@@ -56,9 +85,8 @@ pub(crate) fn parse_url(url_str: &str) -> (String, String, String, String, Strin
             remaining = remaining.strip_prefix("//").unwrap_or(remaining);
         }
     } else if let Some(colon_idx) = remaining.find(':') {
-        // Non-special opaque scheme: `mailto:`, `data:`, `urn:`, etc. The
-        // characters before the colon must look like a scheme. The whole
-        // remainder is the opaque pathname; host/hostname/port stay empty.
+        // A special scheme is always hierarchical (the `//` is optional), so
+        // only the other schemes are opaque.
         let scheme = &remaining[..colon_idx];
         let scheme_ok = !scheme.is_empty()
             && scheme
@@ -70,19 +98,30 @@ pub(crate) fn parse_url(url_str: &str) -> (String, String, String, String, Strin
                 .all(|c| c.is_ascii_alphanumeric() || c == '+' || c == '-' || c == '.');
         if scheme_ok {
             protocol = format!("{}:", scheme.to_ascii_lowercase());
-            pathname = remaining[colon_idx + 1..].to_string();
-            return (
-                protocol,
-                String::new(),
-                String::new(),
-                String::new(),
-                pathname,
-                search,
-                hash,
-            );
+            if is_special_url_scheme(&protocol) {
+                remaining = &remaining[colon_idx + 1..];
+            } else {
+                pathname = remaining[colon_idx + 1..].to_string();
+                return (
+                    protocol,
+                    String::new(),
+                    String::new(),
+                    String::new(),
+                    pathname,
+                    search,
+                    hash,
+                );
+            }
         }
     }
 
+    let special = is_special_url_scheme(&protocol);
+    if special && protocol != "file:" {
+        // A special scheme skips the whole separator run. `file:` keeps its
+        // extra separators as path (`file:////x` is the path `//x`).
+        remaining = remaining.trim_start_matches('/');
+    }
+
     // For file: URLs, the rest is the pathname — unless an authority was
     // present and its first segment is a non-empty host (`file://host/path`,
     // i.e. a UNC share). `file:///path` / `file:/path` are hostless. #2975
@@ -188,9 +227,83 @@ pub(crate) fn parse_url(url_str: &str) -> (String, String, String, String, Strin
         }
     }
 
+    let pathname = normalize_path(&pathname, protocol == "file:");
+
     (protocol, host, hostname, port, pathname, search, hash)
 }
 
+/// Apply the WHATWG path state to a hierarchical path: the `file:` drive-letter
+/// canonicalization, then dot-segment removal.
+pub(crate) fn normalize_path(path: &str, file_scheme: bool) -> String {
+    let drive = if file_scheme {
+        normalize_file_drive_letter(path)
+    } else {
+        None
+    };
+    let path = drive.as_deref().unwrap_or(path);
+    // No `.` and no `%` means no dot segment: `%2e` has no literal dot.
+    if !path.starts_with('/') || !path.contains(['.', '%']) {
+        return path.to_string();
+    }
+
+    let parts: Vec<&str> = path[1..].split('/').collect();
+    let final_index = parts.len() - 1;
+    let mut segments: Vec<&str> = Vec::with_capacity(parts.len());
+    for (index, part) in parts.iter().enumerate() {
+        let is_final = index == final_index;
+        if is_double_dot_segment(part) {
+            let keeps_drive = file_scheme
+                && segments.len() == 1
+                && is_normalized_windows_drive_letter(segments[0]);
+            if !keeps_drive {
+                segments.pop();
+            }
+            // A trailing dot segment leaves an empty segment behind: `/a/..`
+            // is `/` while `/a/b/..` is `/a/`.
+            if is_final {
+                segments.push("");
+            }
+        } else if is_single_dot_segment(part) {
+            if is_final {
+                segments.push("");
+            }
+        } else {
+            segments.push(part);
+        }
+    }
+    format!("/{}", segments.join("/"))
+}
+
+fn normalize_file_drive_letter(path: &str) -> Option<String> {
+    let rest = path.strip_prefix('/')?;
+    let (first, tail) = match rest.find('/') {
+        Some(index) => (&rest[..index], &rest[index..]),
+        None => (rest, ""),
+    };
+    let bytes = first.as_bytes();
+    if bytes.len() == 2 && bytes[0].is_ascii_alphabetic() && bytes[1] == b'|' {
+        Some(format!("/{}:{}", &first[..1], tail))
+    } else {
+        None
+    }
+}
+
+fn is_single_dot_segment(segment: &str) -> bool {
+    segment == "." || segment.eq_ignore_ascii_case("%2e")
+}
+
+fn is_double_dot_segment(segment: &str) -> bool {
+    segment == ".."
+        || segment.eq_ignore_ascii_case(".%2e")
+        || segment.eq_ignore_ascii_case("%2e.")
+        || segment.eq_ignore_ascii_case("%2e%2e")
+}
+
+fn is_normalized_windows_drive_letter(segment: &str) -> bool {
+    let bytes = segment.as_bytes();
+    bytes.len() == 2 && bytes[0].is_ascii_alphabetic() && bytes[1] == b':'
+}
+
 /// Resolve a relative URL against a base URL
 pub(crate) fn resolve_url(url_str: &str, base_str: &str) -> String {
     // If url_str is already absolute, return it
@@ -360,11 +473,9 @@ pub(crate) fn create_url_object(url_string: &str) -> *mut ObjectHeader {
     // IPv4-shorthand hosts also canonicalize to dotted-quad. We reuse the
     // `url`-crate-backed `whatwg_canonicalize_host` (reparses `http://<host>/`),
     // which is the same path the hostname setter uses. Only special hierarchical
-    // schemes get host canonicalization; opaque schemes have no host.
-    let is_special = matches!(
-        protocol.as_str(),
-        "http:" | "https:" | "ws:" | "wss:" | "ftp:"
-    );
+    // schemes get host canonicalization; opaque schemes have no host, and
+    // `file:` is one of them for this purpose too (`file://SERVER/x` → `server`).
+    let is_special = is_special_url_scheme(&protocol);
     if is_special && !hostname.is_empty() {
         if let Some(canon) = super::whatwg_canonicalize_host(&hostname) {
             hostname = canon.clone();
@@ -601,25 +712,41 @@ pub(crate) fn is_valid_absolute_url(s: &str) -> bool {
     if scheme_end >= s.len() || s.as_bytes()[scheme_end] != b':' {
         return false;
     }
-    // For hierarchical schemes (http, https, ws, wss, ftp, file) WHATWG
-    // requires `//` before the authority. For others (mailto:, data:, etc.)
-    // Node accepts anything non-empty after `:`. Match that for ergonomics.
     let after_scheme = &s[scheme_end + 1..];
     let scheme = &s[..scheme_end];
-    let needs_authority = matches!(scheme, "http" | "https" | "ws" | "wss" | "ftp" | "file");
-    if needs_authority {
-        if !after_scheme.starts_with("//") {
-            return false;
+    if is_special_scheme_name(scheme) {
+        let folded;
+        let after_scheme = if after_scheme.contains('\\') {
+            folded = after_scheme.replace('\\', "/");
+            folded.as_str()
+        } else {
+            after_scheme
+        };
+        // The `//` is optional for a special scheme, and `file:` also allows a
+        // hostless path.
+        if scheme == "file" {
+            if let Some(authority) = after_scheme.strip_prefix("//") {
+                let authority = authority.split(['/', '?', '#']).next().unwrap_or_default();
+                if !is_valid_url_authority(scheme, authority) {
+                    return false;
+                }
+            }
+        } else {
+            let rest = after_scheme.trim_start_matches('/');
+            let authority = rest.split(['/', '?', '#']).next().unwrap_or_default();
+            if !is_valid_url_authority(scheme, authority) {
+                return false;
+            }
         }
-        let authority = after_scheme[2..]
-            .split(['/', '?', '#'])
-            .next()
-            .unwrap_or_default();
-        if !is_valid_url_authority(scheme, authority) {
+    } else if let Some(authority) = after_scheme.strip_prefix("//") {
+        // `\` is a forbidden host code point, so `foo://a\b` is not a URL.
+        let authority = authority.split(['/', '?', '#']).next().unwrap_or_default();
+        let host = authority
+            .rsplit_once('@')
+            .map_or(authority, |(_, host)| host);
+        if host.contains('\\') {
             return false;
         }
-    } else if after_scheme.is_empty() {
-        return false;
     }
     true
 }
@@ -646,7 +773,10 @@ pub(super) fn normalize_file_drive_authority(input: &str) -> String {
         .unwrap_or(authority_and_path.len());
     let authority = &authority_and_path[..drive_end];
     let drive_bytes = authority.as_bytes();
-    if drive_bytes.len() != 2 || !drive_bytes[0].is_ascii_alphabetic() || drive_bytes[1] != b':' {
+    if drive_bytes.len() != 2
+        || !drive_bytes[0].is_ascii_alphabetic()
+        || !matches!(drive_bytes[1], b':' | b'|')
+    {
         return input.to_string();
     }
 

crates/perry-runtime/src/url/url_class.rs

@@ -3,10 +3,10 @@
 use super::*;
 
 use super::parse::{
-    create_url_object, is_valid_absolute_url, rebuild_url_host, rebuild_url_href, resolve_url,
-    throw_invalid_url, URL_FIELD_COUNT, URL_HASH, URL_HOST, URL_HOSTNAME, URL_HREF, URL_ORIGIN,
-    URL_PASSWORD, URL_PATHNAME, URL_PORT, URL_PROTOCOL, URL_SEARCH, URL_SEARCH_PARAMS,
-    URL_USERNAME,
+    create_url_object, is_special_url_scheme, is_valid_absolute_url, normalize_path,
+    rebuild_url_host, rebuild_url_href, resolve_url, throw_invalid_url, URL_FIELD_COUNT, URL_HASH,
+    URL_HOST, URL_HOSTNAME, URL_HREF, URL_ORIGIN, URL_PASSWORD, URL_PATHNAME, URL_PORT,
+    URL_PROTOCOL, URL_SEARCH, URL_SEARCH_PARAMS, URL_USERNAME,
 };
 use super::search_params::{
     create_url_search_params_object, parse_query_string, URL_SEARCH_PARAMS_OWNER,
@@ -73,13 +73,6 @@ fn should_strip_url_ascii_whitespace(b: u8) -> bool {
     matches!(b, b'\t' | b'\n' | b'\r')
 }
 
-fn is_special_url_scheme(protocol: &str) -> bool {
-    matches!(
-        protocol,
-        "ftp:" | "file:" | "http:" | "https:" | "ws:" | "wss:"
-    )
-}
-
 fn normalize_protocol_value(raw: &str) -> Option<String> {
     let scheme = raw.strip_suffix(':').unwrap_or(raw);
     let mut chars = scheme.chars();
@@ -258,12 +251,9 @@ pub extern "C" fn js_url_get_href(url: *mut ObjectHeader) -> f64 {
     crate::object::js_object_get_field_f64(url, URL_HREF)
 }
 
-/// Issue #650: `url.pathname = value` setter. Per WHATWG, the setter
-/// normalizes a leading `/` for hierarchical schemes — but Node's actual
-/// runtime behavior on `u.pathname = "changed"` produces pathname
-/// `"/changed"` ONLY when the URL has an authority component, leaving
-/// opaque (non-hierarchical) URLs alone. We follow Node: prepend `/`
-/// when the URL has a non-empty `host`.
+/// Issue #650: `url.pathname = value` setter. Re-runs the parser's path states,
+/// so the value is folded, rooted and normalized as in construction (#12864);
+/// an opaque URL keeps whatever it is given.
 #[no_mangle]
 pub extern "C" fn js_url_set_pathname(url: *mut ObjectHeader, value: f64) {
     if url.is_null() {
@@ -272,12 +262,25 @@ pub extern "C" fn js_url_set_pathname(url: *mut ObjectHeader, value: f64) {
     let raw = coerce_url_setter_value(value);
     unsafe {
         let host = get_string_content(crate::object::js_object_get_field_f64(url, URL_HOST));
-        let normalized = if !host.is_empty() && !raw.starts_with('/') {
+        let protocol =
+            get_string_content(crate::object::js_object_get_field_f64(url, URL_PROTOCOL));
+        let hierarchical = !host.is_empty() || protocol == "file:";
+        let raw = if is_special_url_scheme(&protocol) {
+            raw.replace('\\', "/")
+        } else {
+            raw
+        };
+        let normalized = if hierarchical && !raw.starts_with('/') {
             format!("/{}", raw)
         } else {
             raw
         };
         let encoded = percent_encode_path(&normalized);
+        let encoded = if hierarchical {
+            normalize_path(&encoded, protocol == "file:")
+        } else {
+            encoded
+        };
         js_object_set_field_f64(url, URL_PATHNAME, create_string_f64(&encoded));
         rebuild_url_href(url);
     }