Duplicate response headers are overwritten instead of merged per RFC 7230

[umishra1504]

Summary

When a server returns multiple HTTP headers with the same name, only the last value is preserved. According to RFC 7230, duplicate headers should be merged into a comma-separated string (e.g., "v1, v2"), but instead previous values are completely overwritten.

Environment

• pyodide-http version: latest
• Pyodide version: (as used in Langchain sandbox)

Reproduction

Server Response (multiple headers with same name)

HTTP/1.1 200 OK
X-Custom-Header: value1
X-Custom-Header: value2
X-Custom-Header: value3
Set-Cookie: cookie1=abc; Path=/
Set-Cookie: cookie2=def; Path=/
Set-Cookie: cookie3=xyz; Path=/
Content-Length: 0

Python Code

import requests

response = requests.get("https://example.com/api/endpoint")

# Test with custom header
print(response.headers.get("X-Custom-Header"))

# Test with Set-Cookie
print(response.headers.get("Set-Cookie"))

Expected Output (per RFC 7230)

Duplicate headers should be merged with comma separator:

response.headers.get("X-Custom-Header")
# "value1, value2, value3"

response.headers.get("Set-Cookie")
# "cookie1=abc; Path=/, cookie2=def; Path=/, cookie3=xyz; Path=/"

Actual Output

Only the last header value is preserved:

response.headers.get("X-Custom-Header")
# "value3"  ❌ Only last value!

response.headers.get("Set-Cookie")
# "cookie3=xyz; Path=/"  ❌ Only last cookie!

Expected Behavior (per RFC 7230)

RFC 7230 Section 3.2.2 states:

A recipient MAY combine multiple header fields with the same field name into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field value to the combined field value in order, separated by a comma.

Suggested Fix

Option 1: Merge duplicate headers with comma separator

headers = {}
for name, value in js_headers:
    if name in headers:
        headers[name] = headers[name] + ", " + value  # ✅ Merge with comma
    else:
        headers[name] = value

Option 2: Use urllib3's HTTPHeaderDict

from urllib3._collections import HTTPHeaderDict

headers = HTTPHeaderDict()
for name, value in js_headers:
    headers.add(name, value)  # ✅ Supports getlist() for duplicates

Option 3: Special handling for Set-Cookie

For Set-Cookie specifically, modern browsers provide headers.getSetCookie():

// JavaScript - returns array of all Set-Cookie values
const setCookies = response.headers.getSetCookie();

This could be used to properly populate response.cookies in Python.

Impact

This bug breaks compatibility with:

1. Any API returning multiple headers with the same name

2. Authentication systems returning multiple Set-Cookie headers, including:

   • OpenText ALM (Application Lifecycle Management)
   • Enterprise SSO systems
   • OAuth implementations
   • Any service using multiple session/auth cookies

3. APIs using multiple values for:

   • Link headers (pagination)
   • WWW-Authenticate headers
   • Custom headers with multiple values

Current Workaround

There is no workaround from within the Pyodide environment. For cookie-based authentication, users must:

1. Authenticate externally (using local Python or other tools)
2. Manually extract all cookies
3. Pass pre-authenticated cookies as hardcoded values

This is impractical for production applications.

Test Case

def test_duplicate_headers_merged():
    """Duplicate headers should be merged per RFC 7230."""
    import requests
    
    # Assuming a test endpoint that returns duplicate headers
    response = requests.get("https://httpbin.org/response-headers", params={
        "X-Test": ["value1", "value2"]
    })
    
    # Should be comma-merged, not overwritten
    assert response.headers.get("X-Test") == "value1, value2"

References

• RFC 7230 Section 3.2.2 - Field Order
• MDN: Headers.getSetCookie()
• Python requests: CaseInsensitiveDict
• urllib3 HTTPHeaderDict