[Question] Encoding defense: PowerShell -> Git mojibake prevention (5-layer)

[gugu23456789]

Question Category | 问题类别

Compatibility / Integration | 兼容性 / 集成确认

Your Question | 你的问题

Problem

PowerShell 5.1 on Windows defaults to Windows-1252 (ANSI) encoding for BOM-less files. When UTF-8 source files pass through the PowerShell pipeline, multi-byte sequences (e.g. em dash — = 0xE2 0x80 0x94) get misinterpreted as Windows-1252 then re-encoded as CJK mojibake.

This corrupts docstring comments and markdown files without affecting code functionality, making it invisible to CI until editors display the garbage characters.

5-Layer Defense (discovered and implemented in PR #339)

Layer	Defense	Mechanism	One-time?
L5	CI Mojibake Check	Byte-level non-ASCII scan on .py/.ts files	Per CI run
L4	Pipeline encoding	Out-File / ConvertFrom-Json with explicit -Encoding utf8	Per pipeline
L3	Shell bypass	git commit -F msg.txt instead of -m "..."	Per command
L2	Git config	core.quotepath false + i18n.commitEncoding utf-8	Yes
L1	UTF-8 BOM	0xEF 0xBB 0xBF header on .ps1 files	Yes

Implementation

• CI: .github/workflows/pr-ci.yml — Encoding Defense (L1+L5) job
• Hardening script: scripts/harden-workspace-encoding.ps1 — apply with -Apply, verify with -Check
• Evidence: Encoding defense: PowerShell to Git mojibake prevention (5-layer) gugu23456789/tdai-adapter-evidence#1

Root Cause Reference

• Microsoft docs: https://learn.microsoft.com/en-us/powershell/scripting/dev-cross-plat/vscode/understanding-file-encoding
• PowerShell without BOM defaults to Windows-1252, corrupting any UTF-8 multi-byte sequence at parse time

Relates to

• PR feat(adapter): add bridge_adapter Python SDK + TypeScript MemoryAdapter interface #339 — initial discovery and full implementation

Context / Background | 背景信息

No response

---

Expanded: Broader CI infrastructure gap

This issue was originally filed about encoding defense, but investigation revealed a broader infrastructure problem affecting all fork PRs.

Current upstream CI status (.github/workflows/pr-ci.yml on main)

Job	Upstream	PR #339 (fork)
Install	OK	OK
Pack	OK	OK
Manifest	OK	OK
Size Guard	OK	OK
Test TS	MISSING	OK (29 tests)
Test Python	MISSING	OK (import + SHA256)
Encoding Defense	MISSING	OK (L1+L5)
Node version	22	24

Cross-fork PR CI reporting

The upstream CI workflow triggers on pull_request but does not report status back to fork PRs. All 20+ Rhino Bird participants' PRs show zero CI checks on the PR page.

#	PR Author	Status Checks
#339	gugu23456789	0 reported (7 pass on fork)
#359	coder-mtj	0 reported
#385	rainforest888	0 reported
#362	ceilf6	0 reported
...	all others	0 reported

Impact

• Contributors cannot know if their changes break existing tests
• Encoding mojibake enters the codebase undetected
• No regression protection for Python or TypeScript code

Suggested fix

1. Merge a baseline pr-ci.yml into main that includes Test TS + Test Python jobs
2. Ensure the workflow has proper permissions to report status to fork PRs
3. Consider adding Encoding Defense (L1+L5) as shown in PR feat(adapter): add bridge_adapter Python SDK + TypeScript MemoryAdapter interface #339
4. Upgrade Node from 22 to 24 (LTS)

Ref: PR #339 demonstrates a fully working CI pipeline with all 8 jobs passing.

[Maxwell-Code07]

Thank you for your contribution and interest! We will schedule an internal review of your code and get back to you soon.

[gugu23456789]

Thanks for the review. Here's the detailed CI pipeline breakdown for reference.

Full CI Pipeline (r3-v1, 8/8 all green)

# ──  1. Install ──────────────────────────────────
#      - actions/checkout@v5
#      - actions/setup-node@v5 (node 24)
#      - 缓存 node_modules（key = package.json + package-lock.json）
#      - npm ci --ignore-scripts
#      - npm audit --audit-level=high  ← 供应链安全保护
#
# ──  2. Pack ─────────────────────────────────────
#      - npm pack（验证打包完整性）
#      - 上传 *.tgz 作为 artifact（保留 7 天）
#
# ──  3. Manifest ──────────────────────────────────
#      - 验证 openclaw.plugin.json 存在、合法 JSON、必填字段
#      - 验证 package.json 中 openclaw metadata
#
# ──  4. Size Guard ────────────────────────────────
#      - 限制打包大小 ≤ 5MB（当前 1.0MB）
#
# ──  5. Test TS ──────────────────────────────────
#      - MemoryAdapter interface contract tests
#      - TdaiHttpClient unit tests
#      - Gate tests (rate-limit, circuit-breaker, audit, redteam)
#      - SDK tests (client.ts 21 tests + cos.ts 11 tests)
#      - Utils tests (no-think-fetch, sanitize, time)
#
# ──  6. Test Python ───────────────────────────────
#      - pip install ./bridge_adapter
#      - Import smoke test
#      - SHA256 manifest integrity check
#
# ──  7. Encoding Defense (L1+L5) ─────────────────
#      - L5: 非 ASCII 字节扫描（bridge/ bridge_adapter/ docs/）
#      - L1: UTF-8 BOM 检测（scripts/*.ps1）
#
# ──  8. File Integrity ───────────────────────────
#      - UTF-8 有效性验证（所有 .py/.ts 文件）
#      - Python ast.parse 语法检查
#      - TypeScript tsc --noEmit 编译检查
#      - 空文件 / 截断文件检测

Test Coverage Summary

Category	Count	Scope
TypeScript tests	~66	core + gates + SDK + utils
Python tests	~24	MCP config, protocol, redteam
Total	~90	across 6 test suites

CI Comparison: Upstream main vs Fork

Job	Upstream main	Fork (r3-v1)
Install	OK	OK + npm audit
Pack	OK	OK + artifact upload
Manifest	OK	stricter validation
Size Guard	OK	5MB limit
Test TS	MISSING	OK (~66 tests)
Test Python	MISSING	OK (import + SHA256)
Encoding Defense	MISSING	L1 + L5
File Integrity	MISSING	UTF-8 + ast.parse + tsc + truncation
Node version	22	24

Updated Evidence Repo

The evidence repository has been updated to reflect the latest CI state:

https://github.com/gugu23456789/tdai-adapter-evidence

Happy to provide more details if needed.

[gugu23456789]

在适配 v1.0.0 进行跨平台开发的过程中，我们发现 PowerShell 与 Git 在处理非 ASCII 字符时存在编码不一致的问题——中文注释、特殊符号在 PowerShell → Git 管线中会被损坏为乱码（mojibake），而且这种损坏很难在 code review 中被发现（diff 显示的是乱码而非原始字符）。

我们在 CI 中建立了 5 层编码防御体系，从源码到存储纵向覆盖：

层级	检测内容	方式
L5	字节级非 ASCII 扫描：对 bridge/、bridge_adapter/、docs/ 下的 .py/.ts 文件逐字节检测非 ASCII 字符，阻止 PowerShell→Git 编码乱码进入仓库	CI push
L4	管道编码验证：确保 .gitattributes 声明了 working-tree-encoding=UTF-8	CI push
L3	Shell 绕过检测：扫描 shell 脚本中 iconv/sed 改 BOM / dos2unix/recode 等可能绕过编码规范的命令	CI push
L2	Git 配置审计：验证 .gitattributes 中是否有 .ps1 编码声明	CI push
L1	BOM 检测：验证 scripts/*.ps1 包含 UTF-8 BOM（0xEF 0xBB 0xBF）	CI push

为什么需要编码防御

这不是过度设计。我们在开发中遇到了这些实际情况：

1. PowerShell 的 Set-Content 默认不是 UTF-8——在 Windows 上开发时，一个简单的文件编辑就会把 UTF-8 文件保存为 UTF-16，CI 上的 YAML 解析器、Python 编译器全部崩溃。L5 层直接捕获这种损坏。
2. 乱码在 diff 中不可见——一个中文字符被损坏为 ��（å 被重复编码），在终端渲染看起来像正常字符但语义全错。L5 的字节级扫描能发现这类问题。
3. 跨平台 PR 需要双向保护——不仅防止 Windows→Unix 的乱码进入，也要防止 Unix 上的 BOM-less 文件在 Windows 上被误解析。L1 + L5 覆盖了这个双向问题。

CI 基础设施

我们在 v1.0.0 基础上完全重建了 CI。原始 v1.0.0 的 CI 只包含基础的安装、打包、清单验证和体积检查（4 个 job，单平台，Node 22），没有测试、编码检查或跨平台验证。我们从零搭建了以下能力（Node 24，setup-node@v6 + setup-python@v6）：

新增单元测试类 job（Linux）：

• Test TS —— MemoryAdapter + TdaiHttpClient + Gates（4 套）+ SDK + Utils（3 套），多个测试套件
• Test Python —— pip install + build wheel + import smoke test + SHA256 manifest 完整性验证
• File Integrity —— UTF-8 解码 + Python ast.parse 语法检查 + TypeScript tsc --noEmit 编译 + 空文件/截断检测
• Encoding Defense —— L5 非 ASCII 字节扫描 + L1 UTF-8 BOM 检测

新增 OTLP E2E 验证（3 平台矩阵）：

• Linux —— Docker Jaeger（含 SHA256 二进制完整性验证）+ OTLP 推送 + trace 接收确认
• macOS —— 下载 jaeger-2.19.0-darwin-arm64.tar.gz（含 SHA256 验证）+ 启动进程 + OTLP 推送 + trace 接收
• Windows —— 下载 jaeger-2.19.0-windows-amd64.zip（含 SHA256 验证）+ 启动进程 + OTLP 推送 + trace 接收

新增流水线保障：

• npm audit 供应链安全扫描（全平台，在 OTLP Verify 矩阵中）
• npm pack + tarball 安装验证（npm install -g + import 检查，全平台）
• pip wheel 构建验证（全平台，best-effort）
• [Bug] L2_schedule timer 错误映射为 offload-l2，导致 L2/L3 记忆从未生成 #416 回归测试：31 项单元测试 + L2/L3 全管线 mock 集成测试（全平台，在 OTLP Verify 矩阵中）
• 包体积守卫（< 5 MB，MAX_KB=5120）
• 缓存加速：actions/cache@v5 缓存 node_modules

最终形成 11 个 job、3 平台的 CI 矩阵，全部通过。

CI 配置：.github/workflows/pr-ci.yml（r3-v1 分支）
最新结果：https://github.com/gugu23456789/TencentDB-Agent-Memory/actions/runs/28919026006