diff --git a/docs/designs/vpto-tied-copy-materialization-design-zh.md b/docs/designs/vpto-tied-copy-materialization-design-zh.md new file mode 100644 index 0000000000..51db5c104d --- /dev/null +++ b/docs/designs/vpto-tied-copy-materialization-design-zh.md @@ -0,0 +1,406 @@ +# VPTO tied-operand copy 物化设计 + +## 1. 背景 + +VPTO 使用 SSA 表达 vector value,但部分 A5 vector 指令在硬件层属于 two-address instruction。以 `pto.vmula` 和 `pto.vmadd` 为例,其 accumulator 输入与结果输出具有 tied-operand constraint,必须分配到同一个物理寄存器;指令执行后,该寄存器中的旧值被结果覆盖。 + +当多个具有 tied-operand constraint 的指令共享同一个 tied operand 时,SSA 仍允许它们独立读取同一个 value: + +```mlir +%r0 = pto.vmula %acc, %x0, %y, %mask : ... +%r1 = pto.vmula %acc, %x1, %y, %mask : ... +%r2 = pto.vmula %acc, %x2, %y, %mask : ... +``` + +LLVM lowering 之后,Bisheng 必须为其中一部分 use 插入真实 VMOV,保证每条破坏性指令得到可独立更新的寄存器副本。当前 PTOAS scheduler 在 VPTO IR 上工作,因而看不到这些后端新增指令,造成以下偏差: + +- 调度 DAG 缺少实际存在的 VMOV node 和依赖; +- target resource 和 latency 模型低估实际 vector pipe 工作量; +- register-pressure 模型看不到 VMOV 产生的新物理寄存器; +- PTOAS 与 Bisheng 的 scheduler 实际面对不同的指令序列。 + +Issue #1327 的 precise FP32 division 修正链中,同一个 `%55` 分别作为三条 `pto.vmula` 的 accumulator,是该问题的直接实例。 + +## 2. 目标与非目标 + +### 2.1 目标 + +本设计引入一个独立于 scheduler 实现的 VPTO pass,在 emission-ready VPTO 上: + +1. 显式表达完整物理 vector-register copy; +2. 识别具有 tied-operand constraint 的 operation; +3. 在共享 tied operand 需要独立寄存器时插入 `pto.vmov`; +4. 保证通用 canonicalize、CSE 和 DCE 不会删除或合并已物化的 VMOV; +5. 为后续 scheduler 提供完整、稳定的 operation 视野。 + +设计还必须允许 `vpto-mov` 先于 `vpto-sched-2` 合入。物化 pass 的正确性和测试不依赖 scheduler 的 DAG、strategy、region 或 pressure tracker。 + +### 2.2 非目标 + +本阶段不负责: + +- 在 `vpto-mov` 中实现或修改 scheduler 的选点策略; +- 改变现有 canonicalize、CSE、reduction combine 等 pass 的相对顺序; +- 依赖 Bisheng 后端继续为 tied-operand constraint 隐式插入 copy; +- 为所有未来可能具有 tied-operand constraint 的指令一次性建模; +- 跨 block 或跨控制流路径求最少 VMOV; +- 将 `pto.vmov` 暴露为 PTODSL/VMI 的通用用户 API。 + +首期覆盖已确认具有该约束的 `pto.vmula`、`pto.vmadd`、`pto.vaxpy`、`pto.chistv2`、`pto.dhistv2` 和 `pto.vusqz`。新增指令必须先补齐 tied-operand 接口和目标语义,不能仅在 pass 中按 operation name 特判。 + +### 2.3 指令审计 + +不能仅凭 operation 是否有名为 `acc` 的 operand 判断 tied-operand constraint。本设计以 A5 emitter 选择的 intrinsic form 和参数位置为主要依据:使用 merging-form 的指令由第一个向量参数提供 destination/merge carrier,结果会复用并覆盖该参数对应的物理寄存器。历史 `pto.vmov` emitter 会把同一个 input 同时传入 merging operand 和 source operand,也印证了这一调用约定。 + +当前 VPTO 指令面的审计结果如下: + +| operation | tied operand | tied result | 判定依据 | +| --- | --- | --- | --- | +| `pto.vmula` | `acc`(operand 0) | `result`(result 0) | multiply-accumulate 的 accumulator 是 read-modify-write destination,lowering 使用 merging-form | +| `pto.vmadd` | `acc`(operand 0) | `result`(result 0) | `%acc` 是 destination-as-source multiplicand,lowering 使用 merging-form | +| `pto.vaxpy` | `src1`(operand 1) | `result`(result 0) | 硬件契约为 `dst = alpha * src + dst`,emitter 将 `%src1` 调整到第一个 destination/merge 参数 | +| `pto.chistv2` | `acc`(operand 0) | `result`(result 0) | A5 cumulative histogram 语义原地更新 destination,lowering 使用 merging-form | +| `pto.dhistv2` | `acc`(operand 0) | `result`(result 0) | A5 frequency histogram 语义原地更新 destination,lowering 使用 merging-form | +| `pto.vusqz` | `src`(operand 0) | `result`(result 0) | `%src` 在 VPTO 语义中是 carrier,lowering 将其作为 merging destination,指令执行后该物理寄存器被结果覆盖 | + +### 2.4 `vpto-mov` PR 的边界 + +`vpto-mov` 提供: + +- `pto.vmov` IR 定义、verifier、文档和两套 VPTO LLVM emitter lowering; +- tied-operand operation interface; +- 与 scheduler 无关的 physical-view root 查询工具; +- copy 物化 pass 及独立 lit 测试; +- 手写 `pto.vmov` 的 parse/verify/emission 测试。 + +该 PR 注册 pass,使其可由 `pto-test-opt` 独立运行,但不在默认 `ptoas` pipeline 中自动启用。这样它可以在 active scheduler 合入前独立评审,同时不改变当前默认产物。 + +## 3. `pto.vmov` IR 契约 + +### 3.1 语法 + +新增无 mask、单输入、单结果的物理复制 operation: + +```mlir +%copy = pto.vmov %input + : !pto.vreg -> !pto.vreg +``` + +verifier 要求: + +- input 和 result 都是 `!pto.vreg<...>`; +- input 与 result 类型完全相同; +- 类型是目标支持的单个物理 vector register 表示。 + +首期不支持 mask operand、跨类型 copy、predicate-register copy 或多寄存器 aggregate。跨类型的同一物理值 view 继续由 `pto.vbitcast` 表达。 + +### 3.2 语义 + +`pto.vmov` 表示一次必须保留的完整物理寄存器复制: + +- 读取 `%input` 对应的全部物理 lane; +- 产生内容相同、但可由后续 destructive consumer 独立覆盖的物理寄存器; +- 在硬件上产生一条真实 VMOV; +- 不读写普通 memory; +- 在 A5 上占用 vector pipeline。 + +该 operation 不带 mask。即使后续 tied operation 是 masked form,也必须先复制完整 tied operand,保证该指令执行前获得内容完整且可独立覆盖的物理寄存器。历史上曾存在带 mask 的 `pto.vmov`,其 emitter 使用 merging form;该形式混合了“条件更新 destination”与“创建独立物理副本”两种语义,不能直接恢复为本 pass 的物化 primitive。 + +两套 emitter 应分别选择各自 CANN 版本的 full-register VMOV intrinsic。预期命名族为: + +| emitter | 预期 intrinsic family | +| --- | --- | +| legacy / CANN 9.0.0-beta.1 | `llvm.hivm.vmov.v` | +| CANN 9.0.0 | `llvm.hivm.vmov.x.v` | + +实现时必须用对应工具链头文件或最小编译用例确认最终 spelling、prototype 和支持类型;不能复用历史 `.m` merging form 的调用约定。 + +### 3.3 不可折叠性 + +以下两条 VMOV 即使 source 相同,也代表两个不同的物理副本: + +```mlir +%copy0 = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +%copy1 = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +``` + +因此 `pto.vmov` 不得带 `Pure`,不得提供 fold/canonicalization 将其替换为 input,也不得声明为对通用变换完全无 effect。其 IR effect 契约必须使 `wouldOpBeTriviallyDead` 为 false,并使 CSE 不具备合并两个 VMOV 的依据。 + +与此同时,`VPTOSchedulingOpInterface` 必须明确报告: + +- scheduling class 为 `Schedulable`; +- ordinary `memoryBehavior` 为 `None`; +- execution pipe 为 `PIPE_V`。 + +当前实现可沿用 `mem_bar`、`sprclr` 和 ctrl-register operation 的处理方式:通用 IR 变换保守保留 operation,而 VPTO scheduling semantics 单独声明其不访问普通 memory。不得为了保护 VMOV 而移动或删除已有 CSE pass。 + +## 4. Tied-operand 接口 + +新增 operation interface,例如 `VPTOTiedOperandOpInterface`,用于描述 operation-local 的物理约束。首期每个实现者只支持一组 tied operand/result pair,接口至少提供: + +- tied operand index; +- tied result index。 + +首期登记关系为: + +| operation | tied operand | tied result | +| --- | --- | --- | +| `pto.vmula` | `acc`(operand 0) | `result`(result 0) | +| `pto.vmadd` | `acc`(operand 0) | `result`(result 0) | +| `pto.vaxpy` | `src1`(operand 1) | `result`(result 0) | +| `pto.chistv2` | `acc`(operand 0) | `result`(result 0) | +| `pto.dhistv2` | `acc`(operand 0) | `result`(result 0) | +| `pto.vusqz` | `src`(operand 0) | `result`(result 0) | + +接口只陈述“结果必须复用哪个输入寄存器”的事实,不决定: + +- 是否需要插 VMOV; +- 哪一个 destructive tied use 成为 owner; +- scheduler 应生成哪种 edge; +- operation 在何处调度。 + +如果未来一个 operation 有多组 tied operand/result pair,应扩展接口返回 pair 列表,而不是继续增加按 operation name 的分支。 + +## 5. Physical view 等价关系 + +本节只处理 copy 物化分析所需的物理寄存器同一性,不负责 `pto.vbitcast`/`pto.pbitcast` 自身的调度成本、latency 或 pressure 建模。当 tied operand 经过零开销 view 时,多个不同 SSA value 仍可能引用同一个物理寄存器;pass 必须识别这一关系,避免把它们错误地分别选为 owner。 + +`vpto-mov` 新增位于 IR/通用 transform support 层的查询工具,例如: + +```text +getPhysicalRegisterViewRoot(value) +``` + +它只沿已确认的零开销 view 向上查找,并使用 visited set 防止异常 IR 形成循环。首期支持: + +- `pto.vbitcast`; +- `pto.pbitcast`(供统一工具使用,尽管首期不物化 predicate copy)。 + +该工具不把以下关系混为 view: + +- `pto.vmov` input/result:它们是两个不同物理寄存器; +- tied operand/result:二者在指令执行点复用寄存器,但表示不同 SSA version; +- 普通 elementwise producer/result。 + +该工具本身不引用 scheduler 类型。`vpto-sched-2` 是否复用该工具属于 scheduler PR 的实现选择,不作为 `vpto-mov` 的合入依赖。 + +## 6. Copy 物化 pass + +### 6.1 名称和输入契约 + +建议 pass 名称: + +```text +pto-vpto-materialize-tied-operand-copies +``` + +C++ factory 建议为: + +```text +createVPTOMaterializeTiedOperandCopiesPass() +``` + +pass 以 `func::FuncOp` 为根运行,遍历 emission-ready `pto.vecscope` 和 `pto.strict_vecscope` 内的 block。它没有 scheduler mode,不读取 scheduler analysis,也不要求 scheduler 已经运行。 + +首期目标语义限定为 A5。独立调用时,缺失或不匹配的 target arch 应给出明确诊断,而不是静默生成目标不支持的 VMOV。 + +### 6.2 基本算法 + +对每个 block 执行以下步骤: + +1. 收集实现 tied-operand 接口的 operation; +2. 对每个 tied operand 求 physical-view root; +3. 收集该 root 及其零开销 view 的所有 material use; +4. 判断是否存在可以直接更新原寄存器的 owner; +5. 为其余 destructive tied use 在对应 operation 前插入 `pto.vmov`; +6. 仅将该 use 的 tied operand 替换为 VMOV result。 + +material use 指真实读取或破坏物理寄存器的 operation。`pto.vbitcast`/`pto.pbitcast` 本身只传播 view,不作为 material use;它们的下游 use 继续计入 root。 + +### 6.3 Owner 选择 + +本文将 tied operand 对应物理寄存器会被结果覆盖的使用点称为 destructive tied use;将直接复用并覆盖原寄存器的 destructive tied use 称为 original-register owner,后文简称 owner。一个 destructive tied use 只有同时满足以下条件才能成为 owner: + +- root 的所有 use 和 view use 都可在同一 block 内完整分析; +- root 不 live-out 到其他 block、region 或函数边界; +- 按 block 原始顺序,该 destructive tied use 是 root 的最后一个 material use。 + +满足条件时,最后一个 destructive tied use 直接使用原 operand,其他 destructive tied use 各得到一个 VMOV。若共有 `N` 个 destructive tied use 且不存在其他更晚 use,物化 `N - 1` 条 VMOV: + +```mlir +%copy0 = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +%r0 = pto.vmula %copy0, %x0, %y, %mask : ... + +%copy1 = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +%r1 = pto.vmula %copy1, %x1, %y, %mask : ... + +// 最后一个 material use,可以覆盖原寄存器。 +%r2 = pto.vmula %acc, %x2, %y, %mask : ... +``` + +若最后一个 material use 是普通 read,或存在 cross-block/live-out use,则没有安全 owner,所有 destructive tied use 都必须复制: + +```mlir +%copy0 = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +%r0 = pto.vmula %copy0, %x0, %y, %mask : ... +%copy1 = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +%r1 = pto.vmula %copy1, %x1, %y, %mask : ... +%later = pto.vabs %acc, %mask : ... +``` + +这种保守策略可能比全局最优方案多生成 VMOV,但不依赖控制流调度分析,并保证 `vpto-mov` 可以独立正确工作。跨 block 的最小化不在首期范围内。 + +### 6.4 插入位置 + +每条 VMOV 紧邻对应 destructive consumer 之前插入。这样即使没有 scheduler: + +- 原 block 顺序仍满足所有 read-before-destructive-write 关系; +- VMOV result 到 destructive consumer 存在直接 SSA data dependency; +- pass 输出可直接进入 emitter。 + +后续 scheduler 可以将 VMOV 提前以隐藏 latency,但必须遵守第 8 节定义的新增物理依赖。 + +### 6.5 幂等性 + +pass 必须幂等: + +- pass 自己插入的单 use VMOV result 再次运行时不产生额外 copy; +- 用户手写 VMOV 的 result 作为新的 physical root 分析; +- 若一个手写 VMOV result 又被多个 destructive tied use 共享,仍按普通 root 规则物化; +- 第二次运行后的 IR 与第一次相同。 + +不得通过临时 attribute 标记“已处理”;结构化 IR 本身应足以判断。 + +## 7. Pipeline 位置 + +本设计不调整任何已有 pass 的相对顺序。scheduler 集成后的目标位置为: + +```text +... soft-lib expansion / inline + -> canonicalize + -> CSE + -> VPTOMaterializeTiedOperandCopiesPass + -> VPTOSchedulerPass + -> VPTOCombineReductionsPass + -> CSE + -> PTOValidateVPTOEmissionIR +``` + +选择该位置的原因是: + +- 所有可能新增首期 tied operation 的 lowering 已完成; +- 物化之前仍可正常优化 SSA; +- scheduler 能看到完整 VMOV; +- 物化之后的 CSE 不会合并 VMOV,因为不可折叠性由 operation 自身保证。 + +`vpto-mov` PR 只注册独立 pass,不进行上述 driver 接线。`vpto-sched-2` PR 完成接线。pass 本身不区分 `off`、`analyze` 或 `on`;具体模式是否组合该 pass 是 driver 的 pipeline policy,不进入物化算法。 + +## 8. Scheduler 后续适配契约 + +本节定义 `vpto-sched-2` 必须消费的事实,但不属于 `vpto-mov` 的实现范围。 + +### 8.1 DAG 依赖 + +VMOV 到其 destructive consumer 已由 SSA 建立 data edge。除此之外,当某个 destructive tied use 被选为 original-register owner 时,scheduler 必须保证 owner 在该 root 的所有其他 material read 之后执行,包括: + +- 从原 root 创建其他副本的 VMOV; +- 读取原 root 的普通 vector operation; +- 通过 `pto.vbitcast` view 读取同一物理 root 的 operation。 + +这些关系应表现为 `Anti/Must` edge,reason 应稳定标识 original-register owner,便于 trace 和测试。不能只依赖原始文本顺序,否则 scheduler 可能把 owner 提前并覆盖仍需复制或读取的 tied operand。 + +### 8.2 Resource 和 latency + +`pto.vmov` 是真实 vector micro-op,不能按零 micro-op 或零 latency 建模。首版可以使用 A5 vector-move 参数;最终 write latency 和 resource occupancy 必须用 CA trace 或工具链资料验证。现有文档中的 9-cycle `RV_VLD` proxy 只能作为待验证初值,不能直接当作 VMOV 实测结论。 + +### 8.3 Register pressure + +- `pto.vmov` result 是新的 vector-register pressure unit; +- VMOV input 的 live range 至少延续到该 VMOV; +- tied operand/result 应按寄存器复用约束处理,不能同时重复计为两个独立寄存器; +- scheduler 提前 VMOV 时必须计入被拉长的 copy-result live range。 + +### 8.4 Schedule verification + +调度结果校验至少检查: + +- 每条 VMOV 在其 destructive consumer 之前; +- owner 在同 root 的其他 material read 之后; +- view chain 不绕过 physical-root 依赖; +- schedule apply 后 tied operand/result 约束仍成立。 + +## 9. 错误处理和保守边界 + +遇到以下情况时,pass 不应猜测最优 owner: + +- tied operand 的 view/use 穿过 block 或嵌套 region; +- use 属于无法识别的 region control flow; +- operand 或 result 不是支持的单物理 `!pto.vreg`; +- tied-operand interface 返回越界或不匹配的 operand/result; +- target 不支持所需 full-register VMOV。 + +对于可以通过“给所有 destructive tied use 插 copy”保证正确的 cross-block/live-out 情形,采用保守物化。对于 IR/interface/target 契约本身非法的情形,pass 失败并给出包含 operation name 和原因的诊断,不得静默跳过。 + +## 10. 测试计划 + +### 10.1 ODS 和 verifier + +- `pto.vmov` parse/print round trip; +- input/result 类型相同的合法用例; +- 非 vreg、类型不一致和非单物理寄存器的非法用例; +- 六个首期 operation 的 tied-operand interface 查询,包括 `vaxpy` 的 operand 1 和其余 operation 的 operand 0; +- `vaddcs`/`vsubcs` 不实现 tied-operand interface; +- 当前不存在同名 VPTO operation 的 FMA/MSUB mnemonic 不进入 operation registry 或 pass name-based fallback;未来新增 operation 时验证并登记其 destination/result tied pair; + +### 10.2 物化 pass + +- 三个共享 `vmula` 生成两条 VMOV,最后一个为 owner; +- 三个共享 `vmadd` 的同类用例; +- `vmula` 与 `vmadd` 混合共享 acc; +- `vaxpy` 共享 `src1` 的用例; +- `chistv2`/`dhistv2` 共享 acc 的用例; +- `vaddcs`/`vsubcs` 构造不会被 CSE 合并的两级 carry/borrow 链,共享 vector input 时不插入 VMOV; +- `vusqz` 共享 carrier 的用例; +- 末尾存在普通 read 时所有 destructive tied use 都复制; +- live-out/cross-block 时所有 destructive tied use 都复制; +- 经一层和多层 `vbitcast` 后仍识别同一 physical root; +- 已有单 use VMOV 不重复物化; +- 手写共享 VMOV result 仍会继续物化; +- pass 连续运行两次保持相同 IR; +- CSE 在 pass 后运行时不合并两个同源 VMOV; +- DCE 不删除无 SSA user 但必须保留的 VMOV。 + +### 10.3 Emitter + +- legacy emitter 生成 full-register VMOV intrinsic; +- CANN 9.0.0 emitter 生成对应 full-register VMOV intrinsic; +- 支持类型矩阵逐项编译; +- 输出交给 Bisheng 后确实保留 VMOV,不重新退化为由后端隐式插入 copy; +- masked tied operation 前的完整复制保持原始 tied operand 内容和结果语义; +- `chistv2` 和 `dhistv2` 分别通过 A5 Bisheng 编译,并在 CA/NPU 上完成真实执行和严格 compare; + +### 10.4 Scheduler PR 后续测试 + +- DAG 包含 VMOV node、SSA edge 和 owner anti-edge; +- `vbitcast` 为零开销 view,VMOV 为真实 pressure unit; +- scheduler apply 后 emission 顺序满足 tied-operand constraint; +- issue #1327 最小 IR 中,PTOAS 与 Bisheng 看到相同 VMOV 数量; +- A5 CA/NPU 上功能 compare 通过,并记录性能相对默认路径和 Bisheng MI scheduler 的对比。 + +## 11. 验收标准 + +`vpto-mov` 可以先合入的必要条件: + +- 独立 pass 不引用 `VPTOScheduler` 目录下的类型或实现; +- 三个共享 destructive tied use 的基本用例稳定产生 `N - 1` 条 VMOV; +- cross-block/live-out 用例采用正确的保守策略; +- VMOV 经现有 canonicalize/CSE/DCE 后仍保持数量和相互独立性; +- 两套 emitter 均通过 intrinsic 编译验证; +- pass 可由 `pto-test-opt` 独立运行,默认 `ptoas` pipeline 行为不变。 + +完整功能在 `vpto-sched-2` 合入时的附加标准: + +- scheduler 输入包含全部显式 VMOV; +- DAG 和 schedule verifier 防止 owner 覆盖尚未读取的 tied operand; +- pressure tracker 区分 VMOV copy 与 bitcast view; +- issue #1327 的功能结果不变,并获得可解释、可复现的性能对比。 diff --git a/docs/isa/micro-isa/06-unary-vector-ops.md b/docs/isa/micro-isa/06-unary-vector-ops.md index 8eff4fcec3..a55bda490a 100644 --- a/docs/isa/micro-isa/06-unary-vector-ops.md +++ b/docs/isa/micro-isa/06-unary-vector-ops.md @@ -8,9 +8,11 @@ Element-wise operations that take one vector input and produce one vector output ## Common Operand Model - `%input` is the source vector register value. -- `%mask` is the predicate operand. For this family, inactive lanes follow the +- `%mask`, where present, is the predicate operand. Inactive lanes follow the predication behavior of the selected instruction form: zeroing forms zero-fill inactive lanes, while merging forms preserve the destination value. + `pto.vmov` is the exception in this group: it has no mask and copies every + lane. - `%result` is the destination vector register value. Unless stated otherwise, `%result` has the same lane count and element type as `%input`. @@ -161,6 +163,26 @@ for (int i = 0; i < N; i++) ## Movement +### `pto.vmov` + +- **syntax:** `%result = pto.vmov %input : !pto.vreg -> !pto.vreg` +- **A5 types:** ui8, si8, ui16, si16, f16, bf16, ui32, si32, f32, + si64 + +```c +for (int i = 0; i < N; i++) + dst[i] = src[i]; +``` + +- **inputs:** `%input` supplies every lane of one physical vector register. +- **outputs:** `%result` contains the same lane values in an independent + physical vector register. +- **constraints and limitations:** Source and result types MUST match and MUST + represent one full A5 vector register. This operation has no mask: it always + copies the complete register. It represents a required hardware copy and is + therefore not removed or merged with another `pto.vmov`, even when the two + operations have the same input. + ## Typical Usage ```mlir diff --git a/docs/vpto-spec.md b/docs/vpto-spec.md index 730cb0ecb8..0ff5d29077 100644 --- a/docs/vpto-spec.md +++ b/docs/vpto-spec.md @@ -1323,7 +1323,7 @@ This section provides a categorized overview of all PTO micro Instruction operat | 3 | [Vector Load/Store](isa/micro-isa/03-vector-load-store.md) | UB↔vreg data movement with various access patterns | ~23 | `pto.vlds`, `pto.vldsx2`, `pto.vgather2`, `pto.vsts`, `pto.vstsx2`, `pto.vscatter`, `pto.sprclr`, `pto.sprsti`, `pto.sprsts`, etc. | | 4 | [Predicate Load/Store](isa/micro-isa/04-predicate-load-store.md) | UB↔mask register movement | 5 | `pto.plds`, `pto.pldi`, `pto.psts`, `pto.psti`, `pto.pstu` | | 5 | [Materialization & Predicate Ops](isa/micro-isa/05-materialization-predicate.md) | Scalar broadcast, predicate generation and manipulation | ~20 | `pto.vbr`, `pto.vdup`, `pto.pset_b*`, `pto.pge_b*`, `pto.plt_b*`, `pto.pltm_b*`, `pto.ppack`, `pto.punpack`, `pto.pnot`, `pto.psel`, etc. | -| 6 | [Unary Vector Ops](isa/micro-isa/06-unary-vector-ops.md) | Single-input element-wise operations | 7 | `pto.vabs`, `pto.vneg`, `pto.vexp`, `pto.vln`, `pto.vsqrt`, `pto.vrelu`, `pto.vnot` | +| 6 | [Unary Vector Ops](isa/micro-isa/06-unary-vector-ops.md) | Single-input element-wise and register-copy operations | 8 | `pto.vabs`, `pto.vneg`, `pto.vexp`, `pto.vln`, `pto.vsqrt`, `pto.vrelu`, `pto.vnot`, `pto.vmov` | | 7 | [Binary Vector Ops](isa/micro-isa/07-binary-vector-ops.md) | Two-input element-wise operations | 14 | `pto.vadd`, `pto.vsub`, `pto.vmul`, `pto.vdiv`, `pto.vmax`, `pto.vmin`, `pto.vmadd`, `pto.vand`, `pto.vor`, `pto.vxor`, `pto.vshl`, `pto.vshr`, `pto.vaddc`, `pto.vsubc` | | 8 | [Vec-Scalar Ops](isa/micro-isa/08-vec-scalar-ops.md) | Vector-scalar operations | 9 | `pto.vadds`, `pto.vmuls`, `pto.vmaxs`, `pto.vmins`, `pto.vlrelu`, `pto.vshls`, `pto.vshrs`, `pto.vaddcs`, `pto.vsubcs` | | 9 | [Conversion Ops](isa/micro-isa/09-conversion-ops.md) | Type conversion with rounding/saturation control | 4 | `pto.vcvt`, `pto.vtrc`, `pto.vbitcast`, `pto.pbitcast` | @@ -1379,6 +1379,7 @@ This section provides a categorized overview of all PTO micro Instruction operat | Operation | Group | Description | |-----------|-------|-------------| | Type Conversion | 9 | `pto.vcvt`, `pto.vbitcast`, `pto.pbitcast` | +| Full vector-register copy | 6 | `pto.vmov` | | Interleave/Deinterleave | 12 | `pto.vintlv`, `pto.vdintlv` | | Interleave/Deinterleave (not A5) | 12 | `pto.vintlvv2`, `pto.vdintlvv2` | diff --git a/include/PTO/IR/VPTOInterfaces.td b/include/PTO/IR/VPTOInterfaces.td index 868338c123..534d5f83d9 100644 --- a/include/PTO/IR/VPTOInterfaces.td +++ b/include/PTO/IR/VPTOInterfaces.td @@ -53,6 +53,25 @@ def VPTOSchedulingOpInterface : OpInterface<"VPTOSchedulingOpInterface"> { ]; } +def VPTOTiedOperandOpInterface + : OpInterface<"VPTOTiedOperandOpInterface"> { + let description = [{ + Describes one operation-local physical register constraint where a result + must reuse and overwrite one operand's register. Copy materialization and + scheduling policy are consumers of this fact and are not part of the + interface contract. + }]; + let cppNamespace = "::mlir::pto"; + let methods = [ + InterfaceMethod< + "Return the operand index whose physical register is reused.", + "unsigned", "getTiedOperandIndex">, + InterfaceMethod< + "Return the result index tied to the operand.", + "unsigned", "getTiedResultIndex"> + ]; +} + def PTO_MadSemanticOpInterface : OpInterface<"MadSemanticOpInterface"> { let description = [{ Interface for semantic MAD-family ops. The interface is only a uniform view diff --git a/include/PTO/IR/VPTOOps.td b/include/PTO/IR/VPTOOps.td index bda2b16ab5..000d18a96e 100644 --- a/include/PTO/IR/VPTOOps.td +++ b/include/PTO/IR/VPTOOps.td @@ -2340,7 +2340,11 @@ def PTO_VcgmaxOp : PTO_UnaryVecOp<"vcgmax">; def PTO_VcgminOp : PTO_UnaryVecOp<"vcgmin">; def PTO_VcpaddOp : PTO_UnaryVecOp<"vcpadd">; -class PTO_HistogramOp : PTO_VectorMicroOp { +class PTO_HistogramOp + : PTO_VectorMicroOp + ]> { let arguments = (ins PTO_VectorType:$acc, PTO_VectorType:$source, @@ -2405,7 +2409,8 @@ def PTO_VandOp : PTO_BinaryVecOp<"vand">; def PTO_VorOp : PTO_BinaryVecOp<"vor">; def PTO_VxorOp : PTO_BinaryVecOp<"vxor">; -class PTO_TernaryVecOp : PTO_VectorMicroOp { +class PTO_TernaryVecOp traits = []> + : PTO_VectorMicroOp { let arguments = (ins PTO_VectorType:$acc, PTO_VectorType:$lhs, @@ -2421,7 +2426,9 @@ class PTO_TernaryVecOp : PTO_VectorMicroOp { }]; } -def PTO_VmaddOp : PTO_TernaryVecOp<"vmadd">; +def PTO_VmaddOp : PTO_TernaryVecOp<"vmadd", [ + DeclareOpInterfaceMethods +]>; def PTO_VaddcOp : PTO_VectorMicroOp<"vaddc", [Pure]> { let arguments = (ins @@ -2666,6 +2673,25 @@ def PTO_VbitcastOp : PTO_VectorMicroOp<"vbitcast", [Pure]> { }]; } +def PTO_VmovOp : PTO_VectorMicroOp<"vmov", []> { + let summary = "Materialized full physical vector-register copy"; + let description = [{ + Copies all lanes of one physical vector register into an independent + destination register. The operation is intentionally not Pure: two copies + of the same input represent distinct physical registers and must not be + folded, merged, or removed. + }]; + + let arguments = (ins PTO_VectorType:$input); + let results = (outs PTO_VectorType:$result); + + let hasVerifier = 1; + + let assemblyFormat = [{ + $input attr-dict `:` type($input) `->` type($result) + }]; +} + def PTO_VciOp : PTO_VectorMicroOp<"vci", [Pure]> { let arguments = (ins AnyTypeOf<[AnyInteger, AnyFloat], "integer/float scalar">:$index, @@ -3407,7 +3433,10 @@ def PTO_VselrOp : PTO_VectorMicroOp<"vselr", [Pure]> { def PTO_VsqzOp : PTO_UnaryVecOp<"vsqz">; -def PTO_VusqzOp : PTO_VectorMicroOp<"vusqz", [Pure]> { +def PTO_VusqzOp : PTO_VectorMicroOp<"vusqz", [ + Pure, + DeclareOpInterfaceMethods +]> { let arguments = (ins PTO_VectorType:$src, PTO_MaskTypeConstraint:$mask @@ -3550,7 +3579,10 @@ def PTO_VmullOp : PTO_VectorMicroOp<"vmull", [Pure]> { }]; } -def PTO_VmulaOp : PTO_VectorMicroOp<"vmula", [Pure]> { +def PTO_VmulaOp : PTO_VectorMicroOp<"vmula", [ + Pure, + DeclareOpInterfaceMethods +]> { let arguments = (ins PTO_VectorType:$acc, PTO_VectorType:$lhs, @@ -3614,7 +3646,10 @@ def PTO_VexpdifOp : PTO_VectorMicroOp<"vexpdif", [Pure]> { }]; } -def PTO_VaxpyOp : PTO_VectorMicroOp<"vaxpy", [Pure]> { +def PTO_VaxpyOp : PTO_VectorMicroOp<"vaxpy", [ + Pure, + DeclareOpInterfaceMethods +]> { let arguments = (ins PTO_VectorType:$src0, PTO_VectorType:$src1, diff --git a/include/PTO/IR/VPTOPhysicalRegister.h b/include/PTO/IR/VPTOPhysicalRegister.h new file mode 100644 index 0000000000..9acbab606b --- /dev/null +++ b/include/PTO/IR/VPTOPhysicalRegister.h @@ -0,0 +1,34 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- VPTOPhysicalRegister.h - VPTO physical register views ---*- C++ -*-===// +// +// This file defines scheduler-independent helpers for identifying SSA values +// that are zero-cost views of the same VPTO physical register. +// +//===----------------------------------------------------------------------===// + +#ifndef MLIR_DIALECT_PTO_IR_VPTOPHYSICALREGISTER_H +#define MLIR_DIALECT_PTO_IR_VPTOPHYSICALREGISTER_H + +#include "mlir/IR/Operation.h" +#include "mlir/IR/Value.h" + +namespace mlir::pto { + +/// Return true when `op` only changes the SSA type view of one physical vector +/// or predicate register. +bool isPhysicalRegisterView(Operation *op); + +/// Follow zero-cost physical-register views to their shared SSA root. A real +/// copy such as pto.vmov always starts a new root. +Value getPhysicalRegisterViewRoot(Value value); + +} // namespace mlir::pto + +#endif // MLIR_DIALECT_PTO_IR_VPTOPHYSICALREGISTER_H diff --git a/include/PTO/Transforms/Passes.h b/include/PTO/Transforms/Passes.h index b115f4a901..4c70698e83 100644 --- a/include/PTO/Transforms/Passes.h +++ b/include/PTO/Transforms/Passes.h @@ -117,6 +117,7 @@ std::unique_ptr createVPTOPtrCastCleanupPass(); std::unique_ptr createVPTOCombineReductionsPass(); std::unique_ptr createVPTOOptimizeVcvtPass(); std::unique_ptr createVPTOMaskSimplifyPass(); +std::unique_ptr createVPTOMaterializeTiedOperandCopiesPass(); std::unique_ptr createVPTOSchedulerPass(const VPTOSchedulerOptions &options = {}); LogicalResult validateVPTOAuthoringIR(ModuleOp module, diff --git a/include/PTO/Transforms/Passes.td b/include/PTO/Transforms/Passes.td index ca5e10b4f0..c9a44f1e35 100644 --- a/include/PTO/Transforms/Passes.td +++ b/include/PTO/Transforms/Passes.td @@ -1272,6 +1272,23 @@ def PTOValidateVPTOEmissionIR "mlir::scf::SCFDialect"]; } +def VPTOMaterializeTiedOperandCopies + : Pass<"pto-vpto-materialize-tied-operand-copies", "func::FuncOp"> { + let summary = "Materialize physical copies for destructive VPTO tied operands"; + let description = [{ + Finds A5 vector operations whose result must reuse one input physical + register. For each block-local physical-view root, the last destructive + material use may update the original register; all other destructive uses + receive an explicit full-register `pto.vmov` copy. Cross-block, live-out, + and later ordinary reads use the conservative all-copy form. The pass is + independent of the VPTO scheduler and is idempotent. + }]; + let constructor = + "mlir::pto::createVPTOMaterializeTiedOperandCopiesPass()"; + let dependentDialects = ["mlir::func::FuncDialect", + "mlir::pto::PTODialect"]; +} + def VPTOScheduler : Pass<"pto-vpto-scheduler", "ModuleOp"> { let summary = "Analyze emission-ready VPTO scheduling regions"; let description = [{ diff --git a/lib/PTO/IR/CMakeLists.txt b/lib/PTO/IR/CMakeLists.txt index f2c8fcce7e..707dd155f3 100644 --- a/lib/PTO/IR/CMakeLists.txt +++ b/lib/PTO/IR/CMakeLists.txt @@ -18,6 +18,7 @@ add_mlir_dialect_library(PTOIR PTO.cpp VPTO.cpp VPTOAddressSemantics.cpp + VPTOPhysicalRegister.cpp VPTOScheduling.cpp VMI.cpp VPTOUbOps.cpp diff --git a/lib/PTO/IR/VPTO.cpp b/lib/PTO/IR/VPTO.cpp index 44ace4c9aa..1251bca69b 100644 --- a/lib/PTO/IR/VPTO.cpp +++ b/lib/PTO/IR/VPTO.cpp @@ -43,6 +43,19 @@ using namespace mlir; using namespace mlir::pto; +unsigned Chistv2Op::getTiedOperandIndex() { return 0; } +unsigned Chistv2Op::getTiedResultIndex() { return 0; } +unsigned Dhistv2Op::getTiedOperandIndex() { return 0; } +unsigned Dhistv2Op::getTiedResultIndex() { return 0; } +unsigned VmaddOp::getTiedOperandIndex() { return 0; } +unsigned VmaddOp::getTiedResultIndex() { return 0; } +unsigned VusqzOp::getTiedOperandIndex() { return 0; } +unsigned VusqzOp::getTiedResultIndex() { return 0; } +unsigned VmulaOp::getTiedOperandIndex() { return 0; } +unsigned VmulaOp::getTiedResultIndex() { return 0; } +unsigned VaxpyOp::getTiedOperandIndex() { return 1; } +unsigned VaxpyOp::getTiedResultIndex() { return 0; } + static llvm::cl::opt disableVPTOAlignChainVerification( "vpto-disable-align-chain-verification", llvm::cl::desc("Disable !pto.align linear-chain verifier checks"), @@ -7073,6 +7086,41 @@ LogicalResult VbitcastOp::verify() { return success(); } +LogicalResult VmovOp::verify() { + bool invalidInput = + failed(verifyVRegTypeLike(*this, getInput().getType(), "input type")); + bool invalidResult = + failed(verifyVRegTypeLike(*this, getResult().getType(), "result type")); + if (invalidInput || invalidResult) { + return failure(); + } + Type inputType = getInput().getType(); + Type resultType = getResult().getType(); + if (inputType != resultType) { + return emitOpError( + "requires input and result to have identical vector types"); + } + + Type elementType = cast(inputType).getElementType(); + bool isSupportedFloat = + elementType.isF16() || elementType.isBF16() || elementType.isF32(); + if (isSupportedFloat) { + return success(); + } + auto integerType = dyn_cast(elementType); + if (!integerType) { + return emitOpError("requires f16/bf16/f32 or integer vector element type"); + } + unsigned width = integerType.getWidth(); + if (width == mlir::pto::kValue8 || width == 16 || + width == mlir::pto::kValue32 || + (width == 64 && !integerType.isUnsigned())) { + return success(); + } + return emitOpError("requires 8/16/32-bit integer or non-unsigned 64-bit " + "integer vector element type"); +} + LogicalResult PdintlvB8Op::verify() { if (failed(verifyMaskTypeWithGranularityLike(*this, getLhs().getType(), "lhs type", "b8")) || diff --git a/lib/PTO/IR/VPTOPhysicalRegister.cpp b/lib/PTO/IR/VPTOPhysicalRegister.cpp new file mode 100644 index 0000000000..113ab6be4d --- /dev/null +++ b/lib/PTO/IR/VPTOPhysicalRegister.cpp @@ -0,0 +1,42 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- VPTOPhysicalRegister.cpp - VPTO physical register views -----------===// + +#include "PTO/IR/VPTOPhysicalRegister.h" + +#include "PTO/IR/PTO.h" + +#include "llvm/ADT/SmallPtrSet.h" + +using namespace mlir; +using namespace mlir::pto; + +bool mlir::pto::isPhysicalRegisterView(Operation *op) { + return isa_and_nonnull(op); +} + +Value mlir::pto::getPhysicalRegisterViewRoot(Value value) { + llvm::SmallPtrSet visited; + while (auto result = dyn_cast(value)) { + Operation *owner = result.getOwner(); + if (!visited.insert(owner).second) { + break; + } + if (auto view = dyn_cast(owner)) { + value = view.getInput(); + continue; + } + if (auto view = dyn_cast(owner)) { + value = view.getInput(); + continue; + } + break; + } + return value; +} diff --git a/lib/PTO/IR/VPTOScheduling.cpp b/lib/PTO/IR/VPTOScheduling.cpp index a74c6c9419..36c14e41a9 100644 --- a/lib/PTO/IR/VPTOScheduling.cpp +++ b/lib/PTO/IR/VPTOScheduling.cpp @@ -126,7 +126,7 @@ static void setStaticAccessRange(Operation *op, VPTOMemoryAccess &access) { /// Pure: vector-memory barriers and register-state effects must remain visible /// to the scheduler and to general IR transformations. static bool hasKnownNoOrdinaryMemoryAccess(Operation *op) { - return isa(op); + return isa(op); } static void collectMemoryAccesses(Operation *op, diff --git a/lib/PTO/Transforms/CMakeLists.txt b/lib/PTO/Transforms/CMakeLists.txt index 05067d9e7c..29ca32bb8b 100644 --- a/lib/PTO/Transforms/CMakeLists.txt +++ b/lib/PTO/Transforms/CMakeLists.txt @@ -47,6 +47,7 @@ add_mlir_dialect_library(PTOTransforms VPTOCombineReductions.cpp VPTOOptimizeVcvt.cpp VPTOMaskSimplify.cpp + VPTOMaterializeTiedOperandCopies.cpp VPTOExpandWrapperOps.cpp VPTOSoftPostUpdate.cpp PTOPrintAddressAnalysis.cpp diff --git a/lib/PTO/Transforms/VPTOCANN900LLVMEmitter.cpp b/lib/PTO/Transforms/VPTOCANN900LLVMEmitter.cpp index 0019778dd1..30fba97d2b 100644 --- a/lib/PTO/Transforms/VPTOCANN900LLVMEmitter.cpp +++ b/lib/PTO/Transforms/VPTOCANN900LLVMEmitter.cpp @@ -3741,6 +3741,15 @@ static FailureOr buildVmulaCallee(MLIRContext *context, return buildCANN900SignedModeTypedCallee(context, resultType, "vmula", "m"); } +static FailureOr buildVmovCallee(MLIRContext *context, + Type resultType) { + std::string vec = getCANN900VectorTypeFragment(resultType); + if (vec.empty()) { + return failure(); + } + return StringAttr::get(context, "llvm.hivm.vmov.x." + vec).getValue(); +} + static FailureOr buildVmullCallee(MLIRContext *context, Type resultType) { return buildLaneTypedCallee(context, resultType, "vmull", ""); @@ -8445,6 +8454,42 @@ class LowerVcvtOpPattern final : public OpConversionPattern { LoweringState &state; }; +class LowerVmovOpPattern final : public OpConversionPattern { +public: + explicit LowerVmovOpPattern(TypeConverter &typeConverter, + MLIRContext *context, LoweringState &state) + : OpConversionPattern(typeConverter, context), state(state) {} + + LogicalResult + matchAndRewrite(pto::VmovOp op, pto::VmovOp::Adaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + FailureOr calleeName = + buildVmovCallee(op.getContext(), op.getResult().getType()); + if (failed(calleeName)) { + return rewriter.notifyMatchFailure(op, "unsupported vmov VPTO signature"); + } + + Type resultType = + this->getTypeConverter()->convertType(op.getResult().getType()); + Value input = adaptor.getInput(); + bool hasInvalidTypes = !resultType || input.getType() != resultType; + if (hasInvalidTypes) { + return rewriter.notifyMatchFailure(op, "failed to convert vmov types"); + } + + auto funcType = + rewriter.getFunctionType(TypeRange{resultType}, TypeRange{resultType}); + auto call = rewriter.create( + op.getLoc(), *calleeName, TypeRange{resultType}, ValueRange{input}); + state.plannedDecls.push_back(PlannedDecl{calleeName->str(), funcType}); + rewriter.replaceOp(op, call.getResults()); + return success(); + } + +private: + LoweringState &state; +}; + class LowerVbitcastOpPattern final : public OpConversionPattern { public: @@ -11329,7 +11374,8 @@ static void populateVPTOOpLoweringPatterns(VPTOTypeConverter &typeConverter, LowerVciOpPattern, LowerVexpdifOpPattern, LowerVbitsortOpPattern, LowerVmrgsort4OpPattern, LowerVtrcOpPattern, LowerVcvtOpPattern, - LowerVbitcastOpPattern, LowerPbitcastOpPattern, + LowerVmovOpPattern, LowerVbitcastOpPattern, + LowerPbitcastOpPattern, LowerPredicateLoadOpPattern, LowerPredicateLoadOpPattern, LowerPredicateStoreOpPattern, @@ -11456,7 +11502,7 @@ static void configureVPTOOpLoweringTarget(ConversionTarget &target, pto::VaxpyOp, pto::VmulscvtOp, pto::VciOp, pto::VexpdifOp, pto::VbitsortOp, pto::Vmrgsort4Op, pto::VtrcOp, pto::VcvtOp, - pto::VbitcastOp, + pto::VmovOp, pto::VbitcastOp, pto::VcmpOp, pto::VcmpsOp, pto::CopyGmToUbufOp, pto::CopyUbufToGmOp, pto::CopyUbufToUbufOp, pto::CopyCbufToUbufOp, diff --git a/lib/PTO/Transforms/VPTOLLVMEmitter.cpp b/lib/PTO/Transforms/VPTOLLVMEmitter.cpp index c3d5cbf206..33e4705974 100644 --- a/lib/PTO/Transforms/VPTOLLVMEmitter.cpp +++ b/lib/PTO/Transforms/VPTOLLVMEmitter.cpp @@ -4563,6 +4563,20 @@ static FailureOr buildVmulaCallee(MLIRContext *context, return buildLaneTypedCallee(context, resultType, "vmula", ".m"); } +static FailureOr buildVmovCallee(MLIRContext *context, + Type resultType) { + std::string element = + getElementTypeFragment(getElementTypeFromVectorLike(resultType)); + auto lanes = getElementCountFromVectorLike(resultType); + bool isUnsupportedType = element.empty() || !lanes; + if (isUnsupportedType) { + return failure(); + } + return StringAttr::get(context, "llvm.hivm.vmov.v" + + std::to_string(*lanes) + element) + .getValue(); +} + static FailureOr buildVmullCallee(MLIRContext *context, Type resultType) { return buildLaneTypedCallee(context, resultType, "vmull", ""); @@ -10478,6 +10492,42 @@ class LowerVcvtOpPattern final : public OpConversionPattern { LoweringState &state; }; +class LowerVmovOpPattern final : public OpConversionPattern { +public: + explicit LowerVmovOpPattern(TypeConverter &typeConverter, + MLIRContext *context, LoweringState &state) + : OpConversionPattern(typeConverter, context), state(state) {} + + LogicalResult + matchAndRewrite(pto::VmovOp op, pto::VmovOp::Adaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + FailureOr calleeName = + buildVmovCallee(op.getContext(), op.getResult().getType()); + if (failed(calleeName)) { + return rewriter.notifyMatchFailure(op, "unsupported vmov VPTO signature"); + } + + Type resultType = + this->getTypeConverter()->convertType(op.getResult().getType()); + Value input = adaptor.getInput(); + bool hasInvalidTypes = !resultType || input.getType() != resultType; + if (hasInvalidTypes) { + return rewriter.notifyMatchFailure(op, "failed to convert vmov types"); + } + + auto funcType = + rewriter.getFunctionType(TypeRange{resultType}, TypeRange{resultType}); + auto call = rewriter.create( + op.getLoc(), *calleeName, TypeRange{resultType}, ValueRange{input}); + state.plannedDecls.push_back(PlannedDecl{calleeName->str(), funcType}); + rewriter.replaceOp(op, call.getResults()); + return success(); + } + +private: + LoweringState &state; +}; + class LowerVbitcastOpPattern final : public OpConversionPattern { public: @@ -13535,7 +13585,8 @@ static void populateVPTOOpLoweringPatterns(VPTOTypeConverter &typeConverter, LowerVciOpPattern, LowerVexpdifOpPattern, LowerVbitsortOpPattern, LowerVmrgsort4OpPattern, LowerVtrcOpPattern, LowerVcvtOpPattern, - LowerVbitcastOpPattern, LowerPbitcastOpPattern, + LowerVmovOpPattern, LowerVbitcastOpPattern, + LowerPbitcastOpPattern, LowerPredicateLoadOpPattern, LowerPredicateLoadOpPattern, LowerPredicateStoreOpPattern, @@ -13732,7 +13783,7 @@ static void configureVPTOOpLoweringTarget(ConversionTarget &target, pto::VaxpyOp, pto::VmulscvtOp, pto::VciOp, pto::VexpdifOp, pto::VbitsortOp, pto::Vmrgsort4Op, pto::VtrcOp, pto::VcvtOp, - pto::VbitcastOp, + pto::VmovOp, pto::VbitcastOp, pto::VcmpOp, pto::VcmpsOp, pto::CopyGmToUbufOp, pto::CopyUbufToGmOp, pto::CopyUbufToUbufOp, pto::CopyCbufToUbufOp, diff --git a/lib/PTO/Transforms/VPTOMaterializeTiedOperandCopies.cpp b/lib/PTO/Transforms/VPTOMaterializeTiedOperandCopies.cpp new file mode 100644 index 0000000000..25bcef9c50 --- /dev/null +++ b/lib/PTO/Transforms/VPTOMaterializeTiedOperandCopies.cpp @@ -0,0 +1,256 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- VPTOMaterializeTiedOperandCopies.cpp - Materialize VPTO copies ----===// +// +// This pass makes physical copies explicit before destructive vector +// operations whose result must reuse one input register. It is deliberately +// independent of the VPTO scheduler. +// +//===----------------------------------------------------------------------===// + +#include "PTO/IR/PTO.h" +#include "PTO/IR/VPTOPhysicalRegister.h" +#include "PTO/Transforms/Passes.h" + +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "mlir/IR/BuiltinOps.h" +#include "mlir/Pass/Pass.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/MapVector.h" +#include "llvm/ADT/SmallPtrSet.h" +#include "llvm/ADT/SmallVector.h" + +#include + +namespace mlir { +namespace pto { +#define GEN_PASS_DEF_VPTOMATERIALIZETIEDOPERANDCOPIES +#include "PTO/Transforms/Passes.h.inc" +} // namespace pto +} // namespace mlir + +using namespace mlir; +using namespace mlir::pto; + +namespace { + +struct TiedUse { + Operation *operation = nullptr; + unsigned operandIndex = 0; +}; + +using TiedUseGroups = llvm::MapVector>; + +static StringAttr findTargetArchitecture(Operation *operation) { + for (Operation *current = operation; current; + current = current->getParentOp()) { + auto module = dyn_cast(current); + if (!module) { + continue; + } + if (auto target = + module->getAttrOfType(pto::kPTOTargetArchAttrName)) { + return target; + } + } + return {}; +} + +static bool isInsideVectorScope(Operation *operation) { + return operation->getParentOfType() || + operation->getParentOfType(); +} + +static FailureOr validateTiedUse(Operation *operation) { + auto tied = dyn_cast(operation); + if (!tied) { + return failure(); + } + + unsigned operandIndex = tied.getTiedOperandIndex(); + unsigned resultIndex = tied.getTiedResultIndex(); + if (operandIndex >= operation->getNumOperands()) { + operation->emitOpError("tied-operand interface returned operand index ") + << operandIndex << " for " << operation->getNumOperands() + << " operands"; + return failure(); + } + if (resultIndex >= operation->getNumResults()) { + operation->emitOpError("tied-operand interface returned result index ") + << resultIndex << " for " << operation->getNumResults() << " results"; + return failure(); + } + + Type operandType = operation->getOperand(operandIndex).getType(); + Type resultType = operation->getResult(resultIndex).getType(); + bool hasNonVectorType = + !isa(operandType) || !isa(resultType); + if (hasNonVectorType) { + operation->emitOpError( + "tied operand and result must both be single physical !pto.vreg values"); + return failure(); + } + if (operandType != resultType) { + operation->emitOpError( + "tied operand and result must have identical !pto.vreg types"); + return failure(); + } + return TiedUse{operation, operandIndex}; +} + +static LogicalResult collectTiedUseGroups(func::FuncOp function, + TiedUseGroups &groups) { + WalkResult result = function.walk([&](Operation *operation) { + bool isTiedOperation = isa(operation); + bool isInVectorScope = isInsideVectorScope(operation); + if (!isInVectorScope || !isTiedOperation) { + return WalkResult::advance(); + } + FailureOr tiedUse = validateTiedUse(operation); + if (failed(tiedUse)) { + return WalkResult::interrupt(); + } + Value tiedOperand = operation->getOperand(tiedUse->operandIndex); + groups[getPhysicalRegisterViewRoot(tiedOperand)].push_back(*tiedUse); + return WalkResult::advance(); + }); + return failure(result.wasInterrupted()); +} + +struct RootUseAnalysis { + bool allUsesInBlock = true; + SmallVector materialUsers; +}; + +static RootUseAnalysis analyzeRootUses(Value root, Block *expectedBlock) { + RootUseAnalysis analysis; + SmallVector worklist{root}; + llvm::DenseSet visitedValues; + llvm::SmallPtrSet visitedMaterialUsers; + + while (!worklist.empty()) { + Value value = worklist.pop_back_val(); + if (!visitedValues.insert(value).second) { + continue; + } + for (OpOperand &use : value.getUses()) { + Operation *owner = use.getOwner(); + bool isViewOperand = + isPhysicalRegisterView(owner) && use.getOperandNumber() == 0; + if (isViewOperand) { + analysis.allUsesInBlock &= owner->getBlock() == expectedBlock; + worklist.append(owner->getResults().begin(), owner->getResults().end()); + continue; + } + + analysis.allUsesInBlock &= owner->getBlock() == expectedBlock; + if (visitedMaterialUsers.insert(owner).second) { + analysis.materialUsers.push_back(owner); + } + } + } + return analysis; +} + +static Operation *findLastMaterialUser(const RootUseAnalysis &analysis, + Block *block) { + if (!analysis.allUsesInBlock) { + return nullptr; + } + Operation *last = nullptr; + for (Operation *user : analysis.materialUsers) { + Block *userBlock = user->getBlock(); + if (userBlock != block) { + return nullptr; + } + if (!last || last->isBeforeInBlock(user)) { + last = user; + } + } + return last; +} + +static std::optional chooseOwner(Value root, + ArrayRef tiedUses) { + if (tiedUses.empty()) { + return std::nullopt; + } + Block *block = tiedUses.front().operation->getBlock(); + RootUseAnalysis analysis = analyzeRootUses(root, block); + Operation *lastMaterialUser = findLastMaterialUser(analysis, block); + if (!lastMaterialUser) { + return std::nullopt; + } + for (const TiedUse &tiedUse : tiedUses) { + if (tiedUse.operation == lastMaterialUser) { + return tiedUse; + } + } + return std::nullopt; +} + +static bool isSameTiedUse(const TiedUse &lhs, const TiedUse &rhs) { + return lhs.operation == rhs.operation && lhs.operandIndex == rhs.operandIndex; +} + +static void materializeCopies(TiedUseGroups &groups, MLIRContext *context) { + OpBuilder builder(context); + for (auto &entry : groups) { + Value root = entry.first; + SmallVectorImpl &tiedUses = entry.second; + std::optional owner = chooseOwner(root, tiedUses); + for (const TiedUse &tiedUse : tiedUses) { + if (owner && isSameTiedUse(*owner, tiedUse)) { + continue; + } + Value operand = tiedUse.operation->getOperand(tiedUse.operandIndex); + builder.setInsertionPoint(tiedUse.operation); + auto copy = builder.create(tiedUse.operation->getLoc(), + operand.getType(), operand); + tiedUse.operation->setOperand(tiedUse.operandIndex, copy.getResult()); + } + } +} + +struct VPTOMaterializeTiedOperandCopiesPass + : public pto::impl::VPTOMaterializeTiedOperandCopiesBase< + VPTOMaterializeTiedOperandCopiesPass> { + void runOnOperation() override { + func::FuncOp function = getOperation(); + StringAttr target = findTargetArchitecture(function); + if (!target) { + function.emitError( + "VPTO tied-copy materialization requires target architecture 'a5', " + "but neither this function's module nor an enclosing module defines " + "'pto.target_arch'"); + return signalPassFailure(); + } + StringRef targetName = target.getValue(); + if (targetName != "a5") { + function.emitError( + "VPTO tied-copy materialization requires target architecture 'a5', " + "but module targets '") + << targetName << "'"; + return signalPassFailure(); + } + + TiedUseGroups groups; + if (failed(collectTiedUseGroups(function, groups))) { + return signalPassFailure(); + } + materializeCopies(groups, &getContext()); + } +}; + +} // namespace + +std::unique_ptr +mlir::pto::createVPTOMaterializeTiedOperandCopiesPass() { + return std::make_unique(); +} diff --git a/test/lit/vpto/vmov_ir_semantics.pto b/test/lit/vpto/vmov_ir_semantics.pto new file mode 100644 index 0000000000..5740fe230c --- /dev/null +++ b/test/lit/vpto/vmov_ir_semantics.pto @@ -0,0 +1,46 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: pto-test-opt %s -canonicalize -cse | FileCheck %s +// RUN: pto-test-opt %s '-pto-vpto-scheduler=mode=analyze' > /dev/null 2> %t +// RUN: FileCheck %s --check-prefix=SCHED < %t + +// CHECK-LABEL: func.func @copies_are_not_folded_or_eliminated( +// CHECK-COUNT-2: pto.vmov %arg0 : !pto.vreg<64xf32> -> !pto.vreg<64xf32> +// SCHED-LABEL: vpto-scheduler: function=copies_are_not_folded_or_eliminated +// SCHED: nodes=2 edges=0 +// SCHED-COUNT-2: op=pto.vmov semantic=schedulable sched-class=vector known=true +// SCHED: coverage schedulable=2 structural=0 boundary=0 unsupported=0 unclassified=0 + +module attributes {pto.target_arch = "a5"} { + func.func @copies_are_not_folded_or_eliminated(%input: !pto.vreg<64xf32>) { + pto.vecscope { + %copy0 = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + %copy1 = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + } + return + } + + // Parsing all supported element types also covers the vmov verifier's + // single-physical-vector type matrix. + func.func @supported_types(%u8: !pto.vreg<256xui8>, %s8: !pto.vreg<256xsi8>, %u16: !pto.vreg<128xui16>, %s16: !pto.vreg<128xsi16>, %f16: !pto.vreg<128xf16>, %bf16: !pto.vreg<128xbf16>, %u32: !pto.vreg<64xui32>, %s32: !pto.vreg<64xsi32>, %f32: !pto.vreg<64xf32>, %s64: !pto.vreg<32xsi64>) { + pto.vecscope { + %copy0 = pto.vmov %u8 : !pto.vreg<256xui8> -> !pto.vreg<256xui8> + %copy1 = pto.vmov %s8 : !pto.vreg<256xsi8> -> !pto.vreg<256xsi8> + %copy2 = pto.vmov %u16 : !pto.vreg<128xui16> -> !pto.vreg<128xui16> + %copy3 = pto.vmov %s16 : !pto.vreg<128xsi16> -> !pto.vreg<128xsi16> + %copy4 = pto.vmov %f16 : !pto.vreg<128xf16> -> !pto.vreg<128xf16> + %copy5 = pto.vmov %bf16 : !pto.vreg<128xbf16> -> !pto.vreg<128xbf16> + %copy6 = pto.vmov %u32 : !pto.vreg<64xui32> -> !pto.vreg<64xui32> + %copy7 = pto.vmov %s32 : !pto.vreg<64xsi32> -> !pto.vreg<64xsi32> + %copy8 = pto.vmov %f32 : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + %copy9 = pto.vmov %s64 : !pto.vreg<32xsi64> -> !pto.vreg<32xsi64> + } + return + } +} diff --git a/test/lit/vpto/vmov_verify_invalid.pto b/test/lit/vpto/vmov_verify_invalid.pto new file mode 100644 index 0000000000..768f1d31e3 --- /dev/null +++ b/test/lit/vpto/vmov_verify_invalid.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: pto-test-opt %s -split-input-file -verify-diagnostics + +module { + func.func @mismatched_types(%input: !pto.vreg<64xf32>) { + pto.vecscope { + // expected-error@+1 {{requires input and result to have identical vector types}} + %copy = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xi32> + } + return + } +} + +// ----- + +module { + func.func @unsupported_unsigned_i64(%input: !pto.vreg<32xui64>) { + pto.vecscope { + // expected-error@+1 {{requires 8/16/32-bit integer or non-unsigned 64-bit integer vector element type}} + %copy = pto.vmov %input : !pto.vreg<32xui64> -> !pto.vreg<32xui64> + } + return + } +} diff --git a/test/lit/vpto/vmov_vpto_llvm.pto b/test/lit/vpto/vmov_vpto_llvm.pto new file mode 100644 index 0000000000..73a851a336 --- /dev/null +++ b/test/lit/vpto/vmov_vpto_llvm.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --cann-output-version=9.0.0 --pto-arch=a5 --pto-backend=vpto --emit-vpto-llvm-ir %s -o - 2>&1 | FileCheck %s --check-prefix=CANN900 +// RUN: ptoas --cann-output-version=9.0.0-beta.1 --pto-arch=a5 --pto-backend=vpto --emit-vpto-llvm-ir %s -o - 2>&1 | FileCheck %s --check-prefix=BETA1 + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @vmov_f32(%input: !pto.vreg<64xf32>, %destination: !pto.ptr) attributes {pto.kernel} { + %c0 = arith.constant 0 : index + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + %copy = pto.vmov %input : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + pto.vsts %copy, %destination[%c0], %mask : !pto.vreg<64xf32>, !pto.ptr, !pto.mask + } + return + } +} + +// CANN900-LABEL: define void @vmov_f32_mix_aiv +// CANN900: call <64 x float> @llvm.hivm.vmov.x.v64f32(<64 x float> +// BETA1-LABEL: define void @vmov_f32_mix_aiv +// BETA1: call <64 x float> @llvm.hivm.vmov.v64f32(<64 x float> diff --git a/test/lit/vpto/vpto_materialize_tied_operand_copies.pto b/test/lit/vpto/vpto_materialize_tied_operand_copies.pto new file mode 100644 index 0000000000..96d47f17c9 --- /dev/null +++ b/test/lit/vpto/vpto_materialize_tied_operand_copies.pto @@ -0,0 +1,115 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: pto-test-opt %s -pto-vpto-materialize-tied-operand-copies | FileCheck %s + +module attributes {pto.target_arch = "a5"} { + // CHECK-LABEL: func.func @shared_vmula( + func.func @shared_vmula(%acc: !pto.vreg<64xf32>, %lhs0: !pto.vreg<64xf32>, %lhs1: !pto.vreg<64xf32>, %lhs2: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[COPY0:.*]] = pto.vmov %arg0 : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + // CHECK-NEXT: pto.vmula %[[COPY0]], %arg1, %arg4, %{{.*}} + %r0 = pto.vmula %acc, %lhs0, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK: %[[COPY1:.*]] = pto.vmov %arg0 : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + // CHECK-NEXT: pto.vmula %[[COPY1]], %arg2, %arg4, %{{.*}} + %r1 = pto.vmula %acc, %lhs1, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vmula %arg0, %arg3, %arg4, %{{.*}} + %r2 = pto.vmula %acc, %lhs2, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @shared_vmadd( + func.func @shared_vmadd(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[MADD_COPY0:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.vmadd %[[MADD_COPY0]], %arg1, %arg2, %{{.*}} + %r0 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK: %[[MADD_COPY1:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.vmadd %[[MADD_COPY1]], %arg1, %arg2, %{{.*}} + %r1 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vmadd %arg0, %arg1, %arg2, %{{.*}} + %r2 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @mixed_view_root( + func.func @mixed_view_root(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + %view0 = pto.vbitcast %acc : !pto.vreg<64xf32> -> !pto.vreg<64xi32> + %view1 = pto.vbitcast %view0 : !pto.vreg<64xi32> -> !pto.vreg<64xf32> + // CHECK: %[[VIEW_COPY:.*]] = pto.vmov %{{.*}} + // CHECK-NEXT: pto.vmula %[[VIEW_COPY]], %arg1, %arg2, %{{.*}} + %r0 = pto.vmula %view1, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vmadd %arg0, %arg1, %arg2, %{{.*}} + %r1 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @vaxpy_uses_operand_one( + func.func @vaxpy_uses_operand_one(%src: !pto.vreg<64xf32>, %dst: !pto.vreg<64xf32>, %alpha: f32) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[AXPY_COPY:.*]] = pto.vmov %arg1 + // CHECK-NEXT: pto.vaxpy %arg0, %[[AXPY_COPY]], %arg2, %{{.*}} + %r0 = pto.vaxpy %src, %dst, %alpha, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, f32, !pto.mask -> !pto.vreg<64xf32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vaxpy %arg0, %arg1, %arg2, %{{.*}} + %r1 = pto.vaxpy %src, %dst, %alpha, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, f32, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @histogram_tied_acc( + func.func @histogram_tied_acc(%acc: !pto.vreg<128xui16>, %src: !pto.vreg<256xui8>, %bin: i32) { + pto.vecscope { + %mask = pto.pset_b8 "PAT_ALL" : !pto.mask + // CHECK: %[[HIST_COPY:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.chistv2 %[[HIST_COPY]], %arg1, %{{.*}}, %arg2 + %r0 = pto.chistv2 %acc, %src, %mask, %bin : !pto.vreg<128xui16>, !pto.vreg<256xui8>, !pto.mask, i32 -> !pto.vreg<128xui16> + // CHECK-NOT: pto.vmov + // CHECK: pto.dhistv2 %arg0, %arg1, %{{.*}}, %arg2 + %r1 = pto.dhistv2 %acc, %src, %mask, %bin : !pto.vreg<128xui16>, !pto.vreg<256xui8>, !pto.mask, i32 -> !pto.vreg<128xui16> + } + return + } + + // CHECK-LABEL: func.func @vusqz_tied_carrier( + func.func @vusqz_tied_carrier(%src: !pto.vreg<64xi32>, %mask0: !pto.mask, %mask1: !pto.mask) { + pto.vecscope { + // CHECK: %[[USQZ_COPY:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.vusqz %[[USQZ_COPY]], %arg1 + %r0 = pto.vusqz %src, %mask0 : !pto.vreg<64xi32>, !pto.mask -> !pto.vreg<64xi32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vusqz %arg0, %arg2 + %r1 = pto.vusqz %src, %mask1 : !pto.vreg<64xi32>, !pto.mask -> !pto.vreg<64xi32> + } + return + } + + // CHECK-LABEL: func.func @carry_ops_are_not_tied( + func.func @carry_ops_are_not_tied(%lhs: !pto.vreg<64xui32>, %rhs: !pto.vreg<64xui32>, %carry_in: !pto.mask) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK-NOT: pto.vmov + // CHECK: pto.vaddcs + %sum0, %carry0 = pto.vaddcs %lhs, %rhs, %carry_in, %mask : !pto.vreg<64xui32>, !pto.vreg<64xui32>, !pto.mask, !pto.mask -> !pto.vreg<64xui32>, !pto.mask + // CHECK-NEXT: pto.vsubcs + %sum1, %carry1 = pto.vsubcs %lhs, %rhs, %carry0, %mask : !pto.vreg<64xui32>, !pto.vreg<64xui32>, !pto.mask, !pto.mask -> !pto.vreg<64xui32>, !pto.mask + } + return + } +} diff --git a/test/lit/vpto/vpto_materialize_tied_operand_copies_conservative.pto b/test/lit/vpto/vpto_materialize_tied_operand_copies_conservative.pto new file mode 100644 index 0000000000..6277316fa4 --- /dev/null +++ b/test/lit/vpto/vpto_materialize_tied_operand_copies_conservative.pto @@ -0,0 +1,75 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: pto-test-opt %s -pto-vpto-materialize-tied-operand-copies | FileCheck %s + +module attributes {pto.target_arch = "a5"} { + // CHECK-LABEL: func.func @later_ordinary_read( + func.func @later_ordinary_read(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[COPY0:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.vmula %[[COPY0]] + %r0 = pto.vmula %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK: %[[COPY1:.*]] = pto.vmov %arg0 + // CHECK-NEXT: pto.vmadd %[[COPY1]] + %r1 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK: pto.vabs %arg0 + %later = pto.vabs %acc, %mask : !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @cross_block_use( + func.func @cross_block_use(%condition: i1, %acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + scf.if %condition { + // CHECK: scf.if + // CHECK: %[[THEN_COPY:.*]] = pto.vmov %arg1 + // CHECK-NEXT: pto.vmula %[[THEN_COPY]] + %r0 = pto.vmula %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } else { + // CHECK: } else { + // CHECK: %[[ELSE_COPY:.*]] = pto.vmov %arg1 + // CHECK-NEXT: pto.vmadd %[[ELSE_COPY]] + %r1 = pto.vmadd %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + } + return + } + + // CHECK-LABEL: func.func @single_use_existing_copy( + func.func @single_use_existing_copy(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[EXISTING:.*]] = pto.vmov %arg0 + // CHECK-NOT: pto.vmov + // CHECK: pto.vmula %[[EXISTING]] + %copy = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + %r0 = pto.vmula %copy, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } + + // CHECK-LABEL: func.func @shared_existing_copy( + func.func @shared_existing_copy(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + // CHECK: %[[ROOT:.*]] = pto.vmov %arg0 + %copy = pto.vmov %acc : !pto.vreg<64xf32> -> !pto.vreg<64xf32> + // CHECK: %[[CHILD:.*]] = pto.vmov %[[ROOT]] + // CHECK-NEXT: pto.vmula %[[CHILD]] + %r0 = pto.vmula %copy, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + // CHECK-NOT: pto.vmov + // CHECK: pto.vmadd %[[ROOT]] + %r1 = pto.vmadd %copy, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } +} diff --git a/test/lit/vpto/vpto_materialize_tied_operand_copies_idempotent.pto b/test/lit/vpto/vpto_materialize_tied_operand_copies_idempotent.pto new file mode 100644 index 0000000000..144f57cf56 --- /dev/null +++ b/test/lit/vpto/vpto_materialize_tied_operand_copies_idempotent.pto @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: pto-test-opt %s -pto-vpto-materialize-tied-operand-copies -pto-vpto-materialize-tied-operand-copies | FileCheck %s + +// CHECK-LABEL: func.func @idempotent( +// CHECK: pto.vmov +// CHECK-NEXT: pto.vmula +// CHECK: pto.vmov +// CHECK-NEXT: pto.vmula +// CHECK-NOT: pto.vmov +// CHECK: pto.vmula +// CHECK-NOT: pto.vmov + +module attributes {pto.target_arch = "a5"} { + func.func @idempotent(%acc: !pto.vreg<64xf32>, %lhs: !pto.vreg<64xf32>, %rhs: !pto.vreg<64xf32>) { + pto.vecscope { + %mask = pto.pset_b32 "PAT_ALL" : !pto.mask + %r0 = pto.vmula %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + %r1 = pto.vmula %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + %r2 = pto.vmula %acc, %lhs, %rhs, %mask : !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.vreg<64xf32>, !pto.mask -> !pto.vreg<64xf32> + } + return + } +} diff --git a/test/lit/vpto/vpto_materialize_tied_operand_copies_target.pto b/test/lit/vpto/vpto_materialize_tied_operand_copies_target.pto new file mode 100644 index 0000000000..b58f32b611 --- /dev/null +++ b/test/lit/vpto/vpto_materialize_tied_operand_copies_target.pto @@ -0,0 +1,25 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not pto-test-opt %s -split-input-file -pto-vpto-materialize-tied-operand-copies 2>&1 | FileCheck %s + +// CHECK: error: VPTO tied-copy materialization requires target architecture 'a5', but neither this function's module nor an enclosing module defines 'pto.target_arch' +module { + func.func @missing_target() { + return + } +} + +// ----- + +// CHECK: error: VPTO tied-copy materialization requires target architecture 'a5', but module targets 'a3' +module attributes {pto.target_arch = "a3"} { + func.func @wrong_target() { + return + } +}