feat: add wrappers for ATB and ACLNN fused operators.

[yingxudeng]

No description provided.

[XuZhang99]

remove torch::Tensor active_tensor(ActivationParams& params) and add params.output = npu::active(params.input, params.act_mode) here for npu device.

[yingxudeng]

auto output = torch::empty(
    {batch_size,
     intermediate_size_ / parallel_args_.tp_group_->world_size()},
    gate_up.options());

This is a good modification. However, as described, the current code's output still allocates space preemptively. For NPU operators, they typically allocate their own space and return the result. This unavoidable difference still forces the external calling code to use an #if block to skip space allocation specifically for the NPU case.

To standardize the external calling code, I personally recommend aligning with the NPU's behavior: allocate the space within the operator wrapper/layer and then return it. This approach allows for a unified code structure for all external calls.

[XuZhang99]

so don't add active_tensor and fused_layernorm_tensor these two func in ops_api.h, because no other platform will use such api.
put they in npu_ops_api.h and call them directly in npu layer.

[yingxudeng]

Regarding the code snippet above: if we implement the changes as suggested, we would need to introduce #if directives here to skip memory allocation, since the NPU operator handles this internally.

Could we instead consider moving the memory allocation logic for MLU and CUDA into their respective kernel wrappers? This would make the behavior more similar to PyTorch and allow us to unify the calling code here.

(PS: I haven't modified the CUDA or MLU code yet.)

[XuZhang99]

same as above

[yingxudeng]

Similar to the previous comment.

[XuZhang99]

std::optional<torch::Tensor> position_ids already exists.

[yingxudeng]

During the implementation, I noticed that position_ids are set to empty during the prefill stage, so I initially added position. However, I see that the latest CUDA code addresses the same issue using a different approach. To ensure consistency, I plan to align my implementation with the CUDA method.

[XuZhang99]

For activation ops on npu, revert this commit(refactor: standardize interface for active kernel execution.), and this is what you only need to do:

#elif defined(USE_NPU)
  # make params.output become a null tensor 
  params.output = torch::Tensor();
  params.output = npu::active(params.input, params.act_mode);

[yingxudeng]

#474 (comment) Regarding GPU/MLU code, this has no performance impact, as it simply utilizes pre-allocated memory space. However, for NPU code, repeated calls to operations such as RMS normalization may lead to frequent memory allocations and immediate deallocations within the external framework. Could this pattern of repeatedly allocating and promptly discarding memory potentially affect performance on NPU architectures?

[XuZhang99]

auto output = torch::empty_like(input);
output is already created, why create it again?

[XuZhang99]

#474 (comment) Regarding GPU/MLU code, this has no performance impact, as it simply utilizes pre-allocated memory space. However, for NPU code, repeated calls to operations such as RMS normalization may lead to frequent memory allocations and immediate deallocations within the external framework. Could this pattern of repeatedly allocating and promptly discarding memory potentially affect performance on NPU architectures?

in dense_mlp.cpp:

torch::Tensor output;
if(Device::type!="npu"){
    output = torch::empty(
        {batch_size,
         intermediate_size_ / parallel_args_.tp_group_->world_size()},
        gate_up.options());
}

btw, you need to learn more about memory management in torch.

[XuZhang99]

norm ops for npu need to support fused and non-fused mode.

[yingxudeng]

#474 (comment) Regarding GPU/MLU code, this has no performance impact, as it simply utilizes pre-allocated memory space. However, for NPU code, repeated calls to operations such as RMS normalization may lead to frequent memory allocations and immediate deallocations within the external framework. Could this pattern of repeatedly allocating and promptly discarding memory potentially affect performance on NPU architectures?

in dense_mlp.cpp:

torch::Tensor output;
if(Device::type!="npu"){
    output = torch::empty(
        {batch_size,
         intermediate_size_ / parallel_args_.tp_group_->world_size()},
        gate_up.options());
}

btw, you need to learn more about memory management in torch.

#474 (comment) Regarding GPU/MLU code, this has no performance impact, as it simply utilizes pre-allocated memory space. However, for NPU code, repeated calls to operations such as RMS normalization may lead to frequent memory allocations and immediate deallocations within the external framework. Could this pattern of repeatedly allocating and promptly discarding memory potentially affect performance on NPU architectures?

in dense_mlp.cpp:

torch::Tensor output;
if(Device::type!="npu"){
    output = torch::empty(
        {batch_size,
         intermediate_size_ / parallel_args_.tp_group_->world_size()},
        gate_up.options());
}

btw, you need to learn more about memory management in torch.

Thank you for your review. Moving forward, I will replace the unavoidable #if defined macros with runtime checks based on Device::type. Regarding the description of memory allocation and de-allocation, please feel free to disregard it; it was translated by LLM and is not entirely accurate. In short, for NPU models, I want to avoid having an external output = torch::empty.