Add sycl examples

[ZhaoqiongZ]

Add the example to implement muladd op with SyclExtension

[dvrogozh]

LGTM

[ZhaoqiongZ]

Hi @goldsborough @zou3519 , could you help review this PR? Thank you!

[zou3519]

What is Sycl?

[dvrogozh]

What is Sycl?

Sycl is a C++ abstraction for heterogeneous programming allowing to write and execute code (kernels) on GPU devices. That's the way Intel uses to wrote kernels for Intel GPUs. A number of PyTorch aten operators are written using Sycl. And writing Sycl kernels is exposed to PyTorch users thru CPP Extension API. We are trying to update PyTorch tutorials/examples to demonstrate and encourage users to use Sycl to write custom kernels. See https://www.khronos.org/sycl/. Should we add a note on what is SYCL in the project readme?

[zou3519]

extension-cpp is supposed to be the "base repository" that people clone to do C++/CUDA extensions. Any special hardwares (or other workflows) is out of scope

[dvrogozh]

@zou3519, xpu is now one of the official backends included in pytorch. It's backed by SYCL to generate device specific code and pytorch API to allow users create custom operators. Further, pytorch tutorial educates on usage of this API for C++ and CUDA (https://github.com/pytorch/tutorials/blob/main/advanced_source/cpp_custom_ops.rst). The SYCL cpp extension API we've added to pytorch extends this functionality adding SYCL into existing landscape. Obvious step to consider is to extend existing C++ and CUDA tutorial and example to also cover SYCL. It seems you are against this proposal. Can you, please, provide your reasoning to avoid adding SYCL in this tutorial and example and suggest alternative approach which will help to extend pytorch ecosystem to cover devices other than CUDA and other vendors?

fyi, proposal to change tutorial is here:

• Update cpp custom ops docs with sycl extension and xpu support tutorials#3391

It's currently draft as it depends on this PR in extension-cpp repository.

[zou3519]

and suggest alternative approach which will help to extend pytorch ecosystem to cover devices other than CUDA and other vendors?

I would recommend having a separate extension-sycl repository showing users how to do a sycl extension for XPU. The main reasons are:

1. if we end up supporting "all backends" in this repository, then this repo will get pretty bloated and difficult to work with. It'll be easier and faster for users to just clone an extension-sycl repo instead of having them clone extension-cpp and deleting the e.g. TPU and CUDA code.
2. There's no code that is being reused between the SYCL and the CUDA extensions, so I don't see a good reason for putting all the code into one repo.