How to deploy LLMs on SA8295

[Leonslam]

Thank you for your sharing, I want to deploy LLMs on SA8295, does the project support ? And What are the prerequiste?
For examples, the android version? QNN version? Which LLM should I choose, and so on. Looking forward to your reply.

[SorryMaker2022]

First and foremost, we do support SA8295.
The prerequistes depends your device vendor.

• We do not require a specific Android version.
• The QNN version for this project is 2.29, we have been working on migrating it to 2.39, trying to enable some new features, but sadly most new QNN features, like blockwise quantization, are eligible only on newer SoCs. So keeping the old version is fine.
• For LLMs, you should check your SKU memory size. We suggest Llama 3 series, 1B / 3B / 8B models.

We have noticed that on some specific devkit SKUs, you will need to flash the firmware or push some specific libraries to make the project work, but to our knowledge this highly depends on your vendor SKU. So we are not giving a unified tutorial here. You should talk to your hardware vendor about it.

[Leonslam]

@SorryMaker2022 Thank you for your reply. But what does the "SKU memory" mean, cpu ram or other things ?

[SorryMaker2022]

Exactly CPU RAM. I remember the SA8295 devkit offers 25GB, but it may vary.

It is suggested to check the history issues for more info.