8 min readparanine
Serving LLMs on Ascend NPUs: what actually changes from CUDA
Import constraints mean much of the accelerator capacity arriving in Pakistan is Huawei Ascend, not Nvidia. The differences are real, they sit in the toolchain rather than the maths, and almost none of them should reach an application team.
Nearly every piece of inference advice online assumes a CUDA device. The serving frameworks default to it, the quantisation recipes are validated on it, and the deployment guides are written for it. That assumption does not survive contact with the hardware actually reaching Pakistan, which is largely Huawei's Ascend line.
The useful question is not which is better in the abstract. It is which parts of the difference a team has to care about, and which parts belong to whoever operates the pool.
The maths is the same. The toolchain is not.
An Ascend NPU is a capable matrix engine, and a transformer is a transformer. What differs is everything around the computation. Kernels target CANN rather than CUDA. Operator coverage is not identical, so a model that runs unmodified on a GPU can need conversion or a patched operator before it runs on an NPU. The profiling and debugging tools are a different set, with different names and different output.
None of that is a reason to avoid the hardware. It is a reason to make sure only one team ever has to learn it.
Where the friction actually appears
Every item on that list is real work, and every item is the kind of work that should happen once, behind an endpoint, rather than in each application that wants to call a model.
- Model conversion: a checkpoint usually needs an explicit conversion step rather than loading as-is
- Operator gaps: an unsupported operator is a porting task, not a configuration flag
- Framework maturity: serving stacks ported onto Ascend carry more edges than those written for it
- Recipe transfer: quantisation and batching settings tuned on a GPU are a starting point, not an answer
This is what a gateway is for
A gateway is an abstraction over which accelerator served a request. An application sends the OpenAI chat completions shape to a base URL with a route id in the model field. Whether that route resolved to an NPU pool inside Pakistan or to a global pass-through route is an operational decision, not an application one, and it can change without a code change on the caller's side.
That is the whole argument for putting one in front of heterogeneous hardware. Not that NPUs are hard, but that the difficulty should be paid once, by the people who chose to take it on.
What we are not claiming
We are not publishing throughput comparisons between Ascend and Nvidia here, because a number quoted without the model, the batch shape, the sequence length and the serving stack beside it is marketing rather than measurement. When we have figures measured on P/9 hardware under a stated configuration, they will be published with that configuration attached.
What can be said now is architectural: the capacity available in the country is largely NPU, the toolchain difference is real and surmountable, and an application should never have to know which one answered.