Urdu LLM API: how to test a model on Urdu and Roman Urdu

An LLM API takes Urdu, in Urdu script or Roman Urdu, as ordinary text in a standard chat request: there is no separate Urdu API to find. What differs is how well each model handles the Urdu your users actually write, and the dependable way to choose is to test candidates on real messages, because a model card's multilingual results may not include Urdu at all.

This is a method, not a leaderboard: P/9 publishes no Urdu score for any route.

Urdu arrives in three forms, and each is a separate test

People write Urdu in Urdu script, in Roman Urdu typed with Latin letters, and in either mixed with English. A good result on one form is not a measurement of the others, and Roman Urdu has no standard spelling: nahi also arrives as nahin or nai, so scoring it by exact string match marks correct answers wrong.

Decide which forms you accept and which you reply in before comparing models. One message in each form follows, used again below; both Urdu versions borrow the English word order.

• Urdu script: آپ کا آرڈر بھیج دیا گیا ہے اور تین دن میں آپ تک پہنچ جائے گا۔
• Roman Urdu: Aap ka order bhej diya gaya hai aur teen din mein aap tak pohanch jaye ga.
• English meaning: Your order has been sent and will reach you in three days.

Right-to-left is an interface problem, not a model problem

Unicode text is stored in logical order, first letter first; the bidirectional algorithm reorders Urdu only for display. The model receives the characters in that order and no display direction, so Urdu that looks scrambled on screen is a rendering fault to rule out first, and reversing a string to fix the look sends the model its words backwards.

Set dir="rtl" on elements that hold Urdu script rather than relying on dir="auto", which follows the first strongly directional character: a reply opening with an English product name gets a left-to-right base direction. Use a font with Nastaliq coverage, such as Noto Nastaliq Urdu.

Your users' messages are the Urdu test, not the model card

OpenAI's gpt-oss model card reports multilingual results on MMMLU, a professionally human-translated version of MMLU in 14 languages. Hindi and Arabic are among them; Urdu is not. That is an absence of evidence rather than evidence of weakness. Where a card does report Belebele, a reading comprehension benchmark covering 122 language variants, its urd_Arab and urd_Latn rows are a first filter for Urdu script and romanised Urdu.

The decision belongs to a few dozen real messages judged by a fluent reader, plus three rules specific to Urdu:

• Sample the three forms in the proportions your traffic shows, not in equal thirds
• Swap personal details for realistic stand-ins, such as a different Pakistani name, rather than deleting them, so the set still tests names
• Score script, meaning, register and figures separately, so a fluent reply with a wrong amount cannot pass on fluency

Choosing an LLM API by your own eval: the general method

Urdu script costs more tokens than English: a measurement

Counted locally with tiktoken 0.9.0, OpenAI's open-source tokenizer library, Urdu script took more tokens than English in every comparison below. Each item gives o200k_base (which tiktoken maps to GPT-4o), then cl100k_base (GPT-4).

The encoding matters too: the first Urdu sentence was 19 tokens in the larger o200k_base vocabulary and 59 in cl100k_base. Roman Urdu is not automatically cheaper either; under o200k_base it beat Urdu script only on the second sentence, where three of its seven words are English. P/9's gpt-oss 120B routes use o200k_harmony, which tiktoken's source builds from o200k_base's vocabulary and splitting rules plus chat-format special tokens, so ordinary text splits the same way there. Other models' tokenizers may not.

• English, 13 and 13: Your order has been sent and will reach you in three days.
• Urdu script, 19 and 59: the same sentence, as above
• Roman Urdu, 22 and 26: the same sentence, as above
• English, 7 and 7: Please tell me my account balance.
• Urdu script, 16 and 34: براہ کرم میرے اکاؤنٹ کا بیلنس بتا دیں۔
• Roman Urdu with English words, 8 and 9: Please mere account ka balance bata dein.

Measure the difference on the route you will call

Send each version as an otherwise identical synchronous request and compare usage.prompt_tokens, from the usage object P/9's documentation defines as what a synchronous call is billed on. The chat template is identical in each request, so the differences between the counts are the differences between the sentences.

Replies count the same way, and every chat route in the catalogue prices output above input, so the ratio lands on the dearer meter.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.paranine.com/v1",
    api_key=os.environ["P9_API_KEY"],
)

samples = {
    "English": "Your order has been sent and will reach you in three days.",
    "Urdu script": "آپ کا آرڈر بھیج دیا گیا ہے اور تین دن میں آپ تک پہنچ جائے گا۔",
    "Roman Urdu": "Aap ka order bhej diya gaya hai aur teen din mein aap tak pohanch jaye ga.",
}

for label, text in samples.items():
    reply = client.chat.completions.create(
        model="paranine/gpt-oss-120b(Global)",
        messages=[{"role": "user", "content": text}],
        max_tokens=16,  # only prompt_tokens matters here
    )
    print(label, reply.usage.prompt_tokens)

What a million tokens costs in rupees, and how to price a whole workload

Normalise Urdu letters before you count or compare

Unicode's chart marks keheh (U+06A9), heh goal (U+06C1) and Farsi yeh (U+06CC) for Urdu. Each has a separate Arabic counterpart, and text typed on an Arabic keyboard layout can contain that instead.

Swapping those three letters in the first Urdu sentence changed nine characters and moved its o200k_base count from 19 tokens to 27 (under cl100k_base it went from 59 to 57). Worse for an evaluation, the two versions no longer match as strings, and none of Unicode's four normalisation forms, NFC, NFD, NFKC and NFKD, maps one onto the other. Use an explicit map, checked by a fluent reader:

• Arabic kaf ك (U+0643) to keheh ک (U+06A9)
• Arabic yeh ي (U+064A) to Farsi yeh ی (U+06CC), checking word endings, where Urdu also writes yeh barree ے (U+06D2)
• Arabic heh ه (U+0647) only with judgement: it can stand for heh goal ہ (U+06C1) or for heh doachashmee ھ (U+06BE), which forms aspirated letters such as بھ

Choose the reply script in code, then say it in the prompt

In an Urdu product, any character from the Arabic block, U+0600 to U+06FF, marks a message as Urdu script; without one, it is Roman Urdu or English. That check is deterministic, and an instruction to mirror the user is not.

State the script, the form of address and the rule for figures, and show one exchange in the target form. Below, the example asks "When will my order come?" and answers "Your order will arrive in three days"; the new message asks "When will I get my refund?". Keep examples to form and tone: a model can repeat their details, such as those three days, where they do not apply.

{
  "model": "paranine/gpt-oss-120b(Global)",
  "messages": [
    {
      "role": "system",
      "content": "You answer customers of an online shop in Pakistan. Reply only in Urdu script. Address the customer as آپ. Write numbers in Western digits. Copy names exactly as the customer wrote them."
    },
    { "role": "user", "content": "میرا آرڈر کب آئے گا؟" },
    { "role": "assistant", "content": "آپ کا آرڈر تین دن میں پہنچ جائے گا۔" },
    { "role": "user", "content": "مجھے ریفنڈ کب ملے گا؟" }
  ]
}

Check numbers, names and register first

A fluent Urdu reply with one wrong figure is the failure to design against, because fluent text is the text nobody rereads. Give each its own rows in the evaluation.

• Digits: Urdu text uses Western digits, the Extended Arabic-Indic digits ۰ to ۹ that Unicode lists for Urdu, or both. Name one in the prompt and check figures come back unchanged
• Number words: ڈیڑھ is one and a half and ڈھائی two and a half, while سوا, ساڑھے and پونے add a quarter, add a half or take a quarter away, so پونے تین is 2.75 and ڈیڑھ لاکھ is 150,000
• Names: one name is spelled Muhammad, Mohammad or Muhammed, so tell the model to copy names exactly as written
• Register: address customers as آپ, not تم, and check that the vocabulary matches your product, such as اکاؤنٹ rather than the more formal کھاتہ

Where P/9 fits, and where you need something else

On P/9 an Urdu feature is the same integration as any other: one OpenAI-compatible endpoint, the route in the model string, rupee rates per million tokens, and each call's token counts and cost in the request log, which keeps no prompt or completion text. Urdu customer messages can carry names and CNIC numbers, so location matters: P/9's PK NPU-native routes, served from its own Ascend 910B capacity in Pakistan, were marked coming soon on 17 September 2026, and the catalogue marks the pool behind every route.

For spoken Urdu, P/9 is not the whole answer: its API is chat completions, with no transcription endpoint. Transcribe first with a speech model whose published languages include Urdu, such as OpenAI's open-source Whisper, then send the transcript.

What a sovereign AI gateway is, and why the location of a request matters

The short version

Urdu through an LLM API is ordinary text. The work is the choices around it, from script and model to tokens, letters and display, each measured rather than assumed.