Which open-source LLM API is best? The one your own eval picks

There is no single best open-source LLM API, and no single best model. The right choice is the cheapest model that passes an evaluation built from your own prompts, bought from a provider whose latency, billing and data terms fit the workload: pick the task, shortlist two or three models, test them on real inputs, price the workload rather than the token, and keep a second route ready.

Leaderboards measure someone else's prompts on someone else's setup. What follows is the method, the general trade-offs, and a fair map of where to buy, including when a provider other than P/9 is the better answer.

Start with the task, because the task decides what to measure

A document extractor, a support assistant, a coding agent and a search index want different things from a model. Write the task down first: what goes in, what must come out, how fast, and how wrong it is allowed to be.

• Extraction and classification: output that parses every time, where a smaller model may clear the bar
• Assistants: instruction following and time to first token, because a person is waiting
• Coding and agents: tool calling and multi-step reasoning, where larger reasoning models are most likely to earn their rate
• Long documents: a context window that fits the real input, as the provider accepts it rather than as the weights were published
• Search: an embedding model, which returns vectors rather than text

What model size, reasoning and licence actually trade off

Headline parameter counts mislead, because many open-weight models are mixtures of experts that use only part of the network for each token. gpt-oss 120B's model card publishes 117B total and 5.1B active parameters; Kimi K3's publishes 2.8T and 104B. The active figure is closer to what each token costs to compute, and neither says how a model does on your task.

Reasoning models think before the visible answer, and that thinking is billed as output tokens, the dearer half of every chat route in the P/9 catalogue. Two similar rates can therefore produce very different bills for the same prompt.

Check context and licence before the eval. gpt-oss 120B publishes a 131,072-token window under Apache 2.0; Kimi K3 publishes 1,048,576 tokens under its own Kimi K3 License. A published window is not a promise about what a provider accepts, and the licence matters the day you consider hosting the weights yourself.

Build the eval from your own prompts, not from a leaderboard

Published benchmarks are run by the lab on the lab's prompts, and they describe the weights rather than your product. P/9's model cards print the labs' own scores and say they were not re-measured on P/9 hardware: useful for choosing what to test, not for choosing between candidates.

An eval set is a file of real inputs with a way to judge each output. Take a few dozen to a couple of hundred prompts from actual traffic, keep the awkward cases, and write down what a pass looks like before any model answers. Commit the file: it is also the test for next quarter's model.

• Checkable outputs: JSON that validates, the right label, the right figure pulled from a document
• Judged outputs: a short rubric a colleague can apply in seconds
• Known failures: prompts a previous model got wrong, which separate candidates better than easy ones
• Scrubbed inputs: names, account numbers and CNICs removed before any prompt is sent for testing

Run two or three candidates through the same prompts

Send every prompt to every candidate with identical settings and record, per call, whether it passed, the tokens in and out, and the time taken. Behind an OpenAI-compatible endpoint that is one loop.

The three chat routes open on P/9 on 17 September 2026 share one endpoint and one key. A header starting X-P9-Meta- is recorded against each request and searchable in the request log, which keeps eval traffic apart from production.

import os, time
from openai import OpenAI

client = OpenAI(
    base_url="https://api.paranine.com/v1",
    api_key=os.environ["P9_API_KEY"],
)

CANDIDATES = [
    "paranine/gpt-oss-120b(Global)",
    "paranine/GLM-5.2",
    "paranine/Kimi-K3(Global)",
]

# eval_set: your own prompts, each with a check that returns True or False
for model in CANDIDATES:
    for case in eval_set:
        started = time.monotonic()
        reply = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": case["prompt"]}],
            extra_headers={"X-P9-Meta-eval_run": "shortlist-1"},
        )
        answer = reply.choices[0].message.content or ""
        print(
            model,
            case["check"](answer),
            reply.usage.prompt_tokens,
            reply.usage.completion_tokens,
            round(time.monotonic() - started, 2),
        )

Price the workload, not the token

Take 200 prompts and assume, for the arithmetic only, 800 input and 400 completion tokens each on every candidate, thinking included. At the rates P/9 published on 17 September 2026, one pass costs about Rs 19 on gpt-oss 120B, Rs 177 on GLM-5.2 and Rs 513 on Kimi K3: about Rs 708 for all three, inside the Rs 1,000 of credits a new organisation starts with.

The spread is the argument for measuring. A cheaper route is only cheaper if it passes, because failures get retried, escalated or lost. Divide each candidate's cost by the prompts it passed, using the token counts the run recorded rather than the assumption above.

The catalogue: every route, its pool and its current rate per million tokens in rupees

Keep a second route ready before you need one

Models are deprecated, routes go down and prices change, on their schedule rather than yours. The eval has already named the fallback: the runner-up that passed. Keep its id in configuration, retry once on it after a timeout or a server error, and log that it happened. It can even sit on another provider.

On P/9, scope the production key to the primary and the fallback and nothing else. A request for any other route is refused with a 403, so a misconfigured fallback cannot quietly bill a model nobody chose.

Scoping a key to specific models, with spend caps and expiry

Five places to buy open-model inference, and what each trades away

Each kind of provider trades breadth, control and location differently. Every example named here accepts the OpenAI chat completions request shape, so none of them is a permanent choice.

• Aggregators: OpenRouter documents hundreds of models behind one endpoint, favouring cheaper providers that have stayed up unless you pin or exclude them. The widest catalogue for the least work
• Inference hosts: Together AI, Fireworks AI and DeepInfra serve popular open-weight models at per-token rates listed in dollars. A good fit when your model is on their list and a dollar bill is no obstacle
• The labs themselves: DeepSeek, for one, documents its API as OpenAI-compatible. Closest to the team that trained the model, at the cost of one account and one bill per lab
• Dedicated or self-hosted hardware, paid for by time whether busy or idle: Together AI bills dedicated deployments per minute, P/9 reserves Ascend NPUs racked in Pakistan per hour, and your own accelerators can run vLLM. Cheaper than per-token pricing only when traffic keeps the hardware busy
• A local gateway: one OpenAI-compatible API in the local currency, with the pool behind each route stated. For Pakistan that is P/9: rupee billing, with a narrower catalogue than an aggregator's

Choosing an inference provider when location is a requirement

What P/9 adds for a team in Pakistan, and what is not open yet

P/9 bills in rupees from prepaid credits, at rates published per million tokens. Keys can be capped in rupees, scoped to routes and given an expiry, and the request log shows the settled cost of every call. If the blocker is a dollar card or an exchange-rate line, that removes it for the price of a base URL and a key.

What is not open deserves a date. On 17 September 2026 the catalogue lists sixteen routes, and the three open chat routes (gpt-oss 120B, Kimi K3 and GLM-5.2) are all on the global pass-through pool. The PK NPU-native routes, including the only embedding model, are priced but marked coming soon, as are ten further partner routes.

What an LLM gateway does, and what changes when it runs in Pakistan

When another provider is the better choice

In each of these cases P/9 is not the right answer today. Keep the eval either way, so revisiting the choice is a re-run rather than a project.

• The model your eval picked is not in the catalogue, or is listed but not open yet
• You need embeddings now: P/9's only embedding route is not open
• You need fine-tuning or file uploads: P/9 has no endpoint for either
• Your data must be processed inside Pakistan today: the in-country pool is not open, and the open routes are served elsewhere
• You have no residency requirement and dollar billing is no obstacle: a global aggregator's larger catalogue is often the better answer

What a sovereign AI gateway is, and when a team actually needs one

The short version

Pick the task, shortlist from published characteristics, test on your own prompts and divide cost by passes. Keep the runner-up as the fallback and re-run the file whenever a model, a price or a provider changes. The best LLM API for a startup is whichever survives that loop at the lowest cost per passed task.