Skip to content

Meter model tokens and export models to projects - #4499

Open
peterschmidt85 wants to merge 9 commits into
masterfrom
pr_meter_tokens_at_model_endpoint
Open

peterschmidt85 wants to merge 9 commits into
masterfrom
pr_meter_tokens_at_model_endpoint

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

A project can now export its models to other projects. On a gateway, every project gets its own model endpoint, and the model endpoint meters the tokens of each request.

What changes for users

A model endpoint per project

On a gateway, a project's models are at https://gateway.<domain>/<project>, an OpenAI-compatible API with /models and /chat/completions. dstack apply shows it below the service URL:

$ dstack apply -f qwen-service.dstack.yml
...
Service is published at:
  https://qwen-service.example.com
Model Qwen/Qwen3.8-27B-FP8 is published at:
  https://gateway.example.com/team-a

Members of the project access the model there with their own dstack tokens.

Before, the Model ... is published at line had the service URL plus /v1. Without a gateway, it still does.

Models in exports

An export can hold models, by name. Members of an importer project access an exported model at their own project's model endpoint. The gateway itself doesn't have to be exported.

$ dstack export create my-models --model Qwen/Qwen3.8-27B-FP8 --importer team-b
 NAME       FLEETS  GATEWAYS  MODELS                IMPORTERS
 my-models  -       -         Qwen/Qwen3.8-27B-FP8  team-b

In team-b:

$ dstack import list -v
 NAME              FLEETS  GATEWAYS  MODEL                 URL
 team-a/my-models  -       -         Qwen/Qwen3.8-27B-FP8  https://gateway.example.com/team-b

$ curl https://gateway.example.com/team-b/chat/completions \
    -H "Authorization: Bearer $DSTACK_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen3.8-27B-FP8", "messages": [{"role": "user", "content": "Hello"}]}'

A model can be exported before any service declares it, and it stays exported when its service is stopped and applied again. Importer projects can access it while a service of the exporter project runs it with the openai format on a gateway whose domain has no ${{ run.project_name }}. See Exports.

Token metering

The model endpoint meters the input, cached input and output tokens of every request to an openai model, per project, user and model, also for a stream that the client stops. dstack itself stores nothing. The server collects the counts from the gateway through a hook, which dstack Factory registers to store and show them.

Chat completions pass through

For openai models, the model endpoint forwards the request and the service's response as they are. Requests with stream_options, max_completion_tokens, tool calls or content parts used to get 422, and responses lost usage in stream chunks and prompt_tokens_details. Now they work, and an error of the service reaches the client with the service's own status and body.

UI

  • The Models page lists a project's own and imported models, with their status.
  • A model's page is about the model, not about a run. Its Endpoint button shows how to access the model at the user's project.
  • The Exports tab lists the model's exports and the projects that import it (where the server has exports).
  • The Metering tab shows the model's tokens by day or month and by project (dstack Factory and dstack Sky).
  • The chat shows a thinking model's reasoning.

The screenshots are from the end-to-end test, where exporter-proj exports the model to importer-proj and importer2-proj.

The Models page:

The Models page

A model's page:

A model's page

The Exports tab:

The Exports tab of a model's page

The Metering tab:

The Metering tab of a model's page

Breaking changes

  • The model endpoint moves from https://gateway.<domain> to https://gateway.<domain>/<project>. Requests to the old address get 404.
  • For an openai model on a gateway, the model URL that dstack shows is now the model endpoint. It was the service URL plus /v1. This is the URL in dstack apply, in service.model.base_url of the API and in Run.service_model of the Python API. The service URL itself doesn't change. The model endpoint always takes a dstack token of a project member, also when the service has auth: false.
  • For openai models, an error response of the service is returned as it is, so the client gets the engine's own error. Before, dstack put its body into {"detail": ...}.
  • A service on a gateway can't be named gateway, because that host is the model endpoint's.
  • A model page's address in the UI takes the model's name where it took the run's name.

Implementation highlights

  • Metering on the gateway. GatewayProxyMeter sums the tokens per hour, user, project, service and export, and writes them to batch files every 5 seconds. The server collects the batches with POST /api/metering/collect, from hooks of the gateway replica pipeline (register_process_running_gateway_replica_hook). dstack registers no hook.
  • Pass-through. forward_chat_completions replaces the typed OpenAI client for openai models. It asks the service for token counts on every stream chunk, and removes them from what the client gets unless the client asked for them.
  • One HTTP client per service in the model proxy ([Bug]: Gateway model proxy creates an HTTP client per request and never closes it #4489). It is an httpx2 client, in the gateway's model endpoint and in the in-server proxy: a request to a service is cancelled when its client disconnects, and with httpx some cancelled requests never released their connections (17 of 30,000 in a local test, none with httpx2). So httpx2 is now in the server extra too.
  • Project paths. nginx sends gateway.<domain>/<project>/... to the gateway app, which finds the project's models, own and imported. The project name in the path is not case-sensitive, and an unknown project gets 404.
  • Imports on the gateway. For each exported model that a running service can serve, GatewayReplicaPipeline registers an import per importer project on the gateway, and removes it when the export, the importer or the service goes away.
  • Which models can be served to importer projects. The rules are in server/services/exported_models.py. dstack refuses an export, an in-place service update or a gateway domain change that would break them.
  • Access. The gateway's model endpoint takes the members of the project in the path and global admins, as the in-server proxy does. Before, a gateway also took any user who could view a public project.
  • Gateway shutdown. When a gateway restarts or is updated, the gateway app cancels the requests still in flight after 10 seconds and writes a last batch, so that their tokens are metered (--timeout-graceful-shutdown 10, TimeoutStopSec=30).
  • Database. One migration adds exported_models.
  • Frontend. The Models list reads runs page by page: every active run, and the latest 1,000 runs when Active models is off.

Known issues

Tests

  • Unit tests pass locally (5,213 on SQLite, 6,781 with Postgres), and 76 frontend tests (npm run test:auth).
  • End to end on a test deployment with a real gateway and Qwen/Qwen3.8-27B-FP8 on SGLang: the steps of the user flow below through the API, the CLI and the UI.

Design

The PR implements Steps 1 to 2.3 of the metering plan:

Storing the metering records and the metering API are dstack Factory's part, in a PR of its own.

🤖 Generated with Claude Code

Andrey Cheptsov and others added 3 commits November 19, 2026 15:16
- Step 1: meter tokens at the gateway's model endpoint
- Step 2.1: serve the model endpoint at /<project>
- Step 2.2: export models to other projects
- Step 2.3: imported models and usage in the UI

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Tabs: Details, Exports and Metering. Exports lists the model's exports
  with the projects that import the model, as expandable rows.
- The Code button becomes Endpoint. Its dialog has a Project select when
  several of the user's projects have the model, and stays available for
  a project's own stopped model, with the URL of its last run.
- General shows Name, Project, Run and Status. The URL, the type and the
  chat's System box go.
- Metering: the table is Tokens. The filter, the time range and the
  Day/Month switch are on one row, rows are per period and project with a
  filter by user, and the last row is the total.
- The chat keeps its messages in the browser, shows the user's avatar
  and has a Clear button.
- A model's address keeps the slashes of the model's name.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Total row of the Tokens table used a public Cloudscape colour, which
equals the column headers' colour in light mode only. The headers' colour
is an internal token, so the row gets its two values from the stylesheet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterschmidt85
peterschmidt85 marked this pull request as ready for review November 19, 2026 23:18
Andrey Cheptsov and others added 6 commits November 20, 2026 15:35
The tab is Metering, while its folder, component, helpers and locale keys
said usage. They say metering now. The texts of the tab don't change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`/api/registry/imports/list` returned the gateway's stored `ImportedModel`,
so the server imported a model of the gateway's repo. The route now has
its own schema, `ImportListItem`, with the same four fields.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The exports checks and the in-server proxy each read the `model` of a
service from `ServiceSpec.options`, with different validation. Both now
call `ServiceSpec.get_model_options()`.

Also fixes the error of an export whose service runs on a gateway without
a domain: it said the domain has `${{ run.project_name }}`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The model endpoint cancels the request to a service when its client
disconnects, and the requests to a service share one client. With httpx,
some cancelled requests never release their connections. Cancelling
through an anyio scope, as the code did, doesn't prevent that.

Measured locally with 300 requests at a time over a Unix socket, a third
cancelled before the response and a third during it, 30,000 requests per
case:

- httpx, anyio scope: 17 connections never released
- httpx, Task.cancel(): 0 (1 of 9,000 with 60 requests at a time)
- httpx2, either: 0

So `ServiceClient` is an httpx2 client, and the request is cancelled as a
plain task. The in-server proxy shares `ServiceClient`, so httpx2 joins
the `server` extra. anyio is no longer a direct dependency.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`POST /api/metering/collect` had no router test: one test now checks
that it returns the batches after the given one.

Drops `test_only_project_admin_can_add_models`, which repeats what the
export tests already check for every update.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant