Repository navigation
Meter model tokens and export models to projects - #4499
Open
peterschmidt85 wants to merge 9 commits into
Open
peterschmidt85 wants to merge 9 commits into
peterschmidt85 wants to merge 9 commits into
Conversation
- Step 1: meter tokens at the gateway's model endpoint - Step 2.1: serve the model endpoint at /<project> - Step 2.2: export models to other projects - Step 2.3: imported models and usage in the UI Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- Tabs: Details, Exports and Metering. Exports lists the model's exports with the projects that import the model, as expandable rows. - The Code button becomes Endpoint. Its dialog has a Project select when several of the user's projects have the model, and stays available for a project's own stopped model, with the URL of its last run. - General shows Name, Project, Run and Status. The URL, the type and the chat's System box go. - Metering: the table is Tokens. The filter, the time range and the Day/Month switch are on one row, rows are per period and project with a filter by user, and the last row is the total. - The chat keeps its messages in the browser, shows the user's avatar and has a Clear button. - A model's address keeps the slashes of the model's name. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Total row of the Tokens table used a public Cloudscape colour, which equals the column headers' colour in light mode only. The headers' colour is an internal token, so the row gets its two values from the stylesheet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
peterschmidt85
marked this pull request as ready for review
November 19, 2026 23:18
The tab is Metering, while its folder, component, helpers and locale keys said usage. They say metering now. The texts of the tab don't change. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`/api/registry/imports/list` returned the gateway's stored `ImportedModel`, so the server imported a model of the gateway's repo. The route now has its own schema, `ImportListItem`, with the same four fields. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The exports checks and the in-server proxy each read the `model` of a
service from `ServiceSpec.options`, with different validation. Both now
call `ServiceSpec.get_model_options()`.
Also fixes the error of an export whose service runs on a gateway without
a domain: it said the domain has `${{ run.project_name }}`.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The model endpoint cancels the request to a service when its client disconnects, and the requests to a service share one client. With httpx, some cancelled requests never release their connections. Cancelling through an anyio scope, as the code did, doesn't prevent that. Measured locally with 300 requests at a time over a Unix socket, a third cancelled before the response and a third during it, 30,000 requests per case: - httpx, anyio scope: 17 connections never released - httpx, Task.cancel(): 0 (1 of 9,000 with 60 requests at a time) - httpx2, either: 0 So `ServiceClient` is an httpx2 client, and the request is cancelled as a plain task. The in-server proxy shares `ServiceClient`, so httpx2 joins the `server` extra. anyio is no longer a direct dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`POST /api/metering/collect` had no router test: one test now checks that it returns the batches after the given one. Drops `test_only_project_admin_can_add_models`, which repeats what the export tests already check for every update. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A project can now export its models to other projects. On a gateway, every project gets its own model endpoint, and the model endpoint meters the tokens of each request.
What changes for users
A model endpoint per project
On a gateway, a project's models are at
https://gateway.<domain>/<project>, an OpenAI-compatible API with/modelsand/chat/completions.dstack applyshows it below the service URL:Members of the project access the model there with their own
dstacktokens.Before, the
Model ... is published atline had the service URL plus/v1. Without a gateway, it still does.Models in exports
An export can hold models, by name. Members of an importer project access an exported model at their own project's model endpoint. The gateway itself doesn't have to be exported.
$ dstack export create my-models --model Qwen/Qwen3.8-27B-FP8 --importer team-b NAME FLEETS GATEWAYS MODELS IMPORTERS my-models - - Qwen/Qwen3.8-27B-FP8 team-bIn
team-b:$ dstack import list -v NAME FLEETS GATEWAYS MODEL URL team-a/my-models - - Qwen/Qwen3.8-27B-FP8 https://gateway.example.com/team-b $ curl https://gateway.example.com/team-b/chat/completions \ -H "Authorization: Bearer $DSTACK_TOKEN" \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3.8-27B-FP8", "messages": [{"role": "user", "content": "Hello"}]}'A model can be exported before any service declares it, and it stays exported when its service is stopped and applied again. Importer projects can access it while a service of the exporter project runs it with the
openaiformat on a gateway whosedomainhas no${{ run.project_name }}. See Exports.Token metering
The model endpoint meters the input, cached input and output tokens of every request to an
openaimodel, per project, user and model, also for a stream that the client stops.dstackitself stores nothing. The server collects the counts from the gateway through a hook, which dstack Factory registers to store and show them.Chat completions pass through
For
openaimodels, the model endpoint forwards the request and the service's response as they are. Requests withstream_options,max_completion_tokens, tool calls or content parts used to get 422, and responses lostusagein stream chunks andprompt_tokens_details. Now they work, and an error of the service reaches the client with the service's own status and body.UI
Modelspage lists a project's own and imported models, with their status.Endpointbutton shows how to access the model at the user's project.Exportstab lists the model's exports and the projects that import it (where the server has exports).Meteringtab shows the model's tokens by day or month and by project (dstack Factory and dstack Sky).The screenshots are from the end-to-end test, where
exporter-projexports the model toimporter-projandimporter2-proj.The
Modelspage:A model's page:
The
Exportstab:The
Meteringtab:Breaking changes
https://gateway.<domain>tohttps://gateway.<domain>/<project>. Requests to the old address get 404.openaimodel on a gateway, the model URL thatdstackshows is now the model endpoint. It was the service URL plus/v1. This is the URL indstack apply, inservice.model.base_urlof the API and inRun.service_modelof the Python API. The service URL itself doesn't change. The model endpoint always takes adstacktoken of a project member, also when the service hasauth: false.openaimodels, an error response of the service is returned as it is, so the client gets the engine's own error. Before,dstackput its body into{"detail": ...}.gateway, because that host is the model endpoint's.Implementation highlights
GatewayProxyMetersums the tokens per hour, user, project, service and export, and writes them to batch files every 5 seconds. The server collects the batches withPOST /api/metering/collect, from hooks of the gateway replica pipeline (register_process_running_gateway_replica_hook).dstackregisters no hook.forward_chat_completionsreplaces the typed OpenAI client foropenaimodels. It asks the service for token counts on every stream chunk, and removes them from what the client gets unless the client asked for them.httpx2client, in the gateway's model endpoint and in the in-server proxy: a request to a service is cancelled when its client disconnects, and withhttpxsome cancelled requests never released their connections (17 of 30,000 in a local test, none withhttpx2). Sohttpx2is now in theserverextra too.gateway.<domain>/<project>/...to the gateway app, which finds the project's models, own and imported. The project name in the path is not case-sensitive, and an unknown project gets 404.GatewayReplicaPipelineregisters an import per importer project on the gateway, and removes it when the export, the importer or the service goes away.server/services/exported_models.py.dstackrefuses an export, an in-place service update or a gateway domain change that would break them.--timeout-graceful-shutdown 10,TimeoutStopSec=30).exported_models.Modelslist reads runs page by page: every active run, and the latest 1,000 runs whenActive modelsis off.Known issues
Tests
npm run test:auth).Qwen/Qwen3.8-27B-FP8on SGLang: the steps of the user flow below through the API, the CLI and the UI.Design
The PR implements Steps 1 to 2.3 of the metering plan:
Storing the metering records and the metering API are dstack Factory's part, in a PR of its own.
🤖 Generated with Claude Code