Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 34 additions & 8 deletions contributing/PROXY.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

`dstack-proxy` acts as a reverse proxy and load balancer for services.

The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx. It routes requests based on the `/proxy/services/<project>/<service>` path. It performs load balancing by selecting a random service replica for each forwarded request.
The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx2. It routes requests based on the `/proxy/services/<project>/<service>` path. It performs load balancing by selecting a random service replica for each forwarded request.

The gateway uses Nginx. It automatically maintains Nginx configs for each service in `/etc/nginx/sites-enabled/*`. Each service is published at a subdomain. Nginx performs load balancing using the round-robin method. Nginx forwards requests directly to service replicas, so traffic does not go through the gateway app.

Expand All @@ -30,9 +30,17 @@ The in-server proxy validates the tokens by querying the correct token from the

On gateways, Nginx makes a headers-only subrequest to the gateway app to check authorization for each incoming request. The gateway app then makes a request to the `dstack` server to validate the token. Responses from the `dstack` server are cached for 60 seconds. If the `dstack` server does not respond, the incoming request is denied.

For requests to models, the gateway app also needs the user to meter the tokens for, so it requests the user of each token from the `dstack` server and caches it for an hour.

### OpenAI-compatible API

The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project, convert between OpenAI and TGI request formats, and forward requests to the correct service.
The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project and forward requests to the correct service.

Requests to models with the `openai` format and their responses are not converted. `dstack-proxy` reads `model` and `stream` from the request and `usage` from the response, and forwards the bodies as they are, with two exceptions for streaming requests: it sets `stream_options` to get token counts on every chunk (see [Token metering](#token-metering)), and it removes from the stream the `usage` that the client did not ask for. Of the service's response headers, it keeps `Content-Type` and `Retry-After`. Requests to models with the `tgi` format are converted between the OpenAI and TGI formats through typed schemas.

Only members of the project and global admins can use the interface.

On gateways, the models of a project include the models it imports from other projects through exports. Where exports are enabled, `GatewayReplicaPipeline` registers an import on the gateway's replicas for each exported model that a service on the gateway can serve to importer projects and each importer project: the importer project, the export, the exporter project, and the model's name. The gateway keeps one import per importer project and model name, and registering another one replaces it. When a request arrives, the gateway app finds the model and its service in the exporter project by that name, so the import follows the model to whichever service has registered it. An import is skipped if the importer project has its own model with the same name, and while the exporter project has no such model with the `openai` format.

The in-server proxy forwards requests directly to service replicas.

Expand All @@ -42,16 +50,25 @@ sequenceDiagram
dstack server->>Service replica: http+unix:///path/to/ssh/socket
```

The gateway uses Nginx to forward requests so they are included in access logs and service stats.
The gateway uses Nginx to forward requests so they are included in access logs and service stats. The project is part of the URL path, so one domain serves the models of every project on the gateway.

```mermaid
sequenceDiagram
User->>Nginx: http(s)://gateway.<gateway>/
User->>Nginx: http(s)://gateway.<gateway>/<project>/
Nginx->>Gateway app: http://localhost:8000/api/models/<project>/
Gateway app->>Nginx: http://<service>.<gateway>/
Nginx->>Service replica: http+unix:///path/to/ssh/socket
```

### Token metering

`dstack-proxy` meters the tokens of requests to models with the `openai` format: input, cached input, and output tokens per hour, project, user, service, and export. The project is the one in the URL of the request. For a model that the project imports, the service is the exporter project's, and the export is the one that the request came through. For a project's own models, there is no export.

- For streaming requests, it asks the model for token counts on every chunk (`stream_options.include_usage` and `stream_options.continuous_usage_stats`) and adds what is new since the previous chunk as each chunk arrives. So a stream stopped by the client is metered up to its last chunk. `usage` that the client did not ask for is removed from the stream. If the model reports per-choice counts on each chunk, as vLLM and SGLang do, a stopped stream with `n` above 1 is metered with the tokens of its longest choice.
- For non-streaming requests, the tokens are added when the response ends. If the client disconnects earlier, `dstack-proxy` closes the connection to the service, and nothing is metered.

Metering is only supported on gateways. The gateway app keeps the counts in memory, writes them every 5 seconds to a numbered batch file in `~/dstack/metering/`, and returns the batches from `POST /api/metering/collect`. The caller passes the number of the last batch it has saved, and the gateway deletes the batches up to that number. Batches that nobody collects are deleted after 7 days. The open-source `dstack` server does not collect metering. An extension of the server can collect it in a hook that `GatewayReplicaPipeline` calls each time it processes a running gateway replica (`register_process_running_gateway_replica_hook` in `server/background/pipeline_tasks/gateway_replicas.py`).

### Stats collector

`dstack-proxy` collects service usage stats that are then used by the `dstack` server for autoscaling. Stats collection is only supported on gateways and is implemented by reading `/var/log/nginx/dstack.access.log`
Expand Down Expand Up @@ -81,11 +98,11 @@ Gateway-to-server communication happens over an SSH connection that is establish

The in-server proxy repo implementation fetches the models from the database. It doesn't write to the database, as all the details about services and replicas are written by the `dstack` server.

The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services and replicas are registered.
The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services, replicas, and imports are registered.

## Dependency injection

When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo and for auth checks. The in-server proxy and the gateway provide their own implementations for both interfaces.
When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo, for auth checks, and for token metering. The in-server proxy and the gateway provide their own implementations for these interfaces.

```mermaid
classDiagram
Expand All @@ -100,6 +117,12 @@ classDiagram
}
BaseProxyAuthProvider <|-- ServerProxyAuthProvider
BaseProxyAuthProvider <|-- GatewayProxyAuthProvider

class BaseProxyMeter {
<<abstract>>
}
BaseProxyMeter <|-- ServerProxyMeter
BaseProxyMeter <|-- GatewayProxyMeter
```

`dstack-proxy` then uses dependency injection to select the relevant implementation. Both the in-server proxy and the gateway provide an "injector" class that is used to obtain concrete interface implementations.
Expand All @@ -110,22 +133,25 @@ classDiagram
<<abstract>>
+get_repo() BaseProxyRepo
+get_auth_provider() BaseProxyAuthProvider
+get_meter() BaseProxyMeter
}
class ServerProxyDependencyInjector {
+get_repo() ServerProxyRepo
+get_auth_provider() ServerProxyAuthProvider
+get_meter() ServerProxyMeter
}
class GatewayDependencyInjector {
+get_repo() GatewayProxyRepo
+get_auth_provider() GatewayProxyAuthProvider
+get_meter() GatewayProxyMeter
}
ProxyDependencyInjector <|-- ServerProxyDependencyInjector
ProxyDependencyInjector <|-- GatewayDependencyInjector
```

An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, etc.
An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, `get_proxy_meter`, etc.

There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, etc.
There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, `get_gateway_proxy_meter`, etc.

## Gateway operations

Expand Down
3 changes: 3 additions & 0 deletions frontend/jest.auth.config.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ module.exports = {
'<rootDir>/src/App/auth.test.tsx',
'<rootDir>/src/App/login.test.tsx',
'<rootDir>/src/services/preset.test.tsx',
'<rootDir>/src/services/run.test.ts',
'<rootDir>/src/libs/run.test.ts',
'<rootDir>/src/pages/Models/**/*.test.{ts,tsx}',
],
transform: {
'\\.[jt]sx?$': 'babel-jest',
Expand Down
4 changes: 3 additions & 1 deletion frontend/product.config.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@ const oss = {
id: 'oss',
name: 'dstack',
hasEvents: false,
/** Whether the server has exports. Without them, it refuses to list a project's imports. */
hasExports: false,
hasBilling: false,
hasPresets: false,
/** Whether this product allows templates, even when none are configured. */
hasTemplates: false,
isSky: false,
};
const enterprise = { ...oss, id: 'enterprise', name: 'dstack Enterprise', hasEvents: true, hasTemplates: true };
const enterprise = { ...oss, id: 'enterprise', name: 'dstack Enterprise', hasEvents: true, hasExports: true, hasTemplates: true };
const factory = { ...enterprise, id: 'factory', name: 'dstack Factory', hasBilling: true, hasPresets: true };
const sky = { ...factory, id: 'sky', name: 'dstack Sky', isSky: true };

Expand Down
22 changes: 19 additions & 3 deletions frontend/src/App/login.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -148,9 +148,25 @@ test('the token page does not wait for provider discovery', () => {
});

test('Factory inherits Enterprise features and Sky inherits Factory features', () => {
expect(getProductConfig('enterprise')).toMatchObject({ hasEvents: true, hasBilling: false, hasPresets: false });
expect(getProductConfig('enterprise')).toMatchObject({
hasEvents: true,
hasExports: true,
hasBilling: false,
hasPresets: false,
});
for (const version of ['factory', 'sky']) {
expect(getProductConfig(version)).toMatchObject({ hasEvents: true, hasBilling: true, hasPresets: true });
expect(getProductConfig(version)).toMatchObject({
hasEvents: true,
hasExports: true,
hasBilling: true,
hasPresets: true,
});
}
expect(getProductConfig(undefined)).toMatchObject({ id: 'oss', hasEvents: false, hasBilling: false, hasPresets: false });
expect(getProductConfig(undefined)).toMatchObject({
id: 'oss',
hasEvents: false,
hasExports: false,
hasBilling: false,
hasPresets: false,
});
});
9 changes: 9 additions & 0 deletions frontend/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,10 @@ export const API = {
GPUS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/gpus/list`,
// GPUS
TEMPLATES_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/templates/list`,
// IMPORTS
IMPORTS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/imports/list`,
// EXPORTS
EXPORTS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/exports/list`,
},

BACKENDS: {
Expand Down Expand Up @@ -196,6 +200,11 @@ export const API = {
LIST: () => `${API.PRESET.BASE()}/list`,
},

METERING: {
BASE: () => `${API.BASE()}/metering`,
LIST: () => `${API.METERING.BASE()}/list`,
},

USER_PUBLIC_KEYS: {
BASE: () => `${API.BASE()}/users/public_keys`,
LIST: () => `${API.USER_PUBLIC_KEYS.BASE()}/list`,
Expand Down
18 changes: 18 additions & 0 deletions frontend/src/components/UserAvatar/index.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import React from 'react';
import Avatar from 'react-avatar';
import { product } from 'product';

type Props = {
userName?: string;
// In pixels
size: number;
};

/**
* A user's avatar: their GitHub one in dstack Sky, else their initials.
*/
export const UserAvatar: React.FC<Props> = ({ userName, size }) => {
const avatarProps = product.isSky ? { githubHandle: userName } : { name: userName };

return <Avatar {...avatarProps} size={`${size}px`} round />;
};
1 change: 1 addition & 0 deletions frontend/src/components/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,4 +105,5 @@ export type { IProps as ButtonProps } from './Button';
export { Code } from './Code';
export { Hotspot } from './Hotspot';
export { TokenVisibilityButton } from './TokenVisibilityButton';
export { UserAvatar } from './UserAvatar';
export type { IProps as TabsProps } from './Tabs';
6 changes: 2 additions & 4 deletions frontend/src/layouts/AppLayout/index.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import React from 'react';
import Avatar from 'react-avatar';
import { createPortal } from 'react-dom';
import { useTranslation } from 'react-i18next';
import { useNavigate } from 'react-router-dom';
Expand All @@ -17,6 +16,7 @@ import {
SideNavigation,
Tabs,
TopNavigation,
UserAvatar,
} from 'components';

import { DISCORD_URL, DOCS_URL } from 'consts';
Expand Down Expand Up @@ -125,8 +125,6 @@ const AppLayout: React.FC<{ children: React.ReactNode }> = ({ children }) => {

const isVisibleInfoTab = helpPanelContent.header || helpPanelContent.footer || helpPanelContent.body;

const avatarProps = product.isSky ? { githubHandle: userName } : { name: userName };

const onChangeSystemModeToggle: SideNavigationProps['onFollow'] = (event) => {
event.preventDefault();

Expand Down Expand Up @@ -193,7 +191,7 @@ const AppLayout: React.FC<{ children: React.ReactNode }> = ({ children }) => {
type: 'menu-dropdown',
text: (
<div className={styles.userAvatar}>
<Avatar {...avatarProps} size="40px" />
<UserAvatar userName={userName} size={40} />
</div>
),
items: profileActions,
Expand Down
25 changes: 25 additions & 0 deletions frontend/src/libs/run.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { getExtendedModelFromRun } from './run';

const BASE_URL = 'https://gateway.example.com/exporter-proj';

const makeRun = (status: IRun['status']) =>
({
id: 'run-id',
project_name: 'exporter-proj',
user: 'alice',
submitted_at: '2026-12-03T09:02:00+00:00',
status,
run_spec: { run_name: 'qwen-service' },
service: {
url: '/proxy/services/exporter-proj/qwen-service/',
model: { name: 'Qwen/Qwen3.6-27B-FP8', type: 'chat', base_url: BASE_URL },
},
}) as IRun;

describe('Model of a run', () => {
test.each(['running', 'terminated'] as const)('the model of a %s run has the URL of the run', (status) => {
const model = getExtendedModelFromRun(makeRun(status));

expect(model?.endpoints).toEqual([{ project_name: 'exporter-proj', base_url: BASE_URL }]);
});
});
48 changes: 35 additions & 13 deletions frontend/src/libs/run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,18 @@ export const getStatusIconColor = (
}
};

/**
* A finished run shows the status and the termination reason of its latest job submission.
*/
export const getRunStatus = (run: IRun) => {
const isFinished = finishedRunStatuses.includes(run.status);

return {
status: isFinished ? (run.latest_job_submission?.status ?? run.status) : run.status,
terminationReason: isFinished ? run.latest_job_submission?.termination_reason : null,
};
};

export const getRunStatusMessage = (run: IRun): string => {
if (finishedRunStatuses.includes(run.status) && run.latest_job_submission?.status_message) {
return capitalize(run.latest_job_submission.status_message);
Expand Down Expand Up @@ -93,22 +105,32 @@ export const getRunPriority = (run: IRun): number | null => {
};

export const getExtendedModelFromRun = (run: IRun): IModelExtended | null => {
if (!run?.service?.model) return null;
const model = run?.service?.model;

if (!model) return null;

const { status, terminationReason } = getRunStatus(run);

return {
...(run.service?.model ?? {}),
id: run.id,
name: model.name,
project_name: run.project_name,
run_name: run?.run_spec.run_name ?? 'No run name',
user: run.user,
resources: run.latest_job_submission?.job_provisioning_data?.instance_type?.resources
? formatResources(run.latest_job_submission.job_provisioning_data.instance_type.resources)
: null,
price: run.latest_job_submission?.job_provisioning_data?.price ?? null,
submitted_at: run.submitted_at,
repository: getRepoNameFromRun(run),
backend: run.latest_job_submission?.job_provisioning_data?.backend ?? null,
region: run.latest_job_submission?.job_provisioning_data?.region ?? null,
// A finished run keeps its URL: the model is back at it when the service is applied again
endpoints: [{ project_name: run.project_name, base_url: model.base_url }],
run: {
id: run.id,
run_name: run?.run_spec.run_name ?? 'No run name',
status,
termination_reason: terminationReason,
status_message: getRunStatusMessage(run),
user: run.user,
resources: run.latest_job_submission?.job_provisioning_data?.instance_type?.resources
? formatResources(run.latest_job_submission.job_provisioning_data.instance_type.resources)
: null,
price: run.latest_job_submission?.job_provisioning_data?.price ?? null,
submitted_at: run.submitted_at,
repository: getRepoNameFromRun(run),
backend: run.latest_job_submission?.job_provisioning_data?.backend ?? null,
},
};
};

Expand Down
Loading
Loading