diff --git a/contributing/PROXY.md b/contributing/PROXY.md index cc5c2f6831..f903506b9e 100644 --- a/contributing/PROXY.md +++ b/contributing/PROXY.md @@ -12,7 +12,7 @@ `dstack-proxy` acts as a reverse proxy and load balancer for services. -The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx. It routes requests based on the `/proxy/services//` path. It performs load balancing by selecting a random service replica for each forwarded request. +The in-server proxy uses a custom reverse proxy implementation based on FastAPI and httpx2. It routes requests based on the `/proxy/services//` path. It performs load balancing by selecting a random service replica for each forwarded request. The gateway uses Nginx. It automatically maintains Nginx configs for each service in `/etc/nginx/sites-enabled/*`. Each service is published at a subdomain. Nginx performs load balancing using the round-robin method. Nginx forwards requests directly to service replicas, so traffic does not go through the gateway app. @@ -30,9 +30,17 @@ The in-server proxy validates the tokens by querying the correct token from the On gateways, Nginx makes a headers-only subrequest to the gateway app to check authorization for each incoming request. The gateway app then makes a request to the `dstack` server to validate the token. Responses from the `dstack` server are cached for 60 seconds. If the `dstack` server does not respond, the incoming request is denied. +For requests to models, the gateway app also needs the user to meter the tokens for, so it requests the user of each token from the `dstack` server and caches it for an hour. + ### OpenAI-compatible API -The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project, convert between OpenAI and TGI request formats, and forward requests to the correct service. +The OpenAI interface emulates the real OpenAI API for chat completion models. It can list running models in the project and forward requests to the correct service. + +Requests to models with the `openai` format and their responses are not converted. `dstack-proxy` reads `model` and `stream` from the request and `usage` from the response, and forwards the bodies as they are, with two exceptions for streaming requests: it sets `stream_options` to get token counts on every chunk (see [Token metering](#token-metering)), and it removes from the stream the `usage` that the client did not ask for. Of the service's response headers, it keeps `Content-Type` and `Retry-After`. Requests to models with the `tgi` format are converted between the OpenAI and TGI formats through typed schemas. + +Only members of the project and global admins can use the interface. + +On gateways, the models of a project include the models it imports from other projects through exports. Where exports are enabled, `GatewayReplicaPipeline` registers an import on the gateway's replicas for each exported model that a service on the gateway can serve to importer projects and each importer project: the importer project, the export, the exporter project, and the model's name. The gateway keeps one import per importer project and model name, and registering another one replaces it. When a request arrives, the gateway app finds the model and its service in the exporter project by that name, so the import follows the model to whichever service has registered it. An import is skipped if the importer project has its own model with the same name, and while the exporter project has no such model with the `openai` format. The in-server proxy forwards requests directly to service replicas. @@ -42,16 +50,25 @@ sequenceDiagram dstack server->>Service replica: http+unix:///path/to/ssh/socket ``` -The gateway uses Nginx to forward requests so they are included in access logs and service stats. +The gateway uses Nginx to forward requests so they are included in access logs and service stats. The project is part of the URL path, so one domain serves the models of every project on the gateway. ```mermaid sequenceDiagram - User->>Nginx: http(s)://gateway./ + User->>Nginx: http(s)://gateway.// Nginx->>Gateway app: http://localhost:8000/api/models// Gateway app->>Nginx: http://./ Nginx->>Service replica: http+unix:///path/to/ssh/socket ``` +### Token metering + +`dstack-proxy` meters the tokens of requests to models with the `openai` format: input, cached input, and output tokens per hour, project, user, service, and export. The project is the one in the URL of the request. For a model that the project imports, the service is the exporter project's, and the export is the one that the request came through. For a project's own models, there is no export. + +- For streaming requests, it asks the model for token counts on every chunk (`stream_options.include_usage` and `stream_options.continuous_usage_stats`) and adds what is new since the previous chunk as each chunk arrives. So a stream stopped by the client is metered up to its last chunk. `usage` that the client did not ask for is removed from the stream. If the model reports per-choice counts on each chunk, as vLLM and SGLang do, a stopped stream with `n` above 1 is metered with the tokens of its longest choice. +- For non-streaming requests, the tokens are added when the response ends. If the client disconnects earlier, `dstack-proxy` closes the connection to the service, and nothing is metered. + +Metering is only supported on gateways. The gateway app keeps the counts in memory, writes them every 5 seconds to a numbered batch file in `~/dstack/metering/`, and returns the batches from `POST /api/metering/collect`. The caller passes the number of the last batch it has saved, and the gateway deletes the batches up to that number. Batches that nobody collects are deleted after 7 days. The open-source `dstack` server does not collect metering. An extension of the server can collect it in a hook that `GatewayReplicaPipeline` calls each time it processes a running gateway replica (`register_process_running_gateway_replica_hook` in `server/background/pipeline_tasks/gateway_replicas.py`). + ### Stats collector `dstack-proxy` collects service usage stats that are then used by the `dstack` server for autoscaling. Stats collection is only supported on gateways and is implemented by reading `/var/log/nginx/dstack.access.log` @@ -81,11 +98,11 @@ Gateway-to-server communication happens over an SSH connection that is establish The in-server proxy repo implementation fetches the models from the database. It doesn't write to the database, as all the details about services and replicas are written by the `dstack` server. -The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services and replicas are registered. +The gateway maintains its own in-memory storage repo. A copy of the repo is also stored in `~/dstack/state-v2.json`. The copy is updated on every write operation and is used for data recovery after restarts. The repo is populated when new services, replicas, and imports are registered. ## Dependency injection -When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo and for auth checks. The in-server proxy and the gateway provide their own implementations for both interfaces. +When a module has to be implemented differently for the in-server proxy and the gateway, `dstack-proxy` uses interfaces with multiple implementations. For example, there are common interfaces for the storage repo, for auth checks, and for token metering. The in-server proxy and the gateway provide their own implementations for these interfaces. ```mermaid classDiagram @@ -100,6 +117,12 @@ classDiagram } BaseProxyAuthProvider <|-- ServerProxyAuthProvider BaseProxyAuthProvider <|-- GatewayProxyAuthProvider + + class BaseProxyMeter { + <> + } + BaseProxyMeter <|-- ServerProxyMeter + BaseProxyMeter <|-- GatewayProxyMeter ``` `dstack-proxy` then uses dependency injection to select the relevant implementation. Both the in-server proxy and the gateway provide an "injector" class that is used to obtain concrete interface implementations. @@ -110,22 +133,25 @@ classDiagram <> +get_repo() BaseProxyRepo +get_auth_provider() BaseProxyAuthProvider + +get_meter() BaseProxyMeter } class ServerProxyDependencyInjector { +get_repo() ServerProxyRepo +get_auth_provider() ServerProxyAuthProvider + +get_meter() ServerProxyMeter } class GatewayDependencyInjector { +get_repo() GatewayProxyRepo +get_auth_provider() GatewayProxyAuthProvider + +get_meter() GatewayProxyMeter } ProxyDependencyInjector <|-- ServerProxyDependencyInjector ProxyDependencyInjector <|-- GatewayDependencyInjector ``` -An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, etc. +An instance of the relevant injector class is stored in the FastAPI global app state and can be accessed from FastAPI path operations. There are helper functions that can be used as FastAPI path operation dependencies to obtain the injector or a module implementation: `get_injector`, `get_proxy_repo`, `get_proxy_auth_provider`, `get_proxy_meter`, etc. -There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, etc. +There are also similar gateway-specific helper functions to obtain gateway-specific module implementations: `get_gateway_proxy_repo` (guaranteed to return the gateway repo, which has more methods than the base repo interface), `get_nginx`, `get_stats_collector`, `get_gateway_proxy_meter`, etc. ## Gateway operations diff --git a/frontend/jest.auth.config.cjs b/frontend/jest.auth.config.cjs index c8616f64a9..3a1d157759 100644 --- a/frontend/jest.auth.config.cjs +++ b/frontend/jest.auth.config.cjs @@ -7,6 +7,9 @@ module.exports = { '/src/App/auth.test.tsx', '/src/App/login.test.tsx', '/src/services/preset.test.tsx', + '/src/services/run.test.ts', + '/src/libs/run.test.ts', + '/src/pages/Models/**/*.test.{ts,tsx}', ], transform: { '\\.[jt]sx?$': 'babel-jest', diff --git a/frontend/product.config.cjs b/frontend/product.config.cjs index 4d9e48a975..81e15e919b 100644 --- a/frontend/product.config.cjs +++ b/frontend/product.config.cjs @@ -3,13 +3,15 @@ const oss = { id: 'oss', name: 'dstack', hasEvents: false, + /** Whether the server has exports. Without them, it refuses to list a project's imports. */ + hasExports: false, hasBilling: false, hasPresets: false, /** Whether this product allows templates, even when none are configured. */ hasTemplates: false, isSky: false, }; -const enterprise = { ...oss, id: 'enterprise', name: 'dstack Enterprise', hasEvents: true, hasTemplates: true }; +const enterprise = { ...oss, id: 'enterprise', name: 'dstack Enterprise', hasEvents: true, hasExports: true, hasTemplates: true }; const factory = { ...enterprise, id: 'factory', name: 'dstack Factory', hasBilling: true, hasPresets: true }; const sky = { ...factory, id: 'sky', name: 'dstack Sky', isSky: true }; diff --git a/frontend/src/App/login.test.tsx b/frontend/src/App/login.test.tsx index 39e127d7f2..63139f87d6 100644 --- a/frontend/src/App/login.test.tsx +++ b/frontend/src/App/login.test.tsx @@ -148,9 +148,25 @@ test('the token page does not wait for provider discovery', () => { }); test('Factory inherits Enterprise features and Sky inherits Factory features', () => { - expect(getProductConfig('enterprise')).toMatchObject({ hasEvents: true, hasBilling: false, hasPresets: false }); + expect(getProductConfig('enterprise')).toMatchObject({ + hasEvents: true, + hasExports: true, + hasBilling: false, + hasPresets: false, + }); for (const version of ['factory', 'sky']) { - expect(getProductConfig(version)).toMatchObject({ hasEvents: true, hasBilling: true, hasPresets: true }); + expect(getProductConfig(version)).toMatchObject({ + hasEvents: true, + hasExports: true, + hasBilling: true, + hasPresets: true, + }); } - expect(getProductConfig(undefined)).toMatchObject({ id: 'oss', hasEvents: false, hasBilling: false, hasPresets: false }); + expect(getProductConfig(undefined)).toMatchObject({ + id: 'oss', + hasEvents: false, + hasExports: false, + hasBilling: false, + hasPresets: false, + }); }); diff --git a/frontend/src/api.ts b/frontend/src/api.ts index 5092cc9f9c..975859fe93 100644 --- a/frontend/src/api.ts +++ b/frontend/src/api.ts @@ -126,6 +126,10 @@ export const API = { GPUS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/gpus/list`, // GPUS TEMPLATES_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/templates/list`, + // IMPORTS + IMPORTS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/imports/list`, + // EXPORTS + EXPORTS_LIST: (projectName: IProject['project_name']) => `${API.BASE()}/project/${projectName}/exports/list`, }, BACKENDS: { @@ -196,6 +200,11 @@ export const API = { LIST: () => `${API.PRESET.BASE()}/list`, }, + METERING: { + BASE: () => `${API.BASE()}/metering`, + LIST: () => `${API.METERING.BASE()}/list`, + }, + USER_PUBLIC_KEYS: { BASE: () => `${API.BASE()}/users/public_keys`, LIST: () => `${API.USER_PUBLIC_KEYS.BASE()}/list`, diff --git a/frontend/src/components/UserAvatar/index.tsx b/frontend/src/components/UserAvatar/index.tsx new file mode 100644 index 0000000000..df5db40fef --- /dev/null +++ b/frontend/src/components/UserAvatar/index.tsx @@ -0,0 +1,18 @@ +import React from 'react'; +import Avatar from 'react-avatar'; +import { product } from 'product'; + +type Props = { + userName?: string; + // In pixels + size: number; +}; + +/** + * A user's avatar: their GitHub one in dstack Sky, else their initials. + */ +export const UserAvatar: React.FC = ({ userName, size }) => { + const avatarProps = product.isSky ? { githubHandle: userName } : { name: userName }; + + return ; +}; diff --git a/frontend/src/components/index.ts b/frontend/src/components/index.ts index 288f810065..f51a0ede01 100644 --- a/frontend/src/components/index.ts +++ b/frontend/src/components/index.ts @@ -105,4 +105,5 @@ export type { IProps as ButtonProps } from './Button'; export { Code } from './Code'; export { Hotspot } from './Hotspot'; export { TokenVisibilityButton } from './TokenVisibilityButton'; +export { UserAvatar } from './UserAvatar'; export type { IProps as TabsProps } from './Tabs'; diff --git a/frontend/src/layouts/AppLayout/index.tsx b/frontend/src/layouts/AppLayout/index.tsx index 42fb524fc1..e528335283 100644 --- a/frontend/src/layouts/AppLayout/index.tsx +++ b/frontend/src/layouts/AppLayout/index.tsx @@ -1,5 +1,4 @@ import React from 'react'; -import Avatar from 'react-avatar'; import { createPortal } from 'react-dom'; import { useTranslation } from 'react-i18next'; import { useNavigate } from 'react-router-dom'; @@ -17,6 +16,7 @@ import { SideNavigation, Tabs, TopNavigation, + UserAvatar, } from 'components'; import { DISCORD_URL, DOCS_URL } from 'consts'; @@ -125,8 +125,6 @@ const AppLayout: React.FC<{ children: React.ReactNode }> = ({ children }) => { const isVisibleInfoTab = helpPanelContent.header || helpPanelContent.footer || helpPanelContent.body; - const avatarProps = product.isSky ? { githubHandle: userName } : { name: userName }; - const onChangeSystemModeToggle: SideNavigationProps['onFollow'] = (event) => { event.preventDefault(); @@ -193,7 +191,7 @@ const AppLayout: React.FC<{ children: React.ReactNode }> = ({ children }) => { type: 'menu-dropdown', text: (
- +
), items: profileActions, diff --git a/frontend/src/libs/run.test.ts b/frontend/src/libs/run.test.ts new file mode 100644 index 0000000000..5133f5261b --- /dev/null +++ b/frontend/src/libs/run.test.ts @@ -0,0 +1,25 @@ +import { getExtendedModelFromRun } from './run'; + +const BASE_URL = 'https://gateway.example.com/exporter-proj'; + +const makeRun = (status: IRun['status']) => + ({ + id: 'run-id', + project_name: 'exporter-proj', + user: 'alice', + submitted_at: '2026-12-03T09:02:00+00:00', + status, + run_spec: { run_name: 'qwen-service' }, + service: { + url: '/proxy/services/exporter-proj/qwen-service/', + model: { name: 'Qwen/Qwen3.6-27B-FP8', type: 'chat', base_url: BASE_URL }, + }, + }) as IRun; + +describe('Model of a run', () => { + test.each(['running', 'terminated'] as const)('the model of a %s run has the URL of the run', (status) => { + const model = getExtendedModelFromRun(makeRun(status)); + + expect(model?.endpoints).toEqual([{ project_name: 'exporter-proj', base_url: BASE_URL }]); + }); +}); diff --git a/frontend/src/libs/run.ts b/frontend/src/libs/run.ts index d0773e15b3..6d5d6f26a9 100644 --- a/frontend/src/libs/run.ts +++ b/frontend/src/libs/run.ts @@ -65,6 +65,18 @@ export const getStatusIconColor = ( } }; +/** + * A finished run shows the status and the termination reason of its latest job submission. + */ +export const getRunStatus = (run: IRun) => { + const isFinished = finishedRunStatuses.includes(run.status); + + return { + status: isFinished ? (run.latest_job_submission?.status ?? run.status) : run.status, + terminationReason: isFinished ? run.latest_job_submission?.termination_reason : null, + }; +}; + export const getRunStatusMessage = (run: IRun): string => { if (finishedRunStatuses.includes(run.status) && run.latest_job_submission?.status_message) { return capitalize(run.latest_job_submission.status_message); @@ -93,22 +105,32 @@ export const getRunPriority = (run: IRun): number | null => { }; export const getExtendedModelFromRun = (run: IRun): IModelExtended | null => { - if (!run?.service?.model) return null; + const model = run?.service?.model; + + if (!model) return null; + + const { status, terminationReason } = getRunStatus(run); return { - ...(run.service?.model ?? {}), - id: run.id, + name: model.name, project_name: run.project_name, - run_name: run?.run_spec.run_name ?? 'No run name', - user: run.user, - resources: run.latest_job_submission?.job_provisioning_data?.instance_type?.resources - ? formatResources(run.latest_job_submission.job_provisioning_data.instance_type.resources) - : null, - price: run.latest_job_submission?.job_provisioning_data?.price ?? null, - submitted_at: run.submitted_at, - repository: getRepoNameFromRun(run), - backend: run.latest_job_submission?.job_provisioning_data?.backend ?? null, - region: run.latest_job_submission?.job_provisioning_data?.region ?? null, + // A finished run keeps its URL: the model is back at it when the service is applied again + endpoints: [{ project_name: run.project_name, base_url: model.base_url }], + run: { + id: run.id, + run_name: run?.run_spec.run_name ?? 'No run name', + status, + termination_reason: terminationReason, + status_message: getRunStatusMessage(run), + user: run.user, + resources: run.latest_job_submission?.job_provisioning_data?.instance_type?.resources + ? formatResources(run.latest_job_submission.job_provisioning_data.instance_type.resources) + : null, + price: run.latest_job_submission?.job_provisioning_data?.price ?? null, + submitted_at: run.submitted_at, + repository: getRepoNameFromRun(run), + backend: run.latest_job_submission?.job_provisioning_data?.backend ?? null, + }, }; }; diff --git a/frontend/src/locale/en.json b/frontend/src/locale/en.json index 78f95503ea..fcbabc499a 100644 --- a/frontend/src/locale/en.json +++ b/frontend/src/locale/en.json @@ -30,6 +30,7 @@ "stop": "Stop", "abort": "Abort", "close": "Close", + "clear": "Clear", "clearFilter": "Clear filter", "server_error": "Server error: {{error}}", "login": "Sign in", @@ -603,9 +604,9 @@ "models": { "model_name": "Name", - "url": "URL", + "status": "Status", + "project": "Project", "gateway": "Gateway", - "type": "Type", "run": "Run", "resources": "Resources", "price": "Price", @@ -613,7 +614,8 @@ "user": "User", "repository": "Repository", "backend": "Backend", - "code": "Code", + "endpoint": "Endpoint", + "active_only": "Active models", "empty_message_title": "No models", "empty_message_text": "No models to display.", "nomatch_message_title": "No matches", @@ -621,14 +623,35 @@ "nomatch_message_button_label": "Clear filter", "details": { - "instructions": "System", - "instructions_description": "Specify system", + "title": "Details", "message_placeholder": "Enter your question", "chat_empty_title": "No messages yet", "chat_empty_message": "Please start a chat", - "run_name": "Run name", - "view_code": "View code", + "thinking": "Thinking", + "reply_cut": "The reply was cut at the limit of {{limit}} tokens.", "view_code_description": "You can use the following code to start integrating your current prompt and settings into your application." + }, + + "exports": { + "title": "Exports", + "name": "Name", + "empty_message_title": "No exports", + "empty_message_text": "No exports to display." + }, + + "metering": { + "title": "Metering", + "tokens": "Tokens", + "day": "Day", + "month": "Month", + "day_utc": "Day (UTC)", + "month_utc": "Month (UTC)", + "input_tokens": "Input tokens", + "cached_input_tokens": "Cached input tokens", + "output_tokens": "Output tokens", + "total": "Total", + "empty_message_title": "No usage", + "empty_message_text": "No usage to display." } }, diff --git a/frontend/src/pages/Events/List/hooks/useTimeRange.ts b/frontend/src/pages/Events/List/hooks/useTimeRange.ts index 64ba02c1c8..86b6d3244c 100644 --- a/frontend/src/pages/Events/List/hooks/useTimeRange.ts +++ b/frontend/src/pages/Events/List/hooks/useTimeRange.ts @@ -30,7 +30,7 @@ export const relativeOptions: DateRangePickerProps.RelativeOption[] = [ ]; export const useTimeRange = ({ withSearchParams, ascending }: { withSearchParams?: boolean; ascending: boolean }) => { - const { t } = useTranslation(); + const isValidRange = useIsValidRange(); const [searchParams, setSearchParams] = useSearchParams(); const [timeRange, setTimeRange] = useState(() => @@ -94,7 +94,20 @@ export const useTimeRange = ({ withSearchParams, ascending }: { withSearchParams [ascending, start, end], ); - const isValidRange: DateRangePickerProps.ValidationFunction = (range) => { + return { + timeRange, + onChangeTimeRange, + isValidRange, + refreshRelativeTimeRange, + requestParams, + isPastEnd, + } as const; +}; + +export const useIsValidRange = (): DateRangePickerProps.ValidationFunction => { + const { t } = useTranslation(); + + return (range) => { if (range?.type === 'absolute') { if (!range.startDate || !range.endDate) { return { valid: false, errorMessage: t('events.time_range_incomplete') }; @@ -111,15 +124,6 @@ export const useTimeRange = ({ withSearchParams, ascending }: { withSearchParams return { valid: true }; }; - - return { - timeRange, - onChangeTimeRange, - isValidRange, - refreshRelativeTimeRange, - requestParams, - isPastEnd, - } as const; }; const getTimeRangeBounds = (timeRange: DateRangePickerProps.Value | null, now: Date): { start?: Date; end?: Date } => { diff --git a/frontend/src/pages/Models/Details/Exports/helpers.test.ts b/frontend/src/pages/Models/Details/Exports/helpers.test.ts new file mode 100644 index 0000000000..cba5eaf33a --- /dev/null +++ b/frontend/src/pages/Models/Details/Exports/helpers.test.ts @@ -0,0 +1,90 @@ +import { GlobalUserRole } from 'types'; + +import { canListModelExports, getModelExports } from './helpers'; + +import { IModelExtended } from '../../List/types'; + +// exporter-proj runs the model +const model: IModelExtended = { name: 'Qwen/Qwen3.8-Flash-Next', project_name: 'exporter-proj', endpoints: [] }; + +const makeExport = (name: string, projectNames: string[], modelNames: string[]): IExport => ({ + name, + imports: projectNames.map((project_name) => ({ project_name })), + exported_models: modelNames.map((modelName) => ({ name: modelName })), +}); + +const makeImports = (projectName: string, exportProjectName: string, exportName: string, modelNames: string[]) => ({ + project_name: projectName, + imports: [ + { + export: { + name: exportName, + project_name: exportProjectName, + exported_models: modelNames.map((name) => ({ name, base_url: null })), + }, + }, + ], +}); + +const makeUser = (globalRole: TUserRole) => ({ username: 'bob', global_role: globalRole }) as IUser; + +const makeProject = (projectName: string, userNames: string[]) => + ({ + project_name: projectName, + members: userNames.map((username) => ({ user: { username }, project_role: 'user' })), + }) as IProject; + +describe('Model exports', () => { + test("a member of the model's project gets every export of the model, with the projects that import it", () => { + const exports = [ + makeExport('importer2-models', ['importer2-proj'], [model.name]), + makeExport('importer-models', ['importer-proj', 'another-proj'], [model.name, 'other-model']), + makeExport('other-models', ['importer-proj'], ['other-model']), + makeExport('new-models', [], [model.name]), + ]; + + expect(getModelExports(model, exports, [])).toEqual([ + { + export_name: 'importer-models', + projects: [ + { export_name: 'importer-models', project_name: 'another-proj' }, + { export_name: 'importer-models', project_name: 'importer-proj' }, + ], + }, + { + export_name: 'importer2-models', + projects: [{ export_name: 'importer2-models', project_name: 'importer2-proj' }], + }, + { export_name: 'new-models', projects: [] }, + ]); + }); + + test("other users get the exports that their own projects import the model through, from the model's project", () => { + const projectsImports = [ + makeImports('importer-proj', 'exporter-proj', 'importer-models', [model.name]), + // A model of the same name that another project exports + makeImports('importer2-proj', 'other-proj', 'other-models', [model.name]), + makeImports('importer3-proj', 'exporter-proj', 'other-models', ['other-model']), + makeImports('another-proj', 'exporter-proj', 'importer-models', [model.name]), + ]; + + expect(getModelExports(model, undefined, projectsImports)).toEqual([ + { + export_name: 'importer-models', + projects: [ + { export_name: 'importer-models', project_name: 'another-proj' }, + { export_name: 'importer-models', project_name: 'importer-proj' }, + ], + }, + ]); + }); + + test("the exports of the model's project are for its members and for global admins", () => { + const projects = [makeProject('exporter-proj', ['alice']), makeProject('importer-proj', ['bob'])]; + + expect(canListModelExports(model, projects, makeUser(GlobalUserRole.USER))).toBe(false); + expect(canListModelExports(model, projects, makeUser(GlobalUserRole.ADMIN))).toBe(true); + expect(canListModelExports(model, [makeProject('exporter-proj', ['bob'])], makeUser(GlobalUserRole.USER))).toBe(true); + expect(canListModelExports(model, projects, null)).toBe(false); + }); +}); diff --git a/frontend/src/pages/Models/Details/Exports/helpers.ts b/frontend/src/pages/Models/Details/Exports/helpers.ts new file mode 100644 index 0000000000..3c79fb6e10 --- /dev/null +++ b/frontend/src/pages/Models/Details/Exports/helpers.ts @@ -0,0 +1,74 @@ +import { GlobalUserRole } from 'types'; + +import { getProjectRoleByUserName } from 'pages/Project/utils'; + +import { IModelExtended } from '../../List/types'; + +export type TModelExportProject = { + export_name: IExport['name']; + // The project that imports the model + project_name: IProject['project_name']; +}; + +export type TModelExport = { + export_name: IExport['name']; + // The projects that import the model through the export + projects: TModelExportProject[]; +}; + +/** + * A project's exports are for its members and for global admins. + */ +export const canListModelExports = (model: IModelExtended, projects: IProject[], user: IUser | null): boolean => { + if (!user) { + return false; + } + + if (user.global_role === GlobalUserRole.ADMIN) { + return true; + } + + return projects.some( + (project) => project.project_name === model.project_name && getProjectRoleByUserName(project, user.username) !== null, + ); +}; + +/** + * The model's exports, each with the projects that import the model through it. `exports` are the exports of + * the model's project, which its members see, so they get every export and project. Other users get the exports + * that their own projects import the model through, from their import lists. + */ +export const getModelExports = ( + model: IModelExtended, + exports: IExport[] | undefined, + projectsImports: TProjectImports[], +): TModelExport[] => { + const hasModel = ({ exported_models }: { exported_models: { name: string }[] }) => + exported_models.some(({ name }) => name === model.name); + + const projectNames = new Map(); + + if (exports) { + exports.filter(hasModel).forEach(({ name, imports }) => + projectNames.set( + name, + imports.map(({ project_name }) => project_name), + ), + ); + } else { + projectsImports.forEach(({ project_name, imports }) => + imports + .filter((item) => item.export.project_name === model.project_name && hasModel(item.export)) + .forEach((item) => + projectNames.set(item.export.name, [...(projectNames.get(item.export.name) ?? []), project_name]), + ), + ); + } + + return Array.from(projectNames) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([export_name, names]) => ({ + export_name, + projects: names.sort((a, b) => a.localeCompare(b)).map((project_name) => ({ export_name, project_name })), + })); +}; diff --git a/frontend/src/pages/Models/Details/Exports/hooks.ts b/frontend/src/pages/Models/Details/Exports/hooks.ts new file mode 100644 index 0000000000..0182b1e849 --- /dev/null +++ b/frontend/src/pages/Models/Details/Exports/hooks.ts @@ -0,0 +1,35 @@ +import { useMemo } from 'react'; + +import { useAppSelector } from 'hooks'; +import { useGetProjectExportsQuery } from 'services/export'; + +import { selectUserData } from 'App/slice'; + +import { canListModelExports, getModelExports } from './helpers'; + +import { IModelExtended } from '../../List/types'; + +/** + * Loads the model's exports and the projects that import the model through them, as far as the user can see + * them. The user's projects and their import lists come from the page, which has loaded them for the model. + */ +export const useModelExports = (model: IModelExtended, projects: IProject[], projectsImports: TProjectImports[]) => { + const userData = useAppSelector(selectUserData); + + const canListExports = canListModelExports(model, projects, userData); + + const { currentData, isFetching, error } = useGetProjectExportsQuery( + { project_name: model.project_name }, + { skip: !canListExports }, + ); + + const modelExports = useMemo(() => { + if (isFetching || error) { + return []; + } + + return getModelExports(model, canListExports ? currentData : undefined, projectsImports); + }, [isFetching, error, model, canListExports, currentData, projectsImports]); + + return { modelExports, isLoading: isFetching, error } as const; +}; diff --git a/frontend/src/pages/Models/Details/Exports/index.tsx b/frontend/src/pages/Models/Details/Exports/index.tsx new file mode 100644 index 0000000000..266728488c --- /dev/null +++ b/frontend/src/pages/Models/Details/Exports/index.tsx @@ -0,0 +1,86 @@ +import React, { useState } from 'react'; +import { useTranslation } from 'react-i18next'; + +import type { TableProps } from 'components'; +import { Alert, Header, ListEmptyMessage, NavigateLink, Table } from 'components'; + +import { useCollection } from 'hooks'; +import { getServerError } from 'libs'; +import { ROUTES } from 'routes'; + +import { TModelExport, TModelExportProject } from './helpers'; +import { useModelExports } from './hooks'; + +import { IModelExtended } from '../../List/types'; + +// A row of the table: an export, and under it the projects that import the model through it +type TItem = TModelExport | TModelExportProject; + +const isExport = (item: TItem): item is TModelExport => 'projects' in item; + +type Props = { + model: IModelExtended; + // The user's projects and their import lists + projects: IProject[]; + projectsImports: TProjectImports[]; +}; + +export const ModelExports: React.FC = ({ model, projects, projectsImports }) => { + const { t } = useTranslation(); + const { modelExports, isLoading, error } = useModelExports(model, projects, projectsImports); + + const [collapsedExports, setCollapsedExports] = useState([]); + + const columns: TableProps.ColumnDefinition[] = [ + { + id: 'name', + header: t('models.exports.name'), + cell: (item) => (isExport(item) ? item.export_name : null), + }, + { + id: 'project', + header: t('models.project'), + cell: (item) => + isExport(item) ? null : ( + {item.project_name} + ), + }, + ]; + + const { items, collectionProps } = useCollection(modelExports, { + filtering: { + empty: error ? ( + {t('common.server_error', { error: getServerError(error) })} + ) : ( + + ), + }, + }); + + return ( + (isExport(item) ? item.export_name : `${item.export_name}/${item.project_name}`)} + // Exports are expanded until the user collapses them + expandableRows={{ + getItemChildren: (item) => (isExport(item) ? item.projects : []), + isItemExpandable: (item) => isExport(item) && item.projects.length > 0, + expandedItems: items.filter(({ export_name }) => !collapsedExports.includes(export_name)), + onExpandableItemToggle: ({ detail }) => + setCollapsedExports((currentExports) => + detail.expanded + ? currentExports.filter((exportName) => exportName !== detail.item.export_name) + : [...currentExports, detail.item.export_name], + ), + }} + loading={isLoading} + loadingText={t('common.loading')} + header={
{t('models.exports.title')}
} + /> + ); +}; diff --git a/frontend/src/pages/Models/Details/Metering/helpers.test.ts b/frontend/src/pages/Models/Details/Metering/helpers.test.ts new file mode 100644 index 0000000000..87ae0ce000 --- /dev/null +++ b/frontend/src/pages/Models/Details/Metering/helpers.test.ts @@ -0,0 +1,150 @@ +import { + DEFAULT_PERIOD, + DEFAULT_TIME_RANGE, + getFilteringOptions, + getMeteringRequestParams, + getMeteringTotal, + getUserOptionsRequestParams, +} from './helpers'; + +const model = { + name: 'Qwen/Qwen3.8-Flash-Next', + project_name: 'exporter-proj', + endpoints: [], +}; + +const now = new Date('2026-11-18T23:30:00Z'); + +describe('Model metering request', () => { + test('asks for the model by day and project in the last 30 days by default', () => { + const params = getMeteringRequestParams({ + model, + timeRange: DEFAULT_TIME_RANGE, + period: DEFAULT_PERIOD, + filters: {}, + now, + }); + + expect(params).toEqual({ + start: '2026-10-19T00:00:00Z', + end: '2026-11-19T00:00:00Z', + group_by: ['day', 'project'], + model_project_name: 'exporter-proj', + model_name: 'Qwen/Qwen3.8-Flash-Next', + }); + }); + + test.each([ + [{ project_name: 'importer-proj' }], + [{ user_name: 'bob' }], + [{ project_name: 'importer-proj', user_name: 'bob' }], + ])('the filter %j goes to the request, and the rows stay by project', (filters) => { + const params = getMeteringRequestParams({ model, timeRange: DEFAULT_TIME_RANGE, period: DEFAULT_PERIOD, filters, now }); + + expect(params).toEqual({ + start: '2026-10-19T00:00:00Z', + end: '2026-11-19T00:00:00Z', + group_by: ['day', 'project'], + model_project_name: 'exporter-proj', + model_name: 'Qwen/Qwen3.8-Flash-Next', + ...filters, + }); + }); + + test.each([ + [{ type: 'relative', amount: 3, unit: 'month' }, '2026-08-18T00:00:00Z', '2026-11-19T00:00:00Z'], + [{ type: 'relative', amount: 7, unit: 'day' }, '2026-11-11T00:00:00Z', '2026-11-19T00:00:00Z'], + [{ type: 'absolute', startDate: '2026-11-18', endDate: '2026-11-18' }, '2026-11-18T00:00:00Z', '2026-11-19T00:00:00Z'], + [{ type: 'absolute', startDate: '2026-10-01', endDate: '2026-10-31' }, '2026-10-01T00:00:00Z', '2026-11-01T00:00:00Z'], + ] as const)('by day, the time range %j covers whole days', (timeRange, start, end) => { + const params = getMeteringRequestParams({ model, timeRange, period: 'day', filters: {}, now }); + + expect(params).toMatchObject({ start, end, group_by: ['day', 'project'] }); + }); + + test.each([ + // The running month + ['2026-11-18', { type: 'relative', amount: 7, unit: 'day' }, '2026-11-01T00:00:00Z', '2026-12-01T00:00:00Z'], + // The seven days cross a month boundary + ['2026-12-03', { type: 'relative', amount: 7, unit: 'day' }, '2026-11-01T00:00:00Z', '2027-01-01T00:00:00Z'], + [ + '2026-11-18', + { type: 'absolute', startDate: '2026-09-03', endDate: '2026-10-10' }, + '2026-09-01T00:00:00Z', + '2026-11-01T00:00:00Z', + ], + ] as const)('by month, on %s the time range %j covers whole months', (today, timeRange, start, end) => { + const params = getMeteringRequestParams({ + model, + timeRange, + period: 'month', + filters: {}, + now: new Date(`${today}T23:30:00Z`), + }); + + expect(params).toMatchObject({ start, end }); + }); +}); + +describe('Model metering filter', () => { + test("asks for the users of the filter's project in the same time range, whatever user the filter has", () => { + const filters = { project_name: 'importer-proj', user_name: 'bob' }; + + const params = getUserOptionsRequestParams({ + model, + timeRange: DEFAULT_TIME_RANGE, + period: DEFAULT_PERIOD, + filters, + now, + }); + + expect(params).toEqual({ + start: '2026-10-19T00:00:00Z', + end: '2026-11-19T00:00:00Z', + group_by: ['day', 'user'], + model_project_name: 'exporter-proj', + model_name: 'Qwen/Qwen3.8-Flash-Next', + project_name: 'importer-proj', + }); + }); + + test('offers each user of the rows once', () => { + const rows = ['2026-11-01', '2026-10-01'].flatMap((month) => + ['carol', 'bob'].map((user_name) => ({ + month, + user_name, + input_tokens: 1, + cached_input_tokens: 0, + output_tokens: 0, + })), + ); + + expect(getFilteringOptions(rows, [])).toEqual([ + { propertyKey: 'user_name', value: 'bob' }, + { propertyKey: 'user_name', value: 'carol' }, + ]); + }); +}); + +describe('Model metering total', () => { + test('sums the rows', () => { + const rows: IMeteringRow[] = [ + { + day: '2026-12-03', + project_name: 'importer-proj', + input_tokens: 50211, + cached_input_tokens: 41980, + output_tokens: 6120, + }, + { + day: '2026-12-03', + project_name: 'exporter-proj', + input_tokens: 8707, + cached_input_tokens: 7688, + output_tokens: 423, + }, + ]; + + expect(getMeteringTotal(rows)).toEqual({ input_tokens: 58918, cached_input_tokens: 49668, output_tokens: 6543 }); + }); +}); diff --git a/frontend/src/pages/Models/Details/Metering/helpers.ts b/frontend/src/pages/Models/Details/Metering/helpers.ts new file mode 100644 index 0000000000..b97f083d7f --- /dev/null +++ b/frontend/src/pages/Models/Details/Metering/helpers.ts @@ -0,0 +1,129 @@ +import { pick as _pick, sumBy as _sumBy, uniq as _uniq } from 'lodash'; +import { add, format, parseISO, startOfMonth, sub } from 'date-fns'; + +import type { DateRangePickerProps, PropertyFilterProps } from 'components'; + +import { IModelExtended } from '../../List/types'; + +export type TMeteringPeriod = Extract; + +export type TMeteringFilters = Pick; + +const DATE_FORMAT = 'yyyy-MM-dd'; + +export const DEFAULT_PERIOD: TMeteringPeriod = 'day'; + +export const DEFAULT_TIME_RANGE: DateRangePickerProps.RelativeOption = { + key: 'last-30-days', + amount: 30, + unit: 'day', + type: 'relative', +}; + +export const relativeOptions: DateRangePickerProps.RelativeOption[] = [ + { key: 'last-7-days', amount: 7, unit: 'day', type: 'relative' }, + DEFAULT_TIME_RANGE, + { key: 'last-3-months', amount: 3, unit: 'month', type: 'relative' }, +]; + +/** + * Metering sums tokens by UTC day, so a time range covers whole days: + * from the start of its first day to the end of its last one. A relative time range ends today. + * By month, it covers whole months: from the start of the month of its first day to the end of the month of its last one. + */ +const getTimeRangeBounds = ( + timeRange: DateRangePickerProps.Value, + period: TMeteringPeriod, + now: Date, +): Pick => { + const today = parseISO(now.toISOString().split('T')[0]); + + const [firstDay, lastDay] = + timeRange.type === 'relative' + ? [sub(today, { [`${timeRange.unit}s`]: timeRange.amount }), today] + : [parseISO(timeRange.startDate), parseISO(timeRange.endDate)]; + + const [start, end] = + period === 'month' + ? [startOfMonth(firstDay), add(startOfMonth(lastDay), { months: 1 })] + : [firstDay, add(lastDay, { days: 1 })]; + + return { + start: `${format(start, DATE_FORMAT)}T00:00:00Z`, + end: `${format(end, DATE_FORMAT)}T00:00:00Z`, + }; +}; + +type TMeteringRequest = { + model: IModelExtended; + timeRange: DateRangePickerProps.Value; + period: TMeteringPeriod; + filters: TMeteringFilters; + now: Date; +}; + +/** + * The table's rows: the model's tokens by day or month and by project. A filter by user narrows them to that user's requests. + */ +export const getMeteringRequestParams = ({ + model, + timeRange, + period, + filters, + now, +}: TMeteringRequest): TMeteringListRequestParams => ({ + ...getTimeRangeBounds(timeRange, period, now), + group_by: [period, 'project'], + model_project_name: model.project_name, + model_name: model.name, + ...filters, +}); + +/** + * The rows don't have users, so the users that the filter offers come from a request of their own: + * the same time range and model by user, in the project of the filter if it has one. + */ +export const getUserOptionsRequestParams = ({ filters, ...request }: TMeteringRequest): TMeteringListRequestParams => ({ + ...getMeteringRequestParams({ ...request, filters: _pick(filters, 'project_name') }), + group_by: [request.period, 'user'], +}); + +/** + * The API returns every row of a request, so the sum of the rows is the total. + */ +export const getMeteringTotal = (rows: IMeteringRow[]): IMeteringRow => ({ + input_tokens: _sumBy(rows, 'input_tokens'), + cached_input_tokens: _sumBy(rows, 'cached_input_tokens'), + output_tokens: _sumBy(rows, 'output_tokens'), +}); + +/** + * Adds the projects and users of `rows` to the filter's `options`. + */ +export const getFilteringOptions = ( + rows: IMeteringRow[], + options: PropertyFilterProps.FilteringOption[], +): PropertyFilterProps.FilteringOption[] => + (['project_name', 'user_name'] as const).flatMap((propertyKey) => + _uniq([ + ...options.filter((option) => option.propertyKey === propertyKey).map(({ value }) => value), + ...rows.map((row) => row[propertyKey] ?? ''), + ]) + .filter(Boolean) + .sort() + .map((value) => ({ propertyKey, value })), + ); + +// A row has the day, or the first day of the month, that it sums +const getMeteringPeriod = (row: IMeteringRow) => row.day ?? row.month ?? ''; + +export const compareMeteringPeriods = (a: IMeteringRow, b: IMeteringRow) => + getMeteringPeriod(a).localeCompare(getMeteringPeriod(b)); + +export const formatMeteringPeriod = (row: IMeteringRow) => { + if (row.day) { + return format(parseISO(row.day), 'MM/dd/yyyy'); + } + + return row.month ? format(parseISO(row.month), 'MM/yyyy') : ''; +}; diff --git a/frontend/src/pages/Models/Details/Metering/hooks.ts b/frontend/src/pages/Models/Details/Metering/hooks.ts new file mode 100644 index 0000000000..5d79fbe564 --- /dev/null +++ b/frontend/src/pages/Models/Details/Metering/hooks.ts @@ -0,0 +1,124 @@ +import { useEffect, useMemo, useState } from 'react'; + +import type { DateRangePickerProps, PropertyFilterProps } from 'components'; + +import { EMPTY_QUERY, tokensToRequestParams } from 'libs/filters'; +import { useLazyGetMeteringQuery } from 'services/metering'; + +import { useIsValidRange } from 'pages/Events/List/hooks/useTimeRange'; + +import { DEFAULT_TIME_RANGE, getFilteringOptions, TMeteringFilters } from './helpers'; + +type RequestParamsKeys = keyof TMeteringFilters; + +const filterKeys: Record = { + PROJECT_NAME: 'project_name', + USER_NAME: 'user_name', +}; + +export const useFilters = () => { + const [propertyFilterQuery, setPropertyFilterQuery] = useState(EMPTY_QUERY); + + const filteringProperties = [ + { + key: filterKeys.PROJECT_NAME, + operators: ['='], + propertyLabel: 'Project', + groupValuesLabel: 'Project values', + }, + { + key: filterKeys.USER_NAME, + operators: ['='], + propertyLabel: 'User', + groupValuesLabel: 'User values', + }, + ]; + + const onChangePropertyFilter: PropertyFilterProps['onChange'] = ({ detail }) => { + const { tokens, operation } = detail; + + const filteredTokens = tokens.filter((token, tokenIndex) => { + return !tokens.some((item, index) => token.propertyKey === item.propertyKey && index > tokenIndex); + }); + + setPropertyFilterQuery({ + operation, + tokens: filteredTokens, + }); + }; + + const filteringRequestParams = useMemo(() => { + return tokensToRequestParams({ + tokens: propertyFilterQuery.tokens, + }) as TMeteringFilters; + }, [propertyFilterQuery]); + + return { + filteringRequestParams, + propertyFilterQuery, + onChangePropertyFilter, + filteringProperties, + } as const; +}; + +/** + * The rows have every project that the filter can narrow them to. The rows that a filter has narrowed + * lack the others, so the options are those of all the rows since the tab opened. + */ +export const useFilteringOptions = (rows?: IMeteringRow[]) => { + const [filteringOptions, setFilteringOptions] = useState([]); + + useEffect(() => { + if (rows) { + setFilteringOptions((options) => getFilteringOptions(rows, options)); + } + }, [rows]); + + return filteringOptions; +}; + +/** + * The rows don't have users, so the users that the filter offers are loaded when its User property opens. + */ +export const useUserFilteringOptions = (requestParams: TMeteringListRequestParams) => { + const [getMetering] = useLazyGetMeteringQuery(); + const [userFilteringOptions, setUserFilteringOptions] = useState([]); + const [filteringStatusType, setFilteringStatusType] = useState(); + + const handleLoadItems: PropertyFilterProps['onLoadItems'] = async ({ detail: { filteringProperty } }) => { + if (filteringProperty?.key !== filterKeys.USER_NAME) { + return; + } + + setFilteringStatusType('loading'); + + // An answer that the tab already has is used again, so typing a name sends no request + const { data: rows = [] } = await getMetering(requestParams, true); + + setUserFilteringOptions(getFilteringOptions(rows, [])); + setFilteringStatusType(undefined); + }; + + return { + userFilteringOptions, + filteringStatusType, + handleLoadItems, + } as const; +}; + +export const useTimeRange = () => { + const isValidRange = useIsValidRange(); + const [timeRange, setTimeRange] = useState(DEFAULT_TIME_RANGE); + + const onChangeTimeRange: DateRangePickerProps['onChange'] = ({ detail }) => { + if (detail.value) { + setTimeRange(detail.value); + } + }; + + return { + timeRange, + onChangeTimeRange, + isValidRange, + } as const; +}; diff --git a/frontend/src/pages/Models/Details/Metering/index.tsx b/frontend/src/pages/Models/Details/Metering/index.tsx new file mode 100644 index 0000000000..6e34655dd6 --- /dev/null +++ b/frontend/src/pages/Models/Details/Metering/index.tsx @@ -0,0 +1,147 @@ +import React, { useMemo, useState } from 'react'; +import { useTranslation } from 'react-i18next'; + +import type { TableProps } from 'components'; +import { Alert, DateRangePicker, Header, ListEmptyMessage, PropertyFilter, SegmentedControl, Table } from 'components'; + +import { useCollection } from 'hooks'; +import { getServerError } from 'libs'; +import { useGetMeteringQuery } from 'services/metering'; + +import { + compareMeteringPeriods, + DEFAULT_PERIOD, + formatMeteringPeriod, + getMeteringRequestParams, + getMeteringTotal, + getUserOptionsRequestParams, + relativeOptions, + TMeteringPeriod, +} from './helpers'; +import { useFilteringOptions, useFilters, useTimeRange, useUserFilteringOptions } from './hooks'; + +import { IModelExtended } from '../../List/types'; + +import styles from './styles.module.scss'; + +// The token counts of a row and of the total row +const TOKENS = ['input_tokens', 'cached_input_tokens', 'output_tokens'] as const; + +export const ModelMetering: React.FC<{ model: IModelExtended }> = ({ model }) => { + const { t } = useTranslation(); + const [period, setPeriod] = useState(DEFAULT_PERIOD); + const { timeRange, onChangeTimeRange, isValidRange } = useTimeRange(); + const { filteringRequestParams, propertyFilterQuery, onChangePropertyFilter, filteringProperties } = useFilters(); + + const [requestParams, userOptionsRequestParams] = useMemo(() => { + const request = { model, timeRange, period, filters: filteringRequestParams, now: new Date() }; + + return [getMeteringRequestParams(request), getUserOptionsRequestParams(request)]; + }, [model, timeRange, period, filteringRequestParams]); + + const { currentData, isFetching, error } = useGetMeteringQuery(requestParams); + + // A failed request shows its error, not the rows of an earlier answer + const rows = useMemo(() => (error ? [] : (currentData ?? [])), [currentData, error]); + // The total of the rows, so it follows the filter, the time range and the period. + // The API returns every row of a request, so the browser sums them + const total = useMemo(() => getMeteringTotal(rows), [rows]); + + const filteringOptions = useFilteringOptions(currentData); + const { userFilteringOptions, filteringStatusType, handleLoadItems } = useUserFilteringOptions(userOptionsRequestParams); + + // The total row's cells look like the column headers + const renderCell = (item: IMeteringRow, content: string) => + item === total ? {content} : content; + + const columns: TableProps.ColumnDefinition[] = [ + { + id: 'period', + header: t(`models.metering.${period}_utc`), + cell: (item) => renderCell(item, item === total ? t('models.metering.total') : formatMeteringPeriod(item)), + sortingComparator: compareMeteringPeriods, + }, + { + id: 'project', + header: t('models.project'), + cell: (item) => item.project_name, + }, + ...TOKENS.map((tokens) => ({ + id: tokens, + header: t(`models.metering.${tokens}`), + cell: (item: IMeteringRow) => renderCell(item, item[tokens].toLocaleString()), + sortingField: tokens, + })), + ]; + + const { items, collectionProps } = useCollection(rows, { + filtering: { + empty: error ? ( + {t('common.server_error', { error: getServerError(error) })} + ) : ( + + ), + }, + sorting: { + defaultState: { sortingColumn: columns[0], isDescending: true }, + }, + }); + + return ( +
{t('models.metering.tokens')}} + filter={ +
+
+ `Use: ${value}`, + }} + filteringOptions={[...filteringOptions, ...userFilteringOptions]} + filteringProperties={filteringProperties} + filteringStatusType={filteringStatusType} + onLoadItems={handleLoadItems} + /> +
+ + + + setPeriod(detail.selectedId as TMeteringPeriod)} + options={[ + { text: t('models.metering.day'), id: 'day' }, + { text: t('models.metering.month'), id: 'month' }, + ]} + /> +
+ } + /> + ); +}; diff --git a/frontend/src/pages/Models/Details/Metering/styles.module.scss b/frontend/src/pages/Models/Details/Metering/styles.module.scss new file mode 100644 index 0000000000..a2012ab8e6 --- /dev/null +++ b/frontend/src/pages/Models/Details/Metering/styles.module.scss @@ -0,0 +1,22 @@ +.selectFilters { + display: flex; + flex-wrap: wrap; + gap: 8px 20px; + + .propertyFilter { + max-width: 640px; + flex-grow: 1; + min-width: 0; + } +} + +// The text of a table's column header. Cloudscape keeps this colour among its internal tokens, +// so its two values are repeated here +.total { + font-weight: 700; + color: #424650; + + :global(.awsui-dark-mode) & { + color: #b4b4bb; + } +} diff --git a/frontend/src/pages/Models/Details/ModelDetails/index.tsx b/frontend/src/pages/Models/Details/ModelDetails/index.tsx new file mode 100644 index 0000000000..87ea1b9b4b --- /dev/null +++ b/frontend/src/pages/Models/Details/ModelDetails/index.tsx @@ -0,0 +1,381 @@ +import React, { useEffect, useRef, useState } from 'react'; +import { useForm } from 'react-hook-form'; +import { useTranslation } from 'react-i18next'; +import reactStringReplace from 'react-string-replace'; +import OpenAI from 'openai'; + +import { + Avatar, + Box, + Button, + ChatBubble, + Code, + ColumnLayout, + Container, + ExpandableSection, + FormTextarea, + Header, + ListEmptyMessage, + NavigateLink, + SpaceBetween, + StatusIndicator, + UserAvatar, +} from 'components'; + +import { useAppSelector, useLocalStorageState, useNotifications } from 'hooks'; +import { ROUTES } from 'routes'; + +import { selectAuthToken, selectUserName } from 'App/slice'; + +import { ModelStatus } from '../../components/ModelStatus'; +import { isModelServed } from '../../helpers'; +import { addChunkToMessage, getModelUrl, MAX_TOKENS } from '../helpers'; + +import { IModelExtended } from '../../List/types'; +import { FormValues, Message, Role } from '../types'; + +import css from './styles.module.scss'; + +const NO_MESSAGES: Message[] = []; + +const MESSAGE_ROLE_MAP: Record = { + tool: 'Tool', + system: 'System', + user: 'User', + assistant: 'Assistant', +}; + +/** + * The thinking of a reply. It is open while the thinking arrives, and collapses when the answer starts. + */ +const Thinking: React.FC<{ text: string; hasAnswer: boolean }> = ({ text, hasAnswer }) => { + const { t } = useTranslation(); + const [expanded, setExpanded] = useState(!hasAnswer); + + useEffect(() => { + if (hasAnswer) { + setExpanded(false); + } + }, [hasAnswer]); + + return ( + setExpanded(detail.expanded)} + > + + {text} + + + ); +}; + +export const ModelDetails: React.FC<{ model: IModelExtended }> = ({ model }) => { + const { t } = useTranslation(); + const token = useAppSelector(selectAuthToken); + const userName = useAppSelector(selectUserName); + + // A user's chat with a model is kept in the browser, so that it survives a reload of the page + const [messages, storeMessages] = useLocalStorageState( + `model-chat/${userName}/${model.project_name}/${model.name}`, + NO_MESSAGES, + ); + + const messagesRef = useRef(messages); + messagesRef.current = messages; + + const setMessages = (update: Message[] | ((messages: Message[]) => Message[])) => { + messagesRef.current = typeof update === 'function' ? update(messagesRef.current) : update; + + try { + storeMessages(messagesRef.current); + } catch { + // The browser's storage is full: the chat goes on, and a reload loses what didn't fit + } + }; + + const [loading, setLoading] = useState(false); + const openai = useRef(); + const textAreaRef = useRef(null); + const chatList = useRef(); + const [pushNotification] = useNotifications(); + + // '...' stands for a reply that is on its way, while nothing of it has arrived yet + const isAwaited = (message: Message, index: number) => + loading && index === messages.length - 1 && !message.content && !message.thinking; + + // The chat needs a model that a service serves. A stopped model keeps its URL for the Endpoint dialog only + const modelUrl = isModelServed(model) ? getModelUrl(model) : ''; + + const scrollChatToBottom = () => { + if (!chatList.current) return; + + const { clientHeight, scrollHeight } = chatList.current; + chatList.current.scrollTo(0, scrollHeight - clientHeight); + }; + + useEffect(() => { + // Without a URL, the client would send the user's token to its default API + openai.current = modelUrl + ? new OpenAI({ + baseURL: modelUrl, + apiKey: token, + dangerouslyAllowBrowser: true, + }) + : undefined; + }, [modelUrl, token]); + + useEffect(() => { + scrollChatToBottom(); + }, [messages]); + + const { handleSubmit, control, setValue, watch } = useForm(); + + const messageText = watch('message'); + + const sendRequest = async (messages: Message[]) => { + if (!openai.current) return Promise.reject('Model not found'); + + return openai.current.chat.completions.create({ + model: model.name, + // A reply's thinking isn't sent back to the model + messages: messages.map(({ role, content }) => ({ role, content })), + stream: true, + max_tokens: MAX_TOKENS, + }); + }; + + const onSubmit = async (data: FormValues) => { + if (!data.message) { + return; + } + + const newMessages: Message[] = [...messages, { role: 'user', content: data.message }]; + + setMessages(newMessages); + setLoading(true); + + try { + const stream = await sendRequest(newMessages); + + setMessages((oldMessages) => [ + ...oldMessages, + { + role: 'assistant', + content: '', + }, + ]); + + setValue('message', ''); + setTimeout(onChangeMessage, 0); + + for await (const chunk of stream) { + setMessages((oldMessages) => { + const newMessages = [...oldMessages]; + + const lastMessage = newMessages.pop(); + + if (!lastMessage) { + return oldMessages; + } + + return [...newMessages, addChunkToMessage(lastMessage, chunk)]; + }); + } + } catch (e) { + pushNotification({ + type: 'error', + content: t('common.server_error', { error: e }), + }); + + clearChat(); + } + + setLoading(false); + + setTimeout(() => { + textAreaRef.current?.querySelector('textarea')?.focus(); + }, 10); + }; + + const clearChat = () => { + setValue('message', ''); + + setMessages([]); + setTimeout(onChangeMessage, 10); + }; + + const renderMessageBody = (content: Message['content']) => { + const LANGUAGE_PATTERN = /^(```[A-Za-z]*)/m; + const PATTERN = /^([A-Za-z \t]*)```([A-Za-z]*)?\n([\s\S]*?)```([A-Za-z \t]*)*$/gm; + const languages: string[] = []; + + const matches = content.match(LANGUAGE_PATTERN); + + if (matches) { + [...matches].forEach((l) => l && languages.push(l.replace(/^```/, ''))); + } + + const replacedStrings = reactStringReplace(content, PATTERN, (match) => { + if (!match) { + return ''; + } + + return {match}; + }); + + return ( + + {replacedStrings.filter((line, index) => { + if ( + languages.includes(line as string) && + typeof replacedStrings[index + 1] !== 'string' && + typeof replacedStrings[index + 1] !== 'undefined' + ) { + return false; + } + + return true; + })} + + ); + }; + + const onChangeMessage = () => { + if (!textAreaRef.current) return; + + const textAreaElement = textAreaRef.current.querySelector('textarea'); + + if (!textAreaElement) return; + + textAreaElement.style.height = 'auto'; + textAreaElement.style.height = textAreaElement.scrollHeight + 'px'; + }; + + // eslint-disable-next-line @typescript-eslint/ban-ts-comment + // @ts-expect-error + const onKeyDown = (event) => { + const isCtrlOrShiftKey = event?.detail?.ctrlKey || event?.detail?.shiftKey; + + if (event?.detail?.keyCode === 13 && !isCtrlOrShiftKey) { + handleSubmit(onSubmit)(); + } else if (event?.detail?.keyCode === 13 && isCtrlOrShiftKey) { + event.preventDefault(); + setValue('message', messageText + '\n'); + setTimeout(onChangeMessage, 0); + } + }; + + return ( +
+
+ {t('common.general')}}> + +
+ {t('models.model_name')} +
{model.name}
+
+ +
+ {t('models.project')} + +
+ + {model.project_name} + +
+
+ + {model.run && ( +
+ {t('models.run')} + +
+ + {model.run.run_name} + +
+
+ )} + +
+ {t('models.status')} + +
+ +
+
+
+
+
+ +
+
+ +
+ +
+ {!messages.length && ( + + )} + + {messages.map((message, index) => ( +
+ + ) : ( + + ) + } + > + + {!!message.thinking && } + + {(!!message.content || isAwaited(message, index)) && + renderMessageBody(message.content || '...')} + + {message.finish_reason === 'length' && ( + + {t('models.details.reply_cut', { limit: MAX_TOKENS.toLocaleString() })} + + )} + + +
+ ))} +
+ +
+ + +
+ +
+
+ +
+ ); +}; diff --git a/frontend/src/pages/Models/Details/ModelDetails/styles.module.scss b/frontend/src/pages/Models/Details/ModelDetails/styles.module.scss new file mode 100644 index 0000000000..3b120d5c03 --- /dev/null +++ b/frontend/src/pages/Models/Details/ModelDetails/styles.module.scss @@ -0,0 +1,67 @@ +.modelDetailsLayout { + display: flex; + flex-direction: column; + height: calc(100vh - 219px); + margin-bottom: -40px; +} + +.general { + flex-shrink: 0; +} + +.modelForm { + position: relative; + flex-grow: 1; + min-height: 300px; + display: grid; + grid-template-columns: 1fr; + grid-template-rows: 60px 1fr auto; + grid-template-areas: "buttons" + "chat" + "form"; +} + +.buttons { + display: flex; + justify-content: flex-end; + align-items: center; + gap: 12px; + + & > button { + flex-shrink: 0; + } +} + +.chat { + overflow-y: auto; + grid-area: chat; + padding: 0 20px 20px; + display: flex; + flex-direction: column; + gap: 20px; + + .message { + p { + white-space: pre-wrap; + } + } +} + +.messageForm { + display: grid; + grid-template-columns: 1fr auto; + column-gap: 20px; + grid-area: form; + border-top: 1px solid #b6bec9; + margin-right: -20px; + padding: 20px; + + textarea { + resize: none !important; + max-height: calc(100vh - 550px); + } + + .buttons { + align-self: flex-end; + } +} diff --git a/frontend/src/pages/Models/Details/helpers.test.ts b/frontend/src/pages/Models/Details/helpers.test.ts new file mode 100644 index 0000000000..7fd58d8aab --- /dev/null +++ b/frontend/src/pages/Models/Details/helpers.test.ts @@ -0,0 +1,165 @@ +import type OpenAI from 'openai'; + +import { ROUTES } from 'routes'; + +import { addChunkToMessage, getCurlModelCode, getPythonModelCode, hasModelMetering } from './helpers'; + +import { IModelExtended } from '../List/types'; +import { Message } from './types'; + +let mockHasBilling: boolean; + +jest.mock('product', () => ({ + product: { + get hasBilling() { + return mockHasBilling; + }, + }, +})); + +// exporter-proj runs the model, and importer-proj imports it +const model: IModelExtended = { + name: 'Qwen/Qwen3.6-27B-FP8', + project_name: 'exporter-proj', + endpoints: [{ project_name: 'importer-proj', base_url: 'https://gateway.example.com/importer-proj' }], +}; + +const makeUser = (globalRole: TUserRole): IUser => ({ + id: 'bob-id', + username: 'bob', + global_role: globalRole, + email: null, + permissions: [], + created_at: '2026-12-03T09:02:00+00:00', + active: true, +}); + +const makeProject = (projectName: string, projectRole: TProjectRole) => + ({ + project_name: projectName, + members: [{ user: { username: 'bob' }, project_role: projectRole }], + }) as IProject; + +beforeEach(() => { + mockHasBilling = true; +}); + +describe('Model address', () => { + test('keeps the slashes of the name and encodes the rest', () => { + expect(ROUTES.MODELS.DETAILS.FORMAT('exporter-proj', 'Qwen/Qwen3.8-27B-FP8')).toBe( + '/projects/exporter-proj/models/Qwen/Qwen3.8-27B-FP8', + ); + expect(ROUTES.MODELS.DETAILS.FORMAT('exporter-proj', 'my model?/v1#x')).toBe( + '/projects/exporter-proj/models/my%20model%3F/v1%23x', + ); + expect(ROUTES.MODELS.DETAILS.METERING.FORMAT('exporter-proj', 'Qwen/Qwen3.8-27B-FP8')).toBe( + '/projects/exporter-proj/models/Qwen/Qwen3.8-27B-FP8?tab=metering', + ); + }); + + test('a name with an empty part between slashes is encoded as a whole', () => { + expect(ROUTES.MODELS.DETAILS.FORMAT('exporter-proj', 'a//b')).toBe('/projects/exporter-proj/models/a%2F%2Fb'); + }); +}); + +describe('Model code', () => { + test('is for the chosen project, else for the first of the projects that have the model', () => { + const twoProjects = { + ...model, + endpoints: [ + ...model.endpoints, + { project_name: 'importer2-proj', base_url: 'https://gateway.example.com/importer2-proj' }, + ], + }; + + expect(getPythonModelCode({ model: twoProjects })).toContain('base_url="https://gateway.example.com/importer-proj"'); + expect(getPythonModelCode({ model: twoProjects, projectName: 'importer2-proj' })).toContain( + 'base_url="https://gateway.example.com/importer2-proj"', + ); + expect(getCurlModelCode({ model: twoProjects, projectName: 'importer2-proj' })).toContain( + 'curl https://gateway.example.com/importer2-proj/chat/completions ', + ); + }); + + test('quotes in the name of a model end neither the Python string nor the curl body', () => { + const quotedModel = { ...model, name: `it's a "model"` }; + + expect(getPythonModelCode({ model: quotedModel })).toContain(`model="it's a \\"model\\"",`); + expect(getCurlModelCode({ model: quotedModel })).toContain(`"model": "it'\\''s a \\"model\\"",`); + }); +}); + +describe('Chat reply', () => { + // A reply's stream, as an engine sends it: a delta in each chunk and, in the last chunk, why the stream ended + const getReply = (chunks: [delta: object, finishReason?: string][]) => + chunks.reduce( + (message, [delta, finishReason = null]) => + addChunkToMessage(message, { + choices: [{ index: 0, delta, finish_reason: finishReason }], + } as OpenAI.Chat.Completions.ChatCompletionChunk), + { role: 'assistant', content: '' }, + ); + + test('an answer that starts with blank lines shows without them', () => { + const reply = getReply([[{ reasoning_content: 'Hmm.' }], [{ content: '\n\n' }], [{ content: 'Paris.' }], [{}, 'stop']]); + + expect(reply.content).toBe('Paris.'); + }); + + test('the answer of a model that does not think', () => { + const reply = getReply([ + [{ role: 'assistant', content: '' }], + [{ content: 'Par' }], + [{ content: 'is.' }], + [{}, 'stop'], + ]); + + expect(reply).toEqual({ role: 'assistant', content: 'Paris.', thinking: '', finish_reason: 'stop' }); + }); + + test.each(['reasoning_content', 'reasoning'])('the thinking in `%s`, and then the answer', (field) => { + const reply = getReply([ + [{ role: 'assistant', content: null, [field]: 'The capital ' }], + [{ content: null, [field]: 'is Paris.' }], + [{ content: 'Paris.', [field]: null }], + [{}, 'stop'], + ]); + + expect(reply).toEqual({ + role: 'assistant', + content: 'Paris.', + thinking: 'The capital is Paris.', + finish_reason: 'stop', + }); + }); + + test('a reply that the limit of tokens cut before its answer', () => { + const reply = getReply([[{ reasoning_content: 'Five words: ' }], [{ reasoning_content: 'hello' }, 'length']]); + + expect(reply).toEqual({ role: 'assistant', content: '', thinking: 'Five words: hello', finish_reason: 'length' }); + }); +}); + +describe('Model metering tab', () => { + test('a global admin gets it', () => { + expect(hasModelMetering(model, [], makeUser('admin'))).toBe(true); + }); + + test.each(['exporter-proj', 'importer-proj'])('an admin of %s gets it', (projectName) => { + expect(hasModelMetering(model, [makeProject(projectName, 'admin')], makeUser('user'))).toBe(true); + }); + + test.each([ + ['exporter-proj', 'user'], + ['importer-proj', 'manager'], + ['other-proj', 'admin'], + ] as const)('a member of %s with the %s role does not get it', (projectName, projectRole) => { + expect(hasModelMetering(model, [makeProject(projectName, projectRole)], makeUser('user'))).toBe(false); + }); + + test('nobody gets it where the server has no metering', () => { + mockHasBilling = false; + + expect(hasModelMetering(model, [makeProject('exporter-proj', 'admin')], makeUser('admin'))).toBe(false); + }); +}); diff --git a/frontend/src/pages/Models/Details/helpers.ts b/frontend/src/pages/Models/Details/helpers.ts index 54d44e7c7e..3705c9bae7 100644 --- a/frontend/src/pages/Models/Details/helpers.ts +++ b/frontend/src/pages/Models/Details/helpers.ts @@ -1,16 +1,76 @@ +import type OpenAI from 'openai'; +import { product } from 'product'; + +import { GlobalUserRole, ProjectUserRole } from 'types'; + +import { getProjectRoleByUserName } from 'pages/Project/utils'; + import { getModelGateway } from '../helpers'; import { IModelExtended } from '../List/types'; +import { Message } from './types'; + +// The most tokens of a reply in the chat and in the code. A thinking model's thinking counts too +export const MAX_TOKENS = 4096; + +/** + * The URL through which the user accesses the model, for the chat and the code: the model endpoint of `projectName`, + * else the first one, which is of the model's own project for its members, else of the first of the user's projects + * that imports it. + */ +export const getModelUrl = (model?: IModelExtended, projectName?: IProject['project_name']) => { + const endpoint = model?.endpoints.find(({ project_name }) => project_name === projectName) ?? model?.endpoints[0]; + + return getModelGateway(endpoint?.base_url); +}; + +type TChatCompletionChunk = OpenAI.Chat.Completions.ChatCompletionChunk; + +// An engine with a reasoning parser streams a model's thinking apart from its answer: +// in `reasoning_content`, or in `reasoning` with some engines. The OpenAI client has neither field +type TChatCompletionDelta = TChatCompletionChunk['choices'][number]['delta'] & { + reasoning_content?: string | null; + reasoning?: string | null; +}; + +/** + * Adds a chunk of a reply's stream to the reply: its answer, its thinking and, at the end, why the stream ended. + */ +export const addChunkToMessage = (message: Message, chunk: TChatCompletionChunk): Message => { + const choice = chunk.choices[0]; + const delta: TChatCompletionDelta | undefined = choice?.delta; + + return { + role: delta?.role ?? message.role, + // A thinking model's answer starts with blank lines, which would show as a gap above it + content: (message.content + (delta?.content ?? '')).trimStart(), + thinking: (message.thinking ?? '') + (delta?.reasoning_content ?? delta?.reasoning ?? ''), + finish_reason: choice?.finish_reason ?? message.finish_reason, + }; +}; + +/** + * An imported model's name comes from another project and can be any string, so the code has it as a JSON string, + * which is a Python string too. + */ +const getModelNameLiteral = (model?: IModelExtended) => JSON.stringify(model?.name ?? ''); + +type TModelCodeParams = { + model?: IModelExtended; + // The project whose model endpoint the code uses + projectName?: IProject['project_name']; + token?: string; +}; -export const getPythonModelCode = ({ model, token }: { model?: IModelExtended | null; token?: string }) => { +export const getPythonModelCode = ({ model, projectName, token }: TModelCodeParams) => { return `from openai import OpenAI client = OpenAI( api_key="${token}", - base_url="${getModelGateway(model?.base_url ?? '')}" + base_url="${getModelUrl(model, projectName)}" ) response = client.chat.completions.create( - model="${model?.name ?? ''}", + model=${getModelNameLiteral(model)}, messages=[ { "role": "user", @@ -18,17 +78,20 @@ response = client.chat.completions.create( }, ], stream=True, - max_tokens=512, + max_tokens=${MAX_TOKENS}, )`; }; -export const getCurlModelCode = ({ model, token }: { model?: IModelExtended | null; token?: string }) => { - const url = getModelGateway(model?.base_url ?? '').replace(/\/$/, '') + '/chat/completions'; +export const getCurlModelCode = ({ model, projectName, token }: TModelCodeParams) => { + const url = getModelUrl(model, projectName).replace(/\/$/, '') + '/chat/completions'; + // The body is in single quotes, which a single quote of the name would end + const modelName = getModelNameLiteral(model).replace(/'/g, `'\\''`); + return `curl ${url} \\ -H "Content-Type: application/json" \\ -H "Authorization: Bearer ${token}" \\ -d '{ - "model": "${model?.name ?? ''}", + "model": ${modelName}, "messages": [ { "role": "user", @@ -36,6 +99,28 @@ export const getCurlModelCode = ({ model, token }: { model?: IModelExtended | nu } ], "stream": true, - "max_tokens": 512 + "max_tokens": ${MAX_TOKENS} }'`; }; + +/** + * The Metering tab is for global admins and for the admins of the project that runs the model + * or of a project that imports it, where the server has the metering API. + */ +export const hasModelMetering = (model: IModelExtended, projects: IProject[], user: IUser | null): boolean => { + if (!product.hasBilling || !user) { + return false; + } + + if (user.global_role === GlobalUserRole.ADMIN) { + return true; + } + + const modelProjectNames = [model.project_name, ...model.endpoints.map(({ project_name }) => project_name)]; + + return projects.some( + (project) => + modelProjectNames.includes(project.project_name) && + getProjectRoleByUserName(project, user.username) === ProjectUserRole.ADMIN, + ); +}; diff --git a/frontend/src/pages/Models/Details/hooks.ts b/frontend/src/pages/Models/Details/hooks.ts new file mode 100644 index 0000000000..6c68da8e8a --- /dev/null +++ b/frontend/src/pages/Models/Details/hooks.ts @@ -0,0 +1,50 @@ +import { useMemo } from 'react'; + +import { useAppSelector } from 'hooks'; + +import { selectUserData } from 'App/slice'; + +import { mergeModels } from '../helpers'; +import { useProjectsImports, useRunModels } from '../hooks'; +import { hasModelMetering } from './helpers'; + +/** + * Loads the model that `projectName` runs as `modelName`: the project's own model or one that the user's projects import. + */ +export const useModel = (projectName: IProject['project_name'], modelName: IModel['name']) => { + const userData = useAppSelector(selectUserData); + + // A project's own model comes from an active run or, when no active run has it, from the project's latest runs + const runModels = useRunModels( + { project_name: projectName }, + (activeModels) => !!activeModels && !activeModels.some(({ name }) => name === modelName), + ); + + // The model's endpoints come from the import lists of all the user's projects + const imports = useProjectsImports(); + + const isLoading = runModels.isFetching || imports.isFetching; + const error = isLoading ? undefined : (runModels.error ?? imports.error); + + const model = useMemo(() => { + if (isLoading || error) { + return; + } + + return mergeModels(runModels.models, imports.projectsImports).find( + (model) => model.project_name === projectName && model.name === modelName, + ); + }, [isLoading, error, runModels.models, imports.projectsImports, projectName, modelName]); + + return { + model, + isLoading, + error, + // Not while a request runs, and not when one has failed + isNotFound: !isLoading && !error && !model, + hasMetering: !!model && hasModelMetering(model, imports.projects, userData), + // The user's projects and their import lists, for the Exports tab + projects: imports.projects, + projectsImports: imports.projectsImports, + } as const; +}; diff --git a/frontend/src/pages/Models/Details/index.tsx b/frontend/src/pages/Models/Details/index.tsx index cad0d56c51..b112cadf28 100644 --- a/frontend/src/pages/Models/Details/index.tsx +++ b/frontend/src/pages/Models/Details/index.tsx @@ -1,77 +1,68 @@ -import React, { useEffect, useMemo, useRef, useState } from 'react'; -import { useForm } from 'react-hook-form'; +import React, { useEffect, useState } from 'react'; import { useTranslation } from 'react-i18next'; -import { useParams } from 'react-router-dom'; -import reactStringReplace from 'react-string-replace'; -import OpenAI from 'openai'; +import { Navigate, useParams, useSearchParams } from 'react-router-dom'; +import { product } from 'product'; import { - Avatar, + Alert, Box, Button, - ChatBubble, Code, - ColumnLayout, Container, ContentLayout, DetailsHeader, - FormTextarea, - Header, - ListEmptyMessage, Loader, Modal, - NavigateLink, + SelectCSD, SpaceBetween, Tabs, TokenVisibilityButton, } from 'components'; -import { useAppSelector, useBreadcrumbs, useNotifications } from 'hooks'; -import { copyToClipboard, riseRouterException, TOKEN_MASK } from 'libs'; -import { getExtendedModelFromRun } from 'libs/run'; +import { useAppSelector, useBreadcrumbs } from 'hooks'; +import { copyToClipboard, getServerError, riseRouterException, TOKEN_MASK } from 'libs'; import { ROUTES } from 'routes'; -import { useGetRunQuery } from 'services/run'; import { selectAuthToken } from 'App/slice'; -import { runIsStopped } from 'pages/Runs/utils'; -import { getModelGateway } from '../helpers'; -import { getCurlModelCode, getPythonModelCode } from './helpers'; +import { ModelExports } from './Exports'; +import { getCurlModelCode, getModelUrl, getPythonModelCode } from './helpers'; +import { useModel } from './hooks'; +import { ModelMetering } from './Metering'; +import { ModelDetails } from './ModelDetails'; -import { IModelExtended } from '../List/types'; -import { FormValues, Message, Role } from './types'; +import styles from './styles.module.scss'; -import css from './styles.module.scss'; - -const MESSAGE_ROLE_MAP: Record = { - tool: 'Tool', - system: 'System', - user: 'User', - assistant: 'Assistant', -}; +enum ModelTab { + Details = 'details', + Exports = 'exports', + Metering = 'metering', +} enum CodeTab { Python = 'python', Curl = 'curl', } -export const ModelDetails: React.FC = () => { +export const ModelDetailsPage: React.FC = () => { const { t } = useTranslation(); const token = useAppSelector(selectAuthToken); - const [messages, setMessages] = useState([]); const [viewCodeVisible, setViewCodeVisible] = useState(false); const [codeTab, setCodeTab] = useState(CodeTab.Python); const [isTokenVisible, setIsTokenVisible] = useState(false); - const [loading, setLoading] = useState(false); + const [codeProjectName, setCodeProjectName] = useState(); const params = useParams(); const paramProjectName = params.projectName ?? ''; - const paramRunName = params.runName ?? ''; - const openai = useRef(); - const textAreaRef = useRef(null); - const chatList = useRef(); - const [pushNotification] = useNotifications(); - - const messageForShowing = useMemo(() => messages.filter((m) => m.role !== 'system'), [messages]); + const paramModelName = params['*'] ?? ''; + const [searchParams] = useSearchParams(); + const isMeteringTab = searchParams.get('tab') === ROUTES.MODELS.DETAILS.METERING.TAB; + // Exports exist only where the server enables them + const isExportsTab = product.hasExports && searchParams.get('tab') === ROUTES.MODELS.DETAILS.EXPORTS.TAB; + + const { model, isLoading, error, isNotFound, hasMetering, projects, projectsImports } = useModel( + paramProjectName, + paramModelName, + ); useBreadcrumbs([ { @@ -87,392 +78,166 @@ export const ModelDetails: React.FC = () => { href: ROUTES.MODELS.LIST, }, { - text: paramRunName, - href: ROUTES.MODELS.DETAILS.FORMAT(paramProjectName, paramRunName), + text: paramModelName, + href: ROUTES.MODELS.DETAILS.FORMAT(paramProjectName, paramModelName), }, ]); - const closeViewCode = () => { - setViewCodeVisible(false); - setIsTokenVisible(false); - }; - - const scrollChatToBottom = () => { - if (!chatList.current) return; - - const { clientHeight, scrollHeight } = chatList.current; - chatList.current.scrollTo(0, scrollHeight - clientHeight); - }; - - const { data: runData, isLoading: isLoadingRun } = useGetRunQuery({ - project_name: paramProjectName, - run_name: paramRunName, - }); - useEffect(() => { - if (runData && runIsStopped(runData.status)) { + if (isNotFound) { riseRouterException(); } - }, [runData]); - - const modelData = useMemo(() => { - if (!runData) { - return; - } - - return getExtendedModelFromRun(runData); - }, [runData]); - - useEffect(() => { - if (!modelData) { - return; - } - - openai.current = new OpenAI({ - baseURL: getModelGateway(modelData.base_url), - apiKey: token, - dangerouslyAllowBrowser: true, - }); - }, [modelData, token]); - - useEffect(() => { - scrollChatToBottom(); - }, [messageForShowing]); - - const { handleSubmit, control, setValue, watch } = useForm(); - - const messageText = watch('message'); - - const sendRequest = async (messages: Message[]) => { - if (!openai.current) return Promise.reject('Model not found'); - - return openai.current.chat.completions.create({ - model: modelData?.name ?? '', - messages, - stream: true, - max_tokens: 512, - }); - }; - - const onSubmit = async (data: FormValues) => { - if (!data.message) { - return; - } - - let newMessages: Message[]; - - const newMessage: Message = { role: 'user', content: data.message }; - const messagesWithoutSystemMessage = messages.filter((m) => m.role !== 'system'); - - if (data.instructions?.length) { - newMessages = [{ role: 'system', content: data.instructions }, ...messagesWithoutSystemMessage, newMessage]; - } else { - newMessages = [...messagesWithoutSystemMessage, newMessage]; - } - - setMessages(newMessages); - setLoading(true); - - try { - const stream = await sendRequest(newMessages); - - setMessages((oldMessages) => [ - ...oldMessages, - { - role: 'assistant', - content: '', - }, - ]); - - setValue('message', ''); - setTimeout(onChangeMessage, 0); - - for await (const chunk of stream) { - setMessages((oldMessages) => { - const newMessages = [...oldMessages]; - - const lastMessage = newMessages.pop(); - - if (!lastMessage) { - return oldMessages; - } - - return [ - ...newMessages, - { - role: chunk.choices[0]?.delta?.role ?? lastMessage?.role, - content: (lastMessage?.content ?? '') + (chunk.choices[0]?.delta?.content ?? ''), - }, - ]; - }); - } - } catch (e) { - pushNotification({ - type: 'error', - content: t('common.server_error', { error: e }), - }); - - clearChat(); - } - - setLoading(false); - - setTimeout(() => { - textAreaRef.current?.querySelector('textarea')?.focus(); - }, 10); - }; - - const clearChat = () => { - setValue('message', ''); - setValue('instructions', ''); - - setMessages([]); - setTimeout(onChangeMessage, 10); - }; - - const renderMessageBody = (content: Message['content']) => { - const LANGUAGE_PATTERN = /^(```[A-Za-z]*)/m; - const PATTERN = /^([A-Za-z \t]*)```([A-Za-z]*)?\n([\s\S]*?)```([A-Za-z \t]*)*$/gm; - const languages: string[] = []; - - const matches = content.match(LANGUAGE_PATTERN); - - if (matches) { - [...matches].forEach((l) => l && languages.push(l.replace(/^```/, ''))); - } - - const replacedStrings = reactStringReplace(content, PATTERN, (match) => { - if (!match) { - return ''; - } + }, [isNotFound]); - return {match}; - }); - - return ( - - {replacedStrings.filter((line, index) => { - if ( - languages.includes(line as string) && - typeof replacedStrings[index + 1] !== 'string' && - typeof replacedStrings[index + 1] !== 'undefined' - ) { - return false; - } - - return true; - })} - - ); - }; - - const onChangeMessage = () => { - if (!textAreaRef.current) return; - - const textAreaElement = textAreaRef.current.querySelector('textarea'); - - if (!textAreaElement) return; - - textAreaElement.style.height = 'auto'; - textAreaElement.style.height = textAreaElement.scrollHeight + 'px'; + const closeViewCode = () => { + setViewCodeVisible(false); + setIsTokenVisible(false); }; - // eslint-disable-next-line @typescript-eslint/ban-ts-comment - // @ts-expect-error - const onKeyDown = (event) => { - const isCtrlOrShiftKey = event?.detail?.ctrlKey || event?.detail?.shiftKey; + const displayedToken = isTokenVisible ? token : TOKEN_MASK; - if (event?.detail?.keyCode === 13 && !isCtrlOrShiftKey) { - handleSubmit(onSubmit)(); - } else if (event?.detail?.keyCode === 13 && isCtrlOrShiftKey) { - event.preventDefault(); - setValue('message', messageText + '\n'); - setTimeout(onChangeMessage, 0); - } - }; + // The user's projects through which the model can be accessed now. The code is for the chosen one, else for the first + const codeProjectOptions = (model?.endpoints ?? []) + .filter(({ base_url }) => base_url) + .map(({ project_name }) => ({ label: project_name, value: project_name })); - const displayedToken = isTokenVisible ? token : TOKEN_MASK; + const codeProjectOption = codeProjectOptions.find(({ value }) => value === codeProjectName) ?? codeProjectOptions[0]; + const codeParams = { model, projectName: codeProjectOption?.value }; - const pythonCode = getPythonModelCode({ model: modelData, token: displayedToken }); + const pythonCode = getPythonModelCode({ ...codeParams, token: displayedToken }); - const curlCode = getCurlModelCode({ model: modelData, token: displayedToken }); + const curlCode = getCurlModelCode({ ...codeParams, token: displayedToken }); const onCopyCode = () => { switch (codeTab) { case CodeTab.Python: - return copyToClipboard(getPythonModelCode({ model: modelData, token })); + return copyToClipboard(getPythonModelCode({ ...codeParams, token })); case CodeTab.Curl: - return copyToClipboard(getCurlModelCode({ model: modelData, token })); + return copyToClipboard(getCurlModelCode({ ...codeParams, token })); } }; - console.log({ modelData }); - - return ( - - {paramRunName} - - } - actionButtons={ - - } - /> - } - > - {isLoadingRun && ( - - - - )} - - {modelData && ( - <> -
-
- {t('common.general')}}> - -
- {t('models.details.run_name')} + const tabs = [ + { + label: t('models.details.title'), + id: ModelTab.Details, + href: ROUTES.MODELS.DETAILS.FORMAT(paramProjectName, paramModelName), + }, + product.hasExports && { + label: t('models.exports.title'), + id: ModelTab.Exports, + href: ROUTES.MODELS.DETAILS.EXPORTS.FORMAT(paramProjectName, paramModelName), + }, + hasMetering && { + label: t('models.metering.title'), + id: ModelTab.Metering, + href: ROUTES.MODELS.DETAILS.METERING.FORMAT(paramProjectName, paramModelName), + }, + ].filter(Boolean) as { + label: string; + id: ModelTab; + href: string; + }[]; -
- - {modelData.run_name} - -
-
+ const activeTabId = isMeteringTab ? ModelTab.Metering : isExportsTab ? ModelTab.Exports : ModelTab.Details; + const modelKey = `${paramProjectName}/${paramModelName}`; -
- {t('models.model_name')} -
{modelData.name}
-
+ // A user who doesn't get the Metering tab doesn't get its page either + if (model && isMeteringTab && !hasMetering) { + return ; + } -
- {t('models.type')} -
{modelData.type}
-
-
-
+ return ( +
+ setViewCodeVisible(true)}> + {t('models.endpoint')} + + } + /> + } + > + {isLoading && ( + + + + )} + + {error && {t('common.server_error', { error: getServerError(error) })}} + + {model && ( + <> + + + {/* Remount tabs on model change so the chat and the filters of one model don't show for another */} + {activeTabId === ModelTab.Details && } + {activeTabId === ModelTab.Exports && ( + + )} + {activeTabId === ModelTab.Metering && } + + )} + + + + + + } + onDismiss={closeViewCode} + > + + {t('models.details.view_code_description')} + +
+
+ + + +
-
-
-
- - - -
- {!messageForShowing.length && ( - - )} - - {messageForShowing.map((message, index) => ( -
- - } - > - {renderMessageBody(message.content || '...')} - + setCodeTab(detail.activeTabId as CodeTab)} + activeTabId={codeTab} + actions={ + codeProjectOptions.length > 1 && ( +
+ setCodeProjectName(detail.selectedOption.value)} + />
- ))} -
- -
- - -
- -
-
- + ) + } + tabs={[ + { + label: 'python', + id: CodeTab.Python, + content: {pythonCode}, + }, + { + label: 'curl', + id: CodeTab.Curl, + content: {curlCode}, + }, + ]} + />
- - - - - } - onDismiss={closeViewCode} - > - - {t('models.details.view_code_description')} - -
-
- - - - -
- - setCodeTab(detail.activeTabId as CodeTab)} - activeTabId={codeTab} - tabs={[ - { - label: 'python', - id: CodeTab.Python, - content: {pythonCode}, - }, - { - label: 'curl', - id: CodeTab.Curl, - content: {curlCode}, - }, - ]} - /> -
-
-
- - )} - + + +
); }; diff --git a/frontend/src/pages/Models/Details/styles.module.scss b/frontend/src/pages/Models/Details/styles.module.scss index 7196f62526..27dd30ecde 100644 --- a/frontend/src/pages/Models/Details/styles.module.scss +++ b/frontend/src/pages/Models/Details/styles.module.scss @@ -1,82 +1,22 @@ -.modelDetailsLayout { - display: flex; - flex-direction: column; - height: calc(100vh - 159px); - margin-bottom: -40px; -} - -.general { - flex-shrink: 0; -} - -.modelForm { - position: relative; - flex-grow: 1; - min-height: 0; - display: grid; - grid-template-columns: 300px 1fr; - grid-template-rows: 60px 1fr auto; - grid-template-areas: "sidebar buttons" - "sidebar chat" - "sidebar form"; -} - -.side { - grid-area: sidebar; - border-right: 1px solid #b6bec9; - padding-right: 20px; - padding-top: 20px; - - textarea { - resize: none !important; - } -} - -.buttons { - display: flex; - justify-content: flex-end; - align-items: center; - gap: 12px; +.page { + height: 100%; - & > button { - flex-shrink: 0; + & [class^="awsui_tabs-content"] { + display: none; } -} -.chat { - overflow-y: auto; - grid-area: chat; - padding: 0 20px 20px; - display: flex; - flex-direction: column; - gap: 20px; + & > [class^="awsui_layout"] { + height: 100%; - .message { - p { - white-space: pre-wrap; + & > [class^="awsui_content"] { + display: flex; + flex-direction: column; + gap: 20px; + height: 100%; } } } -.messageForm { - display: grid; - grid-template-columns: 1fr auto; - column-gap: 20px; - grid-area: form; - border-top: 1px solid #b6bec9; - margin-right: -20px; - padding: 20px; - - textarea { - resize: none !important; - max-height: calc(100vh - 550px); - } - - .buttons { - align-self: flex-end; - } -} - .viewCodeControls { position: relative; display: flex; @@ -88,4 +28,8 @@ right: 20px; top: 80px; } + + .codeProject { + min-width: 240px; + } } diff --git a/frontend/src/pages/Models/Details/types.ts b/frontend/src/pages/Models/Details/types.ts index 82722b7ff6..0f9ccc73f5 100644 --- a/frontend/src/pages/Models/Details/types.ts +++ b/frontend/src/pages/Models/Details/types.ts @@ -1,5 +1,4 @@ export interface FormValues { - instructions?: string; message: string; } @@ -8,4 +7,8 @@ export type Role = 'system' | 'user' | 'assistant' | 'tool'; export interface Message { role: Role; content: string; + // What a thinking model streamed before its answer. The chat shows it, and a request doesn't send it back + thinking?: string; + // Why the reply's stream ended: `length` when the reply was cut at the limit of tokens + finish_reason?: string | null; } diff --git a/frontend/src/pages/Models/List/Preferences/consts.ts b/frontend/src/pages/Models/List/Preferences/consts.ts index d58d869abc..9b14423360 100644 --- a/frontend/src/pages/Models/List/Preferences/consts.ts +++ b/frontend/src/pages/Models/List/Preferences/consts.ts @@ -4,16 +4,16 @@ export const DEFAULT_PREFERENCES: CollectionPreferencesProps.Preferences = { pageSize: 30, contentDisplay: [ { id: 'model_name', visible: true }, - { id: 'type', visible: true }, + { id: 'status', visible: true }, + { id: 'project', visible: true }, { id: 'run', visible: true }, - { id: 'resources', visible: true }, - { id: 'price', visible: true }, { id: 'submitted_at', visible: true }, { id: 'user', visible: true }, - { id: 'repository', visible: true }, - { id: 'backend', visible: true }, // hidden by default - { id: 'url', visible: false }, + { id: 'resources', visible: false }, + { id: 'price', visible: false }, + { id: 'repository', visible: false }, + { id: 'backend', visible: false }, ], wrapLines: false, stripedRows: false, diff --git a/frontend/src/pages/Models/List/Preferences/index.tsx b/frontend/src/pages/Models/List/Preferences/index.tsx index ed546d5949..6e1b24758a 100644 --- a/frontend/src/pages/Models/List/Preferences/index.tsx +++ b/frontend/src/pages/Models/List/Preferences/index.tsx @@ -19,17 +19,16 @@ export const Preferences: React.FC = () => { title: t('common.select_visible_columns'), options: [ { id: 'model_name', label: t('models.model_name'), alwaysVisible: true }, - { id: 'type', label: `${t('models.type')}` }, - + { id: 'status', label: `${t('models.status')}` }, + { id: 'project', label: `${t('models.project')}` }, { id: 'run', label: `${t('models.run')}` }, - { id: 'resources', label: `${t('models.resources')}` }, - { id: 'price', label: `${t('models.price')}` }, { id: 'submitted_at', label: `${t('models.submitted_at')}` }, { id: 'user', label: `${t('models.user')}` }, + // hidden by default + { id: 'resources', label: `${t('models.resources')}` }, + { id: 'price', label: `${t('models.price')}` }, { id: 'repository', label: `${t('models.repository')}` }, { id: 'backend', label: `${t('models.backend')}` }, - // hidden by default - { id: 'url', label: `${t('models.url')}` }, ], }} /> diff --git a/frontend/src/pages/Models/List/Preferences/useModelListPreferences.ts b/frontend/src/pages/Models/List/Preferences/useModelListPreferences.ts index 14f17a8850..a27bc65b42 100644 --- a/frontend/src/pages/Models/List/Preferences/useModelListPreferences.ts +++ b/frontend/src/pages/Models/List/Preferences/useModelListPreferences.ts @@ -5,8 +5,9 @@ import { useLocalStorageState } from 'hooks/useLocalStorageState'; import { DEFAULT_PREFERENCES } from './consts'; export const useModelListPreferences = () => { + // A new key: preferences saved under `model-list-preferences` have the list's old columns and would hide Status and Project const [preferences, setPreferences] = useLocalStorageState( - 'model-list-preferences', + 'model-list-preferences-v2', DEFAULT_PREFERENCES, ); diff --git a/frontend/src/pages/Models/List/hooks.tsx b/frontend/src/pages/Models/List/hooks.tsx index 7994b6e78f..d19da3c66e 100644 --- a/frontend/src/pages/Models/List/hooks.tsx +++ b/frontend/src/pages/Models/List/hooks.tsx @@ -3,10 +3,11 @@ import { useTranslation } from 'react-i18next'; import { useSearchParams } from 'react-router-dom'; import { format } from 'date-fns'; -import type { PropertyFilterProps } from 'components'; +import type { PropertyFilterProps, ToggleProps } from 'components'; import { Button, ListEmptyMessage, NavigateLink, TableProps } from 'components'; import { DATE_TIME_FORMAT } from 'consts'; +import { useLocalStorageState } from 'hooks'; import { EMPTY_QUERY, getNamePatternFilterRequestParams, @@ -18,9 +19,9 @@ import { ROUTES } from 'routes'; import { useLazyGetProjectsQuery } from 'services/project'; import { useLazyGetUserListQuery } from 'services/user'; -import { getModelGateway } from '../helpers'; +import { ModelStatus } from '../components/ModelStatus'; -import { IModelExtended } from './types'; +import { IModelExtended, TModelsFilters } from './types'; export const useColumnsDefinitions = () => { const { t } = useTranslation(); @@ -31,62 +32,64 @@ export const useColumnsDefinitions = () => { header: t('models.model_name'), cell: (item) => { return ( - - {item.name} - + {item.name} ); }, }, { - id: 'type', - header: `${t('models.type')}`, - cell: (item) => item.type, + id: 'status', + header: `${t('models.status')}`, + cell: (item) => , }, { - id: 'url', - header: `${t('models.url')}`, - cell: (item) => getModelGateway(item.base_url), + id: 'project', + header: `${t('models.project')}`, + cell: (item) => ( + {item.project_name} + ), }, { id: 'run', header: `${t('models.run')}`, - cell: (item) => ( - - {item.run_name} - - ), + cell: (item) => + item.run ? ( + + {item.run.run_name} + + ) : ( + '-' + ), }, { id: 'resources', header: `${t('models.resources')}`, - cell: (item) => item.resources, + cell: (item) => item.run?.resources ?? '-', }, { id: 'price', header: `${t('models.price')}`, - cell: (item) => (item.price ? `$${item.price}` : null), + cell: (item) => (item.run?.price ? `$${item.run.price}` : '-'), }, { id: 'submitted_at', header: `${t('models.submitted_at')}`, - cell: (item) => format(new Date(item.submitted_at), DATE_TIME_FORMAT), + cell: (item) => (item.run ? format(new Date(item.run.submitted_at), DATE_TIME_FORMAT) : '-'), }, { id: 'user', header: `${t('models.user')}`, - cell: (item) => {item.user}, + cell: (item) => + item.run ? {item.run.user} : '-', }, { id: 'repository', header: `${t('models.repository')}`, - cell: (item) => item.repository, + cell: (item) => item.run?.repository ?? '-', }, { id: 'backend', header: `${t('models.backend')}`, - cell: (item) => item.backend, + cell: (item) => item.run?.backend ?? '-', }, ]; @@ -125,7 +128,7 @@ export const useEmptyMessages = ({ return { renderEmptyMessage, renderNoMatchMessage } as const; }; -type RequestParamsKeys = keyof Pick; +type RequestParamsKeys = keyof TModelsFilters; const filterKeys: Record = { PROJECT_NAME: 'project_name', @@ -136,7 +139,7 @@ const limit = 100; export const useFilters = () => { const [searchParams, setSearchParams] = useSearchParams(); - + const [onlyActive, setOnlyActive] = useLocalStorageState('model-list-filter-only-active', true); const [filteringOptions, setFilteringOptions] = useState([]); const [filteringStatusType, setFilteringStatusType] = useState(); const [getProjects] = useLazyGetProjectsQuery(); @@ -181,10 +184,14 @@ export const useFilters = () => { }); }; + const onChangeOnlyActive: ToggleProps['onChange'] = ({ detail }) => { + setOnlyActive(detail.checked); + }; + const filteringRequestParams = useMemo(() => { return tokensToRequestParams({ tokens: propertyFilterQuery.tokens, - }) as Partial; + }) as TModelsFilters; }, [propertyFilterQuery]); const handleLoadItems: PropertyFilterProps['onLoadItems'] = async ({ detail: { filteringProperty, filteringText } }) => { @@ -226,6 +233,8 @@ export const useFilters = () => { onChangePropertyFilter, filteringOptions, filteringProperties, + onlyActive, + onChangeOnlyActive, filteringStatusType, handleLoadItems, } as const; diff --git a/frontend/src/pages/Models/List/index.tsx b/frontend/src/pages/Models/List/index.tsx index 769c8bc105..15e3aee841 100644 --- a/frontend/src/pages/Models/List/index.tsx +++ b/frontend/src/pages/Models/List/index.tsx @@ -1,14 +1,15 @@ -import React from 'react'; +import React, { useMemo } from 'react'; import { useTranslation } from 'react-i18next'; -import { Button, Header, Loader, PropertyFilter, Table } from 'components'; +import { Alert, Button, Header, PropertyFilter, Table, Toggle } from 'components'; -import { DEFAULT_TABLE_PAGE_SIZE } from 'consts'; -import { useBreadcrumbs, useCollection, useInfiniteScroll } from 'hooks'; +import { useBreadcrumbs, useCollection } from 'hooks'; +import { getServerError } from 'libs'; import { ROUTES } from 'routes'; -import { useLazyGetModelsQuery } from 'services/run'; import { useModelListPreferences } from './Preferences/useModelListPreferences'; +import { getImportedModelStatus, mergeModels } from '../helpers'; +import { useProjectsImports, useRunModels } from '../hooks'; import { useColumnsDefinitions, useEmptyMessages, useFilters } from './hooks'; import { Preferences } from './Preferences'; @@ -26,6 +27,8 @@ export const List: React.FC = () => { filteringOptions, filteringProperties, filteringRequestParams, + onlyActive, + onChangeOnlyActive, filteringStatusType, handleLoadItems, } = useFilters(); @@ -40,12 +43,38 @@ export const List: React.FC = () => { const { columns } = useColumnsDefinitions(); const [preferences] = useModelListPreferences(); - const { data, isLoading, refreshList, isLoadingMore } = useInfiniteScroll({ - useLazyQuery: useLazyGetModelsQuery, - args: { ...filteringRequestParams, limit: DEFAULT_TABLE_PAGE_SIZE }, + // With `Active models` off, the list also has the models of the latest runs + const { + models: runModels, + isFetching: isFetchingRuns, + error, + refetch: refetchRuns, + } = useRunModels(filteringRequestParams, () => !onlyActive); - getPaginationParams: (lastModel) => ({ prev_submitted_at: lastModel.submitted_at }), - }); + const { + projectsImports, + isFetching: isFetchingImports, + refetch: refetchImports, + } = useProjectsImports(filteringRequestParams); + + const models = useMemo(() => { + // A failed runs request shows its error, not the part of the models that was loaded + if (error) { + return []; + } + + const mergedModels = mergeModels(runModels, projectsImports); + + // With `Active models` on, only the active runs are read, and an imported model is active while a service serves it + return mergedModels.filter((model) => !onlyActive || model.run || getImportedModelStatus(model) === 'running'); + }, [error, runModels, projectsImports, onlyActive]); + + const isLoading = isFetchingRuns || isFetchingImports; + + const refreshModels = () => { + refetchRuns(); + refetchImports(); + }; const isDisabledClearFilter = !propertyFilterQuery.tokens.length; @@ -54,9 +83,13 @@ export const List: React.FC = () => { isDisabledClearFilter, }); - const { items, collectionProps } = useCollection(data, { + const { items, collectionProps } = useCollection(models, { filtering: { - empty: renderEmptyMessage(), + empty: error ? ( + {t('common.server_error', { error: getServerError(error) })} + ) : ( + renderEmptyMessage() + ), noMatch: renderNoMatchMessage(), }, selection: {}, @@ -78,9 +111,9 @@ export const List: React.FC = () => { actions={
+ +
+ + {t('models.active_only')} + +
} preferences={} - footer={} /> ); }; diff --git a/frontend/src/pages/Models/List/types.ts b/frontend/src/pages/Models/List/types.ts index 196aec7387..fbec7ba3d4 100644 --- a/frontend/src/pages/Models/List/types.ts +++ b/frontend/src/pages/Models/List/types.ts @@ -1,12 +1,32 @@ -export interface IModelExtended extends Partial { +export interface IModelEndpoint { + project_name: IProject['project_name']; + base_url: string | null; +} + +export interface IModelRun { id: string; run_name: string; - project_name: string; + status: TJobStatus; + termination_reason?: string | null; + status_message: string; submitted_at: string; user: string; resources: string | null; price: number | null; - region: string | null; repository: string | null; backend: TBackendType | null; } + +export interface IModelExtended { + name: string; + // The project that runs the model + project_name: IProject['project_name']; + // The model endpoints of the user's projects that have the model + endpoints: IModelEndpoint[]; + // An imported model has no run in the user's projects + run?: IModelRun; +} + +export type TModelsFilters = Pick; + +export type TModelsRequestParams = TModelsFilters & Pick; diff --git a/frontend/src/pages/Models/components/ModelStatus/index.tsx b/frontend/src/pages/Models/components/ModelStatus/index.tsx new file mode 100644 index 0000000000..6d5ecca095 --- /dev/null +++ b/frontend/src/pages/Models/components/ModelStatus/index.tsx @@ -0,0 +1,39 @@ +import React from 'react'; +import { useTranslation } from 'react-i18next'; + +import { StatusIndicator } from 'components'; + +import { getStatusIconColor, getStatusIconType } from 'libs/run'; + +import { getImportedModelStatus } from '../../helpers'; + +import { IModelExtended } from '../../List/types'; + +type ModelStatusProps = { + model: IModelExtended; +}; + +export const ModelStatus: React.FC = ({ model }) => { + const { t } = useTranslation(); + + if (model.run) { + const { status, termination_reason, status_message } = model.run; + + return ( + + {status_message} + + ); + } + + const status = getImportedModelStatus(model); + + return ( + + {t(`projects.run.statuses.${status}`)} + + ); +}; diff --git a/frontend/src/pages/Models/helpers.test.ts b/frontend/src/pages/Models/helpers.test.ts new file mode 100644 index 0000000000..1ace879383 --- /dev/null +++ b/frontend/src/pages/Models/helpers.test.ts @@ -0,0 +1,103 @@ +import { getImportedModelStatus, isModelServed, mergeModels } from './helpers'; + +import { IModelExtended } from './List/types'; + +const QWEN = 'Qwen/Qwen3.6-27B-FP8'; + +const getUrl = (projectName: string) => `https://gateway.example.com/${projectName}`; + +const makeModel = (projectName: string): IModelExtended => ({ + name: QWEN, + project_name: projectName, + endpoints: [{ project_name: projectName, base_url: getUrl(projectName) }], + run: { + id: `${projectName}-run-id`, + run_name: 'qwen-service', + status: 'running', + status_message: 'Running', + submitted_at: '2026-12-03T09:02:00+00:00', + user: 'alice', + resources: null, + price: null, + repository: null, + backend: null, + }, +}); + +const makeProjectImports = (projectName: string, baseUrl: string | null = getUrl(projectName)): TProjectImports => ({ + project_name: projectName, + imports: [ + { export: { name: 'models', project_name: 'exporter-proj', exported_models: [{ name: QWEN, base_url: baseUrl }] } }, + ], +}); + +describe('Models helpers', () => { + test("a model that two of the user's projects import is one row", () => { + const models = mergeModels([], [makeProjectImports('importer-proj'), makeProjectImports('importer2-proj')]); + + expect(models).toEqual([ + { + name: QWEN, + project_name: 'exporter-proj', + endpoints: [ + { project_name: 'importer-proj', base_url: getUrl('importer-proj') }, + { project_name: 'importer2-proj', base_url: getUrl('importer2-proj') }, + ], + }, + ]); + }); + + test('an imported model joins the row of its run, and a model of another project with the same name does not', () => { + const models = mergeModels( + [makeModel('exporter-proj'), makeModel('other-proj')], + [makeProjectImports('importer-proj')], + ); + + expect(models.map(({ project_name, run }) => [project_name, run?.id])).toEqual([ + ['exporter-proj', 'exporter-proj-run-id'], + ['other-proj', 'other-proj-run-id'], + ]); + expect(models[0].endpoints.map(({ project_name }) => project_name)).toEqual(['exporter-proj', 'importer-proj']); + expect(models[1].endpoints.map(({ project_name }) => project_name)).toEqual(['other-proj']); + }); + + test('a model takes its run that is not finished, though a later run of it has finished', () => { + const running = makeModel('exporter-proj'); + + const makeFinishedModel = (runId: string): IModelExtended => ({ + ...running, + run: running.run && { ...running.run, id: runId, status: 'terminated' }, + }); + + // The latest run first + const [later, earlier] = [makeFinishedModel('later-run-id'), makeFinishedModel('earlier-run-id')]; + + expect(mergeModels([later, running], [])).toEqual([running]); + expect(mergeModels([later, earlier], [])).toEqual([later]); + }); + + test("a project's own model is served until its run is finished, and keeps its URL afterwards", () => { + const running = makeModel('exporter-proj'); + const stopped: IModelExtended = { ...running, run: running.run && { ...running.run, status: 'terminated' } }; + + expect(isModelServed(running)).toBe(true); + expect(isModelServed(stopped)).toBe(false); + expect(stopped.endpoints[0].base_url).toBe(getUrl('exporter-proj')); + }); + + test('an imported model is served while its import list gives it a URL', () => { + const [running] = mergeModels([], [makeProjectImports('importer-proj')]); + const [stopped] = mergeModels([], [makeProjectImports('importer-proj', null)]); + + expect(isModelServed(running)).toBe(true); + expect(isModelServed(stopped)).toBe(false); + }); + + test('an imported model is running with a URL and stopped without one', () => { + const [running] = mergeModels([], [makeProjectImports('importer-proj')]); + const [stopped] = mergeModels([], [makeProjectImports('importer-proj', null)]); + + expect(getImportedModelStatus(running)).toBe('running'); + expect(getImportedModelStatus(stopped)).toBe('stopped'); + }); +}); diff --git a/frontend/src/pages/Models/helpers.ts b/frontend/src/pages/Models/helpers.ts index 24d297dd26..1f07d354f4 100644 --- a/frontend/src/pages/Models/helpers.ts +++ b/frontend/src/pages/Models/helpers.ts @@ -1,6 +1,12 @@ +import { groupBy as _groupBy, uniqBy as _uniqBy } from 'lodash'; + import { isValidUrl } from 'libs'; -export const getModelGateway = (baseUrl?: IModel['base_url']) => { +import { finishedJobs } from 'pages/Runs/constants'; + +import { IModelEndpoint, IModelExtended } from './List/types'; + +export const getModelGateway = (baseUrl?: IModelEndpoint['base_url']) => { if (!baseUrl) { return ''; } @@ -11,3 +17,50 @@ export const getModelGateway = (baseUrl?: IModel['base_url']) => { return document.location.origin + baseUrl; }; + +/** + * Merges the models of runs and the models that the user's projects import, so that a model shows once. + * A model is the project that runs it and its name. It takes the first of its runs that isn't finished, else its latest run: + * `models` have the latest run first. It takes an endpoint from each of the user's projects that has it. + */ +export const mergeModels = (models: IModelExtended[], projectsImports: TProjectImports[]): IModelExtended[] => { + const importedModels = projectsImports.flatMap(({ project_name, imports }) => + imports.flatMap(({ export: { project_name: exportProjectName, exported_models } }) => + exported_models.map( + ({ name, base_url }): IModelExtended => ({ + name, + project_name: exportProjectName, + endpoints: [{ project_name, base_url }], + }), + ), + ), + ); + + const modelGroups = _groupBy([...models, ...importedModels], ({ project_name, name }) => `${project_name}/${name}`); + + return Object.values(modelGroups).map((sameModels) => { + // A finished run has the status of its latest job submission + const model = sameModels.find(({ run }) => run && !finishedJobs.includes(run.status)) ?? sameModels[0]; + + return { + ...model, + endpoints: _uniqBy( + [model, ...sameModels].flatMap(({ endpoints }) => endpoints), + 'project_name', + ), + }; + }); +}; + +/** + * An importer can't see the exporter's run. Its import list gives the model a URL while a service serves it. + */ +export const getImportedModelStatus = (model: IModelExtended): 'running' | 'stopped' => + model.endpoints.some(({ base_url }) => base_url) ? 'running' : 'stopped'; + +/** + * A model answers requests while a service serves it: a project's own model until its run is finished, + * an imported one while its import list gives it a URL. A model that isn't served keeps the URL of its last run. + */ +export const isModelServed = (model: IModelExtended): boolean => + model.run ? !finishedJobs.includes(model.run.status) : getImportedModelStatus(model) === 'running'; diff --git a/frontend/src/pages/Models/hooks.test.tsx b/frontend/src/pages/Models/hooks.test.tsx new file mode 100644 index 0000000000..34634b424b --- /dev/null +++ b/frontend/src/pages/Models/hooks.test.tsx @@ -0,0 +1,226 @@ +/** @jest-environment node */ +import React from 'react'; +import { Provider } from 'react-redux'; +import { act, create, ReactTestRenderer } from 'react-test-renderer'; +import { configureStore } from '@reduxjs/toolkit'; +import { QueryStatus } from '@reduxjs/toolkit/query'; + +import { importApi } from 'services/import'; +import { projectApi } from 'services/project'; +import { runApi } from 'services/run'; + +import { useModel } from './Details/hooks'; +import { useProjectsImports } from './hooks'; + +let mockHasExports: boolean; + +jest.mock('product', () => ({ + product: { + get hasExports() { + return mockHasExports; + }, + }, +})); + +jest.mock('api', () => ({ + API: { + PROJECTS: { + LIST: () => 'http://localhost/api/projects/list', + IMPORTS_LIST: (project: string) => `http://localhost/api/project/${project}/imports/list`, + }, + RUNS: { + LIST: () => 'http://localhost/api/runs/list', + }, + }, +})); + +jest.mock('App/slice', () => ({ selectUserName: () => 'bob', selectUserData: () => null })); +jest.mock('hooks', () => ({ useAppSelector: (selector: () => unknown) => selector() })); + +const QWEN = 'Qwen/Qwen3.6-27B-FP8'; + +// A run of exporter-proj that serves the model +const qwenRun = { + id: 'run-id', + project_name: 'exporter-proj', + user: 'alice', + submitted_at: '2026-12-03T09:02:00+00:00', + status: 'running', + run_spec: { run_name: 'qwen-service' }, + service: { model: { name: QWEN, type: 'chat', base_url: 'https://gateway.example.com/exporter-proj' } }, +}; + +const createStore = () => + configureStore({ + reducer: { + app: (state = { authData: { token: 'token' } }) => state, + [projectApi.reducerPath]: projectApi.reducer, + [importApi.reducerPath]: importApi.reducer, + [runApi.reducerPath]: runApi.reducer, + }, + middleware: (getDefaultMiddleware) => + getDefaultMiddleware().concat(projectApi.middleware, importApi.middleware, runApi.middleware), + }); + +const jsonResponse = (data: unknown, status = 200) => + new Response(JSON.stringify(data), { status, headers: { 'Content-Type': 'application/json' } }); + +describe('Models hooks', () => { + let store: ReturnType; + let rendered: ReactTestRenderer | undefined; + let fetchMock: jest.SpyInstance; + // The path of each request and, when it has one, its body + let requests: [string, unknown?][]; + // What the server answers + let runs: unknown[]; + let runsStatus: number; + let importsStatuses: Record; + + const renderHook = async (useHook: () => Result) => { + const result = {} as { current: Result }; + + const Harness = () => { + result.current = useHook(); + return null; + }; + + await act(async () => { + rendered = create( + + + , + ); + }); + + // One request follows another, like the import lists after the projects list. So wait until none has run for a while + for (let idleTurns = 0; idleTurns < 3; idleTurns = isFetching() ? 0 : idleTurns + 1) { + await act(async () => { + await new Promise((resolve) => setImmediate(resolve)); + }); + } + + return result; + }; + + const isFetching = () => + [projectApi, importApi, runApi].some(({ reducerPath }) => + Object.values(store.getState()[reducerPath].queries).some((query) => query?.status === QueryStatus.pending), + ); + + // Leaves the page. The cache then drops the answers that nothing uses any more, in a timer that this one follows + const unmount = async () => { + act(() => rendered?.unmount()); + rendered = undefined; + + await act(async () => { + await new Promise((resolve) => setTimeout(resolve)); + }); + }; + + beforeEach(() => { + store = createStore(); + requests = []; + runs = []; + runsStatus = 200; + importsStatuses = {}; + fetchMock = jest.spyOn(globalThis, 'fetch').mockImplementation(async (input) => { + const request = input as Request; + const { pathname } = new URL(request.url); + const loggedRequest: [string, unknown?] = [pathname]; + + requests.push(loggedRequest); + + const body = await request.text(); + + if (body) { + loggedRequest.push(JSON.parse(body)); + } + + if (pathname === '/api/runs/list') { + return jsonResponse(runsStatus === 200 ? runs : { detail: 'error' }, runsStatus); + } + + if (pathname === '/api/projects/list') { + return jsonResponse({ + projects: [{ project_name: 'importer-proj' }, { project_name: 'importer2-proj' }], + total_count: 2, + }); + } + + return jsonResponse({ imports: [], warnings: [] }, importsStatuses[pathname.split('/')[3]]); + }); + }); + + afterEach(async () => { + await unmount(); + store.dispatch(projectApi.util.resetApiState()); + store.dispatch(importApi.util.resetApiState()); + store.dispatch(runApi.util.resetApiState()); + fetchMock.mockRestore(); + }); + + test("requests the import list of each of the user's projects where the server has exports", async () => { + mockHasExports = true; + await renderHook(() => useProjectsImports()); + + expect(requests).toEqual([ + ['/api/projects/list', { username: 'bob', return_total_count: true }], + ['/api/project/importer-proj/imports/list'], + ['/api/project/importer2-proj/imports/list'], + ]); + }); + + test('requests nothing where the server has no exports', async () => { + mockHasExports = false; + await renderHook(() => useProjectsImports()); + + expect(requests).toEqual([]); + }); + + test('leaves out an import list that failed, and keeps the others', async () => { + mockHasExports = true; + importsStatuses = { 'importer2-proj': 403 }; + + const result = await renderHook(() => useProjectsImports()); + + expect(result.current.projectsImports).toEqual([{ project_name: 'importer-proj', imports: [] }]); + }); + + test.each([ + ['is not found when the requests have answered without it', 200, true], + ['is an error, and not "not found", when the runs request has failed', 500, false], + ])('a model %s', async (name, status, isNotFound) => { + mockHasExports = true; + runsStatus = status; + + const result = await renderHook(() => useModel('exporter-proj', QWEN)); + + expect(result.current.isLoading).toBe(false); + expect(result.current.isNotFound).toBe(isNotFound); + expect(!!result.current.error).toBe(!isNotFound); + }); + + test('a model that has started since a visit to its page is not "not found" in any render of the next visit', async () => { + mockHasExports = true; + + const firstVisit = await renderHook(() => useModel('exporter-proj', QWEN)); + + expect(firstVisit.current.isNotFound).toBe(true); + + await unmount(); + runs = [qwenRun]; + + const isNotFoundInRenders: boolean[] = []; + + const nextVisit = await renderHook(() => { + const result = useModel('exporter-proj', QWEN); + + isNotFoundInRenders.push(result.isNotFound); + + return result; + }); + + expect(isNotFoundInRenders).not.toContain(true); + expect(nextVisit.current.model?.name).toBe(QWEN); + }); +}); diff --git a/frontend/src/pages/Models/hooks.ts b/frontend/src/pages/Models/hooks.ts new file mode 100644 index 0000000000..85ec79955c --- /dev/null +++ b/frontend/src/pages/Models/hooks.ts @@ -0,0 +1,79 @@ +import { useMemo } from 'react'; +import { product } from 'product'; + +import { useAppSelector } from 'hooks'; +import { useGetProjectsImportsQuery } from 'services/import'; +import { useGetProjectsQuery } from 'services/project'; +import { useGetModelsQuery } from 'services/run'; + +import { selectUserName } from 'App/slice'; + +import { IModelExtended, TModelsFilters } from './List/types'; + +/** + * Loads the models of runs: of every active run and, where `needsLatestRuns` says so, of the latest runs too. + * The latest runs have the models that no active run has, the stopped ones. + * `needsLatestRuns` gets the models of the active runs once they are loaded. + */ +export const useRunModels = (filters: TModelsFilters, needsLatestRuns: (activeModels?: IModelExtended[]) => boolean) => { + const activeQuery = useGetModelsQuery({ ...filters, only_active: true }); + const activeModels = activeQuery.isFetching ? undefined : activeQuery.currentData; + const latestQuery = useGetModelsQuery(filters, { skip: !needsLatestRuns(activeModels) }); + + const models = useMemo( + () => [...(activeQuery.currentData ?? []), ...(latestQuery.currentData ?? [])], + [activeQuery.currentData, latestQuery.currentData], + ); + + const refetch = () => { + activeQuery.refetch(); + + // A skipped query can't be refetched + if (!latestQuery.isUninitialized) { + latestQuery.refetch(); + } + }; + + return { + models, + isFetching: activeQuery.isFetching || latestQuery.isFetching, + error: activeQuery.error ?? latestQuery.error, + refetch, + } as const; +}; + +/** + * Loads the import lists of the user's projects, or of the project of the filter alone. + * An imported model has no user, so a filter by user leaves the imported models out. + * Exports exist only where the server enables them. Elsewhere there are no imports, and nothing is requested. + */ +export const useProjectsImports = (filters: TModelsFilters = {}) => { + const userName = useAppSelector(selectUserName); + const skip = !product.hasExports || !!filters.username; + + const projectsQuery = useGetProjectsQuery( + { username: userName ?? '' }, + { skip: skip || !!filters.project_name || !userName }, + ); + + const projects = projectsQuery.currentData?.data; + const projectNames = filters.project_name ? [filters.project_name] : projects?.map(({ project_name }) => project_name); + + const importsQuery = useGetProjectsImportsQuery({ project_names: projectNames ?? [] }, { skip: skip || !projectNames }); + + const refetch = () => { + // A skipped query can't be refetched + if (!importsQuery.isUninitialized) { + importsQuery.refetch(); + } + }; + + return { + projects: projects ?? [], + projectsImports: importsQuery.currentData ?? [], + // The projects list is asked by the user's name, so it waits for the name + isFetching: !skip && (projectsQuery.isFetching || importsQuery.isFetching || (!filters.project_name && !userName)), + error: projectsQuery.error, + refetch, + } as const; +}; diff --git a/frontend/src/pages/Runs/Details/RunDetails/index.tsx b/frontend/src/pages/Runs/Details/RunDetails/index.tsx index aff1fced7f..d76c27703e 100644 --- a/frontend/src/pages/Runs/Details/RunDetails/index.tsx +++ b/frontend/src/pages/Runs/Details/RunDetails/index.tsx @@ -11,6 +11,7 @@ import { getRunError, getRunPriority, getRunProbeStatuses, + getRunStatus, getRunStatusMessage, getStatusIconColor, getStatusIconType, @@ -18,7 +19,6 @@ import { import { ROUTES } from 'routes'; import { useGetRunQuery } from 'services/run'; -import { finishedRunStatuses } from 'pages/Runs/constants'; import { runIsStopped } from 'pages/Runs/utils'; import { @@ -67,13 +67,7 @@ export const RunDetails = () => { if (!runData) return null; - const status = finishedRunStatuses.includes(runData.status) - ? (runData.latest_job_submission?.status ?? runData.status) - : runData.status; - - const terminationReason = finishedRunStatuses.includes(runData.status) - ? runData.latest_job_submission?.termination_reason - : null; + const { status, terminationReason } = getRunStatus(runData); const finishedAt = getRunListFinishedAt(runData); diff --git a/frontend/src/pages/Runs/List/hooks/useColumnsDefinitions.tsx b/frontend/src/pages/Runs/List/hooks/useColumnsDefinitions.tsx index 285c29ad9f..9bacfd8eb1 100644 --- a/frontend/src/pages/Runs/List/hooks/useColumnsDefinitions.tsx +++ b/frontend/src/pages/Runs/List/hooks/useColumnsDefinitions.tsx @@ -9,14 +9,13 @@ import { getRepoNameFromRun, getRunError, getRunPriority, + getRunStatus, getRunStatusMessage, getStatusIconColor, getStatusIconType, } from 'libs/run'; import { ROUTES } from 'routes'; -import { finishedRunStatuses } from 'pages/Runs/constants'; - import { getRunListFinishedAt, getRunListItemBackend, @@ -78,12 +77,7 @@ export const useColumnsDefinitions = () => { id: 'status', header: t('projects.run.status'), cell: (item: IRun) => { - const status = finishedRunStatuses.includes(item.status) - ? (item.latest_job_submission?.status ?? item.status) - : item.status; - const terminationReason = finishedRunStatuses.includes(item.status) - ? item.latest_job_submission?.termination_reason - : null; + const { status, terminationReason } = getRunStatus(item); const statusMessage = getRunStatusMessage(item); return ( diff --git a/frontend/src/pages/Runs/constants.ts b/frontend/src/pages/Runs/constants.ts index c38a440c83..9cacc8993a 100644 --- a/frontend/src/pages/Runs/constants.ts +++ b/frontend/src/pages/Runs/constants.ts @@ -2,7 +2,6 @@ export const runStatusForDeleting: TJobStatus[] = ['failed', 'aborted', 'done', export const inActiveRunStatuses: TJobStatus[] = ['failed', 'aborted', 'done', 'terminated']; export const runStatusForStopping: TJobStatus[] = ['submitted', 'provisioning', 'pulling', 'pending', 'running']; export const runStatusForAborting: TJobStatus[] = ['submitted', 'provisioning', 'pulling', 'pending', 'running']; -export const unfinishedRuns: TJobStatus[] = ['running', 'terminating', 'pending']; export const finishedJobs: TJobStatus[] = ['terminated', 'aborted', 'failed', 'done']; // TODO: Replace TJobStatus with TRunStatus and remove all consts above export const finishedRunStatuses: TJobStatus[] = ['done', 'failed', 'terminated']; diff --git a/frontend/src/router.tsx b/frontend/src/router.tsx index 428f80dbd7..ecf25e1797 100644 --- a/frontend/src/router.tsx +++ b/frontend/src/router.tsx @@ -24,7 +24,7 @@ import { EventsList as InstanceEventsList } from 'pages/Instances/Details/Events import { InstanceInspect } from 'pages/Instances/Details/Inspect'; import { InstanceDetails } from 'pages/Instances/Details/InstanceDetails'; import { ModelsList } from 'pages/Models'; -import { ModelDetails } from 'pages/Models/Details'; +import { ModelDetailsPage } from 'pages/Models/Details'; import { CreateProjectWizard, ProjectAdd, @@ -242,7 +242,7 @@ export const router = createBrowserRouter([ }, { path: ROUTES.MODELS.DETAILS.TEMPLATE, - element: , + element: , }, // Events, Enterprise and above diff --git a/frontend/src/routes.ts b/frontend/src/routes.ts index b07ef62589..a56516ee4c 100644 --- a/frontend/src/routes.ts +++ b/frontend/src/routes.ts @@ -139,10 +139,28 @@ export const ROUTES = { MODELS: { LIST: '/models', + // A model's name can have slashes, so it is the rest of the address, and the page's tab is a query parameter DETAILS: { - TEMPLATE: `/projects/:projectName/models/:runName`, - FORMAT: (projectName: string, runName: string) => - buildRoute(ROUTES.MODELS.DETAILS.TEMPLATE, { projectName, runName }), + TEMPLATE: `/projects/:projectName/models/*`, + FORMAT: (projectName: string, modelName: string) => { + const segments = modelName.split('/'); + // An empty part between slashes wouldn't survive in a path + const name = segments.every(Boolean) + ? segments.map(encodeURIComponent).join('/') + : encodeURIComponent(modelName); + + return buildRoute(ROUTES.MODELS.DETAILS.TEMPLATE, { projectName }).replace('*', () => name); + }, + EXPORTS: { + TAB: 'exports', + FORMAT: (projectName: string, modelName: string) => + `${ROUTES.MODELS.DETAILS.FORMAT(projectName, modelName)}?tab=${ROUTES.MODELS.DETAILS.EXPORTS.TAB}`, + }, + METERING: { + TAB: 'metering', + FORMAT: (projectName: string, modelName: string) => + `${ROUTES.MODELS.DETAILS.FORMAT(projectName, modelName)}?tab=${ROUTES.MODELS.DETAILS.METERING.TAB}`, + }, }, }, diff --git a/frontend/src/services/export.ts b/frontend/src/services/export.ts new file mode 100644 index 0000000000..386b56c505 --- /dev/null +++ b/frontend/src/services/export.ts @@ -0,0 +1,25 @@ +import { API } from 'api'; +import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; + +import fetchBaseQueryHeaders from 'libs/fetchBaseQueryHeaders'; + +export const exportApi = createApi({ + reducerPath: 'exportApi', + refetchOnMountOrArgChange: true, + baseQuery: fetchBaseQuery({ + prepareHeaders: fetchBaseQueryHeaders, + }), + + endpoints: (builder) => ({ + getProjectExports: builder.query({ + query: ({ project_name }) => ({ + url: API.PROJECTS.EXPORTS_LIST(project_name), + method: 'POST', + }), + + transformResponse: (response: { exports: IExport[] }) => response.exports, + }), + }), +}); + +export const { useGetProjectExportsQuery } = exportApi; diff --git a/frontend/src/services/import.ts b/frontend/src/services/import.ts new file mode 100644 index 0000000000..93eb8a4eac --- /dev/null +++ b/frontend/src/services/import.ts @@ -0,0 +1,40 @@ +import { API } from 'api'; +import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; + +import fetchBaseQueryHeaders from 'libs/fetchBaseQueryHeaders'; + +export const importApi = createApi({ + reducerPath: 'importApi', + refetchOnMountOrArgChange: true, + baseQuery: fetchBaseQuery({ + prepareHeaders: fetchBaseQueryHeaders, + }), + + endpoints: (builder) => ({ + // One import list request for each of the projects. A list that failed is left out, so that the others show. + // A failure is expected for a public project that the user owns without being its member + getProjectsImports: builder.query({ + queryFn: async ({ project_names }, api, extraOptions, baseQuery) => { + const results = await Promise.all( + project_names.map((projectName) => + baseQuery({ + url: API.PROJECTS.IMPORTS_LIST(projectName), + method: 'POST', + }), + ), + ); + + return { + data: results.flatMap(({ data }, index) => + data ? [{ project_name: project_names[index], imports: (data as { imports: IImport[] }).imports }] : [], + ), + }; + }, + + // A page must not decide "Not Found" on an answer cached by an earlier visit + keepUnusedDataFor: 0, + }), + }), +}); + +export const { useGetProjectsImportsQuery } = importApi; diff --git a/frontend/src/services/metering.ts b/frontend/src/services/metering.ts new file mode 100644 index 0000000000..ec0920dc91 --- /dev/null +++ b/frontend/src/services/metering.ts @@ -0,0 +1,28 @@ +import { API } from 'api'; +import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; + +import fetchBaseQueryHeaders from 'libs/fetchBaseQueryHeaders'; + +export const meteringApi = createApi({ + reducerPath: 'meteringApi', + refetchOnMountOrArgChange: true, + baseQuery: fetchBaseQuery({ + prepareHeaders: fetchBaseQueryHeaders, + }), + + endpoints: (builder) => ({ + getMetering: builder.query({ + query: (body) => { + return { + url: API.METERING.LIST(), + method: 'POST', + body, + }; + }, + + transformResponse: (response: { rows: IMeteringRow[] }) => response.rows, + }), + }), +}); + +export const { useGetMeteringQuery, useLazyGetMeteringQuery } = meteringApi; diff --git a/frontend/src/services/run.test.ts b/frontend/src/services/run.test.ts new file mode 100644 index 0000000000..41091c23cf --- /dev/null +++ b/frontend/src/services/run.test.ts @@ -0,0 +1,102 @@ +/** @jest-environment node */ +import { configureStore } from '@reduxjs/toolkit'; + +import { runApi } from './run'; + +jest.mock('api', () => ({ + API: { + RUNS: { LIST: () => 'http://localhost/api/runs/list' }, + }, +})); + +const createStore = () => + configureStore({ + reducer: { + app: (state = { authData: { token: 'token' } }) => state, + [runApi.reducerPath]: runApi.reducer, + }, + middleware: (getDefaultMiddleware) => getDefaultMiddleware().concat(runApi.middleware), + }); + +// The latest run first. The runs 99 and 100, the last of the first page and the first of the second one, are submitted at the same time +const makeRuns = (count: number, modelRuns: number[]) => + Array.from({ length: count }, (_, index) => { + const submittedAt = Date.UTC(2026, 11, 3, 9) - (index === 100 ? 99 : index) * 60000; + + return { + id: `run-${String(index).padStart(4, '0')}`, + project_name: 'exporter-proj', + user: 'alice', + submitted_at: new Date(submittedAt).toISOString(), + status: 'running', + run_spec: { run_name: `run-${index}` }, + service: modelRuns.includes(index) + ? { model: { name: `model-${index}`, type: 'chat', base_url: 'https://gateway.example.com/exporter-proj' } } + : null, + } as IRun; + }); + +describe('Models of runs', () => { + let store: ReturnType; + let fetchMock: jest.SpyInstance; + let runs: IRun[]; + let requests: TRunsRequestParams[]; + + // A page of the runs list, as the server pages it: after the run of the cursor, + // which for runs submitted at the same time is the run with the next id + const listRuns = ({ limit, prev_submitted_at, prev_run_id }: TRunsRequestParams) => + runs + .filter( + ({ id, submitted_at }) => + !prev_submitted_at || + submitted_at < prev_submitted_at || + (submitted_at === prev_submitted_at && !!prev_run_id && id > prev_run_id), + ) + .slice(0, limit); + + const getModelRuns = async (params: Pick) => { + const models = await store.dispatch(runApi.endpoints.getModels.initiate(params)).unwrap(); + + return models.map(({ run }) => run?.id); + }; + + beforeEach(() => { + store = createStore(); + requests = []; + fetchMock = jest.spyOn(globalThis, 'fetch').mockImplementation(async (input) => { + const params = await (input as Request).json(); + + requests.push(params); + + return new Response(JSON.stringify(listRuns(params)), { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }); + }); + }); + + afterEach(() => { + store.dispatch(runApi.util.resetApiState()); + fetchMock.mockRestore(); + }); + + test('reads the runs page by page, each page after the last run of the page before, until a page is not full', async () => { + runs = makeRuns(250, [100, 249]); + + const page = { only_active: true, limit: 100, job_submissions_limit: 1 }; + + expect(await getModelRuns({ only_active: true })).toEqual([runs[100].id, runs[249].id]); + expect(requests).toEqual([ + page, + { ...page, prev_submitted_at: runs[99].submitted_at, prev_run_id: runs[99].id }, + { ...page, prev_submitted_at: runs[199].submitted_at, prev_run_id: runs[199].id }, + ]); + }); + + test('reads every active run, and the latest 1,000 runs when not only the active ones are asked for', async () => { + runs = makeRuns(1100, [999, 1000]); + + expect(await getModelRuns({ only_active: true })).toEqual([runs[999].id, runs[1000].id]); + expect(await getModelRuns({})).toEqual([runs[999].id]); + }); +}); diff --git a/frontend/src/services/run.ts b/frontend/src/services/run.ts index 600f0e5e62..302a107501 100644 --- a/frontend/src/services/run.ts +++ b/frontend/src/services/run.ts @@ -1,13 +1,17 @@ -import { sortBy as _sortBy } from 'lodash'; import { API } from 'api'; import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; import fetchBaseQueryHeaders from 'libs/fetchBaseQueryHeaders'; import { getExtendedModelFromRun } from '../libs/run'; -import { unfinishedRuns } from '../pages/Runs/constants'; -import { IModelExtended } from '../pages/Models/List/types'; +import { IModelExtended, TModelsRequestParams } from '../pages/Models/List/types'; + +// The largest page of the runs list +const RUNS_PAGE_LIMIT = 100; + +// How many of the latest runs are read for the models that no active run has +const MODELS_LATEST_RUNS_LIMIT = 1000; const reduceInvalidateTagsFromRunNames = (names: Array) => { return names.reduce((accumulator, runName: string) => { @@ -128,32 +132,44 @@ export const runApi = createApi({ }, }), - getModels: builder.query({ - query: (body = {}) => { - return { - url: API.RUNS.LIST(), - method: 'POST', - body, - }; + getModels: builder.query({ + // The runs list can't tell the runs that have a model, so the runs are read page by page: + // every active run, or the latest runs + queryFn: async (params, api, extraOptions, baseQuery) => { + const runs: IRun[] = []; + let lastRun: IRun | undefined; + + do { + const { data, error } = await baseQuery({ + url: API.RUNS.LIST(), + method: 'POST', + body: { + ...params, + limit: RUNS_PAGE_LIMIT, + job_submissions_limit: 1, + prev_submitted_at: lastRun?.submitted_at, + prev_run_id: lastRun?.id, + }, + }); + + if (error) { + return { error }; + } + + const page = data as IRun[]; + + runs.push(...page); + lastRun = page.length === RUNS_PAGE_LIMIT ? page[page.length - 1] : undefined; + } while (lastRun && (params.only_active || runs.length < MODELS_LATEST_RUNS_LIMIT)); + + return { data: runs.flatMap((run) => getExtendedModelFromRun(run) ?? []) }; }, - transformResponse: (runs: IRun[]): IModelExtended[] => { - return ( - _sortBy(runs, [(i) => -i.submitted_at]) - // Should show models of active runs only - .filter((run) => unfinishedRuns.includes(run.status) && run.service?.model) - .reduce((acc, run) => { - const model = getExtendedModelFromRun(run); - - if (model) acc.push(model); - - return acc; - }, []) - ); - }, + // A page must not decide "Not Found" on an answer cached by an earlier visit + keepUnusedDataFor: 0, providesTags: (result) => - result ? [...result.map(({ id }) => ({ type: 'Models' as const, id: id })), 'Models'] : ['Models'], + result ? [...result.map(({ run }) => ({ type: 'Models' as const, id: run?.id })), 'Models'] : ['Models'], }), getMetrics: builder.query({ @@ -184,6 +200,6 @@ export const { useApplyRunMutation, useStopRunsMutation, useDeleteRunsMutation, - useLazyGetModelsQuery, + useGetModelsQuery, useGetMetricsQuery, } = runApi; diff --git a/frontend/src/store.ts b/frontend/src/store.ts index 3bb8153e9c..35e527263d 100644 --- a/frontend/src/store.ts +++ b/frontend/src/store.ts @@ -6,10 +6,13 @@ import notificationsReducer from 'components/Notifications/slice'; import { artifactApi } from 'services/artifact'; import { authApi } from 'services/auth'; import { eventApi } from 'services/events'; +import { exportApi } from 'services/export'; import { fleetApi } from 'services/fleet'; import { gatewayApi } from 'services/gateway'; +import { importApi } from 'services/import'; import { instanceApi } from 'services/instance'; import { mainApi } from 'services/mainApi'; +import { meteringApi } from 'services/metering'; import { presetApi } from 'services/preset'; import { projectApi } from 'services/project'; import { publicKeysApi } from 'services/publicKeys'; @@ -48,6 +51,9 @@ export const store = configureStore({ [publicKeysApi.reducerPath]: publicKeysApi.reducer, [eventApi.reducerPath]: eventApi.reducer, [templateApi.reducerPath]: templateApi.reducer, + [importApi.reducerPath]: importApi.reducer, + [exportApi.reducerPath]: exportApi.reducer, + [meteringApi.reducerPath]: meteringApi.reducer, }, middleware: (getDefaultMiddleware) => @@ -71,6 +77,9 @@ export const store = configureStore({ .concat(eventApi.middleware) .concat(repoApi.middleware) .concat(templateApi.middleware) + .concat(importApi.middleware) + .concat(exportApi.middleware) + .concat(meteringApi.middleware) .concat(mainApi.middleware), }); diff --git a/frontend/src/types/export.d.ts b/frontend/src/types/export.d.ts new file mode 100644 index 0000000000..88df391097 --- /dev/null +++ b/frontend/src/types/export.d.ts @@ -0,0 +1,13 @@ +declare interface IExportImport { + project_name: IProject['project_name']; +} + +declare interface IExportedModel { + name: string; +} + +declare interface IExport { + name: string; + imports: IExportImport[]; + exported_models: IExportedModel[]; +} diff --git a/frontend/src/types/import.d.ts b/frontend/src/types/import.d.ts new file mode 100644 index 0000000000..3d975dda9e --- /dev/null +++ b/frontend/src/types/import.d.ts @@ -0,0 +1,21 @@ +declare interface IImportExportedModel { + name: string; + // The importer project's URL for the model. Empty while no service serves the model + base_url: string | null; +} + +declare interface IImportExport { + name: string; + // The project that exports + project_name: IProject['project_name']; + exported_models: IImportExportedModel[]; +} + +declare interface IImport { + export: IImportExport; +} + +declare type TProjectImports = { + project_name: IProject['project_name']; + imports: IImport[]; +}; diff --git a/frontend/src/types/metering.d.ts b/frontend/src/types/metering.d.ts new file mode 100644 index 0000000000..72d3c60fda --- /dev/null +++ b/frontend/src/types/metering.d.ts @@ -0,0 +1,22 @@ +declare type TMeteringGroupBy = 'day' | 'month' | 'project' | 'user'; + +declare type TMeteringListRequestParams = { + start: string; + end: string; + group_by: TMeteringGroupBy[]; + model_project_name?: IProject['project_name']; + model_name?: IModel['name']; + project_name?: IProject['project_name']; + user_name?: IUser['username']; +}; + +declare interface IMeteringRow { + day?: string | null; + // The first day of the month + month?: string | null; + project_name?: string | null; + user_name?: string | null; + input_tokens: number; + cached_input_tokens: number; + output_tokens: number; +} diff --git a/frontend/webpack/dev.js b/frontend/webpack/dev.js index a286e77f4c..4c9b248571 100644 --- a/frontend/webpack/dev.js +++ b/frontend/webpack/dev.js @@ -16,7 +16,9 @@ module.exports = { port, open: true, hot: true, // enable HMR on the server - historyApiFallback: true, // fixes error 404-ish errors when using react router :see this SO question: https://stackoverflow.com/questions/43209666/react-router-v4-cannot-get-url + // fixes error 404-ish errors when using react router :see this SO question: https://stackoverflow.com/questions/43209666/react-router-v4-cannot-get-url + // A model's page has the model's name in its path, and model names have dots + historyApiFallback: { disableDotRule: true }, headers: { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, PATCH, OPTIONS', diff --git a/mkdocs/docs/concepts/exports.md b/mkdocs/docs/concepts/exports.md index 0f078a4d48..66b5cd125b 100644 --- a/mkdocs/docs/concepts/exports.md +++ b/mkdocs/docs/concepts/exports.md @@ -14,14 +14,17 @@ An export is created in the exporter project and specifies the resources to expo importer projects that will gain access to them. Once an export is created, the importer projects can see the exported resources in their resource lists and use them -for running tasks, dev environments, and services. Imported resources appear with a project prefix +for running tasks, dev environments, and services. Imported fleets and gateways appear with a project prefix (e.g., `team-a/my-fleet`) to distinguish them from the project's own resources. -Currently, [fleets](fleets.md) and [gateways](gateways.md) can be exported. +Currently, [fleets](fleets.md), [gateways](gateways.md), and [models](services.md#model) can be exported. +Exported models are not used in runs: members of the importer projects access them at their project's [model endpoint](#imported-models). ??? info "Limitations" - [Container-based](backends.md#container-based) backends of the exporter project cannot be used for fleet importers' runs. - Runs that mount [network volumes](volumes.md#network-volumes) cannot use imported backend fleets. + - Models cannot be added to [global exports](#global-exports), and an export with models cannot be made global. + - If an imported model has the same name as a model that the importer project runs on the same gateway, the project's own model is used. If two imported models on the same gateway have the same name, the model from the older export is used. ??? info "Required role" The user creating or updating an export must have the project admin role on both the exporter project and @@ -32,27 +35,28 @@ Currently, [fleets](fleets.md) and [gateways](gateways.md) can be exported. ### Create exports Use the `dstack export create` command to create a new export. Specify the fleets to export -with `--fleet`, the gateways to export with `--gateway`, and the importer projects with `--importer`: +with `--fleet`, the gateways to export with `--gateway`, the [models](#export-models) to export with `--model`, +and the importer projects with `--importer`:
```shell $ dstack export create my-export --fleet my-fleet --gateway my-gateway --importer team-b - NAME FLEETS GATEWAYS IMPORTERS - my-export my-fleet my-gateway team-b + NAME FLEETS GATEWAYS MODELS IMPORTERS + my-export my-fleet my-gateway - team-b ```
-`--fleet`, `--gateway`, and `--importer` can be specified multiple times: +`--fleet`, `--gateway`, `--model`, and `--importer` can be specified multiple times:
```shell $ dstack export create shared-gpus --fleet gpu-fleet-1 --fleet gpu-fleet-2 --importer team-b --importer team-c - NAME FLEETS GATEWAYS IMPORTERS - shared-gpus gpu-fleet-1, gpu-fleet-2 - team-b, team-c + NAME FLEETS GATEWAYS MODELS IMPORTERS + shared-gpus gpu-fleet-1, gpu-fleet-2 - - team-b, team-c ``` @@ -66,9 +70,9 @@ Use `dstack export list` (or simply `dstack export`) to list all exports in the ```shell $ dstack export list - NAME FLEETS GATEWAYS IMPORTERS - my-export my-fleet my-gateway team-b - shared-gpus gpu-fleet-1, gpu-fleet-2 - team-b, team-c + NAME FLEETS GATEWAYS MODELS IMPORTERS + my-export my-fleet my-gateway - team-b + shared-gpus gpu-fleet-1, gpu-fleet-2 - - team-b, team-c ``` @@ -76,27 +80,27 @@ $ dstack export list ### Update exports -Use the `dstack export update` command to add or remove fleets, gateways, and importers from an existing export: +Use the `dstack export update` command to add or remove fleets, gateways, models, and importers from an existing export:
```shell $ dstack export update my-export --add-fleet another-fleet --add-importer team-c - NAME FLEETS GATEWAYS IMPORTERS - my-export my-fleet, another-fleet my-gateway team-b, team-c + NAME FLEETS GATEWAYS MODELS IMPORTERS + my-export my-fleet, another-fleet my-gateway - team-b, team-c ```
-To remove a fleet, gateway, or importer: +To remove a fleet, gateway, model, or importer:
```shell $ dstack export update my-export --remove-importer team-b - NAME FLEETS GATEWAYS IMPORTERS - my-export my-fleet, another-fleet my-gateway team-c + NAME FLEETS GATEWAYS MODELS IMPORTERS + my-export my-fleet, another-fleet my-gateway - team-c ``` @@ -126,8 +130,8 @@ Users with the global admin role can mark any export as a global export. Global ```shell $ dstack export create global-export --gateway shared-gateway --global - NAME FLEETS GATEWAYS IMPORTERS - global-export - shared-gateway * + NAME FLEETS GATEWAYS MODELS IMPORTERS + global-export - shared-gateway - * ``` @@ -135,6 +139,32 @@ $ dstack export create global-export --gateway shared-gateway --global Only promoting an export to global requires the global admin role. Regular project admins can add or remove resources, remove global status, or delete the export. +### Export models + +To export a [model](services.md#model), pass its name with `--model`. This is the name set in the `model` property of the service that runs the model: + +
+ +```shell +$ dstack export create my-models --model Qwen/Qwen3.8-27B-FP8 --importer team-b + NAME FLEETS GATEWAYS MODELS IMPORTERS + my-models - - Qwen/Qwen3.8-27B-FP8 team-b + +``` + +
+ +Use `--add-model` and `--remove-model` with `dstack export update` to change the models of an existing export. + +An export holds a model by its name, not by the service that runs it. You can export a model before any service declares it, and the model stays exported when its service is stopped and applied again, or replaced by another service with the same `model`. + +Importer projects can access an exported model only while it runs on a service of the exporter project that: + +1. Uses the `openai` model format (the default). +2. Runs on a [gateway](gateways.md) whose `domain` doesn't include `${{ run.project_name }}`. + +If running services already declare the model and none of them meets both requirements, `dstack` refuses to export it. `dstack` also refuses to move the service in place to run without a gateway or on a gateway with `${{ run.project_name }}` in its domain if that leaves the exported model without such a service. While a running service on a gateway serves an exported model to importer projects, `dstack` refuses to remove the gateway's domain or to add `${{ run.project_name }}` to it. Stopping the service is always allowed. + ## Access imported resources From the importer project's perspective, use `dstack import list` (or simply `dstack import`) to list all imports in the project — i.e., all exports from other projects that this project has been granted access to: @@ -143,8 +173,9 @@ From the importer project's perspective, use `dstack import list` (or simply `ds ```shell $ dstack import list - NAME FLEETS GATEWAYS - team-a/my-export my-fleet, another-fleet my-gateway + NAME FLEETS GATEWAYS MODELS + team-a/my-export my-fleet, another-fleet my-gateway - + team-a/my-models - - Qwen/Qwen3.8-27B-FP8 ``` @@ -187,12 +218,56 @@ fleets:
+### Imported models + +Members of the importer project access an imported model at their project's model endpoint on the gateway where the model runs. The gateway itself doesn't have to be imported. Use `dstack import list -v` to see the URL of each imported model: + +
+ +```shell +$ dstack import list -v + NAME FLEETS GATEWAYS MODEL URL + team-a/my-export my-fleet, another-fleet my-gateway - + team-a/my-models - - Qwen/Qwen3.8-27B-FP8 https://gateway.example.com/team-b + +``` + +
+ +In the UI, imported models show on the `Models` page, next to the project's own models. On a model's page, the `Endpoint` button shows how to call the model at your project's URL, and the `Exports` tab lists the model's exports and the projects that import it. + +The URL is an OpenAI-compatible endpoint with `/models` and `/chat/completions`. Use a `dstack` user token of a member of the importer project as the API key: + +
+ +```shell +$ curl https://gateway.example.com/team-b/chat/completions \ + -H 'Content-Type: application/json' \ + -H 'Authorization: Bearer <user token>' \ + -d '{ + "model": "Qwen/Qwen3.8-27B-FP8", + "messages": [ + { + "role": "user", + "content": "Compose a poem that explains the concept of recursion in programming." + } + ] + }' +``` + +
+ +A model has no URL while no service of the exporter project can serve it to importer projects, e.g. the service is stopped or not started yet. Requests to such a model get a 404 error. Once a service starts, the model becomes accessible within about 20 seconds. + !!! info "Tenant isolation" Exported fleets share the same access model as regular fleets. See [Tenant isolation](../guides/tenant-isolation.md) for details. + Requests to an imported model, including the user token, are sent to the gateway that the exporter project runs the model on. + !!! info "What's next?" 1. Check the [`dstack export` CLI reference](../reference/cli/dstack/export.md) 1. Check the [`dstack import` CLI reference](../reference/cli/dstack/import.md) 1. Learn how to manage [fleets](fleets.md) 1. Learn how to manage [gateways](gateways.md) + 1. Learn how to run models as [services](services.md#model) 1. Read about [projects](projects.md) and project roles diff --git a/mkdocs/docs/concepts/gateways.md b/mkdocs/docs/concepts/gateways.md index c84c19af76..04de84dd2e 100644 --- a/mkdocs/docs/concepts/gateways.md +++ b/mkdocs/docs/concepts/gateways.md @@ -57,6 +57,8 @@ A gateway requires a `domain` to be specified in the configuration before creati Once the gateway is created and assigned a hostname, configure your DNS by adding a wildcard record for `*.` (e.g. `*.example.com`). The record should point to the gateway's hostname and should be of type `A` if the hostname is an IP address (most cases), or of type `CNAME` if the hostname is another domain (load balancers, Kubernetes). +The [models](services.md#model) of a project's services are served at the project's model endpoint on the gateway, `https://gateway./`. Because of that, a service on the gateway can't be named `gateway`. + ??? info "Project name interpolation" You can use the `${{ run.project_name }}` variable to include the service’s project name in the domain name. This is especially useful when [exporting](exports.md) the gateway to multiple projects, as it ensures each importer receives a unique domain name. @@ -68,6 +70,8 @@ Once the gateway is created and assigned a hostname, configure your DNS by addin domain: ${{ run.project_name }}.mycompany.example ``` + With this domain, a project's services are at `..mycompany.example`, and its model endpoint is `https://gateway..mycompany.example/`. + ### Backend You can create gateways with the `aws`, `azure`, `gcp`, or `kubernetes` backends, but that does not limit where services run. A gateway can use one backend while services run on any other backend supported by dstack, including backends where gateways themselves cannot be created. diff --git a/mkdocs/docs/concepts/services.md b/mkdocs/docs/concepts/services.md index a9923e36b0..77756de5fd 100644 --- a/mkdocs/docs/concepts/services.md +++ b/mkdocs/docs/concepts/services.md @@ -754,6 +754,34 @@ See the [reference](../reference/dstack.yml/service.md#probes) for more probe co If the service runs a model with an OpenAI-compatible interface, you can set the [`model`](#model) property to make the model accessible through `dstack`'s chat UI on the `Models` page. In this case, `dstack` will use the service's `/v1/chat/completions` service. +If the service runs with a [gateway](gateways.md), its model is also served at the project's model endpoint, `https://gateway./`. +It's one OpenAI-compatible endpoint with `/models` and `/chat/completions` for all the project's models on that gateway. +`dstack apply` shows it as the model's URL, next to the service's own URL. +Use a `dstack` user token as the API key: + +
+ +```shell +$ curl https://gateway.example.com/main/chat/completions \ + -H 'Content-Type: application/json' \ + -H 'Authorization: Bearer <user token>' \ + -d '{ + "model": "meta-llama/Meta-Llama-3.1-8B-Instruct", + "messages": [ + { + "role": "user", + "content": "Compose a poem that explains the concept of recursion in programming." + } + ] + }' +``` + +
+ +For streaming requests, the model endpoint sets `stream_options.include_usage` and `stream_options.continuous_usage_stats` to get token counts from the model, so the model's API has to accept these options. The client gets only the `usage` it asked for. + +The model endpoint accepts requests of up to 1MB. Larger requests, e.g. with images, get a 413 error there and have to be sent to the service's own URL. + When `model` is set, `dstack` automatically configures [`probes`](#probes) to verify model health. To customize or disable this, set `probes` explicitly. @@ -818,6 +846,8 @@ The rps limit sets the max requests per second, tracked in milliseconds (e.g., ` Limits apply to the whole service (all replicas) and per client (by IP). Clients exceeding the limit get a 429 error. +Requests that reach the service through the project's [model endpoint](#model) are not limited per client: they all count as one client, and a limit partitioned by a client header, such as `Authorization`, doesn't apply to them. + ??? info "Partitioning key" Instead of partitioning requests by client IP address, you can choose to partition by the value of a header. diff --git a/pyproject.toml b/pyproject.toml index 027baa4057..498fa5c18b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -220,7 +220,8 @@ gateway = [ "aiorwlock", "aiocache", "httpx>=0.28.0", - # Gateway authorization needs the pool cancellation and recovery fixes in httpx2 (#4338). + # Gateway authorization and requests to services need the pool cancellation and recovery + # fixes in httpx2 (#4338). "httpx2>=2.13.1", "jinja2", ] @@ -231,6 +232,8 @@ server = [ "aiorwlock", "aiocache", "httpx>=0.28.0", + # Requests to services need the pool cancellation and recovery fixes in httpx2 (#4338). + "httpx2>=2.13.1", "jinja2", "watchfiles", "sqlalchemy[asyncio]>=2.0.0", diff --git a/skills/dstack/SKILL.md b/skills/dstack/SKILL.md index 8b910e0cdb..f8225ab5f6 100644 --- a/skills/dstack/SKILL.md +++ b/skills/dstack/SKILL.md @@ -262,7 +262,7 @@ resources: - Without gateway: `/proxy/services///` - With gateway: `https://./` - Authentication: Unless `auth` is `false`, include `Authorization: Bearer ` on service requests. -- Model endpoint: If `model` is set, `service.model.base_url` from `dstack run get --json` provides the model endpoint. For OpenAI-compatible models (the default, unless format is set otherwise), this will be `service.url` + `/v1`. +- Model endpoint: If `model` is set, `service.model.base_url` from `dstack run get --json` provides the model endpoint. Without a gateway, for OpenAI-compatible models (the default, unless format is set otherwise), this will be `service.url` + `/v1`. With a gateway, it is the project's model endpoint, `https://gateway./`, which serves `/models` and `/chat/completions` and always requires a user token; the model's other routes are at `service.url` + `/v1`. - Example (with gateway): ```bash curl -sS -X POST "https://./v1/chat/completions" \ diff --git a/src/dstack/_internal/cli/commands/export.py b/src/dstack/_internal/cli/commands/export.py index 2b1e17b462..b5548b0e04 100644 --- a/src/dstack/_internal/cli/commands/export.py +++ b/src/dstack/_internal/cli/commands/export.py @@ -50,6 +50,13 @@ def _register(self): help="Gateway name to export (can be specified multiple times)", default=[], ) + create_parser.add_argument( + "--model", + action="append", + dest="models", + help="Model name to export (can be specified multiple times)", + default=[], + ) create_parser.add_argument( "--global", dest="is_global", @@ -108,6 +115,20 @@ def _register(self): help="Gateway name to remove (can be specified multiple times)", default=[], ) + update_parser.add_argument( + "--add-model", + action="append", + dest="add_models", + help="Model name to add (can be specified multiple times)", + default=[], + ) + update_parser.add_argument( + "--remove-model", + action="append", + dest="remove_models", + help="Model name to remove (can be specified multiple times)", + default=[], + ) global_group = update_parser.add_mutually_exclusive_group() global_group.add_argument( "--set-global", @@ -156,6 +177,7 @@ def _create(self, args: argparse.Namespace): importer_projects=args.importers, exported_fleets=args.fleets, exported_gateways=args.gateways, + exported_models=args.models, ) print_exports_table([export]) @@ -172,6 +194,8 @@ def _update(self, args: argparse.Namespace): remove_exported_fleets=args.remove_fleets, add_exported_gateways=args.add_gateways, remove_exported_gateways=args.remove_gateways, + add_exported_models=args.add_models, + remove_exported_models=args.remove_models, ) print_exports_table([export]) @@ -191,6 +215,7 @@ def print_exports_table(exports: list[Export]): table.add_column("NAME", no_wrap=True) table.add_column("FLEETS") table.add_column("GATEWAYS") + table.add_column("MODELS") table.add_column("IMPORTERS") for export in exports: @@ -202,6 +227,9 @@ def print_exports_table(exports: list[Export]): if export.exported_gateways else "-" ) + models = ( + ", ".join([m.name for m in export.exported_models]) if export.exported_models else "-" + ) if export.is_global: importers = "*" else: @@ -213,6 +241,7 @@ def print_exports_table(exports: list[Export]): "NAME": export.name, "FLEETS": fleets, "GATEWAYS": gateways, + "MODELS": models, "IMPORTERS": importers, } add_row_from_dict(table, row) diff --git a/src/dstack/_internal/cli/commands/import_.py b/src/dstack/_internal/cli/commands/import_.py index ddd528bcb5..046fe6f508 100644 --- a/src/dstack/_internal/cli/commands/import_.py +++ b/src/dstack/_internal/cli/commands/import_.py @@ -22,6 +22,11 @@ def _register(self): ) list_parser.set_defaults(subfunc=self._list) + for parser in [self._parser, list_parser]: + parser.add_argument( + "-v", "--verbose", action="store_true", help="Show more information" + ) + delete_parser = subparsers.add_parser( "delete", help="Delete an import", formatter_class=self._parser.formatter_class ) @@ -42,7 +47,7 @@ def _list(self, args: argparse.Namespace): response = self.api.client.imports.list_imports(self.api.project) for warning in response.warnings: warn(warning) - print_imports_table(response.imports) + print_imports_table(response.imports, verbose=args.verbose) def _delete(self, args: argparse.Namespace): parts = args.name.split("/") @@ -66,11 +71,16 @@ def _delete(self, args: argparse.Namespace): console.print(f"Import [code]{args.name}[/] deleted") -def print_imports_table(imports: list[Import]): +def print_imports_table(imports: list[Import], verbose: bool = False): table = Table(box=None) table.add_column("NAME", no_wrap=True) table.add_column("FLEETS") table.add_column("GATEWAYS") + if verbose: + table.add_column("MODEL") + table.add_column("URL") + else: + table.add_column("MODELS") for imp in imports: name = f"{imp.export.project_name}/{imp.export.name}" @@ -84,13 +94,22 @@ def print_imports_table(imports: list[Import]): if imp.export.exported_gateways else "-" ) + exported_models = imp.export.exported_models + models = ", ".join([m.name for m in exported_models]) if exported_models else "-" row = { "NAME": name, "FLEETS": fleets, "GATEWAYS": gateways, + "MODELS": models, + "MODEL": exported_models[0].name if exported_models else "-", + "URL": exported_models[0].base_url if exported_models else None, } add_row_from_dict(table, row) + if verbose: + # Each model has its own row with its URL + for model in exported_models[1:]: + add_row_from_dict(table, {"MODEL": model.name, "URL": model.base_url}) console.print(table) console.print() diff --git a/src/dstack/_internal/core/compatibility/exports.py b/src/dstack/_internal/core/compatibility/exports.py index 945eb2d9d9..cba7c41899 100644 --- a/src/dstack/_internal/core/compatibility/exports.py +++ b/src/dstack/_internal/core/compatibility/exports.py @@ -4,9 +4,15 @@ def get_create_export_excludes(request: CreateExportRequest) -> IncludeExcludeStrDictType: excludes: IncludeExcludeStrDictType = {} + if not request.exported_models: + excludes["exported_models"] = True return excludes def get_update_export_excludes(request: UpdateExportRequest) -> IncludeExcludeStrDictType: excludes: IncludeExcludeStrDictType = {} + if not request.add_exported_models: + excludes["add_exported_models"] = True + if not request.remove_exported_models: + excludes["remove_exported_models"] = True return excludes diff --git a/src/dstack/_internal/core/models/exports.py b/src/dstack/_internal/core/models/exports.py index f888801da7..b24dcb2e82 100644 --- a/src/dstack/_internal/core/models/exports.py +++ b/src/dstack/_internal/core/models/exports.py @@ -17,6 +17,10 @@ class ExportedGateway(CoreModel): name: str +class ExportedModel(CoreModel): + name: str + + class Export(CoreModel): id: uuid.UUID name: str @@ -24,6 +28,7 @@ class Export(CoreModel): imports: list[ExportImport] exported_fleets: list[ExportedFleet] exported_gateways: list[ExportedGateway] = [] + exported_models: list[ExportedModel] = [] class ListExportsResponse(CoreModel): diff --git a/src/dstack/_internal/core/models/imports.py b/src/dstack/_internal/core/models/imports.py index d4e528346c..956a41d1b6 100644 --- a/src/dstack/_internal/core/models/imports.py +++ b/src/dstack/_internal/core/models/imports.py @@ -1,4 +1,5 @@ import uuid +from typing import Optional from dstack._internal.core.models.common import CoreModel @@ -13,12 +14,21 @@ class ImportExportedGateway(CoreModel): name: str +class ImportExportedModel(CoreModel): + name: str + base_url: Optional[str] = None + """`base_url` is the importer project's model endpoint on the gateway that serves the model. + `None` while no service of the exporter project serves the model to importer projects. + """ + + class ImportExport(CoreModel): id: uuid.UUID name: str project_name: str exported_fleets: list[ImportExportedFleet] exported_gateways: list[ImportExportedGateway] = [] + exported_models: list[ImportExportedModel] = [] class Import(CoreModel): diff --git a/src/dstack/_internal/core/models/runs.py b/src/dstack/_internal/core/models/runs.py index a63ead8c68..8f11c60e23 100644 --- a/src/dstack/_internal/core/models/runs.py +++ b/src/dstack/_internal/core/models/runs.py @@ -13,6 +13,7 @@ NetworkMode, RegistryAuth, drop_merged_profile, + validate_extra_ignore, ) from dstack._internal.core.models.configurations import ( DEFAULT_PROBE_METHOD, @@ -42,6 +43,7 @@ ) from dstack._internal.core.models.repos import AnyRunRepoData from dstack._internal.core.models.resources import Memory, ResourcesSpec +from dstack._internal.core.models.services import AnyModel from dstack._internal.core.models.unix import UnixUser from dstack._internal.core.models.volumes import MountPoint from dstack._internal.utils import common as common_utils @@ -650,6 +652,16 @@ class ServiceSpec(CoreModel): def get_domain(self) -> Optional[str]: return urlparse(self.url).hostname + def get_model_options(self) -> Optional[AnyModel]: + """ + Returns the `model` of the service configuration, as the server saved it in `options`. + `None` if the service declares no model. + """ + model_options = self.options.get("openai", {}).get("model") + if model_options is None: + return None + return validate_extra_ignore(AnyModel, model_options) + class RunStatus(str, Enum): PENDING = "pending" diff --git a/src/dstack/_internal/proxy/gateway/app.py b/src/dstack/_internal/proxy/gateway/app.py index 6627e9e3f3..297c3eb93a 100644 --- a/src/dstack/_internal/proxy/gateway/app.py +++ b/src/dstack/_internal/proxy/gateway/app.py @@ -1,5 +1,6 @@ """FastAPI app running on a gateway.""" +import asyncio from contextlib import asynccontextmanager from pathlib import Path from typing import Optional @@ -21,18 +22,24 @@ from dstack._internal.proxy.gateway.repo.state_v1 import migrate_from_state_v1 from dstack._internal.proxy.gateway.routers.auth import router as auth_router from dstack._internal.proxy.gateway.routers.config import router as config_router +from dstack._internal.proxy.gateway.routers.metering import router as metering_router from dstack._internal.proxy.gateway.routers.registry import router as registry_router from dstack._internal.proxy.gateway.routers.services import router as services_router from dstack._internal.proxy.gateway.routers.stats import router as stats_router +from dstack._internal.proxy.gateway.services.metering import FLUSH_INTERVAL, GatewayProxyMeter from dstack._internal.proxy.gateway.services.nginx import Nginx from dstack._internal.proxy.gateway.services.registry import ACCESS_LOG_PATH, apply_all from dstack._internal.proxy.gateway.services.server_client import HTTPMultiClient from dstack._internal.proxy.gateway.services.stats import StatsCollector from dstack._internal.proxy.lib.routers.model_proxy import router as model_proxy_router from dstack._internal.utils.common import run_async +from dstack._internal.utils.logging import get_logger from dstack.version import __version__ +logger = get_logger(__name__) + STATE_FILE = DSTACK_DIR_ON_GATEWAY / "state-v2.json" +METERING_DIR = DSTACK_DIR_ON_GATEWAY / "metering" LEGACY_STATE_FILE = DSTACK_DIR_ON_GATEWAY / "state.json" LEGACY_KEYS_DIR = Path("~/.ssh/projects").expanduser().resolve() @@ -45,13 +52,37 @@ async def lifespan(app: FastAPI): service_conn_pool = await injector.get_service_connection_pool() await run_async(nginx.write_global_conf) await apply_all(repo, nginx, service_conn_pool) + stop_metering = asyncio.Event() + metering_task = asyncio.create_task(_flush_meter(injector.get_meter(), stop_metering)) yield + # By now the requests are over or cancelled, so the last flush has all their tokens. + stop_metering.set() + await metering_task await service_conn_pool.remove_all() -def make_app(repo: Optional[GatewayProxyRepo] = None, nginx: Optional[Nginx] = None) -> FastAPI: +async def _flush_meter(meter: GatewayProxyMeter, stop: asyncio.Event) -> None: + """Flushes the meter every `FLUSH_INTERVAL` and once more after `stop` is set.""" + stopping = False + while not stopping: + try: + await asyncio.wait_for(stop.wait(), timeout=FLUSH_INTERVAL) + stopping = True + except asyncio.TimeoutError: + pass + try: + await meter.flush() + except Exception: + logger.exception("Failed to write a metering batch") + + +def make_app( + repo: Optional[GatewayProxyRepo] = None, + nginx: Optional[Nginx] = None, + meter: Optional[GatewayProxyMeter] = None, +) -> FastAPI: if repo is None: migrate_from_state_v1( v1_file=LEGACY_STATE_FILE, v2_file=STATE_FILE, keys_dir=LEGACY_KEYS_DIR @@ -66,6 +97,7 @@ def make_app(repo: Optional[GatewayProxyRepo] = None, nginx: Optional[Nginx] = N ), nginx=nginx or Nginx(), stats_collector=StatsCollector(ACCESS_LOG_PATH), + meter=meter or GatewayProxyMeter(METERING_DIR), ) # TODO: add CORS only to openai routers once fastapi supports it. @@ -79,6 +111,7 @@ def make_app(repo: Optional[GatewayProxyRepo] = None, nginx: Optional[Nginx] = N app.include_router(auth_router, prefix="/api/auth") app.include_router(config_router, prefix="/api/config") + app.include_router(metering_router, prefix="/api/metering") app.include_router(model_proxy_router, prefix="/api/models") app.include_router(registry_router, prefix="/api/registry") app.include_router(services_router, prefix="/api/services") diff --git a/src/dstack/_internal/proxy/gateway/auth.py b/src/dstack/_internal/proxy/gateway/auth.py index f71f9845d1..d98c67791c 100644 --- a/src/dstack/_internal/proxy/gateway/auth.py +++ b/src/dstack/_internal/proxy/gateway/auth.py @@ -1,29 +1,130 @@ +import uuid +from dataclasses import dataclass +from typing import Optional +from urllib.parse import quote + import httpx2 as httpx from aiocache import cached +from fastapi import status +from pydantic import BaseModel -from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider -from dstack._internal.proxy.lib.errors import UnexpectedProxyError +from dstack._internal.core.models.users import GlobalRole +from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider, ProxyCaller +from dstack._internal.proxy.lib.errors import ProxyError, UnexpectedProxyError _AUTH_CACHE_TTL_SECONDS = 60 -"""Seconds to cache allowed or denied access for each project/token pair.""" +"""Seconds to cache allowed or denied access for each project/token pair, +and the answer that the project does not exist.""" +_USER_ID_CACHE_TTL_SECONDS = 3600 +"""Seconds to cache the user ID of each token. Access is still rechecked +every `_AUTH_CACHE_TTL_SECONDS`, so a revoked token stops working within that time.""" + + +@dataclass(frozen=True) +class _ProjectAccess: + """The server's answer on whether a token can access a project.""" + + project_id: Optional[uuid.UUID] + """`None` if access is denied or the project does not exist.""" + member_ids: frozenset[uuid.UUID] = frozenset() + """The project's members. The server also allows global admins, + and any user if the project is public.""" + project_exists: bool = True + """`False` if the server has no project with this name.""" class GatewayProxyAuthProvider(BaseProxyAuthProvider): def __init__(self, server_client: httpx.AsyncClient) -> None: self._server_client = server_client - @cached(ttl=_AUTH_CACHE_TTL_SECONDS, noself=True, skip_cache_func=lambda r: r is None) async def is_project_member(self, project_name: str, token: str) -> bool: + access = await self._get_project_access(project_name, token) + return access.project_id is not None + + async def get_caller(self, project_name: str, token: str) -> Optional[ProxyCaller]: + access = await self._get_project_access(project_name, token) + if not access.project_exists: + raise ProxyError(f"Project {project_name} not found", status.HTTP_404_NOT_FOUND) + if access.project_id is None: + return None + user_id = await self._get_user_id(token) + if user_id is None: + return None + if user_id not in access.member_ids and not await self._is_global_admin(token): + return None + return ProxyCaller(project_id=access.project_id, user_id=user_id) + + @cached(ttl=_AUTH_CACHE_TTL_SECONDS, noself=True) + async def _get_project_access(self, project_name: str, token: str) -> _ProjectAccess: + # The project name comes from the URL of the user's request, so it is quoted. + resp = await self._request( + f"/api/projects/{quote(project_name, safe='')}/get", token, allow_not_found=True + ) + if resp is None: + return _ProjectAccess(project_id=None) + if resp.status_code == httpx.codes.NOT_FOUND: + return _ProjectAccess(project_id=None, project_exists=False) + project = _Project.model_validate_json(resp.content) + return _ProjectAccess( + project_id=project.project_id, + member_ids=frozenset(member.user.id for member in project.members), + ) + + @cached(ttl=_USER_ID_CACHE_TTL_SECONDS, noself=True, skip_cache_func=lambda r: r is None) + async def _get_user_id(self, token: str) -> Optional[uuid.UUID]: + user = await self._get_user(token) + return user.id if user is not None else None + + @cached(ttl=_AUTH_CACHE_TTL_SECONDS, noself=True) + async def _is_global_admin(self, token: str) -> bool: + user = await self._get_user(token) + return user is not None and user.global_role == GlobalRole.ADMIN + + async def _get_user(self, token: str) -> Optional["_User"]: + resp = await self._request("/api/users/get_my_user", token) + if resp is None: + return None + return _User.model_validate_json(resp.content) + + async def _request( + self, path: str, token: str, allow_not_found: bool = False + ) -> Optional[httpx.Response]: + """ + Returns `None` if the server denies access with the token. + With `allow_not_found`, returns the server's 404 response instead of raising. + """ try: resp = await self._server_client.post( - f"/api/projects/{project_name}/get", - headers={"Authorization": f"Bearer {token}"}, + path, headers={"Authorization": f"Bearer {token}"} ) if resp.status_code == httpx.codes.FORBIDDEN: - return False - resp.raise_for_status() + return None + if not (allow_not_found and resp.status_code == httpx.codes.NOT_FOUND): + resp.raise_for_status() except httpx.HTTPError as e: raise UnexpectedProxyError( f"Failed requesting dstack-server to check access: {e!r}" ) from e - return True + return resp + + +# Only the fields that access checks need are parsed from the server's responses, +# so that changes to the other fields cannot break gateways that are not updated yet. + + +class _MemberUser(BaseModel): + id: uuid.UUID + + +class _Member(BaseModel): + user: _MemberUser + + +class _Project(BaseModel): + project_id: uuid.UUID + members: list[_Member] + + +class _User(BaseModel): + id: uuid.UUID + global_role: str diff --git a/src/dstack/_internal/proxy/gateway/deps.py b/src/dstack/_internal/proxy/gateway/deps.py index 009ce8da15..301b59bf07 100644 --- a/src/dstack/_internal/proxy/gateway/deps.py +++ b/src/dstack/_internal/proxy/gateway/deps.py @@ -3,6 +3,7 @@ from fastapi import Depends, FastAPI, Request from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo +from dstack._internal.proxy.gateway.services.metering import GatewayProxyMeter from dstack._internal.proxy.gateway.services.nginx import Nginx from dstack._internal.proxy.gateway.services.stats import StatsCollector from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider @@ -22,12 +23,14 @@ def __init__( auth: BaseProxyAuthProvider, nginx: Nginx, stats_collector: StatsCollector, + meter: GatewayProxyMeter, ) -> None: super().__init__() self._repo = repo self._auth = auth self._nginx = nginx self._stats_collector = stats_collector + self._meter = meter async def get_repo(self) -> AsyncGenerator[BaseProxyRepo, None]: yield self._repo @@ -41,6 +44,9 @@ def get_nginx(self) -> Nginx: def get_stats_collector(self) -> StatsCollector: return self._stats_collector + def get_meter(self) -> GatewayProxyMeter: + return self._meter + def get_gateway_injector_from_app(app: FastAPI) -> GatewayDependencyInjector: injector = get_injector_from_app(app) @@ -71,3 +77,9 @@ async def get_stats_collector( injector: Annotated[GatewayDependencyInjector, Depends(get_gateway_injector)], ) -> StatsCollector: return injector.get_stats_collector() + + +async def get_gateway_proxy_meter( + injector: Annotated[GatewayDependencyInjector, Depends(get_gateway_injector)], +) -> GatewayProxyMeter: + return injector.get_meter() diff --git a/src/dstack/_internal/proxy/gateway/models.py b/src/dstack/_internal/proxy/gateway/models.py index 80ce0b8abe..f2fc52b455 100644 --- a/src/dstack/_internal/proxy/gateway/models.py +++ b/src/dstack/_internal/proxy/gateway/models.py @@ -15,6 +15,18 @@ class ModelEntrypoint(ImmutableModel): read_timeout: int = DEFAULT_SERVICE_READ_TIMEOUT +class ImportedModel(ImmutableModel): + """A model that the importer project accesses through an export of the exporter project.""" + + importer_project: str + export_id: str + """Metered with each request to the model, like `Service.id`.""" + exporter_project: str + name: str + """The model's name in the exporter project. With `exporter_project`, + finds the model and its service.""" + + class ACMESettings(ImmutableModel): server: Optional[AnyHttpUrl] = None eab_kid: Optional[str] = None diff --git a/src/dstack/_internal/proxy/gateway/repo/repo.py b/src/dstack/_internal/proxy/gateway/repo/repo.py index be0c7cbb45..dee5f00a8f 100644 --- a/src/dstack/_internal/proxy/gateway/repo/repo.py +++ b/src/dstack/_internal/proxy/gateway/repo/repo.py @@ -7,8 +7,12 @@ from aiorwlock import RWLock from pydantic import BaseModel -from dstack._internal.proxy.gateway.models import GlobalProxyConfig, ModelEntrypoint -from dstack._internal.proxy.lib.models import ChatModel, Project, Service +from dstack._internal.proxy.gateway.models import ( + GlobalProxyConfig, + ImportedModel, + ModelEntrypoint, +) +from dstack._internal.proxy.lib.models import ChatModel, OpenAIChatModelFormat, Project, Service from dstack._internal.proxy.lib.repo import BaseProxyRepo from dstack._internal.utils.common import run_async @@ -16,6 +20,8 @@ class State(BaseModel): services: dict[str, dict[str, Service]] = {} models: dict[str, dict[str, ChatModel]] = {} + imports: dict[str, dict[str, ImportedModel]] = {} + """Importer project -> model name -> import.""" entrypoints: dict[str, ModelEntrypoint] = {} projects: dict[str, Project] = {} config: GlobalProxyConfig = GlobalProxyConfig() @@ -75,13 +81,21 @@ async def delete_service(self, project_name: str, run_name: str) -> None: if not project_services: self._state.services.pop(project_name, None) + # Models are registered under lowercase project names (see `routers/registry.py`), + # while the model proxy takes the project name from the request URL as the user typed it. + async def list_models(self, project_name: str) -> list[ChatModel]: async with self.reader(): - return list(self._state.models.get(project_name, {}).values()) + project_name = project_name.lower() + names = list(self._state.models.get(project_name, {})) + names += list(self._state.imports.get(project_name, {})) + # `dict.fromkeys` leaves each name once + models = [self._get_model(project_name, name) for name in dict.fromkeys(names)] + return [model for model in models if model is not None] async def get_model(self, project_name: str, name: str) -> Optional[ChatModel]: async with self.reader(): - return self._state.models.get(project_name, {}).get(name) + return self._get_model(project_name.lower(), name) async def set_model(self, model: ChatModel) -> None: async with self.writer(): @@ -96,6 +110,30 @@ async def delete_models_by_run(self, project_name: str, run_name: str) -> None: if not project_models: self._state.models.pop(project_name, None) + async def list_imports(self) -> list[ImportedModel]: + async with self.reader(): + imports_by_project = ( + project_imports.values() for project_imports in self._state.imports.values() + ) + return list(chain(*imports_by_project)) + + async def set_import(self, imported_model: ImportedModel) -> None: + """Replaces the project's import of the model, if it has one through another export.""" + async with self.writer(): + self._state.imports.setdefault(imported_model.importer_project, {})[ + imported_model.name + ] = imported_model + + async def delete_import(self, importer_project: str, export_id: str, name: str) -> None: + """Deletes the project's import of the model, unless it is through another export.""" + async with self.writer(): + project_imports = self._state.imports.get(importer_project, {}) + imported_model = project_imports.get(name) + if imported_model is not None and imported_model.export_id == export_id: + project_imports.pop(name) + if not project_imports: + self._state.imports.pop(importer_project, None) + async def list_entrypoints(self) -> list[ModelEntrypoint]: async with self.reader(): return list(self._state.entrypoints.values()) @@ -153,3 +191,22 @@ def save(self, state: Optional[State] = None) -> None: temporary_path.replace(self._file) finally: temporary_path.unlink(missing_ok=True) + + def _get_model(self, project_name: str, name: str) -> Optional[ChatModel]: + """ + Returns the project's own model with this name. If the project has none, + returns the model that it imports under this name, with the export set. + + The import is skipped while the exporter project has no model with this name + and the `openai` format, since only requests to models with this format are metered. + """ + model = self._state.models.get(project_name, {}).get(name) + if model is not None: + return model + imported_model = self._state.imports.get(project_name, {}).get(name) + if imported_model is None: + return None + model = self._state.models.get(imported_model.exporter_project, {}).get(name) + if model is None or not isinstance(model.format_spec, OpenAIChatModelFormat): + return None + return model.model_copy(update={"export_id": imported_model.export_id}) diff --git a/src/dstack/_internal/proxy/gateway/resources/nginx/entrypoint.jinja2 b/src/dstack/_internal/proxy/gateway/resources/nginx/entrypoint.jinja2 index ce0160f558..be353b0a3a 100644 --- a/src/dstack/_internal/proxy/gateway/resources/nginx/entrypoint.jinja2 +++ b/src/dstack/_internal/proxy/gateway/resources/nginx/entrypoint.jinja2 @@ -1,7 +1,7 @@ server { server_name {{ domain }}; location / { - proxy_pass http://localhost:{{ proxy_port }}/api/models/{{ project_name }}/; + proxy_pass http://localhost:{{ proxy_port }}/api/models/; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Host $host; proxy_read_timeout {{ read_timeout }}s; diff --git a/src/dstack/_internal/proxy/gateway/resources/systemd/dstack.gateway.service b/src/dstack/_internal/proxy/gateway/resources/systemd/dstack.gateway.service index a8027f451a..fe4bb2b6e3 100644 --- a/src/dstack/_internal/proxy/gateway/resources/systemd/dstack.gateway.service +++ b/src/dstack/_internal/proxy/gateway/resources/systemd/dstack.gateway.service @@ -9,6 +9,8 @@ User=ubuntu Group=ubuntu Restart=always RestartSec=5 +# Above uvicorn's graceful shutdown timeout in start.sh, so that the app can shut down after it +TimeoutStopSec=30 LimitNOFILE=65535 [Install] diff --git a/src/dstack/_internal/proxy/gateway/resources/systemd/start.sh b/src/dstack/_internal/proxy/gateway/resources/systemd/start.sh index bcd39d2159..fcd57fca81 100644 --- a/src/dstack/_internal/proxy/gateway/resources/systemd/start.sh +++ b/src/dstack/_internal/proxy/gateway/resources/systemd/start.sh @@ -8,4 +8,5 @@ else version="blue" echo "$version" > "$root/version" fi -"$root/$version/bin/uvicorn" dstack._internal.proxy.gateway.main:app --loop uvloop +# Without the timeout, uvicorn waits for all open streams to end before it shuts the app down. +"$root/$version/bin/uvicorn" dstack._internal.proxy.gateway.main:app --loop uvloop --timeout-graceful-shutdown 10 diff --git a/src/dstack/_internal/proxy/gateway/routers/metering.py b/src/dstack/_internal/proxy/gateway/routers/metering.py new file mode 100644 index 0000000000..23fb8633ad --- /dev/null +++ b/src/dstack/_internal/proxy/gateway/routers/metering.py @@ -0,0 +1,20 @@ +from typing import Annotated + +from fastapi import APIRouter, Depends + +from dstack._internal.proxy.gateway.deps import get_gateway_proxy_meter +from dstack._internal.proxy.gateway.schemas.metering import ( + CollectMeteringRequest, + CollectMeteringResponse, +) +from dstack._internal.proxy.gateway.services.metering import GatewayProxyMeter + +router = APIRouter() + + +@router.post("/collect") +async def collect_metering( + body: CollectMeteringRequest, + meter: Annotated[GatewayProxyMeter, Depends(get_gateway_proxy_meter)], +) -> CollectMeteringResponse: + return CollectMeteringResponse(batches=await meter.collect(after=body.after, limit=body.limit)) diff --git a/src/dstack/_internal/proxy/gateway/routers/registry.py b/src/dstack/_internal/proxy/gateway/routers/registry.py index 3bdc252c86..255a3f01e3 100644 --- a/src/dstack/_internal/proxy/gateway/routers/registry.py +++ b/src/dstack/_internal/proxy/gateway/routers/registry.py @@ -7,10 +7,14 @@ from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo from dstack._internal.proxy.gateway.schemas.common import OkResponse from dstack._internal.proxy.gateway.schemas.registry import ( + ImportListItem, + ImportListResponse, RegisterEntrypointRequest, + RegisterImportRequest, RegisterReplicaRequest, RegisterServiceRequest, SetServiceIdRequest, + UnregisterImportRequest, UpdateReplicaProxyJumpKeysRequest, UpdateReplicaProxyJumpKeysResponse, ) @@ -160,3 +164,51 @@ async def register_entrypoint( nginx=nginx, ) return OkResponse() + + +@router.get("/imports/list") +async def list_imports( + repo: Annotated[GatewayProxyRepo, Depends(get_gateway_proxy_repo)], +) -> ImportListResponse: + return ImportListResponse( + imports=[ + ImportListItem( + importer_project=imported_model.importer_project, + export_id=imported_model.export_id, + exporter_project=imported_model.exporter_project, + name=imported_model.name, + ) + for imported_model in await repo.list_imports() + ] + ) + + +@router.post("/{importer_project}/imports/register") +async def register_import( + importer_project: str, + body: RegisterImportRequest, + repo: Annotated[GatewayProxyRepo, Depends(get_gateway_proxy_repo)], +) -> OkResponse: + await registry_services.register_import( + importer_project=importer_project.lower(), + export_id=body.export_id, + exporter_project=body.exporter_project.lower(), + name=body.name, + repo=repo, + ) + return OkResponse() + + +@router.post("/{importer_project}/imports/unregister") +async def unregister_import( + importer_project: str, + body: UnregisterImportRequest, + repo: Annotated[GatewayProxyRepo, Depends(get_gateway_proxy_repo)], +) -> OkResponse: + await registry_services.unregister_import( + importer_project=importer_project.lower(), + export_id=body.export_id, + name=body.name, + repo=repo, + ) + return OkResponse() diff --git a/src/dstack/_internal/proxy/gateway/schemas/metering.py b/src/dstack/_internal/proxy/gateway/schemas/metering.py new file mode 100644 index 0000000000..52ea7b525b --- /dev/null +++ b/src/dstack/_internal/proxy/gateway/schemas/metering.py @@ -0,0 +1,35 @@ +import uuid +from datetime import datetime +from typing import Optional + +from pydantic import BaseModel + + +class MeteringRecord(BaseModel): + hour: datetime + """UTC, start of the hour in which the tokens were counted.""" + project_id: uuid.UUID + user_id: uuid.UUID + run_id: uuid.UUID + export_id: Optional[uuid.UUID] = None + """The export that the requests came through. `None` for a project's own models.""" + input_tokens: int + """Includes `cached_input_tokens`.""" + cached_input_tokens: int + output_tokens: int + + +class MeteringBatch(BaseModel): + batch_no: int + created_at: datetime + records: list[MeteringRecord] + + +class CollectMeteringRequest(BaseModel): + after: int + """The number of the last batch that the server has saved.""" + limit: int + + +class CollectMeteringResponse(BaseModel): + batches: list[MeteringBatch] diff --git a/src/dstack/_internal/proxy/gateway/schemas/registry.py b/src/dstack/_internal/proxy/gateway/schemas/registry.py index 56c60e3add..c77f4740d3 100644 --- a/src/dstack/_internal/proxy/gateway/schemas/registry.py +++ b/src/dstack/_internal/proxy/gateway/schemas/registry.py @@ -74,6 +74,30 @@ class RegisterEntrypointRequest(BaseModel): read_timeout: int = DEFAULT_SERVICE_READ_TIMEOUT +class RegisterImportRequest(BaseModel): + export_id: str + exporter_project: str + name: str + """The model's name in the exporter project.""" + + +class UnregisterImportRequest(BaseModel): + export_id: str + name: str + + +class ImportListItem(BaseModel): + importer_project: str + export_id: str + exporter_project: str + name: str + """The model's name in the exporter project.""" + + +class ImportListResponse(BaseModel): + imports: list[ImportListItem] + + class ReplicaProxyJumpKeyUpdate(BaseModel): project_name: str run_name: str diff --git a/src/dstack/_internal/proxy/gateway/services/metering.py b/src/dstack/_internal/proxy/gateway/services/metering.py new file mode 100644 index 0000000000..8256f21954 --- /dev/null +++ b/src/dstack/_internal/proxy/gateway/services/metering.py @@ -0,0 +1,201 @@ +import asyncio +import tempfile +import time +import uuid +from dataclasses import dataclass +from datetime import datetime, timedelta +from pathlib import Path +from typing import Optional + +from pydantic import ValidationError + +from dstack._internal.proxy.gateway.schemas.metering import MeteringBatch, MeteringRecord +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.metering import BaseProxyMeter +from dstack._internal.proxy.lib.models import Service +from dstack._internal.utils.common import get_current_datetime, run_async +from dstack._internal.utils.logging import get_logger + +logger = get_logger(__name__) + +FLUSH_INTERVAL = 5 +"""Seconds between writing the counters to batch files.""" +BATCH_TTL = timedelta(days=7) +"""Batch files that the server has not collected are deleted after this time.""" +OLD_BATCHES_CLEANUP_INTERVAL = 3600 +LAST_BATCH_NO_FILE_NAME = "last_batch_no" + + +@dataclass(frozen=True) +class MeteringRecordKey: + hour: datetime + """UTC, start of the hour in which the tokens were counted.""" + project_id: uuid.UUID + user_id: uuid.UUID + run_id: uuid.UUID + export_id: Optional[uuid.UUID] + """The export that the requests came through. `None` for a project's own models.""" + + +@dataclass +class MeteringRecordValue: + input_tokens: int = 0 + """Includes `cached_input_tokens`.""" + cached_input_tokens: int = 0 + output_tokens: int = 0 + + +class GatewayProxyMeter(BaseProxyMeter): + """ + Counts tokens in memory. `flush` moves the counters to a new batch file in `dir`. + The server collects the batches in the order of their numbers + and confirms the ones it has saved by asking for later ones. + """ + + def __init__(self, dir: Path) -> None: + self._dir = dir + self._counters: dict[MeteringRecordKey, MeteringRecordValue] = {} + # Batch numbers are never reused, even after a restart with all batch files collected, + # since the server only asks for batches with numbers above the ones it has saved. + self._last_batch_no = max([self._read_last_batch_no(), *self._list_batch_nos()]) + self._files_lock = asyncio.Lock() + self._next_old_batches_cleanup_at = time.monotonic() + self._services_without_id: set[str] = set() + + def add( + self, + caller: ProxyCaller, + service: Service, + export_id: Optional[str], + input_tokens: int, + cached_input_tokens: int, + output_tokens: int, + ) -> None: + if service.id is None: + if service.fmt() not in self._services_without_id: + self._services_without_id.add(service.fmt()) + logger.warning("Service %s has no ID, cannot meter its tokens", service.fmt()) + return + key = MeteringRecordKey( + hour=get_current_datetime().replace(minute=0, second=0, microsecond=0), + project_id=caller.project_id, + user_id=caller.user_id, + run_id=uuid.UUID(service.id), + export_id=uuid.UUID(export_id) if export_id is not None else None, + ) + self._add(key, MeteringRecordValue(input_tokens, cached_input_tokens, output_tokens)) + + async def flush(self) -> None: + """ + Writes the counters, if any, to a new batch file and resets them. + Once in `OLD_BATCHES_CLEANUP_INTERVAL`, also deletes the batch files older than `BATCH_TTL`. + """ + if time.monotonic() >= self._next_old_batches_cleanup_at: + self._next_old_batches_cleanup_at = time.monotonic() + OLD_BATCHES_CLEANUP_INTERVAL + async with self._files_lock: + await run_async(self._delete_old_batches) + if not self._counters: + return + counters, self._counters = self._counters, {} + try: + async with self._files_lock: + await run_async(self._write_batch, counters) + except Exception: + # Keep the counts for the next flush. + for key, value in counters.items(): + self._add(key, value) + raise + + async def collect(self, after: int, limit: int) -> list[MeteringBatch]: + """ + Returns up to `limit` batches with numbers above `after`, oldest first, + and deletes the batches up to `after`, which the server has saved. + """ + async with self._files_lock: + return await run_async(self._collect, after, limit) + + def _add(self, key: MeteringRecordKey, value: MeteringRecordValue) -> None: + counter = self._counters.setdefault(key, MeteringRecordValue()) + counter.input_tokens += value.input_tokens + counter.cached_input_tokens += value.cached_input_tokens + counter.output_tokens += value.output_tokens + + def _write_batch(self, counters: dict[MeteringRecordKey, MeteringRecordValue]) -> None: + batch = MeteringBatch( + batch_no=self._last_batch_no + 1, + created_at=get_current_datetime(), + records=[ + MeteringRecord( + hour=key.hour, + project_id=key.project_id, + user_id=key.user_id, + run_id=key.run_id, + export_id=key.export_id, + input_tokens=value.input_tokens, + cached_input_tokens=value.cached_input_tokens, + output_tokens=value.output_tokens, + ) + for key, value in counters.items() + ], + ) + self._write_last_batch_no(batch.batch_no) + self._write_file(self._get_batch_path(batch.batch_no), batch.model_dump_json()) + + def _collect(self, after: int, limit: int) -> list[MeteringBatch]: + if after > self._last_batch_no: + # The server has saved batches that are no longer known here: the files were lost, + # e.g. in a container restart. Number new batches above them, + # or the server would never ask for them. + self._write_last_batch_no(after) + batches = [] + for batch_no in self._list_batch_nos(): + path = self._get_batch_path(batch_no) + if batch_no <= after: + path.unlink(missing_ok=True) + continue + if len(batches) == limit: + break + try: + batches.append(MeteringBatch.model_validate_json(path.read_text())) + except ValidationError: + logger.exception("Skipping invalid metering batch %s", path) + return batches + + def _delete_old_batches(self) -> None: + min_modified_at = (get_current_datetime() - BATCH_TTL).timestamp() + for batch_no in self._list_batch_nos(): + path = self._get_batch_path(batch_no) + if path.stat().st_mtime >= min_modified_at: + # Batches are written in the order of their numbers. + break + path.unlink() + + def _read_last_batch_no(self) -> int: + try: + return int((self._dir / LAST_BATCH_NO_FILE_NAME).read_text()) + except (FileNotFoundError, ValueError): + return 0 + + def _write_last_batch_no(self, batch_no: int) -> None: + self._write_file(self._dir / LAST_BATCH_NO_FILE_NAME, str(batch_no)) + self._last_batch_no = batch_no + + def _list_batch_nos(self) -> list[int]: + if not self._dir.exists(): + return [] + return sorted(int(path.stem) for path in self._dir.glob("*.json") if path.stem.isdigit()) + + def _get_batch_path(self, batch_no: int) -> Path: + return self._dir / f"{batch_no}.json" + + def _write_file(self, path: Path, content: str) -> None: + """Writes the file atomically.""" + self._dir.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile(mode="w", dir=self._dir, delete=False) as f: + temporary_path = Path(f.name) + try: + f.write(content) + f.close() + temporary_path.replace(path) + finally: + temporary_path.unlink(missing_ok=True) diff --git a/src/dstack/_internal/proxy/gateway/services/nginx.py b/src/dstack/_internal/proxy/gateway/services/nginx.py index d55f04ada9..df17e690b0 100644 --- a/src/dstack/_internal/proxy/gateway/services/nginx.py +++ b/src/dstack/_internal/proxy/gateway/services/nginx.py @@ -73,7 +73,6 @@ class ServiceConfig(SiteConfig): class ModelEntrypointConfig(SiteConfig): type: Literal["entrypoint"] = "entrypoint" - project_name: str read_timeout: int diff --git a/src/dstack/_internal/proxy/gateway/services/registry.py b/src/dstack/_internal/proxy/gateway/services/registry.py index c4711cf6ef..f48d9050f8 100644 --- a/src/dstack/_internal/proxy/gateway/services/registry.py +++ b/src/dstack/_internal/proxy/gateway/services/registry.py @@ -131,6 +131,8 @@ async def unregister_service( ) await repo.delete_models_by_run(project_name, run_name) await repo.delete_service(project_name, run_name) + # After the service is deleted, so that no request gets a new client. + await service_conn_pool.remove_nginx_client(service.domain_safe) logger.info("Service %s is unregistered now", service.fmt()) @@ -339,6 +341,43 @@ async def register_model_entrypoint( logger.info("Entrypoint %s is now registered in project %s", domain, project_name) +async def register_import( + importer_project: str, + export_id: str, + exporter_project: str, + name: str, + repo: GatewayProxyRepo, +) -> None: + imported_model = gateway_models.ImportedModel( + importer_project=importer_project, + export_id=export_id, + exporter_project=exporter_project, + name=name, + ) + await repo.set_import(imported_model) + logger.info( + "Import of model %s from project %s is now registered in project %s", + name, + exporter_project, + importer_project, + ) + + +async def unregister_import( + importer_project: str, + export_id: str, + name: str, + repo: GatewayProxyRepo, +) -> None: + await repo.delete_import(importer_project, export_id, name) + logger.info( + "Import of model %s through export %s is now unregistered in project %s", + name, + export_id, + importer_project, + ) + + async def apply_service( service: models.Service, old_service: Optional[models.Service], @@ -501,7 +540,6 @@ async def apply_entrypoint( config = ModelEntrypointConfig( domain=entrypoint.domain, https=entrypoint.https, - project_name=entrypoint.project_name, read_timeout=entrypoint.read_timeout, ) acme = (await repo.get_config()).acme_settings @@ -519,8 +557,10 @@ async def _migrate_cors_enabled(repo: GatewayProxyRepo) -> None: openai_run_names: set[tuple[str, str]] = set() for service in services: for model in await repo.list_models(service.project_name): - if model.run_name == service.run_name and isinstance( - model.format_spec, models.OpenAIChatModelFormat + if ( + model.project_name == service.project_name # not a model the project imports + and model.run_name == service.run_name + and isinstance(model.format_spec, models.OpenAIChatModelFormat) ): openai_run_names.add((service.project_name, service.run_name)) for service in services: diff --git a/src/dstack/_internal/proxy/lib/auth.py b/src/dstack/_internal/proxy/lib/auth.py index 382df2c468..e559df0884 100644 --- a/src/dstack/_internal/proxy/lib/auth.py +++ b/src/dstack/_internal/proxy/lib/auth.py @@ -1,7 +1,26 @@ +import uuid from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import Optional + + +@dataclass(frozen=True) +class ProxyCaller: + """The user who makes a request and the project the request is made in.""" + + project_id: uuid.UUID + user_id: uuid.UUID class BaseProxyAuthProvider(ABC): @abstractmethod async def is_project_member(self, project_name: str, token: str) -> bool: pass + + @abstractmethod + async def get_caller(self, project_name: str, token: str) -> Optional[ProxyCaller]: + """ + Returns the caller if the token belongs to a member of the project or to a global admin, + and `None` otherwise. May raise a 404 `ProxyError` if the project does not exist. + """ + pass diff --git a/src/dstack/_internal/proxy/lib/deps.py b/src/dstack/_internal/proxy/lib/deps.py index c080ca3e4e..c144701f84 100644 --- a/src/dstack/_internal/proxy/lib/deps.py +++ b/src/dstack/_internal/proxy/lib/deps.py @@ -4,8 +4,9 @@ from fastapi import Depends, FastAPI, Request, Security, status from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer -from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider +from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider, ProxyCaller from dstack._internal.proxy.lib.errors import ProxyError, UnexpectedProxyError +from dstack._internal.proxy.lib.metering import BaseProxyMeter from dstack._internal.proxy.lib.repo import BaseProxyRepo from dstack._internal.proxy.lib.services.service_connection import ServiceConnectionPool @@ -32,6 +33,10 @@ def get_repo(self) -> AsyncGenerator[BaseProxyRepo, None]: def get_auth_provider(self) -> AsyncGenerator[BaseProxyAuthProvider, None]: pass + @abstractmethod + def get_meter(self) -> BaseProxyMeter: + pass + async def get_service_connection_pool(self) -> ServiceConnectionPool: return self._service_conn_pool @@ -61,6 +66,12 @@ async def get_proxy_auth_provider( yield provider +async def get_proxy_meter( + injector: Annotated[ProxyDependencyInjector, Depends(get_injector)], +) -> BaseProxyMeter: + return injector.get_meter() + + async def get_service_connection_pool( injector: Annotated[ProxyDependencyInjector, Depends(get_injector)], ) -> ServiceConnectionPool: @@ -77,10 +88,26 @@ async def enforce(self) -> None: if self._token is None or not await self._provider.is_project_member( self._project_name, self._token ): - raise ProxyError( - f"Unauthenticated or unauthorized to access project {self._project_name}", - status.HTTP_403_FORBIDDEN, - ) + raise self._forbidden_error() + + async def get_caller(self) -> ProxyCaller: + """ + Returns who makes the request. Raises the same error as `enforce` + unless the token belongs to a member of the project or to a global admin. + The auth provider may raise a 404 `ProxyError` instead if the project does not exist. + """ + if self._token is None: + raise self._forbidden_error() + caller = await self._provider.get_caller(self._project_name, self._token) + if caller is None: + raise self._forbidden_error() + return caller + + def _forbidden_error(self) -> ProxyError: + return ProxyError( + f"Unauthenticated or unauthorized to access project {self._project_name}", + status.HTTP_403_FORBIDDEN, + ) class ProxyAuth: @@ -103,3 +130,9 @@ async def __call__( if self._auto_enforce: await context.enforce() return context + + +async def get_proxy_caller( + auth: Annotated[ProxyAuthContext, Depends(ProxyAuth(auto_enforce=False))], +) -> ProxyCaller: + return await auth.get_caller() diff --git a/src/dstack/_internal/proxy/lib/metering.py b/src/dstack/_internal/proxy/lib/metering.py new file mode 100644 index 0000000000..8a1fd999ad --- /dev/null +++ b/src/dstack/_internal/proxy/lib/metering.py @@ -0,0 +1,27 @@ +from abc import ABC, abstractmethod +from typing import Optional + +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.models import Service + + +class BaseProxyMeter(ABC): + """Meters the tokens of requests to models.""" + + @abstractmethod + def add( + self, + caller: ProxyCaller, + service: Service, + export_id: Optional[str], + input_tokens: int, + cached_input_tokens: int, + output_tokens: int, + ) -> None: + """ + Adds tokens newly counted for the caller's request to the service. + `export_id` is the export that the request came through, + `None` for a request to the project's own model. + `input_tokens` includes `cached_input_tokens`. + """ + pass diff --git a/src/dstack/_internal/proxy/lib/models.py b/src/dstack/_internal/proxy/lib/models.py index a459ef5603..8d7fd551dc 100644 --- a/src/dstack/_internal/proxy/lib/models.py +++ b/src/dstack/_internal/proxy/lib/models.py @@ -139,3 +139,6 @@ class ChatModel(ImmutableModel): created_at: datetime run_name: str format_spec: Annotated[AnyModelFormat, Field(discriminator="format")] + export_id: Optional[str] = None + """Only set on gateways, for a model that a project imports: the export that the model + comes through. `project_name` and `run_name` are then the exporter project's.""" diff --git a/src/dstack/_internal/proxy/lib/routers/model_proxy.py b/src/dstack/_internal/proxy/lib/routers/model_proxy.py index f348957f4a..a28bab2579 100644 --- a/src/dstack/_internal/proxy/lib/routers/model_proxy.py +++ b/src/dstack/_internal/proxy/lib/routers/model_proxy.py @@ -1,25 +1,36 @@ from typing import Annotated, AsyncIterator, Optional -from fastapi import APIRouter, Depends, status +from fastapi import APIRouter, Depends, Request, status +from fastapi.exceptions import RequestValidationError from fastapi.responses import StreamingResponse +from pydantic import ValidationError -from dstack._internal.proxy.lib.deps import ProxyAuth, get_proxy_repo, get_service_connection_pool +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.deps import ( + get_proxy_caller, + get_proxy_meter, + get_proxy_repo, + get_service_connection_pool, +) from dstack._internal.proxy.lib.errors import ProxyError, UnexpectedProxyError +from dstack._internal.proxy.lib.metering import BaseProxyMeter +from dstack._internal.proxy.lib.models import OpenAIChatModelFormat from dstack._internal.proxy.lib.repo import BaseProxyRepo from dstack._internal.proxy.lib.schemas.model_proxy import ( ChatCompletionsChunk, + ChatCompletionsPassthroughRequest, ChatCompletionsRequest, - ChatCompletionsResponse, Model, ModelsResponse, ) from dstack._internal.proxy.lib.services.model_proxy.model_proxy import get_chat_client +from dstack._internal.proxy.lib.services.model_proxy.passthrough import forward_chat_completions from dstack._internal.proxy.lib.services.service_connection import ( ServiceConnectionPool, get_service_replica_client, ) -router = APIRouter(dependencies=[Depends(ProxyAuth(auto_enforce=True))]) +router = APIRouter(dependencies=[Depends(get_proxy_caller)]) @router.get("/{project_name}/models") @@ -34,31 +45,51 @@ async def get_models( return ModelsResponse(data=data) -@router.post("/{project_name}/chat/completions", response_model=ChatCompletionsResponse) +@router.post("/{project_name}/chat/completions") async def post_chat_completions( project_name: str, - body: ChatCompletionsRequest, + body: ChatCompletionsPassthroughRequest, + request: Request, + caller: Annotated[ProxyCaller, Depends(get_proxy_caller)], repo: Annotated[BaseProxyRepo, Depends(get_proxy_repo)], service_conn_pool: Annotated[ServiceConnectionPool, Depends(get_service_connection_pool)], + meter: Annotated[BaseProxyMeter, Depends(get_proxy_meter)], ): model = await repo.get_model(project_name, body.model) if model is None: raise ProxyError( f"Model {body.model} not found in project {project_name}", status.HTTP_404_NOT_FOUND ) - service = await repo.get_service(project_name, model.run_name) + service = await repo.get_service(model.project_name, model.run_name) if service is None or not service.replicas: raise UnexpectedProxyError( f"Model {model.name} in project {project_name} references run {model.run_name}" " that does not exist or has no replicas" ) + if isinstance(model.format_spec, OpenAIChatModelFormat): + return await forward_chat_completions( + request=request, + body=body, + prefix=model.format_spec.prefix, + service=service, + export_id=model.export_id, + caller=caller, + repo=repo, + service_conn_pool=service_conn_pool, + meter=meter, + ) + # Models with other formats are converted to and from typed schemas and are not metered. + try: + typed_body = ChatCompletionsRequest.model_validate(body.model_dump(exclude_unset=True)) + except ValidationError as e: + raise RequestValidationError(e.errors()) http_client = await get_service_replica_client(service, repo, service_conn_pool) client = get_chat_client(model, http_client) - if not body.stream: - return await client.generate(body) + if not typed_body.stream: + return await client.generate(typed_body) else: return StreamingResponse( - await StreamingAdaptor(client.stream(body)).get_stream(), + await StreamingAdaptor(client.stream(typed_body)).get_stream(), media_type="text/event-stream", headers={"X-Accel-Buffering": "no"}, ) diff --git a/src/dstack/_internal/proxy/lib/schemas/model_proxy.py b/src/dstack/_internal/proxy/lib/schemas/model_proxy.py index 632a490dc4..3822ff368b 100644 --- a/src/dstack/_internal/proxy/lib/schemas/model_proxy.py +++ b/src/dstack/_internal/proxy/lib/schemas/model_proxy.py @@ -1,5 +1,7 @@ from typing import Any, Dict, List, Literal, Optional, Union +from pydantic import BaseModel, ConfigDict + from dstack._internal.core.models.common import CoreModel @@ -27,6 +29,26 @@ class ChatCompletionsRequest(CoreModel): user: Optional[str] = None +class ChatCompletionsStreamOptions(BaseModel): + model_config = ConfigDict(extra="allow") + + include_usage: Optional[bool] = None + continuous_usage_stats: Optional[bool] = None + + +class ChatCompletionsPassthroughRequest(BaseModel): + """ + A chat completions request that is forwarded to the model as is. + Declares only the fields that dstack-proxy reads or sets. + """ + + model_config = ConfigDict(extra="allow") + + model: str + stream: Optional[bool] = None + stream_options: Optional[ChatCompletionsStreamOptions] = None + + class ChatCompletionsChoice(CoreModel): finish_reason: str index: int diff --git a/src/dstack/_internal/proxy/lib/services/model_proxy/clients/openai.py b/src/dstack/_internal/proxy/lib/services/model_proxy/clients/openai.py deleted file mode 100644 index dfd0e526f1..0000000000 --- a/src/dstack/_internal/proxy/lib/services/model_proxy/clients/openai.py +++ /dev/null @@ -1,70 +0,0 @@ -from typing import AsyncIterator - -import httpx -from fastapi import status -from pydantic import ValidationError - -from dstack._internal.core.models.common import validate_json_extra_ignore -from dstack._internal.proxy.lib.errors import ProxyError -from dstack._internal.proxy.lib.schemas.model_proxy import ( - ChatCompletionsChunk, - ChatCompletionsRequest, - ChatCompletionsResponse, -) -from dstack._internal.proxy.lib.services.model_proxy.clients.base import ChatCompletionsClient - - -class OpenAIChatCompletions(ChatCompletionsClient): - def __init__(self, http_client: httpx.AsyncClient, prefix: str): - self._http = http_client - self._prefix = prefix - - async def generate(self, request: ChatCompletionsRequest) -> ChatCompletionsResponse: - try: - resp = await self._http.post( - f"{self._prefix}/chat/completions", json=request.model_dump(exclude_unset=True) - ) - await self._propagate_error(resp) - except httpx.RequestError as e: - raise ProxyError(f"Error requesting model: {e!r}", status.HTTP_502_BAD_GATEWAY) - - try: - return validate_json_extra_ignore(ChatCompletionsResponse, resp.content) - except ValidationError as e: - raise ProxyError(f"Invalid response from model: {e}", status.HTTP_502_BAD_GATEWAY) - - async def stream(self, request: ChatCompletionsRequest) -> AsyncIterator[ChatCompletionsChunk]: - try: - async with self._http.stream( - "POST", - f"{self._prefix}/chat/completions", - json=request.model_dump(exclude_unset=True), - ) as resp: - await self._propagate_error(resp) - - async for line in resp.aiter_lines(): - if not line.startswith("data:"): - continue - data = line[len("data:") :].strip() - if data == "[DONE]": - break - yield self._parse_chunk_data(data) - except httpx.RequestError as e: - raise ProxyError(f"Error requesting model: {e!r}", status.HTTP_502_BAD_GATEWAY) - - @staticmethod - def _parse_chunk_data(data: str) -> ChatCompletionsChunk: - try: - return validate_json_extra_ignore(ChatCompletionsChunk, data) - except ValidationError as e: - raise ProxyError(f"Invalid chunk in model stream: {e}", status.HTTP_502_BAD_GATEWAY) - - @staticmethod - async def _propagate_error(resp: httpx.Response) -> None: - """ - Propagates HTTP error by raising ProxyError if status is not 200. - May also raise httpx.RequestError if there are issues reading the response. - """ - if resp.status_code != 200: - resp_body = await resp.aread() - raise ProxyError(resp_body.decode(errors="replace"), code=resp.status_code) diff --git a/src/dstack/_internal/proxy/lib/services/model_proxy/clients/tgi.py b/src/dstack/_internal/proxy/lib/services/model_proxy/clients/tgi.py index 2b1742acdc..84165558ea 100644 --- a/src/dstack/_internal/proxy/lib/services/model_proxy/clients/tgi.py +++ b/src/dstack/_internal/proxy/lib/services/model_proxy/clients/tgi.py @@ -3,7 +3,7 @@ import uuid from typing import AsyncIterator, Dict, List -import httpx +import httpx2 as httpx import jinja2 import jinja2.sandbox from fastapi import status diff --git a/src/dstack/_internal/proxy/lib/services/model_proxy/model_proxy.py b/src/dstack/_internal/proxy/lib/services/model_proxy/model_proxy.py index 0661f79699..5a19efab7e 100644 --- a/src/dstack/_internal/proxy/lib/services/model_proxy/model_proxy.py +++ b/src/dstack/_internal/proxy/lib/services/model_proxy/model_proxy.py @@ -1,9 +1,8 @@ -import httpx +import httpx2 as httpx from dstack._internal.proxy.lib.errors import UnexpectedProxyError from dstack._internal.proxy.lib.models import ChatModel from dstack._internal.proxy.lib.services.model_proxy.clients.base import ChatCompletionsClient -from dstack._internal.proxy.lib.services.model_proxy.clients.openai import OpenAIChatCompletions from dstack._internal.proxy.lib.services.model_proxy.clients.tgi import TGIChatCompletions @@ -14,10 +13,5 @@ def get_chat_client(model: ChatModel, http_client: httpx.AsyncClient) -> ChatCom chat_template=model.format_spec.chat_template, eos_token=model.format_spec.eos_token, ) - elif model.format_spec.format == "openai": - return OpenAIChatCompletions( - http_client=http_client, - prefix=model.format_spec.prefix, - ) else: raise UnexpectedProxyError(f"Unsupported model format {model.format_spec.format}") diff --git a/src/dstack/_internal/proxy/lib/services/model_proxy/passthrough.py b/src/dstack/_internal/proxy/lib/services/model_proxy/passthrough.py new file mode 100644 index 0000000000..b30b54677a --- /dev/null +++ b/src/dstack/_internal/proxy/lib/services/model_proxy/passthrough.py @@ -0,0 +1,275 @@ +""" +Chat completions for models with the `openai` format. Requests and responses pass through +unchanged, and the tokens that the model reports in `usage` are metered. +""" + +import asyncio +from typing import Annotated, Any, AsyncIterator, Coroutine, Optional, TypeVar + +import httpx2 as httpx +from fastapi import Request, Response, status +from fastapi.responses import StreamingResponse +from pydantic import BaseModel, ConfigDict, Field, ValidationError +from starlette.types import Receive, Scope, Send + +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.errors import ProxyError +from dstack._internal.proxy.lib.metering import BaseProxyMeter +from dstack._internal.proxy.lib.models import Service +from dstack._internal.proxy.lib.repo import BaseProxyRepo +from dstack._internal.proxy.lib.schemas.model_proxy import ( + ChatCompletionsPassthroughRequest, + ChatCompletionsStreamOptions, +) +from dstack._internal.proxy.lib.services.service_connection import ( + ServiceConnectionPool, + get_service_replica_client, +) +from dstack._internal.utils.logging import get_logger + +logger = get_logger(__name__) +T = TypeVar("T") +ModelT = TypeVar("ModelT", bound=BaseModel) + +_MAX_TOKEN_COUNT = 2**31 +"""Counts above this are not real. `usage` that has them is ignored.""" +_TokenCount = Annotated[Optional[int], Field(ge=0, le=_MAX_TOKEN_COUNT)] + + +async def forward_chat_completions( + request: Request, + body: ChatCompletionsPassthroughRequest, + prefix: str, + service: Service, + export_id: Optional[str], + caller: ProxyCaller, + repo: BaseProxyRepo, + service_conn_pool: ServiceConnectionPool, + meter: BaseProxyMeter, +) -> Response: + """ + Forwards the request to the service and returns the service's response, whatever its status. + Meters the tokens of the response as they are counted. If the client disconnects, + closes the connection to the service, so that the model stops generating. + + `service` must have at least one replica. `export_id` is the export that the request + comes through, `None` for a request to the project's own model. + """ + stream_options = body.stream_options or ChatCompletionsStreamOptions() + if body.stream: + # Ask for token counts on every chunk, so that a stream the client stops is metered too. + body = body.model_copy( + update={ + "stream_options": stream_options.model_copy( + update={"include_usage": True, "continuous_usage_stats": True} + ) + } + ) + request_meter = _RequestMeter(meter, caller, service, export_id) + client = await get_service_replica_client(service, repo, service_conn_pool) + upstream_request = client.build_request( + "POST", f"{prefix}/chat/completions", json=body.model_dump(exclude_unset=True) + ) + try: + upstream_response = await _run_until_disconnect( + request, client.send(upstream_request, stream=True) + ) + except httpx.RequestError as e: + raise ProxyError(f"Error requesting model: {e!r}", status.HTTP_502_BAD_GATEWAY) + if body.stream and upstream_response.status_code == status.HTTP_200_OK: + return _UpstreamStreamingResponse( + _stream(upstream_response, stream_options, request_meter), + upstream_response=upstream_response, + media_type="text/event-stream", + headers={"X-Accel-Buffering": "no"}, + ) + try: + content = await _run_until_disconnect(request, upstream_response.aread()) + except httpx.RequestError as e: + raise ProxyError(f"Error requesting model: {e!r}", status.HTTP_502_BAD_GATEWAY) + finally: + await upstream_response.aclose() + if upstream_response.status_code == status.HTTP_200_OK: + request_meter.update(_parse(_Completion, content).usage) + request_meter.log_if_no_counts() + response = Response( + content=content, + status_code=upstream_response.status_code, + media_type=upstream_response.headers.get("content-type"), + ) + if "retry-after" in upstream_response.headers: + response.headers["Retry-After"] = upstream_response.headers["retry-after"] + return response + + +class _UpstreamStreamingResponse(StreamingResponse): + """ + Closes the response of the service when the stream is over, + including when the client disconnects before the stream starts. + """ + + def __init__(self, *args: Any, upstream_response: httpx.Response, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._upstream_response = upstream_response + + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: + try: + await super().__call__(scope, receive, send) + finally: + await self._upstream_response.aclose() + + +class _PromptTokensDetails(BaseModel): + cached_tokens: _TokenCount = None + + +class _Usage(BaseModel): + """Token counts of a request so far. `prompt_tokens` includes `cached_tokens`.""" + + prompt_tokens: _TokenCount = None + completion_tokens: _TokenCount = None + prompt_tokens_details: Optional[_PromptTokensDetails] = None + + +class _Completion(BaseModel): + usage: Optional[_Usage] = None + + +class _Chunk(BaseModel): + model_config = ConfigDict(extra="allow") + + choices: Optional[list[Any]] = None + usage: Optional[_Usage] = None + + +class _RequestMeter: + """ + Meters one request. The counts in `usage` are totals for the request so far, + so each update adds only what is new since the previous one. + """ + + def __init__( + self, + meter: BaseProxyMeter, + caller: ProxyCaller, + service: Service, + export_id: Optional[str], + ) -> None: + self._meter = meter + self._caller = caller + self._service = service + self._export_id = export_id + self._input_tokens = 0 + self._cached_input_tokens = 0 + self._output_tokens = 0 + + def update(self, usage: Optional[_Usage]) -> None: + if usage is None: + return + details = usage.prompt_tokens_details + input_tokens = max((usage.prompt_tokens or 0) - self._input_tokens, 0) + cached_input_tokens = max( + ((details.cached_tokens or 0) if details is not None else 0) + - self._cached_input_tokens, + 0, + ) + output_tokens = max((usage.completion_tokens or 0) - self._output_tokens, 0) + if not (input_tokens or cached_input_tokens or output_tokens): + return + self._input_tokens += input_tokens + self._cached_input_tokens += cached_input_tokens + self._output_tokens += output_tokens + self._meter.add( + self._caller, + self._service, + self._export_id, + input_tokens=input_tokens, + cached_input_tokens=cached_input_tokens, + output_tokens=output_tokens, + ) + + def log_if_no_counts(self) -> None: + if not (self._input_tokens or self._output_tokens): + logger.debug("Service %s returned no token counts", self._service.fmt()) + + +async def _stream( + upstream_response: httpx.Response, + stream_options: ChatCompletionsStreamOptions, + request_meter: _RequestMeter, +) -> AsyncIterator[bytes]: + """ + Yields the stream as the service sends it, without the `usage` that the client + did not ask for in `stream_options`. + """ + keep_last_usage = stream_options.include_usage is True + keep_all_usage = keep_last_usage and stream_options.continuous_usage_stats is True + is_event_dropped = False + + def process(line: str) -> Optional[str]: + nonlocal is_event_dropped + if is_event_dropped and not line: + # The blank line that ended the dropped event + is_event_dropped = False + return None + if not line.startswith("data:"): + return line + chunk = _parse(_Chunk, line.removeprefix("data:")) + request_meter.update(chunk.usage) + if keep_all_usage or "usage" not in chunk.model_fields_set: + return line + if chunk.choices: + return "data: " + chunk.model_dump_json(exclude={"usage"}, exclude_unset=True) + # The last chunk, which the service sends only to report `usage` + if keep_last_usage: + return line + is_event_dropped = True + return None + + rest = "" + try: + async for text in upstream_response.aiter_text(): + *lines, rest = (rest + text).split("\n") + processed = [process(line.removesuffix("\r")) for line in lines] + if data := "".join(line + "\n" for line in processed if line is not None): + yield data.encode() + if (line := process(rest)) is not None and line: + yield line.encode() + request_meter.log_if_no_counts() + except httpx.RequestError as e: + # No standard way to report errors while streaming, + # but we'll at least send them as comments + yield f": {e!r}\n\n".encode() + + +def _parse(model: type[ModelT], data: bytes | str) -> ModelT: + """Returns an empty `model` if `data` isn't what dstack-proxy expects from the service.""" + try: + return model.model_validate_json(data) + except ValidationError: + return model() + + +async def _run_until_disconnect(request: Request, coro: Coroutine[Any, Any, T]) -> T: + """ + Awaits `coro`. If the client disconnects first, cancels `coro` and raises `ProxyError`. + The tokens of the response are then not metered, since nobody receives it. + """ + task = asyncio.ensure_future(coro) + disconnect_watcher = asyncio.ensure_future(_wait_for_disconnect(request)) + try: + await asyncio.wait([task, disconnect_watcher], return_when=asyncio.FIRST_COMPLETED) + finally: + disconnect_watcher.cancel() + if not task.done(): + task.cancel() + # The cancelled request releases its connection before the handler goes on. + await asyncio.wait([task]) + if task.cancelled(): + raise ProxyError("Client disconnected") + return task.result() + + +async def _wait_for_disconnect(request: Request) -> None: + while (await request.receive())["type"] != "http.disconnect": + pass diff --git a/src/dstack/_internal/proxy/lib/services/service_connection.py b/src/dstack/_internal/proxy/lib/services/service_connection.py index d40c6e0d59..56584941ee 100644 --- a/src/dstack/_internal/proxy/lib/services/service_connection.py +++ b/src/dstack/_internal/proxy/lib/services/service_connection.py @@ -5,8 +5,8 @@ from tempfile import TemporaryDirectory from typing import Dict, Optional -import httpx -from httpx import AsyncHTTPTransport +import httpx2 as httpx +from httpx2 import AsyncHTTPTransport from dstack._internal.core.services.ssh.tunnel import ( SSH_DEFAULT_OPTIONS, @@ -37,6 +37,11 @@ class ServiceClient(httpx.AsyncClient): + """ + The client that the requests to a service share. An httpx2 client, since a request is + cancelled when its client disconnects, and httpx then leaves some connections unreleased. + """ + def build_request(self, *args, **kwargs) -> httpx.Request: self.cookies.clear() # the client is shared by all users, don't leak cookies return super().build_request(*args, **kwargs) @@ -211,6 +216,9 @@ def __init__(self) -> None: self.connections: Dict[str, ServiceConnection] = {} self._draining: set[asyncio.Task] = set() self._reconnect_semaphore = asyncio.Semaphore(MAX_CONCURRENT_TUNNEL_RECONNECTS) + self._nginx_clients: Dict[str, ServiceClient] = {} + """Clients that forward requests to services via Nginx, by service domain. + Only used on gateways.""" async def get(self, replica_id: str) -> Optional[ServiceConnection]: return self.connections.get(replica_id) @@ -280,6 +288,30 @@ async def remove_all(self) -> None: logger.error( "Error removing connection to service replica %s: %s", replica_ids[i], exc ) + for domain in list(self._nginx_clients): + await self.remove_nginx_client(domain) + + def get_nginx_client(self, service: Service) -> ServiceClient: + """ + Returns the client that forwards requests to the service via Nginx. + All requests to the service share the client, so that their connections are reused. + """ + domain = service.domain_safe + client = self._nginx_clients.get(domain) + if client is None: + client = self._nginx_clients[domain] = ServiceClient( + base_url="http://127.0.0.1", + headers={"Host": domain}, + timeout=service.read_timeout, + # Each request in flight holds a connection, and their number is not limited. + limits=httpx.Limits(max_connections=None, max_keepalive_connections=20), + ) + return client + + async def remove_nginx_client(self, domain: str) -> None: + client = self._nginx_clients.pop(domain, None) + if client is not None: + await client.aclose() async def get_service_replica_client( @@ -290,11 +322,7 @@ async def get_service_replica_client( """ if service.domain is not None: # Forward to Nginx so that requests are visible to StatsCollector in the access log - return httpx.AsyncClient( - base_url="http://127.0.0.1", - headers={"Host": service.domain}, - timeout=service.read_timeout, - ) + return service_conn_pool.get_nginx_client(service) # Nginx not available, forward directly to the tunnel replica = random.choice(service.replicas) connection = await service_conn_pool.get(replica.id) diff --git a/src/dstack/_internal/proxy/lib/testing/auth.py b/src/dstack/_internal/proxy/lib/testing/auth.py index 0a5153ddd7..0df0c9293f 100644 --- a/src/dstack/_internal/proxy/lib/testing/auth.py +++ b/src/dstack/_internal/proxy/lib/testing/auth.py @@ -1,6 +1,7 @@ +import uuid from typing import Container, Optional -from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider +from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider, ProxyCaller class ProxyTestAuthProvider(BaseProxyAuthProvider): @@ -9,3 +10,16 @@ def __init__(self, project_to_tokens: Optional[dict[str, Container[str]]] = None async def is_project_member(self, project_name: str, token: str) -> bool: return token in self._project_to_tokens.get(project_name, set()) + + async def get_caller(self, project_name: str, token: str) -> Optional[ProxyCaller]: + if not await self.is_project_member(project_name, token): + return None + return make_caller(project_name, token) + + +def make_caller(project_name: str, token: str) -> ProxyCaller: + """IDs are derived from the project name and the token, so tests can tell callers apart.""" + return ProxyCaller( + project_id=uuid.uuid5(uuid.NAMESPACE_DNS, project_name), + user_id=uuid.uuid5(uuid.NAMESPACE_DNS, token), + ) diff --git a/src/dstack/_internal/proxy/lib/testing/common.py b/src/dstack/_internal/proxy/lib/testing/common.py index 5481dcfe8b..64f438ad17 100644 --- a/src/dstack/_internal/proxy/lib/testing/common.py +++ b/src/dstack/_internal/proxy/lib/testing/common.py @@ -3,15 +3,23 @@ from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider from dstack._internal.proxy.lib.deps import ProxyDependencyInjector +from dstack._internal.proxy.lib.metering import BaseProxyMeter from dstack._internal.proxy.lib.models import Project, Replica, Service from dstack._internal.proxy.lib.repo import BaseProxyRepo +from dstack._internal.proxy.lib.testing.metering import ProxyTestMeter class ProxyTestDependencyInjector(ProxyDependencyInjector): - def __init__(self, repo: BaseProxyRepo, auth: BaseProxyAuthProvider) -> None: + def __init__( + self, + repo: BaseProxyRepo, + auth: BaseProxyAuthProvider, + meter: Optional[BaseProxyMeter] = None, + ) -> None: super().__init__() self._repo = repo self._auth = auth + self._meter = meter or ProxyTestMeter() async def get_repo(self) -> AsyncGenerator[BaseProxyRepo, None]: yield self._repo @@ -19,6 +27,9 @@ async def get_repo(self) -> AsyncGenerator[BaseProxyRepo, None]: async def get_auth_provider(self) -> AsyncGenerator[BaseProxyAuthProvider, None]: yield self._auth + def get_meter(self) -> BaseProxyMeter: + return self._meter + def make_project(name: str) -> Project: return Project(name=name, ssh_private_key="secret") diff --git a/src/dstack/_internal/proxy/lib/testing/metering.py b/src/dstack/_internal/proxy/lib/testing/metering.py new file mode 100644 index 0000000000..0c2d34e32b --- /dev/null +++ b/src/dstack/_internal/proxy/lib/testing/metering.py @@ -0,0 +1,38 @@ +from dataclasses import dataclass +from typing import Optional + +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.metering import BaseProxyMeter +from dstack._internal.proxy.lib.models import Service + + +@dataclass(frozen=True) +class MeteredTokens: + caller: ProxyCaller + service: Service + export_id: Optional[str] + input_tokens: int + cached_input_tokens: int + output_tokens: int + + +class ProxyTestMeter(BaseProxyMeter): + """Keeps every `add` call in `added`, in order.""" + + def __init__(self) -> None: + self.added: list[MeteredTokens] = [] + + def add( + self, + caller: ProxyCaller, + service: Service, + export_id: Optional[str], + input_tokens: int, + cached_input_tokens: int, + output_tokens: int, + ) -> None: + self.added.append( + MeteredTokens( + caller, service, export_id, input_tokens, cached_input_tokens, output_tokens + ) + ) diff --git a/src/dstack/_internal/server/background/pipeline_tasks/gateway_replicas.py b/src/dstack/_internal/server/background/pipeline_tasks/gateway_replicas.py index 8dafff69c0..93a678615c 100644 --- a/src/dstack/_internal/server/background/pipeline_tasks/gateway_replicas.py +++ b/src/dstack/_internal/server/background/pipeline_tasks/gateway_replicas.py @@ -2,13 +2,14 @@ import uuid from dataclasses import dataclass, field from datetime import timedelta -from typing import Any, Optional, Sequence +from typing import Any, Awaitable, Callable, Optional, Sequence from httpx import HTTPStatusError from sqlalchemy import and_, delete, or_, select, update from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import ( InstrumentedAttribute, + aliased, joinedload, load_only, raiseload, @@ -24,8 +25,11 @@ from dstack._internal.core.models.common import validate_json_extra_ignore from dstack._internal.core.models.gateways import GatewayReplicaStatus, GatewayStatus from dstack._internal.core.models.instances import RemoteConnectionInfo -from dstack._internal.core.models.runs import JobSpec, JobStatus, RunStatus, ServiceSpec -from dstack._internal.proxy.gateway.schemas.registry import ReplicaProxyJumpKeyUpdate +from dstack._internal.core.models.runs import JobSpec, JobStatus, ServiceSpec +from dstack._internal.proxy.gateway.schemas.registry import ( + ImportListItem, + ReplicaProxyJumpKeyUpdate, +) from dstack._internal.proxy.gateway.schemas.services import ServiceListItem from dstack._internal.server import settings from dstack._internal.server.background.pipeline_tasks.base import ( @@ -44,8 +48,11 @@ from dstack._internal.server.db import get_db, get_session_ctx from dstack._internal.server.models import ( BackendModel, + ExportedModelModel, + ExportModel, GatewayModel, GatewayReplicaModel, + ImportModel, InstanceModel, JobModel, ProjectModel, @@ -56,6 +63,12 @@ from dstack._internal.server.services import events from dstack._internal.server.services import gateways as gateways_services from dstack._internal.server.services.backends import cache as backends_cache +from dstack._internal.server.services.exported_models import ( + can_serve_importer_projects, + get_running_service_filters, + get_service_model, +) +from dstack._internal.server.services.exports import are_exports_enabled from dstack._internal.server.services.gateways import ( get_gateway_configuration, get_gateway_lb_configuration, @@ -686,17 +699,22 @@ async def _process_running_item(item: GatewayReplicaPipelineItem): logger.warning("%s: failed to list services: %r", fmt(replica_model), e) await _commit_update(item, replica_model, update_map={}) return - run_filters = ( - RunModel.gateway_id == gateway_model.id, - RunModel.deleted == False, - RunModel.status.not_in(RunStatus.finished_statuses() + [RunStatus.TERMINATING]), - ) + # Imports are synced only where exports are enabled. + currently_registered_imports = None + if are_exports_enabled(): + currently_registered_imports = await _list_imports(client, replica_model) + await _run_process_running_gateway_replica_hooks(replica_model, connection) + run_filters = (RunModel.gateway_id == gateway_model.id, *get_running_service_filters()) job_filters = (JobModel.status == JobStatus.RUNNING, JobModel.registered == True) + run_fields: list[InstrumentedAttribute[Any]] = [RunModel.id] + if currently_registered_imports is not None: + # The imports that the replica should have follow from the models of its services. + run_fields += [RunModel.project_id, RunModel.service_spec] stmt = ( select(RunModel) .where(*run_filters) .options( - load_only(RunModel.id), + load_only(*run_fields), joinedload(RunModel.jobs).load_only(JobModel.id, JobModel.instance_id), with_loader_criteria(JobModel, and_(*job_filters)), ) @@ -717,6 +735,9 @@ async def _process_running_item(item: GatewayReplicaPipelineItem): ) ) instance_connection_info = res.all() + expected_imports: set[_ImportRef] = set() + if currently_registered_imports is not None: + expected_imports = await _get_expected_imports(session, gateway_model, expected_runs) # Parse keys and compare gateway state without holding a DB connection. instance_proxy_jump_public_keys = {} @@ -735,6 +756,8 @@ async def _process_running_item(item: GatewayReplicaPipelineItem): currently_registered=currently_registered_services, expected=expected_runs, instance_proxy_jump_public_keys=instance_proxy_jump_public_keys, + currently_registered_imports=currently_registered_imports, + expected_imports=expected_imports, ) async with get_session_ctx() as session: run_models_by_id, job_models_by_id = await _load_runs_and_jobs_for_state_sync( @@ -773,6 +796,114 @@ async def _process_running_item(item: GatewayReplicaPipelineItem): ) +_PROCESS_RUNNING_GATEWAY_REPLICA_HOOKS: list[ + Callable[[uuid.UUID, GatewayConnection], Awaitable[None]] +] = [] + + +def register_process_running_gateway_replica_hook( + func: Callable[[uuid.UUID, GatewayConnection], Awaitable[None]], +): + """ + Registers `func` to be called with the ID of a running gateway replica and the connection + to it each time `GatewayReplicaPipeline` processes the replica and the replica responds. + + The pipeline keeps the replica locked during the call, so server replicas do not call `func` + for the same gateway replica at once, unless the lock expires. `func` should not take long, + since the replica's services are synced only after it returns. If `func` raises, + the error is logged, and the replica is processed further. + """ + _PROCESS_RUNNING_GATEWAY_REPLICA_HOOKS.append(func) + + +async def _run_process_running_gateway_replica_hooks( + replica_model: GatewayReplicaModel, connection: GatewayConnection +) -> None: + for hook in _PROCESS_RUNNING_GATEWAY_REPLICA_HOOKS: + try: + await hook(replica_model.id, connection) + except Exception: + logger.exception("%s: process running gateway replica hook failed", fmt(replica_model)) + + +async def _list_imports( + client: GatewayClient, replica_model: GatewayReplicaModel +) -> Optional[list[ImportListItem]]: + """Returns `None` if the imports cannot be listed. They are not synced then.""" + try: + return await client.list_imports() + except Exception as e: + if isinstance(e, HTTPStatusError) and e.response.status_code == 404: + logger.debug( + ( + "%s: got error 404 when listing imports, which indicates a gateway" + " that is not updated yet. Skipping imports sync until the gateway is updated" + ), + fmt(replica_model), + ) + else: + logger.warning("%s: failed to list imports: %r", fmt(replica_model), e) + return None + + +async def _get_expected_imports( + session: AsyncSession, gateway_model: GatewayModel, expected_runs: Sequence[RunModel] +) -> set["_ImportRef"]: + """ + Returns the imports that the replicas of the gateway should have: for each of the runs + that can serve its model to importer projects, one per export of the run's project + that holds the model and per importer project of that export. + + If a project imports models with the same name from several exports, only the import + from the oldest export is returned, since a gateway keeps one import per project and name. + """ + # (project ID, model name) of the services that can serve their models to importer projects + served_models: set[tuple[uuid.UUID, str]] = set() + for run_model in expected_runs: + model = get_service_model(run_model) + if model is not None and can_serve_importer_projects(model, gateway_model): + served_models.add((run_model.project_id, model.name)) + if not served_models: + return set() + exporter_project_alias = aliased(ProjectModel) + importer_project_alias = aliased(ProjectModel) + res = await session.execute( + select( + ExportModel.id, + ExportModel.project_id, + exporter_project_alias.name, + ExportedModelModel.name, + importer_project_alias.name, + ) + .join(ExportModel.project.of_type(exporter_project_alias)) + .join(ExportModel.exported_models) + .join(ExportModel.imports) + .join(ImportModel.project.of_type(importer_project_alias)) + .where( + ExportModel.project_id.in_({project_id for project_id, _ in served_models}), + ExportedModelModel.name.in_({name for _, name in served_models}), + importer_project_alias.deleted == False, + ) + .order_by(ExportModel.created_at, ExportModel.id) + ) + rows = res.all() + # (importer project, model name) -> import + imports: dict[tuple[str, str], _ImportRef] = {} + for export_id, project_id, exporter_project_name, model_name, importer_project_name in rows: + if (project_id, model_name) not in served_models: + continue + # The gateway keeps project names in lower case. + import_ref = _ImportRef( + importer_project=importer_project_name.lower(), + export_id=export_id, + exporter_project=exporter_project_name.lower(), + name=model_name, + ) + # The rows start with the oldest export, whose import is the one to stay. + imports.setdefault((import_ref.importer_project, import_ref.name), import_ref) + return set(imports.values()) + + async def _perform_state_sync( connection: GatewayConnection, gateway_model: GatewayModel, @@ -945,6 +1076,37 @@ async def _perform_state_sync( result.failed_replica_registrations[replica_id] = "Unexpected error" continue result.registered_replicas.add(replica_id) + # Imports are registered before the others are unregistered. When the import of a model + # moves to another export, registering replaces it on the gateway, and unregistering + # the previous one then changes nothing, so the model stays imported all along. + # Failures are only logged: the next sync lists the imports and tries again. + for import_ref in plan.register_imports: + logger.debug("%s: registering import %s", fmt(replica_model), import_ref) + try: + async with connection.client() as client: + await client.register_import( + importer_project=import_ref.importer_project, + export_id=import_ref.export_id, + exporter_project=import_ref.exporter_project, + name=import_ref.name, + ) + except Exception as e: + logger.warning( + "%s: failed to register import %s: %r", fmt(replica_model), import_ref, e + ) + for import_ref in plan.unregister_imports: + logger.debug("%s: unregistering import %s", fmt(replica_model), import_ref) + try: + async with connection.client() as client: + await client.unregister_import( + importer_project=import_ref.importer_project, + export_id=import_ref.export_id, + name=import_ref.name, + ) + except Exception as e: + logger.warning( + "%s: failed to unregister import %s: %r", fmt(replica_model), import_ref, e + ) # Send all replacements together. The gateway keeps working tunnels open and # saves its state once, rather than rewriting the full state for each replica. key_updates = [] @@ -1443,6 +1605,15 @@ class _ServiceRef: run_name: str +@dataclass(frozen=True) +class _ImportRef: + importer_project: str + export_id: uuid.UUID + exporter_project: str + name: str + """The model's name in the exporter project.""" + + @dataclass class _StateSyncPlan: register_services: set[uuid.UUID] = field(default_factory=set) @@ -1453,6 +1624,8 @@ class _StateSyncPlan: set_run_ids: dict[_ServiceRef, uuid.UUID] = field(default_factory=dict) update_replica_proxy_jump_keys: dict[uuid.UUID, set[uuid.UUID]] = field(default_factory=dict) """Run IDs mapped to registered job IDs whose saved gateway proxy_jump key needs replacing.""" + register_imports: set[_ImportRef] = field(default_factory=set) + unregister_imports: set[_ImportRef] = field(default_factory=set) @dataclass @@ -1473,7 +1646,13 @@ def _plan_state_sync( currently_registered: list[ServiceListItem], expected: Sequence[RunModel], instance_proxy_jump_public_keys: dict[uuid.UUID, str], + currently_registered_imports: Optional[list[ImportListItem]], + expected_imports: set[_ImportRef], ) -> _StateSyncPlan: + """ + `currently_registered_imports` is `None` if the gateway replica could not list them. + The plan then leaves the imports as they are. + """ plan = _StateSyncPlan() expected_run_id_to_run = {run.id: run for run in expected} expected_job_id_to_run = {job.id: run for run in expected for job in run.jobs} @@ -1527,6 +1706,19 @@ def _plan_state_sync( plan.register_services.add(run_id) plan.register_replicas[run_id] = {job.id for job in expected_run_id_to_run[run_id].jobs} + if currently_registered_imports is not None: + currently_registered_import_refs = { + _ImportRef( + importer_project=imported_model.importer_project, + export_id=uuid.UUID(imported_model.export_id), + exporter_project=imported_model.exporter_project, + name=imported_model.name, + ) + for imported_model in currently_registered_imports + } + plan.register_imports = expected_imports - currently_registered_import_refs + plan.unregister_imports = currently_registered_import_refs - expected_imports + return plan diff --git a/src/dstack/_internal/server/migrations/versions/2026/11_18_1345_17e6c7ef91dd_add_exportedmodelmodel.py b/src/dstack/_internal/server/migrations/versions/2026/11_18_1345_17e6c7ef91dd_add_exportedmodelmodel.py new file mode 100644 index 0000000000..05248e8551 --- /dev/null +++ b/src/dstack/_internal/server/migrations/versions/2026/11_18_1345_17e6c7ef91dd_add_exportedmodelmodel.py @@ -0,0 +1,50 @@ +"""Add ExportedModelModel + +Revision ID: 17e6c7ef91dd +Revises: 2d2692d5b3c2 +Create Date: 2026-11-18 13:45:02.088337+00:00 + +""" + +import sqlalchemy as sa +import sqlalchemy_utils +from alembic import op + +# revision identifiers, used by Alembic. +revision = "17e6c7ef91dd" +down_revision = "2d2692d5b3c2" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + op.create_table( + "exported_models", + sa.Column("id", sqlalchemy_utils.types.uuid.UUIDType(binary=False), nullable=False), + sa.Column("export_id", sqlalchemy_utils.types.uuid.UUIDType(binary=False), nullable=False), + sa.Column("name", sa.Text(), nullable=False), + sa.ForeignKeyConstraint( + ["export_id"], + ["exports.id"], + name=op.f("fk_exported_models_export_id_exports"), + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id", name=op.f("pk_exported_models")), + sa.UniqueConstraint("export_id", "name", name="uq_exported_models_export_id_name"), + ) + with op.batch_alter_table("exported_models", schema=None) as batch_op: + batch_op.create_index( + batch_op.f("ix_exported_models_export_id"), ["export_id"], unique=False + ) + + # ### end Alembic commands ### + + +def downgrade() -> None: + # ### commands auto generated by Alembic - please adjust! ### + with op.batch_alter_table("exported_models", schema=None) as batch_op: + batch_op.drop_index(batch_op.f("ix_exported_models_export_id")) + + op.drop_table("exported_models") + # ### end Alembic commands ### diff --git a/src/dstack/_internal/server/models.py b/src/dstack/_internal/server/models.py index 27c5311ea1..51adbeb11f 100644 --- a/src/dstack/_internal/server/models.py +++ b/src/dstack/_internal/server/models.py @@ -1370,6 +1370,10 @@ class ExportModel(BaseModel): back_populates="export", cascade=CASCADE_DEFAULT_WITH_DELETE_ORPHAN, ) + exported_models: Mapped[List["ExportedModelModel"]] = relationship( + back_populates="export", + cascade=CASCADE_DEFAULT_WITH_DELETE_ORPHAN, + ) class ImportModel(BaseModel): @@ -1436,6 +1440,26 @@ class ExportedGatewayModel(BaseModel): gateway: Mapped["GatewayModel"] = relationship() +class ExportedModelModel(BaseModel): + __tablename__ = "exported_models" + __table_args__ = ( + UniqueConstraint("export_id", "name", name="uq_exported_models_export_id_name"), + ) + + id: Mapped[uuid.UUID] = mapped_column( + UUIDType(binary=False), primary_key=True, default=uuid.uuid4 + ) + export_id: Mapped[uuid.UUID] = mapped_column( + ForeignKey("exports.id", ondelete="CASCADE"), index=True + ) + export: Mapped["ExportModel"] = relationship() + name: Mapped[str] = mapped_column(Text) + """`name` is the name of the model in the export's project. The export holds the model + by its name and not by the service that declares it, so the model stays exported + when the service is stopped and applied again or replaced by another service. + """ + + class UserPublicKeyModel(BaseModel): __tablename__ = "user_public_keys" __table_args__ = ( diff --git a/src/dstack/_internal/server/routers/exports.py b/src/dstack/_internal/server/routers/exports.py index 7ae340e915..add5906151 100644 --- a/src/dstack/_internal/server/routers/exports.py +++ b/src/dstack/_internal/server/routers/exports.py @@ -43,6 +43,7 @@ async def create_export( importer_project_names=body.importer_projects, exported_fleet_names=body.exported_fleets, exported_gateway_names=body.exported_gateways, + exported_model_names=body.exported_models, ) @@ -66,6 +67,8 @@ async def update_export( remove_exported_fleet_names=body.remove_exported_fleets, add_exported_gateway_names=body.add_exported_gateways, remove_exported_gateway_names=body.remove_exported_gateways, + add_exported_model_names=body.add_exported_models, + remove_exported_model_names=body.remove_exported_models, ) diff --git a/src/dstack/_internal/server/schemas/exports.py b/src/dstack/_internal/server/schemas/exports.py index 74828fb455..50e07ba574 100644 --- a/src/dstack/_internal/server/schemas/exports.py +++ b/src/dstack/_internal/server/schemas/exports.py @@ -7,6 +7,7 @@ class CreateExportRequest(CoreModel): importer_projects: list[str] = [] exported_fleets: list[str] = [] exported_gateways: list[str] = [] + exported_models: list[str] = [] class UpdateExportRequest(CoreModel): @@ -19,6 +20,8 @@ class UpdateExportRequest(CoreModel): remove_exported_fleets: list[str] = [] add_exported_gateways: list[str] = [] remove_exported_gateways: list[str] = [] + add_exported_models: list[str] = [] + remove_exported_models: list[str] = [] class DeleteExportRequest(CoreModel): diff --git a/src/dstack/_internal/server/services/exported_models.py b/src/dstack/_internal/server/services/exported_models.py new file mode 100644 index 0000000000..3bfd8e3d29 --- /dev/null +++ b/src/dstack/_internal/server/services/exported_models.py @@ -0,0 +1,265 @@ +""" +Application logic related to models that projects export to other projects. +""" + +import uuid +from collections.abc import Iterable +from typing import Optional + +from sqlalchemy import ColumnElement, select +from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy.orm import joinedload, load_only, raiseload + +from dstack._internal.core.errors import ServerClientError +from dstack._internal.core.models.common import EntityReference, validate_json_extra_ignore +from dstack._internal.core.models.runs import RunStatus, ServiceSpec +from dstack._internal.core.models.services import AnyModel +from dstack._internal.server.models import ( + BackendModel, + ExportedModelModel, + ExportModel, + GatewayModel, + ProjectModel, + RunModel, +) +from dstack._internal.utils.common import is_gateway_domain_fixed + + +async def check_can_export_models( + session: AsyncSession, export: ExportModel, names: list[str] +) -> None: + """ + Checks that the running services of the export's project that declare the models + can serve them to importer projects. + + A model that no running service declares can be added to the export: + importer projects access it once a service that can serve it to them starts. + """ + services = await _list_services_with_models(session, RunModel.project_id == export.project.id) + for name in names: + services_with_model = [ + (run_model, model) for run_model, model in services if model.name == name + ] + if not services_with_model: + continue + if any( + can_serve_importer_projects(model, run_model.gateway) + for run_model, model in services_with_model + ): + continue + missing = "; ".join( + _get_cannot_serve_message(run_model, model) for run_model, model in services_with_model + ) + raise ServerClientError(f"Cannot add {name!r} to {export.name!r}: {missing}") + + +async def check_can_update_service( + session: AsyncSession, run_model: RunModel, previous_gateway: Optional[GatewayModel] +) -> None: + """ + Checks that the in-place update of the service does not leave a model exported + by its project with no service that can serve the model to importer projects. + + The update is refused only if the service could serve the model before the update, + can't serve it after the update, and no other running service of the project can. + + Call it once the update is applied to `run_model`, with `run_model.gateway` loaded. + `previous_gateway` is the gateway the service ran on before the update. + """ + # The model of a service is not updated in place, so only its gateway changes. + model = get_service_model(run_model) + if model is None: + return + if not can_serve_importer_projects(model, previous_gateway): + return + if can_serve_importer_projects(model, run_model.gateway): + return + export_names = await _list_export_names(session, [(run_model.project_id, model.name)]) + model_export_names = export_names.get((run_model.project_id, model.name)) + if not model_export_names: + return + other_services = await _list_services_with_models( + session, RunModel.project_id == run_model.project_id, RunModel.id != run_model.id + ) + if any( + other_model.name == model.name + and can_serve_importer_projects(other_model, other_run_model.gateway) + for other_run_model, other_model in other_services + ): + return + raise ServerClientError( + f"Cannot update {run_model.run_name!r}: {model.name!r} is exported by" + f" {', '.join(map(repr, model_export_names))}," + " and no other service would serve it to importer projects." + " Start another service for the model, or remove it from the export first." + ) + + +async def check_can_set_gateway_domain( + session: AsyncSession, gateway_model: GatewayModel, domain: Optional[str] +) -> None: + """ + Checks that the new domain of the gateway does not stop a service on the gateway + from serving an exported model to importer projects. A service serves them only + on a gateway with a fixed domain, so the domain can be neither removed + nor given `${{ run.project_name }}`. + + Call it before the domain of `gateway_model` is updated. + """ + if domain is not None and is_gateway_domain_fixed(domain): + return + services = await _list_services_with_models(session, RunModel.gateway_id == gateway_model.id) + serving_services = [ + (run_model, model) + for run_model, model in services + if can_serve_importer_projects(model, run_model.gateway) + ] + export_names = await _list_export_names( + session, [(run_model.project_id, model.name) for run_model, model in serving_services] + ) + for run_model, model in serving_services: + model_export_names = export_names.get((run_model.project_id, model.name)) + if not model_export_names: + continue + raise ServerClientError( + f"Cannot remove the domain of {gateway_model.name!r}" + f" or put ${{{{ run.project_name }}}} into it: {model.name!r} is exported by" + f" {', '.join(map(repr, model_export_names))}, and {run_model.run_name!r}" + " serves it to importer projects from this gateway." + " Move the service to another gateway, or remove the model from the export first." + ) + + +async def get_services_serving_importer_projects( + session: AsyncSession, project_ids: Iterable[uuid.UUID] +) -> dict[tuple[uuid.UUID, str], RunModel]: + """ + Returns the running services of the projects that can serve their models + to importer projects, by project ID and model name, with the gateways they run on. + + If several services of a project declare the same model, returns the last submitted one. + """ + project_ids = list(project_ids) + if not project_ids: + return {} + services = await _list_services_with_models(session, RunModel.project_id.in_(project_ids)) + return { + (run_model.project_id, model.name): run_model + for run_model, model in services + if can_serve_importer_projects(model, run_model.gateway) + } + + +def can_serve_importer_projects(model: AnyModel, gateway_model: Optional[GatewayModel]) -> bool: + """ + Returns `True` if a service that declares `model` and runs on `gateway_model` + can serve the model to the projects that import it. + + Only requests to models with the `openai` format are metered, and only a gateway + with a fixed domain gives every importer project its model endpoint on one host. + """ + return ( + model.format == "openai" + and gateway_model is not None + and gateway_model.wildcard_domain is not None + and is_gateway_domain_fixed(gateway_model.wildcard_domain) + ) + + +def get_service_model(run_model: RunModel) -> Optional[AnyModel]: + """ + Returns the model that the service declares, as it is registered on the gateway. + `None` if the run is not a service or the service declares no model. + """ + if run_model.service_spec is None: + return None + service_spec = validate_json_extra_ignore(ServiceSpec, run_model.service_spec) + return service_spec.get_model_options() + + +def get_running_service_filters() -> list[ColumnElement[bool]]: + """ + Returns the filters of the runs whose services are running: + not deleted, not finished, not terminating. + `GatewayReplicaPipeline` registers these services on gateway replicas. + """ + return [ + RunModel.deleted == False, + RunModel.status.not_in(RunStatus.finished_statuses() + [RunStatus.TERMINATING]), + ] + + +def _get_cannot_serve_message(run_model: RunModel, model: AnyModel) -> str: + if model.format != "openai": + return f"{run_model.run_name!r} declares it with format {model.format}, not openai" + gateway_model = run_model.gateway + if gateway_model is None: + return f"{run_model.run_name!r} runs without a gateway" + gateway_ref = EntityReference( + project=( + gateway_model.project.name + if gateway_model.project_id != run_model.project_id + else None + ), + name=gateway_model.name, + ) + if gateway_model.wildcard_domain is None: + return f"{run_model.run_name!r} runs on {gateway_ref.format()!r}, which has no domain" + return ( + f"{run_model.run_name!r} runs on {gateway_ref.format()!r}," + " which has ${{ run.project_name }} in its domain" + ) + + +async def _list_services_with_models( + session: AsyncSession, *run_filters: ColumnElement[bool] +) -> list[tuple[RunModel, AnyModel]]: + """ + Lists the running services that declare a model, with the gateways they run on, + in the order they were submitted. + """ + res = await session.execute( + select(RunModel) + .where(*run_filters, *get_running_service_filters()) + .options( + load_only(RunModel.project_id, RunModel.run_name, RunModel.service_spec), + joinedload(RunModel.gateway).options( + joinedload(GatewayModel.backend).load_only(BackendModel.type), + joinedload(GatewayModel.project).load_only(ProjectModel.name), + ), + raiseload(RunModel.jobs), + ) + .order_by(RunModel.submitted_at) + ) + services = [] + for run_model in res.scalars().all(): + model = get_service_model(run_model) + if model is not None: + services.append((run_model, model)) + return services + + +async def _list_export_names( + session: AsyncSession, models: Iterable[tuple[uuid.UUID, str]] +) -> dict[tuple[uuid.UUID, str], list[str]]: + """ + Returns the names of the exports that hold the models, oldest export first. + The models are given and returned by the ID of the export's project and the model's name. + """ + models = set(models) + if not models: + return {} + res = await session.execute( + select(ExportModel.project_id, ExportedModelModel.name, ExportModel.name) + .join(ExportedModelModel, ExportedModelModel.export_id == ExportModel.id) + .where( + ExportModel.project_id.in_({project_id for project_id, _ in models}), + ExportedModelModel.name.in_({model_name for _, model_name in models}), + ) + .order_by(ExportModel.created_at) + ) + export_names: dict[tuple[uuid.UUID, str], list[str]] = {} + for project_id, model_name, export_name in res.all(): + if (project_id, model_name) in models: + export_names.setdefault((project_id, model_name), []).append(export_name) + return export_names diff --git a/src/dstack/_internal/server/services/exports.py b/src/dstack/_internal/server/services/exports.py index ecb136bd1a..833d1e5d8c 100644 --- a/src/dstack/_internal/server/services/exports.py +++ b/src/dstack/_internal/server/services/exports.py @@ -16,6 +16,7 @@ Export, ExportedFleet, ExportedGateway, + ExportedModel, ExportImport, ListExportsResponse, ) @@ -26,6 +27,7 @@ from dstack._internal.server.models import ( ExportedFleetModel, ExportedGatewayModel, + ExportedModelModel, ExportModel, FleetModel, GatewayModel, @@ -34,6 +36,7 @@ ProjectRole, UserModel, ) +from dstack._internal.server.services.exported_models import check_can_export_models from dstack._internal.server.services.fleets import list_project_fleet_models from dstack._internal.server.services.gateways import list_project_gateway_models from dstack._internal.server.services.locking import get_locker, string_to_lock_id @@ -113,6 +116,7 @@ async def get_export_model_by_name_for_update( selectinload(ExportModel.exported_gateways) .joinedload(ExportedGatewayModel.gateway) .load_only(GatewayModel.name), + selectinload(ExportModel.exported_models), ) .with_for_update(key_share=True) ) @@ -137,6 +141,7 @@ async def create_export( importer_project_names: list[str], exported_fleet_names: list[str], exported_gateway_names: list[str], + exported_model_names: list[str], ) -> Export: if not are_exports_enabled(): raise ServerClientError(EXPORTS_NOT_SUPPORTED_MESSAGE) @@ -187,10 +192,12 @@ async def create_export( imports=[], exported_fleets=[], exported_gateways=[], + exported_models=[], ) await add_importer_projects(session, user, export, importer_project_names) await add_exported_fleets(session, export, exported_fleet_names) await add_exported_gateways(session, export, exported_gateway_names) + await add_exported_models(session, export, exported_model_names) if is_global: await set_as_global(session, export, user) session.add(export) @@ -211,6 +218,8 @@ async def update_export( remove_exported_fleet_names: list[str], add_exported_gateway_names: list[str], remove_exported_gateway_names: list[str], + add_exported_model_names: list[str], + remove_exported_model_names: list[str], ) -> Export: if set_global: if is_db_sqlite(): @@ -240,6 +249,7 @@ async def update_export( add_importer_project_names, add_exported_fleet_names, add_exported_gateway_names, + add_exported_model_names, set_global, ) ): @@ -253,6 +263,8 @@ async def update_export( and not remove_exported_fleet_names and not add_exported_gateway_names and not remove_exported_gateway_names + and not add_exported_model_names + and not remove_exported_model_names ): raise ServerClientError("No changes specified") if set_global and unset_global: @@ -263,6 +275,14 @@ async def update_export( raise ServerClientError( "Cannot change global status and add/remove importers at the same time" ) + # A global export cannot have models yet, so, like importers, + # models are not changed in the same update as the global status. + if (set_global or unset_global) and ( + add_exported_model_names or remove_exported_model_names + ): + raise ServerClientError( + "Cannot change global status and add/remove models at the same time" + ) add_importer_project_names = list(map(str.lower, add_importer_project_names)) remove_importer_project_names = list(map(str.lower, remove_importer_project_names)) @@ -291,13 +311,23 @@ async def update_export( f"Gateways {add_remove_conflict_gateways} are listed for both addition and removal." " Cannot add and remove at the same time" ) + add_remove_conflict_models = set(add_exported_model_names) & set( + remove_exported_model_names + ) + if add_remove_conflict_models: + raise ServerClientError( + f"Models {add_remove_conflict_models} are listed for both addition and removal." + " Cannot add and remove at the same time" + ) await add_importer_projects(session, user, export, add_importer_project_names) await add_exported_fleets(session, export, add_exported_fleet_names) await add_exported_gateways(session, export, add_exported_gateway_names) + await add_exported_models(session, export, add_exported_model_names) await remove_importer_projects(export, remove_importer_project_names) await remove_exported_fleets(export, remove_exported_fleet_names) await remove_exported_gateways(export, remove_exported_gateway_names) + await remove_exported_models(export, remove_exported_model_names) if unset_global: await unset_as_global(export) if set_global: @@ -316,6 +346,13 @@ async def set_as_global(session: AsyncSession, export: ExportModel, user: UserMo raise ServerClientError("The export is already global") if user.global_role != GlobalRole.ADMIN: raise ForbiddenError("Only global admins can make the export global") + if export.exported_models: + model_names = ", ".join(repr(em.name) for em in export.exported_models) + raise ServerClientError( + f"Cannot make {export.name!r} global: it exports models ({model_names})," + " and models cannot be exported to all projects yet." + " Name its importer projects instead." + ) all_projects = await list_project_models( session, load_only_attrs=[ProjectModel.id, ProjectModel.name] ) @@ -449,6 +486,40 @@ async def remove_exported_gateways(export: ExportModel, names: list[str]) -> Non ] +async def add_exported_models( + session: AsyncSession, export: ExportModel, names: list[str] +) -> None: + if not names: + return + if any(not name.strip() for name in names): + raise ServerClientError("Model names cannot be empty") + if export.is_global: + raise ServerClientError( + f"Cannot add {', '.join(map(repr, names))} to {export.name!r}:" + " the export is global, and models cannot be exported to all projects yet." + " Add the model to an export that names its importer projects instead." + ) + if len(names) != len(set(names)): + raise ServerClientError("Some models are listed for addition more than once") + already_exported = {em.name for em in export.exported_models} & set(names) + if already_exported: + raise ServerClientError( + f"Models {already_exported} are already exported by export {export.name!r}" + ) + await check_can_export_models(session, export, names) + for name in names: + export.exported_models.append(ExportedModelModel(name=name)) + + +async def remove_exported_models(export: ExportModel, names: list[str]) -> None: + if len(names) != len(set(names)): + raise ServerClientError("Some models are listed for removal more than once") + existing = {em.name for em in export.exported_models} + if missing := set(names) - existing: + raise ServerClientError(f"Models {missing} are not exported by export {export.name!r}") + export.exported_models = [em for em in export.exported_models if em.name not in names] + + async def delete_export(session: AsyncSession, project: ProjectModel, name: str) -> None: async with get_export_model_by_name_for_update(session, project, name) as export: if export is None: @@ -477,6 +548,7 @@ async def list_exports(session: AsyncSession, project: ProjectModel) -> ListExpo selectinload(ExportModel.exported_gateways) .joinedload(ExportedGatewayModel.gateway) .load_only(GatewayModel.name), + selectinload(ExportModel.exported_models), ) .order_by(ExportModel.created_at.desc()) ) @@ -519,4 +591,8 @@ def export_model_to_export(export_model: ExportModel) -> Export: ) for exported_gateway_model in export_model.exported_gateways ], + exported_models=[ + ExportedModel(name=exported_model_model.name) + for exported_model_model in export_model.exported_models + ], ) diff --git a/src/dstack/_internal/server/services/gateways/__init__.py b/src/dstack/_internal/server/services/gateways/__init__.py index 678649426d..529d3c75a5 100644 --- a/src/dstack/_internal/server/services/gateways/__init__.py +++ b/src/dstack/_internal/server/services/gateways/__init__.py @@ -70,6 +70,7 @@ from dstack._internal.server.services.backends.store import ( get_project_backend_model_by_type_or_error, ) +from dstack._internal.server.services.exported_models import check_can_set_gateway_domain from dstack._internal.server.services.gateways.connection import GatewayConnection from dstack._internal.server.services.gateways.pool import gateway_connections_pool from dstack._internal.server.services.locking import ( @@ -434,6 +435,7 @@ async def set_gateway_wildcard_domain( raise ResourceNotExistsError() old_domain = gateway.wildcard_domain if old_domain != wildcard_domain: + await check_can_set_gateway_domain(session, gateway, wildcard_domain) gateway.wildcard_domain = wildcard_domain if gateway.configuration is not None: conf = get_gateway_configuration(gateway) @@ -1124,6 +1126,8 @@ async def apply_plan( " Delete it and re-apply." ) + if "domain" in diff: + await check_can_set_gateway_domain(session, gateway_model, new_configuration.domain) gateway_model.wildcard_domain = new_configuration.domain if new_configuration.replicas != current_configuration.replicas: gateway_model.desired_replica_count = ( diff --git a/src/dstack/_internal/server/services/gateways/client.py b/src/dstack/_internal/server/services/gateways/client.py index bf92b7687d..4ba0db7267 100644 --- a/src/dstack/_internal/server/services/gateways/client.py +++ b/src/dstack/_internal/server/services/gateways/client.py @@ -11,8 +11,17 @@ from dstack._internal.core.models.configurations import RateLimit, ServiceConfiguration from dstack._internal.core.models.instances import SSHConnectionParams from dstack._internal.core.models.runs import JobSpec, JobSubmission, get_service_port +from dstack._internal.proxy.gateway.schemas.metering import ( + CollectMeteringRequest, + CollectMeteringResponse, + MeteringBatch, +) from dstack._internal.proxy.gateway.schemas.registry import ( + ImportListItem, + ImportListResponse, + RegisterImportRequest, ReplicaProxyJumpKeyUpdate, + UnregisterImportRequest, UpdateReplicaProxyJumpKeysResponse, ) from dstack._internal.proxy.gateway.schemas.services import ServiceListItem, ServiceListResponse @@ -56,7 +65,7 @@ async def register_service( has_router_replica: bool = False, ): if "openai" in options: - entrypoint = f"gateway.{domain.split('.', maxsplit=1)[1]}" + entrypoint = get_gateway_endpoint_domain(domain) await self.register_openai_entrypoint(project, entrypoint, gateway_https, read_timeout) payload = { @@ -198,6 +207,36 @@ async def register_openai_entrypoint( resp.raise_for_status() self.is_server_ready = True + async def register_import( + self, importer_project: str, export_id: uuid.UUID, exporter_project: str, name: str + ) -> None: + resp = await self._client.post( + self._url(f"/api/registry/{importer_project}/imports/register"), + json=RegisterImportRequest( + export_id=export_id.hex, + exporter_project=exporter_project, + name=name, + ).model_dump(), + ) + resp.raise_for_status() + self.is_server_ready = True + + async def unregister_import( + self, importer_project: str, export_id: uuid.UUID, name: str + ) -> None: + resp = await self._client.post( + self._url(f"/api/registry/{importer_project}/imports/unregister"), + json=UnregisterImportRequest(export_id=export_id.hex, name=name).model_dump(), + ) + resp.raise_for_status() + self.is_server_ready = True + + async def list_imports(self) -> list[ImportListItem]: + resp = await self._client.get(self._url("/api/registry/imports/list")) + resp.raise_for_status() + self.is_server_ready = True + return validate_json_extra_ignore(ImportListResponse, resp.content).imports + async def list_services(self) -> list[ServiceListItem]: resp = await self._client.get(self._url("/api/services/list")) if resp.status_code == 400: @@ -242,10 +281,31 @@ async def collect_stats(self) -> list[ServiceStats]: return [] return TypeAdapter(list[ServiceStats]).validate_python(resp_data) + async def collect_metering( + self, after: int, limit: int, timeout: float + ) -> list[MeteringBatch]: + """ + Returns up to `limit` metering batches with numbers above `after`, oldest first. + The gateway deletes the batches up to `after`, so pass the number of the last saved batch. + """ + resp = await self._client.post( + self._url("/api/metering/collect"), + json=CollectMeteringRequest(after=after, limit=limit).model_dump(), + timeout=timeout, + ) + resp.raise_for_status() + self.is_server_ready = True + return validate_json_extra_ignore(CollectMeteringResponse, resp.content).batches + def _url(self, path: str) -> str: return f"{self.base_url}/{path.lstrip('/')}" +def get_gateway_endpoint_domain(service_domain: str) -> str: + """Returns the domain where the gateway serves the models of the service.""" + return f"gateway.{service_domain.split('.', maxsplit=1)[1]}" + + def gateway_error(data: dict) -> GatewayError: return GatewayError(msg=data["detail"]) diff --git a/src/dstack/_internal/server/services/imports.py b/src/dstack/_internal/server/services/imports.py index 03103bec10..542c549965 100644 --- a/src/dstack/_internal/server/services/imports.py +++ b/src/dstack/_internal/server/services/imports.py @@ -1,15 +1,20 @@ +import uuid + from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import joinedload, selectinload from dstack._internal.core.errors import ResourceNotExistsError, ServerClientError +from dstack._internal.core.models.common import validate_json_extra_ignore from dstack._internal.core.models.imports import ( Import, ImportExport, ImportExportedFleet, ImportExportedGateway, + ImportExportedModel, ListImportsResponse, ) +from dstack._internal.core.models.runs import ServiceSpec from dstack._internal.server.models import ( ExportedFleetModel, ExportedGatewayModel, @@ -18,6 +23,10 @@ GatewayModel, ImportModel, ProjectModel, + RunModel, +) +from dstack._internal.server.services.exported_models import ( + get_services_serving_importer_projects, ) from dstack._internal.server.services.exports import ( EXPORTS_NOT_SUPPORTED_MESSAGE, @@ -25,7 +34,10 @@ are_exports_enabled, get_export_model_by_name_for_update, ) +from dstack._internal.server.services.gateways import get_gateway_configuration from dstack._internal.server.services.projects import get_project_model_by_name +from dstack._internal.server.services.services import get_gateway_endpoint_url +from dstack._internal.utils.common import get_or_error async def list_imports(session: AsyncSession, project: ProjectModel) -> ListImportsResponse: @@ -47,12 +59,20 @@ async def list_imports(session: AsyncSession, project: ProjectModel) -> ListImpo selectinload(ExportModel.exported_gateways) .joinedload(ExportedGatewayModel.gateway) .load_only(GatewayModel.id, GatewayModel.name), + selectinload(ExportModel.exported_models), ) ) .order_by(ImportModel.created_at.desc()) ) import_models = res.scalars().all() - imports = [import_model_to_import(imp) for imp in import_models] + services = await get_services_serving_importer_projects( + session, + project_ids={imp.export.project.id for imp in import_models if imp.export.exported_models}, + ) + imports = [ + import_model_to_import(imp, _get_import_exported_models(imp.export, project, services)) + for imp in import_models + ] warnings = [] if not are_exports_enabled(): if not imports: @@ -96,7 +116,9 @@ async def delete_import( await session.commit() -def import_model_to_import(import_model: ImportModel) -> Import: +def import_model_to_import( + import_model: ImportModel, exported_models: list[ImportExportedModel] +) -> Import: return Import( id=import_model.id, export=ImportExport( @@ -117,5 +139,31 @@ def import_model_to_import(import_model: ImportModel) -> Import: ) for eg in import_model.export.exported_gateways ], + exported_models=exported_models, ), ) + + +def _get_import_exported_models( + export: ExportModel, project: ProjectModel, services: dict[tuple[uuid.UUID, str], RunModel] +) -> list[ImportExportedModel]: + """ + Returns the models of the export with their URLs for the importer `project`. + The URLs are built from `services`, the services that serve the models now, + so they follow the services. + """ + exported_models = [] + for exported_model in export.exported_models: + base_url = None + run_model = services.get((export.project.id, exported_model.name)) + if run_model is not None: + service_spec = validate_json_extra_ignore( + ServiceSpec, get_or_error(run_model.service_spec) + ) + base_url = get_gateway_endpoint_url( + service_url=service_spec.url, + configuration=get_gateway_configuration(get_or_error(run_model.gateway)), + project_name=project.name, + ) + exported_models.append(ImportExportedModel(name=exported_model.name, base_url=base_url)) + return exported_models diff --git a/src/dstack/_internal/server/services/proxy/auth.py b/src/dstack/_internal/server/services/proxy/auth.py index fecc685095..7e3ad438ff 100644 --- a/src/dstack/_internal/server/services/proxy/auth.py +++ b/src/dstack/_internal/server/services/proxy/auth.py @@ -1,7 +1,10 @@ +from typing import Optional + +from fastapi import HTTPException from sqlalchemy.ext.asyncio import AsyncSession -from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider -from dstack._internal.server.security.permissions import is_project_member +from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider, ProxyCaller +from dstack._internal.server.security.permissions import get_project_member, is_project_member class ServerProxyAuthProvider(BaseProxyAuthProvider): @@ -10,3 +13,10 @@ def __init__(self, session: AsyncSession) -> None: async def is_project_member(self, project_name: str, token: str) -> bool: return await is_project_member(self.session, project_name, token) + + async def get_caller(self, project_name: str, token: str) -> Optional[ProxyCaller]: + try: + user, project = await get_project_member(self.session, project_name, token) + except HTTPException: + return None + return ProxyCaller(project_id=project.id, user_id=user.id) diff --git a/src/dstack/_internal/server/services/proxy/deps.py b/src/dstack/_internal/server/services/proxy/deps.py index 558b143ab8..308f6d6562 100644 --- a/src/dstack/_internal/server/services/proxy/deps.py +++ b/src/dstack/_internal/server/services/proxy/deps.py @@ -2,9 +2,11 @@ from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider from dstack._internal.proxy.lib.deps import ProxyDependencyInjector +from dstack._internal.proxy.lib.metering import BaseProxyMeter from dstack._internal.proxy.lib.repo import BaseProxyRepo from dstack._internal.server.db import get_session_ctx from dstack._internal.server.services.proxy.auth import ServerProxyAuthProvider +from dstack._internal.server.services.proxy.metering import ServerProxyMeter from dstack._internal.server.services.proxy.repo import ServerProxyRepo @@ -16,3 +18,6 @@ async def get_repo(self) -> AsyncGenerator[BaseProxyRepo, None]: async def get_auth_provider(self) -> AsyncGenerator[BaseProxyAuthProvider, None]: async with get_session_ctx() as session: yield ServerProxyAuthProvider(session) + + def get_meter(self) -> BaseProxyMeter: + return ServerProxyMeter() diff --git a/src/dstack/_internal/server/services/proxy/metering.py b/src/dstack/_internal/server/services/proxy/metering.py new file mode 100644 index 0000000000..c60c0fcc90 --- /dev/null +++ b/src/dstack/_internal/server/services/proxy/metering.py @@ -0,0 +1,20 @@ +from typing import Optional + +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.metering import BaseProxyMeter +from dstack._internal.proxy.lib.models import Service + + +class ServerProxyMeter(BaseProxyMeter): + """Does nothing: tokens are metered on gateways only.""" + + def add( + self, + caller: ProxyCaller, + service: Service, + export_id: Optional[str], + input_tokens: int, + cached_input_tokens: int, + output_tokens: int, + ) -> None: + pass diff --git a/src/dstack/_internal/server/services/proxy/repo.py b/src/dstack/_internal/server/services/proxy/repo.py index 9551541973..1496ad503b 100644 --- a/src/dstack/_internal/server/services/proxy/repo.py +++ b/src/dstack/_internal/server/services/proxy/repo.py @@ -1,6 +1,5 @@ from typing import List, Optional -import pydantic from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import contains_eager, joinedload @@ -38,8 +37,6 @@ ) from dstack._internal.utils.common import get_or_error -_ANY_MODEL_ADAPTER = pydantic.TypeAdapter(AnyModel) - class ServerProxyRepo(BaseProxyRepo): """ @@ -160,10 +157,9 @@ async def list_models(self, project_name: str) -> List[ChatModel]: for run in res.scalars().all(): service_spec = validate_json_extra_ignore(ServiceSpec, get_or_error(run.service_spec)) model_spec = service_spec.model - model_options_obj = service_spec.options.get("openai", {}).get("model") - if model_spec is None or model_options_obj is None: + model_options = service_spec.get_model_options() + if model_spec is None or model_options is None: continue - model_options = _ANY_MODEL_ADAPTER.validate_python(model_options_obj) model = ChatModel( project_name=project_name, name=model_spec.name, diff --git a/src/dstack/_internal/server/services/proxy/services/service_proxy.py b/src/dstack/_internal/server/services/proxy/services/service_proxy.py index 0fa872be53..61d4e9378c 100644 --- a/src/dstack/_internal/server/services/proxy/services/service_proxy.py +++ b/src/dstack/_internal/server/services/proxy/services/service_proxy.py @@ -1,7 +1,7 @@ from typing import AsyncGenerator, AsyncIterator, Optional import fastapi -import httpx +import httpx2 as httpx from fastapi import status from starlette.requests import ClientDisconnect diff --git a/src/dstack/_internal/server/services/runs/__init__.py b/src/dstack/_internal/server/services/runs/__init__.py index f9033c1f5f..7d85bdce19 100644 --- a/src/dstack/_internal/server/services/runs/__init__.py +++ b/src/dstack/_internal/server/services/runs/__init__.py @@ -57,6 +57,7 @@ from dstack._internal.server.services import events, services from dstack._internal.server.services import projects as projects_services from dstack._internal.server.services import repos as repos_services +from dstack._internal.server.services.exported_models import check_can_update_service from dstack._internal.server.services.gateways import get_gateway_replica_models from dstack._internal.server.services.jobs import ( check_can_attach_job_volumes, @@ -641,12 +642,20 @@ async def apply_plan( and current_resource.run_spec.configuration.type == "service" and run_spec.configuration.gateway != current_resource.run_spec.configuration.gateway ): + # Load the gateway to compare where the service runs before and after the update + await session.refresh(current_resource_model, attribute_names=["gateway"]) + previous_gateway = current_resource_model.gateway await services.assign_service( session=session, run_model=current_resource_model, run_spec=run_spec, is_new_service_submission=False, ) + await check_can_update_service( + session=session, + run_model=current_resource_model, + previous_gateway=previous_gateway, + ) new_deployment_num = current_resource.deployment_num + 1 # FIXME: potentially long write transaction # Avoid getting run_model after update diff --git a/src/dstack/_internal/server/services/services/__init__.py b/src/dstack/_internal/server/services/services/__init__.py index 9b5868d8c4..d40b8cf1e3 100644 --- a/src/dstack/_internal/server/services/services/__init__.py +++ b/src/dstack/_internal/server/services/services/__init__.py @@ -2,6 +2,8 @@ Application logic related to `type: service` runs. """ +from urllib.parse import urlparse + from sqlalchemy.ext.asyncio import AsyncSession from dstack._internal.core.errors import ( @@ -27,6 +29,7 @@ get_project_default_gateway_model, get_project_gateway_model_by_reference, ) +from dstack._internal.server.services.gateways.client import get_gateway_endpoint_domain from dstack._internal.server.services.services.options import get_service_options from dstack._internal.utils.common import interpolate_gateway_domain from dstack._internal.utils.logging import get_logger @@ -139,12 +142,18 @@ async def _assign_service_to_gateway( run_project_name=run_model.project.name, exception_type=GatewayError, ) - service_url = f"{service_protocol}://{run_model.run_name}.{wildcard_domain}" - if isinstance(run_spec.configuration.model, OpenAIChatModel): - model_url = service_url + run_spec.configuration.model.prefix - else: - gateway_protocol = "https" if gateway_configuration.certificate is not None else "http" - model_url = f"{gateway_protocol}://gateway.{wildcard_domain}" + service_domain = f"{run_model.run_name}.{wildcard_domain}" + if service_domain == get_gateway_endpoint_domain(service_domain): + raise ServerClientError( + f"Domain name {service_domain!r} is taken by the gateway's model endpoint." + " Please use a different run name" + ) + service_url = f"{service_protocol}://{service_domain}" + model_url = get_gateway_endpoint_url( + service_url=service_url, + configuration=gateway_configuration, + project_name=run_model.project.name, + ) service_spec = _get_service_spec( configuration=run_spec.configuration, service_url=service_url, @@ -218,6 +227,19 @@ def _get_service_spec( return service_spec +def get_gateway_endpoint_url( + service_url: str, configuration: GatewayConfiguration, project_name: str +) -> str: + """ + Returns the URL where the project accesses models + on the gateway that publishes the service at `service_url`. + """ + service_domain = urlparse(service_url).hostname + assert service_domain is not None + protocol = "https" if configuration.certificate is not None else "http" + return f"{protocol}://{get_gateway_endpoint_domain(service_domain)}/{project_name}" + + def should_configure_service_https_on_gateway_replica( run_spec: RunSpec, configuration: GatewayConfiguration ) -> bool: diff --git a/src/dstack/_internal/server/testing/common.py b/src/dstack/_internal/server/testing/common.py index 73b27b70a1..e1e7c2eff9 100644 --- a/src/dstack/_internal/server/testing/common.py +++ b/src/dstack/_internal/server/testing/common.py @@ -37,6 +37,7 @@ from dstack._internal.core.models.configurations import ( AnyRunConfiguration, DevEnvironmentConfiguration, + ServiceConfiguration, ) from dstack._internal.core.models.duration import OptionalIdleDuration from dstack._internal.core.models.envs import Env @@ -96,7 +97,9 @@ RunSpec, RunStatus, RunTerminationReason, + ServiceSpec, ) +from dstack._internal.core.models.services import AnyModel from dstack._internal.core.models.users import GlobalRole from dstack._internal.core.models.volumes import ( AnyVolumeConfiguration, @@ -115,6 +118,7 @@ EventModel, ExportedFleetModel, ExportedGatewayModel, + ExportedModelModel, ExportModel, FileArchiveModel, FleetModel, @@ -145,7 +149,9 @@ set_default_permissions, ) from dstack._internal.server.services.runner.client import RunnerJobSubmissionMismatchError +from dstack._internal.server.services.services.options import get_service_options from dstack._internal.server.services.users import get_token_hash +from dstack._internal.utils.common import get_or_error, interpolate_gateway_domain def mock_lease(value: Any) -> Mock: @@ -442,6 +448,47 @@ async def create_run( return run +async def create_service_run( + session: AsyncSession, + project: ProjectModel, + repo: RepoModel, + user: UserModel, + gateway: Optional[GatewayModel] = None, + run_name: str = "test-service", + model: Optional[AnyModel] = None, + status: RunStatus = RunStatus.RUNNING, +) -> RunModel: + """ + Creates a run of a service that runs on `gateway` or, without it, on the in-server proxy. + Fills `service_spec` the way submitting the run does, with the options of `model`. + """ + configuration = ServiceConfiguration(commands=["serve"], port=8000, model=model) + run = await create_run( + session=session, + project=project, + repo=repo, + user=user, + gateway=gateway, + run_name=run_name, + status=status, + run_spec=get_run_spec(run_name=run_name, repo_id=repo.name, configuration=configuration), + ) + if gateway is None: + service_url = f"/proxy/services/{project.name}/{run_name}/" + else: + domain = interpolate_gateway_domain( + domain=get_or_error(gateway.wildcard_domain), + run_project_name=project.name, + exception_type=None, + ) + service_url = f"https://{run_name}.{domain}" + run.service_spec = ServiceSpec( + url=service_url, options=get_service_options(configuration) + ).model_dump_json() + await session.commit() + return run + + async def create_job( session: AsyncSession, run: RunModel, @@ -640,6 +687,7 @@ async def create_export( exported_gateways: Optional[list[GatewayModel]] = None, name: str = "test-export", is_global: bool = False, + exported_models: Optional[list[str]] = None, ) -> ExportModel: export = ExportModel( name=name, @@ -650,6 +698,9 @@ async def create_export( exported_gateways=[ ExportedGatewayModel(gateway=gateway) for gateway in (exported_gateways or []) ], + exported_models=[ + ExportedModelModel(name=model_name) for model_name in (exported_models or []) + ], ) session.add(export) await session.commit() diff --git a/src/dstack/_internal/utils/common.py b/src/dstack/_internal/utils/common.py index 205323bfa7..c2db0d9ef3 100644 --- a/src/dstack/_internal/utils/common.py +++ b/src/dstack/_internal/utils/common.py @@ -386,5 +386,15 @@ def interpolate_gateway_domain( raise exception_type(f"Cannot interpolate gateway domain name: {e.args[0]}") from e +def is_gateway_domain_fixed(domain: str) -> bool: + """ + Returns `True` if the gateway domain is the same for all projects, + that is, it has no `${{ run.project_name }}`. + """ + return domain == interpolate_gateway_domain( + domain=domain, run_project_name="", exception_type=None + ) + + def list_enum_values_for_annotation(enum_class: type[enum.Enum]) -> str: return ", ".join(f"`{e.value}`" for e in enum_class) diff --git a/src/dstack/api/server/_exports.py b/src/dstack/api/server/_exports.py index 1e01c670c8..f4077d9379 100644 --- a/src/dstack/api/server/_exports.py +++ b/src/dstack/api/server/_exports.py @@ -42,6 +42,7 @@ def create( importer_projects: List[str] = [], exported_fleets: List[str] = [], exported_gateways: List[str] = [], + exported_models: List[str] = [], ) -> Export: body = CreateExportRequest( name=name, @@ -49,6 +50,7 @@ def create( importer_projects=importer_projects, exported_fleets=exported_fleets, exported_gateways=exported_gateways, + exported_models=exported_models, ) resp = self._request( f"/api/project/{project_name}/exports/create", @@ -69,6 +71,8 @@ def update( remove_exported_fleets: List[str] = [], add_exported_gateways: List[str] = [], remove_exported_gateways: List[str] = [], + add_exported_models: List[str] = [], + remove_exported_models: List[str] = [], ) -> Export: body = UpdateExportRequest( name=name, @@ -80,6 +84,8 @@ def update( remove_exported_fleets=remove_exported_fleets, add_exported_gateways=add_exported_gateways, remove_exported_gateways=remove_exported_gateways, + add_exported_models=add_exported_models, + remove_exported_models=remove_exported_models, ) resp = self._request( f"/api/project/{project_name}/exports/update", diff --git a/src/tests/_internal/proxy/gateway/repo/test_repo.py b/src/tests/_internal/proxy/gateway/repo/test_repo.py index 3db014b52c..a6c11fbe57 100644 --- a/src/tests/_internal/proxy/gateway/repo/test_repo.py +++ b/src/tests/_internal/proxy/gateway/repo/test_repo.py @@ -3,10 +3,26 @@ import pytest from pydantic import AnyHttpUrl -from dstack._internal.proxy.gateway.models import ACMESettings, GlobalProxyConfig, ModelEntrypoint +from dstack._internal.proxy.gateway.models import ( + ACMESettings, + GlobalProxyConfig, + ImportedModel, + ModelEntrypoint, +) from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo from dstack._internal.proxy.lib.testing.common import make_project, make_service -from tests._internal.proxy.lib.routers.test_model_proxy import make_model +from tests._internal.proxy.lib.routers.test_model_proxy import TGI_FORMAT, make_model + + +def make_import( + export_id: str = "export-1", exporter_project: str = "exporter-proj" +) -> ImportedModel: + return ImportedModel( + importer_project="importer-proj", + export_id=export_id, + exporter_project=exporter_project, + name="test-model", + ) @pytest.mark.asyncio @@ -16,6 +32,7 @@ async def test_persist_repo(tmp_path: Path) -> None: srv_1 = make_service("proj-1", "run-1", domain="run-1.gtw.test") srv_2 = make_service("proj-2", "run-2", domain="run-2.gtw.test") model_1 = make_model("proj-1", "model-1", run_name="run-1") + import_1 = make_import() entrypoint_1 = ModelEntrypoint(project_name="proj-1", domain="gateway.gtw.test", https=True) config = GlobalProxyConfig(acme_settings=ACMESettings(server=AnyHttpUrl("https://acme.test"))) file = tmp_path / "state-v2.json" @@ -28,6 +45,7 @@ async def test_persist_repo(tmp_path: Path) -> None: await repo.set_service(srv_1) await repo.set_service(srv_2) await repo.set_model(model_1) + await repo.set_import(import_1) repo = GatewayProxyRepo.load(file) # reload from file assert await repo.get_config() == config @@ -39,6 +57,86 @@ async def test_persist_repo(tmp_path: Path) -> None: srv_2, ] assert await repo.list_models("proj-1") == [model_1] + assert await repo.list_imports() == [import_1] + + +@pytest.mark.asyncio +class TestImports: + async def test_importer_project_gets_model_of_exporter_project(self) -> None: + repo = GatewayProxyRepo() + model = make_model("exporter-proj", "test-model", run_name="run-1") + await repo.set_model(model) + await repo.set_import(make_import()) + + imported_model = model.model_copy(update={"export_id": "export-1"}) + assert await repo.list_models("importer-proj") == [imported_model] + assert await repo.list_models("Importer-Proj") == [imported_model] + assert await repo.get_model("Importer-Proj", "test-model") == imported_model + assert await repo.get_model("exporter-proj", "test-model") == model + assert await repo.list_models("other-proj") == [] + + await repo.delete_import("importer-proj", "export-1", "test-model") + assert await repo.list_models("importer-proj") == [] + assert await repo.get_model("importer-proj", "test-model") is None + + async def test_import_follows_model_to_another_service(self) -> None: + repo = GatewayProxyRepo() + await repo.set_import(make_import()) + # No service of the exporter project has registered the model yet. + assert await repo.list_models("importer-proj") == [] + assert await repo.get_model("importer-proj", "test-model") is None + + await repo.set_model(make_model("exporter-proj", "test-model", run_name="run-1")) + model = await repo.get_model("importer-proj", "test-model") + assert model is not None + assert model.run_name == "run-1" + + # Another service registers the model, and the first service is unregistered. + await repo.set_model(make_model("exporter-proj", "test-model", run_name="run-2")) + await repo.delete_models_by_run("exporter-proj", "run-1") + model = await repo.get_model("importer-proj", "test-model") + assert model is not None + assert (model.project_name, model.run_name) == ("exporter-proj", "run-2") + assert model.export_id == "export-1" + + # Requests to models with the `tgi` format are not metered. + await repo.set_model( + make_model("exporter-proj", "test-model", run_name="run-3", format_spec=TGI_FORMAT) + ) + assert await repo.list_models("importer-proj") == [] + assert await repo.get_model("importer-proj", "test-model") is None + + async def test_own_model_wins_over_import_with_same_name(self) -> None: + repo = GatewayProxyRepo() + own_model = make_model("importer-proj", "test-model", run_name="own-run") + await repo.set_model(own_model) + await repo.set_model(make_model("exporter-proj", "test-model", run_name="run-1")) + await repo.set_import(make_import()) + assert await repo.list_models("importer-proj") == [own_model] + assert await repo.get_model("importer-proj", "test-model") == own_model + + await repo.delete_models_by_run("importer-proj", "own-run") + [model] = await repo.list_models("importer-proj") + assert model.export_id == "export-1" + + async def test_keeps_one_import_per_project_and_model_name(self) -> None: + repo = GatewayProxyRepo() + await repo.set_model(make_model("exporter-proj", "test-model", run_name="run-1")) + await repo.set_model(make_model("exporter-proj-2", "test-model", run_name="run-2")) + await repo.set_import(make_import("export-1", "exporter-proj")) + # The import of the model through another export replaces it. + new_import = make_import("export-2", "exporter-proj-2") + await repo.set_import(new_import) + assert await repo.list_imports() == [new_import] + model = await repo.get_model("importer-proj", "test-model") + assert model is not None + assert (model.project_name, model.export_id) == ("exporter-proj-2", "export-2") + + # Deleting the import through the previous export changes nothing. + await repo.delete_import("importer-proj", "export-1", "test-model") + assert await repo.list_imports() == [new_import] + await repo.delete_import("importer-proj", "export-2", "test-model") + assert await repo.list_imports() == [] @pytest.mark.asyncio diff --git a/src/tests/_internal/proxy/gateway/routers/test_metering.py b/src/tests/_internal/proxy/gateway/routers/test_metering.py new file mode 100644 index 0000000000..ffea2ed1b9 --- /dev/null +++ b/src/tests/_internal/proxy/gateway/routers/test_metering.py @@ -0,0 +1,57 @@ +import uuid +from datetime import datetime, timezone +from pathlib import Path + +import httpx +import pytest +import time_machine + +from dstack._internal.proxy.gateway.app import make_app +from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo +from dstack._internal.proxy.gateway.services.metering import GatewayProxyMeter +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.testing.common import make_service + + +@pytest.mark.asyncio +async def test_collect_metering(tmp_path: Path): + caller = ProxyCaller(project_id=uuid.uuid4(), user_id=uuid.uuid4()) + service = make_service("test-proj", "test-run") + meter = GatewayProxyMeter(tmp_path) + with time_machine.travel(datetime(2026, 11, 20, 10, 59, tzinfo=timezone.utc), tick=False): + for input_tokens in (800, 3): + meter.add( + caller, + service, + None, + input_tokens=input_tokens, + cached_input_tokens=0, + output_tokens=1, + ) + await meter.flush() + app = make_app(repo=GatewayProxyRepo(), meter=meter) + client = httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url="http://test/") + + resp = await client.post("/api/metering/collect", json={"after": 1, "limit": 10}) + + assert resp.status_code == 200 + assert resp.json() == { + "batches": [ + { + "batch_no": 2, + "created_at": "2026-11-20T10:59:00Z", + "records": [ + { + "hour": "2026-11-20T10:00:00Z", + "project_id": str(caller.project_id), + "user_id": str(caller.user_id), + "run_id": str(uuid.UUID(service.id)), + "export_id": None, + "input_tokens": 3, + "cached_input_tokens": 0, + "output_tokens": 1, + } + ], + } + ] + } diff --git a/src/tests/_internal/proxy/gateway/routers/test_registry.py b/src/tests/_internal/proxy/gateway/routers/test_registry.py index 854c26062a..a5081596ae 100644 --- a/src/tests/_internal/proxy/gateway/routers/test_registry.py +++ b/src/tests/_internal/proxy/gateway/routers/test_registry.py @@ -804,7 +804,7 @@ async def test_register(self, tmp_path: Path, system_mocks: Mocks) -> None: assert resp.status_code == 200 assert resp.json() == {"status": "ok"} conf = (tmp_path / "sites-enabled" / "443-gateway.gtw.test.conf").read_text() - assert "proxy_pass http://localhost:8000/api/models/test-proj/;" in conf + assert "proxy_pass http://localhost:8000/api/models/;" in conf assert "listen 80;" in conf assert "listen 443" not in conf assert system_mocks.reload_nginx.call_count == 1 @@ -831,8 +831,56 @@ async def test_register_with_https(self, tmp_path: Path, system_mocks: Mocks) -> ) assert resp.status_code == 200 conf = (tmp_path / "sites-enabled" / "443-gateway.gtw.test.conf").read_text() - assert "proxy_pass http://localhost:8000/api/models/test-proj/;" in conf + assert "proxy_pass http://localhost:8000/api/models/;" in conf assert "listen 80;" in conf assert "listen 443 ssl;" in conf assert system_mocks.reload_nginx.call_count == 1 assert system_mocks.run_certbot.call_count == 1 + + +@pytest.mark.asyncio +class TestImports: + async def test_register_list_unregister(self, tmp_path: Path) -> None: + client = make_client(tmp_path) + export_id = uuid.uuid4().hex + assert (await client.get("/api/registry/imports/list")).json() == {"imports": []} + for _ in range(2): # registering an import again changes nothing + resp = await client.post( + "/api/registry/Importer-Proj/imports/register", + json={ + "export_id": export_id, + "exporter_project": "Exporter-Proj", + "name": "Test/Model", + }, + ) + assert resp.status_code == 200 + assert resp.json() == {"status": "ok"} + imports = { + "imports": [ + { + "importer_project": "importer-proj", + "export_id": export_id, + "exporter_project": "exporter-proj", + "name": "Test/Model", + } + ] + } + resp = await client.get("/api/registry/imports/list") + assert resp.status_code == 200 + assert resp.json() == imports + + # The project imports the model through this export, not through the given one. + resp = await client.post( + "/api/registry/Importer-Proj/imports/unregister", + json={"export_id": uuid.uuid4().hex, "name": "Test/Model"}, + ) + assert resp.status_code == 200 + assert (await client.get("/api/registry/imports/list")).json() == imports + + resp = await client.post( + "/api/registry/Importer-Proj/imports/unregister", + json={"export_id": export_id, "name": "Test/Model"}, + ) + assert resp.status_code == 200 + assert resp.json() == {"status": "ok"} + assert (await client.get("/api/registry/imports/list")).json() == {"imports": []} diff --git a/src/tests/_internal/proxy/gateway/services/test_metering.py b/src/tests/_internal/proxy/gateway/services/test_metering.py new file mode 100644 index 0000000000..e798ffdad5 --- /dev/null +++ b/src/tests/_internal/proxy/gateway/services/test_metering.py @@ -0,0 +1,185 @@ +import json +import os +import uuid +from datetime import datetime, timezone +from pathlib import Path + +import pytest +import time_machine + +from dstack._internal.proxy.gateway.services.metering import GatewayProxyMeter +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.testing.common import make_service + +CALLER = ProxyCaller(project_id=uuid.uuid4(), user_id=uuid.uuid4()) +SERVICE = make_service("test-proj", "test-run") + + +def add(meter: GatewayProxyMeter, input_tokens: int = 1) -> None: + meter.add( + CALLER, SERVICE, None, input_tokens=input_tokens, cached_input_tokens=0, output_tokens=0 + ) + + +@pytest.mark.asyncio +class TestGatewayProxyMeter: + async def test_writes_one_record_per_hour_caller_and_service(self, tmp_path: Path): + meter = GatewayProxyMeter(tmp_path) + other_caller = ProxyCaller(project_id=CALLER.project_id, user_id=uuid.uuid4()) + with time_machine.travel(datetime(2026, 11, 20, 10, 59, tzinfo=timezone.utc), tick=False): + meter.add( + CALLER, SERVICE, None, input_tokens=800, cached_input_tokens=700, output_tokens=5 + ) + meter.add( + CALLER, SERVICE, None, input_tokens=0, cached_input_tokens=0, output_tokens=10 + ) + meter.add( + other_caller, SERVICE, None, input_tokens=3, cached_input_tokens=0, output_tokens=1 + ) + with time_machine.travel( + datetime(2026, 11, 20, 11, 0, 5, tzinfo=timezone.utc), tick=False + ): + # The rest of a request that runs past the turn of the hour + meter.add( + CALLER, SERVICE, None, input_tokens=0, cached_input_tokens=0, output_tokens=7 + ) + await meter.flush() + + assert json.loads((tmp_path / "1.json").read_text()) == { + "batch_no": 1, + "created_at": "2026-11-20T11:00:05Z", + "records": [ + { + "hour": "2026-11-20T10:00:00Z", + "project_id": str(CALLER.project_id), + "user_id": str(CALLER.user_id), + "run_id": str(uuid.UUID(SERVICE.id)), + "export_id": None, + "input_tokens": 800, + "cached_input_tokens": 700, + "output_tokens": 15, + }, + { + "hour": "2026-11-20T10:00:00Z", + "project_id": str(CALLER.project_id), + "user_id": str(other_caller.user_id), + "run_id": str(uuid.UUID(SERVICE.id)), + "export_id": None, + "input_tokens": 3, + "cached_input_tokens": 0, + "output_tokens": 1, + }, + { + "hour": "2026-11-20T11:00:00Z", + "project_id": str(CALLER.project_id), + "user_id": str(CALLER.user_id), + "run_id": str(uuid.UUID(SERVICE.id)), + "export_id": None, + "input_tokens": 0, + "cached_input_tokens": 0, + "output_tokens": 7, + }, + ], + } + + async def test_writes_requests_through_an_export_to_a_record_with_the_export( + self, tmp_path: Path + ): + meter = GatewayProxyMeter(tmp_path) + export_id = uuid.uuid4() + add(meter, input_tokens=1) + for input_tokens in (2, 3): + meter.add( + CALLER, + SERVICE, + export_id.hex, + input_tokens=input_tokens, + cached_input_tokens=0, + output_tokens=0, + ) + await meter.flush() + + records = json.loads((tmp_path / "1.json").read_text())["records"] + assert [(record["export_id"], record["input_tokens"]) for record in records] == [ + (None, 1), + (str(export_id), 5), + ] + (batch,) = await meter.collect(after=0, limit=1) + assert [record.export_id for record in batch.records] == [None, export_id] + + async def test_collects_batches_after_the_given_one_and_deletes_the_saved( + self, tmp_path: Path + ): + meter = GatewayProxyMeter(tmp_path) + for input_tokens in (1, 2, 3, 4): + add(meter, input_tokens) + await meter.flush() + + batches = await meter.collect(after=1, limit=2) + assert [batch.batch_no for batch in batches] == [2, 3] + assert [batch.records[0].input_tokens for batch in batches] == [2, 3] + # The batches after the given one stay until the server confirms them. + assert sorted(path.name for path in tmp_path.glob("*.json")) == [ + "2.json", + "3.json", + "4.json", + ] + + assert [batch.batch_no for batch in await meter.collect(after=3, limit=2)] == [4] + assert await meter.collect(after=4, limit=2) == [] + assert list(tmp_path.glob("*.json")) == [] + + async def test_numbers_new_batches_above_the_ones_the_server_saved(self, tmp_path: Path): + # The gateway's files were lost, so it starts from 1, while the server has saved 41. + meter = GatewayProxyMeter(tmp_path) + add(meter) + await meter.flush() + assert await meter.collect(after=41, limit=100) == [] + add(meter) + await meter.flush() + assert [batch.batch_no for batch in await meter.collect(after=41, limit=100)] == [42] + + async def test_continues_numbering_after_restart(self, tmp_path: Path): + meter = GatewayProxyMeter(tmp_path) + for _ in range(2): + add(meter) + await meter.flush() + # All batches are collected and deleted before the restart. + assert await meter.collect(after=2, limit=100) == [] + + meter = GatewayProxyMeter(tmp_path) + add(meter) + await meter.flush() + assert [batch.batch_no for batch in await meter.collect(after=2, limit=100)] == [3] + + # The number is also taken from the batch files, if they are ahead of the stored one. + (tmp_path / "last_batch_no").unlink() + meter = GatewayProxyMeter(tmp_path) + add(meter) + await meter.flush() + assert [batch.batch_no for batch in await meter.collect(after=2, limit=100)] == [3, 4] + + async def test_deletes_batches_older_than_seven_days(self, tmp_path: Path): + meter = GatewayProxyMeter(tmp_path) + for _ in range(2): + add(meter) + await meter.flush() + eight_days_ago = datetime.now(tz=timezone.utc).timestamp() - 8 * 24 * 3600 + os.utime(tmp_path / "1.json", (eight_days_ago, eight_days_ago)) + + # A restarted gateway deletes old batches on its first flush. + meter = GatewayProxyMeter(tmp_path) + await meter.flush() + assert [path.name for path in tmp_path.glob("*.json")] == ["2.json"] + + async def test_keeps_counts_if_the_batch_cannot_be_written(self, tmp_path: Path): + meter = GatewayProxyMeter(tmp_path / "metering") + (tmp_path / "metering").write_text("not a directory") + add(meter, input_tokens=5) + with pytest.raises(OSError): + await meter.flush() + (tmp_path / "metering").unlink() + add(meter, input_tokens=2) + await meter.flush() + (batch,) = await meter.collect(after=0, limit=100) + assert batch.records[0].input_tokens == 7 diff --git a/src/tests/_internal/proxy/gateway/test_app.py b/src/tests/_internal/proxy/gateway/test_app.py index 335be84fa1..31daea0aaa 100644 --- a/src/tests/_internal/proxy/gateway/test_app.py +++ b/src/tests/_internal/proxy/gateway/test_app.py @@ -1,3 +1,4 @@ +import uuid from pathlib import Path import pytest @@ -5,8 +6,10 @@ from dstack._internal.proxy.gateway.app import lifespan, make_app from dstack._internal.proxy.gateway.models import ModelEntrypoint from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo +from dstack._internal.proxy.gateway.services.metering import GatewayProxyMeter from dstack._internal.proxy.gateway.services.nginx import Nginx from dstack._internal.proxy.gateway.testing.common import Mocks +from dstack._internal.proxy.lib.auth import ProxyCaller from dstack._internal.proxy.lib.testing.common import make_project, make_service @@ -32,3 +35,23 @@ async def test_lifespan(tmp_path: Path, system_mocks: Mocks) -> None: assert system_mocks.open_conn.call_count == 1 assert system_mocks.close_conn.call_count == 0 assert system_mocks.close_conn.call_count == 1 + + +@pytest.mark.asyncio +async def test_lifespan_writes_metered_tokens_on_shutdown( + tmp_path: Path, system_mocks: Mocks +) -> None: + nginx_dir = tmp_path / "nginx" + (nginx_dir / "sites-enabled").mkdir(parents=True) + meter = GatewayProxyMeter(tmp_path / "metering") + app = make_app(repo=GatewayProxyRepo(), nginx=Nginx(nginx_dir=nginx_dir), meter=meter) + async with lifespan(app): + meter.add( + ProxyCaller(project_id=uuid.uuid4(), user_id=uuid.uuid4()), + make_service("test-proj", "test-run"), + None, + input_tokens=9, + cached_input_tokens=0, + output_tokens=1, + ) + assert (tmp_path / "metering" / "1.json").exists() diff --git a/src/tests/_internal/proxy/gateway/test_auth.py b/src/tests/_internal/proxy/gateway/test_auth.py index 3b96306aa4..738d2da719 100644 --- a/src/tests/_internal/proxy/gateway/test_auth.py +++ b/src/tests/_internal/proxy/gateway/test_auth.py @@ -1,3 +1,5 @@ +import uuid +from typing import Optional from unittest.mock import AsyncMock import httpx2 as httpx @@ -5,28 +7,65 @@ import pytest_asyncio from aiocache.base import BaseCache +from dstack._internal.core.models.projects import Member, MemberPermissions, Project +from dstack._internal.core.models.users import GlobalRole, ProjectRole, User, UserPermissions from dstack._internal.proxy.gateway.auth import GatewayProxyAuthProvider from dstack._internal.proxy.gateway.services.server_client import HTTPMultiClient -from dstack._internal.proxy.lib.errors import UnexpectedProxyError +from dstack._internal.proxy.lib.auth import ProxyCaller +from dstack._internal.proxy.lib.errors import ProxyError, UnexpectedProxyError @pytest_asyncio.fixture(autouse=True) async def clear_auth_cache(): - cache = _get_auth_cache() - await cache.clear() + caches = [ + _get_auth_cache(), + _get_cache(GatewayProxyAuthProvider._get_user_id), + _get_cache(GatewayProxyAuthProvider._is_global_admin), + ] + for cache in caches: + await cache.clear() yield - await cache.clear() + for cache in caches: + await cache.clear() + + +def make_user(global_role: GlobalRole = GlobalRole.USER) -> User: + return User( + id=uuid.uuid4(), + username="test-user", + global_role=global_role, + active=True, + permissions=UserPermissions(can_create_projects=True), + ) + + +def make_project(members: Optional[list[User]] = None) -> Project: + owner = make_user() + return Project( + project_id=uuid.uuid4(), + project_name="test", + owner=owner, + backends=[], + members=[ + Member( + user=user, + project_role=ProjectRole.USER, + permissions=MemberPermissions(can_manage_ssh_fleets=True), + ) + for user in [owner, *(members or [])] + ], + ) @pytest.mark.asyncio class TestGatewayProxyAuthProvider: - @pytest.mark.parametrize("status", [200, 403]) + @pytest.mark.parametrize("status", [200, 403, 404]) async def test_caches_project_token_decision_for_sixty_seconds(self, monkeypatch, status): requests = [] def handle(request): requests.append(request) - return httpx.Response(status) + return httpx.Response(status, content=make_project().model_dump_json()) cache = _get_auth_cache() set_value = AsyncMock(wraps=cache.set) @@ -56,7 +95,7 @@ def handle(request): nonlocal requests requests += 1 if requests > 1: - return httpx.Response(200) + return httpx.Response(200, content=make_project().model_dump_json()) if isinstance(failure, int): return httpx.Response(failure) raise failure("temporary failure", request=request) @@ -79,7 +118,132 @@ async def test_no_servers_reaches_auth_error_boundary(self, tmp_path): assert isinstance(exc.value.__cause__, httpx.RequestError) assert exc.value.__cause__.request.url.path == "/api/projects/test/get" + @pytest.mark.parametrize( + ("global_role", "is_member", "is_allowed"), + [ + pytest.param(GlobalRole.USER, True, True, id="member"), + # The server lets any user into a public project. + pytest.param(GlobalRole.USER, False, False, id="non-member-of-public-project"), + pytest.param(GlobalRole.ADMIN, False, True, id="global-admin"), + ], + ) + async def test_get_caller_allows_members_and_global_admins( + self, global_role, is_member, is_allowed + ): + user = make_user(global_role) + project = make_project(members=[user] if is_member else []) + + def handle(request): + if request.url.path == "/api/projects/test/get": + return httpx.Response(200, content=project.model_dump_json()) + assert request.url.path == "/api/users/get_my_user" + assert request.headers["Authorization"] == "Bearer token" + return httpx.Response(200, content=user.model_dump_json()) + + async with httpx.AsyncClient( + transport=httpx.MockTransport(handle), base_url="http://dstack/" + ) as client: + caller = await GatewayProxyAuthProvider(client).get_caller("test", "token") + expected = ProxyCaller(project_id=project.project_id, user_id=user.id) + assert caller == (expected if is_allowed else None) + + async def test_get_caller_rechecks_global_admin_role(self): + user = make_user(GlobalRole.ADMIN) + project = make_project() + + def handle(request): + if request.url.path == "/api/users/get_my_user": + return httpx.Response(200, content=user.model_dump_json()) + return httpx.Response(200, content=project.model_dump_json()) + + async with httpx.AsyncClient( + transport=httpx.MockTransport(handle), base_url="http://dstack/" + ) as client: + provider = GatewayProxyAuthProvider(client) + assert await provider.get_caller("test", "token") is not None + user.global_role = GlobalRole.USER + # The role is cached like access, not for as long as the user ID. + await _get_cache(GatewayProxyAuthProvider._is_global_admin).clear() + assert await provider.get_caller("test", "token") is None + + async def test_parses_only_fields_needed_for_access_checks(self): + # A newer server can return values that this gateway does not know. + project = make_project().model_dump(mode="json") + project["members"][0]["project_role"] = "new-role" + project["new_field"] = "new-value" + async with httpx.AsyncClient( + transport=httpx.MockTransport(lambda request: httpx.Response(200, json=project)), + base_url="http://dstack/", + ) as client: + assert await GatewayProxyAuthProvider(client).is_project_member("test", "token") + + async def test_get_caller_denies_if_server_denies(self): + async with httpx.AsyncClient( + transport=httpx.MockTransport(lambda request: httpx.Response(403)), + base_url="http://dstack/", + ) as client: + assert await GatewayProxyAuthProvider(client).get_caller("test", "token") is None + + async def test_get_caller_raises_not_found_if_project_does_not_exist(self): + requests = 0 + + def handle(request): + nonlocal requests + requests += 1 + return httpx.Response(404) + + async with httpx.AsyncClient( + transport=httpx.MockTransport(handle), base_url="http://dstack/" + ) as client: + provider = GatewayProxyAuthProvider(client) + for _ in range(2): + with pytest.raises(ProxyError) as exc: + await provider.get_caller("missing", "token") + assert exc.value.status_code == 404 + assert requests == 1 + + async def test_quotes_project_name_in_server_request(self): + paths = [] + + def handle(request): + paths.append(request.url.raw_path) + return httpx.Response(404) + + async with httpx.AsyncClient( + transport=httpx.MockTransport(handle), base_url="http://dstack/" + ) as client: + with pytest.raises(ProxyError): + await GatewayProxyAuthProvider(client).get_caller("list?", "token") + assert paths == [b"/api/projects/list%3F/get"] + + async def test_get_caller_requests_user_once_per_token(self): + user = make_user() + project = make_project(members=[user]) + paths = [] + + def handle(request): + paths.append(request.url.path) + if request.url.path == "/api/users/get_my_user": + return httpx.Response(200, content=user.model_dump_json()) + return httpx.Response(200, content=project.model_dump_json()) + + async with httpx.AsyncClient( + transport=httpx.MockTransport(handle), base_url="http://dstack/" + ) as client: + provider = GatewayProxyAuthProvider(client) + for project_name in ("first", "second", "first"): + assert await provider.get_caller(project_name, "token") is not None + assert sorted(paths) == [ + "/api/projects/first/get", + "/api/projects/second/get", + "/api/users/get_my_user", + ] + def _get_auth_cache() -> BaseCache: + return _get_cache(GatewayProxyAuthProvider._get_project_access) + + +def _get_cache(cached_method) -> BaseCache: # `aiocache.cached` sets `cache` on the wrapper at runtime, but its type doesn't declare it - return GatewayProxyAuthProvider.is_project_member.cache # pyright: ignore[reportAttributeAccessIssue] + return cached_method.cache diff --git a/src/tests/_internal/proxy/lib/routers/test_model_proxy.py b/src/tests/_internal/proxy/lib/routers/test_model_proxy.py index a2dfa3257c..d322a9dd7e 100644 --- a/src/tests/_internal/proxy/lib/routers/test_model_proxy.py +++ b/src/tests/_internal/proxy/lib/routers/test_model_proxy.py @@ -1,15 +1,24 @@ +import asyncio +import json from datetime import datetime -from typing import AsyncIterator, Generator +from typing import Any, AsyncIterator, Callable, Generator, Optional from unittest.mock import patch import httpx2 as httpx import openai import pytest from fastapi import FastAPI +from starlette.types import Message, Scope +from dstack._internal.proxy.gateway.models import ImportedModel from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo from dstack._internal.proxy.lib.auth import BaseProxyAuthProvider -from dstack._internal.proxy.lib.models import ChatModel, OpenAIChatModelFormat +from dstack._internal.proxy.lib.models import ( + AnyModelFormat, + ChatModel, + OpenAIChatModelFormat, + TGIChatModelFormat, +) from dstack._internal.proxy.lib.repo import BaseProxyRepo from dstack._internal.proxy.lib.routers.model_proxy import router from dstack._internal.proxy.lib.schemas.model_proxy import ( @@ -22,12 +31,13 @@ ChatMessage, ) from dstack._internal.proxy.lib.services.model_proxy.clients.base import ChatCompletionsClient -from dstack._internal.proxy.lib.testing.auth import ProxyTestAuthProvider +from dstack._internal.proxy.lib.testing.auth import ProxyTestAuthProvider, make_caller from dstack._internal.proxy.lib.testing.common import ( ProxyTestDependencyInjector, make_project, make_service, ) +from dstack._internal.proxy.lib.testing.metering import MeteredTokens, ProxyTestMeter SAMPLE_RESPONSE = "Hello there, how may I assist you today?" @@ -76,23 +86,40 @@ async def stream(self, request: ChatCompletionsRequest) -> AsyncIterator[ChatCom ) +TGI_FORMAT = TGIChatModelFormat(chat_template="", eos_token="") + + def make_model( - project_name: str, name: str, run_name: str, created_at: datetime = datetime.fromtimestamp(0) + project_name: str, + name: str, + run_name: str, + created_at: datetime = datetime.fromtimestamp(0), + format_spec: AnyModelFormat = OpenAIChatModelFormat(format="openai", prefix="/v1"), ) -> ChatModel: return ChatModel( project_name=project_name, name=name, created_at=created_at, run_name=run_name, - format_spec=OpenAIChatModelFormat(format="openai", prefix="/v1"), + format_spec=format_spec, ) -def make_http_client(repo: BaseProxyRepo, auth: BaseProxyAuthProvider) -> httpx.AsyncClient: +def make_app( + repo: BaseProxyRepo, auth: BaseProxyAuthProvider, meter: Optional[ProxyTestMeter] = None +) -> FastAPI: app = FastAPI() - app.state.proxy_dependency_injector = ProxyTestDependencyInjector(repo=repo, auth=auth) + app.state.proxy_dependency_injector = ProxyTestDependencyInjector( + repo=repo, auth=auth, meter=meter + ) app.include_router(router, prefix="/proxy/models") - return httpx.AsyncClient(transport=httpx.ASGITransport(app=app)) + return app + + +def make_http_client( + repo: BaseProxyRepo, auth: BaseProxyAuthProvider, meter: Optional[ProxyTestMeter] = None +) -> httpx.AsyncClient: + return httpx.AsyncClient(transport=httpx.ASGITransport(app=make_app(repo, auth, meter))) def make_openai_client( @@ -170,7 +197,9 @@ async def test_chat_completions(mock_chat_client) -> None: repo = GatewayProxyRepo() await repo.set_project(make_project("test-proj")) await repo.set_service(make_service("test-proj", "test-service")) - await repo.set_model(make_model("test-proj", "test-model", "test-service")) + await repo.set_model( + make_model("test-proj", "test-model", "test-service", format_spec=TGI_FORMAT) + ) client = make_openai_client(repo, auth, "test-proj", auth_token="token") completion = await client.chat.completions.create( model="test-model", @@ -179,13 +208,32 @@ async def test_chat_completions(mock_chat_client) -> None: assert completion.choices[0].message.content == SAMPLE_RESPONSE +@pytest.mark.asyncio +async def test_chat_completions_with_project_name_in_another_case(mock_chat_client) -> None: + auth = ProxyTestAuthProvider({"Test-Proj": {"token"}}) + repo = GatewayProxyRepo() + await repo.set_project(make_project("test-proj")) + await repo.set_service(make_service("test-proj", "test-service")) + await repo.set_model( + make_model("test-proj", "test-model", "test-service", format_spec=TGI_FORMAT) + ) + client = make_openai_client(repo, auth, "Test-Proj", auth_token="token") + completion = await client.chat.completions.create( + model="test-model", + messages=[{"role": "user", "content": "Hi"}], + ) + assert completion.choices[0].message.content == SAMPLE_RESPONSE + + @pytest.mark.asyncio async def test_chat_completions_stream(mock_chat_client) -> None: auth = ProxyTestAuthProvider({"test-proj": {"token"}}) repo = GatewayProxyRepo() await repo.set_project(make_project("test-proj")) await repo.set_service(make_service("test-proj", "test-service")) - await repo.set_model(make_model("test-proj", "test-model", "test-service")) + await repo.set_model( + make_model("test-proj", "test-model", "test-service", format_spec=TGI_FORMAT) + ) client = make_openai_client(repo, auth, "test-proj", auth_token="token") response = await client.chat.completions.create( model="test-model", @@ -255,3 +303,388 @@ async def test_unauthorized_http(headers) -> None: headers=headers, ) assert resp.status_code == 403 + + +CHAT_COMPLETIONS_URL = "http://test-host/proxy/models/test-proj/chat/completions" +AUTH_HEADERS = {"Authorization": "Bearer token"} +ServiceHandler = Callable[[httpx.Request], Any] + + +@pytest.fixture +def mock_service() -> Generator[Callable[[ServiceHandler], None], None, None]: + """Routes the requests that the proxy sends to the service to the given handler.""" + handlers: list[ServiceHandler] = [] + + async def get_service_replica_client(*args, **kwargs) -> httpx.AsyncClient: + return httpx.AsyncClient( + transport=httpx.MockTransport(handlers[0]), base_url="http://test-service" + ) + + with patch( + "dstack._internal.proxy.lib.services.model_proxy.passthrough.get_service_replica_client", + get_service_replica_client, + ): + yield handlers.append + + +async def make_repo() -> GatewayProxyRepo: + repo = GatewayProxyRepo() + await repo.set_project(make_project("test-proj")) + await repo.set_service(make_service("test-proj", "test-service")) + await repo.set_model(make_model("test-proj", "test-model", "test-service")) + return repo + + +def sse(*chunks: dict, done: bool = True) -> bytes: + events = [json.dumps(chunk, separators=(",", ":")) for chunk in chunks] + if done: + events.append("[DONE]") + return "".join(f"data: {event}\n\n" for event in events).encode() + + +def parse_sse(content: bytes) -> list[Any]: + events = content.decode().removesuffix("\n\n").split("\n\n") + assert all(event.startswith("data: ") for event in events) + assert events[-1] == "data: [DONE]" + return [json.loads(event.removeprefix("data: ")) for event in events[:-1]] + + +def chunk(content: str, usage: Optional[dict] = None) -> dict: + result = {"id": "1", "choices": [{"index": 0, "delta": {"content": content}}]} + if usage is not None: + result["usage"] = usage + return result + + +def usage(prompt_tokens: int, completion_tokens: int, cached_tokens: int = 0) -> dict: + return { + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": prompt_tokens + completion_tokens, + "prompt_tokens_details": {"cached_tokens": cached_tokens}, + } + + +@pytest.mark.asyncio +class TestChatCompletionsPassthrough: + async def test_forwards_request_and_response_unchanged(self, mock_service) -> None: + request_body = { + "model": "test-model", + "max_completion_tokens": 100, + "messages": [ + {"role": "user", "content": [{"type": "text", "text": "Weather in Berlin?"}]}, + {"role": "assistant", "content": None, "tool_calls": [{"id": "call-1"}]}, + {"role": "tool", "tool_call_id": "call-1", "content": "Sunny"}, + ], + "tools": [{"type": "function", "function": {"name": "get_weather"}}], + } + # Unusual spacing shows that the response is not parsed and serialized again. + response_body = ( + b'{"id": "1", "choices": [{"message": {"content": "Sunny", "reasoning": "..."}}],' + b' "usage": {"prompt_tokens": 812, "completion_tokens": 140, "total_tokens": 952,' + b' "prompt_tokens_details": {"cached_tokens": 768}}}' + ) + requests = [] + + def handle(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, content=response_body, headers={"content-type": "application/json"} + ) + + mock_service(handle) + auth = ProxyTestAuthProvider({"test-proj": {"token"}}) + meter = ProxyTestMeter() + repo = await make_repo() + service = await repo.get_service("test-proj", "test-service") + assert service is not None + client = make_http_client(repo, auth, meter) + resp = await client.post(CHAT_COMPLETIONS_URL, json=request_body, headers=AUTH_HEADERS) + + assert resp.status_code == 200 + assert resp.content == response_body + assert resp.headers["content-type"] == "application/json" + assert requests[0].url.path == "/v1/chat/completions" + assert json.loads(requests[0].content) == request_body + assert meter.added == [ + MeteredTokens( + caller=make_caller("test-proj", "token"), + service=service, + export_id=None, + input_tokens=812, + cached_input_tokens=768, + output_tokens=140, + ) + ] + + async def test_passes_service_errors_through(self, mock_service) -> None: + error = b'{"error": {"message": "Too many requests"}}' + mock_service( + lambda request: httpx.Response(429, content=error, headers={"Retry-After": "7"}) + ) + auth = ProxyTestAuthProvider({"test-proj": {"token"}}) + meter = ProxyTestMeter() + client = make_http_client(await make_repo(), auth, meter) + resp = await client.post( + CHAT_COMPLETIONS_URL, + json={"model": "test-model", "messages": [], "stream": True}, + headers=AUTH_HEADERS, + ) + assert resp.status_code == 429 + assert resp.content == error + assert resp.headers["Retry-After"] == "7" + assert meter.added == [] + + async def test_does_not_meter_token_counts_that_are_not_real(self, mock_service) -> None: + response_body = b'{"choices": [], "usage": {"prompt_tokens": 9223372036854775808}}' + mock_service(lambda request: httpx.Response(200, content=response_body)) + auth = ProxyTestAuthProvider({"test-proj": {"token"}}) + meter = ProxyTestMeter() + client = make_http_client(await make_repo(), auth, meter) + resp = await client.post( + CHAT_COMPLETIONS_URL, + json={"model": "test-model", "messages": []}, + headers=AUTH_HEADERS, + ) + assert resp.status_code == 200 + assert resp.content == response_body + assert meter.added == [] + + @pytest.mark.parametrize( + ("stream_options", "expected_usage"), + [ + pytest.param(None, [None, None], id="client-set-nothing"), + pytest.param( + {"include_usage": True}, [None, None, usage(9, 2, 4)], id="include-usage" + ), + pytest.param( + {"include_usage": True, "continuous_usage_stats": True}, + [usage(9, 1), usage(9, 2), usage(9, 2, 4)], + id="include-and-continuous-usage", + ), + ], + ) + async def test_stream_has_only_usage_that_client_asked_for( + self, mock_service, stream_options: Optional[dict], expected_usage: list[Optional[dict]] + ) -> None: + requests = [] + + def handle(request: httpx.Request) -> httpx.Response: + requests.append(request) + return httpx.Response( + 200, + content=sse( + chunk("Hello", usage(9, 1)), + chunk(" there", usage(9, 2)), + {"id": "1", "choices": [], "usage": usage(9, 2, 4)}, + ), + headers={"content-type": "text/event-stream"}, + ) + + mock_service(handle) + auth = ProxyTestAuthProvider({"test-proj": {"token"}}) + meter = ProxyTestMeter() + client = make_http_client(await make_repo(), auth, meter) + request_body: dict = {"model": "test-model", "messages": [], "stream": True} + if stream_options is not None: + request_body["stream_options"] = stream_options + resp = await client.post(CHAT_COMPLETIONS_URL, json=request_body, headers=AUTH_HEADERS) + + assert resp.status_code == 200 + chunks = parse_sse(resp.content) + assert [c["choices"][0]["delta"]["content"] for c in chunks[:2]] == ["Hello", " there"] + assert [c.get("usage") for c in chunks] == expected_usage + # The service is always asked for token counts on every chunk. + assert json.loads(requests[0].content)["stream_options"] == { + "include_usage": True, + "continuous_usage_stats": True, + } + # Tokens are metered as they are counted: each chunk adds what is new. + assert [(m.input_tokens, m.cached_input_tokens, m.output_tokens) for m in meter.added] == [ + (9, 0, 1), + (0, 0, 1), + (0, 4, 0), + ] + + async def test_meters_stopped_stream_up_to_its_last_chunk(self, mock_service) -> None: + is_stream_closed = asyncio.Event() + + async def stream() -> AsyncIterator[bytes]: + try: + yield sse(chunk("Hello", usage(9, 1)), chunk(" there", usage(9, 2)), done=False) + await asyncio.Event().wait() # the model keeps generating + finally: + is_stream_closed.set() + + mock_service(lambda request: httpx.Response(200, content=stream())) + meter = ProxyTestMeter() + request = ASGIRequest( + make_app(await make_repo(), ProxyTestAuthProvider({"test-proj": {"token"}}), meter), + {"model": "test-model", "messages": [], "stream": True}, + ) + await asyncio.wait_for(request.is_body_received.wait(), timeout=10) + request.disconnect() + await asyncio.wait_for(request.wait(), timeout=10) + + await asyncio.wait_for(is_stream_closed.wait(), timeout=10) + assert [(m.input_tokens, m.output_tokens) for m in meter.added] == [(9, 1), (0, 1)] + + @pytest.mark.parametrize("stream", [False, True]) + async def test_does_not_meter_request_stopped_before_response( + self, mock_service, stream: bool + ) -> None: + is_request_received = asyncio.Event() + is_request_cancelled = asyncio.Event() + + async def handle(request: httpx.Request) -> httpx.Response: + is_request_received.set() + try: + await asyncio.Event().wait() # the model is generating + except asyncio.CancelledError: + is_request_cancelled.set() + raise + raise AssertionError("unreachable") + + mock_service(handle) + meter = ProxyTestMeter() + request = ASGIRequest( + make_app(await make_repo(), ProxyTestAuthProvider({"test-proj": {"token"}}), meter), + {"model": "test-model", "messages": [], "stream": stream}, + ) + await asyncio.wait_for(is_request_received.wait(), timeout=10) + request.disconnect() + await asyncio.wait_for(request.wait(), timeout=10) + + assert is_request_cancelled.is_set() + assert meter.added == [] + + +@pytest.mark.asyncio +async def test_imported_model_is_served_to_importer_project_members_only(mock_service) -> None: + mock_service(lambda request: httpx.Response(200, json={"usage": usage(9, 2)})) + auth = ProxyTestAuthProvider( + {"importer-proj": {"importer-token"}, "exporter-proj": {"exporter-token"}} + ) + meter = ProxyTestMeter() + service = make_service("exporter-proj", "test-service") + repo = GatewayProxyRepo() + await repo.set_project(make_project("exporter-proj")) + await repo.set_service(service) + await repo.set_model(make_model("exporter-proj", "test-model", "test-service")) + await repo.set_import( + ImportedModel( + importer_project="importer-proj", + export_id="export-1", + exporter_project="exporter-proj", + name="test-model", + ) + ) + client = make_http_client(repo, auth, meter) + body = {"model": "test-model", "messages": []} + + resp = await client.get( + "http://test-host/proxy/models/importer-proj/models", + headers={"Authorization": "Bearer importer-token"}, + ) + assert resp.status_code == 200 + assert [model["id"] for model in resp.json()["data"]] == ["test-model"] + resp = await client.post( + "http://test-host/proxy/models/importer-proj/chat/completions", + json=body, + headers={"Authorization": "Bearer importer-token"}, + ) + assert resp.status_code == 200 + # The request goes to the exporter project's service and is metered + # under the importer project with the export. + assert meter.added == [ + MeteredTokens( + caller=make_caller("importer-proj", "importer-token"), + service=service, + export_id="export-1", + input_tokens=9, + cached_input_tokens=0, + output_tokens=2, + ) + ] + + for project_name, token in ( + ("importer-proj", "exporter-token"), + ("exporter-proj", "importer-token"), + ): + resp = await client.post( + f"http://test-host/proxy/models/{project_name}/chat/completions", + json=body, + headers={"Authorization": f"Bearer {token}"}, + ) + assert resp.status_code == 403 + assert len(meter.added) == 1 + + +@pytest.mark.asyncio +async def test_tgi_chat_completions_are_validated_and_not_metered(mock_chat_client) -> None: + auth = ProxyTestAuthProvider({"test-proj": {"token"}}) + meter = ProxyTestMeter() + repo = GatewayProxyRepo() + await repo.set_project(make_project("test-proj")) + await repo.set_service(make_service("test-proj", "test-service")) + await repo.set_model( + make_model("test-proj", "test-model", "test-service", format_spec=TGI_FORMAT) + ) + client = make_http_client(repo, auth, meter) + body = {"model": "test-model", "messages": [{"role": "user", "content": "Hi"}]} + + resp = await client.post(CHAT_COMPLETIONS_URL, json=body, headers=AUTH_HEADERS) + assert resp.status_code == 200 + assert resp.json()["usage"]["prompt_tokens"] == 9 + assert meter.added == [] + + resp = await client.post( + CHAT_COMPLETIONS_URL, json={**body, "max_completion_tokens": 100}, headers=AUTH_HEADERS + ) + assert resp.status_code == 422 + + +class ASGIRequest: + """ + Sends a chat completions request to the app without an HTTP client, + to control when the client disconnects. + """ + + def __init__(self, app: FastAPI, body: dict) -> None: + self.status: Optional[int] = None + self.is_body_received = asyncio.Event() + self._body = json.dumps(body).encode() + self._is_disconnected = asyncio.Event() + scope: Scope = { + "type": "http", + "asgi": {"version": "3.0", "spec_version": "2.3"}, + "http_version": "1.1", + "method": "POST", + "path": "/proxy/models/test-proj/chat/completions", + "raw_path": b"/proxy/models/test-proj/chat/completions", + "query_string": b"", + "headers": [ + (b"authorization", b"Bearer token"), + (b"content-type", b"application/json"), + ], + } + self._task = asyncio.create_task(app(scope, self._receive, self._send)) + + def disconnect(self) -> None: + self._is_disconnected.set() + + async def wait(self) -> None: + await self._task + + async def _receive(self) -> Message: + if self._body: + body, self._body = self._body, b"" + return {"type": "http.request", "body": body, "more_body": False} + await self._is_disconnected.wait() + return {"type": "http.disconnect"} + + async def _send(self, message: Message) -> None: + if message["type"] == "http.response.start": + self.status = message["status"] + elif message.get("body"): + self.is_body_received.set() diff --git a/src/tests/_internal/proxy/lib/services/test_service_connection.py b/src/tests/_internal/proxy/lib/services/test_service_connection.py index c511658b0c..15f6dec7d4 100644 --- a/src/tests/_internal/proxy/lib/services/test_service_connection.py +++ b/src/tests/_internal/proxy/lib/services/test_service_connection.py @@ -12,9 +12,13 @@ from dstack._internal.core.errors import SSHError from dstack._internal.core.models.instances import SSHConnectionParams from dstack._internal.core.services.ssh.tunnel import SSHTunnel +from dstack._internal.proxy.gateway.repo.repo import GatewayProxyRepo from dstack._internal.proxy.lib.errors import UnexpectedProxyError from dstack._internal.proxy.lib.services import service_connection -from dstack._internal.proxy.lib.services.service_connection import ServiceConnectionPool +from dstack._internal.proxy.lib.services.service_connection import ( + ServiceConnectionPool, + get_service_replica_client, +) from dstack._internal.proxy.lib.testing.common import make_project, make_service @@ -447,6 +451,18 @@ async def test_remove_all_closes_draining_connections_without_waiting( tunnel.aclose.assert_awaited_once() + async def test_gateway_requests_to_service_share_client_until_it_is_removed( + self, service, pool + ): + repo = GatewayProxyRepo() + client = await get_service_replica_client(service, repo, pool) + assert await get_service_replica_client(service, repo, pool) is client + assert client.headers["Host"] == "service.gateway.test" + + await pool.remove_nginx_client("service.gateway.test") + assert client.is_closed + assert await get_service_replica_client(service, repo, pool) is not client + @pytest.mark.skipif(sys.platform != "linux", reason="reads /proc/net/unix") @pytest.mark.asyncio diff --git a/src/tests/_internal/server/background/pipeline_tasks/test_gateway_replicas.py b/src/tests/_internal/server/background/pipeline_tasks/test_gateway_replicas.py index 5e61a972a1..d1f3a82013 100644 --- a/src/tests/_internal/server/background/pipeline_tasks/test_gateway_replicas.py +++ b/src/tests/_internal/server/background/pipeline_tasks/test_gateway_replicas.py @@ -1,11 +1,13 @@ import asyncio +import logging import uuid from datetime import datetime, timedelta, timezone from typing import Optional from unittest.mock import ANY, AsyncMock, MagicMock, Mock, patch import pytest -from sqlalchemy import select +from httpx import HTTPStatusError, Request, Response +from sqlalchemy import delete, select from sqlalchemy.ext.asyncio import AsyncSession from dstack._internal.core.backends.base.compute import ComputeWithGatewaySupport @@ -24,8 +26,11 @@ SSHKey, ) from dstack._internal.core.models.runs import JobStatus, RunStatus, ServiceSpec +from dstack._internal.core.models.services import OpenAIChatModel +from dstack._internal.proxy.gateway.schemas.registry import ImportListItem from dstack._internal.proxy.gateway.schemas.services import ServiceListItem, ServiceListReplicaItem from dstack._internal.server import settings +from dstack._internal.server.background.pipeline_tasks import gateway_replicas from dstack._internal.server.background.pipeline_tasks.gateway_replicas import ( GatewayReplicaFetcher, GatewayReplicaPipeline, @@ -33,14 +38,22 @@ GatewayReplicaWorker, ) from dstack._internal.server.models import ( + ExportedModelModel, + GatewayModel, GatewayReplicaModel, + ImportModel, + ProjectModel, + RunModel, ServiceRegistrationModel, ServiceReplicaRegistrationModel, + UserModel, ) +from dstack._internal.server.services import exports as exports_service from dstack._internal.server.testing.common import ( AsyncContextManager, ComputeMockSpec, create_backend, + create_export, create_fleet, create_gateway, create_gateway_replica, @@ -49,6 +62,7 @@ create_project, create_repo, create_run, + create_service_run, create_user, get_gateway_replica_configuration, get_job_provisioning_data, @@ -755,6 +769,46 @@ async def test_running_to_terminating_when_scaled_in( assert replica.active is False assert replica.status_message == "Scaled in" + async def test_calls_hooks_and_processes_replica_if_hook_fails( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + monkeypatch: pytest.MonkeyPatch, + ): + project = await create_project(session=session) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + ) + replica = await create_gateway_replica( + session=session, + gateway_id=gateway.id, + backend=backend, + status=GatewayReplicaStatus.RUNNING, + ) + _lock_replica(replica) + await session.commit() + _get_client_mock(mock_gateway_connection).list_services.return_value = [] + failing_hook = AsyncMock(side_effect=RuntimeError("hook failed")) + hook = AsyncMock() + monkeypatch.setattr( + gateway_replicas, "_PROCESS_RUNNING_GATEWAY_REPLICA_HOOKS", [failing_hook, hook] + ) + + await worker.process(_replica_to_pipeline_item(replica)) + + connection = mock_gateway_connection.return_value + failing_hook.assert_awaited_once_with(replica.id, connection) + hook.assert_awaited_once_with(replica.id, connection) + await session.refresh(replica) + assert replica.status == GatewayReplicaStatus.RUNNING + assert replica.lock_token is None + def _get_client_mock(mock_gateway_connection: AsyncMock) -> AsyncMock: return mock_gateway_connection.return_value.client.return_value.__aenter__.return_value @@ -2242,6 +2296,357 @@ async def test_does_not_emit_duplicate_unregistration_error_event_for_unchanged_ assert [e.message for e in events_after_second_tick].count(error_message) == 1 +@pytest.mark.asyncio +@pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) +class TestGatewayReplicaWorkerRunningImportsSync: + """ + Covers `_process_running_item`'s sync of imports: registering/unregistering imports + so the gateway has one for each exported model that its services serve + and each importer project. + """ + + @pytest.fixture(autouse=True) + def enable_exports(self, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setattr(exports_service, "_EXPORTS_ENABLED", True) + + async def _create_gateway_and_replica( + self, session: AsyncSession, project: ProjectModel, domain: str = "example.com" + ) -> tuple[GatewayModel, GatewayReplicaModel]: + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + wildcard_domain=domain, + status=GatewayStatus.RUNNING, + ) + replica = await create_gateway_replica( + session=session, + gateway_id=gateway.id, + backend=backend, + status=GatewayReplicaStatus.RUNNING, + ) + _lock_replica(replica) + await session.commit() + return gateway, replica + + async def _create_service_run( + self, + session: AsyncSession, + project: ProjectModel, + user: UserModel, + gateway: GatewayModel, + run_name: str, + model_name: str, + ) -> RunModel: + repo = await create_repo(session=session, project_id=project.id, repo_name=run_name) + return await create_service_run( + session=session, + project=project, + repo=repo, + user=user, + gateway=gateway, + run_name=run_name, + model=OpenAIChatModel(name=model_name, format="openai"), + ) + + async def test_registers_imports_of_exported_models_that_services_serve( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + ): + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="Exporter") + importer_projects = [ + await create_project(session=session, owner=user, name=name) + for name in ("Importer-1", "Importer-2") + ] + deleted_importer_project = await create_project( + session=session, owner=user, name="Deleted-Importer", deleted=True + ) + other_project = await create_project(session=session, owner=user, name="Other") + gateway, replica = await self._create_gateway_and_replica(session, exporter_project) + await self._create_service_run( + session, exporter_project, user, gateway, "exported-service", "exported-model" + ) + await self._create_service_run( + session, exporter_project, user, gateway, "other-service", "not-exported-model" + ) + # The export holds this model, but only another project serves a model with this name. + await self._create_service_run( + session, other_project, user, gateway, "other-project-service", "not-served-model" + ) + export = await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[*importer_projects, deleted_importer_project], + exported_fleets=[], + exported_models=["exported-model", "not-served-model"], + ) + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + client_mock.list_imports.return_value = [] + + await worker.process(_replica_to_pipeline_item(replica)) + + for importer_project_name in ("importer-1", "importer-2"): + client_mock.register_import.assert_any_call( + importer_project=importer_project_name, + export_id=export.id, + exporter_project="exporter", + name="exported-model", + ) + assert client_mock.register_import.call_count == 2 + client_mock.unregister_import.assert_not_called() + + @pytest.mark.parametrize("change", [None, "export", "importer_project", "model", "run"]) + async def test_unregisters_import_when_export_importer_project_model_or_run_changes( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + change: Optional[str], + ): + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="exporter") + importer_project = await create_project(session=session, owner=user, name="importer") + gateway, replica = await self._create_gateway_and_replica(session, exporter_project) + run = await self._create_service_run( + session, exporter_project, user, gateway, "test-service", "test-model" + ) + export = await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + ) + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + client_mock.list_imports.return_value = [ + ImportListItem( + importer_project="importer", + export_id=export.id.hex, + exporter_project="exporter", + name="test-model", + ) + ] + if change == "export": + await session.delete(export) + elif change == "importer_project": + await session.execute(delete(ImportModel).where(ImportModel.export_id == export.id)) + elif change == "model": + await session.execute( + delete(ExportedModelModel).where(ExportedModelModel.export_id == export.id) + ) + elif change == "run": + run.status = RunStatus.TERMINATING + await session.commit() + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_import.assert_not_called() + if change is None: + client_mock.unregister_import.assert_not_called() + else: + client_mock.unregister_import.assert_called_once_with( + importer_project="importer", export_id=export.id, name="test-model" + ) + + async def test_registers_import_from_older_export_and_moves_it_once_export_is_deleted( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + ): + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="exporter") + importer_project = await create_project(session=session, owner=user, name="importer") + gateway, replica = await self._create_gateway_and_replica(session, exporter_project) + await self._create_service_run( + session, exporter_project, user, gateway, "test-service", "test-model" + ) + newer_export, older_export = [ + await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + name=name, + ) + for name in ("newer-export", "older-export") + ] + newer_export.created_at = datetime(2026, 2, 1, tzinfo=timezone.utc) + older_export.created_at = datetime(2026, 1, 1, tzinfo=timezone.utc) + await session.commit() + newer_export_id, older_export_id = newer_export.id, older_export.id + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + client_mock.list_imports.return_value = [] + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_import.assert_called_once_with( + importer_project="importer", + export_id=older_export_id, + exporter_project="exporter", + name="test-model", + ) + client_mock.unregister_import.assert_not_called() + + # Second tick: the older export is deleted, so the import moves to the newer one. + client_mock.reset_mock() + client_mock.list_imports.return_value = [ + ImportListItem( + importer_project="importer", + export_id=older_export_id.hex, + exporter_project="exporter", + name="test-model", + ) + ] + await session.delete(older_export) + await session.refresh(replica) + _lock_replica(replica) + await session.commit() + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_import.assert_called_once_with( + importer_project="importer", + export_id=newer_export_id, + exporter_project="exporter", + name="test-model", + ) + client_mock.unregister_import.assert_called_once_with( + importer_project="importer", export_id=older_export_id, name="test-model" + ) + # Registering replaces the import on the gateway, and unregistering the previous one + # then changes nothing, so the model stays imported. + called = [c[0] for c in client_mock.method_calls] + assert called.index("register_import") < called.index("unregister_import") + + async def test_unregisters_imports_on_gateway_with_project_name_in_domain( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + ): + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="exporter") + importer_project = await create_project(session=session, owner=user, name="importer") + gateway, replica = await self._create_gateway_and_replica( + session, exporter_project, domain="${{ run.project_name }}.example.com" + ) + await self._create_service_run( + session, exporter_project, user, gateway, "test-service", "test-model" + ) + export = await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + ) + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + # Registered while the gateway had a fixed domain. + client_mock.list_imports.return_value = [ + ImportListItem( + importer_project="importer", + export_id=export.id.hex, + exporter_project="exporter", + name="test-model", + ) + ] + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_import.assert_not_called() + client_mock.unregister_import.assert_called_once_with( + importer_project="importer", export_id=export.id, name="test-model" + ) + + async def test_syncs_services_but_not_imports_if_gateway_has_no_import_routes( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + caplog: pytest.LogCaptureFixture, + ): + caplog.set_level(logging.WARNING) + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="exporter") + importer_project = await create_project(session=session, owner=user, name="importer") + gateway, replica = await self._create_gateway_and_replica(session, exporter_project) + await self._create_service_run( + session, exporter_project, user, gateway, "test-service", "test-model" + ) + await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + ) + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + request = Request("GET", "http://gateway/api/registry/imports/list") + client_mock.list_imports.side_effect = HTTPStatusError( + "Not Found", request=request, response=Response(404, request=request) + ) + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_service.assert_called_once() + client_mock.register_import.assert_not_called() + client_mock.unregister_import.assert_not_called() + # The gateway is not updated yet, which is not worth a warning on every sync. + assert "imports" not in caplog.text + await session.refresh(replica) + assert replica.status == GatewayReplicaStatus.RUNNING + assert replica.lock_token is None + + async def test_does_not_sync_imports_if_exports_are_disabled( + self, + test_db, + session: AsyncSession, + worker: GatewayReplicaWorker, + mock_gateway_connection: AsyncMock, + monkeypatch: pytest.MonkeyPatch, + ): + monkeypatch.setattr(exports_service, "_EXPORTS_ENABLED", False) + user = await create_user(session=session) + exporter_project = await create_project(session=session, owner=user, name="exporter") + importer_project = await create_project(session=session, owner=user, name="importer") + gateway, replica = await self._create_gateway_and_replica(session, exporter_project) + await self._create_service_run( + session, exporter_project, user, gateway, "test-service", "test-model" + ) + await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + ) + client_mock = _get_client_mock(mock_gateway_connection) + client_mock.list_services.return_value = [] + + await worker.process(_replica_to_pipeline_item(replica)) + + client_mock.register_service.assert_called_once() + client_mock.list_imports.assert_not_called() + client_mock.register_import.assert_not_called() + client_mock.unregister_import.assert_not_called() + + @pytest.mark.asyncio @pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) class TestGatewayReplicaWorkerProvisioning: diff --git a/src/tests/_internal/server/routers/test_exports.py b/src/tests/_internal/server/routers/test_exports.py index fea2c6229d..a4a6a66c5c 100644 --- a/src/tests/_internal/server/routers/test_exports.py +++ b/src/tests/_internal/server/routers/test_exports.py @@ -5,6 +5,7 @@ from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession +from dstack._internal.core.models.services import AnyModel, OpenAIChatModel, TGIChatModel from dstack._internal.core.models.users import GlobalRole, ProjectRole from dstack._internal.server.models import ExportModel, ImportModel from dstack._internal.server.services import exports as exports_service @@ -19,6 +20,8 @@ create_fleet, create_gateway, create_project, + create_repo, + create_service_run, create_user, get_auth_headers, get_fleet_spec, @@ -169,6 +172,123 @@ async def test_creates_empty_export(self, session: AsyncSession, client: AsyncCl res = await session.execute(select(ExportModel).where(ExportModel.name == "empty-export")) assert res.scalar() is not None + async def test_creates_export_with_models(self, session: AsyncSession, client: AsyncClient): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + wildcard_domain="example.com", + ) + # It is enough that one of the services that declare the model can serve it + # to importer projects. + for run_name, run_gateway in [("service-on-gateway", gateway), ("service", None)]: + await create_service_run( + session=session, + project=project, + user=user, + repo=repo, + gateway=run_gateway, + model=OpenAIChatModel(name="running-model", format="openai"), + run_name=run_name, + ) + + response = await client.post( + f"/api/project/{project.name}/exports/create", + headers=get_auth_headers(user.token), + json={ + "name": "test-export", + # A model can be exported before any service declares it. + "exported_models": ["running-model", "not-running-model"], + }, + ) + assert response.status_code == 200 + assert sorted(m["name"] for m in response.json()["exported_models"]) == [ + "not-running-model", + "running-model", + ] + + res = await session.execute(select(ExportModel).where(ExportModel.name == "test-export")) + export = res.scalar_one() + await session.refresh(export, ["exported_models"]) + assert sorted(m.name for m in export.exported_models) == [ + "not-running-model", + "running-model", + ] + + @pytest.mark.parametrize( + ("model", "gateway_domain", "error"), + [ + pytest.param( + TGIChatModel( + name="test-model", format="tgi", chat_template="test", eos_token="" + ), + "example.com", + "'test-service' declares it with format tgi, not openai", + id="tgi-format", + ), + pytest.param( + OpenAIChatModel(name="test-model", format="openai"), + "${{ run.project_name }}.example.com", + "'test-service' runs on 'test-gateway'," + " which has ${{ run.project_name }} in its domain", + id="project-name-in-gateway-domain", + ), + pytest.param( + OpenAIChatModel(name="test-model", format="openai"), + None, + "'test-service' runs without a gateway", + id="no-gateway", + ), + ], + ) + async def test_rejects_model_if_no_running_service_can_serve_it( + self, + session: AsyncSession, + client: AsyncClient, + model: AnyModel, + gateway_domain: Optional[str], + error: str, + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + repo = await create_repo(session=session, project_id=project.id) + gateway = None + if gateway_domain is not None: + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + name="test-gateway", + wildcard_domain=gateway_domain, + ) + await create_service_run( + session=session, project=project, user=user, repo=repo, gateway=gateway, model=model + ) + + response = await client.post( + f"/api/project/{project.name}/exports/create", + headers=get_auth_headers(user.token), + json={"name": "test-export", "exported_models": ["test-model"]}, + ) + assert response.status_code == 400 + assert ( + response.json()["detail"][0]["msg"] + == f"Cannot add 'test-model' to 'test-export': {error}" + ) + res = await session.execute(select(func.count()).select_from(ExportModel)) + assert res.scalar_one() == 0 + async def test_creates_global_export(self, session: AsyncSession, client: AsyncClient): admin = await create_user(session=session, global_role=GlobalRole.ADMIN) exporter_project = await create_project( @@ -266,6 +386,14 @@ async def test_creates_global_export(self, session: AsyncSession, client: AsyncC "Project 'ExporterProject' cannot import from itself", id="self-import", ), + pytest.param( + { + "name": "test-export", + "exported_models": ["test-model", " "], + }, + "Model names cannot be empty", + id="empty-model-name", + ), pytest.param( { "name": "", @@ -876,6 +1004,24 @@ async def test_can_add_same_entities_as_existing_deleted_ones( "Cannot change global status and add/remove importers at the same time", id="unset-global-with-importer-changes", ), + pytest.param( + { + "name": "test-export", + "set_global": True, + "remove_exported_models": ["test-model"], + }, + "Cannot change global status and add/remove models at the same time", + id="set-global-with-model-changes", + ), + pytest.param( + { + "name": "test-export", + "unset_global": True, + "add_exported_models": ["test-model"], + }, + "Cannot change global status and add/remove models at the same time", + id="unset-global-with-model-changes", + ), ], ) async def test_rejects_invalid_update( @@ -1159,12 +1305,109 @@ async def test_set_global_already_global_returns_400( assert response.status_code == 400 assert "The export is already global" in response.json()["detail"][0]["msg"] + async def test_adds_and_removes_models(self, session: AsyncSession, client: AsyncClient): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["model1", "model2"], + name="test-export", + ) + + response = await client.post( + f"/api/project/{project.name}/exports/update", + headers=get_auth_headers(user.token), + json={ + "name": "test-export", + "add_exported_models": ["model3"], + "remove_exported_models": ["model2"], + }, + ) + assert response.status_code == 200 + assert sorted(m["name"] for m in response.json()["exported_models"]) == [ + "model1", + "model3", + ] + + response = await client.post( + f"/api/project/{project.name}/exports/list", headers=get_auth_headers(user.token) + ) + assert response.status_code == 200 + assert sorted(m["name"] for m in response.json()["exports"][0]["exported_models"]) == [ + "model1", + "model3", + ] + + async def test_cannot_add_model_to_global_export( + self, session: AsyncSession, client: AsyncClient + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + name="my-export", + is_global=True, + ) + + response = await client.post( + f"/api/project/{project.name}/exports/update", + headers=get_auth_headers(user.token), + json={"name": "my-export", "add_exported_models": ["test-model"]}, + ) + assert response.status_code == 400 + assert response.json()["detail"][0]["msg"] == ( + "Cannot add 'test-model' to 'my-export': the export is global," + " and models cannot be exported to all projects yet." + " Add the model to an export that names its importer projects instead." + ) + + async def test_cannot_set_global_export_with_models( + self, session: AsyncSession, client: AsyncClient + ): + admin = await create_user(session=session, global_role=GlobalRole.ADMIN) + project = await create_project(session=session, owner=admin) + export = await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="my-export", + ) + + response = await client.post( + f"/api/project/{project.name}/exports/update", + headers=get_auth_headers(admin.token), + json={"name": "my-export", "set_global": True}, + ) + assert response.status_code == 400 + assert response.json()["detail"][0]["msg"] == ( + "Cannot make 'my-export' global: it exports models ('test-model')," + " and models cannot be exported to all projects yet." + " Name its importer projects instead." + ) + await session.refresh(export) + assert not export.is_global + @pytest.mark.parametrize( "body", [ pytest.param({"add_importer_projects": ["ImporterProject"]}, id="add-importer"), pytest.param({"add_exported_fleets": ["fleet"]}, id="add-fleet"), pytest.param({"add_exported_gateways": ["gateway"]}, id="add-gateway"), + pytest.param({"add_exported_models": ["model"]}, id="add-model"), ], ) async def test_rejects_additions_when_disabled( diff --git a/src/tests/_internal/server/routers/test_gateways.py b/src/tests/_internal/server/routers/test_gateways.py index d5980f5f96..7946cae6e2 100644 --- a/src/tests/_internal/server/routers/test_gateways.py +++ b/src/tests/_internal/server/routers/test_gateways.py @@ -8,7 +8,9 @@ from dstack._internal.core.models.backends.base import BackendType from dstack._internal.core.models.common import ApplyAction from dstack._internal.core.models.gateways import GatewayStatus +from dstack._internal.core.models.services import OpenAIChatModel from dstack._internal.core.models.users import GlobalRole, ProjectRole +from dstack._internal.server.models import GatewayModel, ProjectModel, UserModel from dstack._internal.server.services.projects import add_project_member from dstack._internal.server.testing.common import ( clear_events, @@ -17,12 +19,43 @@ create_gateway, create_gateway_replica, create_project, + create_repo, + create_service_run, create_user, get_auth_headers, list_events, ) from dstack._internal.server.testing.matchers import SomeUUID4Str +CANNOT_REMOVE_DOMAIN_OR_PUT_PROJECT_NAME_INTO_IT_ERROR = ( + "Cannot remove the domain of 'my-gateway' or put ${{ run.project_name }} into it:" + " 'test-model' is exported by 'models', and 'test-service' serves it to importer projects" + " from this gateway." + " Move the service to another gateway, or remove the model from the export first." +) + + +async def create_service_with_exported_model( + session: AsyncSession, project: ProjectModel, user: UserModel, gateway: GatewayModel +) -> None: + repo = await create_repo(session=session, project_id=project.id) + await create_service_run( + session=session, + project=project, + repo=repo, + user=user, + gateway=gateway, + model=OpenAIChatModel(name="test-model", format="openai"), + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="models", + ) + class TestListAndGetGateways: @pytest.mark.asyncio @@ -1380,6 +1413,41 @@ async def test_set_wildcard_domain( events[0].message == "Gateway wildcard domain changed 'old.example' -> 'new.example'" ) + @pytest.mark.asyncio + @pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) + async def test_rejects_wildcard_domain_with_project_name_if_gateway_serves_exported_model( + self, test_db, session: AsyncSession, client: AsyncClient + ): + user = await create_user(session, global_role=GlobalRole.USER) + project = await create_project(session) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + backend = await create_backend(session, project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + name="my-gateway", + wildcard_domain="example.com", + ) + await create_service_with_exported_model(session, project, user, gateway) + response = await client.post( + f"/api/project/{project.name}/gateways/set_wildcard_domain", + json={ + "name": gateway.name, + "wildcard_domain": "${{ run.project_name }}.example.com", + }, + headers=get_auth_headers(user.token), + ) + assert response.status_code == 400 + assert ( + response.json()["detail"][0]["msg"] + == CANNOT_REMOVE_DOMAIN_OR_PUT_PROJECT_NAME_INTO_IT_ERROR + ) + await session.refresh(gateway) + assert gateway.wildcard_domain == "example.com" + @pytest.mark.asyncio @pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) async def test_set_wildcard_domain_missing( @@ -2206,6 +2274,55 @@ async def test_updates_in_place_with_force_apply( await session.refresh(gateway) assert gateway.last_update_at is not None + @pytest.mark.asyncio + @pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) + @pytest.mark.parametrize("domain", ["${{ run.project_name }}.example.com", None]) + async def test_rejects_domain_with_project_name_or_none_if_gateway_serves_exported_model( + self, test_db, session: AsyncSession, client: AsyncClient, domain: Optional[str] + ): + user = await create_user(session, global_role=GlobalRole.USER) + project = await create_project(session) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.ADMIN + ) + backend = await create_backend(session, project.id, backend_type=BackendType.AWS) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + name="my-gateway", + region="us-east-1", + wildcard_domain="example.com", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + await create_service_with_exported_model(session, project, user, gateway) + response = await client.post( + f"/api/project/{project.name}/gateways/apply", + json={ + "plan": { + "spec": { + "configuration": { + "type": "gateway", + "name": "my-gateway", + "backend": "aws", + "region": "us-east-1", + "domain": domain, + } + }, + "current_resource": None, + }, + "force": True, + }, + headers=get_auth_headers(user.token), + ) + assert response.status_code == 400 + assert ( + response.json()["detail"][0]["msg"] + == CANNOT_REMOVE_DOMAIN_OR_PUT_PROJECT_NAME_INTO_IT_ERROR + ) + await session.refresh(gateway) + assert gateway.wildcard_domain == "example.com" + @pytest.mark.asyncio @pytest.mark.parametrize("test_db", ["sqlite", "postgres"], indirect=True) @pytest.mark.parametrize("populate_configuration", [True, False]) diff --git a/src/tests/_internal/server/routers/test_imports.py b/src/tests/_internal/server/routers/test_imports.py index c4915d1375..fd43961b51 100644 --- a/src/tests/_internal/server/routers/test_imports.py +++ b/src/tests/_internal/server/routers/test_imports.py @@ -5,6 +5,8 @@ from sqlalchemy import func, select from sqlalchemy.ext.asyncio import AsyncSession +from dstack._internal.core.models.runs import RunStatus +from dstack._internal.core.models.services import OpenAIChatModel from dstack._internal.core.models.users import GlobalRole, ProjectRole from dstack._internal.server.models import ExportModel, ImportModel from dstack._internal.server.services import exports as exports_service @@ -19,6 +21,8 @@ create_fleet, create_gateway, create_project, + create_repo, + create_service_run, create_user, get_auth_headers, get_fleet_spec, @@ -416,6 +420,149 @@ async def test_not_includes_deleted_fleets(self, session: AsyncSession, client: assert len(imports[0]["export"]["exported_fleets"]) == 1 assert imports[0]["export"]["exported_fleets"][0]["name"] == "fleet" + async def test_lists_imported_models(self, session: AsyncSession, client: AsyncClient): + user = await create_user(session=session, global_role=GlobalRole.USER) + importer_project = await create_project( + session=session, name="ImporterProject", owner=user + ) + await add_project_member( + session=session, project=importer_project, user=user, project_role=ProjectRole.USER + ) + exporter_project = await create_project( + session=session, name="ExporterProject", owner=user + ) + repo = await create_repo(session=session, project_id=exporter_project.id) + backend = await create_backend(session=session, project_id=exporter_project.id) + gateway = await create_gateway( + session=session, + project_id=exporter_project.id, + backend_id=backend.id, + wildcard_domain="example.com", + ) + await create_service_run( + session=session, + project=exporter_project, + repo=repo, + user=user, + gateway=gateway, + model=OpenAIChatModel(name="running-model", format="openai"), + ) + await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["running-model", "not-running-model"], + name="test-export", + ) + # Another project exports a model with the same name but does not run it. + other_exporter_project = await create_project( + session=session, name="OtherExporterProject", owner=user + ) + await create_export( + session=session, + exporter_project=other_exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["running-model"], + name="other-export", + ) + + response = await client.post( + f"/api/project/{importer_project.name}/imports/list", + headers=get_auth_headers(user.token), + ) + assert response.status_code == 200 + exported_models = { + imp["export"]["project_name"]: sorted( + imp["export"]["exported_models"], key=lambda m: m["name"] + ) + for imp in response.json()["imports"] + } + assert exported_models == { + "ExporterProject": [ + {"name": "not-running-model", "base_url": None}, + { + "name": "running-model", + "base_url": "https://gateway.example.com/ImporterProject", + }, + ], + "OtherExporterProject": [{"name": "running-model", "base_url": None}], + } + + async def test_imported_model_url_follows_its_service( + self, session: AsyncSession, client: AsyncClient + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + importer_project = await create_project( + session=session, name="ImporterProject", owner=user + ) + await add_project_member( + session=session, project=importer_project, user=user, project_role=ProjectRole.USER + ) + exporter_project = await create_project( + session=session, name="ExporterProject", owner=user + ) + repo = await create_repo(session=session, project_id=exporter_project.id) + backend = await create_backend(session=session, project_id=exporter_project.id) + gateway = await create_gateway( + session=session, + project_id=exporter_project.id, + backend_id=backend.id, + name="gateway", + wildcard_domain="example.com", + ) + other_gateway = await create_gateway( + session=session, + project_id=exporter_project.id, + backend_id=backend.id, + name="other-gateway", + wildcard_domain="other.example.com", + ) + model = OpenAIChatModel(name="test-model", format="openai") + run = await create_service_run( + session=session, + project=exporter_project, + repo=repo, + user=user, + gateway=gateway, + model=model, + ) + await create_export( + session=session, + exporter_project=exporter_project, + importer_projects=[importer_project], + exported_fleets=[], + exported_models=["test-model"], + name="test-export", + ) + + async def get_base_url() -> Optional[str]: + response = await client.post( + f"/api/project/{importer_project.name}/imports/list", + headers=get_auth_headers(user.token), + ) + assert response.status_code == 200 + return response.json()["imports"][0]["export"]["exported_models"][0]["base_url"] + + assert await get_base_url() == "https://gateway.example.com/ImporterProject" + + # The service is stopped + run.status = RunStatus.TERMINATED + await session.commit() + assert await get_base_url() is None + + # The service is applied again on another gateway + await create_service_run( + session=session, + project=exporter_project, + repo=repo, + user=user, + gateway=other_gateway, + model=model, + ) + assert await get_base_url() == "https://gateway.other.example.com/ImporterProject" + async def test_does_not_include_other_projects_imports( self, session: AsyncSession, client: AsyncClient ): diff --git a/src/tests/_internal/server/routers/test_runs.py b/src/tests/_internal/server/routers/test_runs.py index ef888668a0..a42c6c73e5 100644 --- a/src/tests/_internal/server/routers/test_runs.py +++ b/src/tests/_internal/server/routers/test_runs.py @@ -44,6 +44,7 @@ RunStatus, RunTerminationReason, ) +from dstack._internal.core.models.services import AnyModel, OpenAIChatModel, TGIChatModel from dstack._internal.core.models.users import GlobalRole, ProjectRole from dstack._internal.core.models.volumes import InstanceMountPoint, MountPoint from dstack._internal.server.models import JobModel, RunModel @@ -65,6 +66,7 @@ create_project, create_repo, create_run, + create_service_run, create_user, get_auth_headers, get_fleet_configuration, @@ -3785,9 +3787,11 @@ async def _create_service_run( user, gateway=None, gateway_field=None, + run_name: str = "test-service", + model: Optional[AnyModel] = None, ) -> Tuple[RunModel, RunSpec]: run_spec = get_run_spec( - run_name="test-service", + run_name=run_name, repo_id=repo.name, configuration=ServiceConfiguration( type="service", @@ -3795,6 +3799,7 @@ async def _create_service_run( port=80, gateway=gateway_field, replicas=Range(min=1, max=1), + model=model, ), ) validate_run_spec_and_set_defaults(user, run_spec) @@ -4140,6 +4145,236 @@ async def test_no_reassignment_when_resolved_gateway_is_unchanged( assert "Service assigned to gateway" not in event_messages assert "Service assigned to run without a gateway" not in event_messages + @pytest.mark.asyncio + @pytest.mark.parametrize("new_gateway_field", [False, "project-gateway"]) + async def test_rejects_update_if_no_service_would_serve_exported_model( + self, + test_db, + session: AsyncSession, + client: AsyncClient, + new_gateway_field: Union[bool, str], + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.USER + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="my-gateway", + wildcard_domain="my-gateway.example", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + project_gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="project-gateway", + wildcard_domain="${{ run.project_name }}.project-gateway.example", + ) + await create_gateway_replica( + session=session, backend=backend, gateway_id=project_gateway.id + ) + run_model, run_spec = await self._create_service_run( + session, + project, + repo, + user, + gateway=gateway, + gateway_field="my-gateway", + model=OpenAIChatModel(name="test-model", format="openai"), + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="models", + ) + assert isinstance(run_spec.configuration, ServiceConfiguration) + run_spec.configuration.gateway = new_gateway_field + + response = await self._apply(client, project, user, run_model, run_spec) + assert response.status_code == 400, response.json() + assert response.json()["detail"][0]["msg"] == ( + "Cannot update 'test-service': 'test-model' is exported by 'models'," + " and no other service would serve it to importer projects." + " Start another service for the model, or remove it from the export first." + ) + await session.refresh(run_model) + assert run_model.gateway_id == gateway.id + + @pytest.mark.asyncio + async def test_updates_if_another_service_serves_exported_model( + self, test_db, session: AsyncSession, client: AsyncClient + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.USER + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="my-gateway", + wildcard_domain="my-gateway.example", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + model = OpenAIChatModel(name="test-model", format="openai") + run_model, run_spec = await self._create_service_run( + session, project, repo, user, gateway=gateway, gateway_field="my-gateway", model=model + ) + await create_service_run( + session=session, + project=project, + repo=repo, + user=user, + gateway=gateway, + run_name="other-service", + model=model, + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="models", + ) + assert isinstance(run_spec.configuration, ServiceConfiguration) + run_spec.configuration.gateway = False + + response = await self._apply(client, project, user, run_model, run_spec) + assert response.status_code == 200, response.json() + await session.refresh(run_model) + assert run_model.gateway_id is None + + @pytest.mark.asyncio + async def test_updates_if_service_still_serves_exported_model( + self, test_db, session: AsyncSession, client: AsyncClient + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.USER + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="my-gateway", + wildcard_domain="my-gateway.example", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + other_gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="other-gateway", + wildcard_domain="other-gateway.example", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=other_gateway.id) + run_model, run_spec = await self._create_service_run( + session, + project, + repo, + user, + gateway=gateway, + gateway_field="my-gateway", + model=OpenAIChatModel(name="test-model", format="openai"), + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="models", + ) + assert isinstance(run_spec.configuration, ServiceConfiguration) + run_spec.configuration.gateway = "other-gateway" + + response = await self._apply(client, project, user, run_model, run_spec) + assert response.status_code == 200, response.json() + await session.refresh(run_model) + assert run_model.gateway_id == other_gateway.id + + @pytest.mark.asyncio + @pytest.mark.parametrize( + ("model", "gateway_domain"), + [ + pytest.param( + TGIChatModel( + name="test-model", format="tgi", chat_template="test", eos_token="" + ), + "my-gateway.example", + id="tgi-format", + ), + pytest.param( + OpenAIChatModel(name="test-model", format="openai"), + "${{ run.project_name }}.my-gateway.example", + id="project-name-in-gateway-domain", + ), + ], + ) + async def test_updates_if_service_did_not_serve_exported_model( + self, + test_db, + session: AsyncSession, + client: AsyncClient, + model: AnyModel, + gateway_domain: str, + ): + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user) + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.USER + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + name="my-gateway", + wildcard_domain=gateway_domain, + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + run_model, run_spec = await self._create_service_run( + session, project, repo, user, gateway=gateway, gateway_field="my-gateway", model=model + ) + await create_export( + session=session, + exporter_project=project, + importer_projects=[], + exported_fleets=[], + exported_models=["test-model"], + name="models", + ) + assert isinstance(run_spec.configuration, ServiceConfiguration) + run_spec.configuration.gateway = False + + response = await self._apply(client, project, user, run_model, run_spec) + assert response.status_code == 200, response.json() + await session.refresh(run_model) + assert run_model.gateway_id is None + class TestStopRuns: @pytest.mark.asyncio @@ -4375,7 +4610,7 @@ class TestSubmitService: [("default-gateway", True), ("non-default-gateway", False)], None, "https://test-service.default-gateway.example", - "https://test-service.default-gateway.example/v1", + "https://gateway.default-gateway.example/test-project", True, "test-model", id="submits-to-default-gateway", @@ -4384,7 +4619,7 @@ class TestSubmitService: [("default-gateway", True), ("non-default-gateway", False)], True, "https://test-service.default-gateway.example", - "https://test-service.default-gateway.example/v1", + "https://gateway.default-gateway.example/test-project", True, "test-model", id="submits-to-default-gateway-when-gateway-true", @@ -4393,7 +4628,7 @@ class TestSubmitService: [("default-gateway", True), ("non-default-gateway", False)], "non-default-gateway", "https://test-service.non-default-gateway.example", - "https://test-service.non-default-gateway.example/v1", + "https://gateway.non-default-gateway.example/test-project", True, "test-model", id="submits-to-specified-gateway", @@ -4420,7 +4655,7 @@ class TestSubmitService: [("default-gateway", True)], None, "https://test-service.default-gateway.example", - "https://gateway.default-gateway.example", + "https://gateway.default-gateway.example/test-project", True, { "type": "chat", @@ -4847,6 +5082,10 @@ async def test_interpolates_project_name_in_imported_gateway_domain( response.json()["service"]["url"] == "https://test-service.importer-project.example.com" ) + assert ( + response.json()["service"]["model"]["base_url"] + == "https://gateway.importer-project.example.com/importer-project" + ) @pytest.mark.asyncio async def test_returns_error_if_imported_gateway_domain_has_unknown_variable( @@ -5004,3 +5243,53 @@ async def test_return_error_if_explicitly_specified_gateway_forbids_new_services assert response.json() == { "detail": [{"msg": "Gateway does not accept new services", "code": "error"}] } + + @pytest.mark.asyncio + async def test_return_error_if_service_domain_is_taken_by_gateway_model_endpoint( + self, + test_db, + session: AsyncSession, + client: AsyncClient, + ) -> None: + user = await create_user(session=session, global_role=GlobalRole.USER) + project = await create_project(session=session, owner=user, name="test-project") + await add_project_member( + session=session, project=project, user=user, project_role=ProjectRole.USER + ) + repo = await create_repo(session=session, project_id=project.id) + backend = await create_backend(session=session, project_id=project.id) + gateway = await create_gateway( + session=session, + project_id=project.id, + backend_id=backend.id, + status=GatewayStatus.RUNNING, + wildcard_domain="example.com", + ) + await create_gateway_replica(session=session, backend=backend, gateway_id=gateway.id) + project.default_gateway_id = gateway.id + await session.commit() + + response = await client.post( + f"/api/project/{project.name}/runs/apply", + headers=get_auth_headers(user.token), + json={ + "plan": { + "run_spec": get_service_run_spec(repo_id=repo.name, run_name="gateway"), + "current_resource": None, + }, + "force": False, + }, + ) + + assert response.status_code == 400 + assert response.json() == { + "detail": [ + { + "msg": ( + "Domain name 'gateway.example.com' is taken by the gateway's model" + " endpoint. Please use a different run name" + ), + "code": "error", + } + ] + } diff --git a/src/tests/_internal/server/services/proxy/routers/test_service_proxy.py b/src/tests/_internal/server/services/proxy/routers/test_service_proxy.py index 3df1404b90..b1c5fb3664 100644 --- a/src/tests/_internal/server/services/proxy/routers/test_service_proxy.py +++ b/src/tests/_internal/server/services/proxy/routers/test_service_proxy.py @@ -1,7 +1,7 @@ from typing import Generator, Optional, Tuple from unittest.mock import patch -import httpx +import httpx2 as httpx import pytest from fastapi import FastAPI from fastapi.responses import PlainTextResponse diff --git a/src/tests/_internal/server/services/services/test_services.py b/src/tests/_internal/server/services/services/test_services.py index 1ce03c52a6..5121b083ec 100644 --- a/src/tests/_internal/server/services/services/test_services.py +++ b/src/tests/_internal/server/services/services/test_services.py @@ -12,6 +12,7 @@ from dstack._internal.core.models.runs import RunSpec from dstack._internal.server.services.services import ( _should_show_service_https, + get_gateway_endpoint_url, should_configure_service_https_on_gateway_replica, ) from dstack._internal.server.testing.common import get_run_spec @@ -114,3 +115,17 @@ def test_false_disables_https_regardless_of_gateway_certificate(self) -> None: run_spec = _service_run_spec(https=False) gw = _gateway_config(certificate=LetsEncryptGatewayCertificate()) assert _should_show_service_https(run_spec, gw) is False + + +class TestGetGatewayEndpointUrl: + def test_returns_path_of_given_project(self) -> None: + url = get_gateway_endpoint_url( + "https://test-run.example.com", _gateway_config(), "other-project" + ) + assert url == "https://gateway.example.com/other-project" + + def test_uses_http_when_gateway_has_no_certificate(self) -> None: + url = get_gateway_endpoint_url( + "http://test-run.example.com", _gateway_config(certificate=None), "test-project" + ) + assert url == "http://gateway.example.com/test-project"