Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion mkdocs/blog/posts/amd-pd-disaggregation.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,6 @@ $ curl http://localhost:3000/proxy/services/main/amd-sglang-pd-service/v1/chat/c
Requests are routed to SMG, which selects the prefill and decode workers for each request. The prefill worker processes the prompt, the decode worker continues generation, and Mooncake transfers the KV cache between them over RoCE. `dstack` registers and deregisters workers with SMG as replicas are added or removed, runs the `/health` probe on each replica, and scales each replica group independently.

!!! info "Limitations"
- Currently, only one router replica per service is supported.
- The example uses the SGLang inference backend for prefill and decode workers. vLLM backend support is coming soon.
- Autoscaling supports the RPS metric. TTFT and ITL-based autoscaling support is coming soon.

Expand Down
4 changes: 2 additions & 2 deletions mkdocs/blog/posts/nvidia-dynamo.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ With the latest update, `dstack` added native support for NVIDIA Dynamo with Pre

<!-- more -->

!!! note "Deprecation notice"
The `DSTACK_ROUTER_INTERNAL_IP` environment variable used in this blog post is no longer supported. For the modern approach, see the [NVIDIA Dynamo example](../../docs/examples/inference/dynamo.md).
!!! note "Update"
Since this post was published, `dstack` has removed the `DSTACK_ROUTER_INTERNAL_IP` environment variable used here and lifted most of the limitations listed below: the router replica group can have multiple replicas, and services with a Dynamo router support `retry` and in-place updates. For the current approach, see the [NVIDIA Dynamo example](../../docs/examples/inference/dynamo.md).

## About NVIDIA Dynamo

Expand Down
1 change: 0 additions & 1 deletion mkdocs/blog/posts/smg.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ env:
- MODEL_ID=zai-org/GLM-4.5-Air-FP8
replicas:
- count: 1
# For now replica group with router must have count: 1
commands:
- pip install smg
- |
Expand Down
4 changes: 1 addition & 3 deletions mkdocs/docs/concepts/services.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,6 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:

groups:
- replicas: 1
# For now the router group must have replicas: 1
commands:
- pip install smg
- |
Expand Down Expand Up @@ -788,8 +787,7 @@ scales the service to 4 replicas.
Setting the minimum number of replicas to `0` allows the service to scale down to zero when there are no requests.

Each [replica group](#replica-groups) can set its own `replicas` range and `scaling` rules, so
worker groups scale independently. For now, a group with [`router`](#router) must have
`replicas: 1` and cannot be scaled.
the router and worker groups scale independently.

> The `scaling` property requires creating a [gateway](gateways.md).

Expand Down
2 changes: 0 additions & 2 deletions mkdocs/docs/examples/inference/sglang.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,6 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/

groups:
- replicas: 1
# For now replica group with router must have replicas: 1
commands:
- pip install smg
- |
Expand Down Expand Up @@ -218,7 +217,6 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/

groups:
- replicas: 1
# For now replica group with router must have replicas: 1
python: "3.12"
commands:
- pip install smg
Expand Down
4 changes: 0 additions & 4 deletions src/dstack/_internal/core/models/configurations.py
Original file line number Diff line number Diff line change
Expand Up @@ -1541,10 +1541,6 @@ def validate_at_most_one_router_replica_group(self) -> Self:
router_groups = [g for g in groups if g.router is not None]
if len(router_groups) > 1:
raise ValueError("At most one replica group may specify `router`.")
if router_groups:
router_group = router_groups[0]
if router_group.replicas.min != 1 or router_group.replicas.max != 1:
raise ValueError("For now replica group with `router` must have `replicas: 1`.")
return self


Expand Down
Loading