Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 31 additions & 16 deletions mkdocs/docs/concepts/presets.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ trials: 5
min_context_length: 1048576
max_ttft: 675

# The number of simultaneous requests every benchmark uses
concurrency: 1
# A list of simultaneous request counts every benchmark uses.
concurrency: [4, 8, 16]

# The request shape every benchmark uses (defaults to 1024 and 1024)
input_tokens: 10000
Expand All @@ -55,7 +55,7 @@ To create the preset, pass the configuration to the `dstack apply` command:
$ dstack apply -f preset.dstack.yml --fleet b200-fleet
Create the preset dsv4-flash? [y/n]: y
[2026-08-04 11:38:34] Starting preset creation for deepseek-ai/DeepSeek-V4-Flash. Allowed fleets: b200-fleet.
[2026-08-04 12:31:19] Trial 3 switched from vLLM to SGLang: 319 tok/s per user, 2.2x the baseline.
[2026-08-04 12:31:19] Trial 3 switched from vLLM to SGLang: 319/495/570 tok/s per user at concurrency 4/8/16, 2.1x/1.9x/1.7x the baseline.
[2026-08-04 13:04:52] Final service dsv4-flash-c83375b4-4 verified with context length 1048576.
[2026-08-04 13:12:07] Benchmark via sglang.bench_serving: 32/32 requests succeeded.
```
Expand Down Expand Up @@ -255,9 +255,9 @@ A pulled preset works like any other, and is named `<project>/<name>`:

```shell
$ dstack preset list -a
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
main/dsv4-flash-b200 8f065dde deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=1 tps/user=309 ttft=213ms ctx=1M pulled 2 min ago
qwen35-pro6000 092c792b Qwen/Qwen3.5-397B-A17B io=8K/1K c=64 tps/user=19.6 ttft=3.43s ctx=32K verified (7) 3 days ago
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
main/dsv4-flash-b200 8f065dde deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=4..16 tps/user=319..570 ttft=213ms..465ms ctx=1M pulled 2 min ago
qwen35-pro6000 092c792b Qwen/Qwen3.5-397B-A17B io=8K/1K c=64 tps/user=19.6 ttft=3.43s ctx=32K verified (7) 3 days ago
```

</div>
Expand Down Expand Up @@ -336,8 +336,8 @@ Use `dstack preset` to list presets:

```shell
$ dstack preset list
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
dsv4-flash-b200 c83375b4 deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=1 tps/user=309 ttft=213ms ctx=1M ▂▁██▇ trialing (5/5) 2 min ago
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
dsv4-flash-b200 c83375b4 deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=4..16 tps/user=319..570 ttft=213ms..465ms ctx=1M ▂▁██▇ trialing (5/5) 2 min ago
```

</div>
Expand All @@ -348,20 +348,36 @@ By default, `dstack preset` shows creations that are still running, or the most

```shell
$ dstack preset list -a
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
dsv4-flash-b200 c83375b4 deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=1 tps/user=309 ttft=213ms ctx=1M ▂▁██▇ trialing (5/5) 2 min ago
qwen35-pro6000 092c792b Qwen/Qwen3.5-397B-A17B io=8K/1K c=64 tps/user=19.6 ttft=3.43s ctx=32K ▁▂▅▇█·· verified (7) 3 days ago
qwen36-pro4500 9ab0fa65 Qwen/Qwen3.6-27B io=1K/1K c=8 tps/user=57.1 ttft=499ms ctx=128K ▁▄██▆·█ verified (7) 4 days ago
qwen3-32b-5090 f91d6b60 Qwen/Qwen3-32B io=1K/512 c=8 tps/user=85.8 ttft=368ms ctx=32K ▁▁▅▅▄▅▇▄▇█ verified (10) 2 weeks ago
NAME ID BASE CONSTRAINTS BENCHMARK STATUS SUBMITTED
dsv4-flash-b200 c83375b4 deepseek-ai/DeepSeek-V4-Flash io=10K/1.5K c=4..16 tps/user=319..570 ttft=213ms..465ms ctx=1M ▂▁██▇ trialing (5/5) 2 min ago
qwen35-pro6000 092c792b Qwen/Qwen3.5-397B-A17B io=8K/1K c=16..64 tps/user=19.6..41.2 ttft=1.21s..3.43s ctx=32K ▁▂▅▇█·· verified (7) 3 days ago
qwen36-pro4500 9ab0fa65 Qwen/Qwen3.6-27B io=1K/1K c=2..8 tps/user=57.1..94.3 ttft=312ms..499ms ctx=128K ▁▄██▆·█ verified (7) 4 days ago
qwen3-32b-5090 f91d6b60 Qwen/Qwen3-32B io=1K/512 c=8 tps/user=85.8 ttft=368ms ctx=32K ▁▁▅▅▄▅▇▄▇█ verified (10) 2 weeks ago
```

</div>

The `CONSTRAINTS` column is what the creation was asked for, and `BENCHMARK` is the best trial so far. `tps/user` is the steady decode rate, measured as one second divided by the mean time per output token, so it excludes the time to the first token.

The glyphs after the benchmark are one per trial: height is throughput, a yellow bar is a trial whose benchmark broke a constraint, and a red `·` is one that produced no benchmark at all. The shape shows whether a run converged or wandered.
When `concurrency` is a list, `CONSTRAINTS` shows it as `c=min..max`, and the `BENCHMARK` metrics become ranges over the concurrencies that met the constraints, e.g. `tps/user=min..max` and `ttft=min..max`.

Pass `-w` to watch in realtime, `-v` for more detail, or `--json` for complete preset objects. Filter with `--base` or `--repo`.

!!! info "Sparkline legend"

Each bar in the sparklines represents one trial. The bar height is the trial's highest tok/s across concurrency values that met the constraints.

| Bar color | Semantics |
| --- | --- |
| Red | A trial that broke constraints at every concurrency. |
| Green | The best trial that met the constraints. |
| Gray | A trial that met the constraints but had lower tok/s than the best trial. |
| Yellow | The best trial, if it broke constraints at every concurrency. |

A red `·` represets a trial that produced no benchmark.

For detailed benchmark data at each concurrency, pass `--json` to `dstack preset get`.

To watch in realtime, use `-w` with `dstack preset`. Filter with `--base` or `--repo`. Also, use `-v` for verbose output.

### Delete presets

Expand Down Expand Up @@ -390,7 +406,6 @@ At the same time, it's recommended to create presets using your own agent — ei
## Limitations

* Currently, the agent doesn't upload compiled binaries anywhere; patches compile at runtime
* Doesn't support ranges for `concurrency`

> Report bugs and request features on [GitHub](https://github.com/dstackai/dstack/issues), and ask questions on [Discord](https://discord.gg/u8SmfwPpMd).

Expand Down
9 changes: 4 additions & 5 deletions src/dstack/_internal/cli/commands/preset.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from dstack._internal.cli.models.presets import (
AnyStoredPreset,
PresetListOutput,
PresetSessionEntry,
UnverifiedPreset,
)
from dstack._internal.cli.services.completion import ProjectNameCompleter
Expand Down Expand Up @@ -284,7 +285,7 @@ def _list(self, args: argparse.Namespace) -> None:

def _list_presets_and_sessions(
self, *, base: str | None, repo: str | None
) -> tuple[list[AnyStoredPreset], list[dict]]:
) -> tuple[list[AnyStoredPreset], list[PresetSessionEntry]]:
self._reconcile()
presets = PresetStore().list()
sessions = list_preset_sessions()
Expand Down Expand Up @@ -552,15 +553,13 @@ def _filter_presets(


def _session_matches_model(
session: dict,
session: PresetSessionEntry,
*,
base: str | None,
repo: str | None,
repo_to_base: dict[str, str],
) -> bool:
model = str(session.get("model") or "")
if not model:
return False
model = session.state.model
if repo is not None:
return model == repo
return model == base or repo_to_base.get(model) == base
Expand Down
82 changes: 78 additions & 4 deletions src/dstack/_internal/cli/models/preset_agent.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import uuid
from datetime import datetime
from typing import Annotated, Any, Dict, Literal, Optional, Union
from typing import Annotated, Any, Dict, Literal, Optional, Self, Union

import yaml
from pydantic import (
Expand All @@ -9,11 +9,12 @@
ValidationError,
ValidatorFunctionWrapHandler,
WrapValidator,
model_validator,
)

from dstack._internal.core.models.common import CoreModel
from dstack._internal.core.models.configurations import PresetAgentProvider, ServiceConfiguration
from dstack._internal.core.models.presets import PresetBenchmark
from dstack._internal.core.models.presets import PresetBenchmark, PresetResult


class PresetAgentInvalidService(CoreModel):
Expand Down Expand Up @@ -52,8 +53,7 @@ class PresetAgentSuccess(CoreModel):
trial: PositiveInt
base: Annotated[str, Field(min_length=1)]
model: Annotated[str, Field(min_length=1)]
context_length: PositiveInt
benchmark: PresetBenchmark
result: Annotated[list[PresetResult], Field(min_length=1)]


class PresetAgentFailure(CoreModel):
Expand All @@ -66,6 +66,80 @@ class PresetAgentFailure(CoreModel):
AnyPresetAgentResult = Union[PresetAgentSuccess, PresetAgentFailure]


class PresetTrialBenchmark(PresetBenchmark):
"""A trial's benchmark, which the agent writes into `trials/<n>/trial.json`
after benchmarking the trial at a concurrency, read by the CLI for
`dstack preset list`. `PresetBenchmark.validate_metrics` is overridden
because its rule is for the final report; applied to `trial.json` it would
drop the whole trial from the listing."""

@model_validator(mode="after")
def validate_metrics(self) -> Self:
return self


class PresetTrialResult(CoreModel):
"""One `result` entry of `trials/<n>/trial.json`, as the system prompt
defines it: the agent writes one per concurrency in `constraints.json` once
the trial is benchmarked, and the CLI reads them for `dstack preset list`.
The entry's `learned` field is left out: the CLI does not use it, because
the agent writes it for its own reading in later sessions."""

concurrency: Optional[PositiveInt] = None
context_length: Optional[PositiveInt] = None
benchmark: Optional[PresetTrialBenchmark] = None
failed: bool = False

@model_validator(mode="after")
def fail_without_benchmark(self) -> Self:
"""An entry without a benchmark is one the agent could not measure, which
the prompt defines as failed; the flag is set in case the agent left it out."""
if self.benchmark is None:
self.failed = True
return self


def _entry_or_failed(value: Any, handler: ValidatorFunctionWrapHandler) -> Any:
try:
return handler(value)
except ValidationError:
# A broken entry counts as a failed one, so the other entries of the
# trial are still listed.
return PresetTrialResult(failed=True)


class PresetTrial(CoreModel):
"""The `trials/<n>/trial.json` record as defined in the system_prompt.md.
The agent writes it once trial `<n>` is benchmarked, and its presence is
what marks the trial completed; the CLI reads it for `dstack preset list`."""

resources: Optional[dict[str, Any]] = None
groups: Optional[list[list[dict[str, Any]]]] = None
# Required and non-empty: `system_prompt.md` states that `trial.json` has
# one `result` entry per concurrency value.
result: Annotated[
list[Annotated[PresetTrialResult, WrapValidator(_entry_or_failed)]],
Field(min_length=1),
]

@model_validator(mode="before")
@classmethod
def convert_to_result_list(cls, data: Any) -> Any:
# An old `trial.json` has no `result` list. Its `benchmark`,
# `context_length` and `failed` are at the top level, and a failed one
# may have no `benchmark` at all.
if isinstance(data, dict) and "result" not in data:
data = dict(data)
data["result"] = [
{
key: data.pop(key)
for key in ("benchmark", "context_length", "failed")
if key in data
}
]
return data


PresetSessionStatus = Literal["running", "interrupted", "success", "failed"]

# The session models carry no defaults: `session.json` is this CLI's own state
Expand Down
38 changes: 37 additions & 1 deletion src/dstack/_internal/cli/models/presets.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,14 @@

from pydantic import Field, PositiveInt

from dstack._internal.cli.models.preset_agent import (
PresetSessionState,
PresetSessionStatus,
PresetTrial,
)
from dstack._internal.core.models.common import CoreModel
from dstack._internal.core.models.configurations import PresetConfiguration
from dstack._internal.core.models.presets import PortablePreset
from dstack._internal.core.models.presets import AnyPresetConstraints, PortablePreset


class BasePreset(CoreModel):
Expand Down Expand Up @@ -44,3 +49,34 @@ class PulledPreset(BasePreset, PortablePreset):

class PresetListOutput(CoreModel):
presets: List[AnyStoredPreset]


class PresetTrialsSummary(CoreModel):
"""Built by the CLI for `dstack preset list`."""

count: int
# The trial with the highest tok/s that met the constraints.
best: Optional[PresetTrial] = None
# The trial with the highest tok/s but broke the constraints.
best_failed: Optional[PresetTrial] = None
# One value per trial in order, for the sparkline; None for a trial that
# produced no benchmark.
series: list[Optional[float]]
# Parallel to `series`: True where the trial broke the constraints.
failed: list[bool]
# The hardware of the shown trial, `best` or else `best_failed`; when neither
# exists, of the last trial that recorded it, so the session still shows what
# it ran on.
gpu: Optional[str] = None


class PresetSessionEntry(CoreModel):
"""Built by the CLI for `dstack preset list`."""

id: str
state: PresetSessionState
name: Optional[str]
status: PresetSessionStatus
trials: PresetTrialsSummary
verification: Optional[str]
constraints: Optional[AnyPresetConstraints]
8 changes: 3 additions & 5 deletions src/dstack/_internal/cli/services/presets/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from dstack._internal.core.models.instances import Resources
from dstack._internal.core.models.presets import (
PRESET_EXCLUDED_FIELDS,
PresetBenchmark,
PresetResult,
PresetVerificationReplicaGroup,
)
from dstack._internal.core.models.resources import (
Expand All @@ -31,8 +31,7 @@ def build_preset(
verification_replica_groups: list[PresetVerificationReplicaGroup],
base_model: str,
repo: str,
context_length: int,
benchmark: PresetBenchmark,
result: list[PresetResult],
configuration: PresetConfiguration,
best_trial: int,
preset_id: str,
Expand All @@ -48,12 +47,11 @@ def build_preset(
base=base_model,
id=preset_id,
repo=repo,
context_length=context_length,
best_trial=best_trial,
configuration=configuration,
created_at=created_at,
service=service,
benchmark=benchmark,
result=result,
verified_on=verification_replica_groups,
)

Expand Down
Loading
Loading