Skip to content

feat(api): support ad hoc dashboard filters in the external API - #3659

Open
knudtty wants to merge 9 commits into
aaron/adhoc-follow-upfrom
aaron/adhoc-external-api
Open

knudtty wants to merge 9 commits into
aaron/adhoc-follow-upfrom
aaron/adhoc-external-api

Conversation

@knudtty

@knudtty knudtty commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #3658.

Summary

The external v2 dashboards API (/api/v2/dashboards) can now create, read, and update dashboards with ADHOC filters. Until now it hid them on GET and rejected them on POST/PUT. PromQL external API support is already on main, so PromQL ad hoc filters (sourceType: "promql") are included alongside SQL ones.

  • Schema: ADHOC is a new variant in the external filter schemas, with fields named as stored (sources, not sourceIds) so no translation is needed. Source IDs must be valid ObjectIds. As with STATIC_LIST, isBroadcastEnabled and isVariableEnabled each have one allowed value (true), so callers can omit them.
  • Saved values: savedFilterValues accepts and returns adhoc entries, which hold a filter's conditions.
  • Validation: every ID in sources and appliesToSourceIds must exist and be a kind the filter's sourceType supports: Log, Trace, or Session for sql, PromQL for promql. The internal and external APIs now share one helper (utils/filterSources.ts) for this check, for both ADHOC and PROMETHEUS_LABEL filters. On PUT, references a stored filter with the same ID already made are skipped, so a source that was later deleted or changed kind does not block a GET → PUT round trip. PROMETHEUS_LABEL filters with a missing or non-PromQL source now get the shared error message, "The following filter source IDs do not exist or are not a kind of source the filter supports: …", instead of "PROMETHEUS_LABEL filters require a PromQL source…" or "Could not find the following source IDs…".
  • MCP: unchanged. Its read paths pass a new hideAdhoc option to convertToExternalDashboard, and its input schemas are separate. MCP support is the next PR in this stack.
  • OpenAPI: adds the AdhocFilterInput, AdhocSavedFilterValue, and AdhocFilterCondition schemas and regenerates openapi.json.

How to reproduce

Starting point: yarn dev, with a personal API key from Team Settings and a Logs source ID.

On main:

  1. POST /api/v2/dashboards with filters: [{ "type": "ADHOC", "name": "Conditions", "sourceType": "sql", "sources": ["<logs-source-id>"] }] returns 400.
  2. GET on a dashboard that already has an ad hoc filter, created in the UI, returns filters: [].

On this branch:

  1. The same POST returns 200, and the filter comes back with isBroadcastEnabled: true and isVariableEnabled: true.
  2. Adding savedFilterValues: [{ "type": "adhoc", "name": "Conditions", "conditions": [{ "key": "ServiceName", "operator": "IN", "value": "api" }] }] stores and returns the conditions. Opening the dashboard shows them in the filter input.
  3. Pointing sources at a PromQL source while keeping sourceType: "sql" returns 400 with that source ID in the message.
  4. GET followed by PUT of the response body round-trips unchanged.

How to test on Vercel preview

N/A. API-only change.

Testing

  • make dev-int FILE=src/routers/external-api/__tests__/dashboards.int.test.ts passes, and covers GET, POST for SQL and PromQL, a GET → PUT round-trip, schema rejections, and source-kind rejections.
  • make dev-int FILE=src/mcp/__tests__/dashboards passes, so MCP behavior is unchanged.
  • New unit test for convertToExternalDashboard({ hideAdhoc }).

References

The v2 dashboards API now creates, reads, and updates ADHOC filters (SQL and
PromQL source types) and their adhoc saved conditions. Sources and
appliesToSourceIds must exist and match the filter's sourceType. MCP keeps
hiding ad hoc filters via convertToExternalDashboard's hideAdhoc option.
@knudtty
knudtty requested a review from a team as a code owner November 13, 2026 21:28
@vercel

vercel Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 3:54pm UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 3:54pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c8d6a2d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

* Returns an error message if an ad hoc filter reads from or applies to a
* source of a kind its `sourceType` can't query, or null if none does.
*/
function getAdhocFilterSourceKindError(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Ad hoc source-kind check copies the internal API's and drops its skip for unchanged references, so GET → PUT can fail

getFilterSourceIds and getAdhocFilterSourceKindError repeat getFilterSourceRefs/validateFilterSources in packages/api/src/routers/api/dashboards.ts:68-109. The internal copy skips references the stored filter with the same ID already had, so a source that was later deleted or changed kind doesn't block unrelated saves. This copy checks every ADHOC filter on each PUT (dashboards.ts:3738). So if a UI-created ad hoc filter points at a source that has since been deleted or changed kind, a verbatim GET → PUT now returns 400. That contradicts the PR's claim that "the internal API enforces the same rule". Fix: move the ref-building and existing-ref skip into one shared helper, for example in common-utils filters or packages/api/src/utils/, and pass existingDashboard.filters from the PUT path, as the heatmap and formula gates already do with existingTiles.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor

1 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/mcp/tools/dashboards/saveDashboard.ts:323 — MCP update still re-checks every stored PROMETHEUS_LABEL filter's source, unlike the REST PUT and internal PATCH → Pass existingFilters: existingDashboard.filters ?? [] to validateDashboardTiles here. The value is already loaded at line 314, and assignFilterIds keeps the IDs of existing filters. Without it, an MCP get→save of a dashboard whose PROMETHEUS_LABEL source was later deleted is rejected, while the same round trip through /api/v2/dashboards PUT (dashboards.ts:3743) now succeeds.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The ad hoc filter support is well-scoped and the shared filterSources.ts extraction is sound. The source-reference validation correctly rejects cross-team and wrong-kind source IDs, cannot be bypassed by colliding a new source under an existing filter ID (the skip set is keyed on id:kinds, so a new source produces an unseen key and is re-validated), and the GET → PUT round-trip holds for both SQL and PromQL ad hoc filters.

🟡 P2 — recommended

  • packages/api/src/routers/external-api/v2/utils/dashboards.ts:1811 — On PUT, validateDashboardTiles wraps every incoming filter with translateExternalFilterToFilter({ id: '', ...filter }) and relies on the request body's id winning the spread; a WithId body omitting id (or sending id: "") would silently fall into the create-path branch and skip the stored-reference reconciliation, re-validating sources the stored filter already referenced.
    • Fix: Add an integration test asserting a PUT that omits a filter id is rejected by the WithId schema, and a test confirming a since-deleted/kind-changed source referenced only by an unchanged stored filter still round-trips through PUT.
🔵 P3 nitpicks (2)
  • packages/api/src/routers/external-api/v2/utils/dashboards.ts:808 — The filters array in convertToExternalDashboard is built from a nested ternary ((hideAdhoc ? getMcpCompatibleFilters(...) : (dashboard.filters ?? [])).filter(...).map(...)) that is dense to parse.
    • Fix: Extract the source filter list into a named local before .filter().map().
  • packages/api/src/mcp/tools/dashboards/queryTile.ts:71 — The jscpd bot flagged two clone blocks across the MCP dashboard tools, but these are pre-existing structural repetition of the register-tool pattern; this diff only appended one hideAdhoc: true line to each.
    • Fix: Optionally fold the repeated convertToExternalDashboard + find-tile block into a shared helper as part of this extraction effort; not required.

Reviewers (2 returned; 9 dispatched did not return before synthesis): previous-comments, learnings-researcher. Correctness, security, adversarial, api-contract, kieran-typescript, testing, maintainability, project-standards, and agent-native were dispatched but had not reported at synthesis time; their areas were covered by direct orchestrator analysis of the diff and the underlying common-utils schemas.

Testing gaps:

  • No visible coverage for the stored-reference skip on PUT when a referenced source was deleted or had its kind changed between GET and PUT (the round-trip-still-succeeds case), nor for the inverse where a newly added ad hoc source must be re-validated on PUT.
  • No visible coverage for a SQL ad hoc filter carrying where/whereLanguage surviving a GET → PUT round-trip verbatim.
  • MCP write-path behavior when a dashboard already holds ad hoc filters the agent cannot see (hideAdhoc: true) was not independently confirmed to be clobber-safe.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

2 new clones introduced by this PR.

  • packages/api/src/mcp/tools/dashboards/getDashboardTile.ts:44-59 ↔ packages/api/src/mcp/tools/dashboards/queryTile.ts:71-86 — 16 lines
  • packages/api/src/mcp/tools/dashboards/queryTile.ts:64-83 ↔ packages/api/src/mcp/tools/dashboards/queryTiles.ts:170-189 — 20 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Critical-path files (2) — tenancy, public API, or shipped database config:
    • packages/api/src/routers/external-api/v2/dashboards.ts
    • packages/api/src/routers/external-api/v2/utils/dashboards.ts
  • Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 15
  • Production lines changed: 639 (+ 438 in test files, excluded from tier calculation)
  • Critical-path lines changed: 238
  • Branch: aaron/adhoc-external-api
  • Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 465 passed • 1 skipped • 1150s

Status Count
✅ Passed 465
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

… on PUT

The internal and external dashboard APIs now build filter source references
and report bad ones through one helper. The external PUT passes the stored
filters, so an ad hoc filter whose source was later deleted or changed kind
no longer blocks a GET -> PUT round trip.
…r on PUT

Label filters now go through the same filter source check as ad hoc filters,
so references the stored filter already made are skipped and a GET -> PUT
round trip survives a deleted or changed PromQL source. The wrong-kind
appliesToSourceIds test now asserts the source-kind check caused the 400.
Comment thread packages/api/src/utils/filterSources.ts Resolved
Comment thread packages/api/src/utils/zod.ts Resolved
…d lookup

PromQL ad hoc filters never read where/whereLanguage, so the external API now
rejects them (and GET drops ones the internal API may have stored, keeping a
GET -> PUT round trip valid). getNonPromqlSourceError is replaced by a shared
getInvalidSourceIds that the filter source check also uses.
Comment thread packages/api/src/utils/filterSources.ts Resolved
Comment thread packages/api/src/utils/filterSources.ts Resolved

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — c8d6a2dc Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-oss — c8d6a2dc Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant