Skip to content

fix(registers): give hdbscan a writable array so the State tab works - #6

Merged
Ethane98 merged 2 commits into
sandialabs:mainfrom
adamsrnmsu:fix-state-tab-readonly-hdbscan
Nov 19, 2026
Merged

Ethane98 merged 2 commits into
sandialabs:mainfrom
adamsrnmsu:fix-state-tab-readonly-hdbscan

Conversation

@adamsrnmsu

Copy link
Copy Markdown
Collaborator

Description

The Registers page's State tab (GET /registers/state/state/) returned 400 Bad Request: Cannot create writable memory view from read-only memoryview for every request, so "Register State Over Time" and the other state graphs never rendered.

Cause: with pandas 3 (pinned pandas==3.0.6) copy-on-write, DataFrame.values returns a read-only array, and hdbscan 0.8.44's Cython code cannot take a read-only buffer. cluster_features' single-column fit (values.reshape(-1, 1)) always hit it.

Fix: both hdbscan fits in preprocessors/state_reduction.py go through one helper, _fit, which passes a writable float copy (np.array(values, dtype=float)). Nothing else changes.

Related Issues/PRs

Type of Change

  • Bugfix (fix)
  • Feature (feat)
  • Documentation (docs)
  • Refactor (refactor)
  • Chore (CI, build, dependencies, etc.) (chore)
  • Other (please describe):

Checklist

Testing

  • New tests/test_state_reduction.py. The cluster_features test fails on main with the memoryview error and passes with the fix. Both tests assert the clustering result: two register levels give two labels, and states are numbered 1 then 2 in order of appearance.
  • Full suite (DATASTORE_HOST_URL pointed at a throwaway Elasticsearch 8.19.5): 535 passed, 1 failed. The failure is test_build_paths_uses_var_lib_for_manifest_state, which is macOS-only and fails the same way on main.
  • pylint on state_reduction.py: 9.51 → 9.52. codespell clean.
  • Live: with Modbus/DNP3 register data from a running range in Elasticsearch, the same State request returned 400 before the fix and 200 after, and the tab rendered the clustered states.

Additional Notes

Other hdbscan/sklearn call sites were checked: these two fits are the only hdbscan uses in src.

🤖 Generated with Claude Code

The Registers page's State tab (GET /registers/state/state/) answered every
request with 400 "Cannot create writable memory view from read-only
memoryview". With pandas 3 (pinned 3.0.6) copy-on-write, DataFrame.values
returns a read-only array, and hdbscan 0.8.44's Cython code cannot take a
read-only buffer. cluster_features' single-column fit always hit it;
make_clustered_states_df's fit is routed the same way for safety.

Both fits now go through one helper, _fit, which hands hdbscan a writable
float copy (np.array(values, dtype=float)).

Verified:
- new tests/test_state_reduction.py: the cluster_features test fails on main
  with the memoryview error and passes with the fix; both tests assert the
  clustering result (two levels -> two labels, states numbered 1, 2).
- full pytest: 535 passed, 1 failed (test_build_paths_uses_var_lib_for_
  manifest_state, macOS-only, fails the same on main).
- live: against a range's Elasticsearch data, the same State request
  returned 400 before and 200 after, and the tab rendered clustered states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Ethane98 Ethane98 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test(registers): cover the read-only buffer path in make_clustered_states_df

Follow-up to PR #6. The PR's
test_make_clustered_states_df_numbers_states_in_order_of_appearance passes
on main as well as with the fix, so it does not guard the _fit change in
make_clustered_states_df. Its 2-column frame yields an F-ordered read-only
array from .values, and hdbscan copies that into a writable C-ordered array
before its Cython code runs, so the memoryview error never fires.

  • Add test_make_clustered_states_df_accepts_read_only_float_values: a
    single float column gives a C-contiguous read-only array that hdbscan
    uses as-is. On main it fails with "Cannot create writable memory view
    from read-only memoryview"; with the fix it passes and numbers states
    1, 2.
  • In test_cluster_features_separates_each_registers_levels, assert no
    label is -1 (noise), so "one half all noise, other half one cluster"
    no longer passes.

Verified with pandas 3.0.6, hdbscan 0.8.44, numpy 2.5.4:
tests/test_state_reduction.py passes 3/3 on this branch; against main's
src, the cluster_features test and the new test fail with the memoryview
error, and the state-numbering test passes.

Co-Authored-By: Claude noreply@anthropic.com

Comment thread tests/test_state_reduction.py Resolved
Comment thread tests/test_state_reduction.py Resolved
…ates_df

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adamsrnmsu

Copy link
Copy Markdown
Collaborator Author

Thanks, good catch on the F-ordered copy. I added test_make_clustered_states_df_accepts_read_only_float_values and the no-noise assert as suggested. The new test fails with the memoryview error when make_clustered_states_df skips _fit and passes with it, and tests/test_state_reduction.py is 3/3 on pandas 3.0.6 and hdbscan 0.8.44.

@Ethane98
Ethane98 merged commit b23ecd2 into sandialabs:main Nov 19, 2026
4 checks passed
adamsrnmsu pushed a commit to adamsrnmsu/parseus that referenced this pull request Nov 19, 2026
Drive the real web UI in Chromium: seed Malcolm-shaped Modbus documents
(two IP pairs, three registers, two alternating states) into a unique
Elasticsearch index, load it on /registers/, open an IP pair, tick two
registers and check the chart draws, then check the State tab.

- tests/e2e/ uses pytest-playwright with pytest-django's live_server; the
  fixture deletes its index afterwards.
- e2e tests carry an `e2e` marker and are deselected by default
  (addopts = -m 'not e2e'), so the existing pytest job needs no browser.
  Run them with `pytest -m e2e tests/e2e`.
- New `e2e` CI job (python:3.13 container + Elasticsearch service) installs
  Chromium and uploads failure screenshots as an artifact.
- test_state_tab checks that the State tab loads and draws its chart, which
  works since sandialabs#6 gave hdbscan a writable array.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adamsrnmsu added a commit that referenced this pull request Nov 19, 2026
Drive the real web UI in Chromium: seed Malcolm-shaped Modbus documents
(two IP pairs, three registers, two alternating states) into a unique
Elasticsearch index, load it on /registers/, open an IP pair, tick two
registers and check the chart draws, then check the State tab.

- tests/e2e/ uses pytest-playwright with pytest-django's live_server; the
  fixture deletes its index afterwards.
- e2e tests carry an `e2e` marker and are deselected by default
  (addopts = -m 'not e2e'), so the existing pytest job needs no browser.
  Run them with `pytest -m e2e tests/e2e`.
- New `e2e` CI job (python:3.13 container + Elasticsearch service) installs
  Chromium and uploads failure screenshots as an artifact.
- test_state_tab checks that the State tab loads and draws its chart, which
  works since #6 gave hdbscan a writable array.

Co-authored-by: Ryan Adams <radams2@unm.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants