Repository navigation
fix(registers): give hdbscan a writable array so the State tab works - #6
Conversation
The Registers page's State tab (GET /registers/state/state/) answered every request with 400 "Cannot create writable memory view from read-only memoryview". With pandas 3 (pinned 3.0.6) copy-on-write, DataFrame.values returns a read-only array, and hdbscan 0.8.44's Cython code cannot take a read-only buffer. cluster_features' single-column fit always hit it; make_clustered_states_df's fit is routed the same way for safety. Both fits now go through one helper, _fit, which hands hdbscan a writable float copy (np.array(values, dtype=float)). Verified: - new tests/test_state_reduction.py: the cluster_features test fails on main with the memoryview error and passes with the fix; both tests assert the clustering result (two levels -> two labels, states numbered 1, 2). - full pytest: 535 passed, 1 failed (test_build_paths_uses_var_lib_for_ manifest_state, macOS-only, fails the same on main). - live: against a range's Elasticsearch data, the same State request returned 400 before and 200 after, and the tab rendered clustered states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ethane98
left a comment
There was a problem hiding this comment.
test(registers): cover the read-only buffer path in make_clustered_states_df
Follow-up to PR #6. The PR's
test_make_clustered_states_df_numbers_states_in_order_of_appearance passes
on main as well as with the fix, so it does not guard the _fit change in
make_clustered_states_df. Its 2-column frame yields an F-ordered read-only
array from .values, and hdbscan copies that into a writable C-ordered array
before its Cython code runs, so the memoryview error never fires.
- Add test_make_clustered_states_df_accepts_read_only_float_values: a
single float column gives a C-contiguous read-only array that hdbscan
uses as-is. On main it fails with "Cannot create writable memory view
from read-only memoryview"; with the fix it passes and numbers states
1, 2. - In test_cluster_features_separates_each_registers_levels, assert no
label is -1 (noise), so "one half all noise, other half one cluster"
no longer passes.
Verified with pandas 3.0.6, hdbscan 0.8.44, numpy 2.5.4:
tests/test_state_reduction.py passes 3/3 on this branch; against main's
src, the cluster_features test and the new test fail with the memoryview
error, and the state-numbering test passes.
Co-Authored-By: Claude noreply@anthropic.com
…ates_df Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, good catch on the F-ordered copy. I added test_make_clustered_states_df_accepts_read_only_float_values and the no-noise assert as suggested. The new test fails with the memoryview error when make_clustered_states_df skips _fit and passes with it, and tests/test_state_reduction.py is 3/3 on pandas 3.0.6 and hdbscan 0.8.44. |
Drive the real web UI in Chromium: seed Malcolm-shaped Modbus documents (two IP pairs, three registers, two alternating states) into a unique Elasticsearch index, load it on /registers/, open an IP pair, tick two registers and check the chart draws, then check the State tab. - tests/e2e/ uses pytest-playwright with pytest-django's live_server; the fixture deletes its index afterwards. - e2e tests carry an `e2e` marker and are deselected by default (addopts = -m 'not e2e'), so the existing pytest job needs no browser. Run them with `pytest -m e2e tests/e2e`. - New `e2e` CI job (python:3.13 container + Elasticsearch service) installs Chromium and uploads failure screenshots as an artifact. - test_state_tab checks that the State tab loads and draws its chart, which works since sandialabs#6 gave hdbscan a writable array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drive the real web UI in Chromium: seed Malcolm-shaped Modbus documents (two IP pairs, three registers, two alternating states) into a unique Elasticsearch index, load it on /registers/, open an IP pair, tick two registers and check the chart draws, then check the State tab. - tests/e2e/ uses pytest-playwright with pytest-django's live_server; the fixture deletes its index afterwards. - e2e tests carry an `e2e` marker and are deselected by default (addopts = -m 'not e2e'), so the existing pytest job needs no browser. Run them with `pytest -m e2e tests/e2e`. - New `e2e` CI job (python:3.13 container + Elasticsearch service) installs Chromium and uploads failure screenshots as an artifact. - test_state_tab checks that the State tab loads and draws its chart, which works since #6 gave hdbscan a writable array. Co-authored-by: Ryan Adams <radams2@unm.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Description
The Registers page's State tab (
GET /registers/state/state/) returned400 Bad Request: Cannot create writable memory view from read-only memoryviewfor every request, so "Register State Over Time" and the other state graphs never rendered.Cause: with pandas 3 (pinned
pandas==3.0.6) copy-on-write,DataFrame.valuesreturns a read-only array, and hdbscan 0.8.44's Cython code cannot take a read-only buffer.cluster_features' single-column fit (values.reshape(-1, 1)) always hit it.Fix: both hdbscan fits in
preprocessors/state_reduction.pygo through one helper,_fit, which passes a writable float copy (np.array(values, dtype=float)). Nothing else changes.Related Issues/PRs
state_reduction.pyorwebapp/registers/utility.py, so no conflict.Type of Change
fix)feat)docs)refactor)chore)Checklist
Testing
tests/test_state_reduction.py. Thecluster_featurestest fails onmainwith the memoryview error and passes with the fix. Both tests assert the clustering result: two register levels give two labels, and states are numbered 1 then 2 in order of appearance.DATASTORE_HOST_URLpointed at a throwaway Elasticsearch 8.19.5): 535 passed, 1 failed. The failure istest_build_paths_uses_var_lib_for_manifest_state, which is macOS-only and fails the same way onmain.state_reduction.py: 9.51 → 9.52. codespell clean.Additional Notes
Other hdbscan/sklearn call sites were checked: these two fits are the only hdbscan uses in
src.🤖 Generated with Claude Code