Repository navigation
refactor: bring every Python function under CRAP 10 - #5
Open
adamsrnmsu wants to merge 1 commit into
Open
adamsrnmsu wants to merge 1 commit into
adamsrnmsu wants to merge 1 commit into
Conversation
Why: an ub-mode evaluation with crapper found 152 Python functions in src
with CRAP >= 10 (CRAP = CC^2*(1-cov)^3 + CC, where CC is cyclomatic
complexity and cov is test coverage of the function). A function with
CC >= 10 cannot get under 10 by coverage alone, so those were split into
smaller helpers; the rest got characterization tests that pin current
behavior before any structural change.
What changed, by group (counts are baseline functions with CRAP >= 10):
- pei views, 19: src/parseus/webapp/pei/views.py split into helpers.
- pipeline, 40: datastore client/data/query_api, preprocessors
(reg_state, reg_ops, reg_annotations, transform, peat_normalize, ...),
interfaces (config, ot_device, tasks, common), gpib ingestor,
util/datetime_utils.
- analytics core, 22: Sysmon unapproved-device/ip/user analytics,
host_anomalies rules, NewCommunications, Analytic base, builtin
registration, and 8 functions in analytics/util.py. util.py stays
(analytics are loaded through the parseus.analytics entry-point group,
so plugins may import it); nanquantile, WeightedStandardScaler and
RobustStandardScaler.fit were split into helpers and every function in
the module now has CRAP <= 9.
- analytics OT, 16: ot_internals, ot_network function-code and scpi
analytics; mostly tests with golden data, small splits in src.
- pei rest, project and registers, 29: pei consumers, models,
peat_locks, tasks, signals, peat_process; project views; registers
views and utility.
- alerts and analytics webapp, 26: alerts forms/models/tasks/views,
analytics views/tasks/utility_analytics, populate_analytics.
- About 770 new hermetic tests under tests/.
Verification:
- pytest (DATASTORE_HOST_URL=http://localhost:9201): before 575 passed,
1 failed; after 1348 passed, 1 failed (the same macOS-only /var symlink
test, test_build_paths_uses_var_lib_for_manifest_state)
- crapper src: Python functions with CRAP >= 10: 152 before, 0 after
(worst is 9.6). JavaScript functions are not measured: JS coverage did
not run.
- pylint src/parseus: 8.79 before, 8.94 after (CI gate --fail-under=7.5)
- dryer src: 64 duplicate clones before, 62 after; no new clone pairs
Behavior changes (each pinned by a test):
- modbus_to_regop: an unmapped Modbus function code (for example
MASK_WRITE_REGISTER) used to raise UnboundLocalError; it now returns
only the MODBUS_PRESENT regop.
- reg_state_time_bounds: no longer raises TypeError when start_timestamp
is after every document, or when the input is empty and a
start_timestamp is given; it emits the last known state at
start_timestamp, then the end padding.
- reg_ops_to_reg_state: column_pre_filter accepts a bare regex string or
a callable (a string used to be split into characters and a callable
raised TypeError); value_expiration_sec no longer raises KeyError when
two or more registers expire at once.
- sample_weights: weights were 1000x too small when the index was not
nanosecond precision (pandas 3); the index is now cast to datetime64[ns].
- OTDevice: a record missing host, description or ip no longer raises
TypeError/KeyError; missing fields are None. device_identifier is
unchanged for full records. The private setters and _device_identifier
were removed; find_device compares the public device_identifier.
- DSData block re-query: substitutes the block time bounds for dict-form
bool clauses such as filter: {range} (used to raise RuntimeError:
dictionary changed size during iteration) and skips non-list clauses
such as minimum_should_match (used to raise TypeError).
- SysmonRuleForm: used to crash with ValueError ('has no field named
allowed_windows') when the model rejected an allowed window; model
errors now show on allowed_windows_json, the model clean runs once and
a bad timezone is reported once.
- SysmonRule.clean: a non-string entry in days raised TypeError in the
message join; it now gives the 'invalid day(s)' ValidationError.
- analytics view: only the four analytic_types linked from
analytic_base.html (NetworkSupervised, NetworkUnsupervised,
PeatSupervised, PeatUnsupervised) render a list; any other value,
including 'All', now raises Http404. It used to raise
UnboundLocalError (no template was assigned). Nothing links to 'All'
and analytics_list.html is only an include, so there was no intent to
restore.
- nanquantile: integer quantiles (the documented Union[float, int]) used
to raise a ufunc casting error; they now work. Results for float
quantiles are unchanged.
- pei/tasks.py _finish_scan_after_process_error: a scan stopped by the
user between the two refreshes no longer logs "No supported devices
were identified; marking scan complete."; status handling is unchanged.
- pei views: removed an unreachable status branch; containment checks use
Path.is_relative_to (same results). peat_run_file still serves a
symlink that resolves inside the run dir (pinned by a test, unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Brings every Python function in
srcunder a CRAP score of 10 (CRAP = CC^2 * (1 - coverage)^3 + CC). A crapper run found 152 Python functions at or above 10. Functions with CC >= 10 cannot pass on coverage alone, so they were split into smaller helpers. The rest got characterization tests that pin current behavior before any structural change. About 770 new hermetic tests were added.Groups (baseline function counts): pei views 19; pipeline (datastore, preprocessors, interfaces, gpib ingestor, datetime utils) 40; analytics core 22 (including 8 in
analytics/util.py); analytics OT 16; pei rest, project and registers 29; alerts and analytics webapp 26.parseus.analytics.utilis kept: analytics load through theparseus.analyticsentry-point group, so plugins may import it. Its public names are unchanged, and every function in it is now CRAP <= 9.Related Issues/PRs
src/parseus/preprocessors/reg_ops.pyandsrc/parseus/datastore/query_api.py.Type of Change
fix)feat)docs)refactor)chore)Primarily a refactor. It also includes bug fixes found while pinning behavior; each is listed under Behavior changes below and pinned by a test.
Checklist
Testing
The single failing test,
test_build_paths_uses_var_lib_for_manifest_state, fails on macOS only (/varsymlink) and fails the same way before the change.Additional Notes
JavaScript functions are not measured: JS coverage did not run in the crapper pass. Follow-up.
Behavior changes (each pinned by a test):
modbus_to_regop: an unmapped Modbus function code (for example
MASK_WRITE_REGISTER) used to raise UnboundLocalError; it now returns
only the MODBUS_PRESENT regop.
reg_state_time_bounds: no longer raises TypeError when start_timestamp
is after every document, or when the input is empty and a
start_timestamp is given; it emits the last known state at
start_timestamp, then the end padding.
reg_ops_to_reg_state: column_pre_filter accepts a bare regex string or
a callable (a string used to be split into characters and a callable
raised TypeError); value_expiration_sec no longer raises KeyError when
two or more registers expire at once.
sample_weights: weights were 1000x too small when the index was not
nanosecond precision (pandas 3); the index is now cast to datetime64[ns].
OTDevice: a record missing host, description or ip no longer raises
TypeError/KeyError; missing fields are None. device_identifier is
unchanged for full records. The private setters and _device_identifier
were removed; find_device compares the public device_identifier.
DSData block re-query: substitutes the block time bounds for dict-form
bool clauses such as filter: {range} (used to raise RuntimeError:
dictionary changed size during iteration) and skips non-list clauses
such as minimum_should_match (used to raise TypeError).
SysmonRuleForm: used to crash with ValueError ('has no field named
allowed_windows') when the model rejected an allowed window; model
errors now show on allowed_windows_json, the model clean runs once and
a bad timezone is reported once.
SysmonRule.clean: a non-string entry in days raised TypeError in the
message join; it now gives the 'invalid day(s)' ValidationError.
analytics view: only the four analytic_types linked from
analytic_base.html (NetworkSupervised, NetworkUnsupervised,
PeatSupervised, PeatUnsupervised) render a list; any other value,
including 'All', now raises Http404. It used to raise
UnboundLocalError (no template was assigned). Nothing links to 'All'
and analytics_list.html is only an include, so there was no intent to
restore.
nanquantile: integer quantiles (the documented Union[float, int]) used
to raise a ufunc casting error; they now work. Results for float
quantiles are unchanged.
pei/tasks.py _finish_scan_after_process_error: a scan stopped by the
user between the two refreshes no longer logs "No supported devices
were identified; marking scan complete."; status handling is unchanged.
pei views: removed an unreachable status branch; containment checks use
Path.is_relative_to (same results). peat_run_file still serves a
symlink that resolves inside the run dir (pinned by a test, unchanged).
🤖 Generated with Claude Code