Skip to content

refactor: bring every Python function under CRAP 10 - #5

Open
adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:refactor-crap-under-10
Open

adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:refactor-crap-under-10

Conversation

@adamsrnmsu

Copy link
Copy Markdown
Collaborator

Description

Brings every Python function in src under a CRAP score of 10 (CRAP = CC^2 * (1 - coverage)^3 + CC). A crapper run found 152 Python functions at or above 10. Functions with CC >= 10 cannot pass on coverage alone, so they were split into smaller helpers. The rest got characterization tests that pin current behavior before any structural change. About 770 new hermetic tests were added.

Groups (baseline function counts): pei views 19; pipeline (datastore, preprocessors, interfaces, gpib ingestor, datetime utils) 40; analytics core 22 (including 8 in analytics/util.py); analytics OT 16; pei rest, project and registers 29; alerts and analytics webapp 26.

parseus.analytics.util is kept: analytics load through the parseus.analytics entry-point group, so plugins may import it. Its public names are unchanged, and every function in it is now CRAP <= 9.

Related Issues/PRs

Type of Change

  • Bugfix (fix)
  • Feature (feat)
  • Documentation (docs)
  • Refactor (refactor)
  • Chore (CI, build, dependencies, etc.) (chore)
  • Other (please describe):

Primarily a refactor. It also includes bug fixes found while pinning behavior; each is listed under Behavior changes below and pinned by a test.

Checklist

Testing

export DATASTORE_HOST_URL=http://localhost:9201   # local Elasticsearch
pytest
crapper src
pylint src/parseus
dryer src
Check Before After
Python functions with CRAP >= 10 152 0 (worst 9.6)
pytest 575 passed, 1 failed 1348 passed, 1 failed
pylint src/parseus 8.79 8.94
dryer src duplicate clones 64 62 (no new pairs)

The single failing test, test_build_paths_uses_var_lib_for_manifest_state, fails on macOS only (/var symlink) and fails the same way before the change.

Additional Notes

  • JavaScript functions are not measured: JS coverage did not run in the crapper pass. Follow-up.

  • Behavior changes (each pinned by a test):

  • modbus_to_regop: an unmapped Modbus function code (for example
    MASK_WRITE_REGISTER) used to raise UnboundLocalError; it now returns
    only the MODBUS_PRESENT regop.

  • reg_state_time_bounds: no longer raises TypeError when start_timestamp
    is after every document, or when the input is empty and a
    start_timestamp is given; it emits the last known state at
    start_timestamp, then the end padding.

  • reg_ops_to_reg_state: column_pre_filter accepts a bare regex string or
    a callable (a string used to be split into characters and a callable
    raised TypeError); value_expiration_sec no longer raises KeyError when
    two or more registers expire at once.

  • sample_weights: weights were 1000x too small when the index was not
    nanosecond precision (pandas 3); the index is now cast to datetime64[ns].

  • OTDevice: a record missing host, description or ip no longer raises
    TypeError/KeyError; missing fields are None. device_identifier is
    unchanged for full records. The private setters and _device_identifier
    were removed; find_device compares the public device_identifier.

  • DSData block re-query: substitutes the block time bounds for dict-form
    bool clauses such as filter: {range} (used to raise RuntimeError:
    dictionary changed size during iteration) and skips non-list clauses
    such as minimum_should_match (used to raise TypeError).

  • SysmonRuleForm: used to crash with ValueError ('has no field named
    allowed_windows') when the model rejected an allowed window; model
    errors now show on allowed_windows_json, the model clean runs once and
    a bad timezone is reported once.

  • SysmonRule.clean: a non-string entry in days raised TypeError in the
    message join; it now gives the 'invalid day(s)' ValidationError.

  • analytics view: only the four analytic_types linked from
    analytic_base.html (NetworkSupervised, NetworkUnsupervised,
    PeatSupervised, PeatUnsupervised) render a list; any other value,
    including 'All', now raises Http404. It used to raise
    UnboundLocalError (no template was assigned). Nothing links to 'All'
    and analytics_list.html is only an include, so there was no intent to
    restore.

  • nanquantile: integer quantiles (the documented Union[float, int]) used
    to raise a ufunc casting error; they now work. Results for float
    quantiles are unchanged.

  • pei/tasks.py _finish_scan_after_process_error: a scan stopped by the
    user between the two refreshes no longer logs "No supported devices
    were identified; marking scan complete."; status handling is unchanged.

  • pei views: removed an unreachable status branch; containment checks use
    Path.is_relative_to (same results). peat_run_file still serves a
    symlink that resolves inside the run dir (pinned by a test, unchanged).

🤖 Generated with Claude Code

Why: an ub-mode evaluation with crapper found 152 Python functions in src
with CRAP >= 10 (CRAP = CC^2*(1-cov)^3 + CC, where CC is cyclomatic
complexity and cov is test coverage of the function). A function with
CC >= 10 cannot get under 10 by coverage alone, so those were split into
smaller helpers; the rest got characterization tests that pin current
behavior before any structural change.

What changed, by group (counts are baseline functions with CRAP >= 10):
- pei views, 19: src/parseus/webapp/pei/views.py split into helpers.
- pipeline, 40: datastore client/data/query_api, preprocessors
  (reg_state, reg_ops, reg_annotations, transform, peat_normalize, ...),
  interfaces (config, ot_device, tasks, common), gpib ingestor,
  util/datetime_utils.
- analytics core, 22: Sysmon unapproved-device/ip/user analytics,
  host_anomalies rules, NewCommunications, Analytic base, builtin
  registration, and 8 functions in analytics/util.py. util.py stays
  (analytics are loaded through the parseus.analytics entry-point group,
  so plugins may import it); nanquantile, WeightedStandardScaler and
  RobustStandardScaler.fit were split into helpers and every function in
  the module now has CRAP <= 9.
- analytics OT, 16: ot_internals, ot_network function-code and scpi
  analytics; mostly tests with golden data, small splits in src.
- pei rest, project and registers, 29: pei consumers, models,
  peat_locks, tasks, signals, peat_process; project views; registers
  views and utility.
- alerts and analytics webapp, 26: alerts forms/models/tasks/views,
  analytics views/tasks/utility_analytics, populate_analytics.
- About 770 new hermetic tests under tests/.

Verification:
- pytest (DATASTORE_HOST_URL=http://localhost:9201): before 575 passed,
  1 failed; after 1348 passed, 1 failed (the same macOS-only /var symlink
  test, test_build_paths_uses_var_lib_for_manifest_state)
- crapper src: Python functions with CRAP >= 10: 152 before, 0 after
  (worst is 9.6). JavaScript functions are not measured: JS coverage did
  not run.
- pylint src/parseus: 8.79 before, 8.94 after (CI gate --fail-under=7.5)
- dryer src: 64 duplicate clones before, 62 after; no new clone pairs

Behavior changes (each pinned by a test):
- modbus_to_regop: an unmapped Modbus function code (for example
  MASK_WRITE_REGISTER) used to raise UnboundLocalError; it now returns
  only the MODBUS_PRESENT regop.
- reg_state_time_bounds: no longer raises TypeError when start_timestamp
  is after every document, or when the input is empty and a
  start_timestamp is given; it emits the last known state at
  start_timestamp, then the end padding.
- reg_ops_to_reg_state: column_pre_filter accepts a bare regex string or
  a callable (a string used to be split into characters and a callable
  raised TypeError); value_expiration_sec no longer raises KeyError when
  two or more registers expire at once.
- sample_weights: weights were 1000x too small when the index was not
  nanosecond precision (pandas 3); the index is now cast to datetime64[ns].
- OTDevice: a record missing host, description or ip no longer raises
  TypeError/KeyError; missing fields are None. device_identifier is
  unchanged for full records. The private setters and _device_identifier
  were removed; find_device compares the public device_identifier.
- DSData block re-query: substitutes the block time bounds for dict-form
  bool clauses such as filter: {range} (used to raise RuntimeError:
  dictionary changed size during iteration) and skips non-list clauses
  such as minimum_should_match (used to raise TypeError).
- SysmonRuleForm: used to crash with ValueError ('has no field named
  allowed_windows') when the model rejected an allowed window; model
  errors now show on allowed_windows_json, the model clean runs once and
  a bad timezone is reported once.
- SysmonRule.clean: a non-string entry in days raised TypeError in the
  message join; it now gives the 'invalid day(s)' ValidationError.
- analytics view: only the four analytic_types linked from
  analytic_base.html (NetworkSupervised, NetworkUnsupervised,
  PeatSupervised, PeatUnsupervised) render a list; any other value,
  including 'All', now raises Http404. It used to raise
  UnboundLocalError (no template was assigned). Nothing links to 'All'
  and analytics_list.html is only an include, so there was no intent to
  restore.
- nanquantile: integer quantiles (the documented Union[float, int]) used
  to raise a ufunc casting error; they now work. Results for float
  quantiles are unchanged.
- pei/tasks.py _finish_scan_after_process_error: a scan stopped by the
  user between the two refreshes no longer logs "No supported devices
  were identified; marking scan complete."; status handling is unchanged.
- pei views: removed an unreachable status branch; containment checks use
  Path.is_relative_to (same results). peat_run_file still serves a
  symlink that resolves inside the run dir (pinned by a test, unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant