Skip to content

fix(preprocessors): make reg_state value expiration work and stay in order - #8

Open
adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:fix-regstate-value-expiration
Open

adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:fix-regstate-value-expiration

Conversation

@adamsrnmsu

Copy link
Copy Markdown
Collaborator

Description

reg_ops_to_reg_state(..., value_expiration_sec=N) is meant to blank a register that has not been updated for N seconds. It did not work:

  • With the default name map nothing ever expired. The check looked up reg_last_updated keys, which are raw (src, dst, addr) tuples, in current_state, which is keyed by the renamed columns, so the lookup always missed.
  • With an identity name map expiry fired, but the blanked documents were stamped last update + N after the state had already been yielded past that time, so the output went backwards in time.

Fix: keys go through the name map when checking and blanking, expirations are evaluated before every incoming document, and each expiry is handled as an event in time order (flush the pending state, move the state clock to the expiry time, then blank the register). A register refreshed before its expiry is not expired.

Related Issues/PRs

Type of Change

  • Bugfix (fix)
  • Feature (feat)
  • Documentation (docs)
  • Refactor (refactor)
  • Chore (CI, build, dependencies, etc.) (chore)
  • Other (please describe):

Checklist

Testing

  • New tests/test_reg_state_expiration.py (4 tests): expiry fires with the default and the identity name map, output timestamps never go backwards, the expiry document is stamped at last update + N, and a refreshed register does not expire. 3 of the 4 fail on main; all pass with the fix.
  • Full suite on macOS without an Elasticsearch server: the same 21 failures and 1 error as main and no new ones (Elasticsearch tests that need a server, plus the macOS-only test_build_paths_uses_var_lib_for_manifest_state).

Additional Notes

  • Expirations still pending at the end of the stream are not flushed, as before.
  • With an expiry window set, the expiry scan runs on every incoming document (O(registers) each). Fine for the current, unused feature; a heap would avoid it if it ever matters.

🤖 Generated with Claude Code

…order

Bug: with value_expiration_sec set, registers never expired under the
default name map. With an identity name map they did expire, but the
blanked documents could carry timestamps earlier than documents already
yielded, so output was out of time order.

Root cause: the expiry check looked up reg_last_updated keys, which are
raw (src, dst, addr) tuples, in current_state, which is keyed by the
renamed columns, so the lookup missed whenever the name map was not the
identity. Expiry was also only evaluated when a doc changed a value, and
expiry docs were emitted with reg_update + value_expiration_sec after the
current state had already moved past that time.

Fix: map keys through the name map when checking and blanking, evaluate
expirations before every incoming doc, and treat each expiry as an event
in time order: flush the pending state, advance the state clock to the
expiry time, then blank the register. A register refreshed before its
expiry is not expired.

Verified:
- new tests/test_reg_state_expiration.py: 3 of 4 fail on main, 4 of 4
  pass with the fix
- full pytest: the same 21 failures and 1 error as main and nothing
  new (Elasticsearch tests with no server running, plus the known
  macOS-only test_build_paths_uses_var_lib_for_manifest_state)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant