Repository navigation
fix(preprocessors): make reg_state value expiration work and stay in order - #8
Open
adamsrnmsu wants to merge 1 commit into
Open
adamsrnmsu wants to merge 1 commit into
adamsrnmsu wants to merge 1 commit into
Conversation
…order Bug: with value_expiration_sec set, registers never expired under the default name map. With an identity name map they did expire, but the blanked documents could carry timestamps earlier than documents already yielded, so output was out of time order. Root cause: the expiry check looked up reg_last_updated keys, which are raw (src, dst, addr) tuples, in current_state, which is keyed by the renamed columns, so the lookup missed whenever the name map was not the identity. Expiry was also only evaluated when a doc changed a value, and expiry docs were emitted with reg_update + value_expiration_sec after the current state had already moved past that time. Fix: map keys through the name map when checking and blanking, evaluate expirations before every incoming doc, and treat each expiry as an event in time order: flush the pending state, advance the state clock to the expiry time, then blank the register. A register refreshed before its expiry is not expired. Verified: - new tests/test_reg_state_expiration.py: 3 of 4 fail on main, 4 of 4 pass with the fix - full pytest: the same 21 failures and 1 error as main and nothing new (Elasticsearch tests with no server running, plus the known macOS-only test_build_paths_uses_var_lib_for_manifest_state) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
reg_ops_to_reg_state(..., value_expiration_sec=N)is meant to blank a register that has not been updated for N seconds. It did not work:reg_last_updatedkeys, which are raw(src, dst, addr)tuples, incurrent_state, which is keyed by the renamed columns, so the lookup always missed.last update + Nafter the state had already been yielded past that time, so the output went backwards in time.Fix: keys go through the name map when checking and blanking, expirations are evaluated before every incoming document, and each expiry is handled as an event in time order (flush the pending state, move the state clock to the expiry time, then blank the register). A register refreshed before its expiry is not expired.
Related Issues/PRs
value_expiration_sectoday, so default behavior is unchanged.reg_state.py; whichever merges second needs a rebase.Type of Change
fix)feat)docs)refactor)chore)Checklist
Testing
tests/test_reg_state_expiration.py(4 tests): expiry fires with the default and the identity name map, output timestamps never go backwards, the expiry document is stamped at last update + N, and a refreshed register does not expire. 3 of the 4 fail onmain; all pass with the fix.mainand no new ones (Elasticsearch tests that need a server, plus the macOS-onlytest_build_paths_uses_var_lib_for_manifest_state).Additional Notes
🤖 Generated with Claude Code