Skip to content

fix(datastore): make push() fail loudly and stop mutating shared query - #11

Open
adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:fix-push-retry-safe
Open

adamsrnmsu wants to merge 1 commit into
sandialabs:mainfrom
adamsrnmsu:fix-push-retry-safe

Conversation

@adamsrnmsu

Copy link
Copy Markdown
Collaborator

Description

datastore/query_api.py push() deletes the existing documents by query and then bulk-pushes the new ones. It ignored the False that bulk_push_docs returns on a failed document, so a failed push silently lost the data it had just deleted. The grouped path also appended to and popped from one shared query["bool"]["must"] list inside its loop.

Fix: a fresh query per group, and RuntimeError when bulk_push_docs returns False. The only caller, push_results_to_es, already catches and logs exceptions, so a failure is now logged as an error instead of "results pushed".

True atomicity (push first, then delete only the older documents) is not possible with the current client API: the documents have no ids to exclude, so deleting at or after the minimum timestamp after the push would delete the new documents too. This PR makes the failure loud instead.

Related Issues/PRs

Type of Change

  • Bugfix (fix)
  • Feature (feat)
  • Documentation (docs)
  • Refactor (refactor)
  • Chore (CI, build, dependencies, etc.) (chore)
  • Other (please describe):

Checklist

Testing

  • New tests/test_datastore_push.py (5 tests): ungrouped delete-then-push order, grouped pushes use independent queries, no delete when overwrite_old is False, a failed push raises, input documents unchanged. On the old code the grouped and failed-push tests fail.
  • Full suite on macOS without an Elasticsearch server: the same failures as main and no new ones (Elasticsearch-server tests, plus the macOS-only test_build_paths_uses_var_lib_for_manifest_state).

Additional Notes

With grouping, a failure part-way through the per-group deletes still leaves the earlier groups deleted; this PR only makes the push failure visible.

🤖 Generated with Claude Code

push() deleted existing documents by query and then ignored the False
returned by bulk_push_docs, so a failed push silently lost the deleted
data. The grouped path also appended to and popped from one shared
query["bool"]["must"] list inside a loop.

Fix: build a fresh query per group, and raise RuntimeError when
bulk_push_docs returns False. The only caller
(push_results_to_es) already catches and logs exceptions, so it now
reports the failure instead of "results pushed". True atomicity (push,
then delete only older docs) is not possible with the current client
API: documents have no ids to exclude, so a delete >= min timestamp
after the push would remove the new documents too. Failure is made
loud instead.

Verified:
- tests/test_datastore_push.py: 5 pass; on the old code the grouped
  independent-query and failed-push tests fail.
- Full suite failures match the base (Elasticsearch-only + macOS path).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant