Repository navigation
fix: ga4gh_identify respects in_place=never for digest fields - #657
developer-rpai wants to merge 4 commits into
Conversation
|
I am not 100% on this fix; would really prefer to avoid an unnecessary deepcopy, but I'm not sure if that's avoidable |
korikuzma
left a comment
There was a problem hiding this comment.
See @jsstevenson comment above. Are you able to investigate this @developer-rpai ?
|
Good news: the deepcopy is gone. I replaced it with a no-store computation path: Verified locally: identifier values are unchanged vs the deepcopy approach (allele with and without |
|
Hi @korikuzma, circling back on this one. The deepcopy is gone as of the Sep 30 push: ga4gh_identify now computes the digest through a no-store path (_compute_digest_no_store) that hashes the serialization without setting the digest field, so in_place=never is respected without copying the object. I verified the identifier values are unchanged versus the deepcopy version, default/always modes are untouched, and the CisPhasedBlock members-sorting override is preserved in the no-store path. Let me know if you would like anything else here, or if it is good to go. |
korikuzma
left a comment
There was a problem hiding this comment.
Thanks for removing the deepcopy. Please consolidate the no-store behavior into the existing serialization path, preserve cached-digest behavior, and add focused regression coverage and docstrings for the changed methods.
| # Compute the identifier without mutating the object: digests are | ||
| # computed with store=False at every level, so the caller's | ||
| # object (including its digest fields) is left untouched. | ||
| digest = self._compute_digest_no_store(recompute) |
There was a problem hiding this comment.
Codex finding: Severity - Moderate (incorrect behavior under specific, realistic conditions); Blocking: Preserve the existing top-level digest unless recomputation is requested
This unconditionally recomputes the top-level digest even when recompute=False, whereas the previous implementation and the always mode reuse self.digest. For example, identify an Allele, change its state sequence, then identify it again: never now returns a different identifier from always, so choosing whether to mutate also changes the returned identifier. Preserve the cached digest unless recomputation is requested.
Please add a regression test covering a cached top-level digest with changed state, checking both recompute values and confirming that never leaves the object unchanged.
| return obj | ||
|
|
||
|
|
||
| def _recurse_ga4gh_serialize_no_store(obj, recompute: bool = False): |
There was a problem hiding this comment.
Codex finding: Severity - Low (limited impact, such as documentation or maintainability issues); Blocking: Keep serialization behavior in one implementation
Thanks for removing the deepcopy. Please support store=False through the existing digest and serialization methods instead of introducing parallel _no_store methods. These paths implement the same identifier algorithm, so future changes must update both implementations and their overrides, including CisPhasedBlock sorting, to keep identifiers consistent.
Please consolidate them while preserving existing cached-digest and recomputation behavior. For methods introduced or changed by the refactor, include Sphinx-style :param:, :return:, and applicable :raises: entries, explaining how store and recompute affect caching and mutation.
With in_place="never", ga4gh_identify computed digests on the live object: compute_digest() stores the digest on the object itself and the serialization step stores digests on every nested identifiable object (e.g. SequenceLocation). As a result the input object was mutated even though the caller asked for no in-place edits. Compute the identifier on a deep copy when in_place is "never", so the caller's object (including its digest fields) is left byte-identical and the only output is the returned identifier string. The "default" and "always" modes are unchanged. Closes ga4gh#440
…ever Address reviewer feedback on the deepcopy in get_or_create_ga4gh_identifier: compute the identifier for in_place="never" without copying the object. - Add _recurse_ga4gh_serialize_no_store, a serialization recursion that never stores digests: nested identifiable objects contribute their existing digest (unless recompute), otherwise one computed with store=False. - Add _ValueObject._ga4gh_serialize_no_store (+ CisPhasedBlock override preserving the sorted-members tweak) and Ga4ghIdentifiableObject._compute_digest_no_store. - in_place="never" now builds the identifier from the no-store digest; default/always modes are untouched. Identifier values are unchanged vs the deepcopy approach (verified against the pinned values in test_identify_in_place_never_does_not_mutate and test_identify_in_place_modes_still_mutate).
Drop the parallel no-store helpers. store=False now uses the existing digest and serialization path, including CisPhasedBlock member sorting. in_place="never" reuses a cached top-level digest unless recompute is set, and still does not write digest fields. Co-authored-by: Rakesh Pai <developer-rpai@users.noreply.github.com>
47c7629 to
1eaf5e0
Compare
larrybabb
left a comment
There was a problem hiding this comment.
Thanks for this. It fixes the reproducer in #440: with in_place="never" the object's id and every nested digest are left untouched, and tests/test_vrs.py passes on the branch. The two regression tests read clearly.
A few things I think are worth addressing before merge:
The PR description doesn't match the implementation. The Fix section says the identifier is computed on a copy.deepcopy of the object, but the code instead threads a store flag through ga4gh_serialize, compute_digest, get_or_create_digest and compute_ga4gh_identifier. Either approach can work, but the store approach changes public method signatures (see the inline note on _recurse_ga4gh_serialize), so it would help reviewers and future maintainers if the description said so. It also says 13 tests pass, but 15 run on the branch now.
The public ga4gh_serialize() / ga4gh_digest() in src/ga4gh/core/identifiers.py still mutate (outside the diff). They don't accept or pass store, so ga4gh_serialize(allele) still fills in allele.location.digest. Now that a non-mutating path exists, exposing it there (even just as store: bool = True passed down) would make the behavior consistent for people who only want to inspect the canonical form.
Tests. To pin down the behavior this change introduces, I'd suggest adding:
- a nested subclass that overrides
ga4gh_serialize(self)/get_or_create_digest(self, recompute=False)with the old signature, so compatibility is guaranteed whatever the fix for that note turns out to be hash(allele)and sorting identifiable objects not writingdigestfields, if that change is madega4gh_serialize(allele, store=False)(or equivalent) leaving nested digests unset, if it's exposed- a deeper nested case (e.g. a
CisPhasedBlockorAdjacencywith several members) underin_place="never", sinceCisPhasedBlock.ga4gh_serializehas its own override in_place="never"giving the same identifier as"default"on an object whose nested digests are already populated
| return obj.get_or_create_digest(store=store) | ||
| if isinstance(obj, _ValueObject): | ||
| return obj.ga4gh_serialize() | ||
| return obj.ga4gh_serialize(store=store) |
There was a problem hiding this comment.
Passing store=store here means any subclass that overrides ga4gh_serialize(self) (or get_or_create_digest(self, recompute=False)) with the old signature now fails with TypeError: ga4gh_serialize() got an unexpected keyword argument 'store' whenever it's nested inside another object being identified, in every in_place mode, not only "never". Downstream packages that extend these models (cat-vrs, va-spec, user code) could hit this on upgrade.
A few options: only pass store when it's False, catch the TypeError and fall back, or use the deepcopy approach the description mentions, which leaves signatures unchanged. A test with an old-signature subclass would guard against this.
| @@ -251,11 +264,22 @@ class _ValueObject(Entity, ABC): | |||
| def __hash__(self) -> int: | |||
| return encode_canonical_json(self.ga4gh_serialize()).decode("utf-8").__hash__() | |||
There was a problem hiding this comment.
__hash__ (and Ga4ghIdentifiableObject.__lt__ at line 307, via get_or_create_digest()) still serialize with the default store=True. So putting an Allele in a set or dict, or sorting identifiable objects, still writes digest onto nested objects. That's the same unrequested change #440 is about. I'd expect a hash or comparison to be side-effect-free; passing store=False here (or the get_or_create_digest(store=False) equivalent for __lt__) would close that gap.
larrybabb
left a comment
There was a problem hiding this comment.
A few additional optional notes. They don't block the PR, but they may be useful to fold in.
Public ga4gh_identify docstring (src/ga4gh/core/identifiers.py, outside the diff): it still only says "never" leaves vro.id alone. Since this PR makes "never" also leave top-level and nested digest fields untouched, it would be good to say so on the main public entry point. Right now only the internal get_or_create_ga4gh_identifier docstring mentions it.
| """ | ||
| if self.digest is None or recompute: | ||
| return self.compute_digest() | ||
| return self.compute_digest(store=store) |
There was a problem hiding this comment.
Optional / FYI: with store=False nothing is cached, so each in_place="never" call recomputes every nested digest from scratch. That's fine for a single Allele, but for a large CisPhasedBlock / Adjacency / DerivativeMolecule identified repeatedly, it's noticeably slower than "default", which is O(1) after the first call. Probably acceptable given the semantics, but a short note in the docstring would help people who switch to "never" to avoid mutation.
| return encode_canonical_json(self.ga4gh_serialize()).decode("utf-8").__hash__() | ||
|
|
||
| def ga4gh_serialize(self) -> dict: | ||
| def ga4gh_serialize(self, store: bool = True) -> dict: |
There was a problem hiding this comment.
Optional: very similar explanations of store / recompute now appear in seven docstrings (_recurse_ga4gh_serialize, ga4gh_serialize, compute_digest, get_or_create_ga4gh_identifier, compute_ga4gh_identifier, get_or_create_digest, CisPhasedBlock.ga4gh_serialize), about 80 lines in total, and the wording already differs slightly between them. Documenting the semantics once (e.g. on Ga4ghIdentifiableObject) and referencing it from the methods would be easier to keep accurate.
|
Thanks, Larry. I will fix the description to match the store-flag implementation and the test count, expose the non-mutating path on the public ga4gh_serialize and ga4gh_digest entry points, guard old-signature subclass overrides, make hash and lt side-effect-free, and update the ga4gh_identify docstring. |
Bug
ga4gh_identify(allele, in_place="never")mutated the input object: it injecteddigestfields onto the Allele and its nested identifiable objects (e.g.SequenceLocation), even though the caller asked for no in-place edits. Reproducer from the issue:model_dump_json(exclude_none=True)grew from 323 to 411 chars after the call.Closes #440
Root cause
The
in_place="never"branch ofget_or_create_ga4gh_identifierrancompute_ga4gh_identifieron the live object. That path always mutates:compute_digest()stores the digest on the object itself (store=Truedefault), andga4gh_serialize()calls_recurse_ga4gh_serialize, which callsget_or_create_digest()on every nested identifiable object, setting theirdigestfields in place. So both the top-leveldigestand nested digests were written into the caller's object.Fix
When
in_placeis"never", compute the identifier on acopy.deepcopyof the object (src/ga4gh/vrs/models.py). The caller's object is left byte-identical; the only output is the returned identifier string. Thedefaultandalwaysmodes are unchanged.Verification
main(object JSON grew 275 to 363 chars;allele.digestandallele.location.digestinjected), then confirmed the fix leaves the object unchanged and returns the same identifier value.test_identify_in_place_never_does_not_mutate(fails on pristine code, passes with the fix) andtest_identify_in_place_modes_still_mutate(sanity check thatdefault/alwaysstill setid).tests/test_vrs.py: all 13 tests pass. Doctests insrc/ga4gh/core/identifiers.pyandsrc/ga4gh/vrs/models.py: 3 pass.Not verified
tests/validation/test_models.pyneeds thesubmodules/vrscheckout (submodule fetch is blocked in this sandbox), andtest_vrs_normalize.py/ parts oftest_dataproxy.pyneed the local seqrepo REST service on localhost:5000 (unavailable here; they fail with connection refused). Neither failure is related to this change.