Skip to content

Deduplicate shared tars in flatten_all optimization plans - #12

Open
Mohith26 wants to merge 1 commit into
Physical-Intelligence:mainfrom
Mohith26:fix/flatten-all-plan-dedupe
Open

Mohith26 wants to merge 1 commit into
Physical-Intelligence:mainfrom
Mohith26:fix/flatten-all-plan-dedupe

Conversation

@Mohith26

@Mohith26 Mohith26 commented Nov 3, 2026

Copy link
Copy Markdown

When two targets in the same layer group provide the same tar (for example two SizeHintInfo wrappers around one file), the normal planning path deduplicates it, but the flatten_all path built its file list from the raw group.files. The resulting optimization_plan.json listed the tar twice in the flattened layer, reported summary.input_tar_count: 2 while the group entry said 1, and double-counted known_input_size_bytes. The emitted layer itself was already fine since DefaultInfo goes through a depset, so this only affects the plan.

Repro on main, with :heavy_a and :heavy_b both wrapping the same 8 MiB tar in a packages group and flatten_all = True:

  • summary: input_tar_count: 2
  • layers[0]: 2 inputs, known_input_size_bytes: 16777216
  • groups[0]: input_tar_count: 1

The fix applies the same _deduplicate the non-flatten path already uses. Cross-group duplicates still fail as before.

Testing: added flatten_all_plan_test, which reads the plan from the rule's write action. It fails on main ("Expected 1, but got 2") and passes with the change. bazel test //layers/tests:optimize_layers_test //inference/... //apt/... passes and buildifier format/lint is clean on the changed files. I couldn't run the py_test targets locally (they get killed in my sandbox on unmodified main too), and I didn't run the e2e modules, though e2e/rules_python/optimization_plan.json has no duplicate inputs so its snapshot shouldn't change.

When two targets in one layer group provide the same tar, the normal
planning path deduplicates it, but flatten_all did not. The plan then listed
the tar twice, reported input_tar_count 2 while the group entry said 1, and
double-counted known_input_size_bytes. The emitted layer was already correct
because DefaultInfo uses a depset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant