Skip to content

[Rewriter] Match omitted Transpose perm and support commutative Add in matmul_add_to_gemm - #3211

Open
Hardik K (hardikkaurani) wants to merge 10 commits into
microsoft:mainfrom
hardikkaurani:fix/matmul-add-to-gemm-perm-and-commutative
Open

Hardik K (hardikkaurani) wants to merge 10 commits into
microsoft:mainfrom
hardikkaurani:fix/matmul-add-to-gemm-perm-and-commutative

Conversation

@hardikkaurani

@hardikkaurani Hardik K (hardikkaurani) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR improves the matmul_add_to_gemm rewrite rules by supporting omitted default perm attributes in input Transpose nodes, validating perm attributes safely, and matching commutative Add operand orders.

Problem

  1. Omitted Default Transpose Perm: ONNX allows the perm attribute on Transpose to be omitted. For rank-2 tensors, the default permutation is [1, 0]. Previously, the rewrite rules explicitly required perm=[1, 0], rejecting valid models where the attribute was omitted.

  2. Missing Perm Validation: Function reference attributes were not checked, and explicitly provided permutations were not consistently validated in check().

  3. Commutative Add: Existing rules only matched Add(MatMul(...), C), missing equivalent expressions in the order Add(C, MatMul(...)).

Solution

  • Relaxed Transpose Patterns: Removed hardcoded perm=[1, 0] requirements from TransAMatMulAddToGemm, TransBMatMulAddToGemm, and TransABMatMulAddToGemm.
  • Added Perm Validation: Introduced _is_valid_2d_transpose to accept omitted perm attributes, reject function reference attributes, and validate explicit permutations against [1, 0].
  • Contracting Dimension Validation: Added checks to ensure the inner dimensions of the inputs match.
  • Commutative Add Support: Added four rewrite rules to recognize Add(C, MatMul(...)) across the standard and transposed variants.
  • Regression Tests: Added coverage for omitted and explicit perm attributes, invalid permutations, function reference attributes, commutative operand ordering, and numerical equivalence.

…l_add_to_gemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…dToGemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…dToGemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…ddToGemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…tributes safely

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…o_gemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…se perm in Gemm fusion

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 14, 2026 22:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@hardikkaurani

Hardik K (hardikkaurani) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi Ti-Tai Wang (@titaiwangms),

I've opened this PR to improve the matmul_add_to_gemm rewrite rules:

  • Omitted default Transpose perm: In ONNX, the perm attribute on Transpose is optional and defaults to reversing dimensions (which for 2D inputs yields [1, 0]). Relaxed the patterns in TransAMatMulAddToGemm, TransBMatMulAddToGemm, and TransABMatMulAddToGemm so that valid models with omitted perm attributes are properly matched.
  • Transpose perm validation: Added _is_valid_2d_transpose in check() to ensure perm attributes are validated, non-transposing perms are rejected, and function reference attributes are safely rejected.
  • Contracting dimension check: Added validation ensuring inner contracting dimensions match between inputs.
  • Commutative Add support: Added commutative rewrite rules matching Add(C, MatMul(...)) to fuse regardless of Add operand order.
  • Tests: Added comprehensive regression tests covering omitted perm, explicit perm=[1, 0], invalid perm rejection, reference attribute rejection, commutative Add, and numerical equivalence against original models.

All local tests pass (25 dedicated unit tests and all 396 common rewriter rules). Whenever you have time, I'd appreciate your review and feedback. Thank you!

…l equivalence in Gemm fusion

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new dimension-equality check blocks valid dynamic-shape MatMul–Add fusion by requiring static equality where matching contracting dimensions need only be enforced at runtime.

1 open issue

Review effort: Balanced

…l_add_to_gemm

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…acting dimensions in Gemm fusion

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
@hardikkaurani

Copy link
Copy Markdown
Contributor Author

Hi Ti-Tai Wang (@titaiwangms),

I've pushed updates addressing the Copilot review feedback:

  • Relaxed contracting dimension validation for dynamic shapes: In _matmul_add_to_gemm.py, replaced the strict same_dim equality requirement with a check that rejects only provable concrete dimension mismatches (isinstance(k_a, int) and isinstance(k_b, int) and k_a != k_b). Models with unknown or differently named symbolic contracting dimensions (e.g. [4, "K_a"] and ["K_b", 16]) are now safely permitted to fuse into Gemm.
  • Regression test coverage: Added unit tests in _matmul_add_to_gemm_test.py verifying that dynamic models with differently named symbolic contracting dimensions fuse to Gemm, while concrete contracting dimension mismatches are reliably rejected.

All 27 dedicated tests in _matmul_add_to_gemm_test.py and all 398 common rewriter rules tests pass cleanly. Please let me know if any further changes are needed. Thank you!

@hardikkaurani

Copy link
Copy Markdown
Contributor Author

Thanks for the review and approval, Ti-Tai Wang (@titaiwangms)! All tests and checks are passing cleanly. Ready to be merged whenever convenient.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants