Skip to content

avoid cubic ~ matching in xpath_indirect_adjacent_combinator - #227

Open
itecz26 wants to merge 2 commits into
scrapy:masterfrom
itecz26:indirect-adjacent-first-sibling
Open

itecz26 wants to merge 2 commits into
scrapy:masterfrom
itecz26:indirect-adjacent-first-sibling

Conversation

@itecz26

@itecz26 itecz26 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

a ~ b was translated to a/following-sibling::b, so when many siblings match a their following-sibling sets overlap and libxml2 merges each one in with a linear duplicate scan, which makes matching cubic in the list length (lxml 6.1.3 / libxml2 2.14.6: li ~ li takes 1.5 s on a 19 KiB page of 2000 <li>, 13 s at 4000 and 154 s at 8000, same story for dt ~ dd and li ~ li ~ li, so one fetched page is enough to stall a spider); the siblings after the first match cover those after every later one, so for a plain name test on the left xpath_indirect_adjacent_combinator() now only walks from that one via [position() = 1 or not(preceding-sibling::a[1])] (position() = 1 keeps the first match from each context node, i.e. the context node itself or the first match after each node of a ~ chain), which brings li ~ li down to 2 ms at 8000 siblings and 37 ms at 128000.

That test is only valid when the step it's on is reached through an axis that gives a whole sibling list, a contiguous run of it starting at the context node, or a single node. Steps inside the expression and inside :has() always are (child, descendant, following-sibling), but the first step depends on the prefix, so for the first step the condition is only added when the prefix is descendant-or-self:: (the default), descendant::, child::, self::, parent::, following-sibling::, preceding-sibling:: or empty. Any other prefix (following::, preceding::, ancestor::, ancestor-or-self::, //, ...) keeps the old translation for that step. Old and new translations select the same elements on ~1.2M random selector/document/context-node checks across all of those prefixes.

Not covered: a left side with a condition, e.g. li.a ~ li or li:not(.x) ~ li, is still translated as before and is still cubic, because copying the condition into the predicate would double the XPath at each nested :has() level.

a ~ b was translated to a/following-sibling::b. When many siblings
match a, their following-sibling node-sets overlap, and libxml2 merges
each one into the result with a linear duplicate scan, so matching was
cubic in the number of siblings: li ~ li took 154 s on a list of 8000
items. The siblings after the first match of a in a sibling list
include those after every later one, so only walk from that one, found
with preceding-sibling::a[1], which stops at the nearest match.
position() = 1 keeps the first match from each context node, which
matters when the context node itself matches and has matching siblings
before it, and in ~ chains.

Only a plain name test is handled: a condition would have to be copied
into the predicate, doubling the XPath at each nested :has() level.
@codecov

codecov Bot commented Nov 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (1730a48) to head (abebd6a).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #227   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines         1036      1038    +2     
  Branches       174       175    +1     
=========================================
+ Hits          1036      1038    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@AdrianAtZyte

Copy link
Copy Markdown
Contributor

Thoughts on this LLM feedback?

I think this changes results for some prefix values, since the prefix is prepended after translation and the rewrite assumes the left side was found through a descendant/child-like axis. E.g. with <r><li id='1'/><li id='2'><u><li id='a'/><li id='b'><x id='c'/></li><li id='d'/></u></li><li id='3'/><li id='4'/></r> and c as context node:

css_to_xpath("li ~ li", prefix="following::")         → master: ['4'], PR: []
css_to_xpath("li ~ li", prefix="ancestor-or-self::")  → master: ['d', '3', '4'], PR: ['d']

Could the rewrite be limited to prefixes where it is equivalent? Also, please, mention in the title or description that li.a ~ li is still affected.

@itecz26

itecz26 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

it's right. the position() = 1 test assumes the first step comes from an axis that returns whole sibling lists (or a contiguous run starting at the context node), which following:: and ancestor-or-self:: don't. i'd also flagged ancestor::, following:: and preceding:: as differing in my own checks but left them alone, and that was the wrong call.

pushed e1737ad: the first step now only gets the condition when the prefix is one where it's equivalent (descendant-or-self::, descendant::, child::, self::, parent::, following-sibling::, preceding-sibling::, empty). any other prefix keeps the old translation for that step. later steps and steps inside :has() are always reached through child/descendant/following-sibling, so they keep it. your example now gives ['4'] and ['d', '3', '4'] like master, and it's in the tests. a random old-vs-new comparison (~1.2M checks, also covering ancestor::, ancestor-or-self::, following::, preceding:: and //) found no differences.

also updated the description to say li.a ~ li (any left side with a condition) is still affected.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants