Skip to content

Apply robots specificity to blanket allow rules - #43

Merged
mandiau-matthias merged 1 commit into
mainfrom
fix/robots-specificity
Nov 16, 2026
Merged

mandiau-matthias merged 1 commit into
mainfrom
fix/robots-specificity

Conversation

@mandiau-matthias

Copy link
Copy Markdown
Contributor

Follow-up to #39: the equivalent-allow fix treated every root-covering rule as equal.

Problem

RFC 9309 §2.2.2 chooses the most specific match, with an equally specific allow winning. Disallow: /* (2 octets) is more specific than Allow: / (1 octet), so that pair should still cap. The previous code let any root allow lift any root disallow.

What changed

  • rootCoveringLength recognizes patterns that permit every path: /, /*, /**, /*$ and longer star runs.
  • The probe tracks the longest root allow and disallow, keeps the allow-wins tie rule, and caps only when the disallow is strictly more specific.
  • Adds wildcard cases: Disallow: /* + Allow: / blocks; equal /* allows; more specific /** allows.
  • Clarifies the API wording and changelog.

Before and after

robots.txt for * Before After
Disallow: /*, Allow: / not capped capped at 25
Disallow: /*, Allow: /* not capped not capped
Disallow: /*, Allow: /** not capped not capped
Disallow: /, Allow: / not capped not capped
Disallow: /, Allow: /public capped capped

Verification

  • npm run check — exit 0, 145 unit tests, 5 integration tests, package and bundle checks
  • npm run test:browser — exit 0
  • Focused probe tests pass

Boundary

This is still a heuristic for blanket blocks, not a complete robots policy engine or an authorization check.

The equivalent-allow fix treated every root-covering rule as equal. RFC 9309 chooses the most specific match, so Disallow: /* (2 octets) still blocks when paired with Allow: / (1 octet), while Allow: /** lifts it.

Track the longest root-covering allow and disallow patterns, keep the allow-wins tie rule, and add wildcard cases to the probe harness. Narrow allows still do not lift a blanket disallow.
@mandiau-matthias
mandiau-matthias merged commit b2315f9 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant