Skip to content

Ignore a UTF-8 BOM in robots.txt and refresh status - #44

Merged
mandiau-matthias merged 1 commit into
mainfrom
fix/robots-bom-docs
Nov 16, 2026
Merged

mandiau-matthias merged 1 commit into
mainfrom
fix/robots-bom-docs

Conversation

@mandiau-matthias

Copy link
Copy Markdown
Contributor

Problem

A robots.txt file may start with a UTF-8 BOM. The parser treated the first key as \uFEFFuser-agent, so a blanket disallow was missed and the site scored as open. The launch status also still reported 143 unit tests and said no UI code had changed.

What changed

  • Strips one leading BOM before parsing robots.txt and adds a regression case for \uFEFFUser-agent: * + Disallow: /, which now caps at 25.
  • Adds the browser workflow badge to the README.
  • Updates the maintainer continuity paragraph to 146 unit tests and records that the headless Chrome checks run locally and on Ubuntu.
  • Removes the stale sentence that no UI code had changed or browser review was claimed.

Verification

  • npm run check — exit 0, 146 unit tests, 5 integration tests, package and bundle checks
  • npm run test:browser — exit 0
  • Probe tests — 10 pass, including the BOM case

Boundary

The robots parser remains a heuristic for blanket blocks, not a complete policy engine. This is a parser fix and status correction; no new user or assistive-technology evidence.

A leading BOM made the first User-agent line unrecognized, so a blanket disallow could be missed. Strip it before parsing and pin the case in the probe harness.\n\nAdd the browser workflow badge to the README and update the maintainer continuity paragraph from 143 to 146 unit tests, including the new browser check. No UI behavior changes in this commit.
@mandiau-matthias
mandiau-matthias merged commit d69b4b6 into main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant