Repository navigation
Ignore a UTF-8 BOM in robots.txt and refresh status - #44
Merged
Merged
Conversation
A leading BOM made the first User-agent line unrecognized, so a blanket disallow could be missed. Strip it before parsing and pin the case in the probe harness.\n\nAdd the browser workflow badge to the README and update the maintainer continuity paragraph from 143 to 146 unit tests, including the new browser check. No UI behavior changes in this commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A robots.txt file may start with a UTF-8 BOM. The parser treated the first key as
\uFEFFuser-agent, so a blanket disallow was missed and the site scored as open. The launch status also still reported 143 unit tests and said no UI code had changed.What changed
\uFEFFUser-agent: *+Disallow: /, which now caps at 25.browserworkflow badge to the README.Verification
npm run check— exit 0, 146 unit tests, 5 integration tests, package and bundle checksnpm run test:browser— exit 0Boundary
The robots parser remains a heuristic for blanket blocks, not a complete policy engine. This is a parser fix and status correction; no new user or assistive-technology evidence.