Skip to content

Decode archive paths as UTF-8 first #77 - #79

Open
shashank-tomar0 wants to merge 1 commit into
aboutcode-org:mainfrom
shashank-tomar0:fix/utf7-path-decoding
Open

shashank-tomar0 wants to merge 1 commit into
aboutcode-org:mainfrom
shashank-tomar0:fix/utf7-path-decoding

Conversation

@shashank-tomar0

@shashank-tomar0 shashank-tomar0 commented Nov 8, 2026 •

Copy link
Copy Markdown

Summary

Fixes #77.

Extracting qt6-webengine_6.8.2+dfsg-4.debian.tar.xz put
LICENSES/Qt-GPL-exception-1.0.txt in the extraction root under the garbled
name qt6-webengine-6.8.2Min _nyopeQu _ GPL-exception-1.0.txt.

Root cause

Entry.get_path() decodes libarchive byte pathnames with
commoncode.text.as_unicode(), which uses BeautifulSoup's UnicodeDammit.
For the plain ASCII bytes of
qt6-webengine-6.8.2+dfsg/LICENSES/Qt-GPL-exception-1.0.txt, UnicodeDammit
picks UTF-7 and turns +dfsg/LICENSES/Qt- into CJK characters:

>>> from bs4 import UnicodeDammit
>>> p = b"qt6-webengine-6.8.2+dfsg/LICENSES/Qt-GPL-exception-1.0.txt"
>>> UnicodeDammit(p).original_encoding
'utf-7'
>>> UnicodeDammit(p).unicode_markup
'qt6-webengine-6.8.2\u75fb\u20fc\ub202\u10d4\u844b\uf42dGPL-exception-1.0.txt'

The separators are lost, so the file lands in the wrong directory.

Changes

  • Added as_unicode_path() in src/extractcode/libarchive2.py.
  • Archive paths are now decoded deterministically as UTF-8 first, since that is
    what modern archives use and plain ASCII is valid UTF-8.
  • Paths that are not valid UTF-8 still go through the previous charset
    detection, so existing non-UTF-8 archives keep their current behavior.
  • None and empty paths are handled as before.

Tests

$ python -m pytest tests/test_libarchive2.py -q
5 passed

New tests:

  • test_as_unicode_path_does_not_decode_plain_ascii_as_utf7
  • test_as_unicode_path_decodes_utf8
  • test_as_unicode_path_handles_missing_paths
  • test_extract_tar_keeps_directories_with_a_plus_sign

The last test builds a tar with the exact path from the report and extracts
it through libarchive. With the previous decoder it produced
qt6-webengine-6.8.2Min _nyopeQu _ GPL-exception-1.0.txt in the target root;
with this change it produces the expected
qt6-webengine-6.8.2+dfsg/LICENSES/Qt-GPL-exception-1.0.txt.

The tests/test_archive.py suite was also run. Its four failures are
pre-existing 7-Zip environment failures (system 7z instead of the
extractcode-7z plugin) and reproduce identically on pristine main.

ruff format --check and ruff check pass for the changed files.

Copilot AI balanced review requested due to automatic review settings November 8, 2026 08:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Archive entry paths are bytes. The charset detector used by
commoncode.text.as_unicode can mistake plain ASCII path segments such
as "+dfsg/LICENSES/Qt-" for UTF-7. This corrupts the path and extracts
files in the wrong directory, as reported for the qt6-webengine
archive.

Decode paths as UTF-8 first and only fall back to the charset detector
for bytes that are not valid UTF-8. Add a regression test using the
path from the report.

Signed-off-by: Shashank Tomar <shashank.tomar912@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filepaths get incorrectly decoded leading to files being extracted in the wrong directory

2 participants