Repository navigation
fix: ignore UTF-8 BOM in FileSystemSkillStore files - #13745
Open
Assertive-xxh wants to merge 1 commit into
Open
Assertive-xxh wants to merge 1 commit into
Assertive-xxh wants to merge 1 commit into
Conversation
A SKILL.md that starts with a UTF-8 byte order mark was treated as having no frontmatter, so warm_up() raised a ValueError about a missing description. Bundled text files kept the BOM as a leading U+FEFF. Read SKILL.md and bundled text files with utf-8-sig, which drops a leading BOM and reads BOM-less UTF-8 unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Assertive-xxh
requested review from
davidsbatista
and removed request for
a team
November 14, 2026 03:58
Contributor
|
@Assertive-xxh is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
1AbdullahButt
left a comment
There was a problem hiding this comment.
Looks correct. Here's what I checked.
- These are the only three text reads in the store.
skill_store.py:135,:199and:256were the onlyread_textcalls inhaystack/skill_stores/.SkillToolset(tools/skills/skill_toolset.py) reads nothing itself and goes through the store, sowarm_up()on the toolset is covered too. The PDF and image branches still useread_bytes(), so a BOM-like prefix in a binary file isn't touched. - The cause is as described.
b'\xef\xbb\xbf---\n'.decode('utf-8')keeps'---', andstr.lstrip()doesn't removeU+FEFF. Solines[0]in_parse_frontmatter(:42) isn't---, and the function returns{}. Withutf-8-sig, the same bytes decode to'---'. AU+FEFFin the middle of a file is kept (b'a\xef\xbb\xbfb'gives'ab'), so only a leading BOM is dropped. - Error handling is unchanged.
utf-8-sigraises the sameUnicodeDecodeErrorfor non-UTF-8 bytes, so theexceptatread_skill_filestill turns a Latin-1 or other binary file into the "not a readable asset"ValueError. - The tests reach the right code.
write_text(..., encoding="utf-8-sig")writes a real BOM, and the first test goes through bothwarm_up(vialist_skills) andload_skill. git merge-treeagainstmainis clean.
I didn't run the tests.
Reviewed with help from Claude Code.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
FileSystemSkillStorereports a missingdescriptionfor aSKILL.mdthat starts with a UTF-8 BOM #13744Proposed Changes:
FileSystemSkillStorereadSKILL.mdand bundled text files withencoding="utf-8", which keeps a UTF-8 byte order mark (BOM) as a leadingU+FEFF._parse_frontmatterstrips whitespace and then expects the first line to be---, butstr.lstrip()does not stripU+FEFF. ASKILL.mdsaved with a BOM was therefore treated as having no frontmatter, andwarm_up()raisedValueError: Skill '...' is missing a 'description' in its frontmatter.for a file that has a description. One such file breaks the whole skills directory, includingSkillToolset.warm_up().read_skill_filereturned bundled text files with a leading.The three
read_textcalls (warm_up,load_skill,read_skill_file) now useutf-8-sig, which drops a leading BOM and reads UTF-8 without a BOM unchanged. This is the same approach as #12726 for the text, CSV, Markdown and JSON converters.How did you test it?
test/skill_stores/file_system/test_skill_store.py:test_list_and_load_skill_ignore_utf8_bomandtest_read_skill_file_strips_utf8_bom. With the source frommain(4b4346b) both fail (2 failed, 34 passed); on this branch the file passes.pytest test/skill_stores test/tools -m "not integration": 365 passed, 2 skipped, 12 deselected (Ubuntu 24.04 on WSL2, Python 3.12.3).ruff check,ruff format --check,mypy,scripts/release_note_backticks.py --check,codespell.reno lint(my clone is shallow), the license header check. I used a plain virtual environment with the base dependencies instead of the Hatchtestenvironment, and ran the checks above directly rather than throughpre-commit.Notes for the reviewer
read_skill_fileis included because it has the same cause. Without it a bundled text file with a BOM reaches the model with a leading. I can drop that part if you prefer a narrower change.U+FEFFin the middle of a file, and UTF-16, Latin-1 or binary files behave as before; I comparedmainand this branch on those cases. The one difference I found: a bundled file that consists only of an incomplete BOM (the two bytesEF BB) now reads as an empty string instead of raising "not a readable asset".docs-website/docs/tools/skilltoolset.mdx, since it does not mention encodings. The class docstring now states that files are read as UTF-8 and that a leading BOM is ignored. Happy to add a sentence to the page if you want one.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:and added!in case the PR includes breaking changes.🤖 Generated with Claude Code