Skip to content

fix: ignore UTF-8 BOM in FileSystemSkillStore files - #13745

Open
Assertive-xxh wants to merge 1 commit into
deepset-ai:mainfrom
Assertive-xxh:fix/skill-store-utf8-bom
Open

Assertive-xxh wants to merge 1 commit into
deepset-ai:mainfrom
Assertive-xxh:fix/skill-store-utf8-bom

Conversation

@Assertive-xxh

Copy link
Copy Markdown

Related Issues

Proposed Changes:

FileSystemSkillStore read SKILL.md and bundled text files with encoding="utf-8", which keeps a UTF-8 byte order mark (BOM) as a leading U+FEFF. _parse_frontmatter strips whitespace and then expects the first line to be ---, but str.lstrip() does not strip U+FEFF. A SKILL.md saved with a BOM was therefore treated as having no frontmatter, and warm_up() raised ValueError: Skill '...' is missing a 'description' in its frontmatter. for a file that has a description. One such file breaks the whole skills directory, including SkillToolset.warm_up(). read_skill_file returned bundled text files with a leading .

The three read_text calls (warm_up, load_skill, read_skill_file) now use utf-8-sig, which drops a leading BOM and reads UTF-8 without a BOM unchanged. This is the same approach as #12726 for the text, CSV, Markdown and JSON converters.

How did you test it?

  • Added two unit tests in test/skill_stores/file_system/test_skill_store.py: test_list_and_load_skill_ignore_utf8_bom and test_read_skill_file_strips_utf8_bom. With the source from main (4b4346b) both fail (2 failed, 34 passed); on this branch the file passes.
  • pytest test/skill_stores test/tools -m "not integration": 365 passed, 2 skipped, 12 deselected (Ubuntu 24.04 on WSL2, Python 3.12.3).
  • On the changed files: ruff check, ruff format --check, mypy, scripts/release_note_backticks.py --check, codespell.
  • Not run: the full unit suite, integration tests, Windows and macOS, reno lint (my clone is shallow), the license header check. I used a plain virtual environment with the base dependencies instead of the Hatch test environment, and ran the checks above directly rather than through pre-commit.

Notes for the reviewer

  • read_skill_file is included because it has the same cause. Without it a bundled text file with a BOM reaches the model with a leading . I can drop that part if you prefer a narrower change.
  • Files without a BOM, empty files, a U+FEFF in the middle of a file, and UTF-16, Latin-1 or binary files behave as before; I compared main and this branch on those cases. The one difference I found: a bundled file that consists only of an incomplete BOM (the two bytes EF BB) now reads as an empty string instead of raising "not a readable asset".
  • I did not change docs-website/docs/tools/skilltoolset.mdx, since it does not mention encodings. The class docstring now states that files are read as UTF-8 and that a leading BOM is ignored. Happy to add a sentence to the page if you want one.
  • This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

🤖 Generated with Claude Code

A SKILL.md that starts with a UTF-8 byte order mark was treated as having
no frontmatter, so warm_up() raised a ValueError about a missing
description. Bundled text files kept the BOM as a leading U+FEFF.

Read SKILL.md and bundled text files with utf-8-sig, which drops a leading
BOM and reads BOM-less UTF-8 unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Assertive-xxh
Assertive-xxh requested a review from a team as a code owner November 14, 2026 03:58
@Assertive-xxh
Assertive-xxh requested review from davidsbatista and removed request for a team November 14, 2026 03:58
@vercel

vercel Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

@Assertive-xxh is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@1AbdullahButt 1AbdullahButt left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct. Here's what I checked.

  • These are the only three text reads in the store. skill_store.py:135, :199 and :256 were the only read_text calls in haystack/skill_stores/. SkillToolset (tools/skills/skill_toolset.py) reads nothing itself and goes through the store, so warm_up() on the toolset is covered too. The PDF and image branches still use read_bytes(), so a BOM-like prefix in a binary file isn't touched.
  • The cause is as described. b'\xef\xbb\xbf---\n'.decode('utf-8') keeps '---', and str.lstrip() doesn't remove U+FEFF. So lines[0] in _parse_frontmatter (:42) isn't ---, and the function returns {}. With utf-8-sig, the same bytes decode to '---'. A U+FEFF in the middle of a file is kept (b'a\xef\xbb\xbfb' gives 'ab'), so only a leading BOM is dropped.
  • Error handling is unchanged. utf-8-sig raises the same UnicodeDecodeError for non-UTF-8 bytes, so the except at read_skill_file still turns a Latin-1 or other binary file into the "not a readable asset" ValueError.
  • The tests reach the right code. write_text(..., encoding="utf-8-sig") writes a real BOM, and the first test goes through both warm_up (via list_skills) and load_skill.
  • git merge-tree against main is clean.

I didn't run the tests.

Reviewed with help from Claude Code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileSystemSkillStore reports a missing description for a SKILL.md that starts with a UTF-8 BOM

3 participants