Issues

• v1.0.0-rc.2: canonicalize_file rejects ByteLevel add_prefix_space=true supported by 0.23.2

  huggingface/tokenizers#2460 · Filter by author michaelfeil opened on Sep 26, 2026

• [1.0.0rc2] Encoding.ids_array panics while another thread decodes the ids

  huggingface/tokenizers#2458 · Filter by author nimbrel opened on Sep 26, 2026
  2

  

• Type Stubs for subpackages don't work with basedpyright

  huggingface/tokenizers#2456 · Filter by author LyndonArmitage opened on Sep 24, 2026

• [1.0.0rc2] Possible regressions vs. 0.23.x: Byte atom not found in the vocabulary

  huggingface/tokenizers#2447 · Filter by author xenova opened on Sep 23, 2026
  4

  

• Precompiled normalizer diverges from SentencePiece: shortest-prefix match + discarded remainder silently drops codepoints

  Stale
  huggingface/tokenizers#2334 · Filter by author imhmhm opened on Aug 14, 2026
  6

• [Feature/Security] Support trusted and untrusted special-token spans

  huggingface/tokenizers#2225 · Filter by author kexinoh opened on Jul 21, 2026
  2

• [Security] Load-time process abort when loading a malicious tokenizer.json (BPE merge buffer overrun)

  huggingface/tokenizers#2094 · Filter by author geo-chen opened on Jun 10, 2026
  2

• Release 0.23.1 introduces breaking change in Encoding.overflowing

  huggingface/tokenizers#2091 · Filter by author alistairewj opened on Jun 8, 2026
  4

• Reproducible WordPiece encode bench shows ~5x speedup on 123B fixture — methodology question for maintainers

  Stale
  huggingface/tokenizers#2075 · Filter by author SMC17 opened on May 28, 2026
  1

• Unigram trainer: prune loss uses alternatives.len() instead of alternatives[id].len()

  Stale
  huggingface/tokenizers#2069 · Filter by author hunter-heidenreich opened on May 24, 2026
  1

• BPE trainer: i32 overflow in pair_counts corrupts merge ordering on large code corpora

  Stale
  huggingface/tokenizers#2058 · Filter by author petyb opened on May 14, 2026
  2

• It is important to understand the real cause of the problem.

  huggingface/tokenizers#2027 · Filter by author Nearbe opened on Apr 18, 2026
  3

• Feature request: compact vocabulary remapping for domain-specific training with pretrained tokenizers

  huggingface/tokenizers#2023 · Filter by author saslifat-gif opened on Apr 14, 2026
  2

• [RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)

  Feature Request
  huggingface/tokenizers#1975 · Filter by author nicezic opened on Mar 22, 2026
  5

• Make huggingface-hub an optional dependency (extras)

  Feature Request
  huggingface/tokenizers#1973 · Filter by author clemlesne opened on Mar 20, 2026
  10

• Serialization of Bytelevel for decoder and pre_tokenizer is missleading

  huggingface/tokenizers#1960 · Filter by author ArthurZucker opened on Mar 4, 2026

• Tokenizer encode ignores pre-tokenizer when using right to left (rtl) digit splits with add_tokens

  huggingface/tokenizers#1957 · Filter by author gsaltintas opened on Mar 1, 2026
  1

• encode_batch has suboptimal parallelization on high-core systems

  huggingface/tokenizers#1929 · Filter by author stargazerZJ opened on Jan 14, 2026
  8

• De/tokenization on CUDA

  huggingface/tokenizers#1919 · Filter by author vadimkantorov opened on Dec 26, 2025
  7

• do we have plan support java ，we need you

  bindings
  huggingface/tokenizers#1918 · Filter by author mullerhai opened on Dec 25, 2025
  11

• spm_precompiled dependency bricks rust-analyzer in certain build setups

  huggingface/tokenizers#1917 · Filter by author gvilums opened on Dec 21, 2025
  1

• Decoding of AddedTokens containing Latin-Extended A characters are wrong

  huggingface/tokenizers#1915 · Filter by author andrewivan123 opened on Dec 19, 2025
  3

• [Docs] Encode Inputs rendering issues

  huggingface/tokenizers#1909 · Filter by author ariG23498 opened on Dec 12, 2025
  1

• TemplateProcessing does not apply type_id to overflow encodings

  huggingface/tokenizers#1908 · Filter by author antoine-agthe-unity opened on Dec 9, 2025
  4

• Guide: Compiling tokenizers on Android/Termux

  huggingface/tokenizers#1902 · Filter by author Manamama-Gemini-Cloud-AI-01 opened on Dec 1, 2025
  3