Repository navigation
fix: handle empty masks in last-token pooling - #858
elysiopires wants to merge 2 commits into
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
chrikrah
left a comment
There was a problem hiding this comment.
@elysiopires from my side this is ready to merge at 2518f59. Both new tests fail on the base and pass here.
$ .venv/bin/python -m pytest -q -p no:cacheprovider tests/test_common.py
9 passed in 0.54s # head 2518f59, CPython 3.12
# fastembed/common/utils.py from f91d916 (base), your tests kept
FAILED tests/test_common.py::test_last_token_pooling_with_fully_masked_sequences
FAILED tests/test_common.py::test_last_token_pooling_with_zero_length_sequences
2 failed, 7 passed in 0.48s
$ python -I probe_qwen_tok.py # load_tokenizer() on Qwen/Qwen3-Embedding-0.6B, encode_batch(["", "hello world"])
[[1, 0, 0], [1, 1, 1]]
[['<|endoftext|>', '<|endoftext|>', '<|endoftext|>'], ['hello', 'Ġworld', '<|endoftext|>']]
# not run: an ONNX model end to end
non-blocking: zeros or a ValueError, like the other _pool guards, are both defensible. I'd keep zeros so last-token matches mean_pooling on the same row, at the cost of a silent zero vector in the index. Only a custom tokenizer with no EOS gets here; Qwen3 appends <|endoftext|> even to "".
Worth a one-line comment at utils.py:56 naming that case?
Last-token pooling returns a padding embedding when an entire attention-mask row is zero. With a zero-length sequence it raises an
argmaxerror. This can give a misleading nonzero result for a custom model or tokenizer that returns no unmasked tokens.Return a zero vector for these two cases, consistent with mean pooling on fully masked input. Keep the embedding dtype and preserve the existing behavior for rows with real tokens, including left and right padding.
Validation:
PYTHONPATH=. python -m pytest tests/test_common.py -q --tb=short: 9 passed.These tests use synthetic model outputs; no external model inference was run.