Repository navigation
Conversation
Member
|
@CLCK0622, thanks for fixing this and for thinking about Ctrl-C too! 3.0 rewrote the Python glue in C: calls release the GIL once and never retake it without a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #831. Reported by @clemlesne.
When another Python thread is runnable, the add/search reporter currently waits for the GIL on every item, even without a Python progress callback. Keep empty progress callbacks native and poll Python signals from the existing reporter at 100 ms intervals. Check pending signals again after native work returns, preserving KeyboardInterrupt instead of overwriting it with an operation error. Apply the same policy to sharded search and exact search.
Python progress callbacks retain their per-item notifications, final completion report, and cancellation behavior. The C API and public Python signatures are unchanged. The interval applies at existing progress checkpoints; a long individual item, GIL contention, or remaining workers can delay interruption further.
On Apple M4 / CPython 3.11.15, the issue's seed=0 float32 input gives these median wall times with a busy Python thread and no callback (five samples per build, identical clang -O3 / NumKong=1 / OpenMP=0 builds):
Single-query index.search remains around 7.7 ms under contention because returning to Python still requires a GIL hand-off. Additional seven-sample AB/BA controls cover the callback path and uncontended no-callback path. For contended exact search with a callback, the one-thread median changed by +7.2% in seven samples and -1.7% in a separate 21-sample run; the four-thread seven-sample change was +3.1%. Uncontended callback timings are close. Callback counts and output SHA-256 values match before/after; the full samples are included because contended timings remain sensitive to interpreter scheduling.
Validation on macOS arm64, Python 3.11:
Linux and Windows are left to the upstream CI matrix; the interruption test depends on signal delivery timing and a 15 s subprocess bound, and was only run on macOS arm64.
AI disclosure: Codex assisted with part of the code and this description.