Repository navigation
fix(arrow/cdata): drain tasks after cancel and free per-task handle in async consumer - #1643
AynSiddiqui wants to merge 2 commits into
Conversation
…n async consumer After cancel, keep draining taskQueue and call extract_data with NULL for each task until release closes it, as the async spec asks. asyncTaskExtract now deletes the task's cgo.Handle and frees its slot. Fixes apache#1642 Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, this matches the spec. After cancel, tasks keep being passed to extract_data with NULL until release. Because the drain is deferred before close(recordStream), it runs after it, so readers of the stream aren't delayed. The task's handle and slot are now freed exactly once: exportAsyncProducer only frees them itself when on_next_task fails, in which case the consumer never received the task. On main both new tests fail ("handle not deleted" and "tasks sent after cancel must be extracted with NULL"). Here, go test -race -tags test,assert -count=20 -run TestAsync ./arrow/cdata/ and the full package pass.
One small suggestion inline. A minor point that doesn't need a change: the test producer busy-waits for cancel (while (!g_async_cancelled) {}), keeping a thread spinning inside cgo. The test always cancels, so it's fine, though a short sleep in the loop would be gentler.
| defer func() { | ||
| rec.Release() | ||
| h.Delete() | ||
| C.free(task.private_data) |
There was a problem hiding this comment.
Optional: also set task.private_data = nil here, and have asyncTaskExtract return an error such as EINVAL when private_data is nil on entry. As the description says, a second extract_data call is now a use-after-free. That's a consumer bug, but this would turn it into an error rather than corrupted memory, for two lines.
There was a problem hiding this comment.
Done in 5175832: asyncTaskExtract now returns EINVAL when private_data is nil on entry, and sets task.private_data = nil after freeing it. Added TestAsyncTaskExtractTwice (second call returns EINVAL); without the change it panics on the already-deleted handle.
I left the busy-wait in the test producer as is, since the test always cancels.
… task Clear task.private_data once the task is consumed and return EINVAL when asyncTaskExtract is entered with a NULL private_data, so a repeated call fails instead of using the freed handle. Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
Rationale for this change
Fixes #1642.
Follow-ups noted on #1609, from the C Device Data Interface spec:
cancelthe consumer stopped readingtaskQueue, so tasks the producer still sent were never passed toextract_data. The spec says a consumer that doesn't want a task must still callextract_data(withNULL) so the producer can clean up, and that moreon_next_taskcalls can followcancel. When the queue filled up,on_next_taskalso blocked.asyncTaskExtractreleased the record but never deleted the task'scgo.Handleor freed its heap slot, so every extracted task leaked both.What changes are included in this PR?
asyncTaskQueuekeeps drainingtaskQueueafter it returns, callingextract_datawithNULLfor each task untilreleasecloses the queue. This runs after the record stream is closed, so consumers of the stream are not delayed.asyncTaskExtractdeletes the task'scgo.Handleand frees its slot once it has been used (including theNULLcase).Are these changes tested?
Yes.
TestAsyncDrainTasksAfterCancelruns a producer that sends more tasks than the queue holds aftercanceland checks each is extracted withNULL.TestAsyncTaskExtractNullnow checks the handle is deleted. Those two fail on main and pass here.TestAsyncTaskExtractTwicechecks that a secondextract_datacall on the same task returnsEINVAL.go test -race -count=20 -run TestAsync ./arrow/cdataand thearrow/cdatasuite under-asanpass.Are there any user-facing changes?
No.
Notes:
extract_datamust still be called exactly once per task. A second call returnsEINVAL.releasereturns may be extracted withNULLafterwards, as the main loop already did.Stream, the loop can still block onrecordStream <- ...and never reach the drain. That is pre-existing and out of scope here.