Skip to content

fix(arrow/cdata): drain tasks after cancel and free per-task handle in async consumer - #1643

Open
AynSiddiqui wants to merge 2 commits into
apache:mainfrom
AynSiddiqui:fix/1642-async-drain-and-handle-leak
Open

AynSiddiqui wants to merge 2 commits into
apache:mainfrom
AynSiddiqui:fix/1642-async-drain-and-handle-leak

Conversation

@AynSiddiqui

@AynSiddiqui AynSiddiqui commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixes #1642.

Follow-ups noted on #1609, from the C Device Data Interface spec:

  • After cancel the consumer stopped reading taskQueue, so tasks the producer still sent were never passed to extract_data. The spec says a consumer that doesn't want a task must still call extract_data (with NULL) so the producer can clean up, and that more on_next_task calls can follow cancel. When the queue filled up, on_next_task also blocked.
  • asyncTaskExtract released the record but never deleted the task's cgo.Handle or freed its heap slot, so every extracted task leaked both.

What changes are included in this PR?

  • asyncTaskQueue keeps draining taskQueue after it returns, calling extract_data with NULL for each task until release closes the queue. This runs after the record stream is closed, so consumers of the stream are not delayed.
  • asyncTaskExtract deletes the task's cgo.Handle and frees its slot once it has been used (including the NULL case).

Are these changes tested?

Yes. TestAsyncDrainTasksAfterCancel runs a producer that sends more tasks than the queue holds after cancel and checks each is extracted with NULL. TestAsyncTaskExtractNull now checks the handle is deleted. Those two fail on main and pass here. TestAsyncTaskExtractTwice checks that a second extract_data call on the same task returns EINVAL. go test -race -count=20 -run TestAsync ./arrow/cdata and the arrow/cdata suite under -asan pass.

Are there any user-facing changes?

No.

Notes:

  • extract_data must still be called exactly once per task. A second call returns EINVAL.
  • Tasks still buffered when release returns may be extracted with NULL afterwards, as the main loop already did.
  • If the consumer cancels and stops reading Stream, the loop can still block on recordStream <- ... and never reach the drain. That is pre-existing and out of scope here.

…n async consumer

After cancel, keep draining taskQueue and call extract_data with NULL for
each task until release closes it, as the async spec asks. asyncTaskExtract
now deletes the task's cgo.Handle and frees its slot.

Fixes apache#1642

Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this matches the spec. After cancel, tasks keep being passed to extract_data with NULL until release. Because the drain is deferred before close(recordStream), it runs after it, so readers of the stream aren't delayed. The task's handle and slot are now freed exactly once: exportAsyncProducer only frees them itself when on_next_task fails, in which case the consumer never received the task. On main both new tests fail ("handle not deleted" and "tasks sent after cancel must be extracted with NULL"). Here, go test -race -tags test,assert -count=20 -run TestAsync ./arrow/cdata/ and the full package pass.

One small suggestion inline. A minor point that doesn't need a change: the test producer busy-waits for cancel (while (!g_async_cancelled) {}), keeping a thread spinning inside cgo. The test always cancels, so it's fine, though a short sleep in the loop would be gentler.

Comment thread arrow/cdata/exports.go
defer func() {
rec.Release()
h.Delete()
C.free(task.private_data)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: also set task.private_data = nil here, and have asyncTaskExtract return an error such as EINVAL when private_data is nil on entry. As the description says, a second extract_data call is now a use-after-free. That's a consumer bug, but this would turn it into an error rather than corrupted memory, for two lines.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5175832: asyncTaskExtract now returns EINVAL when private_data is nil on entry, and sets task.private_data = nil after freeing it. Added TestAsyncTaskExtractTwice (second call returns EINVAL); without the change it panics on the already-deleted handle.

I left the busy-wait in the test producer as is, since the test always cancels.

… task

Clear task.private_data once the task is consumed and return EINVAL when
asyncTaskExtract is entered with a NULL private_data, so a repeated call
fails instead of using the freed handle.

Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow/cdata: async consumer doesn't drain tasks after cancel, and exported tasks leak their handle

2 participants