Skip to content

fix(arrow/cdata): don't call the async producer after the handler is released - #1609

Merged
zeroshade merged 2 commits into
apache:mainfrom
AynSiddiqui:fix/1572-async-request-after-release
Nov 19, 2026
Merged

zeroshade merged 2 commits into
apache:mainfrom
AynSiddiqui:fix/1572-async-request-after-release

Conversation

@AynSiddiqui

@AynSiddiqui AynSiddiqui commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixes #1572.

The spec only requires an ArrowAsyncProducer to stay valid until just before the producer calls release on the handler. The handler from CreateAsyncDeviceStreamHandler can still have tasks queued at that point, and asyncTaskQueue called request(1) on the producer for each of them, so a producer that frees itself after release, like the .NET producer in the issue, could still be called.

Separately, asyncTaskExtract wrote through out without a NULL check, although the spec lets a consumer pass NULL when it doesn't want the data.

What changes are included in this PR?

  • cAsyncStreamHandler guards producer with a mutex. release clears it, and asyncTaskQueue now takes the handler and calls request and cancel only while producer is set. Tasks that were already queued are still extracted as before; only the calls into the producer stop.
  • asyncTaskExtract returns 0 without writing when out is NULL. The record is still released.

release now waits while a request or cancel call from the consumer is in progress. The spec text in CDeviceDataInterface.rst only forbids request from synchronously calling on_next_task or on_error, and cancel from calling any handler other than on_error. abi.h adds that, while not cancelled, later calls to on_next_task, on_error or release "should be scheduled by the producer to be called later", but that is a should. If a producer calls release synchronously from inside request, the release callback blocks on h.mu, which is held across that request call, and never returns. This is a known case, next to a producer that holds its own lock while calling release and takes the same lock in request or cancel; both could block here. I didn't see a way to drop the wait without calling a producer that may already be gone, but I'm open to another approach.

Are these changes tested?

Yes, two tests in arrow/cdata/cdata_test.go with helpers in cdata_test_framework.go:

  • TestAsyncNoRequestAfterRelease: a C producer sends three tasks and the end of the stream, then calls release. Its extract_data waits until release has returned, so at least two tasks are still queued when the handler is released. The test counts request calls made after release returns and checks that extract_data is called for all three tasks. On main it fails in 10 of 10 runs; with this change no request call is made after release.
  • TestAsyncTaskExtractNull: calls asyncTaskExtract with a NULL out using a checked allocator. On main it panics with a nil pointer dereference; here it returns 0 and the checked allocator reports no outstanding allocations.

go test -tags test,assert -short ./arrow/cdata/..., the same with noasm, go test -race -tags test -count=20 -run TestAsync ./arrow/cdata/ and golangci-lint run ./arrow/cdata/... pass on Linux amd64 with Go 1.27.1. I didn't run the full module tests or test against a real external producer.

Are there any user-facing changes?

No API changes. A producer can now free its ArrowAsyncProducer after calling release without the consumer calling into it, and consumers can pass NULL to extract_data.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I checked #1572 against the C Device Data Interface spec, and it reads it correctly on both points:

  • The producer must stay valid only until release is called, and it cleans up after release ("ArrowAsyncProducer Lifetime", and the on_schema and producer member docs). So the consumer must not call request or cancel once release has returned.
  • A consumer may pass NULL to extract_data when it doesn't want the data (the extract_data docs and "Result lifetimes").

This change does both. No producer call starts after release returns, release waits for a call that is already in progress, buffered tasks are still extracted so the producer can clean them up, and a NULL out releases the record and returns 0. The new tests fail on main and pass here, and go test -race -tags test,assert -count=20 -run TestAsync ./arrow/cdata/ passes.

Two inline notes. One asks for a correction to the description. The other is a related spec gap that main already has, which can be a follow-up.


// request another batch now that we've processed this one
C.goCallRequest(producer, C.int64_t(1))
h.mu.Lock()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A correction to the description. It says "the spec asks producers not to call release from inside request while not cancelled". I can't find that in the spec.

  • The ArrowAsyncProducer.request docs only forbid request from synchronously calling on_next_task or on_error.
  • cancel must not call any consumer handler other than on_error, so release is ruled out from inside cancel, but not from inside request.

That matters because h.mu is held across this call. If a producer called release synchronously from inside request, the release callback would run on this goroutine and block on h.mu forever. A conforming producer is unlikely to do that, since it calls release after the end of the stream or an error. Could you either correct the spec claim, or list this case as a known deadlock next to the lock-ordering case the description already mentions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I couldn't back that sentence up. I removed it. The description now says what the spec text does: only synchronous on_next_task and on_error calls are forbidden from request, and abi.h only says later release calls "should" be scheduled. It also lists a synchronous release from inside request (deadlock on h.mu) as a known case next to the lock-ordering one.

C.goCallCancel(h.producer)
}
h.mu.Unlock()
return

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not from this PR, but related to the same spec section. On cancellation the goroutine calls cancel and returns at once, so tasks still in taskQueue, and any the producer sends afterwards for requests already made, are never passed to extract_data.

The spec says a consumer that doesn't want a task's data "must still call extract_data so that the producer can perform any required cleanup", passing NULL. It also says the consumer must expect more on_next_task calls after cancel. So the producer never gets to clean those tasks up.

Now that asyncTaskExtract handles a NULL out, the consumer side could drain the queue after cancel and call extract_data with NULL for each task until release closes the channel. That seems better as a follow-up than in this PR. I found this by reading the code and haven't run a conforming producer against it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that's a follow-up and not part of this PR. After cancel the consumer could keep draining taskQueue and call extract_data with NULL for each task until release closes the channel, which asyncTaskExtract now handles. There is a related producer-side gap I found by reading: exportAsyncProducer allocates a handle slot and cgo.Handle per task (exports.go:431) and frees them only when on_next_task fails (exports.go:443-444), while asyncTaskExtract (exports.go:332-345) releases the record but never frees them. I haven't reproduced that at runtime. Happy to open an issue for both once this PR merges.

@AynSiddiqui

Copy link
Copy Markdown
Contributor Author

The Linux ASan failures came from an 8-byte allocation in TestAsyncTaskExtractNull. I reproduced the leak by running that test alone on the PR head. The test now stores the handle slot in Go memory and deletes the cgo.Handle after extraction. With the patch, the arrow/cdata ASan suite passed twice, and both new tests passed ten race-enabled runs.

@zeroshade

Copy link
Copy Markdown
Member

Thanks for the ASan fix and the description update; the new commit looks good and I'll merge once CI is green. Yes, please open an issue covering both follow-ups: draining taskQueue with NULL extracts after cancel, and the per-task handle slot and cgo.Handle that asyncTaskExtract never frees.

@zeroshade
zeroshade merged commit b80f8c1 into apache:main Nov 19, 2026
40 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow/cdata: async consumer calls request() on the producer after the handler has been released

2 participants