Repository navigation
fix(arrow/cdata): don't call the async producer after the handler is released - #1609
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
Thanks. I checked #1572 against the C Device Data Interface spec, and it reads it correctly on both points:
- The producer must stay valid only until
releaseis called, and it cleans up afterrelease("ArrowAsyncProducer Lifetime", and theon_schemaandproducermember docs). So the consumer must not callrequestorcanceloncereleasehas returned. - A consumer may pass
NULLtoextract_datawhen it doesn't want the data (theextract_datadocs and "Result lifetimes").
This change does both. No producer call starts after release returns, release waits for a call that is already in progress, buffered tasks are still extracted so the producer can clean them up, and a NULL out releases the record and returns 0. The new tests fail on main and pass here, and go test -race -tags test,assert -count=20 -run TestAsync ./arrow/cdata/ passes.
Two inline notes. One asks for a correction to the description. The other is a related spec gap that main already has, which can be a follow-up.
|
|
||
| // request another batch now that we've processed this one | ||
| C.goCallRequest(producer, C.int64_t(1)) | ||
| h.mu.Lock() |
There was a problem hiding this comment.
A correction to the description. It says "the spec asks producers not to call release from inside request while not cancelled". I can't find that in the spec.
- The
ArrowAsyncProducer.requestdocs only forbidrequestfrom synchronously callingon_next_taskoron_error. cancelmust not call any consumer handler other thanon_error, soreleaseis ruled out from insidecancel, but not from insiderequest.
That matters because h.mu is held across this call. If a producer called release synchronously from inside request, the release callback would run on this goroutine and block on h.mu forever. A conforming producer is unlikely to do that, since it calls release after the end of the stream or an error. Could you either correct the spec claim, or list this case as a known deadlock next to the lock-ordering case the description already mentions?
There was a problem hiding this comment.
You're right, I couldn't back that sentence up. I removed it. The description now says what the spec text does: only synchronous on_next_task and on_error calls are forbidden from request, and abi.h only says later release calls "should" be scheduled. It also lists a synchronous release from inside request (deadlock on h.mu) as a known case next to the lock-ordering one.
| C.goCallCancel(h.producer) | ||
| } | ||
| h.mu.Unlock() | ||
| return |
There was a problem hiding this comment.
Not from this PR, but related to the same spec section. On cancellation the goroutine calls cancel and returns at once, so tasks still in taskQueue, and any the producer sends afterwards for requests already made, are never passed to extract_data.
The spec says a consumer that doesn't want a task's data "must still call extract_data so that the producer can perform any required cleanup", passing NULL. It also says the consumer must expect more on_next_task calls after cancel. So the producer never gets to clean those tasks up.
Now that asyncTaskExtract handles a NULL out, the consumer side could drain the queue after cancel and call extract_data with NULL for each task until release closes the channel. That seems better as a follow-up than in this PR. I found this by reading the code and haven't run a conforming producer against it.
There was a problem hiding this comment.
Agreed, that's a follow-up and not part of this PR. After cancel the consumer could keep draining taskQueue and call extract_data with NULL for each task until release closes the channel, which asyncTaskExtract now handles. There is a related producer-side gap I found by reading: exportAsyncProducer allocates a handle slot and cgo.Handle per task (exports.go:431) and frees them only when on_next_task fails (exports.go:443-444), while asyncTaskExtract (exports.go:332-345) releases the record but never frees them. I haven't reproduced that at runtime. Happy to open an issue for both once this PR merges.
|
The Linux ASan failures came from an 8-byte allocation in |
|
Thanks for the ASan fix and the description update; the new commit looks good and I'll merge once CI is green. Yes, please open an issue covering both follow-ups: draining |
Rationale for this change
Fixes #1572.
The spec only requires an
ArrowAsyncProducerto stay valid until just before the producer callsreleaseon the handler. The handler fromCreateAsyncDeviceStreamHandlercan still have tasks queued at that point, andasyncTaskQueuecalledrequest(1)on the producer for each of them, so a producer that frees itself afterrelease, like the .NET producer in the issue, could still be called.Separately,
asyncTaskExtractwrote throughoutwithout a NULL check, although the spec lets a consumer pass NULL when it doesn't want the data.What changes are included in this PR?
cAsyncStreamHandlerguardsproducerwith a mutex.releaseclears it, andasyncTaskQueuenow takes the handler and callsrequestandcancelonly whileproduceris set. Tasks that were already queued are still extracted as before; only the calls into the producer stop.asyncTaskExtractreturns 0 without writing whenoutis NULL. The record is still released.releasenow waits while arequestorcancelcall from the consumer is in progress. The spec text inCDeviceDataInterface.rstonly forbidsrequestfrom synchronously callingon_next_taskoron_error, andcancelfrom calling any handler other thanon_error.abi.hadds that, while not cancelled, later calls toon_next_task,on_errororrelease"should be scheduled by the producer to be called later", but that is ashould. If a producer callsreleasesynchronously from insiderequest, the release callback blocks onh.mu, which is held across thatrequestcall, and never returns. This is a known case, next to a producer that holds its own lock while callingreleaseand takes the same lock inrequestorcancel; both could block here. I didn't see a way to drop the wait without calling a producer that may already be gone, but I'm open to another approach.Are these changes tested?
Yes, two tests in
arrow/cdata/cdata_test.gowith helpers incdata_test_framework.go:TestAsyncNoRequestAfterRelease: a C producer sends three tasks and the end of the stream, then callsrelease. Itsextract_datawaits untilreleasehas returned, so at least two tasks are still queued when the handler is released. The test countsrequestcalls made afterreleasereturns and checks thatextract_datais called for all three tasks. On main it fails in 10 of 10 runs; with this change norequestcall is made afterrelease.TestAsyncTaskExtractNull: callsasyncTaskExtractwith a NULLoutusing a checked allocator. On main it panics with a nil pointer dereference; here it returns 0 and the checked allocator reports no outstanding allocations.go test -tags test,assert -short ./arrow/cdata/..., the same withnoasm,go test -race -tags test -count=20 -run TestAsync ./arrow/cdata/andgolangci-lint run ./arrow/cdata/...pass on Linux amd64 with Go 1.27.1. I didn't run the full module tests or test against a real external producer.Are there any user-facing changes?
No API changes. A producer can now free its
ArrowAsyncProducerafter callingreleasewithout the consumer calling into it, and consumers can pass NULL toextract_data.