Repository navigation
fix(arrow/flight): skip metadata-only messages in the record reader - #1549
ShivamSharma43 wants to merge 1 commit into
Conversation
Writer.WriteMetadata sends FlightData messages with app metadata but no IPC header or body. NewRecordReader dropped the first such message while peeking for a descriptor, but handed any later one to ipc.NewMessage, so the empty header reached the flatbuffers decoder and reading failed with "index out of range [3] with length 0". Two heartbeats before the schema were enough to break the stream, and one between batches broke it as well. The message adapter now skips messages that have neither a header nor a body, still picking up any FlightDescriptor they carry. Fixes apache#1517 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, this looks right. I checked metadata-only messages before the schema (2 and 1024), between batches and after the last batch: all fail on main with index out of range [3] with length 0 and read cleanly here, and LatestAppMetadata() always returns the batch's own metadata. The same holds over a real gRPC DoGet. The new test fails on main, and the flight tests, including Flight SQL and the driver, pass locally.
Two optional points:
- Inline, about header-less messages that carry a body.
- Could
ReaderorNewRecordReaderget one doc sentence saying that standalone metadata messages are skipped and theirapp_metadataisn't exposed? C++ and Java hand such messages to the caller after the schema, so someone porting code may expect to see them here.
| // a message with neither an IPC header nor a body only carries | ||
| // app metadata (see Writer.WriteMetadata); there is no IPC message | ||
| // to decode, so skip it | ||
| if len(fd.DataHeader) > 0 || len(fd.DataBody) > 0 { |
There was a problem hiding this comment.
Optional: a message with a body but no header still reaches ipc.NewMessage, and mid-stream that returns the same cryptic arrow/ipc: unknown error while reading: runtime error: index out of range [3] with length 0. Since this loop already inspects both fields, it could return a clear error here instead, something like "FlightData has a body but no IPC header". That's existing behaviour, so it can also be a follow-up.
Rationale for this change
Writer.WriteMetadatasends aFlightDatamessage that hasapp_metadatabut no IPC header or body.NewRecordReaderdropped the first such message while peeking for a descriptor, but the message adapter passed any later one toipc.NewMessage. The empty header then reached the flatbuffers decoder, and reading failed with:So two heartbeats before the schema broke the stream (see #1517). A single metadata-only message between two record batches failed the same way, because
Next()hit the same decode error.What changes are included in this PR?
dataMessageReader.Message()now skips messages that have neither aDataHeadernor aDataBody. It still records anyFlightDescriptorthey carry. Messages with a header, or with a body and no header, follow the same path as before.LatestAppMetadata()still returns the metadata attached to the record batch that was read, not the metadata of a skipped heartbeat.Are these changes tested?
Yes.
TestReaderSkipsMetadataOnlyMessageswrites threeWriteMetadatamessages before the schema and one after each record batch (including after the last one). It then checks that every batch reads back equal, with its ownWriteWithAppMetadatavalue, and that the stream ends cleanly. The test fails onmainwith the error above and passes with this change.collectingFlightStreamWriternow copiesAppMetadatatoo, so the test can check it.go test ./arrow/flight/...passes, with and without-tags assert,test -short.go vet ./arrow/flight/...and gofmt are clean.Are there any user-facing changes?
Readers no longer fail on streams that contain standalone metadata messages. The record-oriented reader skips those messages.
Fixes #1517
AI assistance: I used Claude Code (Anthropic) to write this change and the test. I reviewed the code and ran the tests locally.
🤖 Generated with Claude Code