Skip to content

fix(arrow/flight): skip metadata-only messages in the record reader - #1549

Open
ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:flight-skip-metadata-only-messages
Open

ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:flight-skip-metadata-only-messages

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

Rationale for this change

Writer.WriteMetadata sends a FlightData message that has app_metadata but no IPC header or body. NewRecordReader dropped the first such message while peeking for a descriptor, but the message adapter passed any later one to ipc.NewMessage. The empty header then reached the flatbuffers decoder, and reading failed with:

arrow/flight: could not create flight reader: arrow/ipc: unknown error while reading: runtime error: index out of range [3] with length 0

So two heartbeats before the schema broke the stream (see #1517). A single metadata-only message between two record batches failed the same way, because Next() hit the same decode error.

What changes are included in this PR?

dataMessageReader.Message() now skips messages that have neither a DataHeader nor a DataBody. It still records any FlightDescriptor they carry. Messages with a header, or with a body and no header, follow the same path as before. LatestAppMetadata() still returns the metadata attached to the record batch that was read, not the metadata of a skipped heartbeat.

Are these changes tested?

Yes. TestReaderSkipsMetadataOnlyMessages writes three WriteMetadata messages before the schema and one after each record batch (including after the last one). It then checks that every batch reads back equal, with its own WriteWithAppMetadata value, and that the stream ends cleanly. The test fails on main with the error above and passes with this change. collectingFlightStreamWriter now copies AppMetadata too, so the test can check it.

go test ./arrow/flight/... passes, with and without -tags assert,test -short. go vet ./arrow/flight/... and gofmt are clean.

Are there any user-facing changes?

Readers no longer fail on streams that contain standalone metadata messages. The record-oriented reader skips those messages.

Fixes #1517

AI assistance: I used Claude Code (Anthropic) to write this change and the test. I reviewed the code and ran the tests locally.

🤖 Generated with Claude Code

Writer.WriteMetadata sends FlightData messages with app metadata but no
IPC header or body. NewRecordReader dropped the first such message while
peeking for a descriptor, but handed any later one to ipc.NewMessage,
so the empty header reached the flatbuffers decoder and reading failed
with "index out of range [3] with length 0". Two heartbeats before the
schema were enough to break the stream, and one between batches broke
it as well.

The message adapter now skips messages that have neither a header nor
a body, still picking up any FlightDescriptor they carry.

Fixes apache#1517

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 07:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks right. I checked metadata-only messages before the schema (2 and 1024), between batches and after the last batch: all fail on main with index out of range [3] with length 0 and read cleanly here, and LatestAppMetadata() always returns the batch's own metadata. The same holds over a real gRPC DoGet. The new test fails on main, and the flight tests, including Flight SQL and the driver, pass locally.

Two optional points:

  1. Inline, about header-less messages that carry a body.
  2. Could Reader or NewRecordReader get one doc sentence saying that standalone metadata messages are skipped and their app_metadata isn't exposed? C++ and Java hand such messages to the caller after the schema, so someone porting code may expect to see them here.

// a message with neither an IPC header nor a body only carries
// app metadata (see Writer.WriteMetadata); there is no IPC message
// to decode, so skip it
if len(fd.DataHeader) > 0 || len(fd.DataBody) > 0 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: a message with a body but no header still reaches ipc.NewMessage, and mid-stream that returns the same cryptic arrow/ipc: unknown error while reading: runtime error: index out of range [3] with length 0. Since this loop already inspects both fields, it could return a clear error here instead, something like "FlightData has a body but no IPC header". That's existing behaviour, so it can also be a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go] Flight record reader fails on repeated metadata-only messages before the schema

3 participants