Skip to content

Query failover resends a statement after an undecodable final response #115

Open

Description

@mtopolnik

Problem

With failover=on (the default), QwpQueryClient.execute() sends a statement again after the server has already finished it, whenever the client cannot decode the server's final response. The resend goes through the normal failover path, possibly to another endpoint, so a non-idempotent statement (DDL, INSERT, UPDATE, RENAME TABLE, ...) can run twice.

Reachability (at 82b2e14)

  1. QwpEgressIoThread reports these cases through emitTerminalTransportError, which emits a KIND_TRANSPORT_ERROR event:
    • decodeAndEmitExecDone (lines 435-468): an EXEC_DONE truncated before op_type or mid-varint, or a rows_affected varint longer than 64 bits.
    • decodeAndEmitResultEnd (lines ~493-521): the same failures in RESULT_END.
    • onBinaryMessage (lines 196-227): a frame shorter than the header, or an unknown msg_kind.
  2. In QwpQueryClient, the event loop maps KIND_TRANSPORT_ERROR to probe.markTransportFailure (~line 1954).
  3. The failover loop in execute() (~lines 1736-1800) then demotes the host, reconnects, and calls executeOnce again with the same SQL.

The server sends EXEC_DONE and RESULT_END only after it has finished the statement, so the resend repeats completed work. After an unknown msg_kind or an unreadable header, the client cannot tell whether the statement finished.

Trigger: a server that sends a malformed final frame, through a server bug or a hostile server. The rows_affected = -1 sentinel that QuestDB 9.4.0-10.0.1 send for RENAME TABLE, TRUNCATE and similar statements decodes as -1 in this client, so it does not trigger this path.

What the user sees

The statement runs a second time, and the handler reports the outcome of the resend. For example, RENAME TABLE a TO b succeeds on the first attempt, and the resend then fails because a no longer exists. An INSERT ... SELECT inserts its rows twice.

Suggested fix

Do what the Go client now does (questdb/go-questdb-client#74):

  • Do not resubmit after a final frame (EXEC_DONE, QUERY_ERROR, RESULT_END) the client cannot decode, a frame header it cannot read, or an unknown msg_kind. Report the error instead.
  • A RESULT_BATCH or CACHE_RESET that cannot be decoded can keep failing over like a dropped connection, because the statement is still streaming rows.

The normative client spec (connect/wire-protocols/qwp-client-behavior.md, "Query execution-time failover") says a protocol-level terminal failure triggers a resubmit, so it needs the same exception. The cross-client failover guide already documents the Go behavior (questdb/documentation#600).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions