Resend idempotent requests when a reused connection was closed

[ilyazub]

A persistent client can lose a request to a close it couldn't have seen: the server or a proxy closes the connection just as the next request goes out. The write succeeds into the kernel buffer, and the read fails with couldn't read response headers, ECONNRESET, or SSLError over TLS on OpenSSL 3. A liveness check before reuse (#859) can't catch it, because the socket was fine when the check ran.

The client can't tell whether the server processed the request. zanker made that point in #420: "It's possible to have a request that succeeds on the server, but fails on the client when trying to read the headers." So this only resends requests that are safe to repeat. RFC 9110 §9.2.2 allows that for idempotent methods, and Net::HTTP, urllib3 and Go all do it by default.

A request is resent once, on a new connection, when all of these hold:

• the connection was reused. A fresh connection failing is a real error.
• no response byte arrived. After a status line or a 1xx, nothing is resent.
• the method is GET, HEAD, OPTIONS, TRACE, PUT or DELETE, or the request has an Idempotency-Key or X-Idempotency-Key header
• the body is nil or a String. IO and Enumerable bodies may already be consumed.
• the error isn't a timeout
• the client has no .retriable policy. Those keep their own semantics.

The resend isn't resent again, per RFC 9110: "A client SHOULD NOT automatically retry a failed automatic retry." Features see one on_request and one wrap_response per call, and on_error only for the final failure. The client stays dirty until the resent request completes, so a resend interrupted by Thread#kill or Timeout isn't reused.

Counted by the server, 30 runs per case, the same outcome on MRI 3.4.8 and JRuby 10.1.2.0:

case	main	this branch	requests the server got
GET, close crosses the request	ResponseHeaderError 30/30	200 30/30	2, one resend
POST with Idempotency-Key, close crosses it	ResponseHeaderError 30/30	200 30/30	2 byte-identical copies
POST, close crosses it	ResponseHeaderError 30/30	ResponseHeaderError 30/30	1, never duplicated
GET after an idle close	ResponseHeaderError 30/30	200 30/30	1
POST after an idle close	ResponseHeaderError 30/30	ResponseHeaderError 30/30	0. #859 delivers it once

.retriable doesn't cover this today. It's opt-in, and it retries by exception class without looking at the method, so it resends a POST the server already processed, up to 5 times by default. I can open an issue for that separately with repros.

I ran into this through a commercial HTTPS proxy that cuts a CONNECT tunnel 10.0 s after the last byte it sent the client. Of 54 requests reused at 9.6 to 10.1 s idle, 38 failed with 0 response bytes, though the tunnel was still open when each request went out.

Code:

• Client::ConnectionReuse#transmit and #resend? hold the decision, Connection#response_started? tracks the first response byte, and Request::Idempotency#replayable? the method and body rules. They're separate modules to stay within the Metrics limits; check_premature_eof moved into Connection::Internals for the same reason.
• test/support/scripted_server.rb is a small TCP/TLS server that records each request's bytes and connection, so the tests count deliveries instead of inferring them.
• 44 new tests: each resend case over plain, TLS, close_notify and a proxy tunnel, plus every gate above (no resend after a partial response, a malformed one, a 1xx, a read timeout, on a fresh connection, with an IO body, under .retriable), at most once, and the feature callbacks.
• test_connection_reuse_enabled_socket_issue_raises_for_non_idempotent_request pins down the unkeyed POST.

Validation:

• MRI 3.4.8: 2376 runs, 0 failures on seeds 4242 and 777, 100% line and branch coverage. rubocop and yardstick pass, steep shows the same 4 warnings as main
• JRuby 10.1.2.0: 2375 runs, 0 failures
• mutant: HTTP::Client::ConnectionReuse 95/95 killed and HTTP::Request::Idempotency 63/63. .mutant.yml ignores HTTP::Client*, so I passed them as subjects on the command line. Running it with more than one job needs the test CA fix in Keep the test CA in memory #857.

Open questions:

• replayable? and response_started? are public. Happy to mark them @api private.
• Idempotency-Key is still an IETF draft (-07). Go honors it; I can drop it and keep methods only.
• Net::HTTP lets you turn this off with max_retries = 0. Should there be an option here too?