Skip to content

Implement #5119 Feat/enrich wait for http - #5442

Open
Snooz82 wants to merge 9 commits into
mainfrom
feat/enrich-wait-for-http
Open

Snooz82 wants to merge 9 commits into
mainfrom
feat/enrich-wait-for-http

Conversation

@Snooz82

@Snooz82 Snooz82 commented Nov 2, 2026

Copy link
Copy Markdown
Member

No description provided.

Copilot AI balanced review requested due to automatic review settings November 2, 2026 20:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The first message includes both the full legacy POST body and a raw-byte chunk, causing previously supported 3.5 MB requests to exceed Python gRPC’s 4 MiB receive limit and fail in both capture keywords.

1 open issue
What changed in this PR

Enriches Browser’s HTTP capture keywords with detailed request and response metadata, binary-safe body views, and optional response-body exclusion across the Python, gRPC, and Node layers.

Changes:

  • Adds typed capture results with headers, connection details, and raw, text, and JSON body views.
  • Streams raw payload bytes and supports include_body=False.
  • Adds serialization support and unit and acceptance coverage.
File Description
utest/​test_wait_for_http.py Tests capture metadata, body views, parsing, and logging.
utest/​test_network.py Tests strict handling of non-JSON numeric constants.
utest/​test_json_dotdict.py Tests byte serialization and copy behavior.
protobuf/​playwright.proto Adds byte fields, body exclusion, and request streaming.
node/​playwright-wrapper/​response-util.ts Initializes new byte fields in response helpers.
node/​playwright-wrapper/​network.ts Collects enriched metadata and chunks captured payloads.
node/​playwright-wrapper/​grpc-service.ts Shares streaming handling between capture keywords.
node/​playwright-wrapper/​chunking.ts Adds raw-byte chunking.
node/​playwright-wrapper/​__tests__/​network.test.ts Tests enriched captures and payload streaming.
node/​playwright-wrapper/​__tests__/​chunking.test.ts Tests binary chunk boundaries.
node/​dynamic-test-app/​src/​server.ts Adds endpoints for capture acceptance tests.
Browser/​utils/​json_dotdict.py Adds JSON-serializable dictionaries containing bytes.
Browser/​utils/​data_types.py Defines and documents capture result types.
Browser/​utils/​__init__.py Exports capture types and serialization utility.
Browser/​keywords/​network.py Reassembles streams and exposes enriched capture results.
Browser/​gen_stub.py Imports capture types into generated Python stubs.
atest/​test/​03_Waiting/​captured_http.robot Exercises enriched captures against the Test App.
atest/​library/​http_headers.py Adds a case-insensitive header lookup helper.

Review effort: Balanced

Comment thread node/playwright-wrapper/network.ts Outdated
Comment on lines +141 to +143
...jsonResponse(i === 0 ? JSON.stringify(captured.meta) : '', `${captured.log}, chunk ${i + 1}/${count}`),
postDataBytes: postDataChunks[i] ?? Buffer.alloc(0),
bodyBytes: bodyChunks[i] ?? Buffer.alloc(0),
@Snooz82
Snooz82 force-pushed the feat/enrich-wait-for-http branch from 9cc238b to 5a856f5 Compare November 14, 2026 22:24
…onse

Implements the agreed design of #5119: both keywords mirror Playwright's
Request and Response and keep every legacy key and parsing rule.

- New typed return values CapturedRequest and CapturedResponse, with key
  tables and parsing rules in their docstrings.
- Request: allHeaders, headersArray, postDataBuffer, postDataJSON,
  resourceType, isNavigationRequest, redirectedFrom/To, frame. The
  nested request of Wait For Response now has url and the same keys.
- Response: allHeaders (with set-cookie), headersArray, bytes, text,
  json, httpVersion, serverAddr, securityDetails, fromServiceWorker.
- Wait For Response gets a named-only include_body argument.
- Bodies cross gRPC as raw bytes (new bodyBytes/postDataBytes fields,
  both rpcs stream). Python derives text, json and the legacy body, so
  binary bodies survive (#3320).
- The results are JsonDotDicts: json.dumps writes bytes as base64.
- Fixes: falsy JSON bodies no longer drop the body key, JSON string
  bodies are no longer decoded twice, NaN/Infinity are not parsed as
  JSON (also in HTTP).
Covers set-cookie in allHeaders and headersArray, byte-exact binary
bodies, the text/json views, kept {} and [] bodies, include_body=False,
redirect metadata, connection details, JSON serialization of bytes,
request details for fetch and navigation requests, form post data, and
that Wait For Request returns before a stalled response arrives.

The docs now say that headersArray keeps the original header name case
and that securityDetails may be an empty dictionary without TLS.
- Node signals body and post data with explicit hasBody/hasPostData
  flags instead of keys set to null.
- One headersOf helper builds allHeaders/headersArray for requests and
  responses. The message builder is renamed to capturedMessages, and
  the streaming handler logs the grpc started/succeeded/failed events
  again.
- Python keeps the body keys in RESPONSE_BODY_KEYS/REQUEST_BODY_KEYS.
  Bodies and post data stay out of the INFO log and are logged at
  TRACE: as formatted JSON, as text when UTF-8, otherwise as base64.
- Docs describe the fields of headersArray, frame, serverAddr and
  securityDetails inline, because Libdoc does not render the nested
  TypedDicts. The &{var} = caveat now says that only the outer
  dictionary is copied, so request.postDataBuffer keeps its bytes.
- The headersArray filter in the acceptance test moves to an atest
  library keyword.
@Snooz82
Snooz82 force-pushed the feat/enrich-wait-for-http branch from 5a856f5 to 173d490 Compare November 15, 2026 13:03
…O logs

- Wait For Response reads the body and all extra fields within the
  remaining keyword timeout and fails otherwise.
- include_body=False also skips securityDetails, which WebKit reports
  only after the whole body.
- allHeaders and headersArray are logged at DEBUG only, since they hold
  cookies.
- Body tracing is off outside Robot Framework, empty chunk log lines are
  not written, the json docs are corrected and the form post atest uses
  an existing endpoint.
The Test App gets /api/slow-body, which sends the headers at once and
finishes the body later. With include_body=False the keyword returns
before the body arrives, and reading the body is limited by the timeout.
Both tests bound the elapsed time well below the body duration.
WebKit keeps no body for a response without content, so response.body()
fails and the body became None. A 204, 205 or 304 response or one with
content-length 0 now returns an empty body when it cannot be read.
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants