e2e: capture compare Job/pod status before declaring compare_pvc_data failures #2195

Summary

compare_pvc_data reports a bare "Unexpected comparison result" when the compare
Job does not look clean, then deletes the Job in an always: block before any
diagnostics are captured. A recent CI failure shows the check failing while the
Job's own pod log proves the comparison passed, and the pod-level cause is
unrecoverable from the artifacts. Every future failure of this role lands in the
same dead end.

Evidence (2026-10-28, PR #2193)

• ci/prow/e2e-openshift build 2115435760919252992
  failed with Failures: test_restic_manual_normal.yml (only 1 of 38 playbooks).

• Task compare_pvc_data : Determine whether result was expected failed at
  15:03:04Z with "Unexpected comparison result".

• One second later, the role's own Dump Job log shows the compare container had
  completed successfully:

  Validating contents: PVC1 -> PVC2 ... /mnt2/datafile: OK
  Validating contents: PVC2 -> PVC1 ... /mnt/datafile: OK
  ... all file contents matched
  ... all symlinks matched
  /mnt directory uid:0 gid:1000750000 mode:2775
  /mnt/datafile regular file uid:1000750000 gid:1000750000 mode:664
  Looking for differences:
  ... none found
  

• Timing: Job created 15:02:58Z, Wait for compare to complete returned on its
  first poll at 15:03:03Z, failure declared at 15:03:04Z, Job deleted at
  15:03:05Z. The same commit passed this job 15 hours earlier, so this is a
  flake — but the log cannot say which part of the cluster misbehaved.

Problems in the role

test-e2e/roles/compare_pvc_data/tasks/main.yml:

1. Diagnostics are discarded. Determine whether result was expected
   (lines 137-146) emits a fixed string; res2 (Job conditions, counts) is never
   printed, and the pod's status.containerStatuses[].lastState/reason and
   namespace Events are never collected.
2. The evidence is deleted. The always: block (line 165) deletes the Job
   immediately after the log dump, so by the time dump_logs runs (minutes
   later) neither the Job nor its pod exists.
3. The decision is not authoritative. The wait loop (lines 118-135) stops on
   the first of succeeded == 1 or failed == 1, and the check then fails on
   status.failed > 0 regardless of succeeded or the Job's Complete/Failed
   condition. A contradictory or transient pod count can therefore fail a
   comparison that demonstrably passed, and "the data differed" is
   indistinguishable from "the pod did not complete normally".

Proposal

• Before deciding, capture and print: the Job's status (conditions, active,
  succeeded, failed), each pod's phase and containerStatuses[].state/
  lastState (reason, exit code, message), and the namespace Events for the
  Job/pod.
• Include that snapshot in the failure message (or dump it to a file the
  playbook prints) so the cause survives in the CI log; only then delete the Job.
• Decide pass/fail from the Job's completion state, or at least prefer
  succeeded/Complete over a raw failed-pod count.
• Keep the existing container-log dump.

Acceptance criteria

• A failing compare_pvc_data run prints the Job status/conditions and the
  pod termination state, exit code, and reason in the CI log.
• Failure output distinguishes "data mismatch" from "compare Job/pod did not
  complete normally".
• Pass/fail is based on Job completion state (or prefers succeeded) rather
  than a bare failed-pod count.
• Existing behavior is unchanged for the happy path and for the
  should_fail: true callers (test_restic_with_previous,
  test_restic_with_restoreasof, test_restic_with_restoreoptions,
  test_volumepopulator).
• No new permissions: same namespace-scoped reads only.

Notes

• Not urgent/blocking; the underlying failure was a flake, and the VolSync data
  path was verified correct by the compare itself.
• Related: PR ci: run the Go test suite with -race #2193, e2e: assert sync status, conditions, and resource cleanup in existing playbooks #2188 (e2e assertions), e2e: cover paused, Clone copy method, CLI migration, and a negative path #2190 (e2e coverage).

(AI generated)