Skip to content

Test Hessian-vector products over a sequence of points - #36

Open
glebbelov wants to merge 2 commits into
masterfrom
test-hvcomp-point-sequence
Open

glebbelov wants to merge 2 commits into
masterfrom
test-hvcomp-point-sequence

Conversation

@glebbelov

Copy link
Copy Markdown
Contributor

hvseq reads each fixture with pfgh_read, calls hvcomp once at each of 12 points and compares the products with central differences of the Lagrangian gradient. This covers the stale products that solvers2 returned after x changed (fixed in 5b49061), which derivcheck does not reach: it calls hvcomp once per variable at each of its two points. Without that fix 13 of the new tests fail with solvers2.

hvseq reads each fixture with pfgh_read, calls hvcomp once at each of 12
points and compares the products with central differences of the
Lagrangian gradient.  This covers the stale products that solvers2
returned after x changed (fixed in 5b49061), which derivcheck does not
reach: it calls hvcomp once per variable at each of its two points.
Without that fix 13 of the new tests fail with solvers2.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@glebbelov glebbelov self-assigned this Nov 12, 2026
marcos-dv
marcos-dv previously approved these changes Nov 12, 2026
@fdabrandao

Copy link
Copy Markdown
Member

Review of test/hvseq.c

Bugs

  1. Heap overflow (confirmed from the diff). w is allocated with 4*n reals (line 140), but conval(x, w + n, &ne) (line 154) writes m constraint values starting at w[n]. Any fixture with m > 3*n (e.g. 2 variables, 7 constraints) writes past the end of the buffer. Allocate max(4n, n+m) reals or use a separate buffer for the constraint values.

  2. Test can pass without checking anything. The program returns bad ? 3 : 0, so if every point is skipped because of evaluation errors (skipped == K), it exits 0 and CTest reports a pass. It should fail when no point was actually checked.

  3. Finite-difference step ignores scale. h = 1e-6 is absolute, while derivcheck uses 1e-5*(1+|x|). With large |x| or large gradients, rounding error can exceed the 1e-4 relative tolerance and report mismatches even when hvcomp is correct.

  4. Random points are not kept inside the variable bounds. Unlike derivcheck's make_point, X0 + .3*u*(1+|X0|) is never moved back inside the bounds. Fixtures that use log/sqrt/x^1.5/abs and are only safe inside their bounds can then be skipped silently (which makes Bad output in cplex driver for writeprob option #2 worse) or report mismatches that aren't real bugs.

Design / cleanup

  • lagrgrad and the inline random-number generator duplicate derivcheck's lag_grad/rnd(), and they have already drifted apart (the bounds handling in Remove BUILD_DYNRT_LIBS, install missing headers and export cmake targets #4). A shared helper in test/, or a sequence mode in derivcheck, would keep them in sync and reuse the existing tolerances.
  • lagrgrad calls congrd once per constraint, which is O(m·n) per call; jacval plus a walk over Cgrad is O(nonzeros). This is negligible for the current fixtures.
  • Adding a second checker program changes the aslTestChecker signature in test/CMakeLists.txt; a mode in derivcheck would avoid that.

I'd suggest fixing #1 and #2 before merging, and #3/#4 before more fixtures are added.


Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
🤖 Generated with Claude Code

@fdabrandao

Copy link
Copy Markdown
Member

I found two issues in this change:

  1. [P2] Constraint values can overflow the workspace — test/hvseq.c:40. w holds 4*n values, but conval(x, w + n, ...) writes m values, overflowing whenever m > 3*n. A valid model with one variable and four quadratic constraints reproduces a heap-buffer-overflow under AddressSanitizer in both libraries. Please allocate a separate constraint buffer or size w to accommodate both uses.

  2. [P2] Infinite Hessian products incorrectly pass comparison — test/hvseq.c:68–69. With an infinite hv[j] and finite reference, both the difference and tolerance become infinity, so inf <= inf passes. In a temporary checker copy, injecting hv[0] = Infinity immediately after each hvcomp call for pr34_pow15 reports zero mismatches and exits successfully. Please explicitly reject non-finite results before applying the tolerance.

Validation: all 882 registered tests passed locally on macOS with BUILD_ASL_TESTS=ON, USE_LTO=OFF, and a Debug build, including the 102 new sequence tests. The issues above were confirmed with targeted reproductions.

Co-Authored-By: GPT-6 noreply@openai.com

@fdabrandao

Copy link
Copy Markdown
Member

Follow-up after reading the Codex review:

  • Line-number correction: the line numbers in my earlier comment were positions in the diff, not in the file. In test/hvseq.c, the w allocation is line 40, conval(x, w + n, &ne) is line 54, and return bad ? 3 : 0 is line 77. The overflow is the same one Codex reproduced with ASan.
  • Infinite results: I agree with the infinity finding. It goes both ways: an infinite finite-difference value d with a finite hv[j] also gets inf <= inf and passes. NaN is already rejected by the !(... <= ...) form. A check on both values covers it, e.g. if (!isfinite(d) || !isfinite(hv[j]) || !(fabs(d - hv[j]) <= 1e-4*sc)), or x - x == 0. if C99 isfinite should be avoided.
  • No points checked: related to my point Bad output in cplex driver for writeprob option #2. Since all 102 new tests pass, it would help to confirm that none of them prints skipped= equal to points=. Those tests would currently pass without comparing any products.

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
🤖 Generated with Claude Code

@fdabrandao

Copy link
Copy Markdown
Member

Follow-up on Claude's review, with additional validation:

  • The fixed finite-difference step produces reproducible false failures. For a one-variable fixture minimize f: x^2 with starting value 1e12, both hvseq-asl and hvseq-asl2 report 12/12 mismatching points (hvcomp 2, fd 0, exit 3). At that scale, x +/- 1e-6*p rounds back to x. Both derivcheck variants pass the same fixture. The directional step should account for variable/direction scale and produce distinct representable perturbed points.
  • The all-skipped concern is valid, but none of the current sequence tests skips every point. I ran all 102: pr35_obj_sqrt and evalerr_03_objsqrt_lincons each skip 6/12 points in both libraries; all others skip zero. Individual evaluation-error skips are explicitly documented in test/README.md, so I would preserve those while making insufficient coverage explicit. For a sequence regression, checking only one point also cannot establish behavior across point changes; track successful comparisons at distinct points rather than accepting any run with zero mismatches.
  • Bounds handling needs to preserve smooth finite-difference samples. Keeping the central point inside variable bounds helps avoid domain errors, but the two perturbed points must also remain in the function domain and avoid crossing a kink. Bounds alone do not establish that condition.

The buffer overflow and non-finite-result acceptance remain the two independently reproduced issues in my preceding comment. I would treat the proposed helper consolidation as optional cleanup.

Co-Authored-By: GPT-6 noreply@openai.com

…and non-finite checks

Review findings on the sequence checker:
- constraint values overflowed the workspace when m > 3n: separate buffer;
- a run that skipped every point passed: exit 4 when fewer than two
  points are compared;
- an infinite product or difference passed the tolerance test: non-finite
  values are mismatches;
- the absolute step 1e-6 gave false mismatches at large |x|: the direction
  is scaled with the point and the step reduced near bounds;
- points are moved strictly inside the variable bounds, as in derivcheck;
- hvcomp got a one-element weight vector on models with several
  objectives and read past it: no weight vector is passed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@glebbelov

Copy link
Copy Markdown
Contributor Author

Comments addressed (fixing 1 more issue)

Comment thread test/hvseq.c
}

int main(int argc, char **argv) {
FILE *nl; int K = argc > 2 ? atoi(argv[2]) : 12, n, m, i, k, r;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

npoints is not validated. hvseq stub -1 aborts in Malloc (malloc(18446744073709551600) failure), and hvseq stub 0 or hvseq stub abc (atoi gives 0) runs on zero points and exits 0, because checked < (K<2?K:2) is 0 < 0. CMake always uses the default of 12, so this only affects manual runs. Suggest rejecting K < 1 with the usage message and exit status 2.

Comment thread test/hvseq.c
for (i = 0; i < n_con && !ne; i++) congrd(i, x, w, &ne);
if (ne) return 1;
/* no weight vector: it would need one weight per objective */
hvcomp(hv, p, n_obj ? 0 : -1, 0, n_con ? Y : 0);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only objective 0 is exercised: hvcomp is called with objective index 0 here, and lagrgrad uses objgrd(0, ...), so objectives 1 and later are never covered. The README documents that sequences are covered for hvcomp only, but not this limit. Suggest adding a line to the README's limitations list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants