Skip to content

Documentation vs actual code corrections - #508

Open
MichaelHuth wants to merge 25 commits into
mainfrom
docs/audit-fixes
Open

MichaelHuth wants to merge 25 commits into
mainfrom
docs/audit-fixes

Conversation

@MichaelHuth

@MichaelHuth MichaelHuth commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

A set of fixes from our local AI.

@MichaelHuth MichaelHuth self-assigned this Oct 15, 2026
@MichaelHuth MichaelHuth changed the title Documentation corrections Documentation vs actual code corrections Oct 15, 2026

@Garados007 Garados007 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remaining changes seems solid.

Comment thread helper/autorun-test-IP10-64bit.bat Resolved
Comment thread tests/UnitTests/PackageGlobalsTests.ipf Resolved
Comment thread docu/sphinx/source/guided-tour.rst Resolved
@Garados007 Garados007 assigned MichaelHuth and unassigned Garados007 Nov 6, 2026
Copilot AI balanced review requested due to automatic review settings November 9, 2026 15:11
@MichaelHuth MichaelHuth assigned Garados007 and unassigned MichaelHuth Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues allow retry-eligible REQUIRE failures to retry without the required opt-in and cause package-global checks to fail on valid tracing-enabled framework state.

5 open issues
What changed in this PR

This PR aligns igortest’s documentation and examples with framework behavior, while changing failed REQUIRE assertions to stop only the current test case.

Changes:

  • Updates REQUIRE handling and sorts regex-selected test suites alphabetically.
  • Adds regression tests and expands documentation for discovery, tags, skipping, errors, and coverage.
  • Adds unattended autorun scripts for Igor Pro 9 and 10.
File Description
tests/​UnitTests/​PackageGlobalsTests.ipf Adds package-object allowlist checks.
tests/​UnitTests/​main.ipf Includes the new package-global tests.
tests/​TestResultsTests/​TestOrderTests.ipf Tests regex and explicit suite ordering.
tests/​TestResultsTests/​GeneratorTests.ipf Tests generator tags far above functions.
tests/​TestResultsTests/​FailedTests.ipf Tests continuation after failed REQUIRE.
tests/​TestResultsTests/​ErroredTests.ipf Tests INFO with a pending runtime error.
tests/​TestResultsTests/​EasyTests.ipf Checks exclusion of complex-return functions.
Readme.md Clarifies coverage requirements and archive naming.
procedures/​igortest-reporting.ipf Marks REQUIRE aborts and unregisters background monitoring.
procedures/​igortest-constants.ipf Updates the REQUIRE retry flag comment.
procedures/​igortest-basics.ipf Handles REQUIRE-origin state, sorts suites, and expands API documentation.
procedures/​igortest-assertions.ipf Expands skip assertion documentation.
internal_dev/​IM_Test4.ipf Updates output to reflect REQUIRE continuation.
internal_dev/​example15-background_caught_abort_and_require.ipf Corrects comments about REQUIRE and background monitoring.
helper/​autorun-test-IP9-64bit.bat Adds unattended Igor Pro 9 64-bit execution.
helper/​autorun-test-IP9-32bit.bat Adds unattended Igor Pro 9 32-bit execution.
helper/​autorun-test-IP10-64bit.bat Adds unattended Igor Pro 10 execution.
docu/​sphinx/​source/​index.rst Links the new function-tag reference.
docu/​sphinx/​source/​guided-tour.rst Clarifies the example formula and corrects output samples.
docu/​sphinx/​source/​functiontags.rst Adds tag syntax and behavior reference.
docu/​sphinx/​source/​examples.rst Updates REQUIRE and runtime-error guidance.
docu/​sphinx/​source/​developer.rst Expands package-state lifecycle documentation.
docu/​sphinx/​source/​changelog.rst Records behavior changes and corrects historical text.
docu/​sphinx/​source/​basic.rst Clarifies discovery, REQUIRE, and skipping behavior.
docu/​sphinx/​source/​advanced.rst Expands hooks, selection, coverage, and autorun guidance.
docu/​examples/​example8-uncaught-runtime-errors.ipf Corrects comments about INFO placement.
docu/​examples/​example3-plain.ipf Demonstrates continued execution after failed REQUIRE.

Review effort: Balanced

Comment thread procedures/igortest-basics.ipf Resolved
Comment thread tests/UnitTests/PackageGlobalsTests.ipf Resolved
Comment thread docu/sphinx/source/examples.rst Resolved
Comment thread procedures/igortest-assertions.ipf Resolved
Comment thread procedures/igortest-basics.ipf Resolved
Garados007
Garados007 previously approved these changes Nov 9, 2026
@Garados007 Garados007 assigned MichaelHuth and unassigned Garados007 Nov 9, 2026
@Garados007

Copy link
Copy Markdown
Member

I didn't check the copilot output. The changes itself are fine by me.

A reader of the assertion overview came away believing that REQUIRE stops
only the running test case. It does not: ReportResults sets the run-global
abort flag before aborting, and the test run loop returns as soon as it
sees that flag, so every later test case of the current suite and every
later suite is skipped. The reference page has said the opposite ever
since the sphinx rework, while example3 and the developer notes have
always described the real behaviour, so the two halves of the
documentation contradicted each other.

The abort-code section made the same mistake in the other direction: it
promised that the framework continues with the next test case after
catching an Abort. That holds for a plain Abort written by the test
author, which is what example7 demonstrates, but not for the Abort the
framework raises itself for REQUIRE. The section now separates the two
cases and names the ways to stop a single test case instead.

Writing that up showed the two skip assertions missing from the prose
altogether, and the list of assertions without WARN/CHECK/REQUIRE
variants naming two of five.

Introduced in fb8ef0c (rework documentation, 2018-09-04).
The skip assertions went undocumented from the start in
d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS assertions, 2024-06-06).
The coverage documentation opened with "When running Igor Pro 9 or
newer", which presents an Igor version as the only gate. The real gate is
the Thread Utilities XOP together with build 38812: without them the
tracing branch is not compiled in at all, and RunTest aborts with a
message naming both. The XOP ships in this repository, yet neither the
Readme nor the installation steps said it has to be installed, so a
first attempt at coverage on a fresh setup ends in an abort that the
documentation does not mention.

While in there: the traceWinList parameter has described the result as an
RTF document since the feature landed, although the code has always
written .htm through SaveNotebook. The 10 000-line ceiling per procedure
file was never mentioned either, even though hitting it aborts the run
with a message of its own. And the regular expression published for CI
carried an unescaped dot plus anchors that cannot match, because the
coverage figure is emitted in the same output entry as the covered and
total line counts and therefore needs multiline matching.

Introduced in 615a366 (Add Code Coverage feature, 2021-06-23) and
4e7886c (Add documentation for coverage logging, 2021-07-09); the
coverage output itself arrived with
a653434 (Add coverage output to history and stdout at end of traced run, 2022-01-24).
The data generator section still told readers to put the tag "within four
lines above the test cases Function line". The framework never had such a
limit: GetFunctionTagWave asks ProcedureText for the whole comment block
back to the previous Function and scans every line of it, and no bound
exists anywhere in the history of that code. The number began as three and
was raised to four when the tag syntax was first spelled out; when the
rule was later corrected to "all lines up to the previous Function", three
of the four places were fixed and this one was missed, which is why the
same page stated both rules about 200 lines apart. The two IUTF_SKIP
sentences said "preceding the test case function" in the same loose way.

The deprecated UTF_ tag prefix was the other half of the same gap. The
rename added a compatibility layer that still accepts the old spelling,
but neither the changelog nor the prose ever mentioned it, and one 1.10
entry used the old name thirteen days after the rename landed.

Introduced in 17bbc72 (Added documentation for multi data test cases, 2019-04-12),
raised to four by 91b0399 (Manual 1.3.3 Test Case expanded, maximum tag lines added, 2020-05-22),
and left behind by cbe8cf5 (Documentation: Fix the number of tags, 2022-10-10).
The changelog entry came with 8146b9c (changelog.rst: Update it, 2023-02-28),
after e049435 (Rename: Function tags to have IUTF_ prefix, 2023-02-15).
The six _OVERRIDE hooks were documented with empty parameter lists, so a
reader could not learn that each of them receives a string, let alone
what it holds. The framework has always passed one: the run name to the
run hooks, the procedure window name to the suite hooks, and the full
function name with its multi data suffix to the case hooks. A hook
written without that parameter is a runtime error which cancels the run,
so the page described a signature that does not work.

The developer notes named a global which does not exist: the background
monitor flag is BCKG_Registered, not BCKG_Register. Every other name in
that section checks out against the code, so the section was otherwise
dependable; it now also states outright that it is not exhaustive.

Introduced in fb8ef0c (rework documentation, 2018-09-04) and
974e498 (Docs: Add developer section, 2022-12-13).
The example1 walkthrough told readers to look at line 10 and highlighted
only line 11. The NaN comparison it is about sits on lines 11 and 12. The
prose was correct when it was written: adding the version pragma to every
procedure file inserted a line at the top of the example and moved the
interesting lines down by one, and nothing in that commit updated the
line numbers quoted around it.

The TAP output file was described as tap_'time'.log. The generated name
carries the experiment name, the date and the time, and an existing file
is not overwritten but gets a counter appended.

The guided tour named a_0 as the graphene lattice constant without ever
giving it a value, while the procedure it tests substitutes 0.144, the
carbon-carbon bond distance, with the sqrt(3) that converts one into the
other folded into the square root. Every number involved is consistent;
only the connection was missing, which is enough to make a reader who
checks the code against the formula conclude the code is wrong.

Introduced in fb8ef0c (rework documentation, 2018-09-04); the line
numbers went stale with d590eb4 (Add version pragma to all ipf files, 2023-01-04).
The documented return value of RunTest was "total number of errors", but
the function returns NaN from five places where the run never started: an
unsaved experiment with JUNIT, TAP or tracing requested, an empty or
invalid procedure window list, no test case matching the given pattern,
and a rejected test setup. It also returns RUNTEST_RET_BCKG when the run
suspends for background activity and resumes on reentry. A CI script that
reads the return value as an error count takes -1 for one error and NaN
for nothing at all, so the contract belongs beside the parameter list
that doxygenfunction already renders.

Test case discovery was documented as accepting anything that "returns a
numeric value". FunctionList is called with VALTYPE:1, so a function
declared with /C and returning a complex value is never collected.

The compatibility names kept by the 1.10 rename were never listed
anywhere, although each still works and each is scheduled for removal.

The rest is prose: a sentence about expected failures which lost its
negation, Runtest with the wrong case inside an example invocation,
stdin where stdout was meant, a WARN_* "directive" that is an assertion,
a ref role without its space, and the usual misspellings.

The return value has been the only one documented since
c23cd35 (more work towards the first final release, 2012-12-21); the
discovery rule came with 17bbc72 (Added documentation for multi data
test cases, 2019-04-12) and the compatibility names were kept by
717d4f1 (Rename: procedure files to igortest-*, 2023-02-08) without a
word about them. Most typos date from fb8ef0c (rework documentation,
2018-09-04), one from 573c8f3 (Require UTF_EXPECTED_FAILURE to have
errors, 2023-01-11).
The installation steps told users to expect a file called
UnitTestingFramework-v1.09.zip. The archive is named from the tag, so the
prefix changed when the project was renamed and the current asset is
igortest-v1.10.zip. Users comparing the download against the instructions
found a name that matches neither part.

The wording itself dates from ba5c30a (Update Readme.md, 2023-01-13), but
it was correct then: the archive prefix moved later, in 4c1b419
(create-public-release.sh: Update it, 2023-03-30), which is what made this
sentence stale.
The background-activity documentation told readers that no begin/end hooks run
when the framework drops to the command prompt and resumes later. Only the
BEGIN half is true: the BEGIN hooks sit in the `if(!reentry)` branch, while the
END hooks come after the pending-monitor early return, so they do run -- once,
when the test case really ends. Someone who trusts the stated wording writes a
TEST_CASE_END_OVERRIDE that assumes it never sees a reentry, and gets the
opposite.

RegisterIUTFMonitor's reentryFunc parameter carried the same claim as "no hooks
are called", which is what users of the generated reference read.

Both date from the feature itself, 4f707b1 (Added support for test code running
in background functions, 2019-04-04), and 9f6a6f5 (Added documentation for user
background task support, 2019-04-05). The guard structure in 4f707b1 already put
the BEGIN hooks behind `if(!reentry)` and left the END hooks outside it, so the
wording was wrong from the start rather than made stale by a later change.
Every tag is explained somewhere in the prose, but nothing in the
documentation answers the question a user starts with: which tags exist at
all. Finding out whether a test case can opt out of wave tracking, for
instance, requires already suspecting that it can and then searching the
sources. Two tags are effectively invisible today: IUTF_NO_WAVE_TRACKING is
mentioned only inside the waveTrackingMode paragraph of RunTest's own
documentation, and IUTF_NO_SHUFFLE_TEST_CASE only inside the shuffle section.

The set keeps growing without a list that has to be updated alongside it;
UTF_FTAG_RETRY_FAILED arrived in 89faf11 (Allow rerun of flaky test cases,
2023-02-22) and went straight into prose only.

The page also records what the prose never stated: IUTF_TD_GENERATOR aborts
the run when its value is missing, while TAPDirective and TAPDescription
silently count as absent, and the procedure-tag scan is bounded to the first
20 lines of a file whereas the function-tag block is not.
The automate section tells the reader to put `helper/autorun-test.bat` into
their folder and run it. That file has never existed; helper/ holds scripts
named per Igor Pro version and architecture, and example6 describes them
correctly, so the two pages disagree about what to look for and the documented
procedure cannot be started, let alone finished.

The coverage section has the mirror-image problem. The recipe for regenerating
the htmloutput.txt snapshot was written down -- correctly -- but inside an rst
comment, so it is invisible on the rendered page. A reader looking at the
listing cannot tell where it came from or how to refresh it, which is how a
checked-in snapshot drifts from the real output without anyone having decided
to let that happen.

The script name dates from fb8ef0c (rework documentation, 2018-09-04); the
invisible recipe from eb4ce59 (Added description how to generate
htmloutput.txt included in documentation., 2021-07-17).
The section states that these variables live in root:Packages:igortest: and
lists BCKG_Registered, but registration writes six background-monitor
variables, so someone tracing a reentry finds one of six and no hint that the
others exist. abortFlag is listed without abortFromSkipFlag, the counterpart
that keeps a skipped test case from cancelling the whole run; the pair only
makes sense read together.

The remaining groups are named rather than described one variable at a time:
the COMP_* set behind the *_COMPILATION assertions, and the S* variables a test
case waiting on background activity leaves in the SaveState subfolder. That
second set follows the saved state structure, so spelling it out in prose would
only create another thing to keep in sync.

BCKG_Register, the name the page carried, never existed in the sources -- it was
invented in 974e498 (Docs: Add developer section, 2022-12-13). abortFromSkipFlag
arrived with d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS
assertions, 2024-06-06), which also left the changelog without an entry for the
two new assertions; that entry is added here.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 01:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation incorrectly presents IUTF_EXPECTED_FAILURE as stopping execution, and the new regression test does not verify cancellation of pending reentry after a failed REQUIRE.

2 open issues
5 resolved since last review

Review effort: Balanced

Comment thread procedures/igortest-reporting.ipf Resolved
Comment thread docu/sphinx/source/advanced.rst Resolved
Copilot AI balanced review requested due to automatic review settings November 10, 2026 10:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical abort-handling issues can suppress manual cancellation and prevent suspended test runs from resuming or finalizing after a background-task assertion failure.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread procedures/igortest-basics.ipf Resolved
Comment thread procedures/igortest-reporting.ipf Resolved
A failed REQUIRE set the run-wide abort flag, so a single unmet
precondition discarded the results of every remaining test case and test
suite. That put a per-test-case condition on the flag which exists for
the user's abort button, and left a partially executed run as the only
report.

A REQUIRE only has to guarantee that the assertions after it in the same
test case do not run. The framework already had that mechanism for skips:
SKIP_TESTCASE aborts out of the assertion and marks the origin, so the
catch around the test case can tell the framework's own abort apart from
a user abort. The new abortFromRequireFlag does the same for a failed
REQUIRE and the run continues with the next test case. The abort flag
keeps its meaning, so the sites which cancel the run on it, including
ReportErrorAndAbort and failing test hooks, are untouched.

Retrying a test case which aborted on a REQUIRE stays opt-in through
IUTF_RETRY_REQUIRES, now gated on the new flag as well. As CanRetry runs
after the catch, the flag is cleared only once the retry decision is made,
and before the next attempt or data generator index.

The catch around the test case unregisters the background monitor after
a failed REQUIRE. Without that the run would suspend into a reentry for a
test case which has already failed. The REQUIRE itself must not do it: in
a monitored background task it runs while the run is suspended, and only
the monitor resumes it. There the Abort stops the background task, and on
resuming the test case ends without calling its reentry function.

Only the Abort of the REQUIRE, code -3, is cleared. A test case which
caught that Abort can still be aborted by the user, by a stack overflow or
by AbortOnValue, and those are reported as before.

The IUTF_EXPECTED_FAILURE tag is no longer listed as a way to stop only
the current test case. With the tag a failed REQUIRE does not abort, so
the test case continues. The list names the failed REQUIRE instead.

FailedTests gains a RequireFail/RequireFail_Verify pair. It reports an
error when the behaviour regresses, because the neutralisation of the
REQUIRE's failure is what keeps the test suite clean.

ReentryTests gains RequireFailWithMonitor, which registers a monitor
before its REQUIRE fails, and RequireFailInTask, whose REQUIRE fails in
the monitored background task. Their reentry functions fail as well, so
the verification reports an error if one is still called, and the run
does not finish if it is never resumed. ErroredTests gains
AbortOnValueAfterCaughtRequire.

The flag setting goes back to c23cd35 (more work towards the first
final release, 2012-12-21). 324d953 (Document that a failed REQUIRE
cancels the whole test run, 2026-10-15) documented it as it stood.
basic.rst's second test-case rule now says the return has to be
real-valued, which is narrower than "numeric" in Igor terms. Nothing in
the test suite held that line: the filter argument in GetTestCaseList
reads as an implementation detail, and widening it to VALTYPE:3 collects
both the real and the complex case with no other discovery surprise, so
such a change would pass review as a one-character edit.

EasyTests gains a Function/C case and a sibling which asks the framework
itself for the test cases of that window and requires the complex one to
be absent. The sibling also insists on finding itself, so the check
cannot pass on an empty list.
advanced.rst used to require a data-generator tag within four lines above
the Function line. e208689 replaced that with the rule the code actually
implements, the whole comment block back to the previous function, but
nothing held the new wording. Passing -4 instead of -1 to ProcedureText
is a one-character change which reads as harmless in review, and it would
drop distant tags quietly: a test case whose generator tag went missing
fails with an unrelated message about the data generator, not about the
tag.

GeneratorTests gains a case with fourteen comment lines between its tag
and its Function line, checking that the generated value still arrived.

functiontags.rst now also says where the block ends for the first
function of a procedure window: with no preceding function, a tag in the
file header belongs to it.
Both console listings were written by hand in fb8ef0c (rework
documentation, 2018-09-04) and never regenerated, so they had drifted
from what RunTest prints:

- the suite line said "Unnamed" although the test suite is the
  procedure window, which the tour has just named test0,
- the second listing repeated the "Entering test case" line,
- the assertion text lacked the trailing period,
- the location used a "failed in line 11, procedure \"test0\"" form the
  code does not produce; the current one names the test case function
  and puts the window and line in parentheses,
- the two compared numbers were shown rounded, while
  GetNiceStringForNumber() prints them with UTF_DECIMAL_DIGITS_FP64
  significant digits,
- the end-of-run recap of the failed assertion was missing.

The screenshot below the first listing still showed an Igor Pro 7.08
session with #include "unit-testing" and the old suite line, so it is
replaced by one of the same run in Igor Pro 10.
The helper/ family grows one release at a time, last in 2dd1a2b (Add
autorun scripts for IP8, 2018-09-17), so the automated-run instructions
stop at an Igor version CI has been testing for years while the code
already branches on the next one (igortest-basics.ipf:10-12).

/I is still documented in Igor 9 as the way to force a new instance, so
the IP9 pair is the IP8 pair with the release folder name from the
installer's default location. They also pass /UNATTENDED, which Igor
added in 9.00 to suppress interactions that block an unattended run, for
example the About Autosave dialog shown at launch until the user opts
out of it. The doc says so, as the older scripts lack the flag.

The doc sentence records which versions have a script and what to do for
any other, since a script whose executable is missing only prints a
message rather than running anything.
Completes the family 231f3cb extended only as far as Igor 9. The path is
not a guess: Igor 10's own Getting Started.ihf and Advanced Topics.ihf
both name
C:\Program Files\WaveMetrics\Igor Pro 10 Folder\IgorBinaries_x64\Igor64.exe,
which is the executable on this machine.

Only a 64-bit script, because this Igor 10 install has no
IgorBinaries_Win32 folder while the Igor 9 one does. The doc sentence
lists Igor 10 as 64-bit only without claiming to know why.
SKIP_TESTCASE unregistered the background monitor itself. Called in a
monitored background task, it runs while the run is suspended, and only
the monitor resumes the run. The run therefore never finished: no end
hooks, no remaining test cases, and autorun did not quit. As for a failed
REQUIRE, the catch around the test case now unregisters the monitor, and
on resuming the test case ends without calling its reentry function.

The catch also cleared every abort code once the skip flag was set, and
only the catch reset that flag. A test case which caught the Abort of
SKIP_TESTCASE therefore lost a later user abort, stack overflow or
AbortOnValue. Only code -3 is cleared now, and the flag is reset after
every call of a test case.

ReentryTests gains SkipInTask and ErroredTests gains
AbortOnValueAfterCaughtSkip, both like their REQUIRE counterparts.

Bug introduced in d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS assertions, 2024-06-06)
Copilot AI balanced review requested due to automatic review settings November 10, 2026 11:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@MichaelHuth

Copy link
Copy Markdown
Member Author

@Garados007 I fixed some CoPilot reviews

@t-b

t-b commented Nov 17, 2026

Copy link
Copy Markdown
Member

@Garados007 Merge?

@Garados007

Copy link
Copy Markdown
Member

Yes. I had some issues checking the CI (will create some issues in a moment). Everything is fine and can be merged.

@Garados007

Copy link
Copy Markdown
Member

The CI is almost green with the following exceptions:

  1. pre-commit failed for an unrelated reason. I created pre-commit does not succeed #514 for it.
  2. documentation failed for a snippet which context wasn't changed here. I suggest fixing it outside this PR (this one is very large already). I created Documentation warning: TAP_AreAllFunctionsSkip #515 for it.
  3. The did run the failed Igor runs locally - this was faster than restarting them in the CI. If you want you can rerun them in CI and wait for the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants