Repository navigation
Documentation vs actual code corrections - #508
MichaelHuth wants to merge 25 commits into
Conversation
Garados007
left a comment
There was a problem hiding this comment.
The remaining changes seems solid.
3d50216 to
4f4b53e
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Active issues allow retry-eligible REQUIRE failures to retry without the required opt-in and cause package-global checks to fail on valid tracing-enabled framework state.
5 open issues
Preserve REQUIRE-origin flag until retry eligibility is checked · New Add tracing-created globals to the test allowlists · New Clarify that AbortOnRTE can leave a pending runtime error · New Document that SKIP_TESTCASE preserves an existing failure · New Clarify that testCase filters function names, not procedure windows · New
What changed in this PR
This PR aligns igortest’s documentation and examples with framework behavior, while changing failed REQUIRE assertions to stop only the current test case.
Changes:
- Updates REQUIRE handling and sorts regex-selected test suites alphabetically.
- Adds regression tests and expands documentation for discovery, tags, skipping, errors, and coverage.
- Adds unattended autorun scripts for Igor Pro 9 and 10.
| File | Description |
|---|---|
| tests/UnitTests/PackageGlobalsTests.ipf | Adds package-object allowlist checks. |
| tests/UnitTests/main.ipf | Includes the new package-global tests. |
| tests/TestResultsTests/TestOrderTests.ipf | Tests regex and explicit suite ordering. |
| tests/TestResultsTests/GeneratorTests.ipf | Tests generator tags far above functions. |
| tests/TestResultsTests/FailedTests.ipf | Tests continuation after failed REQUIRE. |
| tests/TestResultsTests/ErroredTests.ipf | Tests INFO with a pending runtime error. |
| tests/TestResultsTests/EasyTests.ipf | Checks exclusion of complex-return functions. |
| Readme.md | Clarifies coverage requirements and archive naming. |
| procedures/igortest-reporting.ipf | Marks REQUIRE aborts and unregisters background monitoring. |
| procedures/igortest-constants.ipf | Updates the REQUIRE retry flag comment. |
| procedures/igortest-basics.ipf | Handles REQUIRE-origin state, sorts suites, and expands API documentation. |
| procedures/igortest-assertions.ipf | Expands skip assertion documentation. |
| internal_dev/IM_Test4.ipf | Updates output to reflect REQUIRE continuation. |
| internal_dev/example15-background_caught_abort_and_require.ipf | Corrects comments about REQUIRE and background monitoring. |
| helper/autorun-test-IP9-64bit.bat | Adds unattended Igor Pro 9 64-bit execution. |
| helper/autorun-test-IP9-32bit.bat | Adds unattended Igor Pro 9 32-bit execution. |
| helper/autorun-test-IP10-64bit.bat | Adds unattended Igor Pro 10 execution. |
| docu/sphinx/source/index.rst | Links the new function-tag reference. |
| docu/sphinx/source/guided-tour.rst | Clarifies the example formula and corrects output samples. |
| docu/sphinx/source/functiontags.rst | Adds tag syntax and behavior reference. |
| docu/sphinx/source/examples.rst | Updates REQUIRE and runtime-error guidance. |
| docu/sphinx/source/developer.rst | Expands package-state lifecycle documentation. |
| docu/sphinx/source/changelog.rst | Records behavior changes and corrects historical text. |
| docu/sphinx/source/basic.rst | Clarifies discovery, REQUIRE, and skipping behavior. |
| docu/sphinx/source/advanced.rst | Expands hooks, selection, coverage, and autorun guidance. |
| docu/examples/example8-uncaught-runtime-errors.ipf | Corrects comments about INFO placement. |
| docu/examples/example3-plain.ipf | Demonstrates continued execution after failed REQUIRE. |
Review effort: Balanced
|
I didn't check the copilot output. The changes itself are fine by me. |
A reader of the assertion overview came away believing that REQUIRE stops only the running test case. It does not: ReportResults sets the run-global abort flag before aborting, and the test run loop returns as soon as it sees that flag, so every later test case of the current suite and every later suite is skipped. The reference page has said the opposite ever since the sphinx rework, while example3 and the developer notes have always described the real behaviour, so the two halves of the documentation contradicted each other. The abort-code section made the same mistake in the other direction: it promised that the framework continues with the next test case after catching an Abort. That holds for a plain Abort written by the test author, which is what example7 demonstrates, but not for the Abort the framework raises itself for REQUIRE. The section now separates the two cases and names the ways to stop a single test case instead. Writing that up showed the two skip assertions missing from the prose altogether, and the list of assertions without WARN/CHECK/REQUIRE variants naming two of five. Introduced in fb8ef0c (rework documentation, 2018-09-04). The skip assertions went undocumented from the start in d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS assertions, 2024-06-06).
The coverage documentation opened with "When running Igor Pro 9 or newer", which presents an Igor version as the only gate. The real gate is the Thread Utilities XOP together with build 38812: without them the tracing branch is not compiled in at all, and RunTest aborts with a message naming both. The XOP ships in this repository, yet neither the Readme nor the installation steps said it has to be installed, so a first attempt at coverage on a fresh setup ends in an abort that the documentation does not mention. While in there: the traceWinList parameter has described the result as an RTF document since the feature landed, although the code has always written .htm through SaveNotebook. The 10 000-line ceiling per procedure file was never mentioned either, even though hitting it aborts the run with a message of its own. And the regular expression published for CI carried an unescaped dot plus anchors that cannot match, because the coverage figure is emitted in the same output entry as the covered and total line counts and therefore needs multiline matching. Introduced in 615a366 (Add Code Coverage feature, 2021-06-23) and 4e7886c (Add documentation for coverage logging, 2021-07-09); the coverage output itself arrived with a653434 (Add coverage output to history and stdout at end of traced run, 2022-01-24).
The data generator section still told readers to put the tag "within four lines above the test cases Function line". The framework never had such a limit: GetFunctionTagWave asks ProcedureText for the whole comment block back to the previous Function and scans every line of it, and no bound exists anywhere in the history of that code. The number began as three and was raised to four when the tag syntax was first spelled out; when the rule was later corrected to "all lines up to the previous Function", three of the four places were fixed and this one was missed, which is why the same page stated both rules about 200 lines apart. The two IUTF_SKIP sentences said "preceding the test case function" in the same loose way. The deprecated UTF_ tag prefix was the other half of the same gap. The rename added a compatibility layer that still accepts the old spelling, but neither the changelog nor the prose ever mentioned it, and one 1.10 entry used the old name thirteen days after the rename landed. Introduced in 17bbc72 (Added documentation for multi data test cases, 2019-04-12), raised to four by 91b0399 (Manual 1.3.3 Test Case expanded, maximum tag lines added, 2020-05-22), and left behind by cbe8cf5 (Documentation: Fix the number of tags, 2022-10-10). The changelog entry came with 8146b9c (changelog.rst: Update it, 2023-02-28), after e049435 (Rename: Function tags to have IUTF_ prefix, 2023-02-15).
The six _OVERRIDE hooks were documented with empty parameter lists, so a reader could not learn that each of them receives a string, let alone what it holds. The framework has always passed one: the run name to the run hooks, the procedure window name to the suite hooks, and the full function name with its multi data suffix to the case hooks. A hook written without that parameter is a runtime error which cancels the run, so the page described a signature that does not work. The developer notes named a global which does not exist: the background monitor flag is BCKG_Registered, not BCKG_Register. Every other name in that section checks out against the code, so the section was otherwise dependable; it now also states outright that it is not exhaustive. Introduced in fb8ef0c (rework documentation, 2018-09-04) and 974e498 (Docs: Add developer section, 2022-12-13).
The example1 walkthrough told readers to look at line 10 and highlighted only line 11. The NaN comparison it is about sits on lines 11 and 12. The prose was correct when it was written: adding the version pragma to every procedure file inserted a line at the top of the example and moved the interesting lines down by one, and nothing in that commit updated the line numbers quoted around it. The TAP output file was described as tap_'time'.log. The generated name carries the experiment name, the date and the time, and an existing file is not overwritten but gets a counter appended. The guided tour named a_0 as the graphene lattice constant without ever giving it a value, while the procedure it tests substitutes 0.144, the carbon-carbon bond distance, with the sqrt(3) that converts one into the other folded into the square root. Every number involved is consistent; only the connection was missing, which is enough to make a reader who checks the code against the formula conclude the code is wrong. Introduced in fb8ef0c (rework documentation, 2018-09-04); the line numbers went stale with d590eb4 (Add version pragma to all ipf files, 2023-01-04).
The documented return value of RunTest was "total number of errors", but the function returns NaN from five places where the run never started: an unsaved experiment with JUNIT, TAP or tracing requested, an empty or invalid procedure window list, no test case matching the given pattern, and a rejected test setup. It also returns RUNTEST_RET_BCKG when the run suspends for background activity and resumes on reentry. A CI script that reads the return value as an error count takes -1 for one error and NaN for nothing at all, so the contract belongs beside the parameter list that doxygenfunction already renders. Test case discovery was documented as accepting anything that "returns a numeric value". FunctionList is called with VALTYPE:1, so a function declared with /C and returning a complex value is never collected. The compatibility names kept by the 1.10 rename were never listed anywhere, although each still works and each is scheduled for removal. The rest is prose: a sentence about expected failures which lost its negation, Runtest with the wrong case inside an example invocation, stdin where stdout was meant, a WARN_* "directive" that is an assertion, a ref role without its space, and the usual misspellings. The return value has been the only one documented since c23cd35 (more work towards the first final release, 2012-12-21); the discovery rule came with 17bbc72 (Added documentation for multi data test cases, 2019-04-12) and the compatibility names were kept by 717d4f1 (Rename: procedure files to igortest-*, 2023-02-08) without a word about them. Most typos date from fb8ef0c (rework documentation, 2018-09-04), one from 573c8f3 (Require UTF_EXPECTED_FAILURE to have errors, 2023-01-11).
The installation steps told users to expect a file called UnitTestingFramework-v1.09.zip. The archive is named from the tag, so the prefix changed when the project was renamed and the current asset is igortest-v1.10.zip. Users comparing the download against the instructions found a name that matches neither part. The wording itself dates from ba5c30a (Update Readme.md, 2023-01-13), but it was correct then: the archive prefix moved later, in 4c1b419 (create-public-release.sh: Update it, 2023-03-30), which is what made this sentence stale.
The background-activity documentation told readers that no begin/end hooks run when the framework drops to the command prompt and resumes later. Only the BEGIN half is true: the BEGIN hooks sit in the `if(!reentry)` branch, while the END hooks come after the pending-monitor early return, so they do run -- once, when the test case really ends. Someone who trusts the stated wording writes a TEST_CASE_END_OVERRIDE that assumes it never sees a reentry, and gets the opposite. RegisterIUTFMonitor's reentryFunc parameter carried the same claim as "no hooks are called", which is what users of the generated reference read. Both date from the feature itself, 4f707b1 (Added support for test code running in background functions, 2019-04-04), and 9f6a6f5 (Added documentation for user background task support, 2019-04-05). The guard structure in 4f707b1 already put the BEGIN hooks behind `if(!reentry)` and left the END hooks outside it, so the wording was wrong from the start rather than made stale by a later change.
Every tag is explained somewhere in the prose, but nothing in the documentation answers the question a user starts with: which tags exist at all. Finding out whether a test case can opt out of wave tracking, for instance, requires already suspecting that it can and then searching the sources. Two tags are effectively invisible today: IUTF_NO_WAVE_TRACKING is mentioned only inside the waveTrackingMode paragraph of RunTest's own documentation, and IUTF_NO_SHUFFLE_TEST_CASE only inside the shuffle section. The set keeps growing without a list that has to be updated alongside it; UTF_FTAG_RETRY_FAILED arrived in 89faf11 (Allow rerun of flaky test cases, 2023-02-22) and went straight into prose only. The page also records what the prose never stated: IUTF_TD_GENERATOR aborts the run when its value is missing, while TAPDirective and TAPDescription silently count as absent, and the procedure-tag scan is bounded to the first 20 lines of a file whereas the function-tag block is not.
The automate section tells the reader to put `helper/autorun-test.bat` into their folder and run it. That file has never existed; helper/ holds scripts named per Igor Pro version and architecture, and example6 describes them correctly, so the two pages disagree about what to look for and the documented procedure cannot be started, let alone finished. The coverage section has the mirror-image problem. The recipe for regenerating the htmloutput.txt snapshot was written down -- correctly -- but inside an rst comment, so it is invisible on the rendered page. A reader looking at the listing cannot tell where it came from or how to refresh it, which is how a checked-in snapshot drifts from the real output without anyone having decided to let that happen. The script name dates from fb8ef0c (rework documentation, 2018-09-04); the invisible recipe from eb4ce59 (Added description how to generate htmloutput.txt included in documentation., 2021-07-17).
The section states that these variables live in root:Packages:igortest: and lists BCKG_Registered, but registration writes six background-monitor variables, so someone tracing a reentry finds one of six and no hint that the others exist. abortFlag is listed without abortFromSkipFlag, the counterpart that keeps a skipped test case from cancelling the whole run; the pair only makes sense read together. The remaining groups are named rather than described one variable at a time: the COMP_* set behind the *_COMPILATION assertions, and the S* variables a test case waiting on background activity leaves in the SaveState subfolder. That second set follows the saved state structure, so spelling it out in prose would only create another thing to keep in sync. BCKG_Register, the name the page carried, never existed in the sources -- it was invented in 974e498 (Docs: Add developer section, 2022-12-13). abortFromSkipFlag arrived with d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS assertions, 2024-06-06), which also left the changelog without an entry for the two new assertions; that entry is added here.
4f4b53e to
310098d
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The documentation incorrectly presents IUTF_EXPECTED_FAILURE as stopping execution, and the new regression test does not verify cancellation of pending reentry after a failed REQUIRE.
2 open issues
5 resolved since last review
Review effort: Balanced
310098d to
7faa0b3
Compare
A failed REQUIRE set the run-wide abort flag, so a single unmet precondition discarded the results of every remaining test case and test suite. That put a per-test-case condition on the flag which exists for the user's abort button, and left a partially executed run as the only report. A REQUIRE only has to guarantee that the assertions after it in the same test case do not run. The framework already had that mechanism for skips: SKIP_TESTCASE aborts out of the assertion and marks the origin, so the catch around the test case can tell the framework's own abort apart from a user abort. The new abortFromRequireFlag does the same for a failed REQUIRE and the run continues with the next test case. The abort flag keeps its meaning, so the sites which cancel the run on it, including ReportErrorAndAbort and failing test hooks, are untouched. Retrying a test case which aborted on a REQUIRE stays opt-in through IUTF_RETRY_REQUIRES, now gated on the new flag as well. As CanRetry runs after the catch, the flag is cleared only once the retry decision is made, and before the next attempt or data generator index. The catch around the test case unregisters the background monitor after a failed REQUIRE. Without that the run would suspend into a reentry for a test case which has already failed. The REQUIRE itself must not do it: in a monitored background task it runs while the run is suspended, and only the monitor resumes it. There the Abort stops the background task, and on resuming the test case ends without calling its reentry function. Only the Abort of the REQUIRE, code -3, is cleared. A test case which caught that Abort can still be aborted by the user, by a stack overflow or by AbortOnValue, and those are reported as before. The IUTF_EXPECTED_FAILURE tag is no longer listed as a way to stop only the current test case. With the tag a failed REQUIRE does not abort, so the test case continues. The list names the failed REQUIRE instead. FailedTests gains a RequireFail/RequireFail_Verify pair. It reports an error when the behaviour regresses, because the neutralisation of the REQUIRE's failure is what keeps the test suite clean. ReentryTests gains RequireFailWithMonitor, which registers a monitor before its REQUIRE fails, and RequireFailInTask, whose REQUIRE fails in the monitored background task. Their reentry functions fail as well, so the verification reports an error if one is still called, and the run does not finish if it is never resumed. ErroredTests gains AbortOnValueAfterCaughtRequire. The flag setting goes back to c23cd35 (more work towards the first final release, 2012-12-21). 324d953 (Document that a failed REQUIRE cancels the whole test run, 2026-10-15) documented it as it stood.
basic.rst's second test-case rule now says the return has to be real-valued, which is narrower than "numeric" in Igor terms. Nothing in the test suite held that line: the filter argument in GetTestCaseList reads as an implementation detail, and widening it to VALTYPE:3 collects both the real and the complex case with no other discovery surprise, so such a change would pass review as a one-character edit. EasyTests gains a Function/C case and a sibling which asks the framework itself for the test cases of that window and requires the complex one to be absent. The sibling also insists on finding itself, so the check cannot pass on an empty list.
advanced.rst used to require a data-generator tag within four lines above the Function line. e208689 replaced that with the rule the code actually implements, the whole comment block back to the previous function, but nothing held the new wording. Passing -4 instead of -1 to ProcedureText is a one-character change which reads as harmless in review, and it would drop distant tags quietly: a test case whose generator tag went missing fails with an unrelated message about the data generator, not about the tag. GeneratorTests gains a case with fourteen comment lines between its tag and its Function line, checking that the generated value still arrived. functiontags.rst now also says where the block ends for the first function of a procedure window: with no preceding function, a tag in the file header belongs to it.
Both console listings were written by hand in fb8ef0c (rework documentation, 2018-09-04) and never regenerated, so they had drifted from what RunTest prints: - the suite line said "Unnamed" although the test suite is the procedure window, which the tour has just named test0, - the second listing repeated the "Entering test case" line, - the assertion text lacked the trailing period, - the location used a "failed in line 11, procedure \"test0\"" form the code does not produce; the current one names the test case function and puts the window and line in parentheses, - the two compared numbers were shown rounded, while GetNiceStringForNumber() prints them with UTF_DECIMAL_DIGITS_FP64 significant digits, - the end-of-run recap of the failed assertion was missing. The screenshot below the first listing still showed an Igor Pro 7.08 session with #include "unit-testing" and the old suite line, so it is replaced by one of the same run in Igor Pro 10.
The helper/ family grows one release at a time, last in 2dd1a2b (Add autorun scripts for IP8, 2018-09-17), so the automated-run instructions stop at an Igor version CI has been testing for years while the code already branches on the next one (igortest-basics.ipf:10-12). /I is still documented in Igor 9 as the way to force a new instance, so the IP9 pair is the IP8 pair with the release folder name from the installer's default location. They also pass /UNATTENDED, which Igor added in 9.00 to suppress interactions that block an unattended run, for example the About Autosave dialog shown at launch until the user opts out of it. The doc says so, as the older scripts lack the flag. The doc sentence records which versions have a script and what to do for any other, since a script whose executable is missing only prints a message rather than running anything.
Completes the family 231f3cb extended only as far as Igor 9. The path is not a guess: Igor 10's own Getting Started.ihf and Advanced Topics.ihf both name C:\Program Files\WaveMetrics\Igor Pro 10 Folder\IgorBinaries_x64\Igor64.exe, which is the executable on this machine. Only a 64-bit script, because this Igor 10 install has no IgorBinaries_Win32 folder while the Igor 9 one does. The doc sentence lists Igor 10 as 64-bit only without claiming to know why.
SKIP_TESTCASE unregistered the background monitor itself. Called in a monitored background task, it runs while the run is suspended, and only the monitor resumes the run. The run therefore never finished: no end hooks, no remaining test cases, and autorun did not quit. As for a failed REQUIRE, the catch around the test case now unregisters the monitor, and on resuming the test case ends without calling its reentry function. The catch also cleared every abort code once the skip flag was set, and only the catch reset that flag. A test case which caught the Abort of SKIP_TESTCASE therefore lost a later user abort, stack overflow or AbortOnValue. Only code -3 is cleared now, and the flag is reset after every call of a test case. ReentryTests gains SkipInTask and ErroredTests gains AbortOnValueAfterCaughtSkip, both like their REQUIRE counterparts. Bug introduced in d3cdc27 (Add SKIP_TESTCASE and SKIP_TESTCASE_EXPECT_FAILS assertions, 2024-06-06)
7faa0b3 to
cebef0e
Compare
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
|
@Garados007 I fixed some CoPilot reviews |
|
@Garados007 Merge? |
|
Yes. I had some issues checking the CI (will create some issues in a moment). Everything is fine and can be merged. |
|
The CI is almost green with the following exceptions:
|



A set of fixes from our local AI.