Skip to content

Flaky test: SegmentManagerCloseRaceTest.testSecondBoundedJoinReapsWorkerFinishingExitCleanups #118

Open

Description

@eugenels

Failed on Windows CI (Azure build 283763, job windows-other-2, run for questdb/questdb#7411, which does not change SegmentManager or this test):

java.lang.AssertionError: close() must not return while the worker is still alive in its exit cleanups -- the second bounded join has to hold close() until they finish
    at ...SegmentManagerCloseRaceTest.lambda$testSecondBoundedJoinReapsWorkerFinishingExitCleanups$0(SegmentManagerCloseRaceTest.java:777)

Cause: a timing assumption in the test. It calls close() right after start(), with the first join cut to 200 ms (SegmentManagerCloseRaceTest.java:723-767), and expects the worker to be idle so it leaves its loop at once. But the worker's first pass provisions a hot spare and syncs it (SegmentManager.java:1067). When that pass takes more than 200 ms, as a slow Windows disk can, the first join times out before the worker has left its loop, so close() takes the hand-off branch and returns with the worker still alive (SegmentManager.java:341). That branch is the intended behaviour for a worker still in a service pass, so only the test is wrong.

Reproduce on main (82b2e14): make the first pass slow by adding this just before manager.start() in the test:

manager.setBeforeInstallSyncHook(() -> {
    try {
        Thread.sleep(400L);
    } catch (InterruptedException ignore) {
    }
});

mvn -pl core test -Dtest='SegmentManagerCloseRaceTest#testSecondBoundedJoinReapsWorkerFinishingExitCleanups' then fails every time with the error above. Without the hook it passes.

Possible fix: let the first pass finish before close(), e.g. right after manager.start():

long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
while (ring.needsHotSpare() && System.nanoTime() < deadline) {
    Thread.sleep(1);
}
Assert.assertFalse("worker never provisioned the hot spare", ring.needsHotSpare());

With this, the test passes even with the 400 ms hook above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    QWPbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions