Repository navigation
System tests of main are broken #168
Description
I see the tests are broken on main. I don't if this is the result of merging this PR
@clrudolphi any idea? I havent found time to check the root cause of the test failures
The connector tests are dynamically building a few of the Reqnroll example sample projects from the ReqnrollExploratoryTestProjects repo, such as the SpecFlowCompatibilityProject. Those projects have hard-wired references to Reqnroll 3.3.4-local:
<PackageReference Include="Reqnroll.SpecFlowCompatibility" Version="3.3.4-local" />
Version 3.3.4 final was released two days ago and so a reference to 3.3.4-local isn't valid.
I am not sure whether those should be updated to the latest released build or to a local build of version 4.0.0 (the next build coming).
Originally posted by @clrudolphi in #167 (comment)
Activity
[-]System tests of main are brokrn[/-][+]System tests of main are broken[/+]304NotModified commentedon Mar 25, 2026
The previous commit of main also fails indeed
https://github.com/reqnroll/Reqnroll.VisualStudio/actions/runs/23549959814
304NotModified commentedon Mar 26, 2026
@clrudolphi could you please check reqnroll/Reqnroll.ExploratoryTestProjects#2?
gasparnagy commentedon May 19, 2026
Sorry for causing a confusion.
In general the idea of the https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects repo was to have a set of projects that can be used for local exploratory (manual) tests. I wanted to keep the projects as real-looking as possible (to be able to detect real integration problems), therefore I did not wanted to use a global
Directory.propsbased version setting.As the projects are used mostly to test the local versions, the projects were generally set to the current local versions and I usually updated that (with the update-versions.ps1 it is easy) and pushed. Sometimes you need to update the versions for local testing anyway.
In order to improve the VS discovery coverage, I introduced a new set of "smoke tests" that clones and goes through the projects in the ExploratoryTestProjects repo and tries to load them to VS. We cannot assume a correct version in the cloned projects, so the test should update the versions to the expected one by calling the ps1 script. This should be done by these lines: https://github.com/reqnroll/Reqnroll.VisualStudio/blob/main/Tests/Connector/Reqnroll.VisualStudio.ReqnrollConnector.Tests/SampleProjectTestBase.cs#L206-L210
So "normally" this should work, even with an old and "-local" versions in the ExploratoryTestProjects repo. And it was at least at the time of the last release.
Something has probably changed, or I have a bug in the connector tests that causes this error now. I will look at it and come back with the details.
304NotModified commentedon May 19, 2026
I think this isn't a problem. But I prefer to have it working on the CI, then it's easier to analyze when it isn't working locally:)
gasparnagy commentedon May 20, 2026
Just for clarification: you mean that we should have the "released" versions in the pushed repo, and use the "-local" locally, right?
304NotModified commentedon May 20, 2026
Yes indeed :)
gasparnagy commentedon May 21, 2026
Analyzed the problem and the build works as expected, it updates the versions of the exploratory test projects to the current released version before running the tests. The build failure comes from the Reqnroll issue reqnroll/Reqnroll#1068 - the missing Reqnroll.SpecFlowCompatibility 3.3.4 package.
Since the plan is to discontinue the support for Reqnroll.SpecFlowCompatibility, I think the most paratactical immediate solution is to exclude those tests from the connector smoke test suite.
In addition to this, I will change the CI build for the VisualStudio project to run weekly as well in addition to running per commit. There is not that much activity on that repo, so we might not realize a problem, like this on time.
I will react on the note of @304NotModified about the pushed version in a separate comment, because that is a slightly different topic.