gitoxide: 12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1
[!NOTE] Except the title, this has not been updated to reflect that the performance test seems now to fail only rarely and never on CI, and that since #1871 the two
gix-pathspec::pathspec searchare passing. The automatically checked list of expected failures, and output oftest-fixtures-windowsCI runs on the main branch, have full current details.
Current behavior 😯
Running tests on Windows in a Git Bash environment (similar to the environment in which they run in Windows on CI, where bash is Git Bash), all tests are able to pass normally. However, this apparently relies on the use of generated archives.
When the tests are run with the environment variable GIX_TEST_IGNORE_ARCHIVES set to 1 rather than unset, 15 tests fail. The failing tests, as quoted from the end of the test run, are:
Summary [ 903.419s] 2363 tests run: 2348 passed (17 slow, 1 leaky), 15 failed, 9 skipped
FAIL [ 7.416s] gix-glob::glob pattern::matching::compare_baseline_with_ours
FAIL [ 3.324s] gix-pathspec::pathspec parse::baseline
FAIL [ 0.015s] gix-pathspec::pathspec parse::valid::glob_negations_are_always_literal
FAIL [ 0.018s] gix-pathspec::pathspec parse::valid::whitespace_in_pathspec
FAIL [ 1.749s] gix-pathspec::pathspec search::files
FAIL [ 3.855s] gix-pathspec::pathspec search::prefixes_are_always_case_sensitive
FAIL [ 181.270s] gix-ref-tests::refs packed::iter::performance
FAIL [ 4.803s] gix-submodule::submodule file::baseline::common_values_and_names_by_path
FAIL [ 30.106s] gix-submodule::submodule file::is_active_platform::pathspecs_matter_even_if_they_do_not_match
FAIL [ 35.376s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_are_considered_active_or_inactive
FAIL [ 22.003s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_override_pathspecs
FAIL [ 15.882s] gix-submodule::submodule file::is_active_platform::without_any_additional_settings_all_are_inactive_if_they_have_a_url
FAIL [ 9.165s] gix-submodule::submodule file::is_active_platform::without_submodule_in_index
FAIL [ 0.193s] gix::gix revision::spec::from_bytes::regex::find_youngest_matching_commit::regex_matches
FAIL [ 0.086s] gix::gix revision::spec::from_bytes::regex::with_known_revision::contained_string_matches_in_unanchored_regex_and_disambiguates_automatically
error: test run failed
The full output is available in this gist, showing a test run at c2753b8, which has the fixes in #1444.
Before those fixes, one other test had been reported as failing here. See https://github.com/Byron/gitoxide/issues/1358#issuecomment-2211890726 and the old gist if interested. The discussion in #1345 is still relevant, though it links to this even older gist.
As noted in comments in #1345, the failure in compare_baseline_with_ours seems particularly interesting, since it involves an unexpected effect of .gitignore pattern matching that is different on Windows.
Expected behavior 🤔
All tests should pass, even when suppressing the use of generated archives by setting GIX_TEST_IGNORE_ARCHIVES=1. Differences between Windows and other platforms should be accounted for when intentional and desirable, or fixed otherwise.
Git behavior
Not fully applicable, since this is about a failure of multiple gitoxide tests when run in a certain way.
However, some discrepancies–intended or unintended–between gitoxide and Git may turn out to be related to some of the failures. So this section may be expanded in the future, or perhaps new issues will be split out from this one.
Steps to reproduce 🕹
I ran the tests on Windows 10.0.19045 (x64) with developer mode enabled so that symlink creation is permitted even without UAC elevation, with git version 2.45.2.windows.1.
I used the current tip of the main branch, which at this time is c2753b8425c285c6b53f46eba9bc3584aa85eb01. When I opened this issue originally, that did not exist, but all experiments described here have been performed again, and the reported results have been updated. I used the latest stable Rust toolchain and cargo-nextest, though this does not seem to affect the results.
Local development environments and CI sometimes differ in relevant ways, so I also verified that the tests all pass when GIX_TEST_IGNORE_ARCHIVES is not set. This may be considered an optional step, but is beneficial because it checks that all needed dependencies are installed and working, and that failures really can be attributed to the effect of GIX_TEST_IGNORE_ARCHIVES. To run the tests that way, I run this in Git Bash:
cargo nextest run --all --no-fail-fast
Then I did a full clean:
gix clean -xde
Then, also in Git Bash, I ran this command, which produced the test output and failures described above:
GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --all --no-fail-fast
The reason the tests must be run in Git Bash is that there is a separate issue where many test failures occur when they are run from a typical PowerShell environment (#1359).
I also ran the tests, with and without GIX_TEST_IGNORE_ARCHIVES=1, on Ubuntu 22.04 LTS (x64) and macOS 14.5 (M1). As expected, all tests passed on those systems, confirming that the failures are Windows-specific.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 21 (21 by maintainers)
Commits related to this issue
- Revert "simplify .gitattributes" This reverts commit dd94f57db82a9bf5833e290b3300a747001bb1eb. One of the changes to `.gitattributes`, possibly that change, has caused the currently committed genera... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Revert "simplify .gitattributes" This reverts commit dd94f57db82a9bf5833e290b3300a747001bb1eb. That change to `.gitattributes`, though intended as a refactoring, caused the currently committed gener... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Revert "simplify .gitattributes" This reverts commit a75e97080df22c9fcc3fd79b9f0b03aeb8f5fbe1. That change to `.gitattributes`, though intended as a refactoring, caused the currently committed gener... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Add Windows CI job with `GIX_TEST_IGNORE_ARCHIVES=1` Some of these fail. We report the step as failing if any fail, which currently at least 14 and usually 15 are expected to (one of them is a perfor... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Add Windows CI job with `GIX_TEST_IGNORE_ARCHIVES=1` Some of these fail. We report the step as failing if any fail, which currently at least 14 and usually 15 are expected to (one of them is a perfor... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Use XML nextest output instead of ad-hoc parsing This eliminates the piping, tricky parsing, and special-casing to preserve colorization, and runs the tests and the XML parsing in the default `pwsh` ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Raise the expected failure count This should let the `test-fixtures-windows` job succeed. As suspected (but not, as of now, explained), the five additional failures when running the tests in `bash` ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Add Windows CI job with `GIX_TEST_IGNORE_ARCHIVES=1` Some of these fail. We report the step as failing if any fail, which currently at least 14 and usually 15 are expected to (one of them is a perfor... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Use XML nextest output instead of ad-hoc parsing This eliminates the piping, tricky parsing, and special-casing to preserve colorization, and runs the tests and the XML parsing in the default `pwsh` ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Raise the expected failure count This should let the `test-fixtures-windows` job succeed. As suspected (but not, as of now, explained), the five additional failures when running the tests in `bash` ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Check the specific Windows `GIX_TEST_IGNORE_ARCHIVES=1` failures This modifies the `test-fixtures-windows` job that tests on Windows with `GIX_TEST_IGNORE_ARCHIVES=1` so that, instead of checking tha... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Check the specific Windows `GIX_TEST_IGNORE_ARCHIVES=1` failures This modifies the `test-fixtures-windows` job that tests on Windows with `GIX_TEST_IGNORE_ARCHIVES=1` so that, instead of checking tha... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Exclude performance test(s) from expected failures This omits tests containing `performance` (and not as part of a larger "word", not even with `_`) from being expected to fail on CI with `GIX_TEST_I... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Use a file with `test-fixtures-windows` expected failures Instead of retrieving them from #1358. (See discussion in #1663.) — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- fix: Run test fixture scripts on Windows with Git Bash Rather than hard-coding `bash` on all systems as the fallback interpreter when a fixture script cannot be run directly, this falls back in an op... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- fix: Run test fixture scripts on Windows with Git Bash Rather than hard-coding `bash` on all systems as the fallback interpreter when a fixture script cannot be run directly, this falls back in an op... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Make `test-fixtures-windows` required for PR auto-merge Since #1663, the `test-fixtures-windows` CI job checks actual failures against a list of specific tests that are known to fail on Windows when ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Make `test-fixtures-windows` required for PR auto-merge Since #1663, the `test-fixtures-windows` CI job checks actual failures against a list of specific tests that are known to fail on Windows when ... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Fix Windows-related `-c` options in `match_baseline_files` The `match_baseline_files.sh` fixture script adds entries to the index using `git update-index --add --cacheinfo`. These entries don't exist... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
- Fix Windows-related `-c` options in `match_baseline_files` The `match_baseline_files.sh` fixture script adds entries to the index using `git update-index --add --cacheinfo`. These entries don't exist... — committed to EliahKagan/gitoxide by EliahKagan 2 years ago
That seems to be the case–I’ve confirmed it does occasionally fail locally (though still never on CI).
I suspect forthcoming changes might make the performance test fail more or less often, and maybe that will shed light on something.
By the way, this new gist shows a recent full local run and the 12 failures that consistently occur. I’ll update this PR description to link to that (or a newer run, if I have one by then) once I decide whether to report the performance test failure in a separate issue.
That seems reasonable. I would think that running it repeatedly locally will reproduce it and it’s just a matter of probability. Since it’s a performance test, I think the failure might be related to IO and some FS lock preventing it from functioning.
The performance test seems different in character from the other failures, because it never fails on CI and often does not fail when run locally. This is not to say that it ought to be ignored, but I am inclined to start thinking of it as a separate issue, considering that all the other known current and past failures due to enabling fixtures on Windows are deterministic.
If I don’t keep seeing it locally, I’ll remove it. If I manage to produce it on at least two machines, I’ll make it into its own issue.
Also, the Git installation on my main Windows 10 development machine is affected by #1849 (https://github.com/git-for-windows/git/issues/5436).
For now, this issue is in a half-updated state: its title is updated and I’ve put a prominent note at the top, but I haven’t made a new gist with current output on main or shown a new test results summary.
That’s great news!
Now I wonder if it would be a viable first step to split out the
multiplecase into its own script only the tests that actually need it would fail for the known reasons. This should already be an improvement. From there one could see if using fake submodules will do the trick here, as the simplest-possible fix.Yes, you are right, it’s actually only one per workflow (file), so that’s a non-issue.
A new test from #1618,
gix-merge::merge tree::run_baseline, also fails on Windows withGIX_TEST_IGNORE_ARCHIVES=1. But I have not added it to the list here, because #1652 includes a fix for it in the changes it proposes.Here are updated runs. The failing tests are the same. This is just to clarify that, since I’m working on some changes in
gix-testtools(in #1591 and elsewhere) and ran this on the current tip of main to compare against.The reason for doing two runs within a short time of each other and to report both in the gist is to check that the number of tests reported as leaky varies across runs rather than having changed due to recent code changes (over time or across branches).
2024-10-22 update: The situation is unchanged at db5c9cf (gitoxide v0.38.0).
As expected per https://github.com/Byron/gitoxide/issues/1358#issuecomment-2209859265, the fixes for #1443 in #1444 have also made this test, originally reported as failing here, pass:
I have edited the description here accordingly, updating the title and summary list of failing tests, linking to a new gist with the current output of a full run, and also revising for clarity and adding some missing details.
While a number of changes have been made to the test suite, this issue holds up without modification: the same tests fail on Windows as before (and as shown above) when
GIT_TEST_IGNORE_ARCHIVES=1, at c1d7c023d595eb04891b65295f001d85c9ba8074.The
gix-ref-tests::refs packed::iter::performancetest may depend on the speed of the test machine, but that was the case before. On my machine, it continues to take about the same amount of time (and fail for taking too long).