gitoxide: 12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1

[!NOTE] Except the title, this has not been updated to reflect that the performance test seems now to fail only rarely and never on CI, and that since #1871 the two gix-pathspec::pathspec search are passing. The automatically checked list of expected failures, and output of test-fixtures-windows CI runs on the main branch, have full current details.

Current behavior 😯

Running tests on Windows in a Git Bash environment (similar to the environment in which they run in Windows on CI, where bash is Git Bash), all tests are able to pass normally. However, this apparently relies on the use of generated archives.

When the tests are run with the environment variable GIX_TEST_IGNORE_ARCHIVES set to 1 rather than unset, 15 tests fail. The failing tests, as quoted from the end of the test run, are:

     Summary [ 903.419s] 2363 tests run: 2348 passed (17 slow, 1 leaky), 15 failed, 9 skipped
        FAIL [   7.416s] gix-glob::glob pattern::matching::compare_baseline_with_ours
        FAIL [   3.324s] gix-pathspec::pathspec parse::baseline
        FAIL [   0.015s] gix-pathspec::pathspec parse::valid::glob_negations_are_always_literal
        FAIL [   0.018s] gix-pathspec::pathspec parse::valid::whitespace_in_pathspec
        FAIL [   1.749s] gix-pathspec::pathspec search::files
        FAIL [   3.855s] gix-pathspec::pathspec search::prefixes_are_always_case_sensitive
        FAIL [ 181.270s] gix-ref-tests::refs packed::iter::performance
        FAIL [   4.803s] gix-submodule::submodule file::baseline::common_values_and_names_by_path
        FAIL [  30.106s] gix-submodule::submodule file::is_active_platform::pathspecs_matter_even_if_they_do_not_match
        FAIL [  35.376s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_are_considered_active_or_inactive
        FAIL [  22.003s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_override_pathspecs
        FAIL [  15.882s] gix-submodule::submodule file::is_active_platform::without_any_additional_settings_all_are_inactive_if_they_have_a_url
        FAIL [   9.165s] gix-submodule::submodule file::is_active_platform::without_submodule_in_index
        FAIL [   0.193s] gix::gix revision::spec::from_bytes::regex::find_youngest_matching_commit::regex_matches
        FAIL [   0.086s] gix::gix revision::spec::from_bytes::regex::with_known_revision::contained_string_matches_in_unanchored_regex_and_disambiguates_automatically
error: test run failed

The full output is available in this gist, showing a test run at c2753b8, which has the fixes in #1444.

Before those fixes, one other test had been reported as failing here. See https://github.com/Byron/gitoxide/issues/1358#issuecomment-2211890726 and the old gist if interested. The discussion in #1345 is still relevant, though it links to this even older gist.

As noted in comments in #1345, the failure in compare_baseline_with_ours seems particularly interesting, since it involves an unexpected effect of .gitignore pattern matching that is different on Windows.

Expected behavior 🤔

All tests should pass, even when suppressing the use of generated archives by setting GIX_TEST_IGNORE_ARCHIVES=1. Differences between Windows and other platforms should be accounted for when intentional and desirable, or fixed otherwise.

Git behavior

Not fully applicable, since this is about a failure of multiple gitoxide tests when run in a certain way.

However, some discrepancies–intended or unintended–between gitoxide and Git may turn out to be related to some of the failures. So this section may be expanded in the future, or perhaps new issues will be split out from this one.

Steps to reproduce 🕹

I ran the tests on Windows 10.0.19045 (x64) with developer mode enabled so that symlink creation is permitted even without UAC elevation, with git version 2.45.2.windows.1.

I used the current tip of the main branch, which at this time is c2753b8425c285c6b53f46eba9bc3584aa85eb01. When I opened this issue originally, that did not exist, but all experiments described here have been performed again, and the reported results have been updated. I used the latest stable Rust toolchain and cargo-nextest, though this does not seem to affect the results.

Local development environments and CI sometimes differ in relevant ways, so I also verified that the tests all pass when GIX_TEST_IGNORE_ARCHIVES is not set. This may be considered an optional step, but is beneficial because it checks that all needed dependencies are installed and working, and that failures really can be attributed to the effect of GIX_TEST_IGNORE_ARCHIVES. To run the tests that way, I run this in Git Bash:

cargo nextest run --all --no-fail-fast

Then I did a full clean:

gix clean -xde

Then, also in Git Bash, I ran this command, which produced the test output and failures described above:

GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --all --no-fail-fast

The reason the tests must be run in Git Bash is that there is a separate issue where many test failures occur when they are run from a typical PowerShell environment (#1359).

I also ran the tests, with and without GIX_TEST_IGNORE_ARCHIVES=1, on Ubuntu 22.04 LTS (x64) and macOS 14.5 (M1). As expected, all tests passed on those systems, confirming that the failures are Windows-specific.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 21 (21 by maintainers)

Commits related to this issue

Most upvoted comments

That seems reasonable. I would think that running it repeatedly locally will reproduce it and it’s just a matter of probability.

That seems to be the case–I’ve confirmed it does occasionally fail locally (though still never on CI).

I suspect forthcoming changes might make the performance test fail more or less often, and maybe that will shed light on something.

By the way, this new gist shows a recent full local run and the 12 failures that consistently occur. I’ll update this PR description to link to that (or a newer run, if I have one by then) once I decide whether to report the performance test failure in a separate issue.

If I don’t keep seeing it locally, I’ll remove it. If I manage to produce it on at least two machines, I’ll make it into its own issue.

That seems reasonable. I would think that running it repeatedly locally will reproduce it and it’s just a matter of probability. Since it’s a performance test, I think the failure might be related to IO and some FS lock preventing it from functioning.

The performance test seems different in character from the other failures, because it never fails on CI and often does not fail when run locally. This is not to say that it ought to be ignored, but I am inclined to start thinking of it as a separate issue, considering that all the other known current and past failures due to enabling fixtures on Windows are deterministic.

If I don’t keep seeing it locally, I’ll remove it. If I manage to produce it on at least two machines, I’ll make it into its own issue.

Also, the Git installation on my main Windows 10 development machine is affected by #1849 (https://github.com/git-for-windows/git/issues/5436).

For now, this issue is in a half-updated state: its title is updated and I’ve put a prominent note at the top, but I haven’t made a new gist with current output on main or shown a new test results summary.

That’s great news!

Now I wonder if it would be a viable first step to split out the multiple case into its own script only the tests that actually need it would fail for the known reasons. This should already be an improvement. From there one could see if using fake submodules will do the trick here, as the simplest-possible fix.

If your experience is the same, then just keeping all the jobs in the ci.yml workflow, which should be done anyway due to their close relatedness and having the same event trigger, should be sufficient to avoid getting extra emails. In that case, any way of organizing the jobs and deciding/expressing their dependences should be fine.

Yes, you are right, it’s actually only one per workflow (file), so that’s a non-issue.

A new test from #1618, gix-merge::merge tree::run_baseline, also fails on Windows with GIX_TEST_IGNORE_ARCHIVES=1. But I have not added it to the list here, because #1652 includes a fix for it in the changes it proposes.

Here are updated runs. The failing tests are the same. This is just to clarify that, since I’m working on some changes in gix-testtools (in #1591 and elsewhere) and ran this on the current tip of main to compare against.

The reason for doing two runs within a short time of each other and to report both in the gist is to check that the number of tests reported as leaky varies across runs rather than having changed due to recent code changes (over time or across branches).


2024-10-22 update: The situation is unchanged at db5c9cf (gitoxide v0.38.0).

As expected per https://github.com/Byron/gitoxide/issues/1358#issuecomment-2209859265, the fixes for #1443 in #1444 have also made this test, originally reported as failing here, pass:

gix::gix object::tree::diff::track_rewrites::copies_in_entire_tree_by_similarity_with_limit

I have edited the description here accordingly, updating the title and summary list of failing tests, linking to a new gist with the current output of a full run, and also revising for clarity and adding some missing details.

While a number of changes have been made to the test suite, this issue holds up without modification: the same tests fail on Windows as before (and as shown above) when GIT_TEST_IGNORE_ARCHIVES=1, at c1d7c023d595eb04891b65295f001d85c9ba8074.

The gix-ref-tests::refs packed::iter::performance test may depend on the speed of the test machine, but that was the case before. On my machine, it continues to take about the same amount of time (and fail for taking too long).