cosmic-ray: HTTP Distributor Tutorial is Broken on Windows

Thanks for doing that. It really helps me keep track of what’s going on.

I’m actually ready to close this issue. I’m nearly certain that #516 addresses your initial problems. It’s not an issue on windows, just an issue with my logic in the HTTP distributor. So I’ll close this issue and sort things out over there.

Could you bundle these documentation issues up into another issue? That’ll make it easier for me to keep track of them. Or, if you’re feeling energetic, you could update the docs yourself; I’m always happy to get these kinds of contributions because they almost always deal with my particular blind spots.

Are there plans to make them behave similarly to the one in Python’s coverage tool?

Not right now. Something like this would presumably require syntactic analysis of the code, so it wouldn’t be trivial (though I don’t think it would be terribly complicated either). There may even be an existing library we could leverage that helps us process the pragmas.

If this is something you’d like, it would be helpful to create another issue for it.

1. In case of interrupts and crashes, is there a possibility that CR won’t be able to undo some of its mutations? Is it good practice to fully check-in projects before doing any mutation testing?

CR tries hard to avoid this possibility, but it might still be possible. Tests are run in a subprocess, so even if they die horribly, the CR worker process should be able to undo the mutations. Generally speaking, though, I’d recommend committing everything prior to testing; this will be important, for example, if you’re using the cr-http-workers tool that clones your repo as part of its operations.

1. I assume you should also run init when you make changes to your config file. I didn’t see this in the docs though, so perhaps it should be added.

This isn’t always strictly necessary. For example, changing your distributor won’t require a re-initialization. Changing your test-command or excluded-modules generally would necessitate a re-init since this would effect which tests are run (thus potentially invalidating existing results).

Is there an option for cr-report to only print the progress summary (the last few lines at the end)?

Not currently, though that should be straightforward to implement if you’re so inclined.

On a *nix system you could do something like:

cr-report session | tail -n 3

1. Is this the correct way to exclude files?

It looks like this feature doesn’t work right now. I’ve create #517 as a reminder to reimplement this as a filter.

1. Is this correct, or is it going to disable other mutation Operators as well? I read in the Excluding Filters section that some RegEx is sometimes needed.

Yes, that looks correct. Regular expressions are an option, not a requirement. In your case of excluding a single operator, what you’ve done is fine.

1. I’m not sure I understand how # pragma: no mutate works in CR.

Our pragma filter only works on lines of code, not code blocks. It’s quite a crude filter, really, but sufficient for many purposes.

Does running init overwrite the existing Session DB (I assume it does), or do I have to delete it as well before I rerun init?

init completely rewrites the session file. You don’t need to delete it.

I assume that CR is robust enough to handle aborts (e.g. Ctrl+C) from the user as well as most crashes and is able to reliably pick up where it left off thanks to the Session DB. Is this correct?

CR is designed to be robust against interrupts and crashes. To a large degree, we rely on sqlite to handle that. When you run exec, it looks for all of the WorkItems with no results, and these tell CR what works needs to be done. Furthermore, we only add those results after a mutation/test is complete. So if CR dies for some reason, any running tests simply die, no results are written to the session, and processing picks back up with those incomplete WorkItems next time.

The bad news is that on the second and subsequent tries, mutation testing constantly fails with the error in the OP.

Ok, then there’s something more mysterious going on. I’ll have to try it out on windows.

We may have to add some sort of check that fetchers.keys() is not empty and, if it is, add a small sleep or something. This feels kludgy, but maybe it’s necessary for some valid reason I’m not aware of.

@abingham Also, small typo in the docs:

Note that your worker must be running in the same directory as you would normally run the tests from.

I’ve updated the docs to add a note that effect.

@abingham I see. I’ve never done this before so I wouldn’t know where this standard would be documented (I assume the Python docs).

I’ll let you know the results as soon as I can. 🙂

I’m happy to help support you as much as I can, though my time is a bit limited right now. CR does work well when all of the stars align; as you’re finding, though, I struggle to keep the documentation up to date, and I don’t do any testing on Windows.

Actually, I think I see what’s going on. The call to asyncio.wait mentioned near the end of the stack trace is to blame. Its documentation clearly states that the first argument fs must not be empty, and indeed the ValueError here is complaining that it is. So in src/cosmic_ray/distribution/http.py around line 75 we seem to have a bit of a race condition where urls has no elements but fetchers.keys() is somehow empty as well.

I think the error is on lines 68-69 where we remove a task from fetchers before appending a URL to urls, creating a window where both urls and fetchers are empty. I thought asyncio would prevent this, but perhaps not.

@FloatingSunfish If you’re feeling adventurous, could you swap lines 68 and 69 in src/cosmic_ray/distribution/http.py, install CR from source, and try again?