conan: [bug] Packages being marked as dirty on upload, when they are not dirty
I have a weird bug here that I am fairly sure is related to the recent changes to enable parallel uploads.
I confirmed that rolling back to conan version 1.20.5, before this feature was implemented, fixed my issues completely.
In the steps to reproduce below, what is happening is that on the very last upload command, my packages are being marked as dirty and conan is deleting them, meaning they dont get uploaded. I believe this is related to the parallel changes because I am seeing logs come out of order, suggesting a threading issue. It seems like what is happening is that its running the follow line of code
Before running this line of code:
This is causing the package to be marked as dirty here:
My suspicion is this yield call. I think perhaps the yield call is causing some weirdness and perhaps its going and executing another upload command before the first one is finished? I am unsure completely how the yield call works under the hood in python but it seems suspect to me without correct mutex protection
Environment Details (include every applicable attribute)
- Operating System+version: Linux, Ubuntu 16
- Conan version: 1.22.0-dev
- Python version: 3.6
Steps to reproduce (Include if Applicable)
Create a package, this also does some intermediate rebuilds in the middle of the dependency graph in my case
conan create . test/test
Compress one of those intermediate packages but dont upload yet
conan upload PkgA/1.0.0@test/test -c --check --skip-upload
Upload that package after checks have been performed and its been compressed
conan upload PkgA/1.0.0@test/test -c --check
Logs (Executed commands with output) (Include/Attach if Applicable)
[2020-01-08T22:13:01.902Z] + conan upload PkgA/1.0.0@test/test:e52c9eb724d0973fa676c1bc641288614396a401 -r R1 -c
[2020-01-08T22:13:02.209Z] Uploading to remote 'R1':
[2020-01-08T22:13:02.209Z]
Uploading PkgA/1.0.0@test/test to remote 'R1'
[2020-01-08T22:13:04.354Z] Recipe is up to date, upload skipped
[2020-01-08T22:13:04.354Z]
Uploading package 1/1: e52c9eb724d0973fa676c1bc641288614396a401 to 'R1'
[2020-01-08T22:13:04.354Z] Compressing package...
[2020-01-08T22:13:37.289Z] WARN: conan upload PkgA/1.0.0@test/test:e52c9eb724d0973fa676c1bc641288614396a401: Removing conan_package.tgz, marked as dirty
Also, this may be related to https://github.com/conan-io/conan/issues/6141 but I am unsure. I will keep digging and upload anything else I find out. Let me know if you need any more information to help debug
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 25 (11 by maintainers)
Finally identified the reason of the WARN about conan_package.tgz being dirty in such “random” order. The thing is that the WARN messages are routed to the
stream_errstream, which would besys.stderr, and then the calling process can get it in different not strictly sequential order.So it seems a non-issue. I will create a new issue to discuss if we could have a fully deterministic output in Conan 2.0 (this might be breaking if someone is using
sys.stderr)Thanks again very much for all the feedback. Lets release 1.22.1 asap.
The build is not being executed concurrently. Each upload is done in serial and not parallel. At least, we aren’t intentionally doing things in parallel unless there is a bug here somewhere. I checked the create.json and it doesn’t look like there are two references there for the same package.
Perhaps a question that may answer this whole problem is, does the
conan uploadcommand compress things before exiting? Or does the command return and do things in the background somehow? I have always assumed and from my experience it will compress things before returning but we are running many upload commands in a loop, one after the other.I am trying to above steps now to check for the dirty file. The build that reliably reproduces this takes a while so I will get back to you later today 😃