azure-pipelines-tasks: "Permission denied" error in conda task
Environment
-
Server - VSTS or TFS on-premises?
VSTS
-
Agent - Hosted or Private: Hosted ‘Ubuntu 16.04’
Issue Description
- task: CondaEnvironment@1
inputs:
createCustomEnvironment: true
environmentName: py3.5
packageSpecs: 'python=3.5'
This task produces permission denied errors which lead to task failure.
Task logs
2018-09-12T16:22:46.9229436Z ##[section]Starting: CondaEnvironment
2018-09-12T16:22:46.9368940Z ==============================================================================
2018-09-12T16:22:46.9388229Z Task : Conda Environment
2018-09-12T16:22:46.9406025Z Description : Create and activate a Conda environment.
2018-09-12T16:22:46.9424863Z Version : 1.138.0
2018-09-12T16:22:46.9443213Z Author : Microsoft Corporation
2018-09-12T16:22:46.9462022Z Help : [More information](https://go.microsoft.com/fwlink/?linkid=873466)
2018-09-12T16:22:46.9480302Z ==============================================================================
2018-09-12T16:22:50.0126863Z [command]/usr/bin/conda update --name base conda --yes
2018-09-12T16:23:14.9062048Z Solving environment: ...working... done
2018-09-12T16:23:20.6343316Z
2018-09-12T16:23:20.7776911Z conda-4.5.11 | 1.0 MB | | 0%
2018-09-12T16:23:20.9581679Z conda-4.5.11 | 1.0 MB | ######## | 80%
2018-09-12T16:23:21.0118534Z conda-4.5.11 | 1.0 MB | #########6 | 97%
2018-09-12T16:23:21.0146714Z conda-4.5.11 | 1.0 MB | ########## | 100%
2018-09-12T16:23:21.0155199Z
2018-09-12T16:23:21.1333663Z openssl-1.0.2p | 3.5 MB | | 0%
2018-09-12T16:23:21.8155494Z openssl-1.0.2p | 3.5 MB | #######5 | 76%
2018-09-12T16:23:21.9711961Z openssl-1.0.2p | 3.5 MB | #########4 | 94%
2018-09-12T16:23:21.9736658Z openssl-1.0.2p | 3.5 MB | ########## | 100%
2018-09-12T16:23:21.9746419Z
2018-09-12T16:23:22.0127922Z certifi-2018.8.24 | 140 KB | | 0%
2018-09-12T16:23:22.0151146Z certifi-2018.8.24 | 140 KB | ########## | 100%
2018-09-12T16:23:22.0169372Z ## Package Plan ##
2018-09-12T16:23:22.0179499Z
2018-09-12T16:23:22.0198916Z environment location: /usr/share/miniconda
2018-09-12T16:23:22.0208202Z
2018-09-12T16:23:22.0226396Z added / updated specs:
2018-09-12T16:23:22.0244424Z - conda
2018-09-12T16:23:22.0254336Z
2018-09-12T16:23:22.0263495Z
2018-09-12T16:23:22.0282046Z The following packages will be downloaded:
2018-09-12T16:23:22.0291571Z
2018-09-12T16:23:22.0309753Z package | build
2018-09-12T16:23:22.0327780Z ---------------------------|-----------------
2018-09-12T16:23:22.0346469Z conda-4.5.11 | py36_0 1.0 MB
2018-09-12T16:23:22.0364764Z openssl-1.0.2p | h14c3975_0 3.5 MB
2018-09-12T16:23:22.0384794Z certifi-2018.8.24 | py36_1 140 KB
2018-09-12T16:23:22.0403797Z ------------------------------------------------------------
2018-09-12T16:23:22.0422713Z Total: 4.6 MB
2018-09-12T16:23:22.0431658Z
2018-09-12T16:23:22.0449678Z The following packages will be UPDATED:
2018-09-12T16:23:22.0459692Z
2018-09-12T16:23:22.0478395Z certifi: 2018.4.16-py36_0 --> 2018.8.24-py36_1
2018-09-12T16:23:22.0497467Z conda: 4.5.4-py36_0 --> 4.5.11-py36_0
2018-09-12T16:23:22.0516599Z openssl: 1.0.2o-h20670df_0 --> 1.0.2p-h14c3975_0
2018-09-12T16:23:22.0525578Z
2018-09-12T16:23:22.0534968Z
2018-09-12T16:23:22.0553223Z Downloading and Extracting Packages
2018-09-12T16:23:22.1194848Z Preparing transaction: ...working... done
2018-09-12T16:23:22.5713236Z Verifying transaction: ...working... done
2018-09-12T16:23:22.5736956Z Executing transaction: ...working... failed
2018-09-12T16:23:22.5757156Z Rolling back transaction: ...working... done
2018-09-12T16:23:22.5768227Z
2018-09-12T16:23:22.5788534Z ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
2018-09-12T16:23:22.5809176Z PermissionError(13, 'Permission denied')
2018-09-12T16:23:22.5829065Z Attempting to roll back.
2018-09-12T16:23:22.5838159Z
2018-09-12T16:23:22.5846988Z
2018-09-12T16:23:22.5867327Z PermissionError(13, 'Permission denied')
2018-09-12T16:23:22.5877280Z
2018-09-12T16:23:22.5886944Z
2018-09-12T16:23:22.8545233Z Prepending PATH environment variable with directory: /usr/bin
2018-09-12T16:23:22.8756428Z [command]/usr/bin/conda create --quiet --prefix /usr/envs/py3.5 --mkdir --yes python=3.5
2018-09-12T16:23:24.1875016Z Solving environment: ...working... done
2018-09-12T16:23:32.6251193Z
2018-09-12T16:23:32.6273323Z ## Package Plan ##
2018-09-12T16:23:32.6282599Z
2018-09-12T16:23:32.6301488Z environment location: /usr/envs/py3.5
2018-09-12T16:23:32.6311118Z
2018-09-12T16:23:32.6332182Z added / updated specs:
2018-09-12T16:23:32.6354775Z - python=3.5
2018-09-12T16:23:32.6364968Z
2018-09-12T16:23:32.6378026Z
2018-09-12T16:23:32.6396858Z The following packages will be downloaded:
2018-09-12T16:23:32.6406238Z
2018-09-12T16:23:32.6426046Z package | build
2018-09-12T16:23:32.6445265Z ---------------------------|-----------------
2018-09-12T16:23:32.6464958Z libstdcxx-ng-8.2.0 | hdf63c60_1 2.9 MB
2018-09-12T16:23:32.6483796Z wheel-0.31.1 | py35_0 63 KB
2018-09-12T16:23:32.6504098Z readline-7.0 | h7b6447c_5 392 KB
2018-09-12T16:23:32.6523845Z tk-8.6.8 | hbc83047_0 3.1 MB
2018-09-12T16:23:32.6545870Z sqlite-3.24.0 | h84994c4_0 1.8 MB
2018-09-12T16:23:32.6564638Z certifi-2018.8.24 | py35_1 139 KB
2018-09-12T16:23:32.6583576Z python-3.5.6 | hc3d631a_0 28.3 MB
2018-09-12T16:23:32.6602030Z setuptools-40.2.0 | py35_0 571 KB
2018-09-12T16:23:32.6622815Z pip-10.0.1 | py35_0 1.8 MB
2018-09-12T16:23:32.6642082Z libgcc-ng-8.2.0 | hdf63c60_1 7.6 MB
2018-09-12T16:23:32.6667305Z ------------------------------------------------------------
2018-09-12T16:23:32.6685232Z Total: 46.7 MB
2018-09-12T16:23:32.6694585Z
2018-09-12T16:23:32.6712739Z The following NEW packages will be INSTALLED:
2018-09-12T16:23:32.6721945Z
2018-09-12T16:23:32.6740633Z ca-certificates: 2018.03.07-0
2018-09-12T16:23:32.6759605Z certifi: 2018.8.24-py35_1
2018-09-12T16:23:32.6778112Z libedit: 3.1.20170329-h6b74fdf_2
2018-09-12T16:23:32.6795872Z libffi: 3.2.1-hd88cf55_4
2018-09-12T16:23:32.6814772Z libgcc-ng: 8.2.0-hdf63c60_1
2018-09-12T16:23:32.6834873Z libstdcxx-ng: 8.2.0-hdf63c60_1
2018-09-12T16:23:32.6853380Z ncurses: 6.1-hf484d3e_0
2018-09-12T16:23:32.6872112Z openssl: 1.0.2p-h14c3975_0
2018-09-12T16:23:32.6891673Z pip: 10.0.1-py35_0
2018-09-12T16:23:32.6910287Z python: 3.5.6-hc3d631a_0
2018-09-12T16:23:32.6929504Z readline: 7.0-h7b6447c_5
2018-09-12T16:23:32.6949241Z setuptools: 40.2.0-py35_0
2018-09-12T16:23:32.6969051Z sqlite: 3.24.0-h84994c4_0
2018-09-12T16:23:32.6989538Z tk: 8.6.8-hbc83047_0
2018-09-12T16:23:32.7008705Z wheel: 0.31.1-py35_0
2018-09-12T16:23:32.7027794Z xz: 5.2.4-h14c3975_4
2018-09-12T16:23:32.7047048Z zlib: 1.2.11-ha838bed_2
2018-09-12T16:23:32.7056267Z
2018-09-12T16:23:32.7073845Z Preparing transaction: ...working... failed
2018-09-12T16:23:32.7083022Z
2018-09-12T16:23:32.7101042Z CondaError: Unable to create prefix directory '/usr/envs/py3.5'.
2018-09-12T16:23:32.7120320Z Check that you have sufficient permissions.
2018-09-12T16:23:32.7129576Z
2018-09-12T16:23:32.8411104Z ##[error]Failed to create Conda environment /usr/envs/py3.5. Error: /usr/bin/conda failed with return code: 1
2018-09-12T16:23:32.9312236Z ##[section]Finishing: CondaEnvironment
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 55 (26 by maintainers)
Commits related to this issue
- use sudo for installs: https://github.com/Microsoft/azure-pipelines-tasks/issues/8299 — committed to eqcorrscan/EQcorrscan by calum-chamberlain 6 years ago
I think you are exactly right. Now that we’ve shipped YAML our focus is on making the dev experience better. I think we have a few things coming you’ll be happy about 😃
I usually just make my YAML changes in a branch and run the builds off of that. I don’t think you should need a separate pipeline.
@brcrista @stas00 we’ve got SSH-into-hosted-agents on the long range radar. Thanks for the +1 on the feature.
Workaround I found from someone else’s azure-pipelines.yml:
I’m also thinking that other than giving access to the shell to debug/solve pipeline problems what would be useful is to be able to edit a single task in place, and rerun just it. Just pop an in browser editor and let me edit the commands. Then once it’s good, let me export it to add to the real yaml file. Otherwise it’s a pretty slow debugging since every time I make a change in the pipeline I have to wait for all previous stages to run first before the problematic one gets run. That also mean that each sub-stage should be re-run-able, as it’s possible that in order to test the current problematic stage, I also need to re-run one stage above.
Also, wrt to modifying the pipeline, there must be a way to isolate the experimentation with the pipeline code from the main pipeline, so while it’s being debugged it doesn’t affect the live PRs/commits. Since they get reported as failed/problematic, whereas the problem is in the pipeline script. I guess we really need 2 pipelines - one “production” and another “develop”, so the “develop” pipeline won’t work like the normal pipeline does. It will not get automatically activated by any event, but only run manually. Once it works well, it can overwrite the production one. I realize that I can probably make that happen in the current setup by creating a new pipeline and changing its configuration to be inactive unless run manually, I haven’t tried that yet. But I thought it’d be a good concept to implement as a process/feature, since I think many others would benefit from it. I hope I communicated my intention clearly.
@brcrista, that did the trick - thank you!
I was wondering what those @X meant, as I have seen some scripts with CondaEnvironment@0, others with CondaEnvironment@1. Now I know that they are version numbers.
I changed to CondaEnvironment@1 and the workaround is no longer needed.
Thank you so much!
Hi @mcg1969. Im not sure its 100% accurate for us to say we moved
condaout of a home directory and into a system wide directory.On the older Hosted Linux Preview pool, the agent process ran as root, and on Hosted Ubuntu 1604 the agent runs as “vsts” but can sudo. I ported the miniconda installation from the old pool to the new one without changing anything. In fact, you can see the installation script used here: https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/linux/scripts/installers/miniconda.sh
The main issue here is the “vsts” user does not exist when the machine image is provisioning so I cant install anything directory to eg /home/vsts . The image generation process linked above aims to generate and generic base image, and then additional customizations like creating the vsts account are done internally as part of the hosted pool service.
Im personally not a miniconda expert by any means, so happy to work with you (or PRs welcome 😉 ) on how we provision miniconda on these machines. If theres some environment variables that can be set to control where it tries to write, for example, that is something that can be changed. Or, if most of the time of an ad-hoc per-build setup is being spent downloading the same tarball, it could be pre-downloaded into a well known location to be picked up by any tasks or scripts used in your build.
You’re absolutely right about that, I think it was just a mix-up as we were making the new agent image. I’m going to see if I can get that fixed after pushing the fix for the task out.