skypilot: AWS SSO no longer works for managed jobs - all FAILED_PRECHECKS

AWS SSO account all managed jobs are now FAILED_PRECHECKS status.

It used to work, but now:

(sky) ➜  skypilot git:(master) sky check
Checking credentials to enable clouds for SkyPilot.
  AWS: enabled [compute, storage]
    Hint [compute, storage]: AWS SSO is set. To ensure multiple clouds work correctly, please use SkyPilot with static credentials (e.g., ~/.aws/credentials) by unsetting the AWS_PROFILE environment variable.
  Azure: enabled [compute, storage]

🎉 Enabled clouds 🎉
  AWS [compute, storage]
  Azure [compute, storage]

Using SkyPilot API server: http://127.0.0.1:46580

(sky) ➜  skypilot git:(master) env | grep AWS
(sky) ➜  skypilot git:(master) 
(sky) ➜  skypilot git:(master) sky jobs launch -d --cloud aws -y --cpus 2+ --memory 4+ -n t-client-server-c-94-77-job "echo hello world; sleep 60"

(sky) ➜  skypilot git:(master) sky jobs logs 1                                                                                                    
Job 1 is already in terminal state FAILED_PRECHECKS. For more details, run: sky jobs logs --controller 1
Failure reason: [sky.exceptions.ResourcesUnavailableError] Task 't-client-server-c-94-77-job' requires AWS which is not enabled. To enable access, change the task cloud requirement or run: sky check aws
(sky) ➜  skypilot git:(master) 

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15

Most upvoted comments

No, I only run managed jobs, not with sky local up . My kubernetes cluster is not publicly accessible so the controller must have been on my local kubernetes .

This seems to be working for me on current master (052e0ea3b129f208a82fd9d6bddbfd39d389e831):

❯ sky jobs launch -d --cloud aws -y --cpus 2+ --memory 4+ -n t-client-server-c-94-77-job "echo hello world; sleep 60"
Command to run: echo hello world; sleep 60
Managed job 't-client-server-c-94-77-job' will be launched on (estimated):
Launching managed job 't-client-server-c-94-77-job' from jobs controller...
Choosing resources for managed jobs controller...
Considered resources (1 node):
-----------------------------------------------------------------------------------------
 CLOUD   INSTANCE     vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
-----------------------------------------------------------------------------------------
 AWS     r6i.xlarge   4       32        -              us-east-1     0.25          ✔     
-----------------------------------------------------------------------------------------
⚙︎ Launching managed jobs controller on AWS us-east-1 (us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1f).
└── Instance is up.
✓ Cluster launched: sky-jobs-controller-2c3d9986.  View logs: sky api logs -l sky-2025-03-25-10-01-06-871725/provision.log
⚙︎ Syncing files.
  Syncing (to 1 node): /var/folders/4t/b03l11jx2lz6krkxv815mhmm0000gn/T/managed-dag-t-client-server-c-94-77-job-owqdb3um -> ~/.sky/managed_jobs/t-client-server-c-94-77-job-6dc0.yaml
✓ Synced file_mounts.  View logs: sky api logs -l sky-2025-03-25-10-01-06-871725/file_mounts.log
⚙︎ Running setup on managed jobs controller.
[2/5] Check & install cloud dependencies on controller: Kubernetes                    bash: https://dl.k8s.io/release/v1.31.6/bin/linux/amd64/kubectl: No such file or directory
  Check & install cloud dependencies on controller: done.                                        
Created symlink /home/ubuntu/.config/systemd/user/default.target.wants/skypilot-dashboard.service → /home/ubuntu/.config/systemd/user/skypilot-dashboard.service.
✓ Setup completed.  View logs: sky api logs -l sky-2025-03-25-10-01-06-871725/setup-*.log
⚙︎ Job submitted, ID: 1