server: Unable to access GCS bucket with workload identity mechanism in GKE
Description Since the version 22.08 it is not possible anymore to get access to the GCS buckets. Triton fails to start with the following error message:
I1107 09:25:28.889159 31 server.cc:259] No server context available. Exiting immediately.
error: creating server: Internal - Unable to create GCS client. Check account credentials.
Triton Information From Triton 22.08, official build.
To Reproduce Here a minimal K8S manifest in order to be able to easily replicate the behavior:
apiVersion: v1
kind: Pod
metadata:
name: workload-identity-test
namespace: "default"
spec:
containers:
- image: nvcr.io/nvidia/tritonserver:22.08-py3
name: workload-identity-test
command: ["sleep","infinity"]
serviceAccountName: <MY_SERVICE_ACCOUNT>
Replace <MY_SERVICE_ACCOUNT> with the name of the created service account to get access to GCS. Once the pod is deployed, run a shell on the pod with:
kubectl exec -it workload-identity-test --namespace default -- /bin/bash
Once done run Triton with:
tritonserver --model-store=<GCS_BUCKET_ADDRESS>
Replace <GCS_BUCKET_ADDRESS> with the name of the bucket where the models are stored.
Expected behavior Being able to use the GKE workload identity mechanism.
Thanks in advance.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 15 (6 by maintainers)
Sorry, it looks like we have only resolved the public GCS bucket issues. @kthui Could you please take a look into this?
I have filed a ticket for this issue.