pipelines: Question: Tasks not getting cached
👋 I am new to Kubeflow, I used kfctl to install kbf. Pipelines work fine, but caching is not working. Tasks never get skipped. There is nothing in the logs either:
kubectl logs -n kubeflow --selector app=cache-server
2020/10/28 17:13:34 Initing client manager....
2020/10/28 17:13:34 Database created
2020/10/28 17:13:34 execution_caches
I made sure that mutatingwebhookconfiguration exists and that is setup as mentioned in https://www.kubeflow.org/docs/pipelines/caching/
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 22 (13 by maintainers)
I figured out the problem. When i deleted the kubeflow pipelines deployment to recreate is from scratch following This, it left behind the mutatingwebhookconfiguration for the namespace (I wasn’t aware that it was a cluster-wide resource). Manually deleting that before reinstalling fixed the cache server issue.
I imagine it had been left from when i had done the kfp uninstall/reinstall before, and had mismatch between TLS certs. Not certain of this, however.
Also, i saw no events or errors in status messages on either of the two resources.
Thank you for all of your help!
Its actually 1.17.4
Thanks Bobgy, good info. I updated my prior comment to clarify that I’m using a kubeflow pipelines standalone deployment (v1.0.4).