kubeflow: Can not select namespace in central dashboard - CentralDashboard can't talk to kfam; Use DNS and not environment variables
/kind bug
What steps did you take and what happened:
namespace doesn’t have a list even I create profile. namespaces were created in backend.
Checking central dashboard logs and notice we have this problem. aws platform is not support here which makes entire dashboard unavailable.
k logs -f centraldashboard-7bd77d6dd4-trhhg
> kubeflow-centraldashboard@0.0.2 start /app
> npm run serve
> kubeflow-centraldashboard@0.0.2 serve /app
> node dist/server.js
Initializing Kubernetes configuration
"aws" is not a supported platform for Metrics
Using Profiles service at http://localhost:8084/kfam
Server listening on port http://localhost:8082
Unable to get environment info: Unexpected error getting environment info
Error: connect ECONNREFUSED 127.0.0.1:8084
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Unable to get environment info: Unexpected error getting environment info
Error: connect ECONNREFUSED 127.0.0.1:8084
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Unable to get environment info: Unexpected error getting environment info
Error: connect ECONNREFUSED 127.0.0.1:8084
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Unable to get environment info: Unexpected error getting environment info
Error: connect ECONNREFUSED 127.0.0.1:8084
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)
Environment:
- Kubeflow version: v0.6.2-rc0
- kfctl version: (use
kfctl version): v0.6.1-rc.2-1-g3a37cbc6 - Kubernetes platform: (e.g.
minikube) aws - Kubernetes version: (use
kubectl version): 1.13.7 - OS (e.g. from
/etc/os-release): darwin
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 2
- Comments: 41 (33 by maintainers)
Commits related to this issue
- Bug fixes targeting #3900 - Default Kfam address now localhost / profiles-kfam based on env - Default port for kfam 8084 -> 8081 - Port for webpack dev proxy 8081 -> 8080 — committed to avdaredevil/kubeflow by avdaredevil 5 years ago
- [CentralDashboard v2] Registration flow for basic auth (#3899) * Feature Enhancements | Bug Fixes | Basic Auth Reg: - Registration View logic added for non-identity networks - Custom note added for b... — committed to kubeflow/kubeflow by avdaredevil 5 years ago
- Updates: - Fix for #3900 by supporting empty `USERID_HEADER` - Debug route expanded to include identity headers — committed to avdaredevil/kubeflow by avdaredevil 5 years ago
- [Centraldashboard v2] Added a /debug route to dashboard for python e2e from Kubeflow (#3980) * Added a /debug route to dashboard for python e2e from Kubeflow * Updates: - Fix for #3900 by supporting... — committed to kubeflow/kubeflow by avdaredevil 5 years ago
- Bug fixes targeting #3900 - Default Kfam address now localhost / profiles-kfam based on env - Default port for kfam 8084 -> 8081 - Port for webpack dev proxy 8081 -> 8080 — committed to abhi-g/kubeflow by avdaredevil 5 years ago
- Updates: - Fix for #3900 by supporting empty `USERID_HEADER` - Debug route expanded to include identity headers — committed to abhi-g/kubeflow by avdaredevil 5 years ago
- Cherry pick of #3899 #3933 #3971 #3980 on v0.6-branch. (#3990) * Feature Enhancements | Bug Fixes | Basic Auth Reg: - Registration View logic added for non-identity networks - Custom note added for ... — committed to kubeflow/kubeflow by abhi-g 5 years ago
- [CentralDashboard v2] Registration flow for basic auth (#3899) * Feature Enhancements | Bug Fixes | Basic Auth Reg: - Registration View logic added for non-identity networks - Custom note added for b... — committed to StatCan/kubeflow by avdaredevil 5 years ago
- [Centraldashboard v2] Added a /debug route to dashboard for python e2e from Kubeflow (#3980) * Added a /debug route to dashboard for python e2e from Kubeflow * Updates: - Fix for #3900 by supporting... — committed to StatCan/kubeflow by avdaredevil 5 years ago
- [CentralDashboard v2] Registration flow for basic auth (#3899) * Feature Enhancements | Bug Fixes | Basic Auth Reg: - Registration View logic added for non-identity networks - Custom note added for b... — committed to StatCan/kubeflow by avdaredevil 5 years ago
- [Centraldashboard v2] Added a /debug route to dashboard for python e2e from Kubeflow (#3980) * Added a /debug route to dashboard for python e2e from Kubeflow * Updates: - Fix for #3900 by supporting... — committed to StatCan/kubeflow by avdaredevil 5 years ago
Facing the same issue with 0.6.2rc1 kfctl_k8s_istio.yaml. The config was the same for master.
I am experiencing the same issue when deploying on an On-Premise cluster on kubeflow version 0.6.1 and kubernetes version 1.14.3. There are no errors in the log but the namespace list is empty on the dashboard.
I can see and use other user’s（namespace）servers when I log in by a common user. How to isolate by multi-user. (0.6.2-rc.1)
@jlewi Yes you are correct in spotting where the issue lies. I borrowed the pattern from what we did in setting up the Metadata UI, which itself was derived from the Pipelines UI. I think the difference in those products is that the order of resources is explicitly defined such as in https://github.com/kubeflow/manifests/blob/master/metadata/base/kustomization.yaml, but it still seems like it could be a potential issue depending on when the service actually becomes available.
In any case, I left a comment on #3880 which @avdaredevil already had open to address some MUI issues. The suggestion of using the hostname based on the service name should work and because
centraldashboardandprofiles-kfamget deployed in the same namespace, the service name is all that’s needed to resolve via DNS.Hi @Jeffwan ! The problem is the following:
@avdaredevil how are we going to deal with this? cc @jlewi
/priority p0