kubeflow: crud-web-apps 1.3 Could not find CSRF cookie XSRF-TOKEN in the request

Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions. I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error.

certificate.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

@jmdinis That is indeed because the manifests use a very outdated version of cert-manager, which is also not compatible with KFServing so you should probably update it.

You can use this yaml instead in the mean time:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com

@kimwnasptd While I agree that the web apps not working as expected in HTTP mode needs to be investigated, saying a certificate is highly platform dependent is simply not true. If this were indeed the case, the Kubeflow components would not be able to use cert-manager themselves. Over the past few weeks there have probably been around 30 clusters setup using that certificate manifest, mostly on AWS but also on-prem, while making the ArgoFlow for AWS distribution I started ready for enterprise deployments. The reason I left this issue open is because I knew many people would run into this problem. It is also objectively less work and better practice to use a dummy certificate and HTTPS than it is to change to insecure cookies for every web app. There is also an infinitely higher chance that somebody forgets to change their manifests back to APP_SECURE_COOKIES: "true" everywhere when moving away from a PoC deployment, than when using a dummy certificate. Whereas forgetting to change the issuerRef for the kubeflow gateway is something people would instantly notice when moving out of the PoC stage.

I think the reason is Jupyter uses Volume web app during creating a new notebook server. So you need to set this APP_SECURE_COOKIES env to false in Volume web app as well.

I listed here where I set it (e.g., Tensorboard Web App, Jupyter Web App, Katib UI, even Central Dashboard):

• https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/jupyter/jupyter-web-app/upstream/base/deployment.yaml#L30
• https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/volumes-web-app/upstream/base/deployment.yaml#L23
• https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/katib/upstream/components/ui/ui.yaml#L32
• https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/tensorboard/tensorboards-web-app/upstream/base/deployment.yaml#L23
• https://github.com/kubeflow/manifests/blob/v1.3-branch/apps/centraldashboard/upstream/base/deployment.yaml#L42

Note: This is not recommended in production deployment.

@thesuperzapper I did by having the ingress gateway use HTTPS. For the Argo deployment I’m using the Kubeflow-self-signing-issuer to generate a very for the ingress gateway to avoid people running into this problem. I’m guessing a lot of people are going to run into this so that is why I hadn’t closed the issue yet.

@jalola Just for general information, the Jupyter Web App does not use the Volumes Web App during the creation of a new notebook server. All the web apps are completely independent. I think the better and easier solution to this problem is to expose the Istio ingressgateway with HTTPS. If you don’t have a domain or a proper cluster issuer setup in cert-manager, you can use the kubeflow-self-signing-issuer which is also used by various components and a placeholder domain.

deploy this certificate resource:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com

Then change the kubeflow-gateway to the following:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

Thanks @DavidSpek, but the name of ClusterIssuer in my local cluster is “kubeflow-self-signing-issuer” not “gateways-issuer”. Who run into this problem may check it first by “kubectl get ClusterIssuer -n istio-system”.

@jalola Just for general information, the Jupyter Web App does not use the Volumes Web App during the creation of a new notebook server. All the web apps are completely independent. I think the better and easier solution to this problem is to expose the Istio ingressgateway with HTTPS. If you don’t have a domain or a proper cluster issuer setup in cert-manager, you can use the kubeflow-self-signing-issuer which is also used by various components and a placeholder domain.

deploy this certificate resource:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    name: gateways-issuer
    kind: ClusterIssuer
  commonName: kubeflow.example.com
  dnsNames:
    - kubeflow.example.com

Then change the kubeflow-gateway to the following:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

@nakfour here’s what worked for me:

Firstly, remove cert-manager and Istio-system namespaces.

Secondly install cert-manager and istio by running the following commands from @DavidSpek’s answer above:

Finally, wait a couple minutes till tls secret gets deployed you can check with kubectl get secret -n istio-system

Once deployed, delete the istio-ingressgateway pod so it restarts:

kubectl delete pod istio-ingressgateway-xxx -n istio-system

Full credits to @DavidSpek for this solution.

I agree with @DavidSpek that this should be the default, since it’s causing problems for everyone and it’ll be a pretty easy fix to implement pre-release of 1.3

hi @DavidSpek I am facing this issue too. I deployed a kubeflow cluster on Google Kubernetes Engine using the kubeflow v1.3 manifest. Initially, the istio-ingressway was set to NodePort, I changed that to LoadBalancer and could access the central dashboard. But couldn’t create jupyter notebook server due to the error here.

I have tried the solutions above and nothing is working.

When I check the ingressgateway pod, i get this error Readiness probe failed: Get "http://10.8.2.12:15021/healthz/ready": dial tcp 10.8.2.12:15021: connect: connection refused

What step can i take to expose my service on HTTPS on GKE. my cert-manager

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: istio-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: istio-ingressgateway.istio-system.svc
  # Use ipAddresses if your LoadBalancer issues an IP
  ipAddresses:
  - <external-ip>
  # Use dnsNames if your LoadBalancer issues a hostname (eg on AWS)
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: istio-ingressgateway-certs

kubeflow-gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    tls:
      httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

Thanks

Here the is the scripted version of Kubeflow deployment that includes @davidspek and @geewynn gateway and certs fixes.

Also see full instructions on how to deploy of Kubeflow locally in DinD including nvidia-gpu-operator in Linux or WSL2 with qbo community edition or kind

Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions. I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error. certificate.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

Maybe, this comment should be insert into Kubeflow Manifests installation document as Kubeflow HTTPS setup section or just a example, that more Kubeflow initial user will get a better experience. In fact, the Kubeflow HTTPS setup way troubled me 2 days, becuase I don’t know how to solve “Could not find CSRF cookie XSRF-TOKEN in the request” problem.

tks, it works for me! I installed kubernetes 1.21.13 cluster in my 3 physical servers, and installed kubeflow 1.5.1 according to “https://github.com/kubeflow/manifests/tree/v1.5-branch”.

Here is how I did the fix in KF 1.4 and K8 1.21

Thanks @DavidSpek and @geewynn, I had this issue too and I solved it using your suggestions. I am using K8s on bare-metal server, and I am using NodePort to access ingress-gateway. Apply the yaml below and access https with the node port of the target port 443 of the istio-ingressgateway.In addition, since it is a self-signed certificate, Browser displays a certificate-related error, but if you ignore it and access to central dashboard, you can use the KF without the 403 error.

certificate.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  commonName: <DOMAIN_NAME> #  Ex) kubeflow.mydomain.com 
  issuerRef:
    kind: ClusterIssuer
    name: kubeflow-self-signing-issuer
  secretName: kubeflow-ingressgateway-certs

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    # Upgrade HTTP to HTTPS
    # tls:
    #   httpsRedirect: true
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kubeflow-ingressgateway-certs

Maybe, this comment should be insert into Kubeflow Manifests installation document as Kubeflow HTTPS setup section or just a example, that more Kubeflow initial user will get a better experience. In fact, the Kubeflow HTTPS setup way troubled me 2 days, becuase I don’t know how to solve “Could not find CSRF cookie XSRF-TOKEN in the request” problem.

Thanks @DavidSpek, @geewynn, and @rky0930. Because of your suggestion,I solve the issue I faced during running katib random example.

I deployed a kubeflow cluster on on premise k8s using the kubeflow v1.4 manifest. I am using MetalLB(on premise LoadBalancer) to access ingress-gateway

Below are the steps which I take to expose my service on https.

You can set up the MetalLB configuration using this link.

1. create gateways-issuer.yaml file.

gateways-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kubeflow-ingressgateway-certs
  namespace: istio-system
spec:
  secretName: kubeflow-ingressgateway-certs
  issuerRef:
    kind: ClusterIssuer
    name: gateways-issuer
  commonName: istio-ingressgateway.istio-system.svc
  ipAddresses:
  - 192.168.35.100
  - 192.168.35.101
  - 192.168.35.102

1. apply gateways-issuer.yaml file

kubectl apply -f gateways-issuer.yaml

Then change the kubeflow-gateway to the following:

kubectl edit gateway/kubeflow-gateway -n kubeflow 

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"kubeflow-gateway","namespace":"kubeflow"},"spec":{"selector":{"istio":"ingressgateway"},"servers">
  creationTimestamp: "2021-12-12T14:39:15Z"
  generation: 2
  name: kubeflow-gateway
  namespace: kubeflow
  resourceVersion: "69316"
  selfLink: /apis/networking.istio.io/v1beta1/namespaces/kubeflow/gateways/kubeflow-gateway
  uid: e7bb620a-0aa2-465f-9db2-4765c8eeb535
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
# add below parts
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: kubeflow-ingressgateway-certs
      mode: SIMPLE

After editing kubeflow-gateway gateway, you can access https://192.168.35.100 Also, you can create the notebook without any issues.

Let’s not use this issue to provide the instructions and support around enabling HTTPS, since this is highly platform dependent.

To make the app work with plain http, although this is not encouraged, you can set the APP_SECURE_COOKIES: "false" ENV Var in each app. If after this change your app is still not working with http then lets further discuss and try to understand why the apps don’t behave as expected.

@Dimiftb By executing the commands after what I had sent you you messed up the configuration. I did just realize that the certificate is setup for a more recent version of cert-manager than what is bundled with kubeflow (I’m trying to get the manifest working group to upgrade this). You probably do not need to remove the Istio resources before running the below commands.

Please execute the following commands, and nothing else (all the needed Istio resources are included): First: kustomize build github.com/argoflow/argoflow/cert-manager | kubectl apply -f - Then: kustomize build github.com/argoflow/argoflow/kubeflow/common/istio | kubectl apply -f -

Then wait and once it’s deployed you should see it working. DO NOT deploy kubeflow-istio-resources or change the gateway after running these commands, they are already setup correctly.