faas-netes: Helm Chart is generating incorrect Istio Objects - Policy is deprecated from Istio v1.4/5

Ok, I had a change to retry today and … 💪 success!

$ curl http://localhost:8080/function/env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=env-6dc77984d8-ssh48
fprocess=env
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
HOME=/home/app
Http_User_Agent=curl/7.68.0
Http_X_Forwarded_Host=localhost:8080
Http_Content_Length=0
Http_Accept=*/*
Http_X_Call_Id=d732b7a5-c1f0-443e-8596-efe2c87c9daa
Http_X_Start_Time=1618133183910952047
Http_Accept_Encoding=gzip
Http_X_Envoy_Attempt_Count=1
Http_X_B3_Spanid=2445da32d16249d7
Http_X_Forwarded_For=127.0.0.1:53382
Http_X_Forwarded_Proto=http
Http_X_B3_Traceid=6e374f868082ff89fd4d00fa8878c8b6
Http_X_Request_Id=71a8262b-3a08-4ed7-b2bf-600ed01ae7e6
Http_X_Forwarded_Client_Cert=By=spiffe://cluster.local/ns/openfaas-fn/sa/default;Hash=3575313a26a287d0f823273a79a383086d3d0518642008fb382166adfbf88c65;Subject="";URI=spiffe://cluster.local/ns/openfaas/sa/openfaas-controller
Http_X_B3_Parentspanid=fd4d00fa8878c8b6
Http_X_B3_Sampled=0
Http_Method=GET
Http_ContentLength=0
Http_Path=/
Http_Host=env.openfaas-fn.svc.cluster.local:8080

Renaming the port on the nats Service seems to be the missing piece. Once I did that, everything just started working as expected.

So the next step is to make the changes to the chart right?

@howardjohn i am going to take a look at this tomorrow

Ok, looks like this is what is needed to get latest Istio working. End result is everything is injected with mTLS, prometheus works, etc.

istioctl install -y

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
kubectl label namespace openfaas istio-injection=enabled --overwrite
kubectl label namespace openfaas-fn istio-injection=enabled --overwrite

# Enforce mTLS in openfaas namespaces
cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas-fn
spec:
  mtls:
    mode: STRICT
---
EOF

helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set functionNamespace=openfaas-fn \
    --set generateBasicAuth=true \
    --set gateway.directFunctions=true \ # REQUIRED!
    --set openfaasPRO=false
PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && \
echo "OpenFaaS admin password: $PASSWORD"

# Edit prometheus config kubernetes-pods:
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
# Replace all prometheus.io. with prometheus.io/ in all deployments
# Rename NATS port to "tcp" OR add "appProtocol: TCP" OR inject NATS. Or all 3

kubectl port-forward svc/gateway -n openfaas 31112:8080
# open ui, login with admin:$PASSWORD, deploy `env` function
curl http://127.0.0.1:31112/function/env

# Setup gateway, if you want. You could also use Ingress, but the one in helm chart is a deprecated extensions/v1beta1 version so I didn't want to use that
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: openfaas
  namespace: openfaas
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "openfaas/openfaas.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: openfaas
  namespace: openfaas
spec:
  hosts:
  - "openfaas.example.com"
  gateways:
  - openfaas/openfaas
  http:
  - route:
    - destination:
        host: gateway
EOF
curl <INGRESS_IP>/function/env -H "Host: openfaas.example.com"

I think the changes required (to nats service + prometheus config) should be compatible with non-istio as well.

Thanks, I will see if I can dig into this a bit.

Does Istio still require exec probes?

Nope, should no longer be needed

Hi @howardjohn,

Thank you for reaching out 👍 - your help would be really appreciated.

Whilst we have a broad understanding of service-mesh and hands-on experience with Linkerd, the person who added and maintained the configurations for Istio cannot help with this any more and the alternative is removing Istio instructions.

I’ve reached out to Craig Box a few times to see if folks on the Istio team could help us bring the configs back up to date. OpenFaaS isn’t using any particularly advanced or esoteric features of Istio and doesn’t integrate to the API directly, but users have asked for a working configuration for mTLS.

OpenFaaS can be installed with arkade or helm (arkade is easier) https://docs.openfaas.com/deployment/kubernetes/ - just like Knative, each workload container i.e. “function” is deployed as a Deployment, and then a Pod.

The direct_functions / directFunctions flag has to be set to set to true to use a service mesh, so that the gateway uses the Service name of the function, if not, the endpoints are looked up and randomly accessed by endpoint IP (which I think would invalidate the use of mTLS, or just not work)

We also have settings to switch to and from HTTP and exec probes. Does Istio still require exec probes? (HTTP tend to be more efficient)

A PR would be great, if you are able to give some broad instructions for deploying it with Istio, that would help us verify the behaviour too.

If you’re open to helping, let me know if you need anything here or on CNCF/K8s Slack.

Alex