istio: Connection Failure to a MySQL Service

Hi,

We are also facing this issue, and found that it is related to the default MeshPolicy. Do you get a CONFLICT when running this command?

istioctl authn tls-check mysqldb.qq2.svc.cluster.local

Deleting the default MeshPolicy solved this issue right away:

kubectl delete meshpolicies.authentication.istio.io default

@venilnoronha can this be an issue with the PERMISSIVE mode for tls?

version 1.6.0 - same problem, but none of the above tricks did not help

I\m using Istio 1.5.2 and havving same issue - none of solutions described under below docs helped https://istio.io/faq/security/#mysql-with-mtls

Any idea how to solve it or troubleshoot further?

I was able to get things working by applying this DestinationRule

cat <<EOF | kubectl apply -f -
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "istio-client-mtls"
spec:
  host: "*.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

And this MeshPolicy:

cat <<EOF | kubectl apply -f -
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
  name: "default"
spec:
  peers:
  - mtls:
        mode: PERMISSIVE
EOF

I’m able to connect to mysql from another pod after this was applied.

Hi @incfly , I got connection failure from istio 1.1.4 & 1.1.5 without the default mesh policy, but it works in 1.1.2.

# mysql -h 10.1.226.216 -P 33061 -u root -p
Enter password:
ERROR 2026 (HY000): SSL connection error: socket layer receive error

This costs me several hours to investigate the cause, and suddenly it works when I rename the port name for mysql-33061 to tcp.

# mysql -h 10.0.137.145 -P 33061 -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.7.26 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

Do we have any changes from 1.1.2 on this part?

@rshriram

if you specify port name as “mysql-xx” we will ignore PERMISSIVE mode

What the hell? Who thinks this is acceptable? Networking should not act totally different based on the name. This is already being done with the http and https prefixes and has caused tons of confusion because it’s not something anyone else does. This is insanity. How about making things work right and not changing the behavior based on a name? I’d prefer to have Istio stick around, but choices like this are going to make Linkerd2 (or another project) become the winning service mesh.

I’ve encountered this issue, version 1.9. Fixed it using

apiVersion: v1
kind: Service
metadata:
  name: mysql-test
spec:
  ports:
    - port: 3306
      protocol: TCP
      targetPort: 3306
      # name: http   # It was http which caused error, because it requires https. To make it work, specify https or no to specify it.
  selector:
    app: mysql-test

Istio reads the name of ports. If it is http, Istio configs it as http protocol. If it is https, Istio configs it as https protocol. See: https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/

Should we consider using a Kubernetes Annotation in addition to the port name to control behavior?

If I saw something like this I would understand that this port is special.

      annotations:
        sidecar.istio.io/mysql-protocol-port: 3306

We could do something clever like 3306 is considered MySQL protocol by default. If the user runs MySQL protocol on a different port this annotation would be used. If the user runs HTTPS we could override that destination with something like:

      annotations:
        sidecar.istio.io/https-protocol-port: 443,3306

I am not proposing we throw away the current behavior, but it new users are often surprised that the name of the port affects how traffic is allowed to flow.

@incfly @mabushey

The crucial edit to the default DestinationRule in my case was host: "*.local" -> host: "*.svc.cluster.local", that was the final missing piece that I had overlooked when looking at the DestinationRule described in https://github.com/istio/istio/issues/10062#issuecomment-453119184 (thanks @Bessonov).

It’s not obvious to me why the previous configuration worked for HTTP, and sometimes TCP (redis), but the new configuration works for mysql/redis/misc TCP services. Now I’m worried I need to add my own sidecar to snoop on the network traffic to make sure it’s actually being encrypted consistently.

All versions of Istio above 1.0.2 (ie 1.0.3 and 1.0.4) have this bug.

In addition to that, we could add a special handling for istioctl authn tls-check for mysql service. If the k8s service define the port name as mysql-xx, istioctl will ensure combination of no authn.PERMISSIVE configured on MySQL pod. This will not help if customers define the service port name as “tcp”.

/cc @diemtvu