istio: TLS connection failures after Istio 1.3 upgrade
Bug description After upgrading to Istio 1.3 our services are losing their database connections after what seems to be fixed intervals. This causes errors in the application, as it cannot write data using the already open connection - causing a RollBack of the transaction. Downgrading to istio 1.2.5 resolves the problem. Our setup is as follows. AWS EKS. Istio 1.3 with mTLS and global.outboundTrafficPolicy.mode=REGISTRY_ONLY. AWS RDS Postgresql 9.4. Pods access RDS via a VirtualService and ServiceEntry:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
spec:
hosts:
- fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
- fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
location: MESH_EXTERNAL
ports:
- name: tcp-5432
number: 5432
protocol: tcp
resolution: NONE
And VirtualService to allow access to the RDS instances are via SNI inspection.
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
hosts:
- fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
- fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
tls:
- match:
- port: 5432
sniHosts:
- fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
route:
- destination:
host: fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
port:
number: 5432
- match:
- port: 5432
sniHosts:
- fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
route:
- destination:
host: fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
port:
number: 5432
Logs from istio-proxy for affected services: Service A: Custom JPA java service
[{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59968","duration":"287272","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:44.315Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59966","authority":"-","path":"-"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59926","duration":"291214","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:40.373Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59924","authority":"-","path":"-"}
,{"upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:60066","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:60068","duration":"283127","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:48.461Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:60046","duration":"285386","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:46.203Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:60044","authority":"-","path":"-"}
,{"upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59954","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59956","duration":"288130","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:43.459Z","method":"-","request_id":"-"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59922","duration":"291252","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:40.336Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59920","authority":"-","path":"-"}
,{"start_time":"2019-09-17T09:18:38.459Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59888","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59890","duration":"293131","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-"}
,{"bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59852","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59854","duration":"295434","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:36.155Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-"}
,{"downstream_remote_address":"REDACTED_IP1:59816","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59818","duration":"297304","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:34.286Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"3424","istio_policy_status":"-","bytes_sent":"6445","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59806","duration":"297837","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:33.752Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5193","istio_policy_status":"-","bytes_sent":"5952","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59804","authority":"-","path":"-"}
,{"start_time":"2019-09-17T09:18:33.461Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"190283","istio_policy_status":"-","bytes_sent":"136744","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59800","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59802","duration":"298128","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-"}]
Service B: Keycloak IdP
2019-09-17T08:28:58.915759Z info FLAG: --applicationPorts="[8080,7600]"
2019-09-17T08:28:58.915794Z info FLAG: --binaryPath="/usr/local/bin/envoy"
2019-09-17T08:28:58.915801Z info FLAG: --concurrency="2"
2019-09-17T08:28:58.915806Z info FLAG: --configPath="/etc/istio/proxy"
2019-09-17T08:28:58.915812Z info FLAG: --connectTimeout="10s"
2019-09-17T08:28:58.915817Z info FLAG: --controlPlaneAuthPolicy="MUTUAL_TLS"
2019-09-17T08:28:58.915823Z info FLAG: --controlPlaneBootstrap="true"
2019-09-17T08:28:58.915828Z info FLAG: --customConfigFile=""
2019-09-17T08:28:58.915832Z info FLAG: --datadogAgentAddress=""
2019-09-17T08:28:58.915837Z info FLAG: --disableInternalTelemetry="false"
2019-09-17T08:28:58.915845Z info FLAG: --discoveryAddress="istio-pilot.istio-system:15011"
2019-09-17T08:28:58.915849Z info FLAG: --dnsRefreshRate="300s"
2019-09-17T08:28:58.915854Z info FLAG: --domain="our_ns.svc.cluster.local"
2019-09-17T08:28:58.915859Z info FLAG: --drainDuration="45s"
2019-09-17T08:28:58.915862Z info FLAG: --envoyAccessLogService=""
2019-09-17T08:28:58.915865Z info FLAG: --envoyMetricsServiceAddress=""
2019-09-17T08:28:58.915869Z info FLAG: --help="false"
2019-09-17T08:28:58.915875Z info FLAG: --id=""
2019-09-17T08:28:58.915879Z info FLAG: --ip=""
2019-09-17T08:28:58.915887Z info FLAG: --lightstepAccessToken=""
2019-09-17T08:28:58.915892Z info FLAG: --lightstepAddress=""
2019-09-17T08:28:58.915896Z info FLAG: --lightstepCacertPath=""
2019-09-17T08:28:58.915900Z info FLAG: --lightstepSecure="false"
2019-09-17T08:28:58.915904Z info FLAG: --log_as_json="false"
2019-09-17T08:28:58.915908Z info FLAG: --log_caller=""
2019-09-17T08:28:58.915913Z info FLAG: --log_output_level="default:info"
2019-09-17T08:28:58.915917Z info FLAG: --log_rotate=""
2019-09-17T08:28:58.915921Z info FLAG: --log_rotate_max_age="30"
2019-09-17T08:28:58.915926Z info FLAG: --log_rotate_max_backups="1000"
2019-09-17T08:28:58.915930Z info FLAG: --log_rotate_max_size="104857600"
2019-09-17T08:28:58.915934Z info FLAG: --log_stacktrace_level="default:none"
2019-09-17T08:28:58.915945Z info FLAG: --log_target="[stdout]"
2019-09-17T08:28:58.915949Z info FLAG: --mixerIdentity=""
2019-09-17T08:28:58.915954Z info FLAG: --parentShutdownDuration="1m0s"
2019-09-17T08:28:58.915958Z info FLAG: --pilotIdentity=""
2019-09-17T08:28:58.915964Z info FLAG: --proxyAdminPort="15000"
2019-09-17T08:28:58.915968Z info FLAG: --proxyComponentLogLevel="misc:error"
2019-09-17T08:28:58.915972Z info FLAG: --proxyLogLevel="warning"
2019-09-17T08:28:58.915977Z info FLAG: --serviceCluster="keycloak.our_ns"
2019-09-17T08:28:58.915981Z info FLAG: --serviceregistry="Kubernetes"
2019-09-17T08:28:58.915985Z info FLAG: --statsdUdpAddress=""
2019-09-17T08:28:58.915990Z info FLAG: --statusPort="15020"
2019-09-17T08:28:58.915993Z info FLAG: --templateFile=""
2019-09-17T08:28:58.915997Z info FLAG: --trust-domain=""
2019-09-17T08:28:58.916003Z info FLAG: --zipkinAddress="jaeger-collector.jaeger.svc.cluster.local:9411"
2019-09-17T08:28:58.916037Z info Version root@2795c9cf-d4de-11e9-bd7e-6a558b8effed-docker.io/istio-1.3.0-c2bd59595ce699b31d0f931885f023028ff7902b-Clean
2019-09-17T08:28:58.943630Z info Obtained private IP [REDACTED_IP2]
2019-09-17T08:28:58.943721Z info Proxy role: &model.Proxy{ClusterID:"", Type:"sidecar", IPAddresses:[]string{"REDACTED_IP2", "REDACTED_IP2"}, ID:"keycloak-79c6949b47-rkq54.our_ns", Locality:(*core.Locality)(nil), DNSDomain:"our_ns.svc.cluster.local", TrustDomain:"cluster.local", PilotIdentity:"", MixerIdentity:"", ConfigNamespace:"", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)}
2019-09-17T08:28:58.943744Z info PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2019-09-17T08:28:58.944424Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istio-pilot.istio-system:15011
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: keycloak.our_ns
statNameLength: 189
tracing:
zipkin:
address: jaeger-collector.jaeger.svc.cluster.local:9411
2019-09-17T08:28:58.944445Z info Monitored certs: []string{"/etc/certs/key.pem", "/etc/certs/root-cert.pem", "/etc/certs/cert-chain.pem"}
2019-09-17T08:28:58.944457Z info waiting 2m0s for /etc/certs/key.pem
2019-09-17T08:28:58.944476Z info waiting 2m0s for /etc/certs/root-cert.pem
2019-09-17T08:28:58.944489Z info waiting 2m0s for /etc/certs/cert-chain.pem
2019-09-17T08:28:58.944596Z info PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2019-09-17T08:28:58.947534Z info Starting proxy agent
2019-09-17T08:28:58.947830Z info Opening status port 15020
2019-09-17T08:28:58.948228Z info watching /etc/certs for changes
2019-09-17T08:28:58.948241Z info Received new config, resetting budget
2019-09-17T08:28:58.948620Z info Reconciling retry (budget 10)
2019-09-17T08:28:58.948640Z info Epoch 0 starting
2019-09-17T08:28:58.995583Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster keycloak.our_ns --service-node sidecar~REDACTED_IP2~keycloak-79c6949b47-rkq54.our_ns~our_ns.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-09-17 08:28:59.375][17][warning][config] [external/envoy/source/server/options_impl.cc:193] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead.
[2019-09-17 08:28:59.942][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-09-17 08:28:59.942][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
2019-09-17T08:29:00.314844Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-17T08:29:02.686142Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-09-17 08:29:02.969][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:03.026][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:05.601][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:05.681][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T08:29:05.938502Z info Envoy proxy is ready
{"bytes_sent":"380382","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45437","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:43976","duration":"85433","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:39.511Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"41400","istio_policy_status":"-"}
{"duration":"4489","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:31:04.940Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52733","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55605","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:31:46.178Z","method":"-","request_id":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33211","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:32:25.920Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:33:03.264Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49435","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45225","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:33:45.970Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:34:28.333Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45403","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55973","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:35:10.447Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:35:41.326Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52953","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54037","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:36:12.330Z","method":"-","request_id":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42125","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:36:48.683Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35561","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:37:19.359Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:59319","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:37:45.857Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"start_time":"2019-09-17T08:38:20.362Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48135","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
[2019-09-17 08:39:01.187][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 08:39:01.396][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:01.399][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:03.980][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:04.074][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40159","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:38:53.569Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55745","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:39:37.187Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"127.0.0.6:54749","duration":"630932","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:16.427Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"11008","istio_policy_status":"-","bytes_sent":"1630862","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:54298","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"response_flags":"-","start_time":"2019-09-17T08:29:47.915Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"7954","istio_policy_status":"-","bytes_sent":"6804","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:54038","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54040","duration":"599445","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54220","duration":"591253","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:56.107Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"50656","istio_policy_status":"-","bytes_sent":"80635","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:54218","authority":"-","path":"-"}
{"downstream_remote_address":"REDACTED_IP2:54058","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54060","duration":"598377","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:48.983Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"46839","istio_policy_status":"-","bytes_sent":"109279","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42353","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:39:56.699Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:40:18.921Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43153","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55957","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:40:56.963Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:41:29.230Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48199","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"start_time":"2019-09-17T08:42:08.310Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38977","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51779","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:42:46.318Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:43:26.624Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49625","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
[2019-09-17 08:44:04.179][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 08:44:04.553][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:04.556][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:06.980][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:07.074][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"downstream_remote_address":"REDACTED_IP2:46989","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:43:59.052Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:44:15.164Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:50565","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"duration":"288926","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:40:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758666","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:60416","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:53535"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:47849","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:44:43.287Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54909","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:45:06.671Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:35132","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"407","upstream_local_address":"-","duration":"408","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T08:45:35.612Z","method":"GET","request_id":"b96a034e-126e-9da6-967c-9ebf840bbdc8"}
{"upstream_local_address":"-","duration":"410","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T08:45:35.612Z","method":"GET","request_id":"2d50ff24-d14e-99e8-a3eb-a52e4e0e9362","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:35134","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"410"}
{"upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:45:37.769Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58129","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43687","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:46:05.550Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60717","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:46:37.382Z","method":"-","request_id":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:16.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52239","authority":"-","path":"-"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33501","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:32.458Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53405","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:59.695Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55685","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:48:43.431Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 08:49:07.179][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 08:49:07.322][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:07.326][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:10.078][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:10.172][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:59579","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:49:19.431Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"requested_server_name":"-","bytes_received":"1981","istio_policy_status":"-","bytes_sent":"4304","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:44344","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:44346","duration":"257828","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:35.619Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-"}
{"bytes_sent":"181114","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:43784","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:43786","duration":"286962","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:06.485Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"71237","istio_policy_status":"-"}
{"bytes_sent":"758696","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:34884","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:58615","duration":"292027","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34675","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:49:44.478Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43731","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:50:13.377Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48033","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:50:56.574Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58363","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:51:27.790Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55183","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:51:43.506Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53303","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:52:22.047Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:52:47.413Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49921","authority":"-","path":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:53:23.653Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55607","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
[2019-09-17 08:54:10.279][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 08:54:10.398][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:10.473][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:13.075][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:13.079][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T08:54:00.007Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53253","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:54:26.540Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54817","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:37594","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:51887","duration":"294920","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:50:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758732"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:54:59.622Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40555","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:55:34.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33221","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
[2019-09-17 08:56:04.944][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 08:56:05.402][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:05.405][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:07.782][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:07.874][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T08:55:58.154Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:39047","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35527","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:56:38.221Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"303518","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:40302","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:55337","duration":"109727","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:55:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"2048","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46505","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:57:20.669Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:57:36.995Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:57243","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55567","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:58:02.340Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"9","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.2","response_flags":"-","start_time":"2019-09-17T08:58:34.216Z","method":"GET","request_id":"7728adbe-b55b-96f2-b9f2-ae62ce916117","upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.64.219","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.64.219:0","authority":"saml.fut.trifork.com","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"8"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.75.43","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1462","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.43:0","authority":"saml.fut.trifork.com","path":"/auth/realms/master/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"215","upstream_local_address":"-","duration":"215","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.2","response_flags":"-","start_time":"2019-09-17T08:58:34.281Z","method":"GET","request_id":"be41771d-9537-9411-9e00-aecdd3aa2400"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:58:39.033Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35205","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55051","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:59:04.941Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:59:49.911Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40611","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:00:06.252Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40747","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"response_flags":"-","start_time":"2019-09-17T09:00:16.688Z","method":"GET","request_id":"cc4e4316-bd98-9070-bdfe-0e2bfd6004f4","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1789","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.34.96:49514","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/.well-known/uma2-configuration","protocol":"HTTP/1.1","upstream_service_time":"28","upstream_local_address":"-","duration":"28","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"72","istio_policy_status":"-","bytes_sent":"2651","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.34.96:49514","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"914","upstream_local_address":"-","duration":"915","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:21.996Z","method":"POST","request_id":"7a26ffbe-7e33-9f3e-91d8-314fde3e3a2d"}
{"downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"8","upstream_local_address":"-","duration":"9","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:25.809Z","method":"GET","request_id":"e3e42a11-7827-9ad4-b080-bbed2fdb5538","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local"}
{"bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"378","upstream_local_address":"-","duration":"379","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:25.847Z","method":"POST","request_id":"95072a05-a3d8-924f-8314-c8dfc48ea165","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-"}
{"bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"441","upstream_local_address":"-","duration":"441","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.242Z","method":"GET","request_id":"6afa974c-f86e-907b-9f15-0dfd3bb54143","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"5","upstream_local_address":"-","duration":"5","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:26.700Z","method":"GET","request_id":"649bbb76-436c-9d27-a4b8-c9fe7e0ff515","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"91","upstream_local_address":"-","duration":"91","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.760Z","method":"POST","request_id":"2af6d861-a2ad-9d3f-b71d-da876696071f","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-","bytes_sent":"1825"}
{"downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"17","upstream_local_address":"-","duration":"17","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.859Z","method":"GET","request_id":"6d6de6a0-b6e3-9284-a9f1-564ca1892766","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local"}
{"protocol":"HTTP/1.1","upstream_service_time":"4","upstream_local_address":"-","duration":"4","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:27.475Z","method":"GET","request_id":"274ddff2-47f0-952e-81a2-ee1fee60da88","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs"}
{"bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"122","upstream_local_address":"-","duration":"123","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.526Z","method":"POST","request_id":"0cf41600-100c-9ac7-9ef4-305a2e6bc324","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-"}
{"bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"21","upstream_local_address":"-","duration":"22","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.657Z","method":"GET","request_id":"6683f498-32b4-9467-affe-53f3001d1a10","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"4","upstream_local_address":"-","duration":"4","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:27.708Z","method":"GET","request_id":"90dcf838-ad89-9b77-a12a-fb1687b1a47d","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"-","start_time":"2019-09-17T09:00:27.775Z","method":"POST","request_id":"15130842-5e0f-91da-8bf2-f42ae6a1a66f","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-","bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"133","upstream_local_address":"-","duration":"133","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"15","upstream_local_address":"-","duration":"16","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.917Z","method":"GET","request_id":"6c745f50-88fd-92af-92ea-f0e1dbf0d445","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"540"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56977","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:00:43.563Z","method":"-","request_id":"-"}
[2019-09-17 09:01:07.975][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:01:08.081][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:01:08.086][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:01:10.372203Z info Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://127.0.0.1:15000/server_info: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2019-09-17 09:01:10.478][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:01:10.573][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:01:10.673142Z info Envoy proxy is ready
{"duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:07.027Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52197","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:33.254Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55639","authority":"-","path":"-"}
{"bytes_sent":"758816","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:41402","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:41907","duration":"292326","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:57:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-"}
{"downstream_remote_address":"REDACTED_IP2:58898","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:58900","duration":"199439","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:58:34.308Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"51089","istio_policy_status":"-","bytes_sent":"42209","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:56.357Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45489","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:02:35.383Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36387","authority":"-","path":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58539","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:03:13.660Z","method":"-","request_id":"-"}
{"upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:03:52.751Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56261","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.64.219:0","authority":"saml.fut.trifork.com","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"347","upstream_local_address":"-","duration":"348","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"PostmanRuntime/7.15.2","response_flags":"-","start_time":"2019-09-17T09:04:39.855Z","method":"POST","request_id":"ea4c4f82-19b1-94f8-b66b-008b9b5a236b","upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.64.219","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"62","istio_policy_status":"-","bytes_sent":"2576"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33007","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:04:36.017Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:37641","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:05:13.459Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:05:35.612Z","method":"GET","request_id":"24030340-b20d-9809-afb1-3cf57cc3211d","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:45980","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"2","upstream_local_address":"-","duration":"3","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default"}
{"start_time":"2019-09-17T09:05:35.612Z","method":"GET","request_id":"15eaf9ec-b9c6-9a8f-a2d6-19947382984e","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:45978","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"10","upstream_local_address":"-","duration":"10","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-"}
{"downstream_remote_address":"REDACTED_IP2:55527","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:05:30.766Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
[2019-09-17 09:06:10.578][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:06:10.615][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:10.618][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:12.974][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:12.978][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:32895","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:06:12.071Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38191","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:06:34.317Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"136362","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:04:39.889Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5155","istio_policy_status":"-","bytes_sent":"11328","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:37478","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:37480"}
{"requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758846","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:44114","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:49777","duration":"294831","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:02:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38037","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:17.270Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48681","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:37.730Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43129","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:59.789Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:47021","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:08:40.069Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:41619","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:09:20.058Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43947","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:09:55.122Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34237","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:10:11.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:10:38.885Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53343","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
[2019-09-17 09:11:13.079][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:11:13.237][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:13.242][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:15.576][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:15.672][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49673","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:11:21.573Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758869","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:46816","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:59963","duration":"297426","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:07:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36457","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:11:58.589Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"response_flags":"UF,URX","start_time":"2019-09-17T09:12:41.650Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48467","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46187","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:09.464Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42705","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:27.491Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52555","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:57.743Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53757","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:14:17.473Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51633","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"9997","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:14:49.186Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35791","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:15:33.250Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56953","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:15:53.640Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:16:15.683][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:16:16.092][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:16:16.095][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:16:18.371980Z info Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://127.0.0.1:15000/server_info: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2019-09-17 09:16:18.380][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:16:18.473][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:16:18.574380Z info Envoy proxy is ready
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56963","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:16:17.372Z","method":"-","request_id":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55795","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:16:45.723Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:49116","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:49118","duration":"115170","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:15:06.477Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"3923","istio_policy_status":"-","bytes_sent":"4711"}
{"upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:49518","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:41719","duration":"300227","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:12:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5376","istio_policy_status":"-","bytes_sent":"796906"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60021","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:17:11.890Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"downstream_remote_address":"REDACTED_IP2:58899","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10011","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:17:30.788Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56157","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:18:06.637Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:18:41.270Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56171","authority":"-","path":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34729","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:19:18.033Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:57949","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:19:59.122Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51397","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:20:19.743Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60499","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:20:44.023Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:21:18.575][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:21:18.900][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:18.972][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:21.278][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:21.374][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T09:21:24.904Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58719","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45621","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:21:42.869Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:17:16.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"759108","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:52366","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:60615","duration":"288127","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-"}
{"start_time":"2019-09-17T09:22:21.314Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56619","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35463","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:22:42.451Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:22:59.781Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:32907","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:41053","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:23:20.542Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:23:37.246Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36351","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"downstream_remote_address":"REDACTED_IP2:42211","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:24:09.564Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46049","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:24:46.707Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:25:16.841][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:25:17.084][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:17.087][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:19.582][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:19.677][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"response_flags":"UF,URX","start_time":"2019-09-17T09:25:22.159Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46273","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"upstream_local_address":"-","duration":"20","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:25:35.612Z","method":"GET","request_id":"5aa1e428-4b46-925c-b5d0-7c6fb588f4a4","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:56816","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"19"}
{"bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:56814","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"30","upstream_local_address":"-","duration":"30","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:25:35.612Z","method":"GET","request_id":"c178f8fd-ea29-936a-8c46-8528d52b52d7","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58027","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:25:38.035Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"607347","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:55066","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:45605","duration":"226507","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:22:16.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"4096","istio_policy_status":"-"}
{"downstream_remote_address":"REDACTED_IP2:43933","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:26:19.820Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42991","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:26:54.796Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38733","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:27:38.326Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:01.526Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40857","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35657","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:20.225Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36679","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:45.935Z","method":"-","request_id":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:29:23.707Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55843","authority":"-","path":"-"}
[2019-09-17 09:30:19.778][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-09-17 09:30:20.153][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:30:20.156][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
One thing that jumps out at me is that durations for the closed are very similar (~287000 for Service A, ~10000 for Service B). Also they seem to be followed up by a reconfiguration of Envoy due to gRPC config stream closed: 13,
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure [ ] Docs [ ] Installation [x] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure
Expected behavior Connections should not close unexpectedly
Steps to reproduce the bug
Version (include the output of istioctl version --remote
and kubectl version
)
$ istioctl version # after downgrade
client version: 1.2.4
citadel version: 1.2.5
galley version: 1.2.5
ingressgateway version: 1.2.5
pilot version: 1.2.5
policy version: 1.2.5
sidecar-injector version: 1.2.5
telemetry version: 1.2.5
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T16:54:35Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-eks-5ac0f1", GitCommit:"5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1", GitTreeState:"clean", BuildDate:"2019-08-20T22:39:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed? Helm
Environment where bug was observed (cloud vendor, OS, etc) AWS EKS.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 79 (25 by maintainers)
So far things are looking good with the daily build mentioned above after ~7 hours. No connection drops observed.
I made the change at ~ 08:55 below - (i.e the blue
87444c67d
replicaset) and restarted deployments after that. Very low usage environment; and relatively small mesh; so can just see less spikey pilot CPU usage.We’ve seen MySql connections being dropped and we also have Redis usage; but I haven’t looked into either of them in detail with respect to this issue. Probably because on app side we have quite robust recovery mechanisms within their connection pools.
The reason I’ve been looking into this issue is that one of our services is a non-DB, non-cache TLS/TCP custom protocol and the app-layer connection/session pooling mechanism is not as robust as standard DB/redis pools so the Istio instability underneath the app had exposed an app-level problem for us.
To make it a bit more visual in a relatively low usage environment - the graph below from Istio Workload shows instability starting at exactly the point that we upgraded to Istio 1.3 in this environment 😦
@Shahard2 In each deployment of our services. I don’t think there is any Istio config to do this at mesh level. For that I’ve opened a feature request.
Hi all, I am really confused about the handling of this issue. We for now solved this by a bad workaround and ping our DB every minute, which is a really shitty solution in Production.
There is a pretty similiar thread here https://github.com/istio/istio/issues/19321 and I ve the feeling both have been closed pretty fast. We really tried also to check if this is by TCP settings in OS, K8s etc. but it didnt change anything.
“Something” in Istio is currently forcing TCP disconnects and affects the communication between services. Can you guys maybe make a double check on this, too? It shall be easily reproducable.
Thank you all for testing! just to be clear, our daily images are not really suited for production usage, and change every day, so they may not be stable. I recommended them only to test out this issue - use at your own risk.
On Fri, Oct 4, 2019, 4:11 AM Lasse Højgaard notifications@github.com wrote:
Setting env var
PILOT_DISABLE_XDS_MARSHALING_TO_ANY
totrue
on theistio-pilot
deployment did indeed resolve this issue.Seeing some incredible CPU spikes every 5 minutes though in a zero traffic staging cluster. With just a request per second or so, the pilot HPA maxes out the number of pods for a minute or so, and then back down.
Looking forward to @nrjpoddar’s fix so we don’t have to do this in production 😅
(
PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true
added around 00:40)@chadlwilson good point, probably not the same issue. They were seeing similar error messages so I thought I would mention that in case others were running into that.
Maybe similar issue to https://github.com/istio/istio/issues/17383#issuecomment-535459139, can you try setting that option in pilot mentioned there. Note my warning there - pilot will use about 10x CPU when you set that
@Shahard2 For us, using annotation
traffic.sidecar.istio.io/excludeOutboundPorts
is working, for now.@marziman OK, although the way you initially described it as being when the connections were idle which is a bit different to this. If you are seeing it regardless of the connections being used/idle, and you can see a correlation to when pilot is pushing config (based on proxy logs) and then a drop shortly after that then it might be something similar.
We haven’t moved to
1.6.x
yet (still on1.5.x
since our migrations must involve zero downtime thus have to be done carefully) so I can’t speak to whether the issue might have been re-introduced there, but it seems unlikely to be a branch issue to me because the fix was in envoy itself, which master would be regularly updated, and the release branches are regularly cut from master.The commenter in https://github.com/istio/istio/issues/17139#issuecomment-684118616 also said that they weren’t affected by this issue in a subsequent comment. @courcelm - are you perhaps able to share what your issue was in the end, and how you resolved it, if it wasn’t caused by this issue?
In any case, I feel it might be more productive to open & describe your issue independently with appropriate context as to your config, even if you refer to this issue as appearing similar. You may also want to describe your upgrade path and whether you restarted everything cleanly, because to my knowledge skip-version upgrades (especially over that many versions - you’ve moved over a year of releases there) are not really supported/validated, so depending on how you executed the upgrade I wonder if the control plane and/or proxies got into a messy state.
@JoshuaFox At least from a user perspective, we have not had this issue resurface on any of the Envoy versions used by Istio
1.4.[23456789]
or1.5.[457]
versions we have deployed, and we would notice it due to the way our application services and connection pooling work.@ckcd This log is normal. The grpc connection proxy----pilot disconnects at a period of 30 minutes by default
It would be good to understand how to reproduce it and if possible we can write some tests. This issue feels like something that can prop up again if serialization is not stable.
Sorry if my original message was confusing – PILOT_DISABLE_XDS_MARSHALING_TO_ANY should be
false
. We have established that if we set it totrue
the issue is resolved, but pilot takes a major performance hit. What we want to verify is if we set it tofalse
with the new images if its resolved as well.Thanks for trying this out!
@chadlwilson @jmagnusson on the master branch we do have this change. Can one of you try out master images? That is,
gcr.io/istio-release/proxyv2:master-latest-daily
and with the defaultPILOT_DISABLE_XDS_MARSHALING_TO_ANY=false
.I don’t want to go through all the effort + waiting for a 1.3 release to find out it doesn’t fix the problem.
Also if anyone has a way I could easily reproduce this on my own cluster that would be very much appreciated
Arr you having similar issues with other external services? For us Redis also sees sporadic connectivity issues since istio 1.3.1 (through Google Cloud Memorystore)