vector: SSL handshake failed when sources is kafka

A note for the community

  • Please vote on this issue by adding a πŸ‘ reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

SSL handshake failed

Configuration

[sources.kafka_log_topics]
type = "kafka"
bootstrap_servers = "x.x.x.x:9093,x.x.x.x:9093,x.x.x.x:9093"
group_id = "nginxLogGroup"
#librdkafka_options.client.id = "vector_node1"
auto_offset_reset = "latest"
topics = ["hftopic"]
decoding.codec = "json"
tls.enabled = true
#tls.alpn_protocols = ["TLS1.2"]
tls.ca_file = "/opt/certs/ca.crt"
tls.crt_file = "/opt/certs/client.crt"
tls.key_file = "/opt/certs/client.key"
#tls.key_pass = "xxxxxxxxxxx"
# tls.verify_hostname = false
#tls.verify_certificate = false


[transforms.reduce_logs]
type = "filter"
inputs = ["kafka_log_topics"]
condition = '''
  msg = parse_json!(.message)
  starts_with(string!(msg.http_authorization), "xxxxxxxx") && parse_int!(msg.body_bytes_sent) > 0
'''


[transforms.test_logs_transform]
type = "remap"
inputs = ["reduce_logs"]
source = """
  . = parse_json!(.message)

  del(.bytes_sent)
  del(.http_user_agent)
  del(.remote_addr)
  del(.request)
  del(.status)

  epoch = parse_int!(split!(.msec, ".")[0])
  del(.msec)
  ym_str = format_timestamp!(from_unix_timestamp!(epoch), "%Y%m", "Asia/Shanghai")
  .ym = parse_int!(ym_str)

  .http_authorization = replace(string!(.http_authorization), "tlspxy ", "")
  .http_authorization = parse_int!(.http_authorization)

  .body_bytes_sent = parse_int!(.body_bytes_sent)
"""

[sinks.file]
inputs = ["test_logs_transform"]
type = "file"
encoding.codec = "json"
path = "/tmp/test.out"



2023-11-16T02:49:24.350699Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://x.x.x.x:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:(unknown function) error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 6ms in state SSL_HANDSHAKE)    
2023-11-16T02:49:25.349162Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://x.x.x.x:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:(unknown function) error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 6ms in state SSL_HANDSHAKE)    
2023-11-16T02:49:26.347125Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://x.x.x.x:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:(unknown function) error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 4ms in state SSL_HANDSHAKE)    
2023-11-16T02:49:26.347168Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 3/3 brokers are down    
2023-11-16T02:49:27.348611Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://x.x.x.x:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:(unknown function) error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 5ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)    
2023-11-16T02:49:28.349455Z ERROR source{component_kind="source" component_id=kafka_log_topics component_type=kafka}:kafka_source: rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://x.x.x.x:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1890:(unknown function) error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 6ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed) 


### Version

vector 0.34.0 (x86_64-unknown-linux-gnu c909b66 2023-11-07 15:07:26.748571656)

### Debug Output

_No response_

### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 1
  • Comments: 17 (7 by maintainers)

Most upvoted comments

It’s OK when I build vector from master branch source code

# cat vector.toml
[sources.kafka-log-topics]
type = "kafka"
bootstrap_servers = "x.x.x.x:9093,x.x.x.x:9093,x.x.xx:9093"
group_id = "nginxLogGroup"
auto_offset_reset = "latest"
topics = ["hftopic"]
decoding.codec = "json"
tls.enabled = true
tls.ca_file = "/opt/certs/ca.crt"
tls.crt_file = "/opt/certs/client.crt"
tls.key_file = "/opt/certs/client.key"
tls.verify_hostname = false
tls.verify_certificate = true


[transforms.reduce_logs]
type = "remap"
inputs = ["kafka-log-topics"]
source = """
  del(.@timestamp)
  del(.headers)
  del(.message_key)
  del(.offset)
  del(.partition)
  del(.source_type)
  del(.timestamp)
  del(.topic)
"""


[sinks.file]
#inputs = ["test_logs_transform"]
inputs = ["reduce_logs"]
type = "file"
encoding.codec = "json"
path = "/tmp/test.out"