VictoriaLogs: Jaeger v2 via opentelemetry-operator connecting to the VictoriaLogs ElasticSearch backend doesn't work properly

Is your question request related to a specific component?

VictoriaLogs

Describe the question in detail

Note: I have a discussion going on the Jaeger repo here: https://github.com/orgs/jaegertracing/discussions/6756#discussioncomment-12252595 which asks much of the same questions, nothing so far from them.

When deploying Jaeger v2 via the opentelemetry-operator (e.g., using https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/config-elasticsearch.yaml) and attempting to use VictoriaLogs’ ElasticSearch backend (e.g., https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/#elasticsearch), the Jaeger pod fails with:

Error: failed to start extensions: failed to initialize storage 'victoria_logs': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available
2025/02/19 16:04:43 failed to start extensions: failed to initialize storage 'victoria_logs': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available

However, we see in the VictoriaLogs pod:

{"ts":"2025-02-19T16:09:47.640Z","level":"warn","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:419","msg":"remoteAddr: \"10.244.3.43:48262\"; requestURI: /insert/elasticsearch; unsupported path requested: \"/insert/elasticsearch\""}

The path appears to be correct based on the VictoriaLogs documentation, (and given that I have provided that path correctly to other compatible agents, like Vector). In addition, network connectivity is not blocked (there are no NetworkPolicies in the cluster at all) since we’re seeing errors in the VL pods from the Jaeger pods without issue.

If anyone has any thoughts on this I’d welcome them.

Troubleshooting docs

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@AndrewChubatiuk hmm, I’d recommend putting a note on the “data ingestion” page about ElasticSearch to that effect, then. It’s not clear from the docs (that I can find) that this is so.

I mean it makes logical sense, but still.

Using VL as a trace backend is difficult for now then as there’s no easy way to visualize the data unless VL and the VL plugin for Grafana is extended because as of now, if I try to use the Jaeger plugin, it will use the Jaeger Query component which of course only knows about the storage as “ElasticSearch” and can’t actually search inside it.

I can do native LogsQL from the “Traces” panel in Grafana, but I’m not sure how best to construct a meaningful query. If someone can give me an example one, I’m happy to test it.

victorialogs now only supports es data ingestion, search API is not supported

please try this one victoriametrics/victoria-logs:heads-vlinsert-extra-path-for-es-ping-request-0-ga34e578ae-scratch