beyla: Beyla high memory usage?

Hello, this is a great project!

I’ve been trying Beyla 1.2.1 out on my k8s cluster using k8s service discovery. I noticed a fairly high amount of usage of beyla itself per instrumented process, when I specified many containers to be instrumented.

In specific, when I configured beyla to instrument about 25 containers, it was using about 739 MiB of memory.

Logs
time=2024-02-10T03:36:33.844Z level=INFO msg="Grafana Beyla" Version=992041d5 "OpenTelemetry SDK Version"=1.18.0
time=2024-02-10T03:36:33.958Z level=INFO msg="using hostname" component=traces.ReadDecorator function=instance_ID_hostNamePIDDecorator hostname=beyla-skr24
time=2024-02-10T03:36:33.962Z level=INFO msg="Starting main node" component=beyla.Instrumenter
time=2024-02-10T03:36:36.492Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/app/ns pid=3037425
time=2024-02-10T03:36:36.492Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/app/traQ pid=20002
time=2024-02-10T03:36:36.493Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/bin/caddy pid=16460
time=2024-02-10T03:36:36.493Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/share/elasticsearch/jdk/bin/java pid=7543
time=2024-02-10T03:36:36.493Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/lib/jvm/java-17-amazon-corretto/bin/java pid=17390
time=2024-02-10T03:36:36.493Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/bin/caddy pid=3037465
time=2024-02-10T03:36:36.493Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3037224 child=[] exec=/app/ns
time=2024-02-10T03:36:36.493Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/bin/caddy pid=16441
time=2024-02-10T03:36:36.494Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-aarch64/bin/controller pid=15976
time=2024-02-10T03:36:36.494Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3036389 child=[] exec=/app/ns
time=2024-02-10T03:36:36.494Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/bin/caddy pid=16715
time=2024-02-10T03:36:36.494Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/bin/busybox pid=16390
time=2024-02-10T03:36:36.494Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3036382 child=[] exec=/app/ns
time=2024-02-10T03:36:36.494Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/local/bin/node pid=15435
time=2024-02-10T03:36:36.495Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3035857 child=[] exec=/usr/bin/caddy
time=2024-02-10T03:36:36.495Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/sbin/tini pid=15377
time=2024-02-10T03:36:36.495Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/ns-bin/ns pid=3036872
time=2024-02-10T03:36:36.495Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3036268 child=[] exec=/app/ns
time=2024-02-10T03:36:36.495Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=16276 child=[] exec=/usr/lib/jvm/java-17-amazon-corretto/bin/java
time=2024-02-10T03:36:36.496Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/sbin/tini pid=15308
time=2024-02-10T03:36:36.496Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/bin/tini pid=7325
time=2024-02-10T03:36:36.496Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3036726 child=[] exec=/usr/bin/caddy
time=2024-02-10T03:36:36.496Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=15437 child=[] exec=/usr/local/bin/node
time=2024-02-10T03:36:36.497Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=7543 child=[15142] exec=/usr/share/elasticsearch/jdk/bin/java
time=2024-02-10T03:36:36.497Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=3036242 child=[] exec=/app/ns
time=2024-02-10T03:36:36.614Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:36.876Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:37.073Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:37.542Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:37.726Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:37.985Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T03:36:38.562Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
W0210 03:46:33.954207 2395129 reflector.go:456] k8s.io/client-go/informers/factory.go:150: watch of *v1.Pod ended with: couldn't queue object: was expecting a Pod. Got: *kube.PodInfo

image

When I limited beyla to instrument only 1 container, it was using only about 177 MiB of memory.

Logs
time=2024-02-10T04:05:44.128Z level=INFO msg="Grafana Beyla" Version=992041d5 "OpenTelemetry SDK Version"=1.18.0
time=2024-02-10T04:05:44.235Z level=INFO msg="using hostname" component=traces.ReadDecorator function=instance_ID_hostNamePIDDecorator hostname=beyla-6mf5s
time=2024-02-10T04:05:44.236Z level=INFO msg="Starting main node" component=beyla.Instrumenter
time=2024-02-10T04:05:45.464Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/app/traQ pid=20002
time=2024-02-10T04:05:45.584Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
time=2024-02-10T04:13:17.733Z level=ERROR msg="closing traces provider" component=otel.TracesReporter !BADKEY="context canceled"
time=2024-02-10T04:13:17.734Z level=ERROR msg="closing traces exporter" component=otel.TracesReporter error="context canceled"
time=2024-02-10T04:13:17.734Z level=INFO msg="exiting auto-instrumenter" component=beyla.Instrumenter

image

My configurations are here, just fyi: daemon-set.yaml, beyla config

So my question is: how much memory is beyla expected to consume (per instrumented process)? And, is this memory comsuption in expected range? Also I think it would be a good idea to add some notes about expected memory consumption in the docs, as I couldn’t find it.

Please tell me if I am missing something to look for or configure. Thank you!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (11 by maintainers)

Most upvoted comments

If you open a new issue it has a template asking for what we need to debug better. If what you provided is not enough, we can look closer afer that.

Thanks again @motoki317, we are still investigating this, we’ll come back here with a fix.

Hi @motoki317, thanks for reporting this, it does seem excessive. We’ll look into this next week.