beyla: Beyla failing to capture HTTP calls

We are using Beyla to instrument a Java based application which is receiving calls from outside. The application is acting as a server and it gets called by a client (through ingress-nginx). The results have some discrepancies in them, because only a very small fraction of the calls the client is sending to the application are being captured by Beyla. We have tried running Beyla with BEYLA_PRINT_TRACES=true to log every single call/trace and this does not seem to show the missing calls - nor are they present as Prometheus metrics.

Beyla configuration

routes:
  unmatched: heuristic
discovery:
  services:
    - exe_path: .*(zulu|elasticsearch).*

Service discovery does not seem to be the problem, as some calls are present - therefore the executable is correctly found and instrumented. Other than the Beyla logs missing the relevant traces, there are no abnormalities in the logs.

Here’s an example of calls from the application (from the client-side logs) vs. what is captured by Beyla (Beyla application logs).

Client Call Beyla Log
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.system.SystemService/ping POST /pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.system.SystemService/ping
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.security.AuthenticationService/authenticate
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/unlockSessionRequest
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.Auth.AuthService/isAuthAuthenticationEnabled
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.PosTransactionService/getOrCreateTransaction
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.PosTransactionService/removeBinaryDataFromTransaction POST /pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.PosTransactionService/removeBinaryDataFromTransaction
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.customer.CustomerTransactionService/removeCustomerFromTransactionByTypeCode POST /pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.customer.CustomerTransactionService/removeCustomerFromTransaction
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.api.service.StartUtilsService/isCustomerVerificationRequired POST /pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.api.service.StartUtilsService/isCustomerVerificationRequired
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.PosTransactionService/createAndAdransactionBinaryData POST /pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.transaction.PosTransactionService/createAndAdransactionBinaryData
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.service.pharmacy.PharmacyService/isPrescription POST /pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.service.pharmacy.PharmacyService/isPrescription
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.common.CommonService/updateTransactionExtensions POST /pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.common.CommonService/updateTransactionExtensions
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.masterdata.item.ItemQueryService/loadItemDOByPosItemId POST /pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.masterdata.item.ItemQueryService/loadItemDOByPosItemId
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.work_order.WorkOrderService/createWorkOrder POST /pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.work_order.WorkOrderService/createWorkOrder
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.api.service.StartUtilsService/adransactionExtensionToCurrentTransaction POST /pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.api.service.StartUtilsService/adransactionExtensionToCurrentTransaction
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.system.SystemService/simplePing
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.plugin.start.api.service.StartUtilsService/getStartConfig
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.cst.company.pos.api.service.Auth.AuthService/getAuthServiceConfig
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/setLastServerStatus
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.masterdata.MasterDataService/getBusinessUnit
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/createAnonymousLogin
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.masterdata.MasterDataService/getDefaultCurrencyForBusinessUnit
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/setPosSessionData
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/setLanguageSettings
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/logoutIfNecessary
https://application.example.com:443/pos-service/tenants/001/stores/1003/services/com.company.pos.api.service.session.PosSessionService/logoutIfAnonymous

There seems to be no pattern, neither in the length of the path nor in its content to explain why certain calls are not captured. Interestingly, on subsequent runs of the same test we also get different results, meaning a call that was not captured before is now suddenly present and vice versa.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (13 by maintainers)

Most upvoted comments

Thanks again Martijn! I put out a PR to allow us to load the debug BPF on 5.10. I’ll also repeat my tests on 5.10 to see if that makes a difference, perhaps the missing events issue is related to lower kernel versions…