serilog-sinks-grafana-loki: Stops send log after Loki Error 'entry out of order' (Http status 400)

When our application has some load (a lot of broker messages to consume - using Rebus worker threads), the sink suddenly stops sending logs to Loki.

To investigate, we configured the application to have 2 sinks enabled: console and this sink, and Promtail configured to send console log to Loki. We found this line on Loki log:

level=warn ts=2021-08-17T21:57:55.768341371Z caller=grpc_logging.go:38 method=/logproto.Pusher/Push duration=1.39409ms err="rpc error: code = Code(400) desc = entry with timestamp 2021-08-17 21:56:54.73 +0000 UTC ignored, reason: 'entry out of order' for stream: {SourceContext=\"Rebus.Retry.ErrorTracking.InMemErrorTracker\", app=\"myapp\", container=\"myapp\", errorNumber=\"1\", messageId=\"52xxb042-xxaa-xxxx-813a-31fd6810bb25\", namespace=\"default\", node_name=\"aks-agentpool\", pod=\"myapp-7bd5d4fbcb-gjvxm\"},\ntotal ignored: 1 out of 4" msg="gRPC\n"

After this error, the sink does not recover and stops sending logs to Loki.

A similar problem (entry out of order) is discussed here)

We are using version 6.0.1

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (17 by maintainers)

Most upvoted comments

Loki is reporting (one pod):

Staging: Max 271 log lines per second Production: Max 901 log lines per second

Thank you, @raphaelquati!

I have national holidays in my country, so I will return to investigation next week, after 25 of August