bee: Bee API stops to return data on GET requests without evident reason

Context

Using Bee 1.5.1 with Docker.

Summary

I’ve a node that at some point started to timeout get requests, even if content should be cached.

This is an extract of logs:

time="2022-04-27T15:59:58Z" level=info msg="api access" duration=0.013540011 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=96387 status=200 uri=/bzz/a1384bd71fcecf3a706d777075b56e3d6063dd876bba5617c985ba4eff3311e3/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.145, 10.0.0.74"
time="2022-04-27T16:00:08Z" level=info msg="api access" duration=1.003372393 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=93 status=200 uri="/feeds/F60Fbc6A72dde156f24A08C414197AE3CF5399E0/40daa16a08bb37fc91dcfb3402af8193e327e741209d6f86a8127cbd4b1befaa?type=sequence" user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.35.201, 10.0.0.74"
time="2022-04-27T16:00:09Z" level=info msg="api access" duration=0.01309673 ip=10.0.8.11 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=96387 status=200 uri=/bzz/a1384bd71fcecf3a706d777075b56e3d6063dd876bba5617c985ba4eff3311e3/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.145, 10.0.0.73"
time="2022-04-27T16:00:22Z" level=info msg="api access" duration=2.008228539 ip=10.0.8.11 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=93 status=200 uri="/feeds/3270fF5B8a32e1E39b12311ea5a9Eab325425137/40daa16a08bb37fc91dcfb3402af8193e327e741209d6f86a8127cbd4b1befaa?type=sequence" user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.98, 10.0.0.74"
time="2022-04-27T16:00:26Z" level=info msg="api access" duration=60.005563862 ip=10.0.8.11 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=23 status=200 uri=/bzz/5d2a835a77269dc7bb1fb6be7b12407326cf6dcde4bd14f41b92be9d82414421/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.142, 10.0.0.73"
time="2022-04-27T16:00:26Z" level=info msg="api access" duration=60.00241233 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=0 status=206 uri=/bzz/d88f68aa5b157ce6bda355d8bd54179df264a899c03bf5bdf0d4569f20a6933b/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.148, 10.0.0.74"
time="2022-04-27T16:00:41Z" level=error msg="bzz download: invalid path"
time="2022-04-27T16:00:41Z" level=info msg="api access" duration=60.003016437 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=60 status=404 uri=/bzz/7eb77f7d0c2d17d9e05036f154b4d26091ba3e7d0ccfe8ebda49cda2bb94cd9b/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.101, 10.0.0.74"
time="2022-04-27T16:00:47Z" level=info msg="api access" duration=60.003261465 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=0 status=206 uri=/bzz/cf492b4f50da74f9789d02ad32d741d8642b77e76194a925f8ee0dd1450d3990/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.95, 10.0.0.74"
time="2022-04-27T16:01:20Z" level=error msg="feed get: lookup error"
time="2022-04-27T16:01:20Z" level=info msg="api access" duration=60.005268886 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=64 status=404 uri="/feeds/73963b959078051daB5535029FCB7AAcfc87Cf9E/40daa16a08bb37fc91dcfb3402af8193e327e741209d6f86a8127cbd4b1befaa?type=sequence" user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.134, 10.0.0.74"
time="2022-04-27T16:01:26Z" level=info msg="api access" duration=60.002606399 ip=10.0.8.11 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=23 status=200 uri=/bzz/5d2a835a77269dc7bb1fb6be7b12407326cf6dcde4bd14f41b92be9d82414421/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.142, 10.0.0.73"
time="2022-04-27T16:01:26Z" level=info msg="api access" duration=60.001224132 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=0 status=206 uri=/bzz/d88f68aa5b157ce6bda355d8bd54179df264a899c03bf5bdf0d4569f20a6933b/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.148, 10.0.0.73"
time="2022-04-27T16:01:26Z" level=info msg="api access" duration=60.001309699 ip=10.0.8.247 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=23 status=200 uri=/bzz/5d2a835a77269dc7bb1fb6be7b12407326cf6dcde4bd14f41b92be9d82414421/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.142, 10.0.0.74"
time="2022-04-27T16:01:26Z" level=info msg="api access" duration=59.996773946 ip=10.0.8.11 method=GET proto=HTTP/1.0 referrer="https://app.etherna.io/" size=0 status=206 uri=/bzz/d88f68aa5b157ce6bda355d8bd54179df264a899c03bf5bdf0d4569f20a6933b/ user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" x-forwarded-for="185.159.157.9, 151.139.95.148, 10.0.0.73"

You can see that logs reports duration that started from low values, and without any evident reason pass to ~60 seconds, where the network components rise a timeout. It’s not a network fault, because a simple bee restart solved the issue.

What is strange is that the status is anyway often a 2xx code, but data is not returned, and connection is closed with the timeout.

Expected behavior

API should properly return data, specially with a success result code.

Actual behavior

API stops to return data, even if it should be available, and connections are closed by the timeout exception that drops it.

Steps to reproduce

Don’t know, it’s random.

Possible solution

Don’t know.