fluent-bit: DNS query cancelled error on v1.8.3 when sending to http-intake.logs.datadoghq.eu (works in v1.8.2)
Bug Report
Describe the bug After upgrading to v1.8.3 td-agent-bit does not work when sending logs to http-intake.logs.datadoghq.eu
To Reproduce
- Example log message if applicable:
[2021/08/12 12:08:16] [ info] [engine] started (pid=662)
[2021/08/12 12:08:16] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/08/12 12:08:16] [debug] [storage] [cio stream] new stream registered: tail.0
[2021/08/12 12:08:16] [ info] [storage] version=1.1.1, initializing...
[2021/08/12 12:08:16] [ info] [storage] in-memory
[2021/08/12 12:08:16] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/08/12 12:08:16] [ info] [cmetrics] version=0.1.6
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] inotify watch fd=24
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] scanning path /f/src/json_logs/*.log
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] inode=3280056 with offset=3050 appended as /f/src/json_logs/ml-training.log
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] scan_glob add(): /fx.log, inode 3280056
[2021/08/12 12:08:16] [debug] [input:tail:tail.0] 1 new files found on path ‘/f/*.log'
[2021/08/12 12:08:16] [debug] [datadog:datadog.0] created event channels: read=26 write=27
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] scheme: https://
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] api_key: xxxxxx
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] uri: /v1/input/xxxxx
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] host: http-intake.logs.datadoghq.eu
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] port: 443
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] json_date_key: timestamp
[2021/08/12 12:08:18] [debug] [output:datadog:datadog.0] compress_gzip: 1
[2021/08/12 12:08:18] [debug] [router] match rule tail.0:datadog.0
[2021/08/12 12:08:18] [ info] [sp] stream processor started
[2021/08/12 12:08:18] [debug] [input:tail:tail.0] inode=3280056 file=/f/x.log promote to TAIL_EVENT
[2021/08/12 12:08:18] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3280056 watch_fd=1 name=/f/x.log
[2021/08/12 12:08:53] [debug] [input:tail:tail.0] inode=3280056 events: IN_MODIFY
[2021/08/12 12:08:57] [debug] [task] created task=0x7f9de9637a00 id=0 OK
[2021/08/12 12:08:57] [ warn] [net] getaddrinfo(host='http-intake.logs.datadoghq.eu', err=24): DNS query cancelled
[2021/08/12 12:08:57] [debug] [upstream] connection #-1 failed to http-intake.logs.datadoghq.eu:443
[2021/08/12 12:08:57] [debug] [out coro] cb_destroy coro_id=0
[2021/08/12 12:08:57] [debug] [retry] new retry created for task_id=0 attempts=1
[2021/08/12 12:08:57] [ warn] [engine] failed to flush chunk '662xxxx.flb', retry in 10 seconds: task_id=0, input=tail.0 > output=datadog.0 (out_id=0)
[2021/08/12 12:09:07] [ warn] [net] getaddrinfo(host='http-intake.logs.datadoghq.eu', err=24): DNS query cancelled
[2021/08/12 12:09:07] [debug] [upstream] connection #-1 failed to http-intake.logs.datadoghq.eu:443
[2021/08/12 12:09:07] [debug] [out coro] cb_destroy coro_id=1
[2021/08/12 12:09:07] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2021/08/12 12:09:07] [ warn] [engine] chunk ‘xxx.flb' cannot be retried: task_id=0, input=tail.0 > output=datadog.0
[2021/08/12 12:09:07] [debug] [task] destroy task=0x7f9de9637a00 (task_id=0)
- Steps to reproduce the problem: RUN apt-get -y install td-agent-bit /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/td-agent-bit.conf --log_file=/tmp/fluentbit.log
Expected behavior The logs in the logfile should be send to datadog
Your Environment
- Version used: 1.8.3
- Environment name and version : Docker version 19.03.13-ce
- Server type and version: Docker on AWS Linux image
- Filters and plugins: none
Additional context This error appeared after i updated to version 1.8.3, and if i go back to 1.8.2 it works again Might only be related to datadog eu
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 7
- Comments: 16 (6 by maintainers)
@farvour/everyone this has been fixed in 1.8.7: https://fluentbit.io/announcements/v1.8.7/
I will close this issue. Please re-open if needed.
@farvour my testing suggested it works in 1.8.4 or lower: https://github.com/fluent/fluent-bit/issues/4050#issuecomment-918719115
@edsiper there are multiple reports from AWS and non-AWS users that DNS resolution is still broken in some cases. IMO, this is a critical issue that deserves to be a top priority.