fluent-bit: Cannot configure Stackdriver output plugin

Bug Report

Describe the bug I have followed the configuration guide for Stackdriver in the manual, but have had no success in establishing a connection to Stackdriver.

To Reproduce

Install fluent-bit on an Ubuntu 16.04 LTS box
Create a service account following Google’s instructions and copy the JSON key into /etc/google/auth/

Modify /etc/td-agent-bit/td-agent-bit.conf to include:

[OUTPUT]
    Name  stackdriver
    Match *
    google_service_credentials /etc/google/auth/________.json

Restart the agent to reload the configuration via systemctl restart td-agent-bit.service

Note that the authorisation phase of connecting to Stackdriver fails via systemctl status td-agent-bit.service:

Sep 11 02:24:10 hostname td-agent-bit[16981]: [2018/09/11 02:24:10] [ info] [engine] started (pid=16981)
Sep 11 02:24:10 hostname td-agent-bit[16981]: [2018/09/11 02:24:10] [error] [oauth2] could not get an upstream connection
Sep 11 02:24:10 hostname td-agent-bit[16981]: [2018/09/11 02:24:10] [error] [out_stackdriver] error retrieving oauth2 access token
Sep 11 02:24:10 hostname td-agent-bit[16981]: [2018/09/11 02:24:10] [ warn] [out_stackdriver] token retrieval failed

Expected behavior I expected authentication to succeed against Stackdriver.

Your Environment

Version used: 0.14.1
Configuration: Default configuration but with the [OUTPUT] section as described above. I had to comment out the Plugins_File plugins.conf line as this file does not exist by default and I couldn’t find any documentation on the intended contents of such a file. (I also attempted putting the [OUTPUT] config for stackdriver into this file, as well as just leaving the file blank)
Environment name and version (e.g. Kubernetes? What version?): Our own VPS
Server type and version: N/A
Operating System and version: Ubuntu 16.04.3 LTS
Filters and plugins: Default configuration; no filters, just the stackdriver output plugin.

Additional context I’m trying to use fluent-bit to consume and send through server stats from a VPS we have, that is not part of our Google Cloud cluster.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 40 (15 by maintainers)

Commits related to this issue

out_bigquery: enable 'ipv6 mode' support (#761) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 5 years ago
out_bigquery: enable 'ipv6 mode' support (#761) Signed-off-by: Eduardo Silva <eduardo@treasure-data.com> — committed to fluent/fluent-bit by edsiper 5 years ago
linux: update GPG keys for 1.8.15 refs (#761) Signed-off-by: Patrick Stephens <pat@calyptia.com> — committed to rawahars/fluent-bit by patrick-stephens 2 years ago

Most upvoted comments

thanks everyone for the report, I’ve added ipv6 mode to out_bigquery on 466191c3

edsiper on Jul 19, 2019

+1, I am hit by this too. I get a 200 when I do the curl with the JWT token copied from the logs, and the same oauth error from fluentbit logs.

sudharsh on Jan 29, 2019

@varun-da Just in response to your own reply before, definitely understand that it is a likely cause, but the first thing we checked off in this issue was connectivity from the box to those two addresses. I can confirm I still have connectivity.

I’m still hitting the issue, though:

Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [ info] [engine] started (pid=15149)
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [debug] [out_stackdriver] JWT signature:
Jan 09 09:00:15 hostname td-agent-bit[15149]: removed
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [error] [oauth2] could not get an upstream connection
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [error] [out_stackdriver] error retrieving oauth2 access token
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [ warn] [out_stackdriver] token retrieval failed
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [debug] [router] match rule cpu.0:stdout.0
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [debug] [router] match rule cpu.0:stackdriver.0
Jan 09 09:00:15 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
Jan 09 09:00:16 hostname td-agent-bit[15149]: [2019/01/09 09:00:15] [debug] [input cpu.0] [mem buf] size = 317

Cheers for the assistance!

theFroh on Jan 9, 2019