benthos: Can't output from kinesis-balanced or kinesis

I have the following config and benthos 3.11.0:

input: kinesis_balanced: stream: what-a-great-stream start_from_oldest: false dynamodb_table: table-for-benthos dynamodb_billing_mode: PAY_PER_REQUEST region: us-east-1 credentials: profile: what-a-great-profile output: file: path: outfile.txt delimiter: ""

Using the same AWS profile, on us-east-1, I’m able to put records on this stream, get records from it, and when I scan dynamodb table-for-benthos I see leases are being updated by benthos as long as it’s running. The file, outfile.txt is getting created, and if I swap the input to stdin it works.

I’m not getting any errors, and I’m not getting any output from kinesis.

I’m so close to living the dream, I can taste it.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (18 by maintainers)

Most upvoted comments

Really, we shouldn’t create any resources by default as it has the potential to cause unexpected billing. We ought to just check whether dynamodb_table is specified and error out on start up if it isn’t, similar to the stream field.

@Jeffail absolutely