telegraf: Avro processor doesnt support Unions

Use Case

Telegraf Avro processor doesnt support unions, here is the sample schema and encoded string.

schemaJSON := `
	{
		"type": "record",
		"name": "test",
		"fields": [
			{"name": "field1", "type": "int"},
			{"name": "field2", "type": ["null", "string"]}
		]
	}`

// Example Avro message with a union type messageBytes := []byte(“\x02\x0Ahello”)

Expected behavior

Avro processor to support unions

Actual behavior

Avro processor failed with error " Could not convert map[string:abc] to string for tag “field2”: type “map[string]interface {}” unsupported".

https://github.com/influxdata/telegraf/blob/master/plugins/parsers/avro/parser.go#L143

the conversion should handle type map[string]interface{}.

Additional info

https://github.com/influxdata/telegraf/pull/11816

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 18 (18 by maintainers)

Most upvoted comments

I’ve been talking to my colleague @afausti about what the desired behavior is, and we’re getting pretty close to being ready to start an implementation. (The reason for this weird-from-the-outside division of labor is that Angelo understands what we’re doing with Kafka and Avro way better than I do, but I’m the person with the most Golang experience on our team.)

athornton on Jun 21, 2023

Did some work on this today. I didn’t like skip–I turned that into flatten which is more descriptive of what happens. It’s too bad that the test suite only lets you push a single metric at a time (and I am not feeling ambitious enough to rewrite it), but I have single-metric test cases for nullable and any.

Currently at https://github.com/lsst-sqre/telegraf/tree/features/avro-unions-redux and once I get it built on Linux (there’s a bizarre dependency problem on Amazon Linux that I don’t have on Mac, but I can’t jam a Darwin binary into a Linux container and have it run, obv) so I can test in our containerized environment, I’ll PR it.

athornton on Aug 30, 2023

OK. I have (it’s still being tested) a fix for this, because this is something we want at Rubin too. It’s not ready to be PRed yet but if you want to look at my fork/branch, please do, and also please feel free to try docker.io/lsstsqre/telegraf:avrounions .

The fork/branch is at https://github.com/lsst-sqre/telegraf/tree/features/avro-unions-redux

The option you will want to use is avro_union_mode = "nullable"

athornton on Sep 12, 2023

If you are referring to the goavro Union function where it returns nil: that only happens if the name passed in is “null” and the datum, the actual data, is nil (not null) (e.g. goavro.Union("null", nil)).

powersj on May 25, 2023