trino: Kafka Avro queries fail with Trino 466
Queries for kafka avro topics started to fail with trino 466. Going through the commit history, reverting them one by one and testing locally, it seems that the failure is caused by this change: https://github.com/trinodb/trino/commit/bc9a630f664296ecb3e5cc563d5d81808fa91744#diff-35d599b3308307f88d6f91ca19ff6a4be4ae482af1f6aa9142dfe57e1697e7f2R267. Talking to @wendigo , he suggested that trino expects all data on the different sources are utf-8 encoded. I believe this is the case for my test data. These are Avro events produced using confluent schema registry. The error is on the client side (only jdbc clients, it works fine with python clients):
Error fetching results Invalid UTF-8 start byte 0xa0 at [Source: REDACTED (
StreamReadFeature. INCLUDE_SOURCE_IN_LOCATIONdisabled); line: 1, column: 26278]
Running the query with trino 465 works fine:
And checking the event itself using:
kafkacat -C -b localhost:29092 -t north-am-pending-transfer-summary-v2-StreamingEngine.compacted.v4.transfers -o 0 -p 2 -c1 | hexdump -C
Produces:
00000010 65 33 38 34 2d 34 62 39 38 2d 61 64 64 61 2d 65 |e384-4b98-adda-e|
00000020 65 34 36 66 38 62 62 33 38 36 65 02 a0 9b c7 92 |e46f8bb386e.....|
00000030 95 65 00 00 02 0c 0e 45 55 52 5f 47 42 50 08 30 |.e.....EUR_GBP.0|
00000040 2e 36 34 0e 45 55 52 5f 45 55 52 08 31 2e 30 30 |.64.EUR_EUR.1.00|
00000050 0e 55 53 44 5f 47 42 50 08 30 2e 38 31 0e 55 53 |.USD_GBP.0.81.US|
00000060 44 5f 55 53 44 08 31 2e 30 30 0e 43 41 44 5f 47 |D_USD.1.00.CAD_G|
00000070 42 50 08 30 2e 37 31 0e 43 41 44 5f 43 41 44 08 |BP.0.71.CAD_CAD.|
00000080 31 2e 30 30 00 02 e4 95 80 ed b3 b7 25 b0 d7 dc |1.00........%...|
00000090 e9 b3 b7 25 00 00 02 00 00 00 00 00 00 59 40 02 |...%.........Y@.|
000000a0 06 4e 45 57 00 02 06 47 42 50 02 06 45 55 52 00 |.NEW...GBP..EUR.|
000000b0 02 00 00 00 00 00 00 00 00 00 02 92 e2 9d ed b3 |................|
000000c0 b7 25 00 00 00 00 00 00 02 9e 9b c7 92 95 65 00 |.%............e.|
000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000000f0 00 02 00 00 02 48 63 31 62 37 65 61 62 38 2d 32 |.....Hc1b7eab8-2|
00000100 62 38 36 2d 34 31 65 30 2d 61 36 36 63 2d 39 62 |b86-41e0-a66c-9b|
00000110 62 62 37 65 31 39 39 65 34 63 0a |bb7e199e4c.|
0000011b
My understanding is that the schema registry producer embeds some “control bits” within the event payload, but I am not sure this is something to worry about. Any thoughts @hashhar ?
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (7 by maintainers)
This is the same code. This will work in JDBC too
@akhawatrahTW This ignores improperly encoded utf-8 strings. You can still wrap the column with base64_encode and get it this way