trino: Kafka Avro queries fail with Trino 466

Queries for kafka avro topics started to fail with trino 466. Going through the commit history, reverting them one by one and testing locally, it seems that the failure is caused by this change: https://github.com/trinodb/trino/commit/bc9a630f664296ecb3e5cc563d5d81808fa91744#diff-35d599b3308307f88d6f91ca19ff6a4be4ae482af1f6aa9142dfe57e1697e7f2R267. Talking to @wendigo , he suggested that trino expects all data on the different sources are utf-8 encoded. I believe this is the case for my test data. These are Avro events produced using confluent schema registry. The error is on the client side (only jdbc clients, it works fine with python clients):

Error fetching results Invalid UTF-8 start byte 0xa0 at [Source: REDACTED (StreamReadFeature. INCLUDE_SOURCE_IN_LOCATION disabled); line: 1, column: 26278]

Running the query with trino 465 works fine:

767e8063-e384-4b98-adda-ee46f8bb386e,1738004817616,HISTORICAL,REAL_TIME,"{EUR_GBP=0.64, USD_GBP=0.81, USD_USD=1.00, CAD_GBP=0.71, CAD_CAD=1.00, EUR_EUR=1.00}","{id=82315712660850, profileId=82315709224408, userId=null, receivedAmount=null, invoiceValue=100.0, state=NEW, states=[], sourceCurrency=GBP, targetCurrency=EUR, currentStateId=null, targetValue=0.0, sourceValue=null, targetRecipientId=82315712903305, originalRecipient=null, refundRecipientId=null, fixedRate=null, fixRateReason=null, actualMidMktRate=null, paymentReference=null, submitTime=1738004817615, suspendTime=null, receiveTime=null, matchTime=null, cancelTime=null, transferredTime=null, effectiveRate=null, fixTypeTarget=null, receiveChannel=null, fee=null, compensatedFee=null, feeType=null, paymentPurpose=null, senderName=null, senderEmail=null, transferRequestId=null, incomingPartnerPaymentId=null, securityCode=null, securityHash=null, useSecurityCode=null, linkingHash=null, offeredDeliveryTime=null, quoteId=null, quoteUuid=null, creationUuid=null, fees=[], matchedSourceValue=null, matchedTargetValue=null, discount=null, clientKey=null, attributes=[], externalReferences=[]}",,{},{},{},,c1b7eab8-2b86-41e0-a66c-9bbb7e199e4c,2,0,false,    H767e8063-e384-4b98-adda-ee46f8bb386e��ǒ�e  EUR_GBP0.64EUR_EUR1.00USD_GBP0.81USD_USD1.00CAD_GBP0.71CAD_CAD1.00 䕀�%���鳷%        Y@NEW GBPEUR          ���%      ��ǒ�e                                    Hc1b7eab8-2b86-41e0-a66c-9bbb7e199e4c,{},282,false,82315712660850,14,2025-01-27 19:06:58.336

And checking the event itself using: kafkacat -C -b localhost:29092 -t north-am-pending-transfer-summary-v2-StreamingEngine.compacted.v4.transfers -o 0 -p 2 -c1 | hexdump -C

Produces:

00000010  65 33 38 34 2d 34 62 39  38 2d 61 64 64 61 2d 65  |e384-4b98-adda-e|
00000020  65 34 36 66 38 62 62 33  38 36 65 02 a0 9b c7 92  |e46f8bb386e.....|
00000030  95 65 00 00 02 0c 0e 45  55 52 5f 47 42 50 08 30  |.e.....EUR_GBP.0|
00000040  2e 36 34 0e 45 55 52 5f  45 55 52 08 31 2e 30 30  |.64.EUR_EUR.1.00|
00000050  0e 55 53 44 5f 47 42 50  08 30 2e 38 31 0e 55 53  |.USD_GBP.0.81.US|
00000060  44 5f 55 53 44 08 31 2e  30 30 0e 43 41 44 5f 47  |D_USD.1.00.CAD_G|
00000070  42 50 08 30 2e 37 31 0e  43 41 44 5f 43 41 44 08  |BP.0.71.CAD_CAD.|
00000080  31 2e 30 30 00 02 e4 95  80 ed b3 b7 25 b0 d7 dc  |1.00........%...|
00000090  e9 b3 b7 25 00 00 02 00  00 00 00 00 00 59 40 02  |...%.........Y@.|
000000a0  06 4e 45 57 00 02 06 47  42 50 02 06 45 55 52 00  |.NEW...GBP..EUR.|
000000b0  02 00 00 00 00 00 00 00  00 00 02 92 e2 9d ed b3  |................|
000000c0  b7 25 00 00 00 00 00 00  02 9e 9b c7 92 95 65 00  |.%............e.|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000f0  00 02 00 00 02 48 63 31  62 37 65 61 62 38 2d 32  |.....Hc1b7eab8-2|
00000100  62 38 36 2d 34 31 65 30  2d 61 36 36 63 2d 39 62  |b86-41e0-a66c-9b|
00000110  62 62 37 65 31 39 39 65  34 63 0a                 |bb7e199e4c.|
0000011b

My understanding is that the schema registry producer embeds some “control bits” within the event payload, but I am not sure this is something to worry about. Any thoughts @hashhar ?

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

This is the same code. This will work in JDBC too

@akhawatrahTW This ignores improperly encoded utf-8 strings. You can still wrap the column with base64_encode and get it this way