gosnowflake: SNOW-1337235 SHOW has inconsistent JSON responses that can't be converted to Arrow correctly

Please answer these questions before submitting your issue. In order to accurately debug the issue this information is required. Thanks!

  1. What version of GO driver are you using?

1.9

  1. What operating system and processor architecture are you using?

Windows x64

  1. What version of GO are you using? run go version in your console

go version go1.21.3 windows/amd64

4.Server version:* E.g. 1.90.1 You may get the server version by running a query:

SELECT CURRENT_VERSION();

8.15.0

  1. What did you do?

    If possible, provide a recipe for reproducing the error. A complete runnable program is good.

The issue was first noticed and logged as an ADBC issue on https://github.com/apache/arrow-adbc/issues/1454.

The problem seems to be that when a SHOW command is run that contains a large number of results, the error arrow/ipc: could not read message schema: arrow/ipc: could not read message metadata: unexpected EOF is received.

Upon digging in to this further, the behavior of the call is inconsistent. If the call is included with a LIMIT command to restrict the number of records to, say, 100, then things work as expected:

image

And the 100 values are returned in the JSON:

image

There are no Arrow record batches returned from the ArrowStreamLoader class because it is nil. The downstream calls all succeed as expected.

However, a larger number of records (of indeterminate size) in the result causes that same request to fail. For example, if limited to 700, only 562 records are returned:

image

image

And those records end with a “,” and not EOF because the JSON string isn’t loaded correctly.

Now, ArrowStreamLoader reports there is 1 record batch. This leads to downstream callers failing:

image

Because the string can’t be finished being read in

image

  1. What did you expect to see?

    The correct number of tables from SHOW TABLES should be sent, no matter how many there are.

  2. Can you set logging to DEBUG and collect the logs?

    https://community.snowflake.com/s/article/How-to-generate-log-file-on-Snowflake-connectors

  3. What is your Snowflake account identifier, if any? (Optional)

I included screenshots that contain QueryIDs. Hopefully that helps.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 17 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Ad 1. ArrowStreamLoader which is a return value of QueryArrowStream has a function JSONData() [][]*string. Maybe you can decide on these values? If not, we can add additional function returning some info about types.

The problem there is that it doesn’t correctly manage when the JSON data is chunked. It returns the JSON data from the initial response, and it’s still difficult to tell when you need to call JSONData vs using the Arrow interface since it is entirely dependent upon the query that was sent in which the caller doesn’t know. This inconsistency where metadata queries can’t return Arrow when requested is problematic because there’s no way for callers like ADBC to know if the query supported Arrow or not.

Ad 2. Not yet, but we are planing to add this feature to remaining drivers probably later this year. Customers ask for it.

You will likely run into this problem again when you do that as customers are going to almost certainly dislike the inconsistency. Personally, the preferred solution here would be to add support to return Arrow for these metadata queries.

But for now, at minimum something should be added to ArrowStreamLoader to explicitly be able to denote whether or not a consumer needs to use JSONData or not and whether they should expect JSON data from the batches