gosnowflake: SNOW-1337235 SHOW has inconsistent JSON responses that can't be converted to Arrow correctly
Please answer these questions before submitting your issue. In order to accurately debug the issue this information is required. Thanks!
- What version of GO driver are you using?
1.9
- What operating system and processor architecture are you using?
Windows x64
- What version of GO are you using?
run
go versionin your console
go version go1.21.3 windows/amd64
4.Server version:* E.g. 1.90.1 You may get the server version by running a query:
SELECT CURRENT_VERSION();
8.15.0
-
What did you do?
If possible, provide a recipe for reproducing the error. A complete runnable program is good.
The issue was first noticed and logged as an ADBC issue on https://github.com/apache/arrow-adbc/issues/1454.
The problem seems to be that when a SHOW command is run that contains a large number of results, the error arrow/ipc: could not read message schema: arrow/ipc: could not read message metadata: unexpected EOF is received.
Upon digging in to this further, the behavior of the call is inconsistent. If the call is included with a LIMIT command to restrict the number of records to, say, 100, then things work as expected:
And the 100 values are returned in the JSON:
There are no Arrow record batches returned from the ArrowStreamLoader class because it is nil. The downstream calls all succeed as expected.
However, a larger number of records (of indeterminate size) in the result causes that same request to fail. For example, if limited to 700, only 562 records are returned:
And those records end with a “,” and not EOF because the JSON string isn’t loaded correctly.
Now, ArrowStreamLoader reports there is 1 record batch. This leads to downstream callers failing:
Because the string can’t be finished being read in
-
What did you expect to see?
The correct number of tables from SHOW TABLES should be sent, no matter how many there are.
-
Can you set logging to DEBUG and collect the logs?
https://community.snowflake.com/s/article/How-to-generate-log-file-on-Snowflake-connectors
-
What is your Snowflake account identifier, if any? (Optional)
I included screenshots that contain QueryIDs. Hopefully that helps.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 17 (8 by maintainers)
Commits related to this issue
- fix(go/adbc/driver/snowflake): workaround snowflake metadata-only limitations (#1790) Workaround to fix #1454 until snowflake addresses https://github.com/snowflakedb/gosnowflake/issues/1110 with a ... — committed to apache/arrow-adbc by zeroshade 3 years ago
- fix(go/adbc/driver/snowflake): workaround snowflake metadata-only limitations (#1790) Workaround to fix #1454 until snowflake addresses https://github.com/snowflakedb/gosnowflake/issues/1110 with a ... — committed to meowcraft-dev/arrow-adbc by zeroshade 3 years ago
The problem there is that it doesn’t correctly manage when the JSON data is chunked. It returns the JSON data from the initial response, and it’s still difficult to tell when you need to call
JSONDatavs using the Arrow interface since it is entirely dependent upon the query that was sent in which the caller doesn’t know. This inconsistency where metadata queries can’t return Arrow when requested is problematic because there’s no way for callers like ADBC to know if the query supported Arrow or not.You will likely run into this problem again when you do that as customers are going to almost certainly dislike the inconsistency. Personally, the preferred solution here would be to add support to return Arrow for these metadata queries.
But for now, at minimum something should be added to
ArrowStreamLoaderto explicitly be able to denote whether or not a consumer needs to useJSONDataor not and whether they should expect JSON data from the batches