gosnowflake: SNOW-1896153: Non deterministic errors with concurrent context aware queries in v1.12.1

  1. What version of GO driver are you using?

1.12.1 and 1.12.0

  1. What operating system and processor architecture are you using?

Debian Bullseye x86

  1. What version of GO are you using?

go1.23.3

  1. Server version:* E.g. 1.90.1

9.1.0

  1. What did you do?

We are loading data into snowflake via the golang data. Our strategy is to use PUT... SQL commands to upload parquet files to an internal stage. then we use a COPY INTO... statement to publish the data. We leverage the database/sql golang abstraction . So generally our code will look like

package example

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "sync"
    "time"

    _ "github.com/snowflakedb/gosnowflake" // Snowflake driver
)

func UploadFilesAndCopy(db *sql.DB, stageName, tableName string, files []string) error {
    // Create a context that will be used for all operations
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()

    // A channel that will hold the file paths we want to upload
    fileCh := make(chan string)

    // We'll use a WaitGroup to ensure all goroutines finish
    var wg sync.WaitGroup

    // Start a fixed number of worker goroutines to process the file uploads
    workerCount := 5
    for i := 0; i < workerCount; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for file := range fileCh {
                putQuery := fmt.Sprintf(
                    "PUT file://%s @%s AUTO_COMPRESS=TRUE OVERWRITE=TRUE",
                    file, stageName,
                )
                if _, err := db.ExecContext(ctx, putQuery); err != nil {
                    log.Printf("failed to PUT file %q to stage %q: %v", file, stageName, err)
                }
            }
        }()
    }

    // Send file names into the channel for workers to pick up
    go func() {
        defer close(fileCh) // Close once we're done sending
        for _, file := range files {
            fileCh <- file
        }
    }()

    // Wait for all workers to finish uploading
    wg.Wait()

    // Now that all files are in the stage, run the COPY INTO command
    copyQuery := fmt.Sprintf(`COPY INTO %s FROM @%s FILE_FORMAT = (TYPE = PARQUET)`, tableName, stageName)
    if _, err := db.ExecContext(ctx, copyQuery); err != nil {
        return fmt.Errorf("failed to COPY INTO %s: %w", tableName, err)
    }

    return nil
}

We’re experiencing issues with both 1.12.0 and 1.12.1. On 1.12.0, if context cancellation occurs, other running queries will fail with:

level=error msg="error: 000605: Identified SQL statement is not currently executing." func="gosnowflake.(*snowflakeConn).queryContextInternal" file="connection.go:410"

On 1.12.1 this goes away and we get the correct "context canceled" error message. However, we start experiencing non deterministic errors where the PUT commands will sometimes return an error error: 000605: Identified SQL statement is not currently executing. No context has been canceled in this case.

Going through the release notes and looking at the PRs for what changed, it seems like the #1248 may have introduced some kind of data race into the driver. We were able to get these errors to stop happening across our fleet by not sharing a context between the goroutines and creating a new child context for each spawned worker.

  1. What did you expect to see?

I expect the PUT queries to work concurrently as they did on 1.12.0 and the cancel context error message to reflect the behavior of 1.12.1.

  1. Can you set logging to DEBUG and collect the logs?

Not right now, this happens in production environments where its against our policy to collect these logs.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

@sfc-gh-dszmolka I’ve attached debug logs. I redacted anything we’d consider sensitive but again there isn’t any production data in this snowflake account so we should be fine. The debug logs do seem to reinforce my hunch that this has to do with context cancellation. To re-iterate, we managed to solve this in prod for ourselves by generating a new child context before every use of ExecContext() or QueryContext(). Like:


func () error {
	newCtx, _ = context.WithCancel(ctx) // forces new child context to isolate from concurrent queries

	_, err = db.ExecContext(newCtx, query)
	if err != nil {
		return err
	}

        return nil
}

using db.ExecContext(ctx, query) is what causes this error to non deterministically emerge.

debug_snowflake_logs.txt

Understand the situation, thank you ! When sharing logs, our recommendation is to please make sure nothing sensitive is shared over the public internet so please either sanitize it, or you can also upload to a private GH repository where you can grant my user access, or really any other ways of sharing a file which doesn’t end up sharing with everybody.

@sfc-gh-dszmolka I am setting the log level to debug for gosnowflake and I’ll monitor for the next time this error appears. What is lucky is that I can get this to fail in our nightlies every so often so that environment isn’t particularly sensitive. So I can probably share the debug logs here directly if that is simpler.

If that’s not possible, would you be able to provide a runnable reproducer (not a snippet) which I can run on my end in loops in hope it fails for me too ? This would be the most helpful perhaps, of all other options.

What’s hard about this is that our platform is a bit complex. There are a number of different pipelines executing concurrently on that same instance and how those pipelines are configured is dynamic based on state in our system and the query engine. We support over a dozen databases and their drivers, so the code also isn’t unique to the Snowflake driver. There is a lot of conditional logic depending on what driver we’re using. . There arealso a number of layers of abstractions to facilitate dynamic pipeline generation between the *sql.DB instance and how it actually gets used. Its not really a static set of code I can just rip out and simply hand to you.

@sfc-gh-dszmolka I apologize, I assumed earlier that any error on our nightly CI was the error noted in this issue. That was not true, I’ve waited for our CI to surface this exact issue and it did this morning. The error was Snowflake statement is not executing: Driver Error: 000605: Identified SQL statement is not currently executing from the driver and the session ID was 251775377707022 .

I’ll try to reproduce it in our nightly CI and give you something to work with tomorrow if I can reproduce it. It happens non deterministically so I’ll have to just keep running the battery until I get it to happen.