zipline: Bug: throw er; // Unhandled 'error' event when using s3 as datasource

What happened?

I’ve tried to deploy zipline with a minio s3 bucket as data source. When trying to start the container, the error below was thrown in the logs.

Following S3 ENV variables:

  DATASOURCE_S3_ACCESS_KEY_ID: [my-id]
  DATASOURCE_S3_SECRET_ACCESS_KEY: [my-key]
  DATASOURCE_S3_ENDPOINT: [my-minio-endpoint]
  DATASOURCE_S3_BUCKET: zipline
  DATASOURCE_S3_FORCE_S3_PATH: "false"
  DATASOURCE_S3_REGION: eu-finland

It can’t be minio, as other applications can easily use other buckts in there.

Version

latest (ghcr.io/diced/zipline:latest)

What browser(s) are you seeing the problem on?

No response

Zipline Logs

2023-05-08 08:08:34,588 PM info  [datasource] using S3(zipline) datasource
2023-05-08 08:08:34,617 PM info  [database::migrations] establishing database connection
2023-05-08 08:08:34,618 PM info  [database::migrations] ensuring database exists, if not creating database - may error if no permissions
2023-05-08 08:08:34,741 PM info  [database::migrations] exiting migrations engine - database is up to date
2023-05-08 08:08:35,817 PM info  [server] listening on 0.0.0.0:3000
2023-05-08 08:08:35,817 PM info  [server] started production zipline@3.7.0 server

node:events:491
      throw er; // Unhandled 'error' event
      ^
S3Error
    at Object.parseError (/zipline/node_modules/minio/dist/main/xml-parsers.js:75:11)
    at /zipline/node_modules/minio/dist/main/transformers.js:167:22
    at DestroyableTransform._flush (/zipline/node_modules/minio/dist/main/transformers.js:91:10)
    at DestroyableTransform.prefinish (/zipline/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (node:events:513:28)
    at prefinish (/zipline/node_modules/readable-stream/lib/_stream_writable.js:611:14)
    at finishMaybe (/zipline/node_modules/readable-stream/lib/_stream_writable.js:620:5)
    at endWritable (/zipline/node_modules/readable-stream/lib/_stream_writable.js:643:3)
    at Writable.end (/zipline/node_modules/readable-stream/lib/_stream_writable.js:571:22)
    at IncomingMessage.onend (node:internal/streams/readable:705:10)
Emitted 'error' event on Readable instance at:
    at DestroyableTransform.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:1845:127)
    at DestroyableTransform.emit (node:events:513:28)
    at /zipline/node_modules/minio/dist/main/minio.js:1780:33
    at DestroyableTransform.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:573:13)
    at DestroyableTransform.emit (node:events:525:35)
    at done (/zipline/node_modules/readable-stream/lib/_stream_transform.js:192:25)
    at /zipline/node_modules/readable-stream/lib/_stream_transform.js:130:7
    at DestroyableTransform._flush (/zipline/node_modules/minio/dist/main/transformers.js:91:7)
    at DestroyableTransform.prefinish (/zipline/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (node:events:513:28) {
  amzRequestid: null,
  amzId2: null,
  amzBucketRegion: null
}

Node.js v18.15.0

Browser Logs

No response

Additional Info

I’ve tried with both DATASOURCE_S3_FORCE_S3_PATH true and false. There seems to go something wrong when parsing the S3 parameters or something.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

I had almost the same problem, but with IDrive e2 and DATASOURCE_S3_USE_SSL=true solved it. Thanks!

@TheAnachronism Try the DATASOURCE_S3_USE_SSL=true variable.