arrow: [Python] Cannot read data if endpoint is s3 on a "secure" Minio server

Describe the usage question you have. Please include as many useful details as possible.

I would like to read a CSV file from my TLS-secured object storage (minio).

Here is my code and configuration:

from pyarrow import fs, csv, parquet
minio = fs.S3FileSystem(
    endpoint_override="https://localhost:port",
    access_key="user1234",
    secret_key="password1234",
)

dataCSV = minio.open_input_file("bucket/filename.csv")

I get the following error message:

OSError: When reading information for key 'filename.csv' in bucket 'bucket': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 60, SSL peer certificate or SSH remote key was not OK

If I use the same credentials to read the data with duckDB, it works perfectly.

What am I doing wrong?

Component(s)

Python

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Please keep this issue open. The next step is opening a PR that improves S3Options.

Thanks.

It seems that we need to add TLS related options to https://github.com/apache/arrow/blob/e3b0bd1feb63d59cd6fb553af976449397b8348e/cpp/src/arrow/filesystem/s3fs.h#L102 and use it for internal AWS SDK for C++.