arrow: [Python] Cannot read data if endpoint is s3 on a "secure" Minio server
Describe the usage question you have. Please include as many useful details as possible.
I would like to read a CSV file from my TLS-secured object storage (minio).
Here is my code and configuration:
from pyarrow import fs, csv, parquet
minio = fs.S3FileSystem(
endpoint_override="https://localhost:port",
access_key="user1234",
secret_key="password1234",
)
dataCSV = minio.open_input_file("bucket/filename.csv")
I get the following error message:
OSError: When reading information for key 'filename.csv' in bucket 'bucket': AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 60, SSL peer certificate or SSH remote key was not OK
If I use the same credentials to read the data with duckDB, it works perfectly.
What am I doing wrong?
Component(s)
Python
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 16 (9 by maintainers)
Please keep this issue open. The next step is opening a PR that improves
S3Options.Thanks.
It seems that we need to add TLS related options to https://github.com/apache/arrow/blob/e3b0bd1feb63d59cd6fb553af976449397b8348e/cpp/src/arrow/filesystem/s3fs.h#L102 and use it for internal AWS SDK for C++.