python-slack-sdk: File Upload API succeeds despite 408 Request Timeout

Please, priorotize

I’m using curl for uploading file and had the same problem. What worked as a workaround is adding: --limit-rate 5M.

@vishim I know that we are very slow on this but we are still waiting for the progress on the backend engineering side. Once we get the updates that we can publicly share, we will update y’all here.

Having this issue on a node project using the Bolt API, yesterday it worked. Today, it’s failing with 408’s for every single file, nothing succeeds regardless of how many times Bolt retries. File size is only in the 1-2 MB range. Tried switching to Axios to run upload requests instead, same issue. Update - definitely an intermittent issue. Another day later and I don’t see any 408’s at all.

Facing the same issue, trying to upload 60MB file using curl request, failing with stream timeout error.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 61.0M    0     0  100 61.0M      0  50.2M  0:00:01  0:00:01 --:--:-- 50.2M
100 61.0M    0     0  100 61.0M      0  27.5M  0:00:02  0:00:02 --:--:-- 27.5M
100 61.0M    0     0  100 61.0M      0  18.9M  0:00:03  0:00:03 --:--:-- 18.9M
100 61.0M    0     0  100 61.0M      0  14.4M  0:00:04  0:00:04 --:--:-- 14.4M
100 61.0M    0     0  100 61.0M      0  11.6M  0:00:05  0:00:05 --:--:-- 11.6M
100 61.0M    0     0  100 61.0M      0   9.8M  0:00:06  0:00:06 --:--:--     0
100 61.0M    0     0  100 61.0M      0  8655k  0:00:07  0:00:07 --:--:--     0
100 61.0M    0     0  100 61.0M      0  7602k  0:00:08  0:00:08 --:--:--     0
100 61.0M    0     0  100 61.0M      0  6777k  0:00:09  0:00:09 --:--:--     0
100 61.0M  100    14  100 61.0M      1  6117k  0:00:14  0:00:10  0:00:04     2
100 61.0M  100    14  100 61.0M      1  6116k  0:00:14  0:00:10  0:00:04     3
stream timeout

curl https://slack.com/api/files.upload --limit-rate 423K -F token=“xoxb-PUT-YOUR-TOKEN-HERE” -F channels=“#test-channel” -F title=“TESTING! FileName.apk” -F filename=“filename.txt” -F file=“@/path/to/file/test_empty_file.txt”

Thankyou 🙏

I’m using curl for uploading file and had the same problem. What worked as a workaround is adding: --limit-rate 5M.

That moment, when a guy from comments helped more than an official support

Hi all, let me share some updates on this issue.

Firstly, sincere apologies for taking a long time to resolve this issue on the Slack platform side. We do understand that this issue has been critical for many people.

As a solution, we just released v3.19.0, which includes a new method named WebClient#files_upload_v2(). This new method is much stabler and is mostly compatible with the existing files_upload() method. Please migrate to the new method if your app is affected by the server-side performance issue described here. Here is an example code demonstrating how to use the v2 method:

response = client.files_upload_v2(
    file="./logo.png",
    title="New company logo",
    # Note that channels still works but going with channel="C12345" is recommended
    # channels=["C111", "C222"] is no longer supported. In this case, an exception can be thrown 
    channels=["C12345"],
    initial_comment="Here is the latest version of our new company logo :wave:",
)

The new method eliminates the timeouts. In addition, it enables 3rd party app developers to upload multiple files like humans do in Slack clients. This feature addition can be useful for many use cases. Here is a simple example code:

response = client.files_upload_v2(
    file_uploads=[
        {
            "file": "./logo.png",
            "title": "New company logo",
        },
        {
            "content": "Minutes ....",
            "filename": "team-meeting-minutes-2022-03-01.md",
            "title": "Team meeting minutes (2022-03-01)",
        },
    ],
    channel="C12345",
    initial_comment="Here is the latest version of our new company logo :wave:",
)
response.get("files")  # returns the full metadata of all the uploaded files

Please refer to the release notes for more details: https://github.com/slackapi/python-slack-sdk/releases/tag/v3.19.0

I tried to throttle using OkHttp to 500KB/Sec and yet the issue persists for me when uploading 40MB files from the API. Oddly enough we can upload the same files using the slack desktop/android apps without any issues. This issue is so frustrating!

Even we are facing the same issue @seratch. We are not able to upload build artefacts to slack 😢

Keeping you in the loop here @jaypoojara. Let’s follow this thread.

curl https://slack.com/api/files.upload --limit-rate 423K -F token=“xoxb-PUT-YOUR-TOKEN-HERE” -F channels=“#test-channel” -F title=“TESTING! FileName.apk” -F filename=“filename.txt” -F file=“@/path/to/file/test_empty_file.txt”

Maybe a temporary workaround for those who are experiencing this issue intermittently. It retries three times each with a 3s pause.

for file_name in files_list:
    # Try up to three times because Slack file uploads are recently kinda flaky.
    # @see https://github.com/slackapi/python-slack-sdk/issues/1165
    tries = 3
    for i in range(tries):
        if i > 0:
            print(f"Trying upload again for {file_name} (attempt {i+1} of {tries}) ...")
        try:
            # Limit to 20 calls per minute so we don't get rate-limited by Slack.
            # @see https://api.slack.com/docs/rate-limits
            if len(files_list) > 20:
                time.sleep(3)
            upload_result = client.files_upload(file=file_name)
            logger.info(upload_result)
        except SlackApiError as e:
            logger.error(f"Error uploading file {file_name}. Attempt {i+1} of {tries}: {e}")
            if i < tries - 1:
                time.sleep(3)
                continue
            else:
                raise
        break

Edit: We have been using the above for a week now. Normally it does not need more than one try. But occasionally it goes up to three tries, with the third try succeeding.

Is there any correlation with file type? This started happening to us with ~33MB zip file.
Larger text (csv) files never had this issue…but our CSVs hit a ridiculous size and we started zipping them when they were over some threshold and then we started getting this issue. Makes me suspicious that this might relate to file-type rather than just raw size.

Can anyone else commend on the file types you are experiencing the problem with (binary vs text and also mime-type)? Maybe there is a correlation that will make this easier for the devs to repro.

@locomike I’m sorry to say this but I don’t have anything further that I can suggest at this moment. We will continue communicating with the server-side teams.

@iamarjun Thanks for sharing more details!

I’ve checked internal reports and found that other people are also facing this server-side issue with relatively large size files (I know that your file is not so large but it seems that the situation can arise even with mega-byte-level size).

The server-side engineering teams are working hard to eliminate (at least mitigate) the issue now but I cannot tell when the situation can be improved.

The only workaround that I can suggest is that, if you receive the 408 error, your app can check files.list API results with the same token to know whether the upload was actually successful. If there is no new item in the response, your app needs to retry the same files.upload API call.

We’re sorry for the disruption.

Hi @seratch here’s the full log using curl

curl -v https://slack.com/api/files.upload -F token=access-token -F channels= "#ios-ajay-test1" -F file=@file_path

*   Trying 13.126.138.201...
* TCP_NODELAY set
* Connected to slack.com (13.126.138.201) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Slack Technologies, Inc.; CN=slack.com
*  start date: Apr 13 00:00:00 2021 GMT
*  expire date: Apr 18 23:59:59 2022 GMT
*  subjectAltName: host "slack.com" matched cert's "slack.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc26010800)
> POST /api/files.upload HTTP/2
> Host: slack.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 195338476
> Content-Type: multipart/form-data; boundary=------------------------ce8e5e2beebe823d
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
* We are completely uploaded and fine
< HTTP/2 408 
< x-edge-backend: envoy-www
< content-length: 14
< content-type: text/plain
< x-slack-edge-shared-secret-outcome: no-match
< date: Wed, 26 Jan 2022 08:48:33 GMT
< server: envoy
< via: envoy-edge-bom-3mz6
< 
* Connection #0 to host slack.com left intact
stream timeout* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection 0