esp-idf: httpclient has unexpected and unhadled error that causes the call to not return during esp_https_ota (IDFGH-4543)

Environment

• Development Kit: [none]
• Module or chip used: [ESP32-WROOM-32D]
• IDF version (run git describe --tags to find it): v4.2-47-g2532ddd9f
• Build System: [idf.py]
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0
• Operating System: [Linux]
• Using an IDE?: [No]
• Power Supply: [external 3.3V]

Problem Description

Sometimes when I call esp_err_t ret = esp_https_ota(&config); the function never returns. It prints a few log messages referencing EAGAIN. There is no more OTA relevant information in the log.

Expected Behavior

1. esp_https_ota should not block for 10+ hours
2. As a user of esp_https_ota i expect the library to handle any errors or return a suitable esp_err_t
3. It looks like EAGAIN is related to the non-blocking socket API, but esp_https_ota is blocking (as noted here). Thus it seems EAGAIN should not be a possible errno even in the internal implementation of esp_https_ota

Actual Behavior

esp_http_client_config_t config = {
        .url = image_location,
        .cert_pem = (char *)server_cert_pem_start,
        .event_handler = _http_event_handler,
	.timeout_ms = 20000,
};
esp_err_t ret = esp_https_ota(&config);
ESP_LOGI(TAG, "sometimes the above call never returns");

When esp_https_ota does not return, I get log output that looks like this:

D (08:01:36.759) esp_https_ota: Written image length 1104558
D (08:01:36.766) esp_https_ota: Written image length 1104847
D (08:01:36.772) esp_https_ota: Written image length 1105136
D (08:01:36.779) esp_https_ota: Written image length 1105425
D (08:01:36.785) esp_https_ota: Written image length 1105714
E (08:01:56.782) TRANS_SSL: esp_tls_conn_read error, errno=No more processes
W (08:01:56.783) HTTP_CLIENT: esp_transport_read returned:-26880 and errno:11 
D (08:01:56.792) esp_https_ota: Written image length 1105920

No more OTA logs are printed. The MQTT client on the system keeps working.

The issue only occurs on the office Wi-Fi network. When using an alternative wifi network, that is used for development only, the issue went away.

Steps to reproduce

I have not found a good way to reproduce this issue. As it is dependent on the wifi network used, I expect that this is tricky to reproduce. When testing in my environment I ran the code above on startup, patched esp_https_ota to do esp_restart before esp_https_ota_finish. This results in a system that downloads the ota image, then reboots. Eventually, it will hang in esp_https_ota as described above.

Other items

• It seems that someone in the forum has a similar issue: https://www.esp32.com/viewtopic.php?t=17732
• Issue #4394 appears to be very related, with the same errno in the log, and it has a commit that fixed the issue 2 yeas ago

It may be that the issue is a lack of error handling in esp_https_ota. Alternatively, there is an error in the HTTP client that should not occur.

Please let me know if there is any additional information I can provide.