esp-idf: WiFi connected, have IP, but IP stack broken (IDFGH-9)

Hi @burkulesomesh43 & @Ritesh236,

My guess is that this isn’t a bug in ESP-IDF, and is probably related to your application code.

I am not a contributor, but this might help:

• CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST is not compulsory, but it is sensible if you have PSRAM installed. LWIP will then use heap_caps_malloc(sz, MALLOC_CAP_SPIRAM) where it can, reducing internal memory usage.
• The default ESP-IDF malloc() implementation uses heap_caps_malloc(sz, MALLOC_CAP_8BIT). This often results in internal memory being allocated.
• It is possible that you are exhausting internal memory with your application’s malloc() usage. I prefer to use MALLOC_CAP_SPIRAM whenever I can in application code. This is usually enough to solve the WiFi management buffer alloc failures.
• You may want to look at CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL to set a threshold if allocating a size of < threshold then prefer internal memory, else prefer external / PSRAM.
• Also, check out CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL to reserve an internal memory area for only DMA / MALLOC_CAP_INTERNAL allocations.

Read these pages, and it should become clear:

• https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/mem_alloc.html
• https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/external-ram.html

Although the above docs are for ‘latest’, a lot of the fundamentals apply to ESP-IDF v3.

liuzfesp did a good write-up on alloc eb fail: https://github.com/espressif/esp-idf/issues/1021#issuecomment-469502062

Finally, it may be better to post on the ESP32 forums, rather than this issue, as AFAIK it’s not a bug in IDF.

I hope you manage to fix your problem. Good luck.

https://github.com/espressif/esp-idf/issues/3592#issuecomment-517926444

Thanks @talss89 for your valuable feedback.

So, I have already checked few configurations earlier but still we will check it again and get back to you if any query regarding that.

And also, We will create issue over ESP32 community with details for same to get any other help regarding same.

Thanks again for your valuable feedback

Regards, Ritesh Prajapati

Alright, I think I am pretty close to a root cause on this. It appears to be related to memory allocation in the WiFi or TCP stack. I managed to get a new error message that seems to back this up:

W (11714) wifi: alloc eb len=24 type=3 fail
W (11724) wifi: m f null
W (11794) wifi: alloc eb len=36 type=3 fail
W (11794) wifi: mem fail

When my app starts it immediately tries to connect to a stored WiFi AP. Another task is blocking, waiting for WiFi to be connected, and as soon as it is, it tries to connect to MQTT over TLS. And yet another task is trying to send some requests to an HTTPS API if WiFi is connected.

I think what is actually happening is that DMA capable RAM is being exhausted. If I put a long delay in my code before hitting the HTTPS API everything is fine until it tries to connect, then I get the “null” failure. Here is a log with some memory logging included:

| 8BIT  | 32BIT | 32BIT - 8BIT | INTERNAL | SPIRAM |  DMA  | Heap Total | Used  | Free  | SOA Used | Free  |  HW   |
|=======|=======|==============|==========|========|=======|============|=======|=======|==========|=======|=======|
|1695740|1733568|         37828|     60748| 1672820|  22920|     4375996|2678708|1697288|         1|  32767|  32606|
I (2471) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1
I (3231) wifi: state: init -> auth (b0)
I (3231) wifi: state: auth -> assoc (0)
I (3241) wifi: state: assoc -> run (10)
I (3261) wifi: connected with xx, channel 11
S (3271)  xx_wifi.c:181  (0x0000) SYSTEM_EVENT_STA_CONNECTED ssid "xx", bssid xx, channel 11, authmode WIFI_AUTH_WPA2_PSK.
I (5231) event: sta ip: 192.168.10.141, mask: 255.255.255.0, gw: 192.168.10.1
S (5231)  xx_wifi.c:200  (0x0000) SYSTEM_EVENT_STA_GOT_IP IP 192.168.10.141, GW 192.168.10.1, Netmask 255.255.255.0.
I (5361)  xx_mqtt.c:115  (0x0000) Connecting to MQTT at xx, use_tls = 1
I (6241) wifi: pm start, type:1
|1647372|1685200|         38020|     45892| 1639116|   8064|     4375996|2678708|1697288|         1|  32767|  32606|
I (9541)  xx_mqtt.c:108  (0x0000) MQTT subscribed to xx
I (9631)  xx_mqtt.c:108  (0x0000) MQTT subscribed to xx
I (9711)  xx_mqtt.c:108  (0x0000) MQTT subscribed to xx
S (9731)  xx_mqtt.c:167  (0x0000) xx xx xx connected to MQTT.
|1650656|1688484|         37828|     47784| 1640700|   9956|     4375996|2678708|1697288|         1|  32767|  32606|
|1650924|1688752|         37828|     48056| 1640696|  10228|     4375996|2678708|1697288|         1|  32767|  32606|
|1650924|1688752|         37828|     48056| 1640696|  10228|     4375996|2678708|1697288|         1|  32767|  32606|

-- RUNNING HTTPS API REQUEST

W (25721) wifi: alloc eb len=24 type=3 fail
W (25721) wifi: m f null
|1596988|1634816|         37828|     37828| 1596988|      0|     4375996|2678708|1697288|         1|  32767|  32606|
|1592124|1629952|         37828|     37828| 1592124|      0|     4375996|2678708|1697288|         1|  32767|  32606|
|1591720|1629548|         37828|     37828| 1591720|      0|     4375996|2678708|1697288|         1|  32767|  32606|

I’m investigating now if I can find a way to free up more DMA capable RAM to see if that helps.

One other note: It seems like when these TLS connections fail they don’t give back the RAM. Could be a memory leak, or could just be that this is an unrecoverable error and I should be rebooting anyway.

And finally, I’ll note that I’m now on the 3.0.2 release instead of the detached HEAD I was using before.

Thanks, Jason

HI @talss89, no I’m not using AWS IOT, but I am using a custom port of Apache Paho and they are similar, and both use mbedtls. For what it’s worth, I see this error both when connecting to MQTT and when trying to HTTPS connections, and as you note, it does seem to happen as soon as I start to connect to a service, rather than as soon as WiFi connects.

I have long suspected this had something to do with mbedtls, but it’s only a guess.