Adafruit_CircuitPython_ESP32SPI: No recovery after Timed out waiting for SPI char

I have an application that polls a couple of endpoints, with adafruit_requests, every few seconds and makes a nice display. Every few minutes adafruit_requests fails with: “Timed out waiting for SPI char” in adafruit_esp32spi.py line 271. Once this happens the stack becomes permanently borked with all subsequent calls failing with the same error. The only way to recover that I’ve found is to reboot.

This happens with 2 pyportals, 2 physically different networks and multiple USB cables.

Firmware: 1.2.2 and 1.6.1 (current stable) Circuit python: 4.something and 5.3.0 (current stable) Libraries: 4.something and 5.x (current stable)

The following code reproduces the problem about 90% of the time with an occasional success in the second GET and occasionally a different error.

import time
import board
import busio
from digitalio import DigitalInOut
from adafruit_esp32spi import adafruit_esp32spi
import adafruit_esp32spi.adafruit_esp32spi_socket as socket
import adafruit_requests as requests
from secrets import secrets

# Non existent endpoint
BAD_DATA_SOURCE = "http://1.2.3.4/hello"

# Good data source
# GOOD_DATA_SOURCE = "http://httpbin.org/get"
GOOD_DATA_SOURCE = 'http://54.236.246.173/get'

# Start WiFi
esp32_ready = DigitalInOut(board.ESP_BUSY)
esp32_gpio0 = DigitalInOut(board.ESP_GPIO0)
esp32_reset = DigitalInOut(board.ESP_RESET)
esp32_cs = DigitalInOut(board.ESP_CS)

spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset, esp32_gpio0)

requests.set_socket(socket, esp)

print("MAC: " + ':'.join('{:02x}'.format(x) for x in reversed(esp.MAC_address)))
print("Connecting to AP...")
while not esp.is_connected:
  try:
    esp.connect_AP(secrets['ssid'], secrets['password'])
  except RuntimeError as e:
    print("could not connect to AP, retrying: ",e)
    continue
print("Connected to", str(esp.ssid, 'utf-8'), "\tRSSI:", esp.rssi)
print("Firmware vers.", esp.firmware_version)
print("My IP address is", esp.pretty_ip(esp.ip_address))


try:
  r = requests.get(BAD_DATA_SOURCE)
except RuntimeError as e:
  # Timed out waiting for SPI char
  # A failure here is to be expected
  print(e)


# This GET always fails
r = requests.get(GOOD_DATA_SOURCE)
print(r.text)

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 31 (11 by maintainers)

Commits related to this issue

Most upvoted comments

I have been plagued by the ‘Timed out waiting for SPI char’ exception for a long time, both on a Matrix Portal and PyPortal. To try to understand what was going on, I added code to ESP_SPIcontrol._wait_spi_char to increase the timeout to 1 second and to record the times for each request. Here is what I found:

Running for 8 hours with a simple application on a PyPortal, talking via a socket to another system, in 8 hours there were 2.2 million calls to _wait_spi_char. They were essentially all from my application’s calls to socket.connected and socket.available, waiting for messages. Of those 2.2 million calls, 99.5% of them completed in less than 1 ms (average was actually less than 5 us). 10,000 of them took between 1 and 10 ms. But 30 of them took more than 100ms (which is the timeout value coded in _wait_spi_char). All of those would have generated the exception. The longest of those 30 was just 176 ms. Since I had increased the timeout to 1000 ms, none of them failed and the application had no problems. Of interest is that there were none in the range between 10ms and 100ms. Something, (garbage collect perhaps) occasionally causes a relatively long wait.

Running with the timeout at 1 second has eliminated this as a problem for me.

Update all of your libraries, install the CircuitPython beta, and update your ESP32 firmware.

@tannewt : Is there a particular log/info I should try to fetch in order to see how we can fix this issue? I also wonder if the mqtt code should be made more resilient so it does not go off the rails as it does now once this happens; it may actually be a better way to approach this issue.

Not that I can think of. There is a lot of complexity here between the ESP32SPI driver and the nina firmware that I don’t know how to simplify.

I think your idea to be more resilient is a good one. Networking can cause errors at any time and that’s tricky to handle well.