zephyr: drivers: video: video_stm32_dcmi: Detect and recover from: HAL_DCMI_ErrorCallback

Summary

With several of the different sketches that do things like display a camera image to a display of some form. Like Arduino Gigia Display shield, or some TFT display like an ST7796 or ILI9341, As I mentioned in the issue: #92352 I will receive several messages like:

[00:00:12.307,000] <dbg> video_stm32_dcmi: HAL_DCMI_FrameEventCallback: Failed to get buffer from fifo

Which in that issue I would hope to reuse the oldest frame…

But then I often times then get the message: [00:00:14.851,000] <wrn> video_stm32_dcmi: HAL_DCMI_ErrorCallback And the camera completely stops at that point

Describe the solution you’d like

I wish that the callback could correct the issue and recover.

Alternatives

Or wish there was a way for the client code to detect this and call something to recover.

Detecting this might simply be, that I would expect an image to be ready by my timeout… If not then call the recovery…

Additional Context

Most of my testing is done using the ArduinoCore-zephyr setup. Currently running the latest github code. The west update from their points us to a zephyr snaphshot from about a month ago.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 2
  • Comments: 36 (29 by maintainers)

Most upvoted comments

I think I am pretty late on this topic and I couldn’t follow all the discussions, apologies for that. In one of those threads talking about HAL_DCMI_ErrorCallback I think I wrote once that it could be due to the sensor sending data too rapidly. There are two potentially limiting factors on the receiver side:

  1. the DCMI interface itself. The kernel clock of the DCMI must be at least 2.5 time faster than the pixel clock received by the sensor. So it is important to see what is the kernel clock frequency which clock the DCMI and dividing this by 2.5 will give the maximum allowed pixel clock frequency it can handle. (in the CSI context we have been discussing about having helpers to tackle those HW capabilities via usage of the link-frequencies from video-interface.yaml but that’s not really in place yet)

  2. the DMA capabilities itself (and related to the downstream memory access). The DCMI IP receive data and store them into its internal FIFO. Then the DMA goes to read those data and store them somewhere else. So the performance of the DMA and memory access can also be a bottleneck.

To cop with those, it is usually possible to limit the sensor pixel clock rate.

As a matter of fact, the STM32L4 (stm32l4r9i_disco) is also facing this issue (HAL_DCMI_ErrorCallback) when used with the st_b_cam_omv shield. I haven’t got time to do that cleanly and post a PR for that, however just for explanatiion / demo it is possible to make it work (and use UVC sample) by simply forcing the OV5640 PLL frequency lower:

diff --git a/drivers/video/ov5640.c b/drivers/video/ov5640.c
index 664d99b9543..3997b9a4c1e 100644
--- a/drivers/video/ov5640.c
+++ b/drivers/video/ov5640.c
@@ -428,7 +428,7 @@ static const struct ov5640_reg init_params_dvp[] = {
        {0x3814, 0x31},
        {0x3815, 0x31},
        {0x3034, 0x1a},
-       {0x3035, 0x11},
+       {0x3035, 0x21},
        {0x3036, 0x64},
        {0x3037, 0x13},
        {0x3038, 0x00},

That done, OV5640 streaming into STM32L4R9I_DISCO is working just fine.

Thanks,

I should have mentioned, that the overlay I am using has:

&timers1 {
	status = "okay";
	st,prescaler = <0>;

	pwm1: pwm {
		status = "okay";
		pinctrl-0 = <&tim1_ch1_pa8 &tim1_ch2_pj11>;
		pinctrl-names = "default";
	};
};
...
&pwm1 {
	/* Camera source clock */
	pwmclock: pwmclock {
		status = "okay";
		compatible = "pwm-clock";
		clock-frequency = <0>;
		#clock-cells = <1>;
		pwms = <&pwm1 1 PWM_HZ(12000000) PWM_POLARITY_NORMAL>;
		/* 12MHz for 7670, default is also 12MHz */
	};
};

And yes the xclk, is the clock signal and running at 12mhz more or less:

Image

Will drill down farther…

Another update: I now have a version that runs on MBED, in particular I have it displaying currently on an ST7796 display. (Display code is slow, did first pass port the zephyr version I did to MBED). I made changes to Arducam library, My version is up at: https://github.com/KurtE/Arducam_dvp/tree/experiments_debug My sketch and ST7706 code are up in my github project: https://github.com/KurtE/Arduino_GIGA-stuff I am currently configured to use SDRam for the frame buffer. So far it has run for several minutes without failing.

Note: There are a lot of differences on how this Camera code works on MBED versus on Zephyr. On Zephyr you have multiple buffers defined, one is handed to the Camera, that is constantly is pounding the SDRAM with each frame. It then process an interrupt when each frame completes, and it then tries to grab a frame buffer from the free list, where it memcpy the data from the camera. The Arduino wrapper in the ArduinoCore-zephyr or my sketch then needs to swap the bytes for each pixel and then the code writes the data out of SPI to the TFT or over the xxx to write to the GigaDisplayShield… So lots of code hitting the SDRAM maybe at the same time…

With the MBED: when I call:
int err = cam.grabFrame(fb); The code does:

    if (HAL_DCMI_Start_DMA(&hdcmi, DCMI_MODE_SNAPSHOT,
                (uint32_t) framebuffer, framesize / 4) != HAL_OK) {
        if (_debug) {
            _debug->println("HAL_DCMI_Start_DMA FAILED!");
        }
    }

    // Wait until camera frame is ready.
    for (uint32_t start = millis(); (hdcmi.Instance->CR & DCMI_CR_CAPTURE);) {

So it starts the camera in SNAPSHOT mode (one frame), and then waits up to the timeout in a hard loop until the DCMI_CR_CAPTURE bit is cleared, when the hardware detects the end of the frame and stops…

Then the application or camera library if necessary swaps the bytes and then the application does whatever it needs to do, like send the bytes out to the display.

So with the read an image, write to display code, only one thing is touching the SDRAM. And by default if your code does not set the fb memory like: fb.setBuffer(fb_buf); The camera code will allocate the memory using malloc (not SDRAM).

Wondering if the Video/Camera code on Zephyr should have the ability for SnapShot mode?

Maybe the problem with ov5640 is the master clock is not enough ? For CSI2 mode on RT1170-EVK, the clock frequency is 24 MHz. For DVP mode on ST boards, I don’t know … just a guess …

Sorry everyone, was busy… watering garden.

I tried the a version of my zephyr sketch/… Made a copy of the GC2145 sketch… edited overlay and config… It is up at: https://github.com/KurtE/zephyr_test_sketches/tree/master/ov5640_to_ili9341 Set up with Portenta H7…

I am not getting past first base with it. That is I don’t believe that I have the reset and Powerdown pins configured properly…

Currently I have:

&i2c3 {
	status = "okay";

	ov5640: ov5640@3c {
		compatible = "ovti,ov5640";
		reg = <0x3c>;
		status = "okay";

		reset-gpios = <&gpioe 3 GPIO_ACTIVE_LOW>;
		powerdown-gpios = <&gpiod 5 GPIO_ACTIVE_LOW>;

		port {
			ov5640_ep_out: endpoint {
				remote-endpoint-label = "dcmi_ep_in";
			};
		};
	};

};

&dcmi {
	status = "okay";
	/* ext-sdram = <&sdram1>; */
	pinctrl-0 = <&dcmi_hsync_pa4 &dcmi_pixclk_pa6 &dcmi_vsync_pi5
		     &dcmi_d0_ph9 &dcmi_d1_ph10 &dcmi_d2_ph11 &dcmi_d3_ph12
		     &dcmi_d4_ph14 &dcmi_d5_pi4 &dcmi_d6_pi6 &dcmi_d7_pi7>;
	pinctrl-names = "default";
	dmas = <&dma1 0 75 (STM32_DMA_PERIPH_TO_MEMORY | STM32_DMA_PERIPH_NO_INC |
			    STM32_DMA_MEM_INC | STM32_DMA_PERIPH_32BITS | STM32_DMA_MEM_32BITS |
			    STM32_DMA_PRIORITY_HIGH) STM32_DMA_FIFO_1_4>; //FULL for 7670, default FIFO_1_4

	port {
		dcmi_ep_in: endpoint {
			remote-endpoint-label = "ov5640_ep_out";
			bus-width = <8>;
			hsync-active = <0>;
			vsync-active = <0>;
			pclk-sample = <0>;
		};
	};
};


/ {
	chosen {
		zephyr,camera = &dcmi;
		zephyr,console = &usart6;
		zephyr,shell-uart = &usart6;
		zephyr,cdc-acm-uart0 = &usart6;
	};

};

I edited the cfg to have CONFIG_VIDEO_OV5640=y instead of CONFIG_VIDEO_GC2145=y

The issue is the I2C communications is NAKing out.

Image

I believe the startup is supposed to look like:

Image

The init code for ov5640 has:

static int ov5640_init(const struct device *dev)
{
	const struct ov5640_config *cfg = dev->config;
	struct video_format fmt = {0};
	uint16_t chip_id;
	int ret;

	if (!device_is_ready(cfg->i2c.bus)) {
		LOG_ERR("Bus device is not ready");
		return -ENODEV;
	}

#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(reset_gpios)
	if (cfg->reset_gpio.port != NULL && !gpio_is_ready_dt(&cfg->reset_gpio)) {
		LOG_ERR("%s: device %s is not ready", dev->name, cfg->reset_gpio.port->name);
		return -ENODEV;
	}
#endif

#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(powerdown_gpios)
	if (cfg->powerdown_gpio.port != NULL && !gpio_is_ready_dt(&cfg->powerdown_gpio)) {
		LOG_ERR("%s: device %s is not ready", dev->name, cfg->powerdown_gpio.port->name);
		return -ENODEV;
	}
#endif

	/* Power up sequence */
#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(powerdown_gpios)
	if (cfg->powerdown_gpio.port != NULL) {
		ret = gpio_pin_configure_dt(&cfg->powerdown_gpio, GPIO_OUTPUT_ACTIVE);
		if (ret) {
			return ret;
		}
	}
#endif

#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(reset_gpios)
	if (cfg->reset_gpio.port != NULL) {
		ret = gpio_pin_configure_dt(&cfg->reset_gpio, GPIO_OUTPUT_ACTIVE);
		if (ret) {
			return ret;
		}
	}
#endif

	k_sleep(K_MSEC(5));

#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(powerdown_gpios)
	if (cfg->powerdown_gpio.port != NULL) {
		gpio_pin_set_dt(&cfg->powerdown_gpio, 0);
	}
#endif

	k_sleep(K_MSEC(1));

#if DT_ANY_INST_HAS_PROP_STATUS_OKAY(reset_gpios)
	if (cfg->reset_gpio.port != NULL) {
		gpio_pin_set_dt(&cfg->reset_gpio, 0);
	}
#endif

I know with some setups, We have had some luck with defaulting some of these pins to either INPUT or INPUT_PULLUP or INPUT_PULLDOWN… INPUT was nice if the cameras have their own PU/PD resistors to put them in their normal powered up and not reset setate…

Trying to now to setup on a teensy and see what this signals are doing on it…

Avoiding the memcpy() operation would make most sense, and maybe support for SNAPSHOT mode is the way. The memcpy() was convenient as a way to get the driver going, but for the way the video APIs work, swapping the buffer would make more sense than copying it…

I will follow-up about the bus-type on https://github.com/arduino/ArduinoCore-zephyr/issues/152, it is possible that it is one reason for which the OV5640 stays silent.

@avolmat-st - @KurtE

That done, OV5640 streaming into STM32L4R9I_DISCO is working just fine.

As @KurtE mentioned we are both playing around with the different cameras on the Arduino GIGA and the Arduino Portenta H7. While @KurtE has been working directly with the Zephyr examples I have been primarily working with the ArduinoCore-Zephry through the Arduino IDE. And as could be expected we are both seeing the same issues with the Cameras.

Right not I am trying to get the OV5640 working with both the GIGA and the H7 with just about zero luck.

Issue:

  1. With the GIGA Camera does not even start and have try several clocks - 6, 8, 10, 12 Mhz.
  2. With the H7 Camera seems to start at those frequencies but no frames are received - not even seeing anything in the debug window. No dcmi info at all. Some other debug info I added is that no frames are being dequeued so assuming no frames are being queued up so wondering if camera is really started.

The one thing I am curious about is what bus-type is being used for the GIGA and H7 which I asked on the arduinoCore-zephyr: https://github.com/arduino/ArduinoCore-zephyr/issues/152.

With the other camera’s no bus-type is specified so not 100% sure. Note tried both the GIGA and H7 with no bus specified and with bus-type specified as parallel. Neither works.

The other thing I noticed is that you all specify the configuration for the each support framesize. While what we do is something similar to what is done in the GC2145:


uint8_t OV5640::setFramesize(int w, int h) {
    uint16_t sensor_w = 0;
    uint16_t sensor_h = 0;

    if ((w == 0) || (h == 0))
        return 1; // not valid

    _width = w;
    _height = h;

    if (_debug)
        debug.printf("W/H: %d/%d\n", w, h);

    uint8_t reg = 0;
    int ret = 0;
    // uint16_t w = resolution[framesize][0];
    // uint16_t h = resolution[framesize][1];

    // Not a multiple of 8. The JPEG encoder on the OV5640 can't handle this.
    if ((_format == JPEG) && ((w % 8) || (h % 8))) {
        if (_debug)
            debug.print("JPEG not multiple of 8: failed to set.......");
        return 1;
    }

    // Readout speed too fast. The DCMI_DMAConvCpltUser() line callback overhead
    // is too much to handle the line transfer speed. If we were to slow the
    // pixclk down these resolutions would work. As of right now, the image
    // shakes and scrolls with the current line transfer speed. Note that
    // there's an overhead to the DCMI_DMAConvCpltUser() function. It's not the
    // memory copy operation that's too slow. It's that there's too much
    // overhead in the DCMI_DMAConvCpltUser() method to even have time to start
    // the line transfer. If it were possible to slow the line readout speed of
    // the OV5640 this would enable these resolutions below. However, there's
    // nothing in the datasheet that when modified does this.
    // if (((_format == GRAYSCALE) || (_format == BAYER) || (_format == JPEG)))
    // {
    //    return 1;
    //}

    // Invalid resolution.
    if ((w > ACTIVE_SENSOR_WIDTH) || (h > ACTIVE_SENSOR_HEIGHT)) {
        return 1;
    }

    // Step 0: Clamp readout settings.

    readout_w = max(readout_w, w);
    readout_h = max(readout_h, h);

    int readout_x_max = (ACTIVE_SENSOR_WIDTH - readout_w) / 2;
    int readout_y_max = (ACTIVE_SENSOR_HEIGHT - readout_h) / 2;
    readout_x = max(min(readout_x, readout_x_max), -readout_x_max);
    readout_y = max(min(readout_y, readout_y_max), -readout_y_max);

    // Step 1: Determine readout area and subsampling amount.

    uint16_t sensor_div = 0;

    if ((w > (readout_w / 2)) || (h > (readout_h / 2))) {
        sensor_div = 1;
    } else {
        sensor_div = 2;
    }

    // Step 2: Determine horizontal and vertical start and end points.

    sensor_w = readout_w +
               DUMMY_WIDTH_BUFFER; // camera hardware needs dummy pixels to sync
    sensor_h = readout_h +
               DUMMY_HEIGHT_BUFFER; // camera hardware needs dummy lines to sync

    uint16_t sensor_ws =
        max(min((((ACTIVE_SENSOR_WIDTH - sensor_w) / 4) + (readout_x / 2)) * 2,
                ACTIVE_SENSOR_WIDTH - sensor_w),
            -(DUMMY_WIDTH_BUFFER / 2)) +
        DUMMY_COLUMNS; // must be multiple of 2
    uint16_t sensor_we = sensor_ws + sensor_w - 1;

    uint16_t sensor_hs =
        max(min((((ACTIVE_SENSOR_HEIGHT - sensor_h) / 4) - (readout_y / 2)) * 2,
                ACTIVE_SENSOR_HEIGHT - sensor_h),
            -(DUMMY_HEIGHT_BUFFER / 2)) +
        DUMMY_LINES; // must be multiple of 2
    uint16_t sensor_he = sensor_hs + sensor_h - 1;

    // Step 3: Determine scaling window offset.

    float ratio = min((readout_w / sensor_div) / ((float)w),
                      (readout_h / sensor_div) / ((float)h));

    uint16_t w_mul = w * ratio;
    uint16_t h_mul = h * ratio;
    uint16_t x_off = ((sensor_w / sensor_div) - w_mul) / 2;
    uint16_t y_off = ((sensor_h / sensor_div) - h_mul) / 2;

    // Step 4: Compute total frame time.

    hts_target = sensor_w / sensor_div;

    uint16_t sensor_hts = calculate_hts(w);
    uint16_t sensor_vts = calculate_vts(sensor_h / sensor_div);

    uint16_t sensor_x_inc =
        (((sensor_div * 2) - 1) << 4) |
        (1 << 0); // odd[7:4]/even[3:0] pixel inc on the bayer pattern
    uint16_t sensor_y_inc =
        (((sensor_div * 2) - 1) << 4) |
        (1 << 0); // odd[7:4]/even[3:0] pixel inc on the bayer pattern

    if (_debug) {
        debug.println("Setting FrameSize:");
        debug.println("  Step 0:");
        debug.printf("   readout_w: %d, readout_h: %d, readout_x_max: %d, "
                     "readout_x_max: %d\n",
                     readout_w, readout_h, readout_x_max, readout_y_max);
        debug.printf("   readout_x: %d, readout_y: %d\n", readout_x, readout_y);
        debug.println("  Step 1:");
        debug.printf("   sensor_div: %d\n", sensor_div);
        debug.println("  Step 2:");
        debug.printf(
            "  sensor_w: %d, sensor_h: %d, sensor_ws: %d, sensor_we: %d\n",
            sensor_w, sensor_h, sensor_ws, sensor_we);
        debug.printf("  sensor_hs: %d, sensor_hs: %d\n", sensor_hs, sensor_he);
        debug.printf("  Step 3:");
        debug.printf("  ratio: %f, w_mul: %d, h_mul: %d\n", ratio, w_mul,
                     h_mul);
        debug.printf("  x_off: %d, y_off: %d\n", x_off, y_off);
        debug.printf("  Step 4:");
        debug.printf("  hts_target: %d, sensor_hts: %d, sensor_vts: %d: %d\n",
                     hts_target, sensor_hts, sensor_vts);
        debug.printf("  sensor_x_inc: %d, sensor_x_inc:%d\n", sensor_x_inc,
                     sensor_x_inc);
    }

    // Step 5: Write regs.

    ret |= cameraWriteRegister(TIMING_HS_H, sensor_ws >> 8);
    ret |= cameraWriteRegister(TIMING_HS_L, sensor_ws);

    ret |= cameraWriteRegister(TIMING_VS_H, sensor_hs >> 8);
    ret |= cameraWriteRegister(TIMING_VS_L, sensor_hs);

    ret |= cameraWriteRegister(TIMING_HW_H, sensor_we >> 8);
    ret |= cameraWriteRegister(TIMING_HW_L, sensor_we);

    ret |= cameraWriteRegister(TIMING_VH_H, sensor_he >> 8);
    ret |= cameraWriteRegister(TIMING_VH_L, sensor_he);

    ret |= cameraWriteRegister(TIMING_DVPHO_H, w >> 8);
    ret |= cameraWriteRegister(TIMING_DVPHO_L, w);

    ret |= cameraWriteRegister(TIMING_DVPVO_H, h >> 8);
    ret |= cameraWriteRegister(TIMING_DVPVO_L, h);

    ret |= cameraWriteRegister(TIMING_HTS_H, sensor_hts >> 8);
    ret |= cameraWriteRegister(TIMING_HTS_L, sensor_hts);

    ret |= cameraWriteRegister(TIMING_VTS_H, sensor_vts >> 8);
    ret |= cameraWriteRegister(TIMING_VTS_L, sensor_vts);

    ret |= cameraWriteRegister(TIMING_HOFFSET_H, x_off >> 8);
    ret |= cameraWriteRegister(TIMING_HOFFSET_L, x_off);

    ret |= cameraWriteRegister(TIMING_VOFFSET_H, y_off >> 8);
    ret |= cameraWriteRegister(TIMING_VOFFSET_L, y_off);

    ret |= cameraWriteRegister(TIMING_X_INC, sensor_x_inc);
    ret |= cameraWriteRegister(TIMING_Y_INC, sensor_y_inc);

    ret |= cameraReadRegister(TIMING_TC_REG_20, reg);
    ret |=
        cameraWriteRegister(TIMING_TC_REG_20, (reg & 0xFE) | (sensor_div > 1));

    ret |= cameraReadRegister(TIMING_TC_REG_21, reg);
    ret |=
        cameraWriteRegister(TIMING_TC_REG_21, (reg & 0xFE) | (sensor_div > 1));

    ret |= cameraWriteRegister(VFIFO_HSIZE_H, w >> 8);
    ret |= cameraWriteRegister(VFIFO_HSIZE_L, w);

    ret |= cameraWriteRegister(VFIFO_VSIZE_H, h >> 8);
    ret |= cameraWriteRegister(VFIFO_VSIZE_L, h);

    return ret;
}

where:


#define BLANK_LINES 8
#define DUMMY_LINES 6

#define BLANK_COLUMNS 0
#define DUMMY_COLUMNS 8

#define SENSOR_WIDTH 2624
#define SENSOR_HEIGHT 1964

#define ACTIVE_SENSOR_WIDTH (SENSOR_WIDTH - BLANK_COLUMNS - (2 * DUMMY_COLUMNS))
#define ACTIVE_SENSOR_HEIGHT (SENSOR_HEIGHT - BLANK_LINES - (2 * DUMMY_LINES))

#define DUMMY_WIDTH_BUFFER 16
#define DUMMY_HEIGHT_BUFFER 8

#define HSYNC_TIME 252
#define VYSNC_TIME 24

static int16_t readout_x = 0;
static int16_t readout_y = 0;

static uint16_t readout_w = ACTIVE_SENSOR_WIDTH;
static uint16_t readout_h = ACTIVE_SENSOR_HEIGHT;

static uint16_t hts_target = 0;

If you all have any thoughts it would help.

One would have to compare the speed of the image sensor to be sure, but good to have this as reference!

It will help to have more image sensor ported and more features in the current sensors, to compare across them. A WIP task!

If the SDRAM is not fast enough to push the data from DCMI, this can lead to a DMA error!

And this is not the first STM32 that seems to have this issue: