esp-idf: Support for 32MB version of ESP32S3 Module is missing (IDFGH-6736)

Environment

• Development Kit: ESP32-S3-DevKitC
• Kit version (for WroverKit/PicoKit/DevKitC): V1
• Module or chip used: ESP32-S3-WROOM-2 (N32R8V)
• IDF version (run git describe --tags to find it): v5.0-dev-1429
• Build System: CMake
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32s3-elf-gcc (crosstool-NG esp-2021r2) 8.4.0
• Operating System: Linux
• Using an IDE?: Yes, Clion with IDF_TARGET=esp32s3 set and export.sh as env script
• Power Supply: USB

Problem Description

1.options in menuconfig does not contains 32Mbyte 2.esptool is unable to flash upper 16Mbit of flash (either with --no-stub or not),   erasing or writing above 0x1000000 with --no-stub will warp to 0x00000000 then bootloader will overwride   writing with stub will resault a error ,showing “opi_flash.c 363” at uart port.   try to patch stub with ER4B cmd in the flash datasheet  https://www.macronix.com/Lists/Datasheet/Attachments/7693/MX25UM25645G,%201.8V,%20256Mb,%20v1.1.pdf  and set addr_len to 32bit will do the same thing as --no-stub,   it seems like the high 8bit addr is never sent out or sent zero 3.partition size can’t be bigger than 16M 4.bootloader is not correctly parsing 32M flash or switch flash to opi_dtr mode 5.spi-flash api works fine, including erasing writing,even mount fatfs ,I don’t under stand why stub can’t behave as normal. 6.mmap to Data MMU can’t be more than 16M   Cache_Get_DROM_MMU_End() always returned 0x00000400,so map can’t be more than 0x00000400/4×64K=16M,   hardcoded Cache_Get_DROM_MMU_End() to Cache_Get_DROM_MMU_End()+128*4 works for me(not 256 to save last 8M for PSRAM)

Expected Behavior

all above works as it should be

Actual Behavior

see Problem Description

Steps to reproduce

1. just run menuconfig
2. esptool.py -c esp32s3 -p /dev/ttyUSB0 -b 2000000 --before=default_reset --after=hard_reset write_flash --no-compress --flash_mode dout --flash_freq 80m --flash_size 32MB 0x1000000 something.bin  then you will find header be overrided
3. add an partittion larger than 0x1000000 in partittion.csv and build
4. just boot up and see the uart output shows you have a 2MB Flash
5. with parttion in latter 16M of flash and format &mount as fatfs ,or just perform a partition read/write/erase,all work as it should be
6. try to map an 16Mbyte partition and will return NO_MEM because get_mmu_region never gave you more than 256 mmu entry

Debug Logs

I already patch a lot at my idf clone …so it looks likes below

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x18 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
Octal Flash Mode Enabled
For OPI Flash, Use Default Flash Boot Mode
mode:SLOW_RD, clock div:1
load:0x3fcd0108,len:0x21b8
load:0x403b6000,len:0xd50
load:0x403ba000,len:0x3498
entry 0x403b6298
I (36) boot: ESP-IDF v5.0-dev-1429-ga10d9abba0-dirty 2nd stage bootloader
I (36) boot: compile time 05:01:58
D (36) bootloader_flash: non-XMC chip detected by SFDP Read (87), skip.
D (43) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
I (49) boot: chip revision: 0
I (53) boot.esp32s3: magic e9
I (57) boot.esp32s3: segments 03
I (61) boot.esp32s3: spi_mode 03
I (65) boot.esp32s3: spi_speed 0f
I (69) boot.esp32s3: spi_size 05
I (72) boot.esp32s3: Boot SPI Speed : 80MHz
I (77) boot.esp32s3: SPI Mode       : SLOW READ
I (83) boot.esp32s3: SPI Mode  (reg): 0x002c0008
I (88) boot.esp32s3: SPI Flash Size : 32MB
D (93) boot: Enabling RTCWDT(9000 ms)
I (96) boot: Enabling RNG early entropy source...
D (102) bootloader_flash: mmu set paddr=00000000 count=1 size=c00 src_addr=8000 src_addr_aligned=0
D (111) boot: mapped partition table 0x8000 at 0x3c008000
D (116) flash_parts: partition table verified, 5 entries
I (122) boot: Partition Table:
I (125) boot: ## Label            Usage          Type ST Offset   Length
D (133) boot: load partition table entry 0x3c008000
D (138) boot: type=1 subtype=2
I (141) boot:  0 nvs              WiFi data        01 02 00009000 00006000
D (148) boot: load partition table entry 0x3c008020
D (153) boot: type=1 subtype=1
I (156) boot:  1 phy_init         RF data          01 01 0000f000 00001000
D (164) boot: load partition table entry 0x3c008040
D (169) boot: type=0 subtype=0
I (172) boot:  2 factory          factory app      00 00 00010000 00100000
D (180) boot: load partition table entry 0x3c008060
D (185) boot: type=1 subtype=81
I (188) boot:  3 storage          Unknown data     01 81 01000000 01000000
I (195) boot: End of partition table
D (200) boot: Trying partition index -1 offs 0x10000 size 0x100000
D (206) esp_image: reading image header @ 0x10000
D (211) bootloader_flash: mmu set block paddr=0x00010000 (was 0xffffffff)
D (217) esp_image: image header: 0xe9 0x07 0x03 0x05 4037993c
V (223) esp_image: loading segment header 0 at offset 0x10018
V (229) esp_image: segment data length 0x17f00 data starts 0x10020
V (235) esp_image: segment 0 map_segment 1 segment_data_offs 0x10020 load_addr 0x3c080020
I (244) esp_image: segment 0: paddr=00010020 vaddr=3c080020 size=17f00h ( 98048) map
D (252) esp_image: free data page_count 0x0000003f
D (257) bootloader_flash: mmu set paddr=00010000 count=2 size=17f00 src_addr=10020 src_addr_aligned=10000
V (290) esp_image: loading segment header 1 at offset 0x27f20
D (290) bootloader_flash: mmu set block paddr=0x00020000 (was 0xffffffff)
V (291) esp_image: segment data length 0x46ac data starts 0x27f28
V (297) esp_image: segment 1 map_segment 0 segment_data_offs 0x27f28 load_addr 0x3fc9d6e0
I (306) esp_image: segment 1: paddr=00027f28 vaddr=3fc9d6e0 size=046ach ( 18092) load
D (314) esp_image: free data page_count 0x0000003f
D (319) bootloader_flash: mmu set paddr=00020000 count=1 size=46ac src_addr=27f28 src_addr_aligned=20000
V (333) esp_image: loading segment header 2 at offset 0x2c5d4
D (334) bootloader_flash: mmu set block paddr=0x00020000 (was 0xffffffff)
V (341) esp_image: segment data length 0x3a3c data starts 0x2c5dc
V (347) esp_image: segment 2 map_segment 0 segment_data_offs 0x2c5dc load_addr 0x40378000
I (356) esp_image: segment 2: paddr=0002c5dc vaddr=40378000 size=03a3ch ( 14908) load
D (364) esp_image: free data page_count 0x0000003f
D (369) bootloader_flash: mmu set paddr=00020000 count=2 size=3a3c src_addr=2c5dc src_addr_aligned=20000
V (383) esp_image: loading segment header 3 at offset 0x30018
D (384) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
V (391) esp_image: segment data length 0x7cd3c data starts 0x30020
V (398) esp_image: segment 3 map_segment 1 segment_data_offs 0x30020 load_addr 0x42000020
I (406) esp_image: segment 3: paddr=00030020 vaddr=42000020 size=7cd3ch (511292) map
D (414) esp_image: free data page_count 0x0000003f
D (419) bootloader_flash: mmu set paddr=00030000 count=8 size=7cd3c src_addr=30020 src_addr_aligned=30000
V (550) esp_image: loading segment header 4 at offset 0xacd5c
D (550) bootloader_flash: mmu set block paddr=0x000a0000 (was 0xffffffff)
V (552) esp_image: segment data length 0x11c9c data starts 0xacd64
V (558) esp_image: segment 4 map_segment 0 segment_data_offs 0xacd64 load_addr 0x4037ba3c
I (566) esp_image: segment 4: paddr=000acd64 vaddr=4037ba3c size=11c9ch ( 72860) load
D (575) esp_image: free data page_count 0x0000003f
D (579) bootloader_flash: mmu set paddr=000a0000 count=2 size=11c9c src_addr=acd64 src_addr_aligned=a0000
V (609) esp_image: loading segment header 5 at offset 0xbea00
D (609) bootloader_flash: mmu set block paddr=0x000b0000 (was 0xffffffff)
V (611) esp_image: segment data length 0x10 data starts 0xbea08
V (617) esp_image: segment 5 map_segment 0 segment_data_offs 0xbea08 load_addr 0x50000000
I (625) esp_image: segment 5: paddr=000bea08 vaddr=50000000 size=00010h (    16) load
D (633) esp_image: free data page_count 0x0000003f
D (638) bootloader_flash: mmu set paddr=000b0000 count=1 size=10 src_addr=bea08 src_addr_aligned=b0000
V (648) esp_image: loading segment header 6 at offset 0xbea18
D (654) bootloader_flash: mmu set block paddr=0x000b0000 (was 0xffffffff)
V (660) esp_image: segment data length 0x28 data starts 0xbea20
V (666) esp_image: segment 6 map_segment 0 segment_data_offs 0xbea20 load_addr 0x600fe000
I (675) esp_image: segment 6: paddr=000bea20 vaddr=600fe000 size=00028h (    40) load
D (683) esp_image: free data page_count 0x0000003f
D (688) bootloader_flash: mmu set paddr=000b0000 count=1 size=28 src_addr=bea20 src_addr_aligned=b0000
V (697) esp_image: image start 0x00010000 end of last section 0x000bea48
D (704) bootloader_flash: mmu set block paddr=0x000b0000 (was 0xffffffff)
D (711) boot: Calculated hash: 5eac53517a2714dda6568d16dab7950b23e799570413871a181fc07ca6326ec7
I (729) boot: Loaded app from partition at offset 0x10000
I (729) boot: Disabling RNG early entropy source...
D (731) boot: Mapping segment 0 as DROM
D (735) boot: Mapping segment 3 as IROM
D (739) boot: calling set_cache_and_start_app
D (744) boot: configure drom and irom and start
V (748) boot: d mmu set paddr=00010000 vaddr=3c080000 size=98048 n=2
V (755) boot: rc=0
V (757) boot: i mmu set paddr=00030000 vaddr=42000000 size=511292 n=8
V (763) boot: rc=0
D (765) boot: start: 0x4037993c
W (780) FLASH: DO NOT USE FOR MASS PRODUCTION! Timing parameters will be updated in future IDF version.                                                                           
I (781) opi psram: vendor id    : 0x0d (AP)
I (784) opi psram: dev id       : 0x02 (generation 3)
I (789) opi psram: density      : 0x03 (64 Mbit)
I (795) opi psram: good-die     : 0x01 (Pass)
I (800) opi psram: Latency      : 0x01 (Fixed)
I (805) opi psram: VCC          : 0x00 (1.8V)
I (810) opi psram: SRF          : 0x01 (Fast Refresh)
I (816) opi psram: BurstType    : 0x01 (Hybrid Wrap)
I (821) opi psram: BurstLen     : 0x01 (32 Byte)
I (826) opi psram: Readlatency  : 0x02 (10 cycles@Fixed)
I (832) opi psram: DriveStrength: 0x00 (1/1)
W (837) PSRAM: DO NOT USE FOR MASS PRODUCTION! Timing parameters will be updated in future IDF version.                                                                           
I (848) spiram: Found 8MB SPI RAM device
I (852) spiram: Speed: 80MHz
I (856) spiram: Initialized, cache is in normal (1-core) mode.
I (862) spiram: mapping psram at 0x3d800000 size 8388608 n=128
I (869) cpu_start: Pro cpu up.
I (872) cpu_start: Starting app cpu, entry point is 0x40379890
I (0) cpu_start: App cpu up.
I (1292) spiram: SPI SRAM memory test OK
I (1301) cpu_start: Pro cpu start user code
I (1301) cpu_start: cpu freq: 240000000 Hz
I (1301) cpu_start: Application information:
I (1304) cpu_start: Project name:     iperf
I (1309) cpu_start: App version:      1
I (1313) cpu_start: Compile time:     Feb  9 2022 05:02:45
I (1320) cpu_start: ELF file SHA256:  38b3c29605f2bf56...
I (1326) cpu_start: ESP-IDF:          v5.0-dev-1429-ga10d9abba0-dirty
I (1333) heap_init: Initializing. RAM available for dynamic allocation:
I (1340) heap_init: At 3FCA59D0 len 0003A630 (233 KiB): D/IRAM
I (1347) heap_init: At 3FCE0000 len 0000EE34 (59 KiB): STACK/DRAM
I (1353) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM
I (1360) heap_init: At 600FE028 len 00001FD8 (7 KiB): RTCRAM
I (1366) spiram: Adding pool of 8192K of external SPI memory to heap allocator
I (1374) spi_flash: detected chip: mxic (opi)
I (1379) spi_flash: flash io: opi_dtr
I (1384) sleep: Configure to isolate all GPIO pins in sleep state
I (1390) sleep: Enable automatic switching of GPIO sleep configuration
I (1398) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (1413) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (1430) example: found partition
Erase 0 MB
Erase 1 MB
Erase 2 MB
Erase 3 MB
Erase 4 MB
Erase 5 MB
Erase 6 MB
Erase 7 MB
Erase 8 MB
Erase 9 MB
Erase 10 MB
Erase 11 MB
Erase 12 MB
Erase 13 MB
Erase 14 MB
Erase 15 MB                                                                              
writing at 0xffff00                                                                      
done                                                                                     
Cache_Get_DROM_MMU_End() returned:0x00000400                                             
E (180146) example: mmap return :0                                                       
I (180148) example: Mapped partition to data memory address 0x3c0a0000                   
reading at 0xfffc00                                                                      
done                                                                                     
I (208503) example: Data matches                                                         
I (208507) example: Unmapped partition from data memory                                  
I (208513) example: Example end                                                          
I (208516) example: Mounting FAT filesystem                                              
W (209426) vfs_fat_spiflash: f_mount failed (13)
I (209426) vfs_fat_spiflash: Formatting FATFS partition, allocation unit size=4096
I (209743) vfs_fat_spiflash: Mounting again
I (209743) example: Opening file
I (209924) example: File written
I (209924) example: Reading file
I (209925) example: Read from file: 'This is written by the device'

    const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA, ESP_PARTITION_SUBTYPE_ANY, "storage");
    assert(partition != NULL);
    ESP_LOGI(TAG, "found partition");
//    static const char store_data[] = "ESP-IDF Partition Memory Map Example";

    // Prepare data to be read later using the mapped address
    for(int i=0;i<16;i++)
    {
        printf("Erase %d MB\n",i);
        ESP_ERROR_CHECK(esp_partition_erase_range(partition, i*1024*1024, 1024*1024));
    }
//    ESP_ERROR_CHECK(esp_partition_write(partition, 0, store_data, sizeof(store_data)));
//    ESP_LOGI(TAG, "Written sample data to partition: %s", store_data);
    int countint_0=0;
    int databuffer[64];
    for(int i=0;i<16*1024*1024;i+=4*64){
        printf("writing at %p\r",(void*)i);
        for(int j=0;j<64;j++)
        {
            databuffer[j]=(countint_0)*(countint_0)^0x5aa5a55a;
            countint_0++;
        }
        esp_partition_write(partition, i, databuffer, sizeof(int)*64);
    }
    printf("\ndone\n");

    const void *map_ptr;
    spi_flash_mmap_handle_t map_handle;

    uint32_t Cache_Get_DROM_MMU_End(void);
    printf("Cache_Get_DROM_MMU_End() returned:0x%08x\n",Cache_Get_DROM_MMU_End());
    esp_err_t ret2 = esp_partition_mmap(partition, 0, partition->size, SPI_FLASH_MMAP_DATA, &map_ptr, &map_handle);
    // Map the partition to data memory
    ESP_LOGE(TAG,"mmap return :%d",ret2);
    ESP_ERROR_CHECK(ret2);
    ESP_LOGI(TAG, "Mapped partition to data memory address %p", map_ptr);

    // Read back the written verification data using the mapped memory pointer
//    char read_data[sizeof(store_data)];
//    memcpy(read_data, map_ptr, sizeof(read_data));
//    ESP_LOGI(TAG, "Read sample data from partition using mapped memory: %s", (char*) read_data);

//    assert(strcmp(store_data, read_data) == 0);
    int countint_1=0;
    for(int i=0;i<16*1024*1024;i+=4){
        if((i%1024)==0)printf("reading at %p\r",(void*)i);
        int bytetowrite=(countint_1)*(countint_1)^0x5aa5a55a;
        countint_1++;
        if(((int*)map_ptr)[i/4]!=bytetowrite)
            break;
    }
    printf("\ndone\n");

Other items if possible