ZLUDA: Meshroom 2023.3.0 fails at DepthMap stage

Hardware : 
        Detected core count : 16
        OpenMP will use 16 cores
        Detected available memory : 22375 Mo

[20:48:00.928483][warning] Cannot get available memory information for CUDA gpu device 0:
         (error code: 801) cudaErrorNotSupported
[20:48:00.918372][warning] CUDA-Enabled GPU.
Device information:
        - id:                      0
        - name:                    AMD Radeon RX 6800 XT [ZLUDA]
        - compute capability:      8.8
        - clock frequency (kHz):   2575000
        - total device memory:     16368 MB 
        - device memory available: 0 MB 
        - per-block shared memory: 65536
        - warp size:               32
        - max threads per block:   1024
        - max threads per SM(X):   2048
        - max block sizes:         {1024,1024,1024}
        - max grid sizes:          {2147483647,65536,65536}
        - max 2D array texture:    {16384,16384}
        - max 3D array texture:    {16384,16384,8192}
        - max 2D linear texture:   {0,0,0}
        - max 2D layered texture:  {16384,16384,65535}
        - number of SM(x)s:        36
        - registers per SM(x):     65536
        - registers per block:     65536
        - concurrent kernels:      yes
        - mapping host memory:     yes
        - unified addressing:      yes
        - texture alignment:       256 byte
        - pitch alignment:         256 byte
CUDA build version: 11.3
[20:48:00.928550][info] Supported CUDA-Enabled GPU detected.
[20:48:00.983995][info] Found 1 image dimension(s): 
[20:48:00.984029][info]         - [4032x3024]
[20:48:00.996774][info] Overall maximum dimension: [4032x3024]
[20:48:00.996837][info] Tiling information: 
        - parameters: 
              - buffer width:  1024 px
              - buffer height: 1024 px
              - padding: 64 px
        - maximum downscale:  4
        - maximum image width:  2016 px
        - maximum image height: 1512 px
        - maximum effective tile width:  896 px
        - maximum effective tile height: 896 px
        - # tiles on X-side: 3
        - # tiles on Y-side: 2
        - effective tile width:  672 px
        - effective tile height: 756 px
        - tile list: 
           - tile (1/6) size: 736x820 px, roi: [x: 0-736, y: 0-820]
           - tile (2/6) size: 736x756 px, roi: [x: 0-736, y: 756-1512]
           - tile (3/6) size: 736x820 px, roi: [x: 672-1408, y: 0-820]
           - tile (4/6) size: 736x756 px, roi: [x: 672-1408, y: 756-1512]
           - tile (5/6) size: 672x820 px, roi: [x: 1344-2016, y: 0-820]
           - tile (6/6) size: 672x756 px, roi: [x: 1344-2016, y: 756-1512]
[20:48:00.996860][info] SGM parameters:
        - scale: 2
        - stepXY: 2
[20:48:00.996880][info] Refine parameters:
        - scale: 1
        - stepXY: 1
[20:48:00.996903][info] Number of GPU devices: 1, number of CPU threads: 16
================================================================================
====================== Command line failed with an error =======================
Could not allocate pinned host memory in /opt/AliceVision_git/src/aliceVision/depthMap/cuda/host/memory.hpp:395, operation not supported: operation not supported
================================================================================

Relevant line of code: https://github.com/alicevision/AliceVision/blob/develop/src/aliceVision/depthMap/cuda/host/memory.hpp

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 21 (5 by maintainers)

Most upvoted comments

I was wrong, this was suffering.

Screenshot 2024-03-26 030047

Meshroom seems to work now. Since it is a rather big change and I don’t really know how to use Meshroom I’d like someone who is interested in this issue to give it a try:

  • It works only on Windows, because of an underlying ROCm issue. You need a recent Adrenalin driver (I’m using 24.2.1, I used 23.X previously and that was too old)
  • You need to compile ZLUDA from this PR: #153
  • You also need to compile Alicevision with a small code change, in src/cmake/Dependencies.cmake change
      ExternalProject_Add(${CCTAG_TARGET}
          GIT_REPOSITORY https://github.com/alicevision/CCTag
          GIT_TAG v1.0.3
    
    to
        ExternalProject_Add(${CCTAG_TARGET}
            GIT_REPOSITORY https://github.com/alicevision/CCTag
            GIT_TAG ee7a02dc329efe1e43a4afad36cb0e37644ba957
    

I can provide the binaries, but I’d prefer if someone built from scratch to double check everything. If you have questions please ask.

I get a different error this time

Choosing device 0: AMD Radeon RX 6800 XT [ZLUDA]
terminate called after throwing an instance of 'std::runtime_error'
  what():  /tmp/AliceVisionDeps_build/popsift/src/popsift/gauss_filter.cu:245
    cudaMemcpyToSymbol failed for Gauss kernel initialization: operation not supported

However FeatureExtraction can be done on the CPU, and with a powerful system speed is not much of an issue. The real blocker is the DepthMap node, which has to use CUDA.

cuGetProcAddress returning 500 is benign pretty much always. There’s a handful of CUDA driver functions which are either too esoteric (nvsci) or just difficult to support cleanly (OS-specific interop: DirectX, EGL) where ZLUDA does not even advertise possibility of supporting them.

It’s not int the logs, but I infer that ZLUDA fails to compile a PTX module at startup, which usually is not hard to fix. I’ll have a look

To anyone still having crashes: Use CUDA 11.0, add the installation path (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin) to your system’s Path, and use this pre-compiled package by natowi.

Works on my 6750 XT.

@aesxsc @bonnyr-f5 I guess your error probably comes from DepthMap using CUDA Runtime, but not bundling it with the app. Extracting it from CUDA installer and dropping it in a path where DepthMap looks for .dll would make the error go away ~but still won’t make DepthMap work because of aforementioned ROCm Windows issue~

Edit: Sorry, I got this wrong way around. It’s Linux ROCm that is broken, not Windows one

@dngulin for some reason AMD enabled support for mipmapped arrays (textures) on Windows ROCm, but not on Linux ROCm. Meshroom relies on this functionality