DreamMat: Model Inference Output Issue | Base Color not Showing

Hi, kudos for the amazing work, and the results shown in paper looks promising! However, I’m facing issues while generating/reproducing the exact output as highlighted in the results. I have created a docker image, and uploaded the provided controlnet weights, as suggested by you.

Now, when I run the inference, the generated 3D model lacks color. Seems like the PBR materials are not generating properly, especially the Albedo output (3rd image in gif), which must include the base color. Here is the ‘a brown basketball’ evaluation.gif.

eval

I am using this command (same as guided), so I don’t know where the issue lies. Maybe the trained model is not working properly, or there is some parameter related issue. python launch.py --config configs/dreammat.yaml --test --gradio --gpu 0 system.prompt_processor.prompt="A brown basketball" system.geometry.shape_init=mesh:load/shapes/objs/basketball.obj system.geometry.shape_init_params=0.6 data.blender_generate=true

I’m aiming to achieve these results: image

Please help me resolve the issue, as I am doing research in the domain of 3D and really impressed by your work. I am looking forward to getting a quick response from your end, so I can incorporate your work in my pipeline accordingly. Thanks!

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 28 (11 by maintainers)

Most upvoted comments

Regarding the array out-of-bounds issue you encountered, this is mainly due to a division by zero when normalizing the sampled ray directions, resulting in NaN. You can resolve this problem by modifying our code. Specifically, change https://github.com/zzzyuqing/DreamMat/blob/92afaa64d5fc53002701995d1f9beef8f7ba3d12/threestudio_dreammat/threestudio/models/materials/dreammat_material.py#L443 to:

directions = directions / (directions.norm(p=2, dim=-1, keepdim=True) + 1e-6)

This modification prevents the division by zero error while ensuring correct normalization of the vectors. I used the model you provided and ran it with the following command python launch.py --config configs/dreammat.yaml --train --gradio --gpu 0 system.prompt_processor.prompt="a wooden box" system.geometry.shape_init=mesh:load/shapes/objs/box.obj trainer.max_steps=4000 system.geometry.shape_init_params=0.6 data.blender_generate=false. The result is as follows:

image

The albedo map we have stored is in linear space, so it is necessary to select the correct color space when rendering in Blender, otherwise the result may appear too dark. Additionally, different rendering engines use different methods of correction for the final rendering results, which may also lead to some variations. image

Currently, our method cannot generate clear and reasonable text on textures; early image generation models also had this issue. You might consider replacing the base stable diffusion model and training your own controlnet.

Hi Yuqing, I’m awaiting your response to my previous comment.

In the meantime, I’ll add my observations regarding conda environment setup.

The order for installation of files/packages/libraries which is mentioned in Dockerfile seems more appropriate and relevant, however, there are some things missing in that, which I’ll try to add:

  1. Create conda environment with required Python version (say 3.10.12) conda create -n DreamMat python=3.10.12

  2. Ensure correct GCC, Cmake and CUDA Toolkit versions in your environment (since they create issues during installation of Tiny CUDA NN ) and then proceed for installation of packages.

  3. Upgrade ninja package. pip install --upgrade pip setuptools ninja

  4. Install PyTorch with Cuda-11.8. pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118

  5. Install the complicated repositories first, one by one.

pip install git+https://github.com/KAIR-BAIR/nerfacc.git@v0.5.2
pip install git+https://github.com/NVlabs/tiny-cuda-nn.git#subdirectory=bindings/torch
  1. Install requirements file. Make sure to comment out the already installed or separately installed repos/packages. pip install -r requirements.txt

  2. If the torch version is changed (which it did in my case), repeat the step 4, again.

  3. Install missing dependencies, set path for eigen3 and install raytracing package

apt-get update && apt-get install -y libeigen3-dev && apt-get install libxi6
export CPLUS_INCLUDE_PATH=/usr/include/eigen3
pip install git+https://github.com/ashawkey/raytracing.git
  1. Install mediapipe package (since it was missing in requirements file) pip install mediapipe

  2. Install Blender and set its path. The path command is changed a bit

export PATH=/path_to_blender/blender-3.2.2-linux-x64:${PATH}
  1. The storage format differs. When training your own ControlNet, you can use the .safetensors format for storage, but this does not affect usage.
  2. The setting of data.blender_generate was designed for my convenience in tuning parameters, such as different iteration counts. If you want different text prompts all in the same folder without having to re-render, you can name the result folders after the mesh names, which eliminates the need for manual copying.
  3. In most cases, you can directly use the default parameters: system.geometry.shape_init_params=1.0 and system.guidance.cond_scale=1.05. The parameter shape_init_params scales the size of the input mesh, while cond_scale is the weight in csd.

I appreciate your prompt response! Basically, after setting up the repo, I just ran the DreamMat inference code from .cmd/run_examples.sh on the provided Objects (3D meshes) located in the ./load/shapes/objs. Ideally, the code shall work because I tried my best to ensure all dependencies/packages are installed. Can you please confirm from the Inference Command I’m using, if the issue can be probed from there? Or let me know any .py file, where I may dig further.

One more thing, does it have something to do with the system.geometry.shape_init_params=0.6 or any other parameters being passed in the inference command (which I pasted earlier)? Actually, the commands in cmd/run_examples.sh are for training and not testing, and I myself had to figure out what the inference commands are. So, I’m not sure if the command I used, will generate the desired results. Any guidance in this perspective would be beneficial regarding the issue observed.