Go to file
2023-07-18 14:41:59 +02:00
assets updated animation gif 2022-10-10 17:10:34 -07:00
config "../exp" -> "./exp" 2023-04-07 13:33:06 +02:00
datasets fix: fucking syntax error, you kidding me ? 2023-04-07 13:32:24 +02:00
docker add docker 2023-02-20 00:27:19 -05:00
models "../exp" -> "./exp" 2023-04-07 13:33:06 +02:00
script fix: change batch size to fit on bigviz 2023-04-07 13:32:44 +02:00
third_party fix: apply same patches as PVD 2023-04-07 10:15:59 +02:00
trainers add scripts to train with clip feat 2023-04-03 17:03:27 -04:00
utils "../exp" -> "./exp" 2023-04-07 13:33:06 +02:00
.gitignore init 2023-01-23 00:14:49 -05:00
build_pkg.py init 2023-01-23 00:14:49 -05:00
default_config.py init 2023-01-23 00:14:49 -05:00
demo.py init 2023-01-23 00:14:49 -05:00
environment.yml fix: missed some deps 2023-04-07 13:31:31 +02:00
LICENSE.txt add license 2023-01-24 11:10:30 -05:00
README.md fix: add link to checkpoints 2023-07-18 14:41:59 +02:00
train_dist.py "../exp" -> "./exp" 2023-04-07 13:33:06 +02:00

LION: Latent Point Diffusion Models for 3D Shape Generation

NeurIPS 2022



  • add pointclouds rendering code used for paper figure, see utils/render_mitsuba_pc.py
  • When opening an issue, please add @ZENGXH so that I can reponse faster!


  • Dependencies:

    • CUDA 11.6
  • Setup the environment Install from conda file

        mamba env create -f environment.yml
        # mamba env update -f environment.yml 
        conda activate LION 
        # Install some other packages (use proxy)
        pip install git+https://github.com/openai/CLIP.git 
        # build some packages first (optional)
        export CUDA_HOME=/usr/local/cuda # just in case rosetta cucks you
        module load compilers
        module load mpfr
        python build_pkg.py

    Tested with conda version 22.9.0

  • Using Docker

    • build the docker with bash ./docker/build_docker.sh
    • launch the docker with bash ./docker/run.sh


run python demo.py, will load the released text2shape model on hugging face and generate a chair point cloud. Download checkpoints from HuggingFace Hub

Released checkpoint and samples

  • will be release soon
  • after download, run the checksum with python ./script/check_sum.py ./lion_ckpt.zip
  • put the downloaded file under ./lion_ckpt/



  • ShapeNet can be downloaded here.
  • Put the downloaded data as ./data/ShapeNetCore.v2.PC15k or edit the pointflow entry in ./datasets/data_path.py for the ShapeNet dataset path.

train VAE

  • run bash ./script/train_vae.sh $NGPU (the released checkpoint is trained with NGPU=4 on A100)
  • if want to use comet to log the experiment, add .comet_api file under the current folder, write the api key as {"api_key": "${COMET_API_KEY}"} in the .comet_api file

train diffusion prior

  • require the vae checkpoint
  • run bash ./script/train_prior.sh $NGPU (the released checkpoint is trained with NGPU=8 with 2 node on V100)

train diffusion prior with clip feat

  • this script trains model for single-view-reconstruction or text2shape task
    • the idea is that we take the encoder and decoder trained on the data as usual (without conditioning input), and when training the diffusion prior, we feed the clip image embedding as conditioning input: the shape-latent prior model will take the clip embedding through AdaGN layer.
  • require the vae checkpoint trained above
  • require the rendered ShapeNet data, you can render yourself or download it from here
    • put the rendered data as ./data/shapenet_render/ or edit the clip_forge_image entry in ./datasets/data_path.py
    • the img data will be read under ./datasets/pointflow_datasets.py with the render_img_path, you may need to cutomize this variable depending of the folder structure
  • run bash ./script/train_prior_clip.sh $NGPU

(Optional) monitor exp

  • (tested) use comet-ml: need to add a file .comet_api under this LION folder, example of the .comet_api file:
{"api_key": "...", "project_name": "lion", "workspace": "..."}
  • (not tested) use wandb: need to add a .wandb_api file, and set the env variable export USE_WB=1 before training
{"project": "...", "entity": "..."}
  • (not tested) use tensorboard, set the env variable export USE_TFB=1 before training
  • see the utils/utils.py files for the details of the experiment logger; I usually use comet-ml for my experiments

evaluate a trained prior

  • download the test data (Table 1) from here, unzip and put it as ./datasets/test_data/
  • download the released checkpoint from above
bash ./script/eval.sh $checkpoint  # will take 1-2 hour 

other test data

  • ShapeNet-Vol test data:
  • table 21 and table 20, point-flow test data

Evaluate the samples with the 1-NNA metrics

  • download the test data from here, unzip and put it as ./datasets/test_data/
  • run python ./script/compute_score.py (Note: for ShapeNet-Vol data and table 21, 20, need to set norm_box=True)


    title={LION: Latent Point Diffusion Models for 3D Shape Generation},
        author={Xiaohui Zeng and Arash Vahdat and Francis Williams and Zan Gojcic and Or Litany and Sanja Fidler and Karsten Kreis},
        booktitle={Advances in Neural Information Processing Systems (NeurIPS)},