PVD/README.md

95 lines
2.8 KiB
Markdown
Raw Normal View History

2021-10-19 20:54:46 +00:00
# Shape Generation and Completion Through Point-Voxel Diffusion
2023-04-06 07:24:37 +00:00
<img src="assets/pvd_teaser.gif" width="100%"/>
2021-10-19 20:54:46 +00:00
2021-11-01 07:12:02 +00:00
[Project](https://alexzhou907.github.io/pvd) | [Paper](https://arxiv.org/abs/2104.03670)
2021-10-19 20:54:46 +00:00
2021-11-01 07:12:02 +00:00
Implementation of Shape Generation and Completion Through Point-Voxel Diffusion
2021-10-19 20:54:46 +00:00
2021-11-01 07:12:02 +00:00
[Linqi Zhou](https://alexzhou907.github.io), [Yilun Du](https://yilundu.github.io/), [Jiajun Wu](https://jiajunwu.com/)
2021-10-19 20:54:46 +00:00
## Requirements:
2023-04-06 07:24:37 +00:00
Install Python environment
```bash
module load conda
module load artifactory
2023-04-06 14:19:01 +00:00
mamba env create --file environment.yml
# mamba env update --file environment.yml
2023-04-11 07:30:14 +00:00
conda activate PVD
module load gcc/11.2.0
module load mpfr/4.0.2
2021-11-01 07:12:02 +00:00
```
Install PyTorchEMD by
2023-04-06 07:24:37 +00:00
```bash
2021-11-01 07:12:02 +00:00
cd metrics/PyTorchEMD
python setup.py install
2023-04-06 07:24:37 +00:00
cp build/**/emd_cuda.cpython-*-x86_64-linux-gnu.so .
2021-11-01 07:12:02 +00:00
```
2021-10-19 20:54:46 +00:00
The code was tested on Unbuntu with Titan RTX.
2021-11-01 07:12:02 +00:00
## Data
2021-10-19 20:54:46 +00:00
2023-04-06 14:19:01 +00:00
For generation, we use ShapeNet point cloud, which can be downloaded [here](https://github.com/stevenygd/PointFlow). Or at `/data/users/lfainsin/ShapeNetCore.v2.PC15k.zip`.
2021-10-19 20:54:46 +00:00
2021-11-01 07:12:02 +00:00
For completion, we use ShapeNet rendering provided by [GenRe](https://github.com/xiumingzhang/GenRe-ShapeHD).
We provide script `convert_cam_params.py` to process the provided data.
For training the model on shape completion, we need camera parameters for each view
which are not directly available. To obtain these, simply run
2021-10-19 20:54:46 +00:00
```bash
2021-11-01 07:12:02 +00:00
$ python convert_cam_params.py --dataroot DATA_DIR --mitsuba_xml_root XML_DIR
```
which will create `..._cam_params.npz` in each provided data folder for each view.
## Pretrained models
Pretrained models can be downloaded [here](https://drive.google.com/drive/folders/1Q7aSaTr6lqmo8qx80nIm1j28mOHAHGiM?usp=sharing). Or at `/data/users/lfainsin/PVD/checkpoints`.
2021-11-01 07:12:02 +00:00
## Training:
```bash
$ python train_generation.py --category car|chair|airplane
2021-10-19 20:54:46 +00:00
```
Please refer to the python file for optimal training parameters.
2021-11-01 07:12:02 +00:00
## Testing:
```bash
$ python train_generation.py --category car|chair|airplane --model MODEL_PATH
```
2021-10-19 20:54:46 +00:00
## Results
2021-11-01 07:21:12 +00:00
Some generation and completion results are as follows.
2023-04-06 07:24:37 +00:00
<img src="assets/gen_comp.gif" width="100%"/>
2021-11-01 07:21:12 +00:00
2021-11-01 07:26:00 +00:00
Multimodal completion on a ShapeNet chair.
2023-04-06 07:24:37 +00:00
<img src="assets/mm_shapenet.gif" width="100%"/>
2021-10-19 20:54:46 +00:00
2021-11-01 07:21:12 +00:00
Multimodal completion on PartNet.
2023-04-06 07:24:37 +00:00
<img src="assets/mm_partnet.gif" width="100%"/>
2021-11-01 07:21:12 +00:00
2021-11-01 07:26:00 +00:00
Multimodal completion on two Redwood 3DScan chairs.
2023-04-06 07:24:37 +00:00
<img src="assets/mm_redwood.gif" width="100%"/>
2021-10-19 20:54:46 +00:00
## Reference
2023-04-06 07:24:37 +00:00
```bibtex
2021-11-01 07:21:12 +00:00
@inproceedings{Zhou_2021_ICCV,
2023-04-06 07:24:37 +00:00
author = {Zhou, Linqi and Du, Yilun and Wu, Jiajun},
title = {3D Shape Generation and Completion Through Point-Voxel Diffusion},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2021},
pages = {5826-5835}
2021-10-19 20:54:46 +00:00
}
```
## Acknowledgement
2021-11-01 07:21:12 +00:00
For any questions related to codes and experiment setting, please contact [Linqi Zhou](linqizhou@stanford.edu) and [Yilun Du](yilundu@mit.edu).