This commit is contained in:
xzeng 2022-09-16 17:58:03 -04:00
parent 2fbb7aced9
commit 1cf956c667
49 changed files with 53 additions and 34 deletions

BIN
assets/LION_interp.mp4 Normal file

Binary file not shown.

BIN
assets/airplane_voxel.mp4 Normal file

Binary file not shown.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 244 KiB

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 217 KiB

After

Width:  |  Height:  |  Size: 2.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 158 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 123 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 17 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

View file

@ -294,7 +294,30 @@ pre {
width: 100%; width: 100%;
} }
} }
* {
box-sizing: border-box;
}
.column {
text-align: center;
float: left;
width: 33.33%;
padding: 5px;
}
/* Clearfix (clear floats) */
.row::after {
content: "";
clear: both;
display: table;
}
/* Responsive layout - makes the three columns stack on top of each other instead of next to each other */
@media screen and (max-width: 500px) {
.column {
width: 100%;
}
}
</style> </style>
<script type="text/javascript" src="../js/hidebib.js"></script> <script type="text/javascript" src="../js/hidebib.js"></script>
@ -327,11 +350,11 @@ pre {
<center> <center>
<div class="author-row-new"> <div class="author-row-new">
<a href="https://www.cs.utoronto.ca/~xiaohui/">Xiaohui Zeng<sup>1,2,3</sup></a>, <a href="https://www.cs.utoronto.ca/~xiaohui/">Xiaohui Zeng<sup>1,2,3</sup></a>,
<a href="http://latentspace.cc/">Arash Vahdat<sup>1</sup></a>,
<a href="https://www.fwilliams.info/">Francis Williams<sup>1</sup></a>, <a href="https://www.fwilliams.info/">Francis Williams<sup>1</sup></a>,
<a href="https://zgojcic.github.io/">Zan Gojcic<sup>1</sup></a>, <a href="https://zgojcic.github.io/">Zan Gojcic<sup>1</sup></a>,
<a href="https://orlitany.github.io/">Or Litany<sup>1</sup></a>, <a href="https://orlitany.github.io/">Or Litany<sup>1</sup></a>,
<a href="https://www.cs.utoronto.ca/~fidler/">Sanja Fidler<sup>1,2,3</sup></a>, <a href="https://www.cs.utoronto.ca/~fidler/">Sanja Fidler<sup>1,2,3</sup></a>,
<a href="http://latentspace.cc/">Arash Vahdat<sup>1</sup></a>,
<a href="https://karstenkreis.github.io/">Karsten Kreis<sup>1</sup></a> <a href="https://karstenkreis.github.io/">Karsten Kreis<sup>1</sup></a>
</div> </div>
</center> </center>
@ -513,30 +536,18 @@ pre {
</div> </div>
<center> <center>
<figure style="width: 40%;"> <figure style="width: 80%;">
<video class="centered" width="30%" controls autoplay muted playsinline class="video-background " > <video class="centered" width="80%" controls muted playsinline class="video-background " >
<source src="assets/airplane_voxel_input_165.mp4#t=0.01" type="video/mp4"> <source src="assets/airplane_voxel.mp4#t=0.14" type="video/mp4">
Your browser does not support the video tag. Your browser does not support the video tag.
</video> </video>
<p class="caption" style="margin-bottom: 24px;" width="30%"> <p class="caption" style="margin-bottom: 24px;" width="30%">
Input voxel grid Left: Input voxel grid. Right: two point clouds generated by LION and the reconstructed mesh.
<!-- Voxel-guided synthesis experiments, on different categories. We run diffuse-denoise in latent space to generate diverse plausible clean shapes (first row, left plane: 250 diffuse-denoise steps; first row, right plane: 200 steps;) --> <!-- Voxel-guided synthesis experiments, on different categories. We run diffuse-denoise in latent space to generate diverse plausible clean shapes (first row, left plane: 250 diffuse-denoise steps; first row, right plane: 200 steps;) -->
</p> </p>
</figure> </figure>
<figure style="width: 40%;">
<video class="centered" width="30%" controls autoplay muted playsinline class="video-background " >
<source src="assets/airplane_voxel_output_165_D200_1.mp4#t=0.001" type="video/mp4">
Your browser does not support the video tag.
</video>
<p class="caption" style="margin-bottom: 24px;" width="30%">
Output point clouds and mesh.
</p>
</figure>
</center> </center>
<br> <br>
@ -545,30 +556,38 @@ pre {
<p>LION can interpolate two shapes by traversing the latent space. The generated shapes are clean and semantically plausible along the entire interpolation path. </p> <p>LION can interpolate two shapes by traversing the latent space. The generated shapes are clean and semantically plausible along the entire interpolation path. </p>
</div> </div>
<figure> <figure>
<video class="centered" width="40%" autoplay loop muted playsinline class="video-background " > <video class="centered" width="100%" controls loop muted playsinline class="video-background " >
<source src="assets/LION_latent_interp.mp4#t=0.001" type="video/mp4"> <source src="assets/LION_interp.mp4#t=0.001" type="video/mp4">
Your browser does not support the video tag. Your browser does not support the video tag.
</video> </video>
<!-- <p class="caption"> <p class="caption">
The sequence above is generated by interpolating the latent variable of the source and target shapes (using the probability flow ODE formulation). Left most shape: the source shape. Right most shape: the target shape. The shapes in middle are interpolated results between source and target shape.
</p> --> </p>
</figure> </figure>
<br> <br>
<h3> Per-sample Text-driven Texture Synthesis </h3> <h3> Per-sample Text-driven Texture Synthesis </h3>
<div class="flex-row"> <div class="flex-row">
<p> <p>
We apply Text2mesh on some generated meshes from LION to additionally synthesize textures in a text-driven manner, leveraging CLIP. We apply Text2mesh on some generated meshes from LION to additionally synthesize textures in a text-driven manner, leveraging CLIP. The original mesh is generated by LION.
</p> </p>
</div> </div>
<figure style="width: 100%;"> <div class="row">
<a> <div class="column">
<img width="100%" src="assets/text2mesh_airplane.png"> <img width="50%" src="assets/text2mesh/strawberries_airplane-rec_3.jpg">
</a> <figcaption align = "center">An airplane made of strawberry</figcaption>
<p class="caption" style="margin-bottom: 24px;"> </div>
Text2mesh results with text prompt "a airplane made of __". All prompts are applied on the same generated shapes. The original mesh is generated by LION. <div class="column">
</p> <img width="50%" src="assets/text2mesh/watermelons_airplane-rec_3.jpg">
</figure> <figcaption align = "center">An airplane made of watermelon</figcaption>
</div>
<div class="column">
<img width="50%" src="assets/text2mesh/fabric_leather_airplane-rec_3.jpg">
<figcaption align = "center">An airplane made of fabric leather</figcaption>
</div>
</div>
<br> <br>
<h3> Single View Reconstruction </h3> <h3> Single View Reconstruction </h3>
@ -584,7 +603,7 @@ pre {
</div> </div>
<figure style="width: 50%;"> <figure style="width: 50%;">
<a> <a>
<img width="100%" src="assets/svr_chair.png"> <img width="100%" src="assets/svr/img2shape_mitsuba_full.jpg">
</a> </a>
<p class="caption" style="margin-bottom: 24px;"> <p class="caption" style="margin-bottom: 24px;">
Single view reconstruction from RGB images of chair. For each input image, LION can generate multi-modal outputs. Single view reconstruction from RGB images of chair. For each input image, LION can generate multi-modal outputs.
@ -592,7 +611,7 @@ pre {
</figure> </figure>
<figure style="width: 50%;"> <figure style="width: 50%;">
<a> <a>
<img width="100%" src="assets/svr_car.png"> <img width="100%" src="assets/svr/img2shape_cari2s_mm_mitsuba_full.jpg">
</a> </a>
<p class="caption" style="margin-bottom: 24px;"> <p class="caption" style="margin-bottom: 24px;">
Single view reconstruction from RGB images of car. For each input image, LION can generate multi-modal outputs. Single view reconstruction from RGB images of car. For each input image, LION can generate multi-modal outputs.
@ -600,7 +619,7 @@ pre {
</figure> </figure>
<figure style="width: 100%;"> <figure style="width: 100%;">
<a> <a>
<img width="100%" src="assets/svr_car_more.png"> <img width="100%" src="assets/svr/img2shape_cari2s_mitsuba_full.jpg">
</a> </a>
<p class="caption" style="margin-bottom: 24px;"> <p class="caption" style="margin-bottom: 24px;">
More single view reconstruction from RGB images of car. More single view reconstruction from RGB images of car.