projet-long/slides.md

---
theme: academic
class: text-white
coverAuthor: Laurent Fainsin, Pierre-Eliot Jourdan, Raphaëlle Monville-Letu, Jade Neav
coverBackgroundUrl: https://plus.unsplash.com/premium_photo-1673553304257-018c85e606f8?ixlib=rb-4.0.3&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8
coverBackgroundSource: unplash
coverBackgroundSourceUrl: https://unsplash.com/photos/g4I556WCJT0
coverDate: "2023-03-09"
themeConfig:
  paginationX: r
  paginationY: t
  paginationPagesDisabled:
    - 1
title: Projet Long
---

<h2 class="opacity-50" style="font-size: 1.9rem;">End of study project</h2>

<h1 style="font-size: 2.3rem;">Sphere detection and multimedia applications</h1>

---

# Contents

<div class="h-100 flex items-center text-2xl">

- Types of spheres
- Automatic sphere detection
- Lighting intensity estimation
- Lightning direction estimation

</div>

<figure class="absolute top-15 right-25 w-35">
  <img src="https://images.pexels.com/photos/13849458/pexels-photo-13849458.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1"/>
  <figcaption class="text-center">Architecture</figcaption>
</figure>

<figure class="absolute top-40 right-75 w-50">
  <img src="https://images.pexels.com/photos/3945321/pexels-photo-3945321.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1"/>
  <figcaption class="text-center">Cinema</figcaption>
</figure>

<figure class="absolute top-72 right-30 w-45">
  <img src="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTzg_yM_NbCIYXfZ55WdtFbAtaF7EUGSKSVBQ&usqp=CAU"/>
  <figcaption class="text-center">3D Reconstruction</figcaption>
</figure>

<a href="https://www.pexels.com" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">pexels</a>

---
class: text-white custombg
---

<style>
.custombg {
  background-repeat: no-repeat;
  background-position: center center;
  background-size: cover;
  background-image: url("/assets/spheres.png");
}
</style>

## Types of spheres

---
class: text-white custombg2
---

<style>
.custombg2 {
  background-repeat: no-repeat;
  background-position: center center;
  background-size: cover;
  background-image: url("https://media.caveacademy.com/wp-content/uploads/2021/05/04000307/cave_prop1002_chrome_v001_r001.jpg");
}
</style>

## Chrome sphere

<a href="https://caveacademy.com/wiki/onset-production/data-acquisition/data-acquisition-training/the-grey-the-chrome-and-the-macbeth-chart/" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">CaveAcademy</a>

---

## Acquisition techniques

<img src="/assets/capture_hdri.jpg" class="m-auto"/>

<a href="https://www.youtube.com/watch?v=kwGZa5qTeAI" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">Louis du Mont</a>

<!-- https://www.youtube.com/watch?v=HCfHQL4kLnw -->

---

## Realistic lighting

<div class="grid grid-cols-2 col-auto m-auto h-100 gap-1">
  <img src="/assets/image-026.png" class="m-auto w-full"/>
  <img src="/assets/image-027.png" class="m-auto w-full"/>
</div>

---
class: text-white custombg3
---

<style>
.custombg3 {
  background-repeat: no-repeat;
  background-position: center center;
  background-size: cover;
  background-image: url("/assets/shiny.jpg");
}
</style>


## Shiny sphere

<a href="https://caveacademy.com/wiki/onset-production/data-acquisition/data-acquisition-training/the-grey-the-chrome-and-the-macbeth-chart/" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">CaveAcademy</a>

---
class: text-white custombg4
---

<style>
.custombg4 {
  background-repeat: no-repeat;
  background-position: center center;
  background-size: cover;
  background-image: url("https://media.caveacademy.com/wp-content/uploads/2021/05/04000316/cave_prop1002_grey_v001_r001.jpg");
}
</style>

## Matte sphere

<a href="https://caveacademy.com/wiki/onset-production/data-acquisition/data-acquisition-training/the-grey-the-chrome-and-the-macbeth-chart/" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">CaveAcademy</a>

---

# Automatic detection of spheres

---

## Model

<div class="h-100 flex items-center">
  <img src="/assets/DETR.png" class="m-auto"/>
</div>

<a href="https://arxiv.org/abs/2005.12872" class="absolute bottom-0 font-extralight mb-1 mr-2 right-0 text-xs">End-to-End Object Detection with Transformers,  arXiv:2005.12872
</a>

---

## Datasets


<div class="grid grid-cols-2 col-auto m-auto h-full">
  <img src="/assets/antoine.webp" class="m-auto h-55"/>
  <img src="/assets/illumination.webp" class="m-auto h-55"/>
  <img src="/assets/compositing.webp" class="m-auto h-55"/>
  <img src="/assets/render.webp" class="m-auto h-55"/>
</div>


---

## Results

---

# Estimation of the lighting intensity in an image

---
 
## Photometric Stereo

<div class="h-100 flex items-center">
  <img src= "https://upload.wikimedia.org/wikipedia/commons/b/b5/Photometric_stereo.png" class="m-auto h-90"/>
</div>

- Estimate the surface normals of an object
- Shiny spheres $\rightarrow$ direction of the lighting

---

## Lambert law

<div class="h-100 flex items-center">
  <img src= "https://img.laserfocusworld.com/files/base/ebm/lfw/image/2019/06/1906LFW_ost_1.5d13a8a881e81.png?auto=format,compress&w=1050&h=590&fit=clip" class="m-auto h-90"/>
</div>


$I(q) = \rho(Q) \times \vec{n}(Q) \cdot \vec{s}(Q)$

$\rho(Q)$ is the albedo

$\vec{n}(Q)$ is the normal vector

$\vec{s}(Q) = \phi \times \vec{s_0}(Q)$, $\vec{s_0}(Q)$ being the direction of the lighting vector

---

## Problem formulation

$N$ lightings, $P$ pixels

$I = M \times S \times D_{\phi}$

$I \in \mathbb{R}^{P \times N} \rightarrow$ gray scale levels

$M \in \mathbb{R}^{P \times 3} \rightarrow$ the albedo and the normals (unknown)

$S \in \mathbb{R}^{3 \times N} \rightarrow$ direction of lightings

$D_{phi} = diag(\phi_1,...,\phi_{N}) \in \mathbb{R}^{ N \times N} \rightarrow$ intensities of lightings (to be determined)

---

## Algorithm 1

<div class="h-100 flex items-center">
  <img src="/assets/algo1.png" class="m-auto h-80"/>
</div>

Intensities : $[\phi_1,...,\phi_{N}]$

New values : $\phi_j + \delta$ and $\phi_j + \delta$, $j \in [1,..,N]$

Mean-squared error : $\underset{\phi_i}{\min} || I - M S D_{\phi} ||_2^2$

Update the value of $\phi_j$

Repeat previous steps

---

## Algorithm 2

Algorithm 1 $\rightarrow$ too long

$$I = M S D_{\phi} \iff M = I(S D_{\phi})^\dagger = I (S D_{\phi})^T [(S D_{\phi})(S D_{\phi})^T]^{-1}$$

Lambert law : 

$$
\begin{align*}
  I &= I (S D_{\phi})^T [(S D_{\phi})(S D_{\phi})^T]^{-1} S D_{\phi} \\
    &= I D_{\phi} S^T S^{-T} D_{\phi}^{-2} S^{-1} S D_{\phi}
\end{align*}
$$

New residual :

$$\underset{\phi_i}{\min} || I - I D_{\phi} S^T S^{-T} D_{\phi}^{-2} S^{-1} S D_{\phi} ||_2^2$$


---

## Generated images

<div class="grid grid-cols-4 col-auto h-110 m-auto">
  <img src="/assets/im2.jpg" class="m-auto h-50"/>
  <img src="/assets/im3.jpg" class="m-auto h-50"/>
  <img src="/assets/im4.jpg" class="m-auto h-50"/>
  <img src="/assets/im5.jpg" class="m-auto h-50"/>
  <img src="/assets/im12.jpg" class="m-auto h-50"/>
  <img src="/assets/im13.jpg" class="m-auto h-50"/>
  <img src="/assets/im14.jpg" class="m-auto h-50"/>
  <img src="/assets/im15.jpg" class="m-auto h-50"/>
</div>

---

## Results (1/2)

<div class="h-100 flex items-center">
  <img src="/assets/residu_4.jpg" class="m-auto w-full"/>
  <img src="/assets/residu2d_3.jpg" class="m-auto w-full"/>
</div>

---

## Results (2/2)

<div class="h-100 flex items-center">
  <img src="/assets/resultats_finaux.jpg" class="m-auto h-110"/>
</div>

---

## Real images

TODO LOLO : mettre les images comme dans les slides avec carre rouge

---

## Results

<div class="h-100 flex items-center">
  <img src="/assets/resultats_finaux_comete.jpg" class="m-auto w-full"/>
  <img src="/assets/resultats_finaux_stsernin.jpg" class="m-auto w-full"/>
</div>

---

# Automatic estimation of lighting vector 

- Creation of data 
- Estimation of light vector with matte balls 
- Training of neural networks

---

## Creation of mask

---

## Generated data with Blender

---

## Estimation of lighting vector for training

---

## Verification estimation of lighting vector 

---

## Which type of neural network ?

---

## Results

---

## Conclusion

---

## Perspectives