416 lines
16 KiB
Plaintext
416 lines
16 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "4Vhg6pn2uDlW"
|
||
},
|
||
"source": [
|
||
"# Dataset des 2 lunes\n",
|
||
"\n",
|
||
"(Avec un nouvel affichage plus joli, merci Arthur !)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "Hw8rHiTKuHL3"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"from sklearn import datasets\n",
|
||
"import matplotlib.pyplot as plt \n",
|
||
"\n",
|
||
"def generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250):\n",
|
||
" num_samples = num_lab + num_unlab + num_test\n",
|
||
" # Génération de 1000 données du dataset des 2 lunes\n",
|
||
" x, y = datasets.make_moons(n_samples=num_samples, noise=0.1, random_state=1)\n",
|
||
"\n",
|
||
" x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=num_test/num_samples, random_state=1)\n",
|
||
" x_train_lab, x_train_unlab, y_train_lab, y_train_unlab = train_test_split(x_train, y_train, test_size=num_unlab/(num_unlab+num_lab), random_state=6)\n",
|
||
"\n",
|
||
" return x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "Ww95atT6uJ4D"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250)\n",
|
||
"\n",
|
||
"print(x_train_lab.shape, x_train_unlab.shape, x_test.shape)\n",
|
||
"print(y_train_lab.shape, y_train_unlab.shape, y_test.shape)\n",
|
||
"\n",
|
||
"# Affichage des données\n",
|
||
"plt.plot(x_train_unlab[y_train_unlab==0,0], x_train_unlab[y_train_unlab==0,1], color=(0.5,0.5,0.5), marker='.', linestyle=' ')\n",
|
||
"plt.plot(x_train_unlab[y_train_unlab==1,0], x_train_unlab[y_train_unlab==1,1], color=(0.5,0.5,0.5), marker='.', linestyle=' ')\n",
|
||
"\n",
|
||
"plt.plot(x_test[y_test==0,0], x_test[y_test==0,1], 'b+')\n",
|
||
"plt.plot(x_test[y_test==1,0], x_test[y_test==1,1], 'r+')\n",
|
||
"\n",
|
||
"plt.plot(x_train_lab[y_train_lab==0,0], x_train_lab[y_train_lab==0,1], 'b.', markersize=30)\n",
|
||
"plt.plot(x_train_lab[y_train_lab==1,0], x_train_lab[y_train_lab==1,1], 'r.', markersize=30)\n",
|
||
"\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "LKODCH2luSPM"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def create_model_2moons():\n",
|
||
"\n",
|
||
" inputs = keras.Input(shape=(2,))\n",
|
||
" x = Dense(20, activation=\"relu\")(inputs)\n",
|
||
" outputs = Dense(1, activation=\"sigmoid\")(x)\n",
|
||
" model = keras.Model(inputs=inputs, outputs=outputs) \n",
|
||
"\n",
|
||
" return model"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "ea7E3-6l3_le"
|
||
},
|
||
"source": [
|
||
"# $\\Pi$-Modèle\n",
|
||
"\n",
|
||
"Nous allons maintenant tenter d'utiliser un 2nd algorithme semi-supervisé supposé être plus efficace, il s'agit de l'algorithme du $\\Pi$-Modèle, dont la version détaillée est présentée ci-dessous (en VO).\n",
|
||
"\n",
|
||
"<img src=\"https://drive.google.com/uc?id=13VhlBYwA6YIYGzKI81Jom_jTiuhOypEg\">\n",
|
||
"<caption><center> Figure 1 : Pseudo-code de l'algorithme du $\\Pi$-Modèle</center></caption>\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "6vaWDKNpYxc0"
|
||
},
|
||
"source": [
|
||
"Ci-dessous, la boucle d'entraînement détaillée est reprise et contient un squelette du code à réaliser pour implémenter le $\\Pi$-Modèle. \n",
|
||
"\n",
|
||
"**Travail à faire :** Complétez le squelette de l'algorithme du $\\Pi$-Modèle pour pouvoir tester ce nouvel algorithme."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "uVK8itsvD72s"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Nombre d'epochs de l'apprentissage\n",
|
||
"epochs = 2000\n",
|
||
"# Nombre de données non-labellisées par batch\n",
|
||
"bs_unlab = 100\n",
|
||
"# Nombre de données labellisées par batch\n",
|
||
"bs_lab = 10\n",
|
||
"# Taille du batch\n",
|
||
"batch_size = bs_lab + bs_unlab\n",
|
||
"# Valeur initiale du paramètre de contrôle de l'importance de la régularisation non-supervisée\n",
|
||
"lambda_t = 0\n",
|
||
"\n",
|
||
"# Données et modèle du problème des 2 clusters\n",
|
||
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250)\n",
|
||
"model = create_model_2moons()\n",
|
||
"\n",
|
||
"# Nombre de batches par epochs\n",
|
||
"steps_per_epochs = int(np.floor(x_train_lab.shape[0]/bs_lab))\n",
|
||
"# Instanciation d'un optimiseur et d'une fonction de coût.\n",
|
||
"optimizer = keras.optimizers.Adam(learning_rate=3e-2)\n",
|
||
"# ICI ON A BESOIN DE DEUX FONCTIONS DE COUT : \n",
|
||
"# L'une pour la partie supervisée de la perte\n",
|
||
"loss_sup = ...\n",
|
||
"# L'autre pour la partie non-supervisée de la perte\n",
|
||
"loss_unsup = ...\n",
|
||
"\n",
|
||
"# Préparation des métriques pour le suivi de la performance du modèle.\n",
|
||
"train_acc_metric = keras.metrics.BinaryAccuracy()\n",
|
||
"val_acc_metric = keras.metrics.BinaryAccuracy()\n",
|
||
"\n",
|
||
"# Indices de l'ensemble non labellisé\n",
|
||
"indices_lab = np.arange(x_train_lab.shape[0]) \n",
|
||
"# Indices de l'ensemble non labellisé\n",
|
||
"indices_unlab = np.arange(x_train_unlab.shape[0]) \n",
|
||
"\n",
|
||
"for epoch in range(epochs):\n",
|
||
"\n",
|
||
" for b in range(steps_per_epochs):\n",
|
||
"\n",
|
||
" # Les données d'un batch sont constituées de l'intégralité de nos données labellisées...\n",
|
||
" x_batch_lab = x_train_lab[indices_lab[b*bs_lab:(b+1)*bs_lab]]\n",
|
||
" y_batch_lab = y_train_lab[indices_lab[b*bs_lab:(b+1)*bs_lab]]\n",
|
||
" y_batch_lab = np.expand_dims(y_batch_lab, 1)\n",
|
||
"\n",
|
||
" # ... ainsi que de données non-labellisées !\n",
|
||
" x_batch_unlab = x_train_unlab[indices[b*bs_unlab:(b+1)*bs_unlab]]\n",
|
||
"\n",
|
||
" # On forme notre batch en concaténant les données labellisées et non labellisées\n",
|
||
" x_batch = np.concatenate((x_batch_lab, x_batch_unlab), axis=0)\n",
|
||
"\n",
|
||
" # On forme également un batch alternatif constitué des mêmes données bruitées\n",
|
||
" # Le bruit ici sera simplement obtenu avec np.rand()\n",
|
||
" # Attention à l'échelle du bruit !\n",
|
||
" x_batch_noisy = ...\n",
|
||
"\n",
|
||
" # Les opérations effectuées par le modèle dans ce bloc sont suivies et permettront\n",
|
||
" # la différentiation automatique.\n",
|
||
" with tf.GradientTape() as tape:\n",
|
||
"\n",
|
||
" # Application du réseau aux données d'entrée\n",
|
||
" y_pred = model(x_batch, training=True)\n",
|
||
" # Ne pas oublier de le faire également sur le 2e batch ! \n",
|
||
" y_pred_noisy = model(x_batch_noisy, training=True) \n",
|
||
"\n",
|
||
" # Calcul de la fonction de perte sur ce batch\n",
|
||
" sup_term = ...\n",
|
||
" unsup_term = ...\n",
|
||
"\n",
|
||
" loss_value = ...\n",
|
||
"\n",
|
||
" # Calcul des gradients par différentiation automatique\n",
|
||
" grads = tape.gradient(loss_value, model.trainable_weights)\n",
|
||
"\n",
|
||
" # Réalisation d'une itération de la descente de gradient (mise à jour des paramètres du réseau)\n",
|
||
" optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
|
||
"\n",
|
||
" # Mise à jour de la métrique\n",
|
||
" train_acc_metric.update_state(np.expand_dims(y_batch_lab, 1), y_pred[0:bs_lab])\n",
|
||
"\n",
|
||
" \n",
|
||
" # Calcul de la précision à la fin de l'epoch\n",
|
||
" train_acc = train_acc_metric.result()\n",
|
||
" # Calcul de la précision sur l'ensemble de validation à la fin de l'epoch\n",
|
||
" val_logits = model(x_test, training=False)\n",
|
||
" val_acc_metric.update_state(np.expand_dims(y_test, 1), val_logits)\n",
|
||
" val_acc = val_acc_metric.result()\n",
|
||
"\n",
|
||
" print(\"Epoch %4d : Loss : %.4f, Acc : %.4f, Val Acc : %.4f\" % (epoch, float(loss_value), float(train_acc), float(val_acc)))\n",
|
||
"\n",
|
||
" # Remise à zéro des métriques pour la prochaine epoch\n",
|
||
" train_acc_metric.reset_states()\n",
|
||
" val_acc_metric.reset_states()\n",
|
||
"\n",
|
||
" # Mise à jour du paramètre de contrôle de l'importance de la régularisation non-supervisée\n",
|
||
" # Il augmente progressivement !\n",
|
||
" if lambda_t < 1:\n",
|
||
" if epoch > 100:\n",
|
||
" lambda_t = lambda_t + 0.001\n",
|
||
"\n",
|
||
"\n",
|
||
" "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "l1dZNTmKYjZs"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from mlxtend.plotting import plot_decision_regions\n",
|
||
"\n",
|
||
"# Affichage des données\n",
|
||
"plt.plot(x_train_unlab[y_train_unlab==0,0], x_train_unlab[y_train_unlab==0,1], 'b.')\n",
|
||
"plt.plot(x_train_unlab[y_train_unlab==1,0], x_train_unlab[y_train_unlab==1,1], 'r.')\n",
|
||
"\n",
|
||
"plt.plot(x_test[y_test==0,0], x_test[y_test==0,1], 'b+')\n",
|
||
"plt.plot(x_test[y_test==1,0], x_test[y_test==1,1], 'r+')\n",
|
||
"\n",
|
||
"plt.plot(x_train_lab[y_train_lab==0,0], x_train_lab[y_train_lab==0,1], 'b.', markersize=30)\n",
|
||
"plt.plot(x_train_lab[y_train_lab==1,0], x_train_lab[y_train_lab==1,1], 'r.', markersize=30)\n",
|
||
"\n",
|
||
"plt.show()\n",
|
||
"\n",
|
||
"# Plot decision boundary\n",
|
||
"plot_decision_regions(x_train_unlab, y_train_unlab, clf=model, legend=2)\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "e2AnvQPl4YTb"
|
||
},
|
||
"source": [
|
||
"# MNIST"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "B_noJPS5f2Td"
|
||
},
|
||
"source": [
|
||
"Pour adapter l'algorithme du $\\Pi$-modèle à MNIST, nous allons devoir remplacer le bruitage des données par de l'augmentation de données.\n",
|
||
"\n",
|
||
"Commencez par remplir l'ImageDataGenerator (à vous de voir comment dans [la documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator)) avec des transformations pertinentes. **Attention** cette étape est cruciale pour obtenir de bons résultats. Il faut intégrer les augmentations les plus fortes possibles, mais être certain qu'elles ne modifient pas le label du chiffre !"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "anl-QTIxgnwf"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from tensorflow.keras.datasets import mnist\n",
|
||
"import numpy as np\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"\n",
|
||
"def generate_mnist_dataset(num_lab = 100):\n",
|
||
"\n",
|
||
" # Chargement et normalisation (entre 0 et 1) des données de la base de données MNIST\n",
|
||
" (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
|
||
"\n",
|
||
" x_train = np.expand_dims(x_train.astype('float32') / 255., 3)\n",
|
||
" x_test = np.expand_dims(x_test.astype('float32') / 255., 3)\n",
|
||
"\n",
|
||
" x_train_lab, x_train_unlab, y_train_lab, y_train_unlab = train_test_split(x_train, y_train, test_size=(x_train.shape[0]-num_lab)/x_train.shape[0], random_state=2)\n",
|
||
"\n",
|
||
" return x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test\n",
|
||
"\n",
|
||
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_mnist_dataset()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "OLKir7N1klkz"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"\n",
|
||
"train_datagen = ImageDataGenerator(\n",
|
||
" ### A COMPLETER\n",
|
||
")\n",
|
||
"\n",
|
||
"# Affichage d'une donnée et de son augmentation\n",
|
||
"x = x_train_lab[0:10]\n",
|
||
"plt.imshow(x[0, : ,: ,0])\n",
|
||
"plt.show()\n",
|
||
"x_aug = train_datagen.flow(x, shuffle=False, batch_size=10).next()\n",
|
||
"plt.imshow(x_aug[0, : ,: ,0])\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "nx9N8ZV-u_fX"
|
||
},
|
||
"source": [
|
||
"Implémentez le réseau LeNet-5 pour la classifications des chiffres manuscrits, en suivant cet exemple : \n",
|
||
"<img src=\"https://www.datasciencecentral.com/wp-content/uploads/2021/10/1lvvWF48t7cyRWqct13eU0w.jpeg\">\n",
|
||
"<caption><center> Figure 2 : Schéma de l'architecture de LeNet-5</center></caption>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "ASNuRBCVvHZe"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from tensorflow.keras.layers import *\n",
|
||
"from tensorflow.keras import Model, Input\n",
|
||
"\n",
|
||
"# A COMPLETER\n",
|
||
"# Ici, on implémentera le modèle LeNet-5 :\n",
|
||
"# 1 couche de convolution 5x5 à 6 filtres suivie d'un max pooling\n",
|
||
"# puis 1 couche de convolution 5x5 à 16 filtres suivie d'un max pooling et d'un Flatten\n",
|
||
"# Enfin 2 couches denses de 120 et 84 neurones, avant la couche de sortie à 10 neurones.\n",
|
||
"def create_model_mnist():\n",
|
||
"\n",
|
||
" inputs = keras.Input(shape=(...))\n",
|
||
"\n",
|
||
" ...\n",
|
||
" \n",
|
||
" outputs = \n",
|
||
"\n",
|
||
" model = keras.Model(inputs=inputs, outputs=outputs) \n",
|
||
"\n",
|
||
" return model"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "D-v1X5Ypv4jz"
|
||
},
|
||
"source": [
|
||
"**Travail à faire**\n",
|
||
"\n",
|
||
"Commencez d'abord par entraîner LeNet-5 sur MNIST de manière supervisée, en **utilisant 100 données labellisées**.\n",
|
||
"\n",
|
||
"Attention, il va vous falloir modifier quelques élements par rapport à ce que nous avons fait dans la séance précédente, notamment la fonction de coût (*SparseCategoricalCrossEntropy*) et les métriques (*SparseCategoricalAccuracy*).\n",
|
||
"\n",
|
||
"Pour comparer de manière juste les versions supervisée et semi-supervisée, n'oubliez pas également d'intégrer l'augmentation de données dans votre apprentissage. Vous devriez obtenir environ 80\\% de bonnes classifications sur l'ensemble de test."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {
|
||
"id": "VAAFtjTv5U1n"
|
||
},
|
||
"source": [
|
||
"**Travail à faire**\n",
|
||
"\n",
|
||
"Reprenez ensuite le code du $\\Pi$-Modèle pour l'adapter à MNIST, en intégrant l'augmentation (à la place du bruitage des données). Vous devriez obtenir un gain significatif avec les bons hyperparamètres ! (jusqu'à environ 97\\%)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"accelerator": "GPU",
|
||
"colab": {
|
||
"collapsed_sections": [],
|
||
"machine_shape": "hm",
|
||
"provenance": [],
|
||
"toc_visible": true
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3.10.8 64-bit",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.10.8"
|
||
},
|
||
"vscode": {
|
||
"interpreter": {
|
||
"hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90"
|
||
}
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0
|
||
}
|