TP-intelligence-artificiell.../IAM2022_Apprentissage_Semi_Supervise_Suite.ipynb
2023-06-23 19:39:56 +02:00

416 lines
16 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "4Vhg6pn2uDlW"
},
"source": [
"# Dataset des 2 lunes\n",
"\n",
"(Avec un nouvel affichage plus joli, merci Arthur !)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Hw8rHiTKuHL3"
},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn import datasets\n",
"import matplotlib.pyplot as plt \n",
"\n",
"def generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250):\n",
" num_samples = num_lab + num_unlab + num_test\n",
" # Génération de 1000 données du dataset des 2 lunes\n",
" x, y = datasets.make_moons(n_samples=num_samples, noise=0.1, random_state=1)\n",
"\n",
" x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=num_test/num_samples, random_state=1)\n",
" x_train_lab, x_train_unlab, y_train_lab, y_train_unlab = train_test_split(x_train, y_train, test_size=num_unlab/(num_unlab+num_lab), random_state=6)\n",
"\n",
" return x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ww95atT6uJ4D"
},
"outputs": [],
"source": [
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250)\n",
"\n",
"print(x_train_lab.shape, x_train_unlab.shape, x_test.shape)\n",
"print(y_train_lab.shape, y_train_unlab.shape, y_test.shape)\n",
"\n",
"# Affichage des données\n",
"plt.plot(x_train_unlab[y_train_unlab==0,0], x_train_unlab[y_train_unlab==0,1], color=(0.5,0.5,0.5), marker='.', linestyle=' ')\n",
"plt.plot(x_train_unlab[y_train_unlab==1,0], x_train_unlab[y_train_unlab==1,1], color=(0.5,0.5,0.5), marker='.', linestyle=' ')\n",
"\n",
"plt.plot(x_test[y_test==0,0], x_test[y_test==0,1], 'b+')\n",
"plt.plot(x_test[y_test==1,0], x_test[y_test==1,1], 'r+')\n",
"\n",
"plt.plot(x_train_lab[y_train_lab==0,0], x_train_lab[y_train_lab==0,1], 'b.', markersize=30)\n",
"plt.plot(x_train_lab[y_train_lab==1,0], x_train_lab[y_train_lab==1,1], 'r.', markersize=30)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LKODCH2luSPM"
},
"outputs": [],
"source": [
"def create_model_2moons():\n",
"\n",
" inputs = keras.Input(shape=(2,))\n",
" x = Dense(20, activation=\"relu\")(inputs)\n",
" outputs = Dense(1, activation=\"sigmoid\")(x)\n",
" model = keras.Model(inputs=inputs, outputs=outputs) \n",
"\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ea7E3-6l3_le"
},
"source": [
"# $\\Pi$-Modèle\n",
"\n",
"Nous allons maintenant tenter d'utiliser un 2nd algorithme semi-supervisé supposé être plus efficace, il s'agit de l'algorithme du $\\Pi$-Modèle, dont la version détaillée est présentée ci-dessous (en VO).\n",
"\n",
"<img src=\"https://drive.google.com/uc?id=13VhlBYwA6YIYGzKI81Jom_jTiuhOypEg\">\n",
"<caption><center> Figure 1 : Pseudo-code de l'algorithme du $\\Pi$-Modèle</center></caption>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6vaWDKNpYxc0"
},
"source": [
"Ci-dessous, la boucle d'entraînement détaillée est reprise et contient un squelette du code à réaliser pour implémenter le $\\Pi$-Modèle. \n",
"\n",
"**Travail à faire :** Complétez le squelette de l'algorithme du $\\Pi$-Modèle pour pouvoir tester ce nouvel algorithme."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uVK8itsvD72s"
},
"outputs": [],
"source": [
"# Nombre d'epochs de l'apprentissage\n",
"epochs = 2000\n",
"# Nombre de données non-labellisées par batch\n",
"bs_unlab = 100\n",
"# Nombre de données labellisées par batch\n",
"bs_lab = 10\n",
"# Taille du batch\n",
"batch_size = bs_lab + bs_unlab\n",
"# Valeur initiale du paramètre de contrôle de l'importance de la régularisation non-supervisée\n",
"lambda_t = 0\n",
"\n",
"# Données et modèle du problème des 2 clusters\n",
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_2moons_dataset(num_lab = 10, num_unlab=740, num_test=250)\n",
"model = create_model_2moons()\n",
"\n",
"# Nombre de batches par epochs\n",
"steps_per_epochs = int(np.floor(x_train_lab.shape[0]/bs_lab))\n",
"# Instanciation d'un optimiseur et d'une fonction de coût.\n",
"optimizer = keras.optimizers.Adam(learning_rate=3e-2)\n",
"# ICI ON A BESOIN DE DEUX FONCTIONS DE COUT : \n",
"# L'une pour la partie supervisée de la perte\n",
"loss_sup = ...\n",
"# L'autre pour la partie non-supervisée de la perte\n",
"loss_unsup = ...\n",
"\n",
"# Préparation des métriques pour le suivi de la performance du modèle.\n",
"train_acc_metric = keras.metrics.BinaryAccuracy()\n",
"val_acc_metric = keras.metrics.BinaryAccuracy()\n",
"\n",
"# Indices de l'ensemble non labellisé\n",
"indices_lab = np.arange(x_train_lab.shape[0]) \n",
"# Indices de l'ensemble non labellisé\n",
"indices_unlab = np.arange(x_train_unlab.shape[0]) \n",
"\n",
"for epoch in range(epochs):\n",
"\n",
" for b in range(steps_per_epochs):\n",
"\n",
" # Les données d'un batch sont constituées de l'intégralité de nos données labellisées...\n",
" x_batch_lab = x_train_lab[indices_lab[b*bs_lab:(b+1)*bs_lab]]\n",
" y_batch_lab = y_train_lab[indices_lab[b*bs_lab:(b+1)*bs_lab]]\n",
" y_batch_lab = np.expand_dims(y_batch_lab, 1)\n",
"\n",
" # ... ainsi que de données non-labellisées !\n",
" x_batch_unlab = x_train_unlab[indices[b*bs_unlab:(b+1)*bs_unlab]]\n",
"\n",
" # On forme notre batch en concaténant les données labellisées et non labellisées\n",
" x_batch = np.concatenate((x_batch_lab, x_batch_unlab), axis=0)\n",
"\n",
" # On forme également un batch alternatif constitué des mêmes données bruitées\n",
" # Le bruit ici sera simplement obtenu avec np.rand()\n",
" # Attention à l'échelle du bruit !\n",
" x_batch_noisy = ...\n",
"\n",
" # Les opérations effectuées par le modèle dans ce bloc sont suivies et permettront\n",
" # la différentiation automatique.\n",
" with tf.GradientTape() as tape:\n",
"\n",
" # Application du réseau aux données d'entrée\n",
" y_pred = model(x_batch, training=True)\n",
" # Ne pas oublier de le faire également sur le 2e batch ! \n",
" y_pred_noisy = model(x_batch_noisy, training=True) \n",
"\n",
" # Calcul de la fonction de perte sur ce batch\n",
" sup_term = ...\n",
" unsup_term = ...\n",
"\n",
" loss_value = ...\n",
"\n",
" # Calcul des gradients par différentiation automatique\n",
" grads = tape.gradient(loss_value, model.trainable_weights)\n",
"\n",
" # Réalisation d'une itération de la descente de gradient (mise à jour des paramètres du réseau)\n",
" optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
"\n",
" # Mise à jour de la métrique\n",
" train_acc_metric.update_state(np.expand_dims(y_batch_lab, 1), y_pred[0:bs_lab])\n",
"\n",
" \n",
" # Calcul de la précision à la fin de l'epoch\n",
" train_acc = train_acc_metric.result()\n",
" # Calcul de la précision sur l'ensemble de validation à la fin de l'epoch\n",
" val_logits = model(x_test, training=False)\n",
" val_acc_metric.update_state(np.expand_dims(y_test, 1), val_logits)\n",
" val_acc = val_acc_metric.result()\n",
"\n",
" print(\"Epoch %4d : Loss : %.4f, Acc : %.4f, Val Acc : %.4f\" % (epoch, float(loss_value), float(train_acc), float(val_acc)))\n",
"\n",
" # Remise à zéro des métriques pour la prochaine epoch\n",
" train_acc_metric.reset_states()\n",
" val_acc_metric.reset_states()\n",
"\n",
" # Mise à jour du paramètre de contrôle de l'importance de la régularisation non-supervisée\n",
" # Il augmente progressivement !\n",
" if lambda_t < 1:\n",
" if epoch > 100:\n",
" lambda_t = lambda_t + 0.001\n",
"\n",
"\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l1dZNTmKYjZs"
},
"outputs": [],
"source": [
"from mlxtend.plotting import plot_decision_regions\n",
"\n",
"# Affichage des données\n",
"plt.plot(x_train_unlab[y_train_unlab==0,0], x_train_unlab[y_train_unlab==0,1], 'b.')\n",
"plt.plot(x_train_unlab[y_train_unlab==1,0], x_train_unlab[y_train_unlab==1,1], 'r.')\n",
"\n",
"plt.plot(x_test[y_test==0,0], x_test[y_test==0,1], 'b+')\n",
"plt.plot(x_test[y_test==1,0], x_test[y_test==1,1], 'r+')\n",
"\n",
"plt.plot(x_train_lab[y_train_lab==0,0], x_train_lab[y_train_lab==0,1], 'b.', markersize=30)\n",
"plt.plot(x_train_lab[y_train_lab==1,0], x_train_lab[y_train_lab==1,1], 'r.', markersize=30)\n",
"\n",
"plt.show()\n",
"\n",
"# Plot decision boundary\n",
"plot_decision_regions(x_train_unlab, y_train_unlab, clf=model, legend=2)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e2AnvQPl4YTb"
},
"source": [
"# MNIST"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B_noJPS5f2Td"
},
"source": [
"Pour adapter l'algorithme du $\\Pi$-modèle à MNIST, nous allons devoir remplacer le bruitage des données par de l'augmentation de données.\n",
"\n",
"Commencez par remplir l'ImageDataGenerator (à vous de voir comment dans [la documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator)) avec des transformations pertinentes. **Attention** cette étape est cruciale pour obtenir de bons résultats. Il faut intégrer les augmentations les plus fortes possibles, mais être certain qu'elles ne modifient pas le label du chiffre !"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "anl-QTIxgnwf"
},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import mnist\n",
"import numpy as np\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"def generate_mnist_dataset(num_lab = 100):\n",
"\n",
" # Chargement et normalisation (entre 0 et 1) des données de la base de données MNIST\n",
" (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
"\n",
" x_train = np.expand_dims(x_train.astype('float32') / 255., 3)\n",
" x_test = np.expand_dims(x_test.astype('float32') / 255., 3)\n",
"\n",
" x_train_lab, x_train_unlab, y_train_lab, y_train_unlab = train_test_split(x_train, y_train, test_size=(x_train.shape[0]-num_lab)/x_train.shape[0], random_state=2)\n",
"\n",
" return x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test\n",
"\n",
"x_train_lab, y_train_lab, x_train_unlab, y_train_unlab, x_test, y_test = generate_mnist_dataset()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OLKir7N1klkz"
},
"outputs": [],
"source": [
"from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
"import matplotlib.pyplot as plt\n",
"\n",
"train_datagen = ImageDataGenerator(\n",
" ### A COMPLETER\n",
")\n",
"\n",
"# Affichage d'une donnée et de son augmentation\n",
"x = x_train_lab[0:10]\n",
"plt.imshow(x[0, : ,: ,0])\n",
"plt.show()\n",
"x_aug = train_datagen.flow(x, shuffle=False, batch_size=10).next()\n",
"plt.imshow(x_aug[0, : ,: ,0])\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nx9N8ZV-u_fX"
},
"source": [
"Implémentez le réseau LeNet-5 pour la classifications des chiffres manuscrits, en suivant cet exemple : \n",
"<img src=\"https://www.datasciencecentral.com/wp-content/uploads/2021/10/1lvvWF48t7cyRWqct13eU0w.jpeg\">\n",
"<caption><center> Figure 2 : Schéma de l'architecture de LeNet-5</center></caption>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ASNuRBCVvHZe"
},
"outputs": [],
"source": [
"from tensorflow.keras.layers import *\n",
"from tensorflow.keras import Model, Input\n",
"\n",
"# A COMPLETER\n",
"# Ici, on implémentera le modèle LeNet-5 :\n",
"# 1 couche de convolution 5x5 à 6 filtres suivie d'un max pooling\n",
"# puis 1 couche de convolution 5x5 à 16 filtres suivie d'un max pooling et d'un Flatten\n",
"# Enfin 2 couches denses de 120 et 84 neurones, avant la couche de sortie à 10 neurones.\n",
"def create_model_mnist():\n",
"\n",
" inputs = keras.Input(shape=(...))\n",
"\n",
" ...\n",
" \n",
" outputs = \n",
"\n",
" model = keras.Model(inputs=inputs, outputs=outputs) \n",
"\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D-v1X5Ypv4jz"
},
"source": [
"**Travail à faire**\n",
"\n",
"Commencez d'abord par entraîner LeNet-5 sur MNIST de manière supervisée, en **utilisant 100 données labellisées**.\n",
"\n",
"Attention, il va vous falloir modifier quelques élements par rapport à ce que nous avons fait dans la séance précédente, notamment la fonction de coût (*SparseCategoricalCrossEntropy*) et les métriques (*SparseCategoricalAccuracy*).\n",
"\n",
"Pour comparer de manière juste les versions supervisée et semi-supervisée, n'oubliez pas également d'intégrer l'augmentation de données dans votre apprentissage. Vous devriez obtenir environ 80\\% de bonnes classifications sur l'ensemble de test."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VAAFtjTv5U1n"
},
"source": [
"**Travail à faire**\n",
"\n",
"Reprenez ensuite le code du $\\Pi$-Modèle pour l'adapter à MNIST, en intégrant l'augmentation (à la place du bruitage des données). Vous devriez obtenir un gain significatif avec les bons hyperparamètres ! (jusqu'à environ 97\\%)"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"machine_shape": "hm",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3.10.8 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
},
"vscode": {
"interpreter": {
"hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90"
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}