Fusion de données multimodales pour l’apprentissage automatique : application au diagnostic médical

Abstract

Nowadays, information about a phenomenon can be acquired using various modalities; and this led to the emergence of the concept of multi-modal data fusion. In fact, one of the application domains that strongly relies on multi-modal data acquisition is radiation therapy (RT), in which computed tomography (CT) imaging and magnetic resonance imaging (MRI) are the main modalities employed. The latter offers superior soft-tissue visualisation, hence it is used for tumour contouring, whereas the former contains the necessary electron density information for radiation dose computation. However, the pipeline of RT treatment planning presents many flaws from which is the excessive radiation exposure due to repetitive CT acquisitions. Recently, an idea to remediate for this main limitation and others has emerged; it consists in synthesising a CT image called pseudo-CT (pCT) from MRI images. In this dissertation, we aim to meet this objective by analysing existing work and proposing two variants of pCT image estimation methods employing multi-modal data fusion and machine learning. The first approach is designated hybrid; and combines image registration, unsupervised learning, image fusion and a new Hounsfield Unit (HU) values correction technique. This approach aims to predict a pCT image from T2-weighted MR images while attempting to reduce prediction error by using information from MR images. The second approach employs multi-modal shape, texture and spatial feature extraction, reduction and fusion of T1-weighted and T2-weighted MR images. In addition, we designed an ensemble learning model with stacked generalisation that builds a mapping from these features to HU values over two levels of learning. Results showed a significant improvement achieved by the proposed methods.