Digital Cultural Heritage Preservation: Enrichment and Reconstruction based on Hierarchical Multimodal CNNs and Image Inpainting Approaches

PhD student: 
External supervisors: 

S. Foufou

Starting date: 
February 2017
Host institution: 

Long-term physical preservation of cultural heritage remains risky and can lead to multiple problems related to destruction and accidental damage. Fully digitizing cultural assets (visually and historically) is only easy when it comes to assets that are rich physically (in a good shape) and all their data is at possession (fully annotated). In the real world, many assets suffer from physical degradation and information loss. Usually, to annotate and curate these assets, heritage institutions refer to art specialists, historians, and other peer institutions. This process is tedious, involves many time and financial resources, and can often be inaccurate.

Our work is focusing on the cost-effective preservation of cultural heritage through advanced machine learning methods. The aim is to provide a technical framework for the enrichment phase of the cultural heritage digital preservation/curation process. Through this thesis, we propose new methods to improve the process of cultural heritage preservation. Our challenges are mainly related to the annotation and enrichment of cultural objects suffering from missing and incomplete data (annotations and visual data) which is often considered ineffective when performed manually. Thus, we propose approaches based on machine learning and deep learning to tackle these challenges. These approaches consist of the automatic completion of missing cultural data. We mainly focus on two types of missing data: textual data (metadata) and visual data.

The first stage is mainly related to the annotation and labeling of cultural objects using deep learning. We have proposed approaches that take advantage of cultural objects visual features as well as partially available textual annotations to perform an effective classification. (1) A first approach is dedicated to the Multimodal Classification of cultural objects where any object can be represented, during classification, with a subset of available metadata in addition to its visual capture. (2) the second proposed approach is related to the Hierarchical Classification of Objects to better meet each cultural type metadata requirements and increase the classification’s performance. The second stage considers the lack of visual information found in certain incomplete and damaged cultural objects. In this case, we proposed an approach based on deep learning through generative models and image data clustering to optimize the image completion process of damaged cultural heritage objects. For our experiments, we collected a large database of cultural objects. We chose to use fine-art paintings in our tests and validations as they were the best in terms of annotations quality.

Keywords : Preservation of Cultural Objects, Deep Learning, Multimodal Classification, Image Completion.