Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Enabling Deep Document Image Analysis with Generative Models
Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Embedded Internet Systems Lab.ORCID iD: 0000-0002-9332-3188
2023 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

Historical documents are a valuable source of cultural knowledge and can provide information about previous events, societies, beliefs, and cultures. They can serve as an excellent source for research in various fields including history, literature, linguistics, and anthropology. Their preservation and analysis pose significant challenges due to the unique characteristics of handwritten scripts, the variability, and the document degradation. With the rise of the Deep Learning era, enormous amounts of annotated data are required to train large models that can efficiently perform tasks on unseen data. Nowadays, digital libraries provide high-quality digitized images for analysis and processing of historical documents. However, collecting and annotating the provided data is an expensive task and requires a lot of expertise from historians and the humanities. Hence, generating synthetic data to enhance the performance of Deep Learning frameworks is a common approach in Computer Vision and, specifically in this thesis, in Document Image Analysis and Recognition (DIAR).

This thesis focuses on leveraging generative models to facilitate DIAR tasks, focusing on historical and handwritten documents, by generating realistic synthetic images that resemble a real distribution and enhance the training of downstream DIAR tasks. The contributions of the thesis include a systematic literature review, a comparison evaluation, and a developed method for handwriting generation.

First, a systematic literature review of existing historical document image datasets, provides summarized information of 65 studies, focusing on different aspects, such as statistics, document type, language, visual, and annotation aspects. The study discusses limitations and promising resources for future research, which refer to the limited dataset size and absence of benchmarks, as well as the lack of standardization in terms of data format and evaluation scheme.

A subsequent contribution is the integration of generated data in a historical document font classification task. Semi-synthetic data are generated with the use of DocCreator, an open-source software, from which different document degradation augmentations are used. A conditional Generative Adversarial Network (GAN) is used to generate fully synthetic data conditioned on a specific sample. The data generated by the two methods areintegrated as additional samples in the training of several Convolutional Neural Networks classifiers and the effect in the performance is examined.

The final contribution of the thesis introduces a new method for generating styled handwritten text images based on Denoising Diffusion Probabilistic Models (DDPM), which is an unexplored method in DIAR. The method manages to capture stylistic and content characteristics of a standard multi-writer handwriting dataset and achieved an improved performance in enhancing writer identification and handwriting text recognition compared to Generative Adversarial Network (GAN)-based methods. The results demonstrate the potential of the generative method for enabling deep document image analysis and pave the way for further research.

As a future direction, this work will aim to progress from generating word images to generating sentence and full document images by conditioning on the content, style, and layout of historical documents. Another future action will be to further extend the proposed method to operate in a few-shot scheme for the writer style condition in order to generate unseen styles. Furthermore, the future work will aim to leverage important features from pre-training with synthetic and real data in order to generalize to historical documents that are a scarce source and adjusting the text encoding parts to different languages and scripts. Finally, the ultimate goal of the future work aims to generate a massive synthetic historical document image database to fill the existing benchmark gap.

Place, publisher, year, edition, pages
Luleå: Luleå University of Technology, 2023.
Series
Licentiate thesis / Luleå University of Technology, ISSN 1402-1757
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering
Research subject
Machine Learning
Identifiers
URN: urn:nbn:se:ltu:diva-96361ISBN: 978-91-8048-303-2 (print)ISBN: 978-91-8048-304-9 (electronic)OAI: oai:DiVA.org:ltu-96361DiVA, id: diva2:1749852
Presentation
2023-06-07, A117, Luleå tekniska universitet, Luleå, 10:00 (English)
Opponent
Supervisors
Available from: 2023-04-12 Created: 2023-04-11 Last updated: 2025-10-21Bibliographically approved
List of papers
1. Investigating the Effect of Using Synthetic and Semi-synthetic Images for Historical Document Font Classification
Open this publication in new window or tab >>Investigating the Effect of Using Synthetic and Semi-synthetic Images for Historical Document Font Classification
2022 (English)In: Document Analysis Systems: 15th IAPR International Workshop, DAS 2022, La Rochelle, France, May 22–25, 2022, Proceedings / [ed] Seiichi Uchida; Elisa Barney; Véronique Eglin, Springer Nature, 2022, p. 613-626Conference paper, Published paper (Refereed)
Abstract [en]

This paper studies the effect of using various data augmentation by synthetization approaches on historical image data, particularly for font classification. Historical document image datasets often lack the appropriate size to train and evaluate deep learning models, motivating data augmentation and synthetic document generation techniques for creating additional data. This work explores the effect of various semi-synthetic and synthetic historical document images, some of which appear as recent trends and others not published yet, on a font classification task. We use 10K patch samples as baseline dataset, derived from the dataset of Early Printed Books with Multiple Font Groups, and increase its size using DocCreator software and Generative Adversarial Networks (GAN). Furthermore, we fine-tune different pre-trained Convolutional Neural Network (CNN) classifiers as a baseline using the original dataset and then compare the performance with the additional semi-synthetic and synthetic images. We further evaluate the performance using additional real samples from the original dataset in the training process. DocCreator, and the additional real samples improve the performance giving the best results. Finally, for the best-performing architecture, we explore different sizes of training sets and examine how the gradual addition of data affects the performance.

Place, publisher, year, edition, pages
Springer Nature, 2022
Series
Lecture Notes in Computer Science, ISSN 0302-9743, E-ISSN 1611-3349 ; 13237
Keywords
Historical document images, Synthetic image generation, Font classification, Convolutional neural networks, Generative Adversarial Networks, Document Image Analysis, Performance evaluation
National Category
Computer graphics and computer vision
Research subject
Machine Learning
Identifiers
urn:nbn:se:ltu:diva-91458 (URN)10.1007/978-3-031-06555-2_41 (DOI)000870314500041 ()2-s2.0-85131123625 (Scopus ID)
Conference
15th IAPR International Workshop on Document Analysis Systems (DAS 2022), La Rochelle, France, May 22-25, 2022
Note

ISBN för värdpublikation: 978-3-031-06554-5, 978-3-031-06555-2

Available from: 2022-06-21 Created: 2022-06-21 Last updated: 2025-10-21Bibliographically approved
2. A survey of historical document image datasets
Open this publication in new window or tab >>A survey of historical document image datasets
2022 (English)In: International Journal on Document Analysis and Recognition, ISSN 1433-2833, E-ISSN 1433-2825, Vol. 25, no 4, p. 305-338Article in journal (Refereed) Published
Abstract [en]

This paper presents a systematic literature review of image datasets for document image analysis, focusing on historical documents, such as handwritten manuscripts and early prints. Finding appropriate datasets for historical document analysis is a crucial prerequisite to facilitate research using different machine learning algorithms. However, because of the very large variety of the actual data (e.g., scripts, tasks, dates, support systems, and amount of deterioration), the different formats for data and label representation, and the different evaluation processes and benchmarks, finding appropriate datasets is a difficult task. This work fills this gap, presenting a meta-study on existing datasets. After a systematic selection process (according to PRISMA guidelines), we select 65 studies that are chosen based on different factors, such as the year of publication, number of methods implemented in the article, reliability of the chosen algorithms, dataset size, and journal outlet. We summarize each study by assigning it to one of three pre-defined tasks: document classification, layout structure, or content analysis. We present the statistics, document type, language, tasks, input visual aspects, and ground truth information for every dataset. In addition, we provide the benchmark tasks and results from these papers or recent competitions. We further discuss gaps and challenges in this domain. We advocate for providing conversion tools to common formats (e.g., COCO format for computer vision tasks) and always providing a set of evaluation metrics, instead of just one, to make results comparable across studies.

Place, publisher, year, edition, pages
Springer Nature, 2022
Keywords
Historical documents, Image datasets, Document image analysis, Machine learning
National Category
Computer graphics and computer vision Software Engineering
Research subject
Machine Learning
Identifiers
urn:nbn:se:ltu:diva-92440 (URN)10.1007/s10032-022-00405-8 (DOI)000833450300001 ()2-s2.0-85135223161 (Scopus ID)
Note

Validerad;2022;Nivå 2;2022-12-01 (marisr)

Available from: 2022-08-11 Created: 2022-08-11 Last updated: 2025-10-21Bibliographically approved
3. WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models
Open this publication in new window or tab >>WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models
Show others...
2023 (English)In: Document Analysis and Recognition - ICDAR 2023, Part II / [ed] Gernot A. Fink, Rajiv Jain, Koichi Kise & Richard Zanibbi, Springer, 2023, p. 384-401Conference paper, Published paper (Refereed)
Abstract [en]

Text-to-Image synthesis is the task of generating an image according to a specific text description. Generative Adversarial Networks have been considered the standard method for image synthesis virtually since their introduction. Denoising Diffusion Probabilistic Models are recently setting a new baseline, with remarkable results in Text-to-Image synthesis, among other fields. Aside its usefulness per se, it can also be particularly relevant as a tool for data augmentation to aid training models for other document image processing tasks. In this work, we present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles, by using class index styles and text content prompts without the need of adversarial training, writer recognition, or text recognition. We gauge system performance with the Fréchet Inception Distance, writer recognition accuracy, and writer retrieval. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.

Place, publisher, year, edition, pages
Springer, 2023
Series
Lecture Notes in Computer Science (LNCS), ISSN 0302-9743, E-ISSN 1611-3349 ; 14188
Keywords
Diffusion Models, Synthetic Image Generation, Text Content Generation, Handwriting Generation, Data Augmentation, Handwriting Text Recognition
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering
Research subject
Machine Learning
Identifiers
urn:nbn:se:ltu:diva-96364 (URN)10.1007/978-3-031-41679-8_22 (DOI)001346405600022 ()2-s2.0-85173585071 (Scopus ID)
Conference
17th International Conference on Document Analysis and Recognition (ICDR2023), San José, CA, USA, August 21-26. 2023
Note

ISBN for host publication: 978-3-031-41678-1 (print), 978-3-031-41679-8 (electronic);

This paper has has previously appeared as a manuscript in a thesis.

Available from: 2023-04-11 Created: 2023-04-11 Last updated: 2025-10-21Bibliographically approved

Open Access in DiVA

fulltext(6808 kB)1589 downloads
File information
File name FULLTEXT02.pdfFile size 6808 kBChecksum SHA-512
7376d74b29821ac3eb1b241c9503e50525a621ba168225c2af64d78a2c92df87258c732ec37bb96a68a887ff8006d35d9e127581c1895942f6f4567885bb2fce
Type fulltextMimetype application/pdf

Authority records

Nikolaidou, Konstantina

Search in DiVA

By author/editor
Nikolaidou, Konstantina
By organisation
Embedded Internet Systems Lab
Other Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 1594 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1236 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf