Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems.

Diffusion-based Handwritten Text Generation (HTG) approaches achieve impressive results on frequent, invocabulary words observed at training time and on regular styles. However, they are prone to memorizing training samples and often struggle with style variability and generation clarity. In particular, standard diffusion models tend to produce artifacts or distortions that negatively affect the readability of the generated text, especially when the style is hard to produce. To tackle these issues, we propose a novel sampling guidance strategy, Dual Orthogonal Guidance (DOG), that leverages an orthogonal projection of a negatively perturbed prompt onto the original positive prompt. This approach helps steer the generation away from artifacts while maintaining the intended content, and encourages more diverse, yet plausible, outputs. Unlike standard Classifier-Free Guidance (CFG), which relies on unconditional predictions and produces noise at high guidance scales, DOG introduces a more stable, disentangled direction in the latent space. To control the strength of the guidance across the denoising process, we apply a triangular schedule: weak at the start and end of denoising, when the process is most sensitive, and strongest in the middle steps. Experimental results on the state-of-the-art DiffusionPen and One-DM demonstrate that DOG improves both content clarity and style variability, even for out-of-vocabulary words and challenging writing styles.

Handwritten Text Generation (HTG) has emerged as a promising remedy for data scarcity that limits the training of Deep Learning (DL) models for Document Image Analysis and Recognition (DIAR) tasks. Current HTG pipelines are mostly based on adversarial training, which can face several issues, such as mode collapse, limiting variability, and in turn, reducing the practical usefulness of the generated data. Moreover, existing evaluation protocols prioritize visual realism rather than usefulness, obscuring the link between generative quality and downstream task performance.

These limitations raise several questions related to which HTG methods truly benefit DIAR tasks, which real datasets should serve as a foundation for synthetic data generation, how Diffusion Models can be adapted for controllable and robust HTG, and whether current evaluation protocols capture downstream utility. Given the success of text-to-image Diffusion Models in generating realistic images from text prompts, their adaptation for handwriting generation offers the potential to produce high-quality, style-aware handwritten data that can directly enhance Handwritten Text Recognition (HTR). The overarching goal of this thesis is to bridge the gap between generation and task-aligned evaluation by showing how controllable diffusion-based HTG, guided by systematic data analysis and evaluated through practical utility, can address these challenges.

The contributions of this thesis are organized along three complementary directions. First, to establish a solid basis for HTG, a comprehensive overview of modern, historical, and synthetic document image datasets and HTG methods is presented. This results in C1, a systematic overview of dataset resources, and C2, a detailed survey of generative paradigms and evaluation practices in HTG, identifying key data and methodological gaps that motivate the development of diffusion-based models. Second, to overcome the instability and limited variability of adversarial methods, three diffusion-based approaches are proposed. C3 (WordStylist), a latent diffusion model enabling verbatim text and style conditioning. C4 (DiffusionPen), a few-shot extension of WordStylist capable of generalizing to unseen writers through hybrid classification and metric-learning style embeddings. Moreover, C5 (Dual Orthogonal Guidance), a sampling-time mechanism that enhances stability while preserving stylistic diversity. These proposed HTG methods demonstrate that diffusion-based models can generate realistic, diverse, and style-consistent handwriting under controllable conditions. Third, recognizing that generative quality should translate into practical utility, C6 introduces a task-aligned evaluation framework that links generation metrics to recognition outcomes. This framework measures content preservation, style preservation, robustness to Out-of-Vocabulary (OOV) content, and variability, providing a practical assessment of whether synthetic handwriting improves DIAR performance. By integrating generation and evaluation, this contribution redefines how HTG success is measured.

In summary, this thesis demonstrates that controllable diffusion-based HTG, grounded in systematic dataset analysis, enabled by robust generative modeling, and evaluated with task-aligned metrics, provides efficient synthetic handwriting pipelines that directly enhance HTR performance. On a broader level, this work bridges the gap between generative modeling and document analysis, setting the stage for future research in task-aware synthesis of document images.

The digitization of historical manuscripts presents significant challenges for Handwritten Text Recognition (HTR) systems, particularly when dealing with small, author-specific collections that diverge from the training data distributions. Handwritten Text Generation (HTG) techniques, which generate synthetic data tailored to specific handwriting styles, offer a promising solution to address these challenges. However, the effectiveness of various HTG models in enhancing HTR performance, especially in low-resource transcription settings, has not been thoroughly evaluated. In this work, we systematically compare three state-of-the-art styled HTG models (representing the generative adversarial, diffusion, and autoregressive paradigms for HTG) to assess their impact on HTR fine-tuning. We analyze how visual and linguistic characteristics of synthetic data influence fine-tuning outcomes and provide quantitative guide-lines for selecting the most effective HTG model. The results of our analysis provide insights into the current capabilities of HTG methods and highlight key areas for further improvement in their application to low-resource HTR.

Handwritten Text Recognition (HTR) is a challenging problem that plays an essential role in digitizing and interpreting diverse handwritten documents. While traditional approaches primarily utilize CNN-RNN (CRNN) architectures, recent advancements based on Transformer architectures have demonstrated impressive results in HTR. However, these Transformer-based systems often involve high-parameter configurations and rely extensively on synthetic data. Moreover, they lack focus on efficiently integrating the ability of Transformer modules to grasp contextual relationships within the data. In this paper, we explore a lightweight integration of Transformer modules into existing CRNN frameworks to address the complexities of HTR, aiming to enhance the context of the sequential nature of the task. We present a hybrid CNN image encoder with intermediate MobileViT blocks that effectively combines the different components in a resource-efficient manner. Through extensive experiments and ablation studies, we refine the integration of these modules and demonstrate that our proposed model enhances HTR performance. Our results on the line-level IAM and RIMES datasets suggest that our proposed method achieves competitive performance with significantly fewer parameters and without integrating synthetic data compared to existing systems. 

Methods for detecting forged handwriting are usually based on the assumption that the forged handwriting is produced by humans. Authentic-looking handwriting, however, can also be produced synthetically. Diffusion-based generative models have recently gained popularity as they produce striking natural images and are also able to realistically mimic a person’s handwriting. It is, therefore, reasonable to assume that these models will be used to forge handwriting in the near future, adding a new layer to handwriting forgery detection. We show for the first time that the identification of synthetic handwritten data is possible by a small Convolutional Neural Network (ResNet18) reaching accuracies of 90%. We further investigate the existence of distinct discriminative features in synthetic handwriting data produced by latent diffusion models that could be exploited to build stronger detection methods. Our experiments indicate that the strongest discriminative features do not come from generation artifacts, letter shapes, or the generative model’s architecture, but instead originate from real-world artifacts in genuine handwriting that are not reproduced by generative methods.

Historical documents are a valuable source of cultural knowledge and can provide information about previous events, societies, beliefs, and cultures. They can serve as an excellent source for research in various fields including history, literature, linguistics, and anthropology. Their preservation and analysis pose significant challenges due to the unique characteristics of handwritten scripts, the variability, and the document degradation. With the rise of the Deep Learning era, enormous amounts of annotated data are required to train large models that can efficiently perform tasks on unseen data. Nowadays, digital libraries provide high-quality digitized images for analysis and processing of historical documents. However, collecting and annotating the provided data is an expensive task and requires a lot of expertise from historians and the humanities. Hence, generating synthetic data to enhance the performance of Deep Learning frameworks is a common approach in Computer Vision and, specifically in this thesis, in Document Image Analysis and Recognition (DIAR).

This thesis focuses on leveraging generative models to facilitate DIAR tasks, focusing on historical and handwritten documents, by generating realistic synthetic images that resemble a real distribution and enhance the training of downstream DIAR tasks. The contributions of the thesis include a systematic literature review, a comparison evaluation, and a developed method for handwriting generation.

First, a systematic literature review of existing historical document image datasets, provides summarized information of 65 studies, focusing on different aspects, such as statistics, document type, language, visual, and annotation aspects. The study discusses limitations and promising resources for future research, which refer to the limited dataset size and absence of benchmarks, as well as the lack of standardization in terms of data format and evaluation scheme.

A subsequent contribution is the integration of generated data in a historical document font classification task. Semi-synthetic data are generated with the use of DocCreator, an open-source software, from which different document degradation augmentations are used. A conditional Generative Adversarial Network (GAN) is used to generate fully synthetic data conditioned on a specific sample. The data generated by the two methods areintegrated as additional samples in the training of several Convolutional Neural Networks classifiers and the effect in the performance is examined.

The final contribution of the thesis introduces a new method for generating styled handwritten text images based on Denoising Diffusion Probabilistic Models (DDPM), which is an unexplored method in DIAR. The method manages to capture stylistic and content characteristics of a standard multi-writer handwriting dataset and achieved an improved performance in enhancing writer identification and handwriting text recognition compared to Generative Adversarial Network (GAN)-based methods. The results demonstrate the potential of the generative method for enabling deep document image analysis and pave the way for further research.

As a future direction, this work will aim to progress from generating word images to generating sentence and full document images by conditioning on the content, style, and layout of historical documents. Another future action will be to further extend the proposed method to operate in a few-shot scheme for the writer style condition in order to generate unseen styles. Furthermore, the future work will aim to leverage important features from pre-training with synthetic and real data in order to generalize to historical documents that are a scarce source and adjusting the text encoding parts to different languages and scripts. Finally, the ultimate goal of the future work aims to generate a massive synthetic historical document image database to fill the existing benchmark gap.

Text-to-Image synthesis is the task of generating an image according to a specific text description. Generative Adversarial Networks have been considered the standard method for image synthesis virtually since their introduction. Denoising Diffusion Probabilistic Models are recently setting a new baseline, with remarkable results in Text-to-Image synthesis, among other fields. Aside its usefulness per se, it can also be particularly relevant as a tool for data augmentation to aid training models for other document image processing tasks. In this work, we present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles, by using class index styles and text content prompts without the need of adversarial training, writer recognition, or text recognition. We gauge system performance with the Fréchet Inception Distance, writer recognition accuracy, and writer retrieval. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.