Automatic detection and recognition of Arabic text in video scenes

Abstract

This thesis focuses on the problem of Arabic text recognition from scene videos, a research area that has gained increasing importance in recent years due to the rapid growth of visual media on the internet, social networks, and video-sharing platforms. Extracting textual information from videos is a key step for many ap- plications, including video indexing, content-based retrieval, assistive technologies, and intelligent surveillance systems. However, Arabic scene text recognition re- mains particularly challenging. The Arabic script is inherently cursive, characters can take multiple shapes depending on their position within a word, and diacritics may appear or disappear depending on writing style. These linguistic characteristics, combined with common video-related issues such as motion blur, low resolution, compression artifacts, varying illumination, and complex backgrounds, make Arabic text recognition significantly more difficult than its Latin counterpart. In addition to these intrinsic challenges, the field suffers from a major limitation in terms of data availability. Existing Arabic scene text datasets are generally small, limited in font diversity, or captured under constrained conditions, which makes them insufficient for training modern deep learning models that typically require large and diverse datasets to generalize well. As a result, many state-of-the-art architectures struggle to achieve robust performance when applied to real-world Arabic video data. The work presented in this thesis aims to address these challenges from two complementary perspectives: data generation and model design. The first major contribution of this research is the design, construction, and public release of a large-scale artificial dataset named IYaD, specifically tailored for Arabic scene text recognition. The dataset was carefully generated to reflect the visual and structural complexity of Arabic text as it appears in natural scenes. It contains approximately 1,400,000 images for a single font and a comparable number of images for sixteen additional fonts, resulting in a dataset with exceptional size and stylistic diversity. Each text sample is rendered in three distinct visual versions and is accompanied by rich annotations, including its Arabic ground-truth label, a corresponding Latin transcription, and the original textual content. Although IYaD is synthetically generated, a detailed complexity analysis and 3multiple comparisons with real-world datasets were conducted to demonstrate that it closely simulates the visual characteristics of natural Arabic scene text. Further more, the dataset is distributed together with a complete set of tools that facilitate preprocessing, augmentation, transformation, and seamless integration into existing deep learning pipelines. By providing both scale and diversity, IYaD fills a critical gap in the available resources for Arabic text recognition and enables more effective training of data-hungry neural architectures. The second main contribution of this thesis is the proposal of a novel Transformerbased architecture for Arabic Scene Text Recognition (ASTR). This work is grounded in a comprehensive comparative study between two different modeling strategies. The first strategy follows a more conventional approach, where convolutional neural networks (CNNs) are used to extract visual feature maps that are subsequently processed by a Transformer encoder. The second strategy removes the convolutional component entirely and relies on a purely Transformer-based model to directly learn visual representations and sequence dependencies. Extensive experiments were carried out using all publicly available Arabic text recognition datasets, as well as the newly introduced IYaD dataset. The results clearly indicate that the fully Transformer-based architecture outperforms the hybrid CNN–Transformer approach across multiple evaluation settings. In particular, when trained on IYaD, the proposed model achieves performance levels that surpass the current state of the art, while also demonstrating strong robustness to variations in font style, image quality, and visual noise. These findings highlight not only the effectiveness of Transformer architectures for Arabic OCR, but also the crucial role that large-scale artificial datasets can play in overcoming the limitations of scarce real-world data. In addition to these core contributions, this thesis provides a comprehensive experimental benchmark for Arabic text recognition. Several well-known OCR models were implemented, trained, and evaluated under consistent and controlled conditions. This benchmark offers a detailed comparison of different architectural choices and training strategies, helping to clarify their respective strengths and weaknesses. It also serves as a valuable reference framework for future research in the domain. Overall, this thesis contributes new datasets, new modeling approaches, and new experimental insights to the field of Arabic scene text recognition. By combining a carefully designed large-scale artificial dataset with a modern Transformer based recognition architecture, it demonstrates a practical and effective path to- ward improving Arabic OCR performance in video-based and real-world scenarios, and it opens promising directions for future research in multilingual and lowresource text recognition.