Training HTR models for historical manuscript recognition

What is an HTR model?

Handwritten Text Recognition (HTR) is a technology designed to automatically recognize handwritten text from digital document images. Unlike classic OCR, which has historically focused mainly on printed documents, HTR has to work with significantly greater variability of characters, words and entire scribal styles. The same letter can have many forms in a manuscript, and its appearance varies not only between individual scribes, but also according to the surrounding characters, the writing instrument used, the period of creation of the document or the physical condition of the original. [1]

Modern HTR systems therefore do not recognize handwriting only as a collection of isolated letters. They use machine learning models that learn from examples the relationship between the visual form of the text and the corresponding sequence of characters. For historical sources, this approach is particularly important, as it allows the creation of models adapted to a certain type of document, period, language or specific scribal hands. [1] [2]

Ground Truth- the basis of training

The most important prerequisite for creating your own HTR model is high-quality training data, usually referred to as Ground Truth (GT). It is a pair formed by an image of a handwritten text and its exact human transcription. During training, the model receives an image of the text together with information about what kind of text is actually found in the given image, and gradually learns to combine the two representations. [2] [3]

Ground Truth quality is critical. Errors in manual transcription are not just "noise" for the model: during training, they can be presented as the correct result for the model to learn. For that reason, the training transcription should not only be as accurate as possible, but also internally consistent. [2]

Consistency applies for example to:

• transcription of upper- and lowercase letters,
• punctuation,
• spaces,
• historical spelling,
• abbreviations and whether to expand them,
• diacritics,
• special characters,
• unreadable or damaged places.

Before starting a more extensive transcription, it is therefore advisable to establish uniform transcription rules. From a machine learning point of view, a simpler and consistently applied system tends to be more beneficial than very complicated rules that are not used consistently in the data. [3]

How much training data is needed

There is no universal answer to the question of how many pages must be manually transcribed. The result depends on the homogeneity of the documents, the number of scribal hands, the language, the quality of digitization, the complexity of the handwriting, and whether the model is created from scratch or is based on an already existing model.

The Transkribus platform documentation gives an approximate initial range of approximately 5,000 to 15,000 words, or roughly 25 to 75 pages of Ground Truth. For manuscript material, it recommends approximately 10,000 words for one scribal hand. Large-scale models based on more than 100,000 words and many hands from a similar period and region may be able to work with manuscripts that were not directly part of the training data. [2] However, these numbers should be understood as a practical recommendation, not as a fixed limit.

More important than the sheer number of pages is their representativeness. If the model is to be applied to, for example, an archival collection containing several scribes, it is not sufficient to generate a large amount of Ground Truth from the handwriting of a single scribe. The training material should contain a reasonable representation of different forms of writing, which the model is supposed to recognize later. [2]

The same applies to documents created over a long period of time. If an author's handwriting has changed over decades, different stages of his writing should be represented in the training data.

Document segmentation

Text recognition is not the only task of the whole process. Even before that, the system must determine where the text is located on the page. A document may contain several text areas, columns, marginal notes, tables, or other elements complicating automatic analysis.

Therefore, when training a recognition model, the image regions or lines to which the respective text belongs are also associated with the transcriptions. Common data formats, such as PAGE XML or ALTO XML, allow transcription and information about the position of the text in the image to be stored together. [3]

The quality of the segmentation can significantly affect the final result. Experimental work shows that the model can achieve significantly better results with properly prepared text lines than when using faulty automatic segmentation. [4] Incorrectly defined line boundaries can therefore cause errors that at first glance appear to be a deficiency of the HTR model itself.

Training and validation data

During training, Ground Truth is usually divided so that not all of it is used directly to teach the model. Part of the material is set aside as a validation set.

The training set is used to adjust model parameters. Validation data, on the other hand, are not used as direct examples for its learning, but allow to observe during training how the model works with material that it is not currently using for its own learning. By default, Transkribus works with the Ground Truth section set aside for validation and recommends around 10% of the available transcripts in its documentation. [5]

The validation set must also be representative. If the training data contains, for example, several scribes, but the validation set is almost exclusively a single one, the calculated success rate may not well characterize the actual applicability of the model on the entire set of documents. [5]

For a more thorough scientific evaluation, in addition to the training and validation set, a separate test set can be created, which will be held aside during model development and used only for the final independent evaluation. Such a separation of training, validation and test data is also used in research experiments with HTR. [6]

CER – Character Error Rate

One of the basic quality indicators of the HTR model is Character Error Rate (CER). It expresses the proportion of character-level errors when comparing auto-recognized text to the correct Ground Truth transcription. The result includes missing, redundant and incorrectly recognized characters. [7]

In simple terms it can be said that:

CER = (substitution + omission + insertion of characters) / number of characters of reference text

The lower the CER, the more accurate the automatic transcription. For example, Transkribus states that a CER of 5% roughly corresponds to a situation where out of every 100 characters, 95 are recognized correctly and five incorrectly. [7]

However, the CER value itself must be interpreted in context. A model with a very low CER on documents similar to its training data may not perform as well on other scribes or another type of archival material. When comparing models, it is therefore important to know on which data the CER was calculated and how this data resembles the documents on which the model is to be deployed.

Model overfitting

One of the risks of training is overfitting, i.e. excessive adaptation to the training data. A model can adapt very well to training examples without learning the more general characteristics of the handwriting as well. The result can be a very low error rate on known data, but significantly worse recognition of new documents.

It's watching the results on the validation set that helps reveal this problem. Modern training procedures can also use early stopping: training is terminated when further epochs no longer bring improvements in the results of the validation data. [5]

So it is not the case that the longer we train the model, the better it must be. What is decisive is its ability to generalize learned properties to previously unseen material.

Using an already existing model

Model creation doesn't always have to start from scratch. If there is a model trained on a similar language, period or scribal style, it can be used as a base model in some HTR systems and further adapted to your own data. [5]

This procedure is usually referred to as fine-tuning or continued training. Its advantage is that the model already contains some knowledge of the character structures and does not need to learn everything exclusively from the newly created Ground Truth. This can reduce the amount of hand-transcribed material required to produce a usable result. However, the benefit of the starting model is not automatic: it must be sufficiently similar to the target material and its suitability must be verified experimentally. [5]

The importance of fine-tuning is also confirmed by experimental studies. For some historical corpora, fitting a suitable pretrained model gave better results than building a model from scratch. [4] [8]

Specialized or general model?

When preparing the HTR project, it is important to determine how wide a range of documents the resulting model should handle.

A specialized model can be focused on, for example, a single scribe, a single office, or a homogeneous series of archival documents. Because it solves a more limited problem, it can achieve very good results on the corresponding material even with a relatively limited amount of Ground Truth.

The general model, on the other hand, is supposed to recognize a larger number of scribes, periods or types of documents. It therefore requires much more varied training data. Its advantage is wider applicability, while its disadvantage may be lower accuracy on specific scribal hands compared to a narrowly specialized model. [2] [9]

An extensive general model can also serve as a basis for further specialization. For archival practice, a combination of a general pre-trained model and a smaller amount of high-quality Ground Truth created directly from a specific archival collection can be effective.

Iterative creation of HTR model

In practice, Ground Truth may not be created entirely by hand before the first training. A procedure in several cycles can be effective:

1. manually transcribe the initial set of representative pages,
2. create the first HTR model,
3. apply it to other documents,
4. manually correct the automatic transcriptions,
5. put the corrected pages among Ground Truth,
6. create a new version of the model,
7. repeat the procedure as needed.

Such a procedure makes it possible to gradually expand the training corpus. However, it is important to emphasize that just fixing auto-transcription alone does not teach a pre-existing model. The corrected data must be used during further training or fine-tuning of the model. [10]

When choosing new pages, it is advisable not to focus only on easy-to-read documents. If the resulting model is to handle the entire archival collection, Ground Truth should gradually be supplemented with representative examples of more difficult manuscripts, different scribes and other typical variants.

Importance of open training data

Training data created for one project may also be of value for other research. The HTR-United initiative therefore creates a catalog of documented datasets intended for automatic transcription and segmentation of historical documents. Data files can be published, for example, in PAGE XML, ALTO XML or as a combination of images and text files, and their metadata also includes terms of further use. [11]

Sharing quality Ground Truth allows not only to reproduce research results, but also to create more general models or use pre-existing data to train models for related material. In the long run, therefore, the most valuable result of the project may not only be the HTR model itself, but also the carefully created and well-documented training corpus.

HTR as part of working with archive data

HTR cannot be understood as a one-time "reading" of a manuscript by a computer. In archival practice, it is more of a process in which digitization, page layout analysis, human transcription, Ground Truth preparation, model training, automatic recognition and result checking are connected.

The main advantage of HTR does not have to be the creation of a perfectly flawless edition of the document. Even a transcript containing a certain number of errors can significantly expand the possibilities of working with extensive archival collections: it allows full-text searching, preliminary orientation in documents and creates a machine-readable basis for further processing of historical data. HTR can thus make available the content of digitized manuscripts, which would otherwise remain available only through time-consuming manual reading of individual documents. [1]

Notes and sources used

[1] MUEHLBERGER, Günter et al.; COLUTTO, Sebastian et al. Transforming scholarship in the archives through handwritten text recognition. Journal of Documentation, 2019. DOI: 10.1108/JD-07-2018-0114.
Article and bibliographic data

[2] READ-COOP. Data Preparation – Training Text Recognition Models. Transkribus documentation. Recommendations for the creation of Ground Truth, the amount of training data and the representation of scribal hands.
Transkribus Documentation – Data Preparation

[3] Kraken OCR. Introduction to Automatic Text Recognition. Kraken system documentation; description of preparation of training data, transcription rules and PAGE XML and ALTO formats.
Kraken – Introduction to Automatic Text Recognition

[4] BÜTTNER, Jonas et al. From Manuscript to Model: Developing HTR for Medieval Greek. Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026), pp. 139–151. Among other things, the study compares training from the beginning with fine-tuning and shows the effect of segmentation on the resulting error rate. DOI: 10.63317/47tkobv5mgbu.

[5] READ-COOP. Model Setup and Training. Transkribus documentation. Description of the training and validation set, base models, epochs and early stopping.