Practical example: creating an HTR model for an archival volume

Let's imagine that we have a digitized 300-page land register. The book was created over several decades and contains entries by several scribes. The goal is not to create a diplomatic edition of the entire source, but to obtain a sufficiently high-quality automatic transcription in which it will be possible to search for names of persons, place names, farmstead numbers and other data.

This example shows a practical procedure for creating your own HTR model.

1. Don't start by transcribing the first fifty pages

The first intuitive procedure is usually simple: open the book at the beginning and gradually transcribe the pages until there is enough Ground Truth.

However, such a selection may produce an unrepresentative training set. For example, if the first 80 pages were written by one scribe and the rest of the book by other scribes, the model will learn the first scribal hand very well, but may do significantly worse on the rest of the book.

First, it is therefore advisable to go through the entire document without detailed transcribing and find out its basic structure.

For each part of the book, we mainly monitor:

• scribal hand changes,
• significant period changes,
• language or spelling changes,
• different types of entries,
• page layout changes,
• tables and form parts,
• damaged or poorly scanned pages.

This might result, for example, in finding that pages 1-90 were mostly written by scribe A, pages 91-210 by scribe B, and the last section contains a mixture of entries by scribes B, C, and D.

This simple analysis is more important for further work than mechanically reaching a certain number of transcribed pages.

2. Select the first Ground Truth set

From a 300-page book, we can select, for example, 40 to 50 pages for the first experiment. However, this is not a universal recommended number; the actual need depends on the amount of text on the page and the variability of the handwriting.

We select pages across the entire book.

An illustrative selection might look like this:

Even this ratio may not be ideal in all circumstances. For example, if scribe D only has a few short entries in the book, it may not be practical to devote the same number of training pages to him as the main scribe.

The goal is to create a representative sample of the material that the model will actually read later.

3. Establish the rules of transcription before creating the Ground Truth

Before we start transcribing tens of pages, it is advisable to create a short document with transcription rules.

For the land register, for example, we set:

• we keep the original spelling;
• we do not modernize historical word forms;
• we preserve uppercase and lowercase letters according to the original, if they can be reliably identified;
• we write punctuation only where it is actually in the document;
• abbreviations are not rewritten in modern form;
• we use a uniform way of writing historical characters;
• we always mark unreadable places in the same way;
• we don't transcribe the text according to what we think "should have been written" based on the context, but according to what can actually be read.

Specific rules may vary from project to project. It is essential that the rules do not change arbitrarily while Ground Truth is being created.

For example, if one person transcribes a historical abbreviation verbatim and the other automatically expands it, the model receives two different responses to the same visual phenomenon.

4. Perform segmentation and check the lines

We will perform a layout analysis on the selected 50 pages.

Automatic segmentation usually speeds up the work significantly, but we will check its result. In particular, we will focus on:

• missing lines,
• two connected lines,
• one line incorrectly divided into several parts,
• marginal notes,
• titles,
• text extending into the binding,
• crossed out and additionally inserted entries.

Only then do we create or correct the transcription itself.

5. Create the first Ground Truth

If we have a suitable generic HTR model for the relevant language, period and handwriting, we can first run it on selected pages and manually correct its output.

If a suitable model does not exist or its results are very poor, we create a transcription manually.

The result must be a pair:

line image → verified line transcription

It is this data that represents the true basis of the future model.

It's not worth saving time at the expense of quality at this stage. If we create 50 pages of inaccurate Ground Truth, we are not getting any better training material than if we create a smaller but carefully reviewed dataset.

6. Set aside a portion of the data for validation

From the prepared Ground Truth, we will allocate a part of the pages on which we will monitor the quality of the model during training.

At 50 pages, for example, we could use:

45 pages for training set
5 pages for validation set

However, the division should not be purely random, if this would result in an unrepresentative validation set. For example, the five validation pages should not only come from Scribe A.

If the goal is a more serious evaluation, it is advisable to set aside a few more pages as a test set at the beginning. We will not use them either for our own training or for continuous decision-making during model development.

7. Train the first model

Custom training can now be started.

If there is a quality model created on similar historical material, it is usually advisable to first try using it as a base model and create a specialized variant with further training.

In addition, depending on the possibilities of the system used, a model without a suitable pre-trained basis can also be experimentally created and the results compared.

The goal of the first training may not yet be to create a definitive model. It is mainly about finding out what the model was able to learn from the prepared data and where it has problems.

8. Do not evaluate the model by only one CER

After completing the training, we get the Character Error Rate value on the validation data. For example, say the model achieves:

CER = 7.2%

The number itself looks promising, but it doesn't say whether the model is suitable for the entire book.

It is therefore necessary to find out which errors make up this 7.2%.

If the model mainly confuses punctuation or some variants of capital letters, it may already be very usable for full-text searches.

However, if it systematically distorts:

• surnames,
• place names,
• numbers,
• dates,
• sums of money,

it can be problematic for the intended historical research even with a relatively good overall CER.

9. Test individual scribes

Now we will take a few pages that the model didn't see at all during training.

For example, the result might look like this:

The average result of the whole model could seem satisfactory. However, the table shows something more substantial: the model handles scribes A and B very well, scribes C poorly, and fails significantly for scribe D.

So adding more random pages from the entire book may not be the solution.

It will be much more useful to create another Ground Truth mainly from the handwriting of scribe D and possibly C.

10. Use the first model to create additional data

We are now applying the first custom model to other representative pages.

For example, we originally spent 20 minutes manually transcribing one page. The first model can already create a transcript, the manual correction of which takes significantly less time.

Corrected pages will become the new Ground Truth.

This creates a work cycle:

manual Ground Truth → model 1 → automatic transcription → fix → extended Ground Truth → model 2

The second model can be trained on, for example, the original 45 training pages supplemented by an additional 20 pages focused mainly on scribes, for whom the first model failed.

11. Repeat the test on the same independent data

After creating the second version of the model, we repeat the evaluation.

For example, we can get:

Such a result would show that the targeted Ground Truth expansion really helped.

The numbers shown are only a model example, not the expected values ​​for an actual archive document.

12. When to stop training

There is no CER limit at which the model is automatically "done".

Purpose decides.

If we want to create a critical edition of a historical source, we will likely require human review of virtually the entire text anyway.

However, if we need to make 300 pages available for full-text searches and subsequent research work, the benefit of further reducing the CER from 5% to 3% may be less than the amount of work required to create additional Ground Truth data.

At some point, therefore, a practical question arises:

Is it still worth investing work in further training, or is it more profitable to run the model on the remaining documents?

The answer depends on the intended use of the resulting data.

13. Run the model on the entire book

Once the model is of acceptable quality on representative test pages, we can use it on the remaining approximately 230-250 pages of the book.

However, the work does not have to end there.

Automatic transcription can be further:

• be corrected manually,
• be indexed,
• be searched in full text,
• be exported to XML or another structured format,
• be used to search for people and places,
• be linked to database records,
• be used as a basis for further automatic processing.

For an archival data project, it is important to keep the relationship between the transcription and the original image of the document in addition to the resulting text. The researcher can thus return to the original at any time and verify the automatically recognized data.

What to take away from the example

The most important principle is not to "transcribe as many pages as possible". The goal is to create as small as possible, but sufficiently representative and high-quality Ground Truth, which corresponds to the real variability of the processed material.

For the first project, the entire procedure can be simplified to:

1. View the entire document.
2. Identify the main scribes and types of entries.
3. Select representative pages.
4. Establish uniform transcription rules.
5. Check segmentation.
6. Create quality Ground Truth.
7. Separate training, validation and possibly test data.
8. Train the first model.
9. Test it on previously unused pages and individual scribes.
10. Supplement Ground Truth especially where the model fails.
11. Retrain the model.
12. Once the quality matches the purpose of the project, apply it to the rest of the document.

This iterative approach has another important advantage. The result of the project is not only a trained HTR model. It also creates a verified Ground Truth corpus that can be preserved, published, used to train future models or combined with other datasets. Therefore, in the long run, a well-made Ground Truth may have more value than a specific version of the HTR model, which may later be replaced by a more technically advanced solution.