Enriching Historical Records: An OCR and AI-Driven Approach for Database Integration

Authors

  • Zahra Abedi University of Leiden
  • Richard M.K. van Dijk University of Leiden
  • Gijs Wijnholds University of Leiden
  • Tessa Verhoef University of Leiden

Abstract

This research digitizes and analyzes the Leidse hoogleraren en lectoren 1575-1815 books written between 1983 and 1985, which contain biographic data about professors and curators of Leiden University. It addresses the central question: ’How can we design an automated pipeline that integrates OCR, LLM-based interpretation, and database linking to harmonize data from historical document images with existing high-quality database records?’ We applied OCR techniques, generative AI decoding constraints that structure data extraction, and database linkage methods to process typewritten historical records into a digital format. OCR achieved a Character Error Rate (CER) of 1.08% and a Word Error Rate (WER) of 5.06%, while JSON extraction from OCR text achieved an average accuracy of 63% and, based on annotated OCR, 65%. This indicates that generative AI somewhat corrects low OCR performance. Our record linkage algorithm linked annotated JSON files with 94% accuracy and OCR-derived JSON files with 81%. This study contributes to digital humanities research by offering an automated pipeline for interpreting digitized historical documents, addressing challenges like layout variability and terminology differences, and exploring the applicability and strength of an advanced generative AI model.

Downloads

Published

2025-07-15

How to Cite

Abedi, Z., van Dijk, R. M., Wijnholds, G., & Verhoef, T. (2025). Enriching Historical Records: An OCR and AI-Driven Approach for Database Integration. Computational Linguistics in the Netherlands Journal, 14, 401–420. Retrieved from https://clinjournal.org/clinj/article/view/208

Issue

Section

Articles