RecordLinkage

Warnings

• breaking Version 0.15 removed several deprecated classes and bumped the minimum required Python version to 3.8 and pandas version to >=1.0. Older code using removed classes or incompatible Python/pandas versions will break.
  Fix: Review the release notes for version 0.15 to update deprecated class usages and ensure your environment meets the new minimum Python (3.8+) and pandas (1.0+) requirements.
• gotcha The choice of blocking keys can introduce significant bias, as true matches that do not share the exact same blocking key will be missed. This is especially problematic with data entry errors or variations.
  Fix: Carefully select blocking keys, considering multiple keys (union of blocks), using more error-tolerant indexing methods (e.g., sorted neighbourhood, phonetic blocking), or applying pre-processing to standardize blocking fields.
• gotcha String comparison performance varies greatly between algorithms. Highly accurate but computationally intensive algorithms like Damerau-Levenshtein can be much slower than Jaro-Winkler or Jaro, especially on large datasets. Ensure the 'jellyfish' library (Rust version) is correctly installed for optimal string comparison speed.
  Fix: Benchmark different string comparison methods for your specific data and choose a balance between accuracy and performance. Prioritize 'jellyfish' for string comparisons and ensure the Rust-backed version is active (check `import jellyfish.rustyfish` does not raise an exception).
• gotcha Effective record linkage heavily depends on prior data cleaning and standardization. Inconsistent formatting, typos, missing values, or variations in how information is recorded (e.g., 'St.' vs 'Street') can severely hinder matching accuracy.
  Fix: Implement robust pre-processing steps using `recordlinkage.standardise` or other data cleaning libraries to normalize names, addresses, dates, and other attributes before attempting to link records.

Install

• pip install recordlinkage Standard Install
• pip install recordlinkage['all'] Install with Recommended and Optional Dependencies

Imports

• recordlinkage

  import recordlinkage

  Main module import.
• Index

  from recordlinkage import Index

  To create an indexer object.
• Block

  from recordlinkage.index import Block

  Direct import for specific indexing algorithms.
• Compare

  from recordlinkage import Compare

  To create a comparator object.
• Exact

  from recordlinkage.compare import Exact

  Direct import for specific comparison measures.

Quickstart

This quickstart demonstrates the core steps of record linkage: generating candidate pairs using blocking on a common attribute (city/town), comparing these pairs using string similarity and exact matches, and then classifying likely matches based on the comparison scores.

import recordlinkage
import pandas as pd

# Dummy data for demonstration
df_a = pd.DataFrame({
    'name': ['John Doe', 'Jane Smith', 'Peter Jones'],
    'city': ['New York', 'Los Angeles', 'Chicago']
}, index=['id_a_1', 'id_a_2', 'id_a_3'])

df_b = pd.DataFrame({
    'full_name': ['Jon Doe', 'Jane Smiht', 'Pete Jones'],
    'town': ['New York', 'Los Angles', 'Chicago']
}, index=['id_b_1', 'id_b_2', 'id_b_3'])

# 1. Indexing: Generate candidate links using blocking
indexer = recordlinkage.Index()
indexer.block(left_on='city', right_on='town')
candidate_links = indexer.index(df_a, df_b)

print(f'Generated {len(candidate_links)} candidate links.')

# 2. Comparing: Compare records on relevant attributes
compare_cl = recordlinkage.Compare()
compare_cl.string('name', 'full_name', method='jarowinkler', label='name_similarity')
compare_cl.exact('city', 'town', label='city_exact')

features = compare_cl.compute(candidate_links, df_a, df_b)

# 3. Classification: Decide on matches based on comparison features
# Simple sum of scores (1 for match, 0 for mismatch)
matches = features[features.sum(axis=1) > 1.5]

print('\nIdentified Matches:')
print(matches.index.to_list())