Splink

4.0.16 · active · verified Mon Apr 13

Splink is a Python package for fast, accurate, and scalable probabilistic record linkage (entity resolution). It enables users to deduplicate and link records from datasets that lack unique identifiers, leveraging unsupervised learning based on the Fellegi-Sunter model. Splink supports various SQL backends like DuckDB, Apache Spark, and AWS Athena, allowing it to scale to datasets of 100 million records or more, and provides a suite of interactive visualizations for model understanding and diagnostics.

Warnings

breaking Splink v5.0 introduces significant breaking changes. Key updates include the removal of the implicit cache mechanism in favor of explicit cache table management functions, removal of 'salting', introduction of 'chunking' for large datasets, and a shift from Bayes Factors to Match Weights (log-odds) for internal probabilistic calculations to improve numerical stability. Additionally, support for the Athena backend is being dropped.
Fix: Review v5.0 documentation and migration guides for updated API calls, cache management, and probabilistic calculation handling. Users relying on Athena should plan for migration to another backend or use Splink v4.x.
breaking Python 3.8 support was dropped in Splink v4.0.12. Older versions of Python are being phased out in alignment with community end-of-life policies.
Fix: Ensure your environment uses Python 3.9 or higher. The current requirement is `>=3.9.0, <4.0.0`.
gotcha Splink performs best with input data containing multiple, non-highly correlated columns. It is not designed for linking single-column 'bag of words' data (e.g., only a company name). High correlation (e.g., city and postcode) can also reduce effectiveness.
Fix: Pre-process data to ensure multiple, diverse columns are used for linkage. Avoid relying on highly correlated features or single 'bag of words' columns for optimal accuracy.
deprecated SQLite backend support is minimal and receives less attention from the development team compared to DuckDB and Spark. It has reasonable but not complete coverage of comparison functions, particularly for array and date comparisons.
Fix: For optimal performance and feature coverage, especially with larger datasets or complex comparisons, consider using DuckDB (default) or other actively supported backends like Spark or PostgreSQL.

Install

pip install splink Base Install (includes DuckDB and SQLite)
pip install 'splink[spark]' For Apache Spark backend
pip install 'splink[athena]' For AWS Athena backend
pip install 'splink[postgres]' For PostgreSQL backend

Imports

Linker
```
from splink import Linker
```
SettingsCreator
```
from splink import SettingsCreator
```
block_on
```
from splink import block_on
```
DuckDBAPI
```
from splink import DuckDBAPI
```
splink_datasets
```
from splink import splink_datasets
```
cl
```
import splink.comparison_library as cl
```

Quickstart

This quickstart demonstrates how to set up a basic Splink deduplication model using DuckDB. It covers defining comparison libraries and blocking rules, estimating parameters for record linkage, and preparing for prediction. It uses a built-in `fake_1000` dataset for convenience.

import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets

db_api = DuckDBAPI()
df = splink_datasets.fake_1000

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.NameComparison("first_name"),
        cl.JaroAtThresholds("surname"),
        cl.DateOfBirthComparison("dob", input_is_string=True),
        cl.ExactMatch("city").configure(term_frequency_adjustments=True),
        cl.EmailComparison("email"),
    ],
    blocking_rules_to_generate_predictions=[
        block_on("first_name", "dob"),
        block_on("surname"),
    ]
)

linker = Linker(df, settings, db_api)

linker.training.estimate_probability_two_random_records_match(
    [block_on("first_name", "surname")], recall=0.7
)
linker.training.estimate_u_using_random_sampling(max_pairs=1e6)
linker.training.estimate_parameters_using_expectation_maximisation(
    block_on("first_name", "surname")
)

# To get the results, e.g., predictions_df = linker.inference.predict()

view raw JSON →