Kaldi-ark loading and writing module

2.18.1 · active · verified Mon Apr 13

kaldiio is a pure Python module for reading and writing Kaldi ark and scp files. It provides utilities for handling various Kaldi object types, including binary/text matrices, vectors, and compressed matrices, by interacting with NumPy arrays. The library also facilitates I/O operations through Unix pipes and supports some extended ark formats, including those used for NumPy, Pickle, WAV, and FLAC files. It is actively maintained with frequent minor and patch releases.

Warnings

gotcha Users often confuse `kaldiio` with older or related libraries like `kaldi_io` (from Karel Vesely) or `torchaudio.kaldi_io`. The `torchaudio.kaldi_io` module, in particular, is deprecated since version 2.8 and will be removed in 2.9, potentially leading to breaking changes if not using `kaldiio` directly.
Fix: Ensure you are importing directly from `kaldiio` (`from kaldiio import ...`) and that it is the installed package, not an indirect dependency or a different library.
breaking The `wav` option, previously available in `ReadHelper` for WAV file processing, was removed in version 2.11.0. Code relying on this option will break.
Fix: For WAV file I/O, use the dedicated `kaldiio.wavio` functions directly or leverage `kaldiio`'s support for reading WAV/FLAC files via extended ark formats if applicable. Review the `ReadHelper` documentation for current functionality.
breaking Version 2.16.0 introduced support for an 'extended ark format'. Older versions of `kaldiio` (prior to 2.16.0) might not be able to correctly read or fully utilize `.ark` files created with these extended formats by newer `kaldiio` versions or other Kaldi tools.
Fix: Upgrade `kaldiio` to the latest version to ensure compatibility with all supported `.ark` formats, especially if you are working with recently generated Kaldi data.
gotcha Kaldi's I/O often involves complex `rspecifiers` and `wspecifiers` which can include UNIX pipes (e.g., `ark:gunzip -c file.ark.gz |`). Incorrect formatting or environment issues with pipe commands are common pitfalls. `kaldiio` explicitly supports 'pipe fashion' for functions like `load_scp` (since v2.15.0) and `ReadHelper`.
Fix: Carefully construct your specifiers. For debugging pipe issues, try running the pipe command directly in the shell to isolate the problem. `kaldiio`'s `open_like_kaldi` and `ReadHelper` are designed to abstract away some of this complexity, but understanding Kaldi's I/O conventions is beneficial.

Install

pip install kaldiio Install stable version

Imports

load_ark
```
from kaldiio import load_ark
```
load_scp
```
from kaldiio import load_scp
```
save_ark
```
from kaldiio import save_ark
```
save_ark is a function directly exposed in the top-level package for convenience, not typically accessed as a submodule.
ReadHelper
```
from kaldiio import ReadHelper
```
WriteHelper
```
from kaldiio import WriteHelper
```
open_like_kaldi
```
from kaldiio import open_like_kaldi
```

Quickstart

This quickstart demonstrates how to write a dictionary of NumPy arrays to a Kaldi .ark file and generate a corresponding .scp file using `save_ark`. It then shows two common methods for reading data: `ReadHelper` for sequential iteration (supporting Kaldi-style specifiers and pipes) and `load_scp` for random access to features by utterance ID.

import numpy as np
from kaldiio import load_scp, ReadHelper, WriteHelper, save_ark

# Example: Create dummy data for writing
utt_ids = ['utt1', 'utt2', 'utt3']
features = {
    'utt1': np.random.rand(100, 40).astype(np.float32),
    'utt2': np.random.rand(120, 40).astype(np.float32),
    'utt3': np.random.rand(90, 40).astype(np.float32)
}

# Write features to an ARK file and generate an SCP file
output_ark_path = 'output.ark'
output_scp_path = 'output.scp'
save_ark(output_ark_path, features, scp=output_scp_path)
print(f"Written to {output_ark_path} and {output_scp_path}")

# Read SCP file using ReadHelper (sequential access, supports pipes)
print("\nReading via ReadHelper (scp:)...")
with ReadHelper(f'scp:{output_scp_path}') as reader:
    for key, matrix in reader:
        print(f"Key: {key}, Matrix shape: {matrix.shape}")

# Read SCP file using load_scp (random access, returns dict-like object)
print("\nReading via load_scp (random access)...")
loaded_features = load_scp(output_scp_path)
print(f"Loaded 'utt2' shape: {loaded_features['utt2'].shape}")

# Cleanup (optional)
import os
os.remove(output_ark_path)
os.remove(output_scp_path)

view raw JSON →