LM Dataformat

LM Dataformat (lm-dataformat) is a Python utility designed for efficient storage and reading of files specifically tailored for large language model (LLM) training. It provides functionalities to archive data with associated metadata and stream documents for processing. The current version is 0.0.20, but the project appears to be abandoned, with no active development or maintenance since its last release in 2021 and last GitHub commit over six years ago.