Hyparquet

1.25.6 · active · verified Sun Apr 19

Hyparquet is a pure JavaScript library for parsing Apache Parquet files directly in web browsers and Node.js environments. It specializes in efficient data retrieval from cloud storage by leveraging HTTP range requests, allowing for direct querying of Parquet files over the network without requiring a server-side intermediary. The library is dependency-free since 2023, offering a lightweight solution. Its current stable version is 1.25.6, with a release cadence that follows active development. Key differentiators include its ability to minimize data fetches through selective row and column filtering, comprehensive support for all Parquet types, encodings, and compression codecs, and inclusion of TypeScript definitions for improved developer experience. It is particularly well-suited for data engineering, data science, and machine learning applications where large datasets stored in Parquet format need to be accessed and processed client-side.

Common errors

```
TypeError: require is not a function
```
cause Attempting to import hyparquet using CommonJS `require()` syntax in a Node.js environment or bundler that expects ES modules.

fix Change `const { ... } = require('hyparquet')` to `import { ... } from 'hyparquet'`. Ensure your Node.js project or bundler is configured for ES modules (e.g., 'type': 'module' in package.json or using .mjs extension).
```
TypeError: Cannot read properties of undefined (reading 'slice')
```
cause The `file` argument passed to `parquetReadObjects` or `parquetMetadataAsync` is not a valid `AsyncBuffer` instance or a compatible object.

fix Ensure `file` is created using `asyncBufferFromUrl({ url })` for remote files or `asyncBufferFromFile('path/to/file.parquet')` for local Node.js files, or provide a custom object implementing the `AsyncBuffer` interface correctly.
```
Uncaught (in promise) TypeError: x.map is not a function
```
cause This error often occurs when attempting to map over a non-array result, sometimes indicating that the Parquet file was malformed, or the schema parsing returned an unexpected structure.

fix Verify the integrity and structure of your Parquet file. Ensure the `parquetSchema` function is used correctly, and its output (e.g., `schema.children`) is what you expect to iterate over. Inspect `metadata` and `schema` objects before mapping.

Warnings

breaking Hyparquet is distributed exclusively as an ES module (ESM). Direct 'require()' calls for CommonJS environments are not supported.
Fix: Use ESM import statements (e.g., `import { ... } from 'hyparquet'`) or dynamic `import()` for compatibility with older Node.js versions. Configure your project to handle ES modules.
gotcha When reading remote Parquet files with `asyncBufferFromUrl`, efficient performance relies on the server supporting HTTP range requests. Without proper server support, the entire file might be downloaded.
Fix: Ensure your HTTP server (e.g., S3, Google Cloud Storage) is configured to handle `Range` headers for partial content requests. Test with small files first.
gotcha The `num_rows` property from Parquet metadata is returned as a `BigInt`. Direct arithmetic operations or comparisons with standard `Number` types may lead to errors or unexpected results.
Fix: Convert `BigInt` values to `Number` using `Number(metadata.num_rows)` when working with smaller numbers, or ensure your application logic correctly handles `BigInt` types for large row counts.
gotcha To optimize data fetching for large remote files, always specify `columns`, `rowStart`, and `rowEnd` parameters in `parquetReadObjects`. Failing to do so will result in downloading and parsing the entire file.
Fix: Define `columns` to specify only the needed columns and `rowStart`/`rowEnd` to limit the row range, reducing network bandwidth and parsing overhead. Example: `parquetReadObjects({ file, columns: ['col1'], rowStart: 0, rowEnd: 100 })`.

Install

npm install hyparquet npm
yarn add hyparquet yarn
pnpm add hyparquet pnpm

Imports

parquetReadObjects
wrong:
```
const { parquetReadObjects } = require('hyparquet')
```
correct:
```
import { parquetReadObjects } from 'hyparquet'
```
Hyparquet is published as an ES module; CommonJS 'require()' is not supported directly. Use dynamic import() or ensure your Node.js environment supports ES modules.
asyncBufferFromUrl
```
import { asyncBufferFromUrl } from 'hyparquet'
```
Used to create an AsyncBuffer from a URL, suitable for browser environments or Node.js network fetches. For local files in Node.js, use asyncBufferFromFile.
parquetMetadataAsync
wrong:
```
import parquetMetadataAsync from 'hyparquet'
```
correct:
```
import { parquetMetadataAsync, parquetSchema } from 'hyparquet'
```
parquetMetadataAsync and parquetSchema are named exports, not default. They are crucial for inspecting file structure and statistics without full data loading.
AsyncBuffer
```
import type { AsyncBuffer } from 'hyparquet'
```
This is a TypeScript type import for custom implementations of the AsyncBuffer interface.

Quickstart

Demonstrates how to fetch and parse a remote Parquet file in a browser or Node.js using HTTP range requests, filtering for specific columns and rows.

import { asyncBufferFromUrl, parquetReadObjects } from 'hyparquet'

async function fetchData() {
  const url = 'https://hyperparam-public.s3.amazonaws.com/bunnies.parquet'
  // Wrap the URL for asynchronous fetching with HTTP range requests
  const file = await asyncBufferFromUrl({ url })
  // Read objects, filtering by specific columns and rows for efficiency
  const data = await parquetReadObjects({
    file,
    columns: ['Breed Name', 'Lifespan'],
    rowStart: 10,
    rowEnd: 20,
  })
  console.log('Fetched data:', data)
}

fetchData().catch(console.error)

view raw JSON →