KafkaJS Snappy Codec (TypeScript)

1.0.3 · active · verified Sun Apr 19

kafkajs-snappy-typescript provides a Snappy compression and decompression codec specifically designed for use with KafkaJS, offering strict TypeScript compatibility. This package serves as a type-safe alternative to `tulios/kafkajs-snappy`, aiming to integrate seamlessly into TypeScript-first KafkaJS applications. It leverages Brooooooklyn's `snappy` Node.js compression library for its core compression logic. The current stable version is 1.0.3, primarily consisting of bug fixes and rebundling efforts to ensure proper NPM publication. As a specialized codec, its release cadence is tied to necessary compatibility updates with KafkaJS or its underlying `snappy` dependency, rather than frequent feature additions. Its key differentiator is its explicit focus on strong typing for KafkaJS environments.

Common errors

```
Error: Unknown compression type
```
cause The Snappy codec was not registered with KafkaJS's `CompressionCodecs` before a producer attempted to send a Snappy-compressed message or a consumer tried to decompress one.

fix Add `CompressionCodecs[CompressionTypes.Snappy] = new SnappyCodec().codec;` to your application startup code, ensuring it runs before KafkaJS operations.
```
node-pre-gyp ERR! build error
```
cause The underlying `snappy` package (which uses native code) failed to compile during installation, often due to missing build tools like `python` or `make`, or incorrect compiler setup.

fix Ensure you have the necessary build tools installed on your system. For `node-gyp`, this typically includes Python 3, `make` (or Xcode on macOS, Visual C++ Build Tools on Windows). Consult the `node-gyp` and `snappy` documentation for detailed prerequisites.
```
TypeError: Cannot read properties of undefined (reading 'codec')
```
cause You might be attempting to access `.codec` on `SnappyCodec` without instantiating it first, or `SnappyCodec` itself might be undefined due to an incorrect import.

fix Ensure you are instantiating the class correctly: `new SnappyCodec().codec`. Also, verify the import path: `import { SnappyCodec } from 'kafkajs-snappy-typescript';`

Warnings

gotcha This package is a codec and must be explicitly registered with KafkaJS via `CompressionCodecs[CompressionTypes.Snappy] = new SnappyCodec().codec;` before producing or consuming Snappy-compressed messages. Failing to register it will result in KafkaJS not knowing how to handle Snappy compression.
Fix: Ensure the codec registration line is executed before any KafkaJS producer or consumer operations that involve Snappy compression.
gotcha The underlying `snappy` package, used for compression, relies on native C++ bindings. This can sometimes lead to compilation issues during `npm install` on certain environments or architectures if build tools (like `node-gyp` dependencies) are not correctly set up. Check the `snappy` package documentation for specific build requirements.
Fix: Refer to the official `snappy` (Brooooooklyn/snappy) package's installation instructions and ensure all necessary system-level build tools (e.g., Python, C/C++ compiler) are installed and configured for `node-gyp`.
gotcha This library is a TypeScript-focused alternative to `tulios/kafkajs-snappy`. While functionally similar, using both or mixing their import styles could lead to type conflicts or confusion.
Fix: Choose one Snappy codec library for KafkaJS (either `kafkajs-snappy-typescript` or `kafkajs-snappy`) and stick with it throughout your project for consistency and to avoid potential conflicts.

Install

npm install kafkajs-snappy-typescript npm
yarn add kafkajs-snappy-typescript yarn
pnpm add kafkajs-snappy-typescript pnpm

Imports

SnappyCodec
wrong:
```
const { SnappyCodec } = require('kafkajs-snappy-typescript');
```
correct:
```
import { SnappyCodec } from 'kafkajs-snappy-typescript';
```
The primary export is a named class, intended for ESM usage with TypeScript. CommonJS `require` should generally be avoided in favor of ES modules.
CompressionTypes
```
import { CompressionTypes, CompressionCodecs } from 'kafkajs';
```
While this package provides the codec, KafkaJS's `CompressionTypes` and `CompressionCodecs` are external imports necessary for codec registration.

Quickstart

This quickstart demonstrates how to register the Snappy codec with KafkaJS and then use it to send and receive messages with Snappy compression.

import { Kafka, CompressionTypes, CompressionCodecs } from 'kafkajs';
import { SnappyCodec } from 'kafkajs-snappy-typescript';

// Register the Snappy codec with KafkaJS
CompressionCodecs[CompressionTypes.Snappy] = new SnappyCodec().codec;

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'],
});

const producer = kafka.producer();
const consumer = kafka.consumer({ groupId: 'test-group' });

async function run() {
  await producer.connect();
  await consumer.connect();

  await producer.send({
    topic: 'test-topic',
    compression: CompressionTypes.Snappy, // Use Snappy compression
    messages: [
      { value: 'Hello KafkaJS with Snappy!' },
    ],
  });
  console.log('Message sent with Snappy compression.');

  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        value: message.value?.toString(),
        headers: message.headers,
      });
    },
  });

  // Disconnect after a short period for demonstration
  setTimeout(async () => {
    await producer.disconnect();
    await consumer.disconnect();
    console.log('Producer and consumer disconnected.');
  }, 5000);
}

run().catch(console.error);

view raw JSON →