UTF-8 Encoder/Decoder for JavaScript

Common errors

• Error: A lone surrogate code point was found.

  cause Attempting to encode a JavaScript string containing an unpaired (lone) surrogate character (e.g., `\uD800` without a trailing `\uDC00` to `\uDFFF`).
  fix Validate and sanitize input strings to ensure they are well-formed Unicode before passing them to `utf8.encode()`. You can replace lone surrogates or use `String.prototype.toWellFormed()` (if targeting environments that support it) or related libraries.

• Error: Malformed UTF-8 data.

  cause The input string provided to `utf8.decode()` contains byte sequences that do not conform to valid UTF-8 encoding rules.
  fix Check the origin and integrity of the byte string being decoded. Ensure it has been correctly encoded as UTF-8. Wrap `utf8.decode()` calls in a `try...catch` block to handle invalid input gracefully, e.g., by logging the error and using a fallback or replacement.

• TypeError: require is not a function (in ES module context) or SyntaxError: Cannot use import statement outside a module (in CJS context)

  cause This package is a CommonJS module. Using `require()` in an ES module context or `import` in a CommonJS context will lead to module resolution errors.
  fix For Node.js, ensure you use `const utf8 = require('utf8');` in CommonJS modules (`.js` files where `"type": "module"` is not set or in `.cjs` files). If working in an ES module environment (`.mjs` files or `"type": "module"` in `package.json`), you may need to use dynamic `import('utf8')` or rely on a bundler like Webpack or Rollup to handle the CommonJS dependency.

Warnings

• gotcha The `utf8.encode()` method will throw an `Error` if the input JavaScript string contains a non-scalar value, specifically a lone surrogate (a `U+D800` to `U+DFFF` code point not part of a valid surrogate pair). This strict behavior is by design, adhering to the Encoding Standard for proper UTF-8.
  Fix: Ensure input strings are 'well-formed' Unicode. Pre-process strings to remove or replace lone surrogates (e.g., with `U+FFFD` replacement character) before calling `utf8.encode()`. For scenarios requiring encoding of non-scalar values, consider using the `WTF-8` library.
• gotcha The `utf8.decode()` method will throw an `Error` when it detects malformed UTF-8 byte sequences in the input `byteString`. This strictness prevents silent data corruption that can occur with lenient decoders.
  Fix: Verify the source of the input `byteString` to ensure it is correctly encoded as UTF-8. Implement robust error handling using `try...catch` blocks around `utf8.decode()` calls to gracefully manage potentially malformed input, or pre-validate the input if possible.

Install

• npm install utf8 npm
• yarn add utf8 yarn
• pnpm add utf8 pnpm

Imports

• utf8
  wrong:

  import utf8 from 'utf8';

  correct:

  const utf8 = require('utf8');

  This package (v3.0.0) is primarily a CommonJS module. Direct ES module imports are not officially supported and will require a bundler or dynamic import for compatibility.
• global utf8 object

  <!-- In browser -->
  <script src="utf8.js"></script>
  <script>
    const encoded = utf8.encode('Hello');
  </script>

  When included directly in a browser via a <script> tag, the library exposes a global `utf8` object on the `window`.
• utf8.encode
  wrong:

  const { encode } = require('utf8'); // Not a named export

  correct:

  const utf8 = require('utf8');
  const encodedString = utf8.encode('Hello, world! 😊');

  The `encode` method is accessed as a property of the main `utf8` object.

Quickstart

This quickstart demonstrates how to encode and decode UTF-8 strings, including those with multi-byte Unicode characters, and how to check the library's version.

const utf8 = require('utf8');

// Example 1: Encoding a basic string
const originalString1 = 'Hello, world! 👋';
const encodedString1 = utf8.encode(originalString1);
console.log(`Original: '${originalString1}'`);
console.log(`Encoded:  '${encodedString1}'`);
const decodedString1 = utf8.decode(encodedString1);
console.log(`Decoded:  '${decodedString1}'`);

// Example 2: Encoding a string with a multi-byte Unicode character
// U+1F60A SMILING FACE WITH SMILING EYES
const originalString2 = 'Smiling face: \uD83D\uDE0A'; 
const encodedString2 = utf8.encode(originalString2);
console.log(`\nOriginal: '${originalString2}'`);
console.log(`Encoded:  '${encodedString2}'`);
const decodedString2 = utf8.decode(encodedString2);
console.log(`Decoded:  '${decodedString2}'`);

// Example 3: Demonstrate version access
console.log(`\nutf8.js version: ${utf8.version}`);