{"id":13158,"library":"evalite","title":"Evalite","description":"Evalite is a TypeScript-native evaluation runner designed for testing Large Language Model (LLM)-powered applications, leveraging the popular Vitest testing framework. Currently at version 0.19.0, the library maintains a rapid release cadence with frequent minor and patch updates, introducing new features and improvements. It differentiates itself by offering a local, API-key-free evaluation environment, making it suitable for development without external service dependencies. Key features include the ability to define evaluations with custom columns that access scores and traces, support for custom base paths for static UI exports, flexible configuration through `evalite.config.ts` for global settings like `testTimeout` and `maxConcurrency`, and a `trialCount` option for assessing variance in non-deterministic LLM outputs. It has also improved its testing pipeline by migrating to Vitest's Annotations API and removing serialization requirements for eval datasets, enhancing flexibility for complex input types.","status":"active","version":"0.19.0","language":"javascript","source_language":"en","source_url":"https://github.com/mattpocock/evalite","tags":["javascript","ai","evals","typescript","vitest"],"install":[{"cmd":"npm install evalite","lang":"bash","label":"npm"},{"cmd":"yarn add evalite","lang":"bash","label":"yarn"},{"cmd":"pnpm add evalite","lang":"bash","label":"pnpm"}],"dependencies":[{"reason":"Evalite is built on top of Vitest and requires it as a peer dependency for running evaluations.","package":"vitest","optional":false},{"reason":"Often used for scoring LLM outputs, as shown in quickstart examples. Not strictly required, but common.","package":"autoevals","optional":true}],"imports":[{"note":"Evalite is primarily designed for ESM usage with TypeScript. CommonJS require() may lead to `TypeError: evalite is not a function` in certain setups.","wrong":"const evalite = require('evalite');","symbol":"evalite","correct":"import { evalite } from 'evalite';"},{"note":"The `defineConfig` utility for `evalite.config.ts` is imported from a specific subpath to avoid bundling issues and clearly separate configuration from runtime eval definition.","wrong":"import { defineConfig } from 'evalite';","symbol":"defineConfig","correct":"import { defineConfig } from 'evalite/config';"},{"note":"Evalite provides its own CLI for running evaluations (`evalite run`, `evalite watch`), which orchestrates Vitest internally. Direct `vitest` commands typically won't trigger Evalite's specific evaluation logic or UI.","wrong":"vitest run --evalite","symbol":"CLI","correct":"evalite run"}],"quickstart":{"code":"/* package.json excerpt */\n/*\n{\n  \"name\": \"my-llm-app\",\n  \"version\": \"0.1.0\",\n  \"description\": \"My LLM powered application\",\n  \"type\": \"module\",\n  \"devDependencies\": {\n    \"evalite\": \"^0.19.0\",\n    \"vitest\": \"^4.0.0\" /* Ensure compatible Vitest version */\n  },\n  \"scripts\": {\n    \"eval\": \"evalite run\",\n    \"eval:watch\": \"evalite watch\"\n  }\n}\n*/\n\n// evalite.config.ts (create this file at your project root)\nimport { defineConfig } from 'evalite/config';\n\nexport default defineConfig({\n  testTimeout: 60000, // LLM calls can be slow, increase timeout if needed\n  maxConcurrency: 5,\n  // Optional: Specify files to run before tests, e.g., for environment variables\n  // setupFiles: ['./evalite.setup.ts'],\n  columns: {\n    'Input Length': ({ input }) => input.query.length,\n    'LLM Output Starts With': ({ output }) => output?.startsWith('Processed:'),\n  },\n});\n\n// my-llm-app.eval.ts (your evaluation file)\nimport { evalite } from 'evalite';\nimport { describe, expect } from 'vitest'; // Vitest utilities are still available\n\n// A dummy LLM function to simulate an actual LLM call\nconst mockLLM = async (input: string): Promise<string> => {\n  // Simulate network delay and LLM processing\n  await new Promise(resolve => setTimeout(resolve, Math.random() * 200 + 100));\n  return `Processed: ${input.toUpperCase()}`;\n};\n\ninterface MyEvalInput {\n  query: string;\n  expectedOutput: string;\n}\n\nevalite<MyEvalInput>('My LLM Application Evaluation Suite', ({ test, evalSuite }) => {\n  // Define the dataset for your evaluations\n  evalSuite.data([\n    { query: 'hello world', expectedOutput: 'PROCESSED: HELLO WORLD' },\n    { query: 'how are you', expectedOutput: 'PROCESSED: HOW ARE YOU' },\n    { query: 'evalite rocks', expectedOutput: 'PROCESSED: EVALITE ROCKS' },\n    { query: 'quick brown fox', expectedOutput: 'PROCESSED: QUICK BROWN FOX' },\n  ]);\n\n  // Define a test for each data point\n  test('LLM output should match expected format and content', async ({ input, expect: evalExpect }) => {\n    const llmResponse = await mockLLM(input.query);\n    evalExpect(llmResponse).toBe(input.expectedOutput);\n    return { output: llmResponse }; // The output will be available to scorers and columns\n  });\n\n  // Optional: Define custom scorers to evaluate LLM outputs quantitatively\n  evalSuite.scorer('ExactMatchScorer', ({ output, expected }) => {\n    const score = output === expected ? 1 : 0;\n    return { score, verdict: score === 1 ? 'Perfect Match' : 'Mismatch' };\n  });\n\n  // Optional: Add custom columns to display scorer results or other data in the UI\n  evalSuite.column('Match Score', ({ scores }) => scores.ExactMatchScorer?.score);\n});\n\n// To run: \n// 1. Install dependencies: npm install evalite vitest\n// 2. Run the evals: npm run eval \n//    (or npm run eval:watch for watch mode and UI at http://localhost:3006)","lang":"typescript","description":"This quickstart demonstrates how to set up `evalite.config.ts`, define an evaluation suite with input data, integrate a mock LLM call, assert expected outputs using Vitest's `expect`, and utilize custom scorers and columns for detailed results. It covers the core `evalite` function and its integration into a project to test LLM-powered applications."},"warnings":[{"fix":"Upgrade Vitest to version 4.x or later: `npm install vitest@latest`.","message":"Evalite upgraded to Vitest v4 in version 0.17.0. Projects using older Vitest versions might encounter compatibility issues or require an upgrade of their Vitest dependency.","severity":"breaking","affected_versions":">=0.17.0"},{"fix":"Ensure your project's Vitest dependency is `3.2.4` or newer: `npm install vitest@latest`.","message":"Version 0.15.0 migrated to Vitest's Annotations API, requiring Vitest 3.2.4 or later. Earlier Vitest versions will cause runtime errors.","severity":"breaking","affected_versions":">=0.15.0"},{"fix":"Create an `evalite.config.ts` file at your project root and use `defineConfig` from `evalite/config` to set global options.","message":"Introduced `evalite.config.ts` in version 0.16.0 for centralized configuration. If upgrading from an earlier version, global options (like `testTimeout`, `maxConcurrency`) previously configured via CLI arguments might need to be migrated to this new configuration file.","severity":"gotcha","affected_versions":">=0.16.0"},{"fix":"Use `evalite export --basePath=/your-path` to ensure correct asset resolution for static UI deployments.","message":"When exporting static UI for hosting on non-root URLs (e.g., S3/CloudFront subpaths), the `evalite export` command now requires a `--basePath` option (since 0.19.0) to correctly resolve assets.","severity":"gotcha","affected_versions":">=0.19.0"},{"fix":"If issues arise, delete `node_modules/.evalite` folder, update `evalite` to the latest version, and rerun evals. Report issues if they persist.","message":"Evalite is an experimental project and its author actively pushes breaking changes. Unexpected behavior might occur between minor versions.","severity":"gotcha","affected_versions":">=0.1.0"}],"env_vars":null,"last_verified":"2026-04-19T00:00:00.000Z","next_check":"2026-07-18T00:00:00.000Z","problems":[{"fix":"Upgrade your `vitest` dependency: `npm install vitest@latest`.","cause":"Your installed `vitest` package is older than the minimum version required by `evalite`.","error":"Error: evalite requires Vitest version X.Y.Z or higher."},{"fix":"Create `evalite.config.ts` in your project root with content like `import { defineConfig } from 'evalite/config'; export default defineConfig({});`.","cause":"After `evalite` v0.16.0, a configuration file (`evalite.config.ts`) is expected at the project root for global settings.","error":"Error: evalite.config.ts not found. Please create one..."},{"fix":"Ensure you are using `import { evalite } from 'evalite';` and that your project is configured for ESM (e.g., `\"type\": \"module\"` in `package.json`).","cause":"This usually indicates an incorrect import, often trying to use CommonJS `require()` syntax or an incorrect named import in an ESM context.","error":"TypeError: (0 , evalite_1.evalite) is not a function"},{"fix":"Try rebuilding `better-sqlite3`: `npm rebuild better-sqlite3` (or `pnpm rebuild better-sqlite3`). You might also need to approve the build: `pnpm approve-builds`.","cause":"This error is related to the `better-sqlite3` dependency that Evalite uses for local data storage, often seen with certain package managers or build environments.","error":"Command failed, Error: Could not locate the bindings file."}],"ecosystem":"npm","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null,"pypi_latest":null,"cli_name":"evalite"}