Evalite

Evalite is a TypeScript-native evaluation runner designed for testing Large Language Model (LLM)-powered applications, leveraging the popular Vitest testing framework. Currently at version 0.19.0, the library maintains a rapid release cadence with frequent minor and patch updates, introducing new features and improvements. It differentiates itself by offering a local, API-key-free evaluation environment, making it suitable for development without external service dependencies. Key features include the ability to define evaluations with custom columns that access scores and traces, support for custom base paths for static UI exports, flexible configuration through `evalite.config.ts` for global settings like `testTimeout` and `maxConcurrency`, and a `trialCount` option for assessing variance in non-deterministic LLM outputs. It has also improved its testing pipeline by migrating to Vitest's Annotations API and removing serialization requirements for eval datasets, enhancing flexibility for complex input types.