Artillery Expectation Plugin

2.24.0 · active · verified Wed Apr 22

Artillery-plugin-expect is an official plugin for Artillery, a powerful open-source load and functional testing tool built with Node.js. This plugin extends Artillery's capabilities by adding support for declarative checks and assertions on HTTP requests directly within YAML test scripts. It enables users to perform functional or acceptance testing alongside performance testing, verifying response status codes, content types, headers, body properties (via JSONPath or JMESPath), and regex matches. The current stable version is 2.24.0, and its development and release cadence are now tightly integrated with the main Artillery project since its codebase was moved into the core repository in December 2022. This plugin is a key differentiator for Artillery, allowing comprehensive API testing (both load and functional) from a single, YAML-driven test definition, making it suitable for CI/CD pipelines to run post-deployment smoke tests.

Common errors

```
Plugin 'expect' was not loaded: Error: Cannot find module 'artillery-plugin-expect'
```
cause This error typically occurs because the `artillery-plugin-expect` npm package is not installed or cannot be found by Artillery in the expected location.

fix Ensure the plugin is installed correctly. If Artillery is installed globally (`npm install -g artillery`), install the plugin globally too (`npm install -g artillery-plugin-expect`). If Artillery is a local project dependency, install the plugin as a local `devDependency` (`npm install --save-dev artillery-plugin-expect`). Also, check your YAML configuration for typos under `config.plugins`.
```
Failed expectation for request GET /api/data: statusCode=200, expected=201, actual=200
```
cause An assertion defined in your `expect` block did not match the actual response received from the target API.

fix Review the details of the failed expectation in the Artillery output. Compare the `expected` value/condition with the `actual` value. Check your API's expected behavior, the test data, and the `expect` definition in your YAML script for any discrepancies. Use the `pretty` or `prettyError` output options for the plugin for more detailed debugging information.
```
Cannot read properties of undefined (reading 'title') in '{{ body.title }}'
```
cause This error often indicates that a JSON property or variable path specified in an `equals` or `matchesRegexp` expectation (e.g., `{{ body.title }}`) does not exist in the HTTP response body, or the response body itself is not valid JSON.

fix Inspect the raw response body from your API to confirm the existence and exact path of the property you are trying to assert. Use `capture` with `json` or `jmespath` to debug what values are being extracted and ensure your JMESPath or JSONPath expressions are correct. Consider using `hasProperty` expectation first to check for existence before asserting on its value.

Warnings

gotcha The `artillery-plugin-expect` functionality is only compatible with the HTTP engine. It cannot be used with other engines like WebSocket or Socket.IO. Additionally, expectations cannot be used within `beforeRequest` or `afterResponse` JavaScript hooks.
Fix: Ensure your scenarios use the HTTP engine. If you need custom logic with non-HTTP engines or within hooks, consider using custom JavaScript functions as steps in your flow to perform assertions manually or log data for later analysis.
gotcha When installing Artillery and its plugins, ensure consistent installation methods. If Artillery is installed globally (e.g., `npm install -g artillery`), plugins like `artillery-plugin-expect` should also be installed globally (`npm install -g artillery-plugin-expect`). A mismatch (global Artillery, local plugin, or vice-versa) can lead to the plugin not being loaded or recognized.
Fix: Verify your `npm` or `yarn` installation strategy. If `artillery` is global, install plugins globally. If `artillery` is a local project dependency, install plugins as local `devDependencies`.
breaking The standalone `artillery-plugin-expect` GitHub repository (artilleryio/artillery-plugin-expect) was archived on December 11, 2022. The plugin's codebase and ongoing development have been moved into the main Artillery repository. While this doesn't directly break existing installations, it means that new issues or contributions for `artillery-plugin-expect` should be directed to the main Artillery project.
Fix: For reporting issues or contributing, refer to the main Artillery repository (artilleryio/artillery). Ensure you are using a recent version of Artillery that bundles this plugin's functionality or uses the latest compatible plugin package.
gotcha By default, a failed expectation will report an error but may not necessarily halt the entire test run immediately. For strict functional testing or CI/CD integration where failures should stop the pipeline, it's often necessary to combine `expect` with Artillery's `config.ensure` section.
Fix: To ensure tests fail decisively on expectation errors, add `ensure` conditions to your `config` section, such as `maxErrorRate: 0` or specific `p99` thresholds, which will cause Artillery to exit with a non-zero code if conditions are not met. The plugin can also report failures as errors in the report via configuration, or you can use the `prettyError` output option.

Install

npm install artillery-plugin-expect npm
yarn add artillery-plugin-expect yarn
pnpm add artillery-plugin-expect pnpm

Imports

Expectation configuration in YAML
```
config:
  plugins:
    expect: {}
```
This plugin is configuration-driven and does not expose JavaScript/TypeScript symbols for direct import into user code. It is enabled and configured declaratively within the `artillery.yml` test script. The configuration object for `expect` can be empty or contain specific options like `pretty`, `json`, `prettyError`, or `silent` for output control.
Adding expectations to a scenario step
```
scenarios:
  - name: My Scenario
    flow:
      - get:
          url: '/api/data'
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: 'items'
            - equals:
                - 'expectedValue'
                - '{{ jsonProp }}'
```
Expectations are defined directly under an `expect` key within an HTTP request step (e.g., `get`, `post`). There are various types of expectations available, such as `statusCode`, `contentType`, `hasProperty`, `equals`, `hasHeader`, `headerEquals`, `matchesRegexp`, and `jmespath`.

Quickstart

This quickstart demonstrates how to install `artillery-plugin-expect` globally and configure a basic Artillery YAML script. It shows how to enable the plugin and add various HTTP expectations (status code, content type, body properties, equality, regex matching) to different request steps within a scenario, including using captured variables in assertions. It also illustrates how to use environments to switch between functional and load testing.

npm install -g artillery artillery-plugin-expect

# Create a file named 'test-api.yml'
# The following YAML defines a simple functional test scenario
# using artillery-plugin-expect.
# It targets a placeholder API and asserts on the response.
---
config:
  target: 'https://jsonplaceholder.typicode.com'
  plugins:
    expect: {}
  environments:
    functional:
      phases:
        - duration: 1
          arrivalCount: 1
    load:
      phases:
        - duration: 10
          arrivalRate: 5

scenarios:
  - name: Get A Todo Item
    flow:
      - get:
          url: '/todos/1'
          expect:
            - statusCode: 200
            - contentType: json
            - hasProperty: 'id'
            - hasProperty: 'title'
            - equals:
                - 'delectus aut autem'
                - '{{ body.title }}'
            - matchesRegexp:
                - 'autem'
                - '{{ body.title }}'
      - get:
          url: '/posts/1'
          capture:
            - json: '$.title'
              as: postTitle
          expect:
            - statusCode: 200
            - contentType: json
            - equals:
                - 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit'
                - '{{ postTitle }}'

# Run the functional test:
# artillery run --environment functional test-api.yml

# Or run a load test (without plugin output):
# artillery run --environment load test-api.yml

view raw JSON →