matchspec Documentation

Getting Started

• Overview What matchspec is and how it fits into your stack.
• Quick Start Run your first eval suite in five minutes.
• Installation Install the CLI and Go package.

Core Concepts

• Datasets Define evaluation inputs and expected outputs.
• Graders Score model outputs: exact match, regex, semantic, LLM-as-judge.
• Harnesses Wire datasets and graders into a runnable eval suite.
• Thresholds Set pass/fail thresholds and statistical confidence.

Reference

• CLI Reference All commands, flags, and exit codes.
• HTTP API Run evals and retrieve results over HTTP.
• Configuration matchspec.yml schema and defaults.
• Go API Programmatic access from Go code.

Guides

• CI/CD Integration Gate deployments in GitHub Actions and other CI systems.
• Custom Graders Write graders for domain-specific correctness definitions.