Promptfoo

Used by engineering teams at OpenAI, Anthropic, and hundreds of AI-forward companies, Promptfoo gives developers a systematic way to measure prompt quality, benchmark model performance across providers, and identify security vulnerabilities in AI applications before they reach production all through simple declarative configuration files that slot neatly into existing CI/CD pipelines.

The evaluation engine at Promptfoo's core enables developers to compare outputs across 50+ LLM providers simultaneously using the same test suite.

When choosing between models, comparing GPT-4o against Claude 3.5 Sonnet, or evaluating a fine-tuned model against its base counterpart, Promptfoo runs all candidates against the same inputs and metrics, producing side-by-side comparisons that make model selection and prompt optimization decisions data-driven rather than intuitive.

Promptfoo's red-teaming capabilities are among the most comprehensive available in the open-source ecosystem.

The built-in vulnerability scanner systematically attempts to jailbreak target models across more than 50 vulnerability categories from prompt injection and data leakage to harmful content generation and unauthorized capability unlocking.

Coverage maps to OWASP LLM Top 10 and NIST AI Risk Management Framework standards, giving security-conscious teams the audit trail they need for compliance and governance documentation.

For teams building agentic AI systems, Promptfoo includes specialized evaluation capabilities for testing LLM agents and RAG (Retrieval-Augmented Generation) systems.