Conversion GemsConversion Gems
Promptfoo logo
Reviewed · Updated 2026-06-16

Promptfoo

Testing platform for evaluating prompt performance and analyzing LLM outputs.

Reviewed by the Conversion Gems editorial team ·
Try Promptfoo
Pricing
Paid
Best for
Developers
Category
AI Development
The bottom line

The leading free, open-source tool for LLM evals and red-teaming; powerful and local, with paid enterprise only for team governance and scale.

8.1
Our score
8.1 / 10
Conversion Gems editorial verdict
Free (MIT); Enterprise custom
Features9/10
9 - evals, red-teaming, 60+ providers, CI/CD, vulnerability scanning and local runs.
Value9/10
9 - free MIT and fully featured.
Ease of use6/10
6 - CLI and config-driven, developer-oriented.
Ecosystem9/10
9 - 60+ providers, used by OpenAI/Anthropic, with broad adoption.
Support6/10
6 - community support on OSS; paid enterprise support.
What it really is

An open-source CLI and library for evaluating and red-teaming LLM apps - testing prompts, models, RAG and agents, plus vulnerability scanning - not a $19/mo SaaS.

Our take

It is a developer-first standard for LLM testing: declarative configs, side-by-side model comparison, CI/CD integration and security red-teaming, running 100% locally so prompts never leave your machine (used by OpenAI and Anthropic). The free MIT tier is fully featured (including 10k red-team probes/month); only enterprise governance and unlimited probes are paid.

Best for
Developers benchmarking prompts, models and RAG
AI teams red-teaming apps for security and vulnerabilities
CI/CD pipelines gating LLM quality and safety
Not good for
Non-developers wanting a no-code UI
Enterprises needing SSO, dashboards and unlimited probes without paying
Teams wanting a fully managed, hosted-only service
Friction report
Time to value
Fast for developers: install the CLI, write a declarative config, and run evals locally or in CI within minutes.
Scale breakpoint
The free tier caps red-team probes at 10k/month, and team governance (SSO, dashboards, collaboration, unlimited probes) requires custom-priced Enterprise.
Walled garden
Low lock-in - MIT, model-agnostic and local-first; only enterprise features are proprietary.

Frequently Asked Questions

Alternatives

Step up

An enterprise eval/observability platform like Langfuse or Braintrust with managed dashboards.

Lighter alternative

A provider's built-in eval tooling if you only test one model.

Ready to try Promptfoo?
Opens the official site — we may earn a commission if you sign up.
Try Promptfoo

Tags

#LLMOps#LLMObservability#ModelEvaluation

Explore related categories

Conversion Gems independently reviews every tool. We may earn a commission if you sign up through our links — it never affects our verdict or ranking.