在 DeepEval中数据集通常就是一系列的 goldens. golden包含llm测试case的所有前置条件。 所以 dataset 就是 golden 的管理器， 使用golden去完成测试用例编写。

初始化数据集

dataset = EvaluationDataset()

添加数据

一个Dataset 中不能同时包含 Golden和 ConversationalGolden

dataset.add_golden(Golden(...))
dataset.add_golden(ConversationalGolden(...))

存储数据集

Confident AI

dataset.push(alias="xxxx")

CSV,JSON,JSONL

注意默认文件名是当前时间字符串, 格式为 %Y%m%d_%H%M%S. 注意保存机制，避免文件名重复。当然，你可以通过 filename 自行指定

dataset.save_as(file_type="csv")
dataset.save_as(file_type="json")
dataset.save_as(file_type="jsonl")

加载数据

Confident AI

dataset.pull(alias="xxxx")

CSV,JSON,JSONL

dataset.add_goldens_from_csv_file(filepath="xxx")
dataset.add_goldens_from_json_file(filepath="xxx")
dataset.add_goldens_from_jsonl_file(filepath="xxx")

生成Goldens

# test.txt

An In-depth Introduction to DeepEval: The Open-Source LLM Evaluation Framework
DeepEval is a powerful, open-source evaluation framework specifically designed for systematically assessing, testing, and validating large language model (LLM) applications, emerging as a foundational tool for AI developers and teams building reliable, production-grade LLM systems. Drawing on cutting-edge academic research and industry best practices, it addresses the critical gap in modern AI development: unlike traditional software, LLM outputs are probabilistic, non-deterministic, and prone to inconsistencies such as hallucinations, irrelevant responses, and unfaithful reasoning, making conventional testing methods ineffective. DeepEval solves this challenge by providing a pytest-style, developer-centric evaluation ecosystem that brings structured, automated, and reproducible testing workflows to LLMs, RAG (Retrieval-Augmented Generation) pipelines, AI agents, chatbots, and custom prompt architectures.
At its core, DeepEval redefines LLM evaluation by popularizing the LLM-as-a-Judge paradigm, a state-of-the-art evaluation methodology that leverages advanced language models to autonomously score and assess the quality of target LLM outputs. Unlike rigid rule-based evaluation tools that rely solely on keyword matching or static metrics, DeepEval’s judge-driven evaluation delivers nuanced, human-like assessments of semantic quality, logical coherence, and contextual accuracy. A key advantage of this framework is its exceptional flexibility: developers are not locked into a single model provider. While it defaults to OpenAI models for evaluation, it seamlessly supports mainstream alternatives including Anthropic, Gemini, and local open-source models via Ollama, enabling fully private, offline evaluation workflows for sensitive enterprise use cases.
DeepEval features a comprehensive suite of research-backed evaluation metrics that cover the most critical pain points of LLM application development, making it a one-stop solution for holistic quality auditing. Its native metric library includes industry-standard benchmarks such as G-Eval, RAGAS, and specialized indicators for core LLM quality dimensions: answer relevancy to ensure responses align with user queries, factual faithfulness to verify outputs match retrieved context and real-world facts, hallucination detection to identify fabricated or unsubstantiated information, task completion rate to measure whether the LLM fulfills intended functional goals, and groundedness to validate alignment with source data. Beyond pre-built metrics, DeepEval supports fully customizable evaluation rubrics, allowing teams to tailor assessment criteria to unique business scenarios, domain-specific requirements, and custom AI agent behaviors.
One of DeepEval’s most distinctive strengths is its developer-friendly design and seamless integration with existing AI development stacks. Modeled after the widely adopted pytest testing framework, it adopts intuitive, lightweight syntax that requires minimal learning curve for software engineers and AI developers. It natively integrates with mainstream LLM development tools and frameworks including LangChain, LlamaIndex, and major cloud AI platforms, enabling end-to-end black-box evaluation of complete LLM applications rather than isolated model outputs. This compatibility allows teams to embed DeepEval directly into CI/CD pipelines, automating continuous evaluation throughout the model development, iteration, and deployment lifecycle. This shift from manual, sporadic quality checks to automated, continuous evaluation drastically reduces human error, accelerates iteration speed, and ensures consistent output quality across model updates and prompt revisions.
In industrial practice, DeepEval has established itself as a trusted, scalable evaluation solution with widespread industry adoption. It is currently used by over 150,000 developers and more than half of Fortune 500 companies, processing over 100 million daily evaluations globally. Its robust A/B testing and statistical analysis tools empower teams to quantitatively compare different model versions, prompt templates, retrieval strategies, and pipeline architectures, accurately identifying high-performance iterations and isolating defective components. Unlike observability-focused AI tools that prioritize post-deployment debugging, DeepEval specializes in pre-production benchmarking and proactive quality control, enabling teams to identify and resolve potential risks such as biased outputs, inaccurate reasoning, and poor contextual adaptation before models go live.
DeepEval also excels in balancing evaluation accuracy, privacy, and controllability. It supports local execution of evaluation models, allowing all assessment processes to run on private enterprise infrastructure without third-party data transmission, which is critical for industries with strict data compliance requirements such as finance, healthcare, and government. Its built-in statistical analyzers uncover deep, subtle performance flaws that manual reviews often miss, including inconsistent response quality across different user query types, marginal factual errors, and degraded reasoning ability in complex tasks. Additionally, its open-source nature ensures full transparency and extensibility: developers can modify core evaluation logic, add custom metrics, and adapt the framework to specialized use cases ranging from enterprise customer service chatbots and professional domain Q&A systems to autonomous AI agents and intelligent document processing tools.
In summary, DeepEval fills a vital void in the LLM engineering ecosystem by standardizing the evaluation of probabilistic AI systems. It transforms LLM development from a trial-and-error, experience-dependent process into a data-driven, standardized, and automated engineering workflow. By combining research-grade evaluation metrics, flexible model compatibility, seamless pipeline integration, and enterprise-level scalability, DeepEval has become an indispensable tool for building trustworthy, stable, and high-performance LLM applications, driving the maturity and industrialization of large language model technology.

注意你需要配置 OpenAI Key 或者自定义DeepEvalLLM 和 DeepEvalBaseEmbeddingModel传入才能工作

需要安装的额外依赖

• chromadb
• langchain
• langchain_community
• langchain_text_splitters

from deepeval.synthesizer import Synthesizer

# 默认使用openai处理提取工作
goldens = Synthesizer().generate_goldens_from_docs(document_paths=['test.txt'])
# 传入自定义的llm_model
goldens = Synthesizer(model=custom_llm_model).generate_goldens_from_docs(
	document_paths=['test.txt'], 
	context_construction_config=ContextConstructionConfig(
		embedder=CustomEmbeddingLLM(),
		critic_model=CustomLLM(),
	)
)
dataset = EvaluationDataset(goldens=goldens)