模型评测·评估

直接按这套走。先跑自动全量，再人工抽 20-50 条，别一上来全看。

1. 文件结构

你的项目/
├── config.yaml
├── data/
│   └── cases.jsonl
├── results/
│   ├── result.jsonl
│   ├── report.json
│   └── human_eval_template.csv
├── run_eval.py
└── requirements.txt

2. 依赖安装

pip install openai pyyaml

3. 配置文件 config.yaml

model:
  # 如果自己用 vLLM 部署，base_url 填 vLLM 的 OpenAI 兼容地址
  base_url: "http://localhost:8000/v1"
  api_key: "EMPTY"
  model_name: "Qwen-72B-Chat"
  temperature: 0
  max_tokens: 512
  timeout: 60

data:
  test_file: "data/cases.jsonl"
  output_dir: "results"
  # 0 表示全部跑完；如果只想先跑 50 条，写 50
  max_cases: 0

tasks:
  classification:
    labels: ["退货", "换货", "维修", "投诉", "咨询", "物流催单", "其他"]
    required_fields: ["category", "confidence", "reason"]
  reply_generation:
    required_fields: ["reply", "action", "category"]
    min_reply_len: 5
    max_reply_len: 200

eval:
  human_sample_size: 20

如果你用阿里云 DashScope，不用自己部署，改这几行：

model:
  base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
  api_key: "sk-你的key"
  model_name: "qwen2.5-72b-instruct"

4. 测试集格式 data/cases.jsonl

每行一个 JSON。expected 是你人工标注的标准答案，给自动打分用。

分类 Case

字段说明：

• task: classification
• input: 原始工单文本
• labels: 可选，这个 Case 的人工标签
• expected.category: 标准分类
• required_fields: 这条 Case 要求模型必须输出的字段

{"id":"cls_001","task":"classification","input":"用户：我买的手机屏幕有问题，用了三天就黑屏了，我要退货","labels":["退货"],"expected":{"category":"退货"},"required_fields":["category","confidence","reason"]}
{"id":"cls_002","task":"classification","input":"用户：快递显示签收但我没收到，你们怎么回事","labels":["物流催单"],"expected":{"category":"物流催单"},"required_fields":["category","confidence","reason"]}
{"id":"cls_003","task":"classification","input":"用户：刚收到货，打开发现外壳有划痕，能换吗","labels":["换货"],"expected":{"category":"换货"},"required_fields":["category","confidence","reason"]}

回复生成 Case

expected.must_contain：回复中必须覆盖的业务关键点。

expected.must_not_contain：红线词或不能对外承诺的话。

{"id":"gen_001","task":"reply_generation","input":{"category":"退货","user_message":"我买的手机屏幕有问题，用了三天就黑屏了，我要退货"},"expected":{"category":"退货","must_contain":["退货","寄回","检测"],"must_not_contain":["一定可以","保证赔偿"]},"required_fields":["reply","action","category"]}
{"id":"gen_002","task":"reply_generation","input":{"category":"物流催单","user_message":"快递显示签收但我没收到，你们怎么回事"},"expected":{"category":"物流催单","must_contain":["核实","物流","重新派送"],"must_not_contain":["一定今天到"]},"required_fields":["reply","action","category"]}
{"id":"gen_003","task":"reply_generation","input":{"category":"换货","user_message":"刚收到货，打开发现外壳有划痕，能换吗"},"expected":{"category":"换货","must_contain":["换货","凭证","寄回"],"must_not_contain":["赔偿现金"]},"required_fields":["reply","action","category"]}

5. Qwen-72B 部署/接入

不要用 base 版，要用 Chat/Instruct 版，否则指令遵循会很差。

自己部署推荐 vLLM

72B BF16 权重大约 145GB，建议 4 张 80G 或更好。显存不够就换 AWQ/GPTQ 量化版。

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server \
  --model /export/models/Qwen-72B-Chat \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --port 8000

启动后，config.yaml 里的 base_url 就是：

base_url: "http://localhost:8000/v1"
api_key: "EMPTY"
model_name: "Qwen-72B-Chat"

6. 自动评测脚本 run_eval.py

下面脚本直接复制使用。它做三件事：

1. 读取 cases.jsonl
2. 逐条调 Qwen-72B
3. 自动打分并生成报告 + 人工抽检 CSV

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Qwen-72B 售后工单分类/回复生成评测脚本
用法：
    python run_eval.py --config config.yaml
    python r