Compare commits
7 commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 020409b30b | |||
|
4297f71e14 | ||
|
|
d4c33502e6 | ||
|
|
c91e7f8c20 | ||
|
|
cb6692c0c0 | ||
|
|
ae809a6006 | ||
|
|
18a5e38796 |
39 changed files with 11839 additions and 586 deletions
8
.gitignore
vendored
8
.gitignore
vendored
|
|
@ -1,6 +1,9 @@
|
|||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Cursor / IDE 本地历史
|
||||
.history/
|
||||
|
||||
# Python
|
||||
310py/
|
||||
.venv/
|
||||
|
|
@ -28,6 +31,11 @@ voc_clustering.sqlite
|
|||
# 流水线输出（含 word_freq.csv、报告 HTML 等）
|
||||
output/
|
||||
|
||||
# 样例报告 HTML（可本地重生成）
|
||||
bikini-trimmer-voc-v*.html
|
||||
voc_业务_2/bikini-trimmer-voc-report.html
|
||||
voc_业务_2/.echarts_cache.js
|
||||
|
||||
# 原始 / 中间 CSV（不纳入版本库）
|
||||
*.csv
|
||||
|
||||
|
|
|
|||
484
README.md
484
README.md
|
|
@ -1,257 +1,309 @@
|
|||
# VOC LLM 结构化分析 (VOC_LLM结构化)
|
||||
# 亚马逊评论 VOC 结构化分析
|
||||
|
||||
> 基于大语言模型（[DeepSeek](https://api.deepseek.com) OpenAI 兼容 API）与本地 MLX 向量的亚马逊 VOC（Voice of Customer）评论分析流水线：合并 CSV → 清洗 → LLM 结构化 → 向量化 → 聚类与词频 → 生成 HTML 分析报告。
|
||||
> 基于 DeepSeek Chat + 本地 MLX 向量的亚马逊站内评论分析流水线。
|
||||
> 仓库内存在 **两套报告方案**，共用同一套数据管道（步骤 1–6），仅在「报告生成」阶段分叉。
|
||||
|
||||
## 📖 目录
|
||||
**仓库**：
|
||||
- https://git.onesvm.com/1svm/amz_review_analyse_Hui（辉哥版本）
|
||||
- https://git.onesvm.com/whoops/amz_review_analyse
|
||||
|
||||
- [核心特性](#-核心特性)
|
||||
- [环境要求](#-环境要求)
|
||||
- [安装指南](#-安装指南)
|
||||
- [使用说明](#-使用说明)
|
||||
- [示例与输出](#-示例与输出)
|
||||
- [项目结构](#-项目结构)
|
||||
- [常见问题](#-常见问题)
|
||||
- [参与贡献](#-参与贡献)
|
||||
- [开源协议](#-开源协议)
|
||||
- [联系方式与鸣谢](#-联系方式与鸣谢)
|
||||
---
|
||||
|
||||
## ✨ 核心特性
|
||||
## 目录
|
||||
|
||||
- **七步全流程编排** — `main_voc分析.py` 一键串联：合并、清洗、结构化、向量化、聚类、词频、HTML 报告
|
||||
- **断点续跑** — 支持 `--from-step` / `--only-step`，从任意步骤恢复，调试时节省 API 成本
|
||||
- **LLM 结构化提取** — 从评论中抽取受众、痛点、方面、观点、情感等字段（`prompts/schema.yaml` 可配置）
|
||||
- **本地向量化** — Apple Silicon 上运行 `Qwen3-Embedding-4B-mxfp8`（MLX），无需云端 Embedding API
|
||||
- **语义聚类** — UMAP + HDBSCAN 多阶段聚类，辅以 LLM 评估簇质量自动调参
|
||||
- **词频分析** — LLM 归纳专有名词 + spaCy 全量词频统计，报告内嵌词云与六类归类
|
||||
- **可编辑 Prompt** — `prompts/` 目录下 Markdown / YAML 热加载，产品运营可直接改话术（见 `prompts/README.md`）
|
||||
- **并行加速** — 聚类与词频在步骤 5–6 由线程池并行执行；结构化批间并行（默认 8 路）
|
||||
- [新旧版本对比](#新旧版本对比)
|
||||
- [流程一：经典版（7 步）](#流程一经典版7-步)
|
||||
- [流程二：辉哥版本（6 步 + 业务报告）](#流程二辉哥版本6-步--业务报告)
|
||||
- [快速开始](#快速开始)
|
||||
- [结构化字段](#结构化字段)
|
||||
- [溯源与归因](#溯源与归因)
|
||||
- [环境要求与安装](#环境要求与安装)
|
||||
- [项目结构](#项目结构)
|
||||
- [常见问题](#常见问题)
|
||||
|
||||
## 🛠 环境要求
|
||||
---
|
||||
|
||||
| 依赖 | 说明 |
|
||||
|------|------|
|
||||
| Python | >= 3.10（推荐 3.12，项目内 `310py`） |
|
||||
| pip / uv | 安装 `requirements.txt` 中的包 |
|
||||
| spaCy 英文模型 | 经 `uv pip` 安装 `en-core-web-sm`（见安装指南，词频步骤必需） |
|
||||
| DeepSeek API Key | 结构化、聚类评估、词频、报告等 Chat 步骤 |
|
||||
| 本地 Embedding 模型 | 目录 `Qwen3-Embedding-4B-mxfp8/`（约 4GB，已 gitignore，需自行下载） |
|
||||
| Apple Silicon | 本地向量化依赖 MLX（M 系列芯片） |
|
||||
## 新旧版本对比
|
||||
|
||||
**Chat API Key**（任选其一，勿提交到 Git）：
|
||||
| 维度 | 经典版（`main` 分支） | 辉哥版本（`辉哥版本` 分支） |
|
||||
|------|----------------------|----------------------------|
|
||||
| **定位** | 通用 VOC 分析报告，偏「数据总览 + LLM 撰写洞察」 | 业务决策报告，偏「Persona / 主题 / 根因 / KANO + 源评论归因」 |
|
||||
| **入口命令** | `main_voc分析.py`（一步跑完 7 步） | `voc_业务_2/run_pipeline.py`（step 1–6 + 自动 `build_report.py`） |
|
||||
| **报告脚本** | `voc_report.py` | `voc_业务_2/build_report.py` |
|
||||
| **报告模板** | 程序内拼装 HTML | `voc_业务_2/template.html` |
|
||||
| **报告输出** | `output/{产品}/{产品}_voc_report.html` | `voc_业务_2/output/{slug}-voc-report.html` |
|
||||
| **结构化 schema** | 旧：含 `audience` 等 4 根字段 | 新：仅 `persona_signals` / `pain_points` / `product_feedback` |
|
||||
| **向量化实体** | 含 `audience` 向量（旧） | `persona_signal` / `pain_point` / `aspect_opinion`（无 audience） |
|
||||
| **聚类阶段** | `3a` 全量痛点 + `3b` 情感分桶 | 同左（共用 `聚类.py`） |
|
||||
| **报告核心模块** | Dashboard KPI、词云、词频六类、LLM 正文洞察、聚类附录 | Persona 卡片、差评/好评主题表、KANO 四象限、JTBD、人群×场景矩阵、根因分析 |
|
||||
| **源评论展示** | 附录「结构化/聚类效果验证」按 `source_row` 抽样 | Persona / 根因卡片内嵌 1–3 / 1–4 条真实评论 + ASIN 链接 |
|
||||
| **引用匹配** | 附录直接展示结构化 JSON | 命中池 `source_rows` → 结构化字段优先 → 原文 keyword 回退 |
|
||||
| **Prompt 配置** | `prompts/`（结构化 + 报告） | 数据层共用 `prompts/`；业务 LLM 见 `voc_业务_2/prompts.yaml` |
|
||||
| **适用场景** | 快速出一份带词频、聚类可视化的综合报告 | 产品定义、竞品对标、根因归因、需逐条溯源的业务汇报 |
|
||||
|
||||
1. 环境变量 `DEEPSEEK_API_KEY`
|
||||
2. 环境变量 `DEEPSEEK_API_KEY_FILE` 指向单行密钥文件
|
||||
3. 项目根目录 `.deepseek_key`（单行，无引号）
|
||||
**共用部分（两版相同）**：步骤 1–6 的脚本、三个 SQLite 库、`merged_reviews_cleaned.csv` 与 `source_row` 行号约定。
|
||||
|
||||
可选：`DEEPSEEK_MODEL`（默认 `deepseek-v4-pro`）、`DEEPSEEK_BASE_URL`（默认 `https://api.deepseek.com`）。
|
||||
**不可混用**：辉哥版本生成的结构化 JSON（无 `audience`）与旧库不兼容；换版本分析时请重跑 step 3（不要 `--keep-db`）。
|
||||
|
||||
## 📦 安装指南
|
||||
---
|
||||
|
||||
1. 克隆项目到本地：
|
||||
## 流程一：经典版（7 步）
|
||||
|
||||
```bash
|
||||
git clone https://git.onesvm.com/whoops/amz_review_analyse.git
|
||||
cd amz_review_analyse # 或你的本地目录名
|
||||
入口：`main_voc分析.py`
|
||||
分支：`main`
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S1[1 合并CSV] --> S2[2 清洗]
|
||||
S2 --> S3[3 结构化LLM]
|
||||
S3 --> S4[4 向量化MLX]
|
||||
S4 --> S5[5 聚类]
|
||||
S4 --> S6[6 词频]
|
||||
S5 --> S7[7 voc_report报告]
|
||||
S6 --> S7
|
||||
```
|
||||
|
||||
2. 创建虚拟环境并安装依赖（推荐）：
|
||||
| 步骤 | 脚本 | 输入 | 输出 | LLM |
|
||||
|:--:|------|------|------|:---:|
|
||||
| **1** | `合并评论数据.py` | 原始 CSV 目录 | `merged_reviews.csv` | — |
|
||||
| **2** | `content清洗.py` | 合并 CSV | `merged_reviews_cleaned.csv` | — |
|
||||
| **3** | `结构化_server.py` | 清洗 CSV | `voc_structured.sqlite` | ✓ |
|
||||
| **4** | `向量化.py` | 结构化库 + CSV | `voc_embeddings.sqlite` | — |
|
||||
| **5** | `聚类.py` | 向量库 | `voc_clustering.sqlite` | ✓ 调参 |
|
||||
| **6** | `词频.py` | 清洗 CSV | `output/word_freq.csv`、`voc_terms.json` | ✓ 术语 |
|
||||
| **7** | `voc_report.py` | 上述全部产物 | `output/{产品}/{产品}_voc_report.html` | ✓ 正文 |
|
||||
|
||||
```bash
|
||||
uv venv 310py --python 3.12
|
||||
uv pip install --python 310py/bin/python -r requirements.txt
|
||||
uv pip install --python 310py/bin/python \
|
||||
"en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"
|
||||
```
|
||||
|
||||
3. 配置 DeepSeek API Key：
|
||||
|
||||
```bash
|
||||
export DEEPSEEK_API_KEY="sk-xxx"
|
||||
# 或在项目根创建 .deepseek_key（已被 .gitignore 忽略）
|
||||
```
|
||||
|
||||
4. 准备本地 Embedding 模型（首次向量化前）：
|
||||
|
||||
将 `Qwen3-Embedding-4B-mxfp8` 放到项目根，或设置 `VOC_EMBED_MODEL_PATH` 指向模型目录。可从 [Hugging Face](https://huggingface.co/mlx-community/Qwen3-Embedding-4B-mxfp8) 下载。
|
||||
|
||||
5. 准备原始评论 CSV 目录（目录内所有 `*.csv` 表头须一致），例如亚马逊导出的 `*_realtime.csv`。
|
||||
|
||||
## 🚀 使用说明
|
||||
|
||||
### 全流程分析
|
||||
**一键运行**：
|
||||
|
||||
```bash
|
||||
./310py/bin/python main_voc分析.py \
|
||||
--input-dir "reviews_export" \
|
||||
--product "cat deterrent indoor" \
|
||||
--industry "Pet Supplies"
|
||||
--product "Bikini Trimmer" \
|
||||
--industry "个人护理"
|
||||
```
|
||||
|
||||
- `--input-dir`：原始 CSV 目录
|
||||
- `--product`：产品名（写入结构化任务与报告路径）
|
||||
- `--industry`：行业名，默认 `-`（可在步骤 3 写入库）
|
||||
- `--keep-db`：保留已有结构化 `voc_*.sqlite`，不覆盖删除
|
||||
|
||||
### 加速（批间并行，默认已开启）
|
||||
|
||||
步骤 3 结构化默认多批并行 Chat 请求；步骤 4 向量为本地 MLX 串行批处理（勿对同一模型多线程）：
|
||||
**断点续跑**：
|
||||
|
||||
```bash
|
||||
# 全流程
|
||||
./310py/bin/python main_voc分析.py --input-dir "reviews_export" --product "产品名"
|
||||
|
||||
# 调低结构化并发（遇 429 时）
|
||||
./310py/bin/python main_voc分析.py --input-dir "reviews_export" --product "产品名" \
|
||||
--struct-workers 4
|
||||
|
||||
# 环境变量：export VOC_STRUCT_WORKERS=8 VOC_EMBED_BATCH_SIZE=16
|
||||
./310py/bin/python main_voc分析.py --from-step 4 # 从向量化续跑
|
||||
./310py/bin/python main_voc分析.py --only-step 7 # 仅重生成 voc_report
|
||||
./310py/bin/python main_voc分析.py --from-step 4 --keep-db # 保留已有 SQLite
|
||||
```
|
||||
|
||||
### 断点续跑
|
||||
|
||||
```bash
|
||||
# 从向量化起续跑（步骤 4 起可省略 --product，自动读结构化库）
|
||||
./310py/bin/python main_voc分析.py --from-step 4 --keep-db
|
||||
|
||||
# 仅重跑词频（复用已有 voc_terms.json）
|
||||
./310py/bin/python main_voc分析.py --from-step 6 --skip-wordfreq-llm
|
||||
|
||||
# 仅重新生成 HTML 报告
|
||||
./310py/bin/python main_voc分析.py --only-step 7
|
||||
```
|
||||
|
||||
### 其他常用参数
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `--clean-intermediates` | 报告成功后删除中间 csv/sqlite，减少磁盘占用 |
|
||||
| `--filter-small-clusters` | 报告仅保留簇内评论占比 ≥ 10% 的簇 |
|
||||
| `--save-llm-raw` | 将报告 LLM 原文保存为 `report_llm_raw.txt`，调试时使用 |
|
||||
|
||||
### 程序式调用
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from main_voc分析 import run_voc_analysis
|
||||
|
||||
result = run_voc_analysis(
|
||||
input_dir=Path("reviews_export"),
|
||||
industry="Pet Supplies",
|
||||
product_name="cat deterrent indoor",
|
||||
from_step=1,
|
||||
clean_databases=True,
|
||||
)
|
||||
print(result["report_html"])
|
||||
```
|
||||
|
||||
### Prompt 验收（无需 API Key）
|
||||
|
||||
```bash
|
||||
./310py/bin/python prompts/smoke.py # 检查 prompt 能否加载
|
||||
./310py/bin/python prompts/smoke.py --live # 联调模型（需 DEEPSEEK_API_KEY）
|
||||
```
|
||||
|
||||
### 可选变体：jieba 词频
|
||||
|
||||
中文或需 jieba 分词时，可使用 `main_voc分析_jieba.py`（词频走 `词频_jieba.py`，其余步骤与主流程一致）。
|
||||
详细参数见 **[main_voc分析.md](main_voc分析.md)**。
|
||||
|
||||
---
|
||||
|
||||
更详细的步骤说明、算法与 SQLite 约定见 **[main_voc分析.md](main_voc分析.md)**。
|
||||
## 流程二：辉哥版本（6 步 + 业务报告）
|
||||
|
||||
## 📸 示例与输出
|
||||
入口：`voc_业务_2/run_pipeline.py`
|
||||
分支：`辉哥版本`
|
||||
|
||||
流程结束后，主要产物如下：
|
||||
数据管道与经典版 **步骤 1–6 完全相同**（内部调用 `main_voc分析.py --only-step N`），**跳过** 经典版 step 7，改为业务报告：
|
||||
|
||||
| 路径 | 说明 |
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S1[1 合并CSV] --> S2[2 清洗]
|
||||
S2 --> S3[3 结构化LLM]
|
||||
S3 --> S4[4 向量化MLX]
|
||||
S4 --> S5[5 聚类]
|
||||
S4 --> S6[6 词频]
|
||||
S5 --> R[build_report业务报告]
|
||||
S6 --> R
|
||||
```
|
||||
|
||||
### 阶段 A：数据管道（step 1–6）
|
||||
|
||||
| 步骤 | 说明 | 产物 |
|
||||
|:--:|------|------|
|
||||
| 1 | 多 CSV 合并 | `merged_reviews.csv` |
|
||||
| 2 | 去重、清洗 | `merged_reviews_cleaned.csv` |
|
||||
| 3 | LLM 结构化（三字段 schema） | `voc_structured.sqlite` |
|
||||
| 4 | 本地 MLX 向量化 | `voc_embeddings.sqlite` |
|
||||
| 5 | UMAP + HDBSCAN 聚类 | `voc_clustering.sqlite` |
|
||||
| 6 | spaCy 词频 + LLM 术语 | `output/word_freq.csv` |
|
||||
|
||||
`run_pipeline.py` 每步默认**清理旧 SQLite**（不加 `--keep-db`），避免与历史 job 混用。
|
||||
|
||||
### 阶段 B：业务报告（`build_report.py`）
|
||||
|
||||
在 SQLite 就绪后，按顺序执行（部分 LLM 任务并发）：
|
||||
|
||||
| 序号 | 模块 | 说明 |
|
||||
|:--:|------|------|
|
||||
| B1 | 加载数据 | 评论、聚类、结构化 extraction、竞品 ASIN 统计 |
|
||||
| B2 | Persona 发现 | LLM 绑定聚类簇 → 计算命中数/占比 → 选取源评论 |
|
||||
| B3 | 差评/好评主题 | LLM 归纳主题 + 结构化字段统计频次 |
|
||||
| B4 | KANO + JTBD + 情感词 | 三任务并发 LLM |
|
||||
| B5 | 人群×场景矩阵 | 依赖 KANO 结果 |
|
||||
| B6 | 根因分析 | 各 Persona 并发 LLM → 系统回填源评论 |
|
||||
| B7 | 渲染 HTML | 填充 `template.html` → 输出报告 |
|
||||
|
||||
**一键运行**：
|
||||
|
||||
```bash
|
||||
cd voc_业务_2
|
||||
# 编辑 config.yaml：input_dir 指向原始 CSV 目录
|
||||
../310py/bin/python run_pipeline.py --input-dir "../你的评论CSV目录"
|
||||
```
|
||||
|
||||
**仅重跑报告**（数据库已就绪）：
|
||||
|
||||
```bash
|
||||
cd voc_业务_2
|
||||
../310py/bin/python build_report.py --product "产品名" --industry "行业"
|
||||
```
|
||||
|
||||
**断点续跑数据管道**：
|
||||
|
||||
```bash
|
||||
../310py/bin/python run_pipeline.py --from-step 4 # 从向量化起
|
||||
../310py/bin/python run_pipeline.py --from-step 5 # 仅重跑聚类
|
||||
```
|
||||
|
||||
业务方法论见 **[VOC分析方法论与报告生成逻辑.md](VOC分析方法论与报告生成逻辑.md)**。
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
git clone https://git.onesvm.com/1svm/amz_review_analyse_Hui.git
|
||||
cd amz_review_analyse_Hui
|
||||
git checkout 辉哥版本 # 业务报告版
|
||||
# git checkout main # 经典 voc_report 版
|
||||
|
||||
uv venv 310py --python 3.12
|
||||
uv pip install --python 310py/bin/python -r requirements.txt
|
||||
export DEEPSEEK_API_KEY="sk-xxx"
|
||||
|
||||
# 辉哥版本（推荐业务使用）
|
||||
cd voc_业务_2 && ../310py/bin/python run_pipeline.py --input-dir "../评论CSV目录"
|
||||
|
||||
# 或经典版
|
||||
./310py/bin/python main_voc分析.py --input-dir "评论CSV目录" --product "产品名"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 结构化字段
|
||||
|
||||
辉哥版本 schema（`prompts/schema.yaml`）：
|
||||
|
||||
```json
|
||||
{
|
||||
"persona_signals": ["sensitive skin", "travel grooming"],
|
||||
"pain_points": ["ingrown hair"],
|
||||
"product_feedback": [
|
||||
{
|
||||
"aspect": "battery life",
|
||||
"opinion": "dies after one use",
|
||||
"sentiment": "Negative",
|
||||
"category": "Function"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
经典版旧 schema 曾含 `audience`（购买关系）；辉哥版本已移除，画像信息统一写入 `persona_signals`。
|
||||
|
||||
---
|
||||
|
||||
## 溯源与归因
|
||||
|
||||
全链路通过 **`source_row`**（与 `merged_reviews_cleaned.csv` 行号一致）关联：
|
||||
|
||||
```
|
||||
评论原文 (CSV)
|
||||
↓ source_row
|
||||
comment_extractions (voc_structured.sqlite)
|
||||
↓
|
||||
embedding_items → cluster_assignments
|
||||
↓
|
||||
Persona.source_rows（命中池）
|
||||
↓
|
||||
build_report.py
|
||||
├── Persona 卡片：结构化 persona_signals/pain_points 优先匹配
|
||||
├── 根因：Negative aspect+opinion 优先匹配
|
||||
└── 附录（仅 voc_report）：结构化 JSON 抽样
|
||||
```
|
||||
|
||||
`voc_业务_2/config.yaml`：
|
||||
|
||||
```yaml
|
||||
persona_quote_max: 3 # Persona 卡片最多展示条数
|
||||
rootcause_quote_max: 4 # 每条根因最多展示条数
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 环境要求与安装
|
||||
|
||||
| 依赖 | 说明 |
|
||||
|------|------|
|
||||
| `merged_reviews.csv` | 多文件合并结果 |
|
||||
| `merged_reviews_cleaned.csv` | 清洗、去重后的评论 |
|
||||
| `voc_structured.sqlite` | LLM 结构化结果 |
|
||||
| `voc_embeddings.sqlite` | 本地 Qwen3 向量（维度见库内 `dimensions` 字段） |
|
||||
| `voc_clustering.sqlite` | 多阶段聚类标签 |
|
||||
| `output/voc_terms.json` | 专有名词 / 停用词 |
|
||||
| `output/word_freq.csv` | 全量词频表 |
|
||||
| `output/{product}/{product}_voc_report.html` | **最终 VOC 分析报告**（词云、词频、分簇、AI 正文） |
|
||||
| Python | ≥ 3.10（推荐 3.12） |
|
||||
| DeepSeek API Key | 结构化、聚类调参、词频、报告 LLM |
|
||||
| `Qwen3-Embedding-4B-mxfp8/` | 本地 MLX 向量（约 4GB，需自行下载） |
|
||||
| Apple Silicon | 向量化依赖 MLX |
|
||||
| spaCy `en_core_web_sm` | 词频步骤 |
|
||||
|
||||
stdout 会打印 JSON 摘要（含 `report_html` 等键）。
|
||||
```bash
|
||||
uv pip install --python 310py/bin/python \
|
||||
"en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"
|
||||
```
|
||||
|
||||
## 📂 项目结构
|
||||
API Key：`DEEPSEEK_API_KEY` 环境变量，或项目根 `.deepseek_key`（勿提交 Git）。
|
||||
|
||||
---
|
||||
|
||||
## 项目结构
|
||||
|
||||
```text
|
||||
VOC_LLM结构化/
|
||||
├── main_voc分析.py # 主流程编排入口（七步）
|
||||
├── main_voc分析_jieba.py # 词频使用 jieba 的变体入口
|
||||
├── main_voc分析.md # 流程与算法详细说明
|
||||
├── voc_llm.py # DeepSeek Chat 密钥与客户端
|
||||
├── local_embedding.py # 本地 MLX Qwen3 向量化
|
||||
├── 合并评论数据.py # 步骤 1：多 CSV 合并
|
||||
├── content清洗.py # 步骤 2：评论清洗与去重
|
||||
├── 结构化_server.py # 步骤 3：LLM 结构化入库
|
||||
├── 结构化_Prompt.py # 结构化 prompt 组装
|
||||
├── 向量化.py # 步骤 4：本地 Embedding 入库
|
||||
├── 聚类.py # 步骤 5：UMAP + HDBSCAN
|
||||
├── 词频.py / 词频_jieba.py # 步骤 6：术语提取 + 词频
|
||||
├── voc_report.py # 步骤 7：HTML 报告生成
|
||||
├── prompts/ # 可编辑 prompt、schema、配置
|
||||
├── Qwen3-Embedding-4B-mxfp8/ # 本地模型（gitignore，需自行放置）
|
||||
├── requirements.txt
|
||||
├── output/ # 报告与词频输出（gitignore）
|
||||
└── README.md # 本文件
|
||||
├── main_voc分析.py # 经典版：七步编排入口
|
||||
├── voc_report.py # 经典版：step 7 报告
|
||||
├── 结构化_server.py / 向量化.py / 聚类.py / 词频.py
|
||||
├── prompts/ # 结构化 Prompt（两版共用）
|
||||
├── voc_业务_2/ # 辉哥版本业务报告
|
||||
│ ├── run_pipeline.py # step 1–6 + 自动 build_report
|
||||
│ ├── build_report.py # 业务 HTML 报告
|
||||
│ ├── report_utils.py # 统计、溯源引用、渲染
|
||||
│ ├── config.yaml / prompts.yaml / template.html
|
||||
│ └── output/ # 业务报告输出
|
||||
└── output/ # 经典版报告 + 词频（gitignore）
|
||||
```
|
||||
|
||||
## ❓ 常见问题
|
||||
|
||||
**Q：提示缺少 `DEEPSEEK_API_KEY`？**
|
||||
A：按上文配置环境变量或 `.deepseek_key`，并确认密钥未提交到仓库。
|
||||
|
||||
**Q：只有 `.dashscope_key` 报错？**
|
||||
A：Chat 已切换为 DeepSeek，DashScope 密钥不能用于 `api.deepseek.com`，请改用 `.deepseek_key`。
|
||||
|
||||
**Q：步骤 4 向量化失败 / 找不到模型？**
|
||||
A：确认 `Qwen3-Embedding-4B-mxfp8/` 在项目根，或设置 `VOC_EMBED_MODEL_PATH`；需在 Apple Silicon + Python 3.10+ 环境。
|
||||
|
||||
**Q：步骤 4 报错找不到 `product`？**
|
||||
A：从步骤 1–3 完整跑过，或确保 `voc_structured.sqlite` 中已有最新 job；步骤 4 起可省略 `--product`。
|
||||
|
||||
**Q：词频步骤报 spaCy 模型缺失？**
|
||||
A：执行 `uv pip install --python 310py/bin/python "en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"`。
|
||||
|
||||
**Q：合并 CSV 失败？**
|
||||
A：确保 `--input-dir` 下所有 CSV 表头完全一致。
|
||||
|
||||
**Q：修改 LLM 话术后报告解析失败？**
|
||||
A：勿修改 `prompts/schema.yaml` 中 `report.markers` 四段标记名；改完运行 `./310py/bin/python prompts/smoke.py` 验收。
|
||||
|
||||
## 🤝 参与贡献
|
||||
|
||||
欢迎提交 Issue 与 Pull Request。建议流程：
|
||||
|
||||
1. Fork 本仓库
|
||||
2. 创建特性分支：`git checkout -b feature/your-feature`
|
||||
3. 提交更改：`git commit -m '简要说明变更'`
|
||||
4. 推送并发起 Pull Request
|
||||
|
||||
修改主流程或 CLI 时，请同步更新 `main_voc分析.md`；修改 `prompts/` 时请遵循 `prompts/README.md` 中的占位符与 schema 约定。
|
||||
|
||||
**安全提醒**：勿提交 `.deepseek_key`、`.dashscope_key`、`.env`、真实评论 CSV、`*.sqlite`、`output/` 及本地模型目录（见 `.gitignore`）。
|
||||
|
||||
## 📄 开源协议
|
||||
|
||||
本项目尚未在仓库中附带 `LICENSE` 文件。若为内部项目，请按组织规范使用；若计划开源，请补充协议文件（如 MIT）并更新本节链接。
|
||||
|
||||
## ✉️ 联系方式与鸣谢
|
||||
|
||||
- **详细技术文档**：[main_voc分析.md](main_voc分析.md)、[prompts/README.md](prompts/README.md)
|
||||
- **项目仓库**：https://git.onesvm.com/whoops/amz_review_analyse
|
||||
|
||||
### 鸣谢
|
||||
|
||||
- [DeepSeek API](https://api.deepseek.com) — Chat 结构化、聚类评估、词频与报告
|
||||
- [mlx-community/Qwen3-Embedding-4B-mxfp8](https://huggingface.co/mlx-community/Qwen3-Embedding-4B-mxfp8) — 本地向量化
|
||||
- [UMAP](https://umap-learn.readthedocs.io/)、[HDBSCAN](https://hdbscan.readthedocs.io/) — 聚类管线
|
||||
- [spaCy](https://spacy.io/) — 英文词频与 NLP
|
||||
|
||||
---
|
||||
|
||||
*README 与 `main_voc分析.py` 七步流程保持一致；深度说明请参阅 `main_voc分析.md`。*
|
||||
## 常见问题
|
||||
|
||||
**Q：两个版本可以共用同一份 SQLite 吗？**
|
||||
A：step 1–6 产物可共用，但结构化库须为**同一 schema**。从经典版（含 audience）切到辉哥版本时，必须重跑 step 3。
|
||||
|
||||
**Q：Persona 显示「命中 N 条」但没有源评论？**
|
||||
A：用最新代码重跑 `build_report.py`；命中池非空时会多层回退，优先结构化字段匹配。
|
||||
|
||||
**Q：选哪个版本？**
|
||||
A：需要 Persona、KANO、根因、源评论归因 → **辉哥版本**；需要词云、词频 Dashboard、LLM 长文洞察 → **经典版**。也可只跑 step 1–6，再分别生成两种报告。
|
||||
|
||||
---
|
||||
|
||||
## 分支与仓库
|
||||
|
||||
| 分支 / 仓库 | 说明 |
|
||||
|-------------|------|
|
||||
| `main` | 经典 7 步 + `voc_report.py` |
|
||||
| **`辉哥版本`** | 业务报告 + 三字段结构化 + 溯源归因 |
|
||||
| `1svm/amz_review_analyse_Hui` | 辉哥版本主仓库 |
|
||||
| `whoops/amz_review_analyse` | 同源备份 |
|
||||
|
||||
---
|
||||
|
||||
## 鸣谢
|
||||
|
||||
- [DeepSeek API](https://api.deepseek.com)
|
||||
- [mlx-community/Qwen3-Embedding-4B-mxfp8](https://huggingface.co/mlx-community/Qwen3-Embedding-4B-mxfp8)
|
||||
- [UMAP](https://umap-learn.readthedocs.io/)、[HDBSCAN](https://hdbscan.readthedocs.io/)
|
||||
|
||||
---
|
||||
|
||||
*算法与 SQLite 表结构详见 [main_voc分析.md](main_voc分析.md)。*
|
||||
|
|
|
|||
815
VOC分析方法论与报告生成逻辑.md
Normal file
815
VOC分析方法论与报告生成逻辑.md
Normal file
|
|
@ -0,0 +1,815 @@
|
|||
# VOC 数据清洗、分析与报告生成通用方法论
|
||||
|
||||
> 文档版本：v1.0 · 2026-06-11
|
||||
> 适用范围：亚马逊任意品类竞品 VOC 分析
|
||||
> 数据来源：卖家精灵 / Shulex 导出的实时评论 CSV
|
||||
> 报告输出：双层结构 HTML 报告（描述层 What + 分析层 Why）
|
||||
|
||||
---
|
||||
|
||||
## 目录
|
||||
|
||||
1. [原始数据结构](#1-原始数据结构)
|
||||
2. [数据清洗规则](#2-数据清洗规则)
|
||||
3. [描述层 What — 分析逻辑](#3-描述层-what--分析逻辑)
|
||||
- 3.1 有效评论统计
|
||||
- 3.2 用户画像（Persona）识别
|
||||
- 3.3 正负反馈主题提取
|
||||
- 3.4 情感关键词分析
|
||||
4. [分析层 Why — 分析逻辑](#4-分析层-why--分析逻辑)
|
||||
- 4.1 KANO 模型需求分类
|
||||
- 4.2 JTBD 动机框架
|
||||
- 4.3 人群 × 场景 × 需求矩阵
|
||||
- 4.4 痛点根因分析
|
||||
5. [报告生成逻辑](#5-报告生成逻辑)
|
||||
- 5.1 HTML 整体结构
|
||||
- 5.2 可视化组件
|
||||
- 5.3 使用场景字段写入规则（核心规则）
|
||||
6. [执行 SOP（逐步操作流程）](#6-执行-sop逐步操作流程)
|
||||
7. [关键阈值与判断规则速查表](#7-关键阈值与判断规则速查表)
|
||||
8. [新品类接入清单](#8-新品类接入清单)
|
||||
|
||||
---
|
||||
|
||||
## 1. 原始数据结构
|
||||
|
||||
### 文件命名规则
|
||||
|
||||
```
|
||||
{ASIN}_realtime.csv
|
||||
```
|
||||
|
||||
每个竞品 ASIN 对应一个独立文件，分析时批量读取同一目录下的全部文件。
|
||||
|
||||
### CSV 字段说明
|
||||
|
||||
| 字段名 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `asin` | string | 亚马逊标准识别号，文件可能带 BOM 头（`\ufeffasin`），读取时须用 `utf-8-sig` 编码 |
|
||||
| `rating` | float（字符串形式） | 评分，取值 `1.0` / `2.0` / `3.0` / `4.0` / `5.0`，须用 `float()` 转换，**不能用 `int()`** |
|
||||
| `title` | string | 评论标题 |
|
||||
| `content` | string | 评论正文（主要分析字段） |
|
||||
| `verified` | string | `"True"` / `"False"`，是否已验证购买 |
|
||||
| `vine` | string | `"True"` / `"False"`，是否为 Vine 评测 |
|
||||
| `review_date` | string | ISO 8601 格式，如 `2026-05-28T00:00:00+00:00` |
|
||||
|
||||
---
|
||||
|
||||
## 2. 数据清洗规则
|
||||
|
||||
### 2.1 有效评论筛选（不可更改的核心规则）
|
||||
|
||||
**只保留以下两类评论，其余全部排除：**
|
||||
|
||||
```python
|
||||
def is_valid(row):
|
||||
return (
|
||||
row.get('verified', '').strip().lower() == 'true'
|
||||
or
|
||||
row.get('vine', '').strip().lower() == 'true'
|
||||
)
|
||||
```
|
||||
|
||||
**排除理由**：未验证且非 Vine 的评论可能包含刷评、竞品恶意差评或未实际购买的猜测，会干扰真实用户体验数据。
|
||||
|
||||
### 2.2 差评 / 好评 / 中性评论定义
|
||||
|
||||
| 分类 | 星级 | 用途 |
|
||||
|---|---|---|
|
||||
| 好评（Positive） | ★★★★★ / ★★★★ | 提取正向主题、魅力型需求、用户满意点 |
|
||||
| 中性（Neutral） | ★★★ | 单独记录，不进入主题频次统计 |
|
||||
| 差评（Negative） | ★★ / ★ | 主要分析对象，提取痛点主题和根因 |
|
||||
|
||||
### 2.3 新买家 vs 复购买家区分规则
|
||||
|
||||
| 类型 | content 字段识别关键词 |
|
||||
|---|---|
|
||||
| **新买家** | `first time` / `just got` / `just bought` / `new to` |
|
||||
| **复购买家** | `reorder` / `bought again` / `second time` / `repurchase` / `keep buying` |
|
||||
|
||||
> 两类买家的差评重点通常不同：
|
||||
> - 新买家 → 效果不符预期、开箱即坏、使用门槛高
|
||||
> - 复购买家 → 某个功能在长期使用后失效、品质下降
|
||||
|
||||
---
|
||||
|
||||
## 3. 描述层 What — 分析逻辑
|
||||
|
||||
### 3.1 有效评论统计
|
||||
|
||||
对每个 ASIN 分别计算，再汇总全市场数据：
|
||||
|
||||
| 指标 | 计算公式 |
|
||||
|---|---|
|
||||
| 有效评论数 | 通过 `is_valid()` 筛选后的总行数 |
|
||||
| 加权平均评分 | `Σ(各ASIN均分 × 各ASIN有效评论数) / 全市场有效评论总数` |
|
||||
| 差评率 | `≤2星评论数 / 有效总数` |
|
||||
| 正评率 | `≥4星评论数 / 有效总数` |
|
||||
| 各星级分布 | 1–5 星各自数量及占比 |
|
||||
|
||||
**市场竞争状态判断**：
|
||||
|
||||
| 加权均分 | 判断 |
|
||||
|---|---|
|
||||
| < 3.5 | 市场存在严重系统性缺陷，是新品进入的明确窗口期 |
|
||||
| 3.5 – 4.0 | 市场有改进空间，部分功能存在普遍短板 |
|
||||
| > 4.0 | 市场整体较成熟，需通过差异化或细分切入 |
|
||||
|
||||
### 3.2 用户画像（Persona）识别
|
||||
|
||||
#### 识别方法
|
||||
|
||||
在评论 `title + content` 字段中搜索特征词，将评论人归入对应 Persona。一条评论可同时归入多个 Persona。
|
||||
|
||||
#### Persona 识别词的建立原则
|
||||
|
||||
1. **阅读全部差评（≤2星）**，找出用户描述自身处境的词汇（"I have... / I am... / As a..."）
|
||||
2. **阅读全部好评（≥4星）**，找出用户描述自身需求背景的词汇
|
||||
3. 从中归纳出 4–6 个差异化的用户群体
|
||||
4. 每个群体设定 5–10 个识别关键词
|
||||
|
||||
#### ⚠️ Persona 必须覆盖的三个分类维度（缺一不可）
|
||||
|
||||
在最终确认 Persona 列表前，必须检查是否已从以下三个维度进行了覆盖，**不能只按其中一个维度拆分就停止**：
|
||||
|
||||
| 维度 | 说明 | 典型信号词 |
|
||||
|---|---|---|
|
||||
| **A. 物理/生理特征** | 用户身体特征决定了产品对他们的效果上限（最易被遗漏） | `thick/dark/coarse hair` / `sensitive skin` / `pregnant` / `curly` / `Latina` / `Type 4 hair` |
|
||||
| **B. 行为/场景** | 用户在什么情境下使用产品 | `travel` / `in the shower` / `gift` / `daily` |
|
||||
| **C. 购买动机/背景** | 用户为何从其他方案切换过来 | `switched from razor` / `too expensive` / `saw on TikTok` / `first time` |
|
||||
|
||||
> **关键原则**：如果你的 Persona 列表里只有场景类和动机类群体，而没有任何一个群体是按身体特征定义的，说明维度 A 被遗漏了，必须重新检查差评中的自我标注词汇。
|
||||
|
||||
#### 自我标注信号强制检查步骤
|
||||
|
||||
在完成初步 Persona 归纳后，**必须**额外执行以下搜索，确认是否有被遗漏的物理特征用户群：
|
||||
|
||||
```
|
||||
搜索差评中所有含以下模式的句子：
|
||||
"I have [adj] [noun]"（如 I have thick hair / I have sensitive skin）
|
||||
"My [noun] is/are [adj]"（如 My skin is super sensitive）
|
||||
"As a [noun/adj person]"（如 As a Latina / As a curly-haired person）
|
||||
"[族裔/肤色/发质形容词]"（如 Latina / dark hair / coarse / Type 4）
|
||||
|
||||
若上述词汇出现 ≥ 5 条，则该物理特征代表一个独立 Persona，必须单独列出。
|
||||
```
|
||||
|
||||
#### Persona 识别词模板格式
|
||||
|
||||
```python
|
||||
PERSONA_KEYWORDS = {
|
||||
'{群体名称A}': ['{关键词1}', '{关键词2}', ...],
|
||||
'{群体名称B}': ['{关键词1}', '{关键词2}', ...],
|
||||
# 根据实际品类补充
|
||||
}
|
||||
```
|
||||
|
||||
#### Persona 占比估算规则
|
||||
|
||||
```
|
||||
占比 = 命中该Persona识别词的评论数 / 有效评论总数
|
||||
四舍五入至整5%
|
||||
```
|
||||
|
||||
> 因一条评论可被多个 Persona 命中，各 Persona 占比之和可超过 100%。
|
||||
|
||||
#### Persona 卡片内容规格（每个群体输出以下信息）
|
||||
|
||||
| 字段 | 来源 | 说明 |
|
||||
|---|---|---|
|
||||
| 群体名称 | 自定义 | 简洁描述身份特征，≤6 字 |
|
||||
| 占比 | 统计计算 | 见上方公式 |
|
||||
| 核心痛点 | 该群体差评 | ≤3 条，原文语义概括 |
|
||||
| 核心需求 | 该群体好评+诉求 | ≤3 条 |
|
||||
| 购买动机 | JTBD 分析 | 用"雇佣产品做什么"句式 |
|
||||
| 代表性引用 | 真实评论原文 | 必须来自实际评论，注明 ASIN |
|
||||
|
||||
### 3.3 正负反馈主题提取
|
||||
|
||||
#### 主题识别关键词组的建立方法
|
||||
|
||||
1. 阅读**全部差评（≤2星）**，记录用户描述问题时的高频词
|
||||
2. 将语义相近的词归为同一主题，形成关键词组
|
||||
3. 每个主题设定 5–10 个关键词
|
||||
4. 覆盖 80%+ 的差评内容（长尾主题可合并为"其他"）
|
||||
|
||||
#### ⚠️ 主题拆分规则：相近但机制不同的问题必须独立成主题
|
||||
|
||||
语义相近不等于根因相同。以下情况**必须拆分为独立主题，不得合并**：
|
||||
|
||||
| 合并后失真的典型例子 | 应该如何拆分 | 原因 |
|
||||
|---|---|---|
|
||||
| "剃效差"（笼统） | ① 留茬/剃不干净 ② 拉扯/扯毛而非切断 | 机制不同：留茬=刀头贴肤不足；拉扯=刀片咬不断粗硬毛，影响人群完全不同 |
|
||||
| "皮肤问题"（笼统） | ① 割伤/出血 ② 摩擦热/灼烧感 ③ 剃须疹/内生毛 | 根因不同，对应不同的工程解决方案 |
|
||||
| "产品损坏"（笼统） | ① 充电失效 ② 配件断裂/脱落 | 分属电气系统和结构系统，受影响时间节点不同（充电=使用初期；断裂=一段时间后） |
|
||||
|
||||
> **判断是否需要拆分的问题**：
|
||||
> "同一主题下的差评，是否描述的是同一个物理/工程原因？"
|
||||
> 如果不是，必须拆开。
|
||||
|
||||
**通用差评主题模板格式**：
|
||||
|
||||
```python
|
||||
NEGATIVE_THEMES = {
|
||||
'{主题名称}': ['{关键词1}', '{关键词2}', ...],
|
||||
# 品类相关主题
|
||||
}
|
||||
```
|
||||
|
||||
**通用好评主题模板格式**：
|
||||
|
||||
```python
|
||||
POSITIVE_THEMES = {
|
||||
'{主题名称}': ['{关键词1}', '{关键词2}', ...],
|
||||
}
|
||||
```
|
||||
|
||||
#### 频次统计规则
|
||||
|
||||
1. 在 `title + content` 中搜索关键词
|
||||
2. 同一评论中同一关键词出现多次，仍计为 1 次（避免重复计数）
|
||||
3. 差评主题只统计 ≤2 星评论；好评主题只统计 ≥4 星评论
|
||||
4. **频次 = 命中该主题的评论条数**（非词语出现总次数）
|
||||
|
||||
#### 主题优先级判定规则
|
||||
|
||||
| 优先级 | 差评频次门槛 | 涉及竞品范围 |
|
||||
|---|---|---|
|
||||
| **P0（立即处理）** | ≥ 总有效差评数的 20% | 80%+ 竞品均出现 |
|
||||
| **P1（短期处理）** | 总有效差评数的 10–20% | 60%+ 竞品出现 |
|
||||
| **P2（中期关注）** | 总有效差评数的 3–10% | 40%+ 竞品出现 |
|
||||
|
||||
> **频次门槛的动态计算**：
|
||||
> `P0 绝对门槛 = 全市场有效差评数 × 20%`
|
||||
> 例：1123 条有效评论，差评率 35% ≈ 393 条差评，P0 门槛 ≈ 79 条
|
||||
|
||||
### 3.4 情感关键词分析
|
||||
|
||||
对所有有效评论进行词频统计，提取高频情感词。
|
||||
|
||||
**输出字段规格**：
|
||||
|
||||
| 字段 | 说明 | 规则 |
|
||||
|---|---|---|
|
||||
| 词汇 | 英文原词或词组 | 保留原文，不翻译 |
|
||||
| 出现频次 | 在有效评论中出现的条数 | 同一评论多次出现计1次 |
|
||||
| 情感极性 | 正面 / 负面 / 中性 | 根据语境判断，同一词在不同语境可有不同极性 |
|
||||
| 含义/使用场景 | 该词汇在评论中的具体语境 | **只写评论中明确出现的内容，不推断** |
|
||||
| 主要关联人群 | 对应的 Persona 名称 | 可多个 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 分析层 Why — 分析逻辑
|
||||
|
||||
### 4.0 Persona 完整性验证（进入分析层前的强制关卡）
|
||||
|
||||
**在开始 KANO / JTBD 分析之前，必须完成以下交叉验证，发现遗漏立即返回 3.2 节补充。**
|
||||
|
||||
#### 验证方法：每个 P0/P1 主题 → 强制归因到 Persona
|
||||
|
||||
为每一个 P0/P1 差评主题填写下表：
|
||||
|
||||
| 差评主题 | 频次 | 该主题的典型描述 | 主要影响哪类用户？ | 对应已有 Persona？ |
|
||||
|---|---|---|---|---|
|
||||
| {主题1} | {N条} | {原文特征} | {用户特征描述} | {Persona名 / ❌未覆盖} |
|
||||
| {主题2} | ... | ... | ... | ... |
|
||||
|
||||
**如果某个 P0/P1 主题在"对应已有 Persona"列填写了 ❌，说明存在遗漏的用户群体，必须新增 Persona。**
|
||||
|
||||
#### 常见漏洞场景
|
||||
|
||||
| 被遗漏的情况 | 漏洞原因 | 补救方式 |
|
||||
|---|---|---|
|
||||
| 身体特征群体（如粗硬发质用户） | 只按场景/动机分群，未检查维度 A（物理特征） | 返回 3.2 节执行自我标注信号强制检查 |
|
||||
| 长期使用复购用户 | 只看差评内容，未注意时间轴（"after months of use"） | 检查含 `months` / `after a while` / `second bottle` 的差评是否形成独立群体 |
|
||||
| 特定人群的特殊需求 | 该群体占比较小但痛点极具体 | 即使占比低（~5%），若痛点独特且无法被其他 Persona 代表，必须单独列出 |
|
||||
|
||||
### 4.1 KANO 模型需求分类
|
||||
|
||||
#### 四种类型定义与判断标准
|
||||
|
||||
| 类型 | 定义 | 判断标准 | 常见错误 |
|
||||
|---|---|---|---|
|
||||
| **基本型（Must-be）** | 不满足→强烈差评；满足→用户不会特别提及或表扬 | ① 差评频次达 P0 级别 ② 80%+ 竞品均出现该缺陷 ③ 好评中几乎不出现"因为做到了 X 所以好评" | 把"剃净度"归为基本型——剃净度好坏都会被用户提及，属期望型 |
|
||||
| **期望型（Performance）** | 做得越好评分越高，做得越差评分越低，线性关系 | ① 好评中被作为"这款优于竞品"的主要理由 ② 差评中作为"原本期待但未达到"的失望点 ③ 用户用程度词描述（`better/worse/not as good as`） | 把"电池续航"归为基本型——续航差才差评，续航超长会被用户特别称赞 |
|
||||
| **魅力型（Attractive）** | 满足→产生超预期惊喜和好评；不满足→用户不会差评 | ① 好评中出现强情感词 `love` / `obsessed` / `amazing` / `didn't expect` / `bonus` ② 该功能在差评中几乎不出现 ③ 竞品普遍缺失，属市场空白 | 把"附赠收纳袋"归为期望型——用户从未因为"没有收纳袋"而差评，属意外惊喜 |
|
||||
| **反向型（Reverse）** | 某些用户认为该功能是负担，反而差评 | ① 差评中出现对某个"功能"的明确抱怨 ② 该内容在好评中也受另一部分人喜爱（说明用户分歧） | 把"产品损坏"归为反向型——没有用户"希望产品能损坏" |
|
||||
|
||||
#### 各类型的输出格式要求
|
||||
|
||||
每个 KANO 条目必须包含以下 5 个字段，缺一不可：
|
||||
|
||||
```
|
||||
需求项：[具体需求描述，动词+名词形式]
|
||||
评论频次证据：[支撑该分类的评论条数及代表性原文片段]
|
||||
主要影响 Persona：[哪类用户群对该需求最敏感]
|
||||
分类原因：[用一句话解释为什么是这个 KANO 类型，而不是其他类型]
|
||||
竞品现状：[现有竞品是否满足，满足程度如何]
|
||||
```
|
||||
|
||||
**示例（基本型）**：
|
||||
```
|
||||
需求项：充电后可正常启动
|
||||
评论频次证据：118条差评（P0级别），"stopped working after a few uses" / "won't charge at all"
|
||||
主要影响 Persona：所有群体，尤其是复购用户（第二台也坏后彻底失去信任）
|
||||
分类原因：充电失效是"有就正常、坏了就1星"的底线需求，好评中没有人因"能充电"而特别表扬
|
||||
竞品现状：全部5款竞品均有此问题，说明是行业普遍工程缺陷
|
||||
```
|
||||
|
||||
**示例（魅力型）**：
|
||||
```
|
||||
需求项：LCD 电量显示
|
||||
评论频次证据：好评中 28条提及，"love that I can see the battery level" / "so convenient"，差评中0条因缺少LCD而差评
|
||||
主要影响 Persona：旅行护理族（出行前确认电量）/ 所有群体
|
||||
分类原因：用户不会因为"没有电量显示"而差评，但有了之后会主动提及并作为推荐理由
|
||||
竞品现状：仅1款（FANKRUAI）有此功能，属差异化空白
|
||||
```
|
||||
|
||||
#### KANO 归类操作步骤
|
||||
|
||||
**步骤一：基本型识别**
|
||||
- 列出所有 P0/P1 差评主题
|
||||
- 检查每个主题对应的好评：如果好评中几乎没有人因"做到了这点"而表扬，确认为基本型
|
||||
- 每个基本型需求必须注明：频次（条数）+ 出现该问题的竞品数量
|
||||
|
||||
**步骤二：期望型 vs 魅力型区分**
|
||||
|
||||
在好评中对每个高频好评主题做以下判断：
|
||||
|
||||
| 判断问题 | 若"是"→ | 若"否"→ |
|
||||
|---|---|---|
|
||||
| 差评中有人因该功能**不够好**而差评？ | 期望型 | 魅力型候选 |
|
||||
| 好评用程度词描述（`better/works great/very`）？ | 期望型 | 魅力型候选 |
|
||||
| 好评中出现 `love/obsessed/amazing/bonus/didn't expect`？ | 魅力型 | 继续判断 |
|
||||
| 竞品普遍缺失，属市场新鲜感？ | 魅力型 | 继续判断 |
|
||||
|
||||
**步骤三：反向型搜索（不得以"未发现"一笔带过）**
|
||||
|
||||
必须主动在差评中搜索以下关键词，并记录每个词的出现频次：
|
||||
|
||||
```
|
||||
搜索词组（在全部有效评论 title+content 中搜索）：
|
||||
过于复杂：too many parts / too complicated / confusing / hard to use
|
||||
过于嘈杂：too loud / so loud / noise / noisy
|
||||
功能多余：don't need / unnecessary / didn't ask for / useless feature
|
||||
操作繁琐：takes too long / too many steps / annoying to clean
|
||||
```
|
||||
|
||||
**结果处理规则**：
|
||||
- 若任意词组频次 ≥ 5 条 → 该功能为反向型，单独列出并附引用
|
||||
- 若所有词组总频次 < 5 条 → 填写："反向型：经主动搜索 [列出搜索词]，出现频次共 [N] 条，低于阈值，本品类暂无明确反向需求"（**禁止直接写"无"或"未发现"**）
|
||||
|
||||
### 4.2 JTBD 动机框架
|
||||
|
||||
> JTBD（Jobs To Be Done）：用户"雇佣"产品来完成什么任务。分析维度：功能性动机、情感性动机、社会性动机。
|
||||
|
||||
**输出格式（每个 Persona 一行）**：
|
||||
|
||||
| 字段 | 说明 | 填写规则 |
|
||||
|---|---|---|
|
||||
| 用户群 | Persona 名称 | — |
|
||||
| 核心 Job | 用户想完成的任务 | 动词+宾语形式，如"用电动工具替代传统方式" |
|
||||
| 功能性动机 | 实用层面的驱动因素 | 必须能从评论中找到佐证句子 |
|
||||
| 情感性动机 | 情绪/心理层面的驱动因素 | 必须能从评论中找到佐证句子 |
|
||||
| 社会性动机 | 他人视角/社交驱动（如无评论佐证则留空） | 可选 |
|
||||
| 购买触发时机 | 什么具体事件让用户决定购买 | 来自评论中的具体描述 |
|
||||
|
||||
### 4.3 人群 × 场景 × 需求矩阵
|
||||
|
||||
矩阵将 Persona、使用场景、KANO 需求分层和当前满意度整合为一张全景视图。
|
||||
|
||||
**列结构**：
|
||||
|
||||
| 列 | 填写来源 |
|
||||
|---|---|
|
||||
| 用户群 | Persona 名称 + 占比 |
|
||||
| 使用场景（When/Where） | **严格遵守场景字段规则（见 5.3 节）** |
|
||||
| 基本型需求 | KANO 基本型 + 该群体 P0 差评 |
|
||||
| 期望型需求 | KANO 期望型 + 该群体 P1 差评 |
|
||||
| 魅力型需求 | KANO 魅力型 + 该群体好评加分点 |
|
||||
| 当前满意度 | 该群体对应评论的均分和好评率综合判断 |
|
||||
|
||||
**满意度评级标准**：
|
||||
|
||||
| 当前满意度 | 对应均分参考 | 显示样式 |
|
||||
|---|---|---|
|
||||
| 高 | ≥ 4.0 | 绿色 |
|
||||
| 中等 | 3.3 – 3.9 | 黄色 |
|
||||
| 低 ⚠ | < 3.3 | 红色 |
|
||||
|
||||
### 4.4 痛点根因分析
|
||||
|
||||
**适用条件**：差评主题达到 P0 或 P1 级别时，必须进行根因分析。
|
||||
|
||||
**分析框架**：
|
||||
|
||||
```
|
||||
根因 N：[工程/设计/材料/体验设计问题名称]
|
||||
→ 导致后果：[差评主题名称] × [频次]
|
||||
→ 失效机制：[从产品结构或工作原理层面解释为什么会出现这个问题]
|
||||
→ 关键引用：[2-3条真实评论原文（英文）— 所属ASIN品牌]
|
||||
```
|
||||
|
||||
**根因分析的层次要求**：
|
||||
|
||||
| 层次 | 示例（错误 → 正确） |
|
||||
|---|---|
|
||||
| 停留在现象层（❌） | "产品质量差" |
|
||||
| 到达机制层（✅） | "充电口防水胶圈未达到IP67标准，浴室蒸汽渗入导致腐蚀" |
|
||||
|
||||
---
|
||||
|
||||
## 5. 报告生成逻辑
|
||||
|
||||
### 5.1 HTML 整体结构
|
||||
|
||||
报告采用**纯 HTML 内嵌 CSS + JS**，无外部文件依赖，单文件可直接分享。
|
||||
|
||||
```
|
||||
{产品关键词}-voc-v{版本号}.html
|
||||
├── <head>
|
||||
│ ├── Chart.js CDN（可视化依赖）
|
||||
│ │ └── https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js
|
||||
│ └── <style> 内嵌 CSS
|
||||
│
|
||||
├── <nav class="top-nav">（固定顶部导航，支持锚点跳转）
|
||||
│ ├── [描述层 What] → 数据总览 / 受众画像 / 正负反馈 / 各ASIN主题分布 / 情感词频
|
||||
│ └── [分析层 Why] → KANO模型 / JTBD动机 / 人群矩阵 / 痛点根因
|
||||
│
|
||||
└── <div class="page">（主体内容）
|
||||
├── 标题 + 副标题 + 阅读指引 callout
|
||||
├── ── 描述层 What ──
|
||||
│ ├── #sec-overview：KPI 总览卡片
|
||||
│ ├── #sec-persona：Persona 卡片网格
|
||||
│ ├── #sec-feedback：全市场差评/好评主题柱状图（汇总）
|
||||
│ ├── #sec-asin-theme：各 ASIN 差评/好评主题频次分布（新增）
|
||||
│ └── #sec-keyword：情感词频表格
|
||||
├── ── 分析层 Why ──
|
||||
│ ├── #sec-kano：KANO 四象限卡片
|
||||
│ ├── #sec-jtbd：JTBD 动机表格
|
||||
│ ├── #sec-matrix：人群×场景×需求矩阵
|
||||
│ └── #sec-rootcause：痛点根因分析块
|
||||
└── <footer>（数据来源声明）
|
||||
```
|
||||
|
||||
**文件存放路径**：`{产品文件夹}/分析报告 HTML/`
|
||||
|
||||
**文件命名**：`{产品关键词（连字符）}-voc-v{版本号}.html`
|
||||
示例：`dog-calming-chews-voc-v1.html`
|
||||
|
||||
### 5.2 可视化组件规格
|
||||
|
||||
#### KPI 卡片（4 个，固定布局）
|
||||
|
||||
| 位置 | 指标 | 颜色规则 |
|
||||
|---|---|---|
|
||||
| 卡片1 | 有效评论总数 | 蓝色（中性） |
|
||||
| 卡片2 | 加权平均评分 | < 3.5 红色 / 3.5–4.0 黄色 / > 4.0 绿色 |
|
||||
| 卡片3 | 正评率（≥4星占比） | 绿色 |
|
||||
| 卡片4 | 差评率（≤2星占比） | 红色 |
|
||||
|
||||
#### 柱状图（使用 Chart.js，水平条形图）
|
||||
|
||||
**差评主题柱状图（全市场汇总）**：
|
||||
|
||||
```javascript
|
||||
new Chart(document.getElementById('negChart'), {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels: ['主题1', '主题2', ...], // 按频次降序排列
|
||||
datasets: [{
|
||||
data: [频次1, 频次2, ...],
|
||||
// 颜色按优先级：P0 = '#ef4444'，P1 = '#f97316'，P2 = '#eab308'
|
||||
backgroundColor: ['#ef4444', '#ef4444', '#f97316', '#f97316', '#eab308', '#eab308'],
|
||||
}]
|
||||
},
|
||||
options: {
|
||||
indexAxis: 'y',
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: { legend: { display: false } },
|
||||
scales: { x: { beginAtZero: true } }
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**好评主题柱状图（全市场汇总）**：配色统一使用绿色系（`#22c55e`）。
|
||||
|
||||
---
|
||||
|
||||
#### 各 ASIN 主题频次分布图（新增，ID: sec-asin-theme）
|
||||
|
||||
**用途**：揭示同一痛点在不同竞品间的严重程度差异，帮助判断哪款竞品在哪个维度最弱/最强。
|
||||
|
||||
**图表类型**：**分组柱状图（Grouped Bar Chart）**，每个主题一组，每组内各 ASIN 一根柱子。
|
||||
|
||||
**差评版 — 数据结构**：
|
||||
|
||||
```javascript
|
||||
new Chart(document.getElementById('asinNegChart'), {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels: ['主题1', '主题2', ...], // x轴：差评主题（P0/P1 主题，按全市场频次降序）
|
||||
datasets: [
|
||||
// 每个 ASIN 一个 dataset
|
||||
{ label: 'ASIN1（品牌名）', data: [主题1频次, 主题2频次, ...], backgroundColor: '#色值' },
|
||||
{ label: 'ASIN2（品牌名）', data: [...], backgroundColor: '#色值' },
|
||||
// ...
|
||||
]
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
plugins: {
|
||||
legend: { display: true, position: 'bottom' }
|
||||
},
|
||||
scales: {
|
||||
x: { beginAtZero: true },
|
||||
y: { beginAtZero: true }
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**好评版 — 与差评版结构完全相同**，labels 换成好评主题，颜色使用绿色系渐变。
|
||||
|
||||
**ASIN 配色规则（固定，每次分析保持一致）**：
|
||||
|
||||
| ASIN 序号 | 差评图颜色 | 好评图颜色 |
|
||||
|---|---|---|
|
||||
| ASIN 1 | `#ef4444`（红） | `#22c55e`（绿） |
|
||||
| ASIN 2 | `#f97316`（橙） | `#86efac`（浅绿） |
|
||||
| ASIN 3 | `#eab308`（黄） | `#4ade80`（草绿） |
|
||||
| ASIN 4 | `#8b5cf6`（紫） | `#2dd4bf`（青绿） |
|
||||
| ASIN 5 | `#3b82f6`（蓝） | `#60a5fa`（浅蓝绿） |
|
||||
|
||||
**数据提取逻辑（Python）**：
|
||||
|
||||
```python
|
||||
def count_theme_per_asin(all_reviews, themes, neg=True):
|
||||
"""
|
||||
按 ASIN 分别统计各主题的差评/好评频次
|
||||
返回格式：{asin: {theme: count}}
|
||||
"""
|
||||
from collections import defaultdict
|
||||
result = defaultdict(lambda: defaultdict(int))
|
||||
for r in all_reviews:
|
||||
if neg and r['rating'] > 2: continue
|
||||
if not neg and r['rating'] < 4: continue
|
||||
text = (r['title'] + ' ' + r['content']).lower()
|
||||
for theme, keywords in themes.items():
|
||||
if any(kw.lower() in text for kw in keywords):
|
||||
result[r['asin']][theme] += 1
|
||||
return dict(result)
|
||||
|
||||
# 转为 Chart.js datasets 格式
|
||||
def to_chartjs_datasets(per_asin_data, asin_labels, theme_order):
|
||||
"""
|
||||
asin_labels: {asin: '品牌名'} 映射
|
||||
theme_order: 主题列表（按全市场频次排序）
|
||||
"""
|
||||
colors = ['#ef4444', '#f97316', '#eab308', '#8b5cf6', '#3b82f6']
|
||||
datasets = []
|
||||
for i, (asin, label) in enumerate(asin_labels.items()):
|
||||
datasets.append({
|
||||
'label': label,
|
||||
'data': [per_asin_data.get(asin, {}).get(t, 0) for t in theme_order],
|
||||
'backgroundColor': colors[i % len(colors)],
|
||||
})
|
||||
return datasets
|
||||
```
|
||||
|
||||
**图表尺寸**：高度建议 `360px`（主题数 ≤ 8）或 `480px`（主题数 > 8）。
|
||||
|
||||
**图表标题规则**：
|
||||
- 差评版：`各竞品差评主题频次对比（≤2星评论）`
|
||||
- 好评版：`各竞品好评主题频次对比（≥4星评论）`
|
||||
|
||||
#### 表格通用样式规则
|
||||
|
||||
| 元素 | CSS 类 | 用途 |
|
||||
|---|---|---|
|
||||
| P0 行 | `.row-r` | 红色背景底色（`#fff5f5`） |
|
||||
| P1 行 | `.row-y` | 黄色背景底色（`#fffbeb`） |
|
||||
| P2 行 | `.row-b` | 蓝色背景底色（`#eff6ff`） |
|
||||
| 机会行 | `.row-g` | 绿色背景底色（`#f0fdf4`） |
|
||||
| 优先级徽章 | `.pill pill-danger` / `.pill-warn` / `.pill-info` | 红/黄/蓝圆角标签 |
|
||||
|
||||
### 5.3 使用场景字段写入规则（核心规则）
|
||||
|
||||
**这是本方法论中最容易出错、最需要严格执行的规则：**
|
||||
|
||||
```
|
||||
【强制规则】
|
||||
所有涉及"使用场景（When/Where）"的字段，
|
||||
只允许填写在评论 content 中能找到原词佐证的场景描述。
|
||||
如果找不到评论佐证，一律填写 "—"。
|
||||
禁止基于产品功能、品类常识或逻辑推断填写任何场景描述。
|
||||
```
|
||||
|
||||
#### 执行步骤
|
||||
|
||||
```
|
||||
Step 1：确定该 Persona 的特征词组
|
||||
Step 2：在该 Persona 对应的评论中搜索场景类词汇
|
||||
（地点词：shower / bathroom / office / car / gym...）
|
||||
（时间词：morning / night / before / after / daily...）
|
||||
（情境词：traveling / pregnant / postpartum / gift...）
|
||||
Step 3：统计每个场景词在该 Persona 评论中的出现条数
|
||||
Step 4：按以下规则决定是否填写
|
||||
```
|
||||
|
||||
| 出现条数 | 处理方式 |
|
||||
|---|---|
|
||||
| ≥ 5 条 | 可以填写该场景 |
|
||||
| 2 – 4 条 | 慎重填写，建议留空或标注"少数提及" |
|
||||
| < 2 条 | 必须留空，填写 `—` |
|
||||
|
||||
#### 常见错误示例
|
||||
|
||||
| 错误写法（❌ 推断） | 正确写法（✅ 仅来自评论） |
|
||||
|---|---|
|
||||
| 夏季前突击整理 | —（无评论提及"before summer"作为使用时机） |
|
||||
| 日常护理 | —（"daily"在评论中属产品使用频率描述，非场景） |
|
||||
| 坐姿/斜靠操作 | —（用户姿势属推断，评论未明确提及） |
|
||||
| 任何场所 | —（无具体场景词，不可用"泛化"代替留空） |
|
||||
| 浴室/淋浴 | ✅（评论中出现 `in the shower` / `bathroom` ≥5条） |
|
||||
| 旅行途中 | ✅（评论中出现 `travel` / `on a trip` ≥5条） |
|
||||
| 孕期护理 | ✅（评论中出现 `pregnant` / `37 weeks pregnant` ≥5条） |
|
||||
|
||||
---
|
||||
|
||||
## 6. 执行 SOP（逐步操作流程）
|
||||
|
||||
### Step 1：数据读取
|
||||
|
||||
```python
|
||||
import csv
|
||||
import os
|
||||
|
||||
def load_reviews(filepath):
|
||||
"""读取单个ASIN的评论CSV，返回有效评论列表"""
|
||||
reviews = []
|
||||
with open(filepath, encoding='utf-8-sig') as f: # utf-8-sig 处理BOM头
|
||||
for row in csv.DictReader(f):
|
||||
if (row.get('verified', '').strip().lower() == 'true' or
|
||||
row.get('vine', '').strip().lower() == 'true'):
|
||||
reviews.append({
|
||||
'asin': row.get('asin', '').strip(),
|
||||
'rating': float(row.get('rating', 0) or 0),
|
||||
'title': row.get('title', ''),
|
||||
'content': row.get('content', ''),
|
||||
'date': row.get('review_date', ''),
|
||||
})
|
||||
return reviews
|
||||
|
||||
def load_all_reviews(directory):
|
||||
"""批量读取目录下所有ASIN的评论"""
|
||||
all_reviews = []
|
||||
for fname in os.listdir(directory):
|
||||
if fname.endswith('_realtime.csv'):
|
||||
all_reviews.extend(load_reviews(os.path.join(directory, fname)))
|
||||
return all_reviews
|
||||
```
|
||||
|
||||
### Step 2：基础统计
|
||||
|
||||
```python
|
||||
def basic_stats(reviews):
|
||||
"""计算有效评论的基础统计指标"""
|
||||
ratings = [r['rating'] for r in reviews if r['rating'] > 0]
|
||||
total = len(ratings)
|
||||
if not total:
|
||||
return {}
|
||||
return {
|
||||
'total': total,
|
||||
'avg': round(sum(ratings) / total, 2),
|
||||
'pos_rate': round(sum(1 for r in ratings if r >= 4) / total, 3),
|
||||
'neg_rate': round(sum(1 for r in ratings if r <= 2) / total, 3),
|
||||
'dist': {i: ratings.count(float(i)) for i in range(1, 6)},
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3：主题频次统计
|
||||
|
||||
```python
|
||||
# 根据品类自定义主题关键词（见第3.3节）
|
||||
NEGATIVE_THEMES = {
|
||||
'{主题名称}': ['{关键词1}', '{关键词2}', ...],
|
||||
}
|
||||
POSITIVE_THEMES = {
|
||||
'{主题名称}': ['{关键词1}', '{关键词2}', ...],
|
||||
}
|
||||
|
||||
def count_themes(reviews, themes, neg=True):
|
||||
"""
|
||||
统计各主题频次
|
||||
neg=True 时只统计差评(≤2星)，neg=False 时只统计好评(≥4星)
|
||||
"""
|
||||
result = {}
|
||||
for theme, keywords in themes.items():
|
||||
count = 0
|
||||
for r in reviews:
|
||||
if neg and r['rating'] > 2: continue
|
||||
if not neg and r['rating'] < 4: continue
|
||||
text = (r['title'] + ' ' + r['content']).lower()
|
||||
if any(kw.lower() in text for kw in keywords):
|
||||
count += 1
|
||||
result[theme] = count
|
||||
return dict(sorted(result.items(), key=lambda x: x[1], reverse=True))
|
||||
```
|
||||
|
||||
### Step 4：Persona 识别
|
||||
|
||||
```python
|
||||
# 根据品类自定义（见第3.2节）
|
||||
PERSONA_KEYWORDS = {
|
||||
'{群体名称}': ['{关键词1}', '{关键词2}', ...],
|
||||
}
|
||||
|
||||
def identify_personas(reviews, persona_keywords):
|
||||
"""统计各Persona的命中评论数量"""
|
||||
result = {p: 0 for p in persona_keywords}
|
||||
for r in reviews:
|
||||
text = (r['title'] + ' ' + r['content']).lower()
|
||||
for persona, kws in persona_keywords.items():
|
||||
if any(kw.lower() in text for kw in kws):
|
||||
result[persona] += 1
|
||||
total = len(reviews)
|
||||
return {p: {'count': c, 'pct': round(c / total * 100)} for p, c in result.items()}
|
||||
```
|
||||
|
||||
### Step 5：场景词验证（填写矩阵前必须执行）
|
||||
|
||||
```python
|
||||
def verify_scene(persona_reviews, scene_keywords, min_count=5):
|
||||
"""
|
||||
验证某个场景词是否在该Persona评论中出现足够多次
|
||||
返回 (是否可填写, 实际出现条数)
|
||||
"""
|
||||
count = 0
|
||||
for r in persona_reviews:
|
||||
text = (r['title'] + ' ' + r['content']).lower()
|
||||
if any(kw.lower() in text for kw in scene_keywords):
|
||||
count += 1
|
||||
return count >= min_count, count
|
||||
|
||||
# 示例用法
|
||||
persona_reviews = [r for r in all_reviews if '粗硬发质' in identify_personas_for_review(r)]
|
||||
can_fill, cnt = verify_scene(persona_reviews, ['shower', 'bathroom', 'in the shower'])
|
||||
scene_text = '浴室/淋浴' if can_fill else '—'
|
||||
```
|
||||
|
||||
### Step 6：报告组装流程
|
||||
|
||||
1. **准备数据**：运行 Step 1–5，收集所有统计结果
|
||||
2. **起草 Canvas**（在 IDE 中生成 `.canvas.tsx` 文件），等待用户确认内容无误
|
||||
3. **用户确认后**：生成 HTML 文件，保存至 `分析报告 HTML/` 目录
|
||||
4. **Canvas 与 HTML 内容必须保持同步**，修改 Canvas 后须同步更新 HTML
|
||||
|
||||
---
|
||||
|
||||
## 7. 关键阈值与判断规则速查表
|
||||
|
||||
| 决策点 | 规则 |
|
||||
|---|---|
|
||||
| **有效评论筛选** | `verified == True` 或 `vine == True` |
|
||||
| **差评定义** | rating ≤ 2.0 |
|
||||
| **好评定义** | rating ≥ 4.0 |
|
||||
| **P0 主题门槛** | 频次 ≥ 有效差评数 × 20%，且 80%+ 竞品出现 |
|
||||
| **P1 主题门槛** | 频次为有效差评数的 10–20%，60%+ 竞品出现 |
|
||||
| **P2 主题门槛** | 频次为有效差评数的 3–10%，40%+ 竞品出现 |
|
||||
| **Persona 占比** | 命中评论数 / 有效总数，四舍五入至整5% |
|
||||
| **场景词可填写门槛** | ≥ 5 条 Persona 评论中出现该场景词，否则填 `—` |
|
||||
| **市场整体判断** | 均分 < 3.5 = 系统性缺陷，新品窗口期 |
|
||||
| **KANO 基本型判断** | P0 级差评，且与低评分强相关 |
|
||||
| **KANO 魅力型判断** | 好评中出现 `love`/`amazing`/`didn't expect`，非差评主题 |
|
||||
| **新买家识别** | content 含 `first time`/`just got`/`just bought`/`new to` |
|
||||
| **复购买家识别** | content 含 `reorder`/`bought again`/`second time`/`repurchase` |
|
||||
| **根因分析触发** | 差评主题达到 P0 或 P1 级别 |
|
||||
| **Persona 引用合规** | 只能引用 content 中确实存在的原文，禁止改写或虚构 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 新品类接入清单
|
||||
|
||||
每次分析新品类时，依次完成以下配置，其余分析框架直接复用：
|
||||
|
||||
```
|
||||
□ 1. 确认 CSV 文件路径和 ASIN 列表
|
||||
□ 2. 阅读**全部差评和全部好评**，按三个维度（A物理特征 / B行为场景 / C购买动机）归纳 Persona，编写 PERSONA_KEYWORDS
|
||||
□ 3. 执行**自我标注信号强制检查**（搜索 "I have [adj]..." / "As a [noun]..." 等模式），确认无遗漏的物理特征群体
|
||||
□ 4. 阅读**全部差评（≤2星）**，归纳差评主题，注意根因不同的问题必须拆分为独立主题，编写 NEGATIVE_THEMES
|
||||
□ 5. 阅读**全部好评（≥4星）**，归纳 4–6 个好评主题，编写 POSITIVE_THEMES
|
||||
□ 6. 运行 Step 1–3，验证频次统计结果与人工阅读印象一致；同时用 count_theme_per_asin() 生成各 ASIN 的主题频次分布数据，用于 #sec-asin-theme 图表
|
||||
□ 7. **Persona 完整性验证**：为每个 P0/P1 主题强制归因到 Persona，有 ❌ 则返回步骤 2 补充
|
||||
□ 8. 对每个 Persona 执行 verify_scene()，确认使用场景字段
|
||||
□ 9. 完成 KANO 归类（基于统计结果，无需额外数据）
|
||||
□ 10. 完成 JTBD 框架（基于 Persona 评论，无需额外数据）
|
||||
□ 11. 撰写根因分析（P0/P1 主题，每条根因附 2-3 条原文引用）
|
||||
□ 12. 生成 Canvas → 用户确认 → 生成 HTML
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*本文档为通用框架，无品类特定数据。新品类分析时，只需填写第 8 节清单中的品类相关配置，其余规则和代码模板均可直接复用。*
|
||||
|
|
@ -24,7 +24,7 @@ PROJECT_ROOT = Path(__file__).resolve().parent
|
|||
DEFAULT_MODEL_PATH = PROJECT_ROOT / "Qwen3-Embedding-4B-mxfp8"
|
||||
# 实测短句 batch=32 仍 <0.5GB增量；长句 500 字符 batch=16 约 1.5GB 峰值
|
||||
DEFAULT_BATCH_SIZE = 16
|
||||
DEFAULT_MAX_TEXT_CHARS = 512
|
||||
DEFAULT_MAX_TEXT_CHARS = 10_000
|
||||
|
||||
|
||||
def _apply_hf_hub_shim() -> None:
|
||||
|
|
|
|||
|
|
@ -3,8 +3,8 @@ VOC 全流程：合并 → 清洗 → 结构化 → 向量化 →（聚类 ∥
|
|||
|
||||
用法::
|
||||
|
||||
# 全流程（--industry 默认 Pet Supplies；写入 sqlite 前默认清理旧库，keep-db不清理）
|
||||
./310py/bin/python main_voc分析.py --input-dir 目录 --product "产品名" --keep-db
|
||||
# 全流程（--product 默认「亚马逊商品」；--industry 默认「-」；写入 sqlite 前默认清理旧库，keep-db 不清理）
|
||||
./310py/bin/python main_voc分析.py --input-dir reviews_export --keep-db
|
||||
--industry "行业名"
|
||||
# 断点续跑（步骤 4 起可省略 --product，自动读 voc_structured.sqlite）
|
||||
./310py/bin/python main_voc分析.py --from-step 5
|
||||
|
|
@ -32,7 +32,7 @@ from pathlib import Path
|
|||
from content清洗 import process_reviews, save_cleaned_reviews
|
||||
from 合并评论数据 import merge_csv_directory
|
||||
from 向量化 import EMBED_DEFAULT_WORKERS, run_embed
|
||||
from 结构化_server import STRUCT_DEFAULT_WORKERS, run_analysis
|
||||
from 结构化_server import DEFAULT_MAX_BATCH_REVIEWS, STRUCT_DEFAULT_WORKERS, run_analysis
|
||||
from 聚类 import run_clustering
|
||||
from voc_llm import require_chat_api_key
|
||||
from voc_report import DEFAULT_CLUSTER_MIN_REVIEW_RATIO, generate_report
|
||||
|
|
@ -56,7 +56,10 @@ WORD_FREQ_CSV = OUTPUT_DIR / "word_freq.csv"
|
|||
|
||||
DEFAULT_MERGED = PROJECT_ROOT / "merged_reviews.csv"
|
||||
DEFAULT_CLEANED = PROJECT_ROOT / "merged_reviews_cleaned.csv"
|
||||
DEFAULT_INDUSTRY = "-"
|
||||
DEFAULT_INDUSTRY = "亚马逊电商"
|
||||
DEFAULT_PRODUCT = "亚马逊商品"
|
||||
|
||||
|
||||
|
||||
|
||||
def _safe_product_dir_name(product_name: str) -> str:
|
||||
|
|
@ -164,6 +167,7 @@ def run_voc_analysis(
|
|||
file_path=str(cleaned_csv),
|
||||
clean_databases=clean_databases,
|
||||
workers=struct_workers,
|
||||
max_batch_reviews=DEFAULT_MAX_BATCH_REVIEWS,
|
||||
)
|
||||
result["structured_job_id"] = ar.get("job_id")
|
||||
result["structured_db"] = str(STRUCTURED_DB)
|
||||
|
|
@ -282,23 +286,19 @@ def _resolve_industry_product(
|
|||
from_step: int,
|
||||
only_step: int | None,
|
||||
) -> tuple[str, str]:
|
||||
"""步骤 4 起可从 structured 库自动读取 product；industry 默认 Pet Supplies。"""
|
||||
"""步骤 4 起若 product 仍为默认值，可从 structured 库自动读取；industry 默认「-」。"""
|
||||
ind = (industry or DEFAULT_INDUSTRY).strip() or DEFAULT_INDUSTRY
|
||||
if product_name:
|
||||
return ind, product_name
|
||||
prod = (product_name or DEFAULT_PRODUCT).strip() or DEFAULT_PRODUCT
|
||||
start_step = only_step if only_step is not None else from_step
|
||||
if start_step >= 4:
|
||||
if not STRUCTURED_DB.is_file():
|
||||
raise SystemExit(
|
||||
f"缺少 {STRUCTURED_DB.name}，无法自动读取 --product"
|
||||
)
|
||||
_, db_ind, prod = _latest_job_meta(STRUCTURED_DB)
|
||||
logger.info("未指定 product，已从 structured 库读取: %r（industry=%r）", prod, db_ind)
|
||||
return db_ind, prod
|
||||
raise SystemExit(
|
||||
f"步骤 1–3 需要 --product；--industry 可省略（默认 {DEFAULT_INDUSTRY}）；"
|
||||
"从步骤 4 起 product 也可省略（自动读 voc_structured.sqlite）"
|
||||
if start_step >= 4 and prod == DEFAULT_PRODUCT and STRUCTURED_DB.is_file():
|
||||
_, db_ind, db_prod = _latest_job_meta(STRUCTURED_DB)
|
||||
logger.info(
|
||||
"product 为默认值，已从 structured 库读取: %r（industry=%r）",
|
||||
db_prod,
|
||||
db_ind,
|
||||
)
|
||||
return db_ind, db_prod
|
||||
return ind, prod
|
||||
|
||||
|
||||
def main() -> None:
|
||||
|
|
@ -314,7 +314,11 @@ def main() -> None:
|
|||
default=DEFAULT_INDUSTRY,
|
||||
help=f"行业（默认 {DEFAULT_INDUSTRY}）",
|
||||
)
|
||||
parser.add_argument("--product", default=None, help="产品名（步骤 4 起可省略，自动读库）")
|
||||
parser.add_argument(
|
||||
"--product",
|
||||
default=DEFAULT_PRODUCT,
|
||||
help=f"产品名（默认 {DEFAULT_PRODUCT}；步骤 4 起若为默认值则自动读库）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--merged-csv",
|
||||
type=Path,
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@
|
|||
## 占位符
|
||||
|
||||
`.md` 文件使用 Python `format` 语法，例如 `{product_name}`、`{industry}`。
|
||||
正文里需要字面量花括号时写双花括号：`{{"audience": "unknown"}}`。
|
||||
正文里需要字面量花括号时写双花括号：`{{"persona_signals": []}}`。
|
||||
|
||||
## 改完怎么验收
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,6 @@
|
|||
本批共 {n_keys} 条评论，用户消息中每条评论以 [C1]、[C2]… 前缀标识。
|
||||
- 只输出一个 JSON 对象；顶层键必须且仅能是：{keys_literal}
|
||||
- 每个顶层键对应一条同前缀评论，不得遗漏、不得新增其他顶层键
|
||||
- 每个键的值是单条结构化对象，仅含 audience、pain_points、product_feedback 三个字段
|
||||
- 每个键的值是单条结构化对象，仅含 persona_signals、pain_points、product_feedback 三个字段
|
||||
- 不要用 JSON 数组作为顶层；不要把多条评论合并进一个对象；不要用 results、data 等包裹层
|
||||
- 不要输出 markdown 代码围栏或任何解释文字
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@
|
|||
- instruction: "示例 1 教学：展示正常长评论如何标准提取，如何准确分类产品优点、缺陷以及物流问题。"
|
||||
review: "Bought this for my 12yo lab who struggles with stairs. It is very soft and helps her get onto the bed easily. But the zipper broke after a week and the shipping box was damaged."
|
||||
output:
|
||||
audience: "12yo lab"
|
||||
persona_signals:
|
||||
- "large breed dog"
|
||||
- "mobility limited pet"
|
||||
pain_points:
|
||||
- "struggles with stairs"
|
||||
- "difficulty getting onto bed"
|
||||
|
|
@ -23,7 +25,8 @@
|
|||
- instruction: "示例 2 教学：防混淆与隐性属性提炼。什么是「产品缺陷（如讲太快）」，它属于 product_feedback，绝不是 pain_points（用户需求）！并提炼出 teaching speed 这个隐性反馈对象。"
|
||||
review: "The course content is good, the teacher speaks too fast, can't keep up."
|
||||
output:
|
||||
audience: "student"
|
||||
persona_signals:
|
||||
- "online course learner"
|
||||
pain_points: []
|
||||
product_feedback:
|
||||
- aspect: "course content"
|
||||
|
|
@ -38,7 +41,9 @@
|
|||
- instruction: "示例 3 教学：深度上下文推理。如何通过动作（抓跳蚤）推理出使用者（pet），如何通过「全家能睡觉」推理出用户需求（失眠），以及如何将「Worth every penny」准确归类为对 Price（价格）的产品反馈。"
|
||||
review: "Finally found something that stops the midnight flea scratching! The chemical smell is a bit strong initially, but it fades. Worth every penny since our family can finally sleep."
|
||||
output:
|
||||
audience: "pet"
|
||||
persona_signals:
|
||||
- "flea problem pet"
|
||||
- "household sleep disruption"
|
||||
pain_points:
|
||||
- "midnight flea scratching"
|
||||
- "sleep deprivation"
|
||||
|
|
@ -52,10 +57,32 @@
|
|||
sentiment: Positive
|
||||
category: Price
|
||||
|
||||
- instruction: "示例 4 教学：保留关键成分与症状等核心具体词汇。绝不能将带有具体成分/病症的词泛化提取。例如遇到「chicken flavor」（鸡肉风味）或「joint pain」（关节疼痛）时，必须保留核心修饰词，绝不能错误缩减提取为「flavor」或「pain」。"
|
||||
- instruction: "示例 4 教学：1) 非英语评论（例如西班牙语）输出必须全英文，禁止保留原文非英文片段，先理解语义再用英文短语表达；2) 「随附配件」不属于 8 个标准类别，演示如何补充一个简洁的英文新类别 'Accessories'。"
|
||||
review: "El producto en sí está muy bien, viene con 2 cristales templados y 2 grips para los joycon. Pero es lo que he leido en algunos comentarios, la funda huele como a tabaco jajaja es raro pero es así"
|
||||
output:
|
||||
persona_signals:
|
||||
- "joycon grip user"
|
||||
pain_points: []
|
||||
product_feedback:
|
||||
- aspect: "product"
|
||||
opinion: "it's fine"
|
||||
sentiment: Positive
|
||||
category: Quality
|
||||
- aspect: "included accessories"
|
||||
opinion: "comes with 2 tempered glass and 2 grips"
|
||||
sentiment: Positive
|
||||
category: Accessories
|
||||
- aspect: "case smell"
|
||||
opinion: "smells like tobacco, strange"
|
||||
sentiment: Negative
|
||||
category: Quality
|
||||
|
||||
- instruction: "示例 5 教学：保留关键成分与症状等核心具体词汇。绝不能将带有具体成分/病症的词泛化提取。例如遇到「chicken flavor」（鸡肉风味）或「joint pain」（关节疼痛）时，必须保留核心修饰词，绝不能错误缩减提取为「flavor」或「pain」。"
|
||||
review: "My elderly cat suffers from severe joint pain. But he is very attracted by the chicken flavor of this supplement! It really helps him walk better."
|
||||
output:
|
||||
audience: "elderly cat"
|
||||
persona_signals:
|
||||
- "senior cat"
|
||||
- "joint pain cat"
|
||||
pain_points:
|
||||
- "severe joint pain"
|
||||
product_feedback:
|
||||
|
|
|
|||
|
|
@ -1,7 +1,19 @@
|
|||
## 语言规则（红线）:
|
||||
- 评论输入可能是英语、西班牙语、法语、德语、日语等**任意语言**。
|
||||
- **所有输出字段值**（persona_signals 每一条、pain_points 每一条、aspect、opinion）**必须是自然英文**。
|
||||
- **禁止**保留原文非英文片段（如 está muy bien、huele como a tabaco、très bon）；先理解语义，再用英文短语表达。
|
||||
|
||||
## 分析要求:
|
||||
|
||||
1. audience (为谁购买):
|
||||
- 提取出实际的使用者，使用简短的英文名词。若无明确提及，请根据上下文推理；若完全无法推理则输出 'unknown'。
|
||||
1. persona_signals (用户画像信号):
|
||||
- **字符串数组**；每条为 2–8 词的英文短语，描述身份、体质、场景或使用背景。
|
||||
- 从评论原文推断，典型信号：
|
||||
- 体质/生理：`sensitive skin`、`coarse thick hair`、`pregnant`、`elderly`
|
||||
- 场景/行为：`travel grooming`、`shower use`、`bikini area`、`first-time buyer`
|
||||
- 自我标注句式：`I have …` / `As a …` / `My skin is …` 须提炼为短语写入此处
|
||||
- **禁止**仅写 `self` / `user` / `unknown` 等泛化词。
|
||||
- 不得把产品缺陷/差评（如 `pulls hair`、`battery dead`）写入 persona_signals；产品体验属于 product_feedback。
|
||||
- 若无任何可区分信号，可输出 `[]`。
|
||||
|
||||
2. pain_points (用户需求):
|
||||
- 仅限提取用户在购买前遇到的外部困扰、疾病、或具体场景（购买前尚未被本产品解决的需求）。
|
||||
|
|
@ -16,6 +28,8 @@
|
|||
- **禁止** Mixed、Ambiguous、Both、Balanced 等自创词。
|
||||
- 同一条反馈褒贬交织时：选**最主要**倾向；或拆成多条 product_feedback 分别标注。
|
||||
- 无法判断倾向时用 Neutral。
|
||||
- category (类别): 仅限 {categories_literal}（禁止 Value、Cost 等自创词；性价比高/物有所值 归入 Price）。
|
||||
- category (类别): 优先归入以下 8 个标准类别之一：{categories_literal}。
|
||||
- 归类提示：性价比/物有所值→'Price'；成分/材质/配方→'Ingredient'；效果/作用/功能/是否好用→'Function'；品牌/正品/可靠性/信任→'Trust'。禁止 Value、Cost 等同义自创词。
|
||||
- 仅当某条反馈确实**无法**归入以上 8 类时，才可补充一个简洁的英文新类别（1-2 个英文单词、首字母大写，如 'Scent'、'Accessories'），不要硬塞或滥造。
|
||||
- 每条 product_feedback 必须同时包含 aspect、opinion、sentiment、category 四个子字段。
|
||||
- 若无提及 product_feedback，输出空列表 []
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
5. 无关评论过滤：
|
||||
- 若某条评论明显与{product_name}无关（其他品类、其他 SKU 或完全跑题），该条输出：{{"audience": "unknown", "pain_points": [], "product_feedback": []}}。
|
||||
- 不得将无关内容填入 audience、pain_points 或 product_feedback。
|
||||
- 若某条评论明显与{product_name}无关（其他品类、其他 SKU 或完全跑题），该条输出：{{"persona_signals": [], "pain_points": [], "product_feedback": []}}。
|
||||
- 不得将无关内容填入 persona_signals、pain_points 或 product_feedback。
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
4. 单条评论对象内的格式与字段约束：
|
||||
- 每个评论对象只能包含 `audience`, `pain_points`, `product_feedback` 这 3 个字段。
|
||||
- 每个评论对象只能包含 `persona_signals`, `pain_points`, `product_feedback` 这 3 个字段。
|
||||
- **绝对不要**在 JSON 中输出 `instruction`、`教学说明` 或其他任何多余字段。
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
4. 格式与字段约束：
|
||||
- **你的 JSON 输出只能包含 `audience`, `pain_points`, `product_feedback` 这 3 个根字段。**
|
||||
- **你的 JSON 输出只能包含 `persona_signals`, `pain_points`, `product_feedback` 这 3 个根字段。**
|
||||
- **绝对不要**在 JSON 中输出 `instruction`、`教学说明` 或其他任何多余字段。
|
||||
- 必须以纯 JSON 格式输出结果，不要包含任何 markdown 标记（如 ```json ）或其他解释性文字。
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
以下为 {n_keys} 段带前缀的英文评论，每段互相独立。请严格按 system 中的「批量输出格式」返回 JSON，顶层键为 {keys_literal}。
|
||||
以下为 {n_keys} 段带前缀的用户评论（可能是任意语言），每段互相独立。所有输出字段值必须为英文。请严格按 system 中的「批量输出格式」返回 JSON，顶层键为 {keys_literal}。
|
||||
|
||||
{tagged_input}
|
||||
|
|
|
|||
|
|
@ -161,7 +161,8 @@ def _report_marker_kwargs(
|
|||
"marker_word_zh": markers[0],
|
||||
"marker_word_category": markers[1],
|
||||
"marker_cluster_names": markers[2],
|
||||
"marker_report_html": markers[3],
|
||||
"marker_opportunity_matrix": markers[3],
|
||||
"marker_report_html": markers[4],
|
||||
"wordcloud_top_n": wf.get("wordcloud_top_n", 180),
|
||||
"word_category_classify_n": wf.get("word_category_classify_n", 180),
|
||||
"outlier_label_zh": rp.get("outlier_label_zh", "未归类"),
|
||||
|
|
@ -283,7 +284,7 @@ def validate_prompt_files() -> List[str]:
|
|||
sch = get_schema()
|
||||
if not word_categories(cfg):
|
||||
errors.append("config.yaml: word_categories 为空")
|
||||
if len(report_markers(sch)) != 4:
|
||||
if len(report_markers(sch)) != 5:
|
||||
errors.append("schema.yaml: report.markers 须为 4 项")
|
||||
if not product_feedback_categories(sch):
|
||||
errors.append("schema.yaml: extraction.product_feedback_categories 为空")
|
||||
|
|
|
|||
|
|
@ -5,4 +5,14 @@
|
|||
2.分受众分析 — 对 top2 受众分别写（1）用户需求（2）正面（3）负面（4）客观（仅用对应 2a/2b stage）。
|
||||
- **（二）全部用户需求与产品反馈**：1.全部受众需求（3a）、2.正面、3.负面、4.客观（3b 三档）；
|
||||
每个 <ul> 只收纳对应 stage 的簇；占比=结构化短语数÷a；同 ul 内按短语数降序。正文禁用「观点 / 评价 / 痛点」。
|
||||
3. **三、改进建议与机会**：保持模版 <h2> 与四条 <ol> 结构；建议须可执行，第 4 条单独写可放大的产品/市场机会；覆盖未满足需求、负面反馈与客观描述中的风险，勿复述本条款文字。
|
||||
- **（三）多维交叉洞察**（新增）：从聚类数据中推断以下维度并进行交叉分析：
|
||||
1.**使用场景推断** — 从评论关键词和需求簇推断用户的典型使用场景（如日常家用、专业工作、礼品/转售、特定季节等），分析不同场景下的需求差异。
|
||||
2.**购买动机分析** — 推断驱动购买的核心动机（功能刚需、性价比、品牌信任、替代竞品、尝新等），每个动机引用具体聚类数据支撑。
|
||||
3.**人群×需求矩阵** — 将 top2 受众与 top 需求/反馈做交叉比对，找出各群体的独有关注点（A群体在意 X 但 B群体不在意）和共性盲区。
|
||||
每条洞察必须引用具体数据占比或簇名，禁止空泛概括。
|
||||
3. **三、改进建议与机会**：须严格按模版结构输出六条 <li>。
|
||||
- 前三条为**短期快赢**建议（0-3月可落地），包括产品本身的微调、包装/说明书优化、Listing与A+页面优化。
|
||||
- 第四条为**中期优化**（3-6月），涵盖配件组合/套装策略、服务升级（售后/质保/教程内容）、差异化定价与变体策略。
|
||||
- 第五条为**长期布局**（6-12月），覆盖新品类延伸、场景化产品线、品牌社群与内容生态等前瞻性方向。
|
||||
- 第六条为**差异化机会**，专门指出竞品未覆盖但本品有潜力占领的空白市场/场景/人群机会。
|
||||
- 每条建议须关联具体数据洞察（引用对应簇名或占比），不可泛泛而谈。
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
{err_block}
|
||||
|
||||
【修正要点】
|
||||
- 四个标记必须按顺序出现：{marker_word_zh}、{marker_word_category}、{marker_cluster_names}、{marker_report_html}（JSON 在前，HTML 最后）
|
||||
- 五个标记必须按顺序出现：{marker_word_zh}、{marker_word_category}、{marker_cluster_names}、{marker_opportunity_matrix}、{marker_report_html}（JSON 在前，HTML 最后）
|
||||
- JSON 段必须是严格合法 JSON：键与字符串均用英文双引号；例如 "words":["dog","cat"]，禁止写成 "words:["dog"] 或 words:[
|
||||
- 不要用 Markdown 代码围栏包裹 JSON；WORD_CATEGORY_JSON 的键仅限：{cats_literal}
|
||||
- WORD_CATEGORY_JSON 必须是**一个** JSON 对象，words 仅来自 Top{word_category_classify_n}；逐词尽量归类、不强制每类凑满；禁止「其他」键；禁止多段 `{{"某类":[...]}}, "另一类":[...]`
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
请严格按以下四段标记输出（标记外不要有任何文字；**顺序不可调换**）：
|
||||
请严格按以下五段标记输出（标记外不要有任何文字；**顺序不可调换**）：
|
||||
|
||||
{marker_word_zh}
|
||||
一行合法 JSON 对象：Top{wordcloud_top_n} 英文词 -> 中文，如 {{"dog":"狗","cat":"猫"}}。
|
||||
|
|
@ -19,6 +19,18 @@
|
|||
簇标签 -1 统一命名为「{outlier_label_zh}」。
|
||||
命名规则：1_audience、3a_pain_global、3b_aspect_opinion_* 写完整业务簇名；
|
||||
2a/2b 只写子主题（程序拼接受众名）；2b 子主题禁止含情感词。
|
||||
【簇名风格红线】每个簇名最多 15 个中文字，必须用日常口语，让运营人员一眼看懂。
|
||||
正确示范：「多猫家庭」「喷头漏液」「猫在床上乱尿」「气味刺鼻」「完全无效」
|
||||
错误示范：「针对床铺沙发等亲密区域的定点排尿抗议行为」「效力根本性不足导致乱尿行为完全无法被中断」
|
||||
|
||||
{marker_opportunity_matrix}
|
||||
一个合法 JSON 数组，恰好 6 个元素，与 REPORT_HTML 中「三、改进建议与机会」的 6 条 <li> 一一对应。
|
||||
每个元素格式为：
|
||||
{{"title":"策略标题（≤15字）","phase":"short|mid|long|diff","impact":1-10,"effort":1-10,"data_ref":"引用的核心数据点（如'猫在床上乱尿 15.5%'）"}}
|
||||
- phase 取值：short=短期快赢，mid=中期优化，long=长期布局，diff=差异化机会
|
||||
- impact=预期业务影响力（10=最大），effort=实施难度（10=最难）
|
||||
- data_ref 须引用报告中具体的簇名或占比数据
|
||||
示例：[{{"title":"升级缓释配方","phase":"short","impact":9,"effort":5,"data_ref":"效果仅持续数小时 6.8%"}}, ...]
|
||||
|
||||
{marker_report_html}
|
||||
（最后输出）完整 HTML 文档，结构遵循上文模版；勿用 Markdown 代码围栏包裹。
|
||||
|
|
|
|||
|
|
@ -1,8 +1,11 @@
|
|||
**输出格式（必须严格遵守）：**
|
||||
1. 只允许输出标准 HTML，不允许 Markdown。
|
||||
2. {marker_report_html} 内必须是完整 HTML 文档，包含 <html>、<head>、<body>；可保留模版中的 <!-- 映射注释 -->，浏览器不会显示。
|
||||
3. 允许标签：<h1><h2><h3><h4><p><ul><ol><li><strong>；列表项格式须为 <strong>纯中文簇名</strong> (XX.X%)：洞察。
|
||||
3. 允许标签：<h1><h2><h3><h4><p><ul><ol><li><strong>；列表项格式须为 <strong>纯中文簇名</strong> (XX.X%)：洞察。簇名必须控制在 15 字以内，用日常口语概括，禁止学术化长句。
|
||||
4. 禁止 Markdown（#、**、``` 等）及 <table> 等未列出的标签。
|
||||
4b. 【文风红线】洞察描述必须直白务实，每句只说一件事。禁止文学性夸张修辞（如"切肤之痛""灾难性崩塌""谋杀式""奇迹心理""令人愉悦"等）。保持客观商业语气。
|
||||
5. 【红线】正文可见文字中禁止出现 1_audience|2、2a_pain_audience_c2|0、stage=、audience_c 等任何机器标识；仅 CLUSTER_NAMES JSON 键可保留原始 id。
|
||||
6. 不要输出解释性前后缀；REPORT_HTML 段内只放 HTML 文档本体。
|
||||
7. 四段标记顺序固定：先三个 JSON 段（WORD_ZH → WORD_CATEGORY → CLUSTER_NAMES），**最后**输出 REPORT_HTML，避免长 HTML 导致 JSON 被截断。
|
||||
7. 五段标记顺序固定：先四个 JSON 段（WORD_ZH → WORD_CATEGORY → CLUSTER_NAMES → OPPORTUNITY_MATRIX），**最后**输出 REPORT_HTML，避免长 HTML 导致 JSON 被截断。
|
||||
8. **（三）多维交叉洞察**的三个子板块各至少包含 3 条 <li> 分析；每条 <li> 必须以 <strong>洞察标题</strong> 开头并引用数据占比。
|
||||
9. **三、改进建议与机会**的 <ol> 必须恰好 6 条 <li>，分别对应短期快赢（3条）、中期优化（1条）、长期布局（1条）、差异化机会（1条），每条以 <strong>策略标题：</strong> 开头。
|
||||
|
|
|
|||
|
|
@ -2,3 +2,17 @@
|
|||
报告必须专业、可读，正文中绝对禁止出现原始聚类 ID、stage 代码名或簇标签编号。
|
||||
展示用语统一：用户需求、产品反馈、产品客观描述；禁止观点、评价、痛点等旧称。
|
||||
你必须严格遵守 HTML 输出规范，禁止 Markdown。
|
||||
|
||||
【语言风格——红线要求】
|
||||
你的读者是业务团队和高管，不是学术论文审稿人。务必：
|
||||
- 用大白话写作，一句话只传达一个核心信息，避免多层从句嵌套。
|
||||
- 簇名和标题限制在 15 个中文字以内，用最日常的语言概括核心要点。
|
||||
正确示范：「猫在床/沙发上乱尿」「产品气味太刺鼻」「喷头漏液」「产品完全无效」
|
||||
错误示范：「针对床铺沙发等亲密区域的定点排尿抗议行为」「效力根本性不足导致乱尿行为完全无法被中断」
|
||||
- 描述要具体但不要堆砌修辞——说清"是什么、有多严重、影响谁"即可，删掉所有文学性修饰（如"切肤之痛""灾难性崩塌""谋杀式破坏"）。
|
||||
- 如果原始评论用了生动的英文原词，可以括号引用（如"stopped peeing"），但不要自行发明夸张措辞。
|
||||
|
||||
你的核心分析原则：
|
||||
1. **多维交叉验证**：不仅分析"用户说了什么"，还要推断其背后的使用场景、用户人群特征、购买动机，并交叉比对不同维度的数据，发现隐藏洞察。
|
||||
2. **机会多元化**：改进方案绝不局限于现有产品本身的修改——应覆盖服务升级、配件组合、场景化套装、全新品类延伸、内容与社群运营等多条路径。方案之间要体现差异化梯度（短期快赢 → 中期优化 → 长期布局）。
|
||||
3. **逻辑递进**：报告整体遵循「数据 → 洞察 → 机会 → 方案」的叙事线，每个结论都能回溯到具体数据支撑。
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
extraction:
|
||||
root_fields:
|
||||
- audience
|
||||
- persona_signals
|
||||
- pain_points
|
||||
- product_feedback
|
||||
product_feedback_fields:
|
||||
|
|
@ -11,7 +11,11 @@ extraction:
|
|||
- opinion
|
||||
- sentiment
|
||||
- category
|
||||
# 8 个标准类别（信任/原料/质量/功效/功能/外观/物流/客服/价格）；
|
||||
# 若评论反馈确实不属于这 8 类，允许模型自行补充简洁的英文新类别
|
||||
product_feedback_categories:
|
||||
- Trust
|
||||
- Ingredient
|
||||
- Quality
|
||||
- Function
|
||||
- Appearance
|
||||
|
|
@ -28,4 +32,5 @@ report:
|
|||
- "===WORD_ZH_JSON==="
|
||||
- "===WORD_CATEGORY_JSON==="
|
||||
- "===CLUSTER_NAMES_JSON==="
|
||||
- "===OPPORTUNITY_MATRIX_JSON==="
|
||||
- "===REPORT_HTML==="
|
||||
|
|
|
|||
497
voc_report.py
497
voc_report.py
|
|
@ -119,10 +119,10 @@ CATEGORY_COLORS: Dict[str, str] = {
|
|||
WORDCLOUD_UNCATEGORIZED_COLOR = "#9ca3af"
|
||||
|
||||
TRANSLATE_CHARS_PER_REQUEST = 200_000
|
||||
TRANSLATE_MAX_OUTPUT_TOKENS = 65536
|
||||
REPORT_MAX_OUTPUT_TOKENS = 65536
|
||||
TRANSLATE_MAX_OUTPUT_TOKENS = 200000
|
||||
REPORT_MAX_OUTPUT_TOKENS = 200000
|
||||
TRANSLATE_LLM_TIMEOUT_SEC = 600.0
|
||||
REPORT_LLM_TIMEOUT_SEC = 600.0
|
||||
REPORT_LLM_TIMEOUT_SEC = 720.0
|
||||
REPORT_PARSE_MAX_RETRIES = 2
|
||||
# JSON 段放前、HTML 放后：输出被 max_tokens 截断时优先保留可解析的 JSON
|
||||
REPORT_MARKERS: Tuple[str, ...] = (
|
||||
|
|
@ -132,9 +132,9 @@ REPORT_MARKERS: Tuple[str, ...] = (
|
|||
"===REPORT_HTML===",
|
||||
)
|
||||
WORD_CATEGORY_ASSIGN_BATCH_SIZE = 40
|
||||
WORD_CATEGORY_ASSIGN_MAX_TOKENS = 8192
|
||||
WORD_CATEGORY_ASSIGN_MAX_TOKENS = 200_000
|
||||
WORD_CATEGORY_ANALYSIS_MAX_WORDS = 25 # 生成 analysis 时每类最多展示的词条数
|
||||
WORD_CATEGORY_ANALYSIS_MAX_TOKENS = 4096
|
||||
WORD_CATEGORY_ANALYSIS_MAX_TOKENS = 200_000
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -169,7 +169,7 @@ def _call_llm_messages(
|
|||
*,
|
||||
model: str | None = None,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 16384,
|
||||
max_tokens: int = REPORT_MAX_OUTPUT_TOKENS,
|
||||
timeout: float = 300.0,
|
||||
reasoning_effort: str | None = None,
|
||||
extra_body: Dict[str, Any] | None = None,
|
||||
|
|
@ -233,7 +233,7 @@ def _call_llm(
|
|||
api_key: str,
|
||||
*,
|
||||
temperature: float = 0.3,
|
||||
max_tokens: int = 16384,
|
||||
max_tokens: int = REPORT_MAX_OUTPUT_TOKENS,
|
||||
timeout: float = 300.0,
|
||||
) -> str:
|
||||
return _call_llm_messages(
|
||||
|
|
@ -321,7 +321,7 @@ def _extraction_to_display_zh(ext: Dict[str, Any]) -> Dict[str, Any]:
|
|||
if not isinstance(pains, list):
|
||||
pains = []
|
||||
return {
|
||||
"受众": str(ext.get("audience", "")).strip(),
|
||||
"画像信号": [str(s).strip() for s in ext.get("persona_signals") or [] if str(s).strip()],
|
||||
"需求/痛点": [str(p).strip() for p in pains if str(p).strip()],
|
||||
"产品反馈": feedback,
|
||||
}
|
||||
|
|
@ -510,10 +510,16 @@ def build_cluster_bundles(
|
|||
stage_ratio = cnt / stage_total if stage_total else 0.0
|
||||
ratio = stage_ratio
|
||||
ratio_global = phrase_cnt / total_reviews if total_reviews else 0.0
|
||||
skip_filter = (
|
||||
stage == STAGE_1_AUDIENCE
|
||||
or stage.startswith(STAGE_2A_PAIN_PREFIX)
|
||||
or stage == STAGE_3A_PAIN
|
||||
)
|
||||
if (
|
||||
min_cluster_review_ratio is not None
|
||||
and stage_total > 0
|
||||
and stage_ratio < min_cluster_review_ratio
|
||||
and not skip_filter
|
||||
):
|
||||
logger.info(
|
||||
"报告跳过小簇 %s / %s：%s 条评论 (本步骤 %.1f%% < %.0f%%)",
|
||||
|
|
@ -977,10 +983,11 @@ def _build_report_prompt(
|
|||
word_freq: List[Tuple[str, int]],
|
||||
top2_audience_clusters: Sequence[int],
|
||||
) -> Tuple[str, str]:
|
||||
wf_table = word_freq[:WORD_FREQ_TABLE_N]
|
||||
wf_classify = word_freq[:WORD_CATEGORY_CLASSIFY_N]
|
||||
wf_table_lines = "\n".join(f"{w}\t{c}" for w, c in wf_table)
|
||||
wf_classify_lines = "\n".join(f"{w}\t{c}" for w, c in wf_classify)
|
||||
n_table = WORD_FREQ_TABLE_N
|
||||
n_classify = WORD_CATEGORY_CLASSIFY_N
|
||||
n_max = max(n_table, n_classify)
|
||||
wf_max = word_freq[:n_max]
|
||||
wf_lines = "\n".join(f"{w}\t{c}" for w, c in wf_max)
|
||||
cats_literal = "、".join(WORD_CATEGORIES)
|
||||
stage_map = _cluster_stage_mapping_guide(
|
||||
bundles, top2_audience_clusters=top2_audience_clusters
|
||||
|
|
@ -992,16 +999,23 @@ def _build_report_prompt(
|
|||
top2_audience_clusters=top2_audience_clusters,
|
||||
)
|
||||
system = build_report_system()
|
||||
if n_table == n_classify:
|
||||
wf_block = (
|
||||
f"【词频 Top{n_max}（word\\tcount，用于辅助撰写报告 + WORD_CATEGORY_JSON 分类，排名 1-{n_max}）】\n"
|
||||
f"{wf_lines}"
|
||||
)
|
||||
else:
|
||||
wf_block = (
|
||||
f"【词频 Top{n_max}（word\\tcount，排名 1-{n_max}）】\n"
|
||||
f"前 {n_table} 行用于辅助理解品类与撰写报告；前 {n_classify} 行**仅**用于 WORD_CATEGORY_JSON。\n"
|
||||
f"{wf_lines}"
|
||||
)
|
||||
user = f"""请基于以下用户评论反馈的聚类结果，撰写一份详细的《{product_name}》产品改进建议报告。
|
||||
|
||||
行业背景：{industry}
|
||||
数据清洗后有效评论总数 a = {total_reviews}
|
||||
|
||||
【词频 Top{WORD_FREQ_TABLE_N}（word\\tcount，辅助理解品类与撰写报告，排名 1-{WORD_FREQ_TABLE_N}）】
|
||||
{wf_table_lines}
|
||||
|
||||
【词频分类词表 Top{WORD_CATEGORY_CLASSIFY_N}（word\\tcount，**仅**用于 WORD_CATEGORY_JSON，排名 1-{WORD_CATEGORY_CLASSIFY_N}）】
|
||||
{wf_classify_lines}
|
||||
{wf_block}
|
||||
|
||||
【聚类流程与报告小节映射】
|
||||
{stage_map}
|
||||
|
|
@ -1878,6 +1892,92 @@ def _build_word_category_map(category_data: Dict[str, Any]) -> Dict[str, List[st
|
|||
return dict(by_word)
|
||||
|
||||
|
||||
def _audience_pie_data(bundles: List[ClusterBundle]) -> List[dict]:
|
||||
"""受众饼图：1_audience 阶段各簇的去重评论数。"""
|
||||
return [
|
||||
{"name": b.cluster_title_zh, "value": b.review_count}
|
||||
for b in bundles
|
||||
if b.stage == "1_audience"
|
||||
]
|
||||
|
||||
|
||||
def _sentiment_bar_data(bundles: List[ClusterBundle]) -> Dict[str, int]:
|
||||
"""情感分布柱状图：3b 阶段按正/负/中汇总结构化短语数。"""
|
||||
agg: Dict[str, int] = {"positive": 0, "negative": 0, "neutral": 0}
|
||||
for b in bundles:
|
||||
for key in agg:
|
||||
if b.stage == f"3b_aspect_opinion_{key}":
|
||||
agg[key] += b.phrase_count
|
||||
return agg
|
||||
|
||||
|
||||
def _pain_top_chart_data(
|
||||
bundles: List[ClusterBundle], *, top_n: int = 10
|
||||
) -> List[dict]:
|
||||
"""全量需求 Top-N 条形图：3a_pain_global 阶段各簇短语数。"""
|
||||
subset = [
|
||||
b for b in bundles
|
||||
if b.stage == STAGE_3A_PAIN and b.cluster_label != -1
|
||||
]
|
||||
subset.sort(key=lambda b: -b.phrase_count)
|
||||
return [
|
||||
{"name": b.cluster_title_zh, "value": b.phrase_count}
|
||||
for b in subset[:top_n]
|
||||
]
|
||||
|
||||
|
||||
def _negative_top_chart_data(
|
||||
bundles: List[ClusterBundle], *, top_n: int = 10
|
||||
) -> List[dict]:
|
||||
"""负面反馈 Top-N 条形图。"""
|
||||
subset = [
|
||||
b for b in bundles
|
||||
if b.stage == STAGE_3B_NEGATIVE and b.cluster_label != -1
|
||||
]
|
||||
subset.sort(key=lambda b: -b.phrase_count)
|
||||
return [
|
||||
{"name": b.cluster_title_zh, "value": b.phrase_count}
|
||||
for b in subset[:top_n]
|
||||
]
|
||||
|
||||
|
||||
def _radar_chart_data(
|
||||
category_data: Dict[str, Any],
|
||||
word_freq: List[Tuple[str, int]],
|
||||
) -> List[dict]:
|
||||
"""词频分类雷达图：各类别词频总和。"""
|
||||
count_map = {w.lower(): c for w, c in word_freq}
|
||||
result: List[dict] = []
|
||||
for cat in WORD_CATEGORIES:
|
||||
block = category_data.get(cat, {})
|
||||
words = block.get("words", []) if isinstance(block, dict) else []
|
||||
total = sum(count_map.get(str(w).lower(), 0) for w in words)
|
||||
result.append({"name": cat, "value": total})
|
||||
return result
|
||||
|
||||
|
||||
def _compute_dashboard_kpis(
|
||||
total_reviews: int,
|
||||
bundles: List[ClusterBundle],
|
||||
) -> Dict[str, Any]:
|
||||
"""Dashboard 四张 KPI 卡片的数据。"""
|
||||
audience_count = sum(
|
||||
1 for b in bundles
|
||||
if b.stage == "1_audience" and b.cluster_label != -1
|
||||
)
|
||||
sent = _sentiment_bar_data(bundles)
|
||||
total_sent = sum(sent.values()) or 1
|
||||
cluster_count = sum(1 for b in bundles if b.cluster_label != -1)
|
||||
return {
|
||||
"total_reviews": total_reviews,
|
||||
"audience_count": audience_count,
|
||||
"positive_ratio": round(sent["positive"] / total_sent * 100, 1),
|
||||
"negative_ratio": round(sent["negative"] / total_sent * 100, 1),
|
||||
"neutral_ratio": round(sent["neutral"] / total_sent * 100, 1),
|
||||
"cluster_count": cluster_count,
|
||||
}
|
||||
|
||||
|
||||
def _wordcloud_color_for_categories(categories: List[str]) -> str:
|
||||
if not categories:
|
||||
return WORDCLOUD_UNCATEGORIZED_COLOR
|
||||
|
|
@ -2163,7 +2263,7 @@ def _render_ai_validation_section(
|
|||
phrases_panel = _wrap_ai_verify_panel("聚类效果验证", phrases_html)
|
||||
return (
|
||||
'<section id="sec-ai-verify">\n'
|
||||
" <h2>AI分析效果验证</h2>\n"
|
||||
" <h2>模型中间过程可视化</h2>\n"
|
||||
' <p class="muted">以下为结构化与聚类结果的抽样展示，点击标题展开查看。</p>\n'
|
||||
f" {structured_panel}\n"
|
||||
f" {phrases_panel}\n"
|
||||
|
|
@ -2179,20 +2279,37 @@ def _assemble_html(
|
|||
phrases_html: str,
|
||||
structured_html: str,
|
||||
wordcloud_data: List[dict],
|
||||
audience_pie_data: List[dict],
|
||||
sentiment_data: Dict[str, int],
|
||||
pain_top_data: List[dict],
|
||||
negative_top_data: List[dict],
|
||||
radar_data: List[dict],
|
||||
kpis: Dict[str, Any],
|
||||
) -> str:
|
||||
"""按固定顺序机器拼装最终 HTML（不由 LLM 决定版块顺序）。"""
|
||||
data_json = json.dumps(wordcloud_data, ensure_ascii=False)
|
||||
"""按固定版块顺序拼装最终 HTML：Dashboard → 洞察报告 → 词频 → 附录。"""
|
||||
page_title = f"{product_name} · 评论分析报告"
|
||||
report_body = _strip_report_top_heading(report_html)
|
||||
report_section = (
|
||||
'<section id="sec-report">\n'
|
||||
" <h2>产品改进建议报告</h2>\n"
|
||||
f" {report_body}\n"
|
||||
"</section>"
|
||||
)
|
||||
ai_verify_section = _render_ai_validation_section(structured_html, phrases_html)
|
||||
# 顺序固定：改进建议 → 词频 → AI 分析效果验证（内含结构化 / 聚类，可展开）
|
||||
|
||||
wc_json = json.dumps(wordcloud_data, ensure_ascii=False)
|
||||
aud_json = json.dumps(audience_pie_data, ensure_ascii=False)
|
||||
sent_json = json.dumps(sentiment_data, ensure_ascii=False)
|
||||
pain_json = json.dumps(pain_top_data, ensure_ascii=False)
|
||||
neg_json = json.dumps(negative_top_data, ensure_ascii=False)
|
||||
radar_json = json.dumps(radar_data, ensure_ascii=False)
|
||||
|
||||
kpi_reviews = f"{kpis['total_reviews']:,}"
|
||||
kpi_audiences = kpis["audience_count"]
|
||||
kpi_pos = kpis["positive_ratio"]
|
||||
kpi_neg = kpis["negative_ratio"]
|
||||
|
||||
body_sections = "\n".join((report_section, wordfreq_html, ai_verify_section))
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
|
|
@ -2202,9 +2319,48 @@ def _assemble_html(
|
|||
<script src="https://cdn.jsdelivr.net/npm/echarts@5/dist/echarts.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/echarts-wordcloud@2/dist/echarts-wordcloud.min.js"></script>
|
||||
<style>
|
||||
:root {{ --c-primary: #2563eb; --c-pos: #16a34a; --c-neg: #dc2626; --c-neu: #6366f1;
|
||||
--c-bg: #f8fafc; --c-border: #e2e8f0; --radius: 12px; }}
|
||||
*, *::before, *::after {{ box-sizing: border-box; }}
|
||||
body {{ font-family: -apple-system, "PingFang SC", "Microsoft YaHei", sans-serif;
|
||||
margin: 0 auto; padding: 24px; line-height: 1.6; color: #222; max-width: 1100px; }}
|
||||
.muted {{ color: #666; font-size: 0.9em; }}
|
||||
margin: 0; padding: 0; line-height: 1.6; color: #1e293b; background: #fff; }}
|
||||
.muted {{ color: #64748b; font-size: 0.9em; }}
|
||||
|
||||
/* --- 导航栏 --- */
|
||||
.report-nav {{ position: sticky; top: 0; z-index: 100; background: rgba(255,255,255,0.92);
|
||||
backdrop-filter: blur(8px); border-bottom: 1px solid var(--c-border);
|
||||
display: flex; align-items: center; gap: 0; padding: 0 24px; overflow-x: auto; }}
|
||||
.report-nav a {{ padding: 14px 18px; font-size: 0.92em; font-weight: 500;
|
||||
color: #475569; text-decoration: none; white-space: nowrap;
|
||||
border-bottom: 2px solid transparent; transition: all 0.2s; }}
|
||||
.report-nav a:hover {{ color: var(--c-primary); }}
|
||||
.report-nav a.active {{ color: var(--c-primary); border-bottom-color: var(--c-primary); }}
|
||||
|
||||
.page-wrap {{ max-width: 1200px; margin: 0 auto; padding: 24px 28px 60px; }}
|
||||
.page-title {{ font-size: 1.6em; font-weight: 700; margin: 0 0 8px; color: #0f172a; }}
|
||||
.page-subtitle {{ color: #64748b; margin: 0 0 32px; font-size: 0.95em; }}
|
||||
|
||||
/* --- Dashboard --- */
|
||||
.kpi-grid {{ display: grid; grid-template-columns: repeat(4, 1fr); gap: 16px; margin-bottom: 28px; }}
|
||||
.kpi-card {{ background: var(--c-bg); border: 1px solid var(--c-border); border-radius: var(--radius);
|
||||
padding: 20px 18px; text-align: center; }}
|
||||
.kpi-value {{ font-size: 2em; font-weight: 700; color: #0f172a; line-height: 1.2; }}
|
||||
.kpi-label {{ font-size: 0.85em; color: #64748b; margin-top: 4px; }}
|
||||
.kpi-card.pos .kpi-value {{ color: var(--c-pos); }}
|
||||
.kpi-card.neg .kpi-value {{ color: var(--c-neg); }}
|
||||
.chart-grid {{ display: grid; grid-template-columns: 1fr 1fr; gap: 20px; margin-bottom: 32px; }}
|
||||
.chart-box {{ background: var(--c-bg); border: 1px solid var(--c-border);
|
||||
border-radius: var(--radius); padding: 16px; }}
|
||||
.chart-box h3 {{ margin: 0 0 8px; font-size: 1em; font-weight: 600; color: #334155; }}
|
||||
.chart-container {{ width: 100%; height: 360px; }}
|
||||
.chart-container.tall {{ height: 420px; }}
|
||||
|
||||
/* --- 各区块通用 --- */
|
||||
section {{ margin-top: 40px; }}
|
||||
section > h2 {{ font-size: 1.3em; border-bottom: 2px solid var(--c-primary);
|
||||
padding-bottom: 10px; color: #0f172a; }}
|
||||
|
||||
/* --- 词频 --- */
|
||||
#wordcloud-chart {{ width: 100%; height: 520px; margin: 16px 0 24px; }}
|
||||
.wc-legend {{ display: flex; flex-wrap: wrap; gap: 8px 14px; margin: 8px 0 12px;
|
||||
font-size: 0.88em; color: #444; }}
|
||||
|
|
@ -2213,27 +2369,15 @@ def _assemble_html(
|
|||
table {{ border-collapse: collapse; width: 100%; margin: 12px 0; }}
|
||||
th, td {{ border: 1px solid #ddd; padding: 8px; text-align: left; }}
|
||||
th {{ background: #f0f0f0; }}
|
||||
details {{ margin: 12px 0 24px; padding: 8px; background: #fafafa; border-radius: 6px; }}
|
||||
#sec-ai-verify {{ margin-top: 40px; }}
|
||||
details.ai-verify-panel {{ margin: 16px 0; padding: 0; border: 1px solid #e5e7eb;
|
||||
background: #fff; }}
|
||||
details.ai-verify-panel > summary {{ padding: 12px 16px; font-size: 1.05em; font-weight: 600;
|
||||
cursor: pointer; list-style: none; background: #f8fafc; border-radius: 6px 6px 0 0; }}
|
||||
details.ai-verify-panel > summary::-webkit-details-marker {{ display: none; }}
|
||||
details.ai-verify-panel > summary::before {{ content: "▸ "; color: #2563eb; }}
|
||||
details.ai-verify-panel[open] > summary::before {{ content: "▾ "; }}
|
||||
details.ai-verify-panel[open] > summary {{ border-bottom: 1px solid #e5e7eb; }}
|
||||
.ai-verify-body {{ padding: 12px 16px 16px; }}
|
||||
.stage-h {{ margin-top: 32px; border-bottom: 1px solid #eee; padding-bottom: 8px; }}
|
||||
.wf-categories {{ margin: 24px 0; }}
|
||||
.wf-cat {{ margin-bottom: 20px; }}
|
||||
.wf-pager {{ margin: 16px 0; }}
|
||||
.wf-tab-bar {{ margin-bottom: 8px; }}
|
||||
.wf-tab {{ margin: 4px 8px 4px 0; padding: 6px 12px; cursor: pointer;
|
||||
border: 1px solid #ccc; background: #fff; border-radius: 4px; }}
|
||||
.wf-tab.active {{ background: #2563eb; color: #fff; border-color: #2563eb; }}
|
||||
.wf-tab.active {{ background: var(--c-primary); color: #fff; border-color: var(--c-primary); }}
|
||||
.wf-nav {{ margin: 12px 0; display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }}
|
||||
.wf-table-wrap {{ max-height: 520px; overflow: auto; border: 1px solid #e5e7eb; border-radius: 8px; }}
|
||||
.wf-table-wrap {{ max-height: 520px; overflow: auto; border: 1px solid var(--c-border); border-radius: 8px; }}
|
||||
.wf-freq-table thead th {{ position: sticky; top: 0; z-index: 1; box-shadow: 0 1px 0 #ddd; }}
|
||||
.wf-freq-table tbody tr:nth-child(even) {{ background: #fafafa; }}
|
||||
.wf-freq-table tbody tr:hover {{ background: #f0f7ff; }}
|
||||
|
|
@ -2241,62 +2385,236 @@ def _assemble_html(
|
|||
.wf-row.wf-visible {{ display: table-row; }}
|
||||
.wf-count {{ min-width: 120px; }}
|
||||
.wf-count .wf-bar {{ display: inline-block; height: 10px; margin-right: 8px;
|
||||
background: linear-gradient(90deg, #93c5fd, #2563eb); border-radius: 2px; vertical-align: middle; }}
|
||||
background: linear-gradient(90deg, #93c5fd, var(--c-primary)); border-radius: 2px; vertical-align: middle; }}
|
||||
.wf-en {{ font-family: ui-monospace, monospace; font-size: 0.92em; }}
|
||||
#sec-wordfreq {{ margin-top: 40px; }}
|
||||
.struct-sample {{ margin: 24px 0; padding: 16px; border: 1px solid #e5e7eb;
|
||||
|
||||
/* --- 附录 --- */
|
||||
details {{ margin: 12px 0 24px; padding: 8px; background: #fafafa; border-radius: 6px; }}
|
||||
details.ai-verify-panel {{ margin: 16px 0; padding: 0; border: 1px solid var(--c-border);
|
||||
background: #fff; }}
|
||||
details.ai-verify-panel > summary {{ padding: 12px 16px; font-size: 1.05em; font-weight: 600;
|
||||
cursor: pointer; list-style: none; background: var(--c-bg); border-radius: 6px 6px 0 0; }}
|
||||
details.ai-verify-panel > summary::-webkit-details-marker {{ display: none; }}
|
||||
details.ai-verify-panel > summary::before {{ content: "\\25B8 "; color: var(--c-primary); }}
|
||||
details.ai-verify-panel[open] > summary::before {{ content: "\\25BE "; }}
|
||||
details.ai-verify-panel[open] > summary {{ border-bottom: 1px solid var(--c-border); }}
|
||||
.ai-verify-body {{ padding: 12px 16px 16px; }}
|
||||
.stage-h {{ margin-top: 32px; border-bottom: 1px solid #eee; padding-bottom: 8px; }}
|
||||
.struct-sample {{ margin: 24px 0; padding: 16px; border: 1px solid var(--c-border);
|
||||
border-radius: 8px; background: #fafafa; }}
|
||||
.struct-sample h4 {{ margin: 0 0 12px; }}
|
||||
.struct-review {{ white-space: pre-wrap; word-break: break-word; margin: 8px 0 16px; }}
|
||||
.struct-json {{ margin: 0; padding: 12px; background: #fff; border: 1px solid #e5e7eb;
|
||||
.struct-json {{ margin: 0; padding: 12px; background: #fff; border: 1px solid var(--c-border);
|
||||
border-radius: 6px; overflow-x: auto; font-size: 0.9em; line-height: 1.5; }}
|
||||
|
||||
@media (max-width: 768px) {{
|
||||
.kpi-grid {{ grid-template-columns: repeat(2, 1fr); }}
|
||||
.chart-grid {{ grid-template-columns: 1fr; }}
|
||||
.report-nav {{ padding: 0 12px; }}
|
||||
.report-nav a {{ padding: 12px 12px; font-size: 0.85em; }}
|
||||
.page-wrap {{ padding: 16px 14px 40px; }}
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>{page_title}</h1>
|
||||
|
||||
<nav class="report-nav" id="main-nav">
|
||||
<a href="#sec-dashboard">数据总览</a>
|
||||
<a href="#sec-report">用户洞察</a>
|
||||
<a href="#sec-wordfreq">词频分析</a>
|
||||
<a href="#sec-ai-verify">附录</a>
|
||||
</nav>
|
||||
|
||||
<div class="page-wrap">
|
||||
<h1 class="page-title">{page_title}</h1>
|
||||
<p class="page-subtitle">数据 \\u2192 洞察 \\u2192 机会 \\u2192 方案</p>
|
||||
|
||||
<!-- ========== 数据总览 ========== -->
|
||||
<section id="sec-dashboard">
|
||||
<h2>数据总览</h2>
|
||||
<div class="kpi-grid">
|
||||
<div class="kpi-card">
|
||||
<div class="kpi-value">{kpi_reviews}</div>
|
||||
<div class="kpi-label">有效评论数</div>
|
||||
</div>
|
||||
<div class="kpi-card">
|
||||
<div class="kpi-value">{kpi_audiences}</div>
|
||||
<div class="kpi-label">受众群体</div>
|
||||
</div>
|
||||
<div class="kpi-card pos">
|
||||
<div class="kpi-value">{kpi_pos}%</div>
|
||||
<div class="kpi-label">正面反馈占比</div>
|
||||
</div>
|
||||
<div class="kpi-card neg">
|
||||
<div class="kpi-value">{kpi_neg}%</div>
|
||||
<div class="kpi-label">负面反馈占比</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="chart-grid">
|
||||
<div class="chart-box">
|
||||
<h3>受众画像分布</h3>
|
||||
<div id="audience-chart" class="chart-container"></div>
|
||||
</div>
|
||||
<div class="chart-box">
|
||||
<h3>产品反馈情感分布</h3>
|
||||
<div id="sentiment-chart" class="chart-container"></div>
|
||||
</div>
|
||||
<div class="chart-box">
|
||||
<h3>用户需求 Top 10</h3>
|
||||
<div id="pain-top-chart" class="chart-container tall"></div>
|
||||
</div>
|
||||
<div class="chart-box">
|
||||
<h3>负面反馈 Top 10</h3>
|
||||
<div id="negative-top-chart" class="chart-container tall"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="chart-grid">
|
||||
<div class="chart-box">
|
||||
<h3>词频分类关注度</h3>
|
||||
<div id="radar-chart" class="chart-container"></div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ========== 用户洞察与行动方案 ========== -->
|
||||
{body_sections}
|
||||
<script>
|
||||
(function() {{
|
||||
const el = document.getElementById('wordcloud-chart');
|
||||
if (el && typeof echarts !== 'undefined') {{
|
||||
const chart = echarts.init(el);
|
||||
chart.setOption({{
|
||||
|
||||
</div><!-- .page-wrap -->
|
||||
|
||||
<script>
|
||||
(function() {{
|
||||
/* --- 工具函数 --- */
|
||||
var E = typeof echarts !== 'undefined' ? echarts : null;
|
||||
function initChart(id, opt) {{
|
||||
var el = document.getElementById(id);
|
||||
if (!el || !E) return null;
|
||||
var c = E.init(el);
|
||||
c.setOption(opt);
|
||||
window.addEventListener('resize', function() {{ c.resize(); }});
|
||||
return c;
|
||||
}}
|
||||
|
||||
/* --- 受众饼图 --- */
|
||||
var audData = {aud_json};
|
||||
if (audData.length) {{
|
||||
initChart('audience-chart', {{
|
||||
tooltip: {{ trigger: 'item', formatter: '{{b}}: {{c}} 条 ({{d}}%)' }},
|
||||
color: ['#3b82f6','#f59e0b','#10b981','#8b5cf6','#ef4444','#06b6d4','#ec4899','#84cc16'],
|
||||
series: [{{
|
||||
type: 'pie', radius: ['30%','70%'], roseType: 'radius',
|
||||
itemStyle: {{ borderRadius: 6, borderColor: '#fff', borderWidth: 2 }},
|
||||
label: {{ formatter: '{{b}}\\n{{d}}%' }},
|
||||
data: audData
|
||||
}}]
|
||||
}});
|
||||
}}
|
||||
|
||||
/* --- 情感柱状图 --- */
|
||||
var sentRaw = {sent_json};
|
||||
initChart('sentiment-chart', {{
|
||||
tooltip: {{ trigger: 'axis' }},
|
||||
xAxis: {{ type: 'category', data: ['\\u6b63\\u9762\\u53cd\\u9988','\\u8d1f\\u9762\\u53cd\\u9988','\\u5ba2\\u89c2\\u63cf\\u8ff0'],
|
||||
axisLabel: {{ fontSize: 13 }} }},
|
||||
yAxis: {{ type: 'value', name: '\\u7ed3\\u6784\\u5316\\u77ed\\u8bed\\u6570' }},
|
||||
series: [{{
|
||||
type: 'bar', barWidth: '45%',
|
||||
data: [
|
||||
{{ value: sentRaw.positive || 0, itemStyle: {{ color: '#16a34a' }} }},
|
||||
{{ value: sentRaw.negative || 0, itemStyle: {{ color: '#dc2626' }} }},
|
||||
{{ value: sentRaw.neutral || 0, itemStyle: {{ color: '#6366f1' }} }}
|
||||
],
|
||||
label: {{ show: true, position: 'top', fontWeight: 'bold' }}
|
||||
}}]
|
||||
}});
|
||||
|
||||
/* --- 需求 Top-N 横向 Bar --- */
|
||||
var painData = {pain_json};
|
||||
if (painData.length) {{
|
||||
initChart('pain-top-chart', {{
|
||||
tooltip: {{ trigger: 'axis', axisPointer: {{ type: 'shadow' }} }},
|
||||
grid: {{ left: '35%', right: '8%', top: 10, bottom: 20 }},
|
||||
yAxis: {{ type: 'category', data: painData.map(function(d){{ return d.name; }}).reverse(),
|
||||
axisLabel: {{ width: 200, overflow: 'truncate', fontSize: 12 }} }},
|
||||
xAxis: {{ type: 'value', name: '\\u77ed\\u8bed\\u6570' }},
|
||||
series: [{{ type: 'bar', data: painData.map(function(d){{ return d.value; }}).reverse(),
|
||||
itemStyle: {{ color: '#3b82f6', borderRadius: [0,4,4,0] }},
|
||||
label: {{ show: true, position: 'right' }} }}]
|
||||
}});
|
||||
}}
|
||||
|
||||
/* --- 负面反馈 Top-N 横向 Bar --- */
|
||||
var negData = {neg_json};
|
||||
if (negData.length) {{
|
||||
initChart('negative-top-chart', {{
|
||||
tooltip: {{ trigger: 'axis', axisPointer: {{ type: 'shadow' }} }},
|
||||
grid: {{ left: '35%', right: '8%', top: 10, bottom: 20 }},
|
||||
yAxis: {{ type: 'category', data: negData.map(function(d){{ return d.name; }}).reverse(),
|
||||
axisLabel: {{ width: 200, overflow: 'truncate', fontSize: 12 }} }},
|
||||
xAxis: {{ type: 'value', name: '\\u77ed\\u8bed\\u6570' }},
|
||||
series: [{{ type: 'bar', data: negData.map(function(d){{ return d.value; }}).reverse(),
|
||||
itemStyle: {{ color: '#dc2626', borderRadius: [0,4,4,0] }},
|
||||
label: {{ show: true, position: 'right' }} }}]
|
||||
}});
|
||||
}}
|
||||
|
||||
/* --- 雷达图 --- */
|
||||
var radarData = {radar_json};
|
||||
if (radarData.length) {{
|
||||
var maxVal = Math.max.apply(null, radarData.map(function(d){{ return d.value; }})) || 1;
|
||||
initChart('radar-chart', {{
|
||||
tooltip: {{}},
|
||||
radar: {{
|
||||
indicator: radarData.map(function(d) {{
|
||||
return {{ name: d.name, max: Math.ceil(maxVal * 1.2) }};
|
||||
}}),
|
||||
shape: 'circle',
|
||||
splitArea: {{ areaStyle: {{ color: ['rgba(37,99,235,0.02)','rgba(37,99,235,0.05)'] }} }}
|
||||
}},
|
||||
series: [{{
|
||||
type: 'radar',
|
||||
data: [{{ value: radarData.map(function(d){{ return d.value; }}),
|
||||
areaStyle: {{ color: 'rgba(37,99,235,0.15)' }},
|
||||
lineStyle: {{ color: '#3b82f6', width: 2 }},
|
||||
itemStyle: {{ color: '#3b82f6' }} }}]
|
||||
}}]
|
||||
}});
|
||||
}}
|
||||
|
||||
/* --- 词云 --- */
|
||||
var wcData = {wc_json};
|
||||
if (wcData.length) {{
|
||||
initChart('wordcloud-chart', {{
|
||||
tooltip: {{
|
||||
show: true,
|
||||
formatter: function(p) {{
|
||||
var d = p.data || {{}};
|
||||
var rank = d.rank ? ('#' + d.rank + ' ') : '';
|
||||
var cnt = d.count != null ? d.count : p.value;
|
||||
var zh = d.zh && d.zh !== d.name ? ('<br/>中文: ' + d.zh) : '';
|
||||
var zh = d.zh && d.zh !== d.name ? ('<br/>\\u4e2d\\u6587: ' + d.zh) : '';
|
||||
var cat = (d.categories && d.categories.length)
|
||||
? ('<br/>分类: ' + d.categories.join('、'))
|
||||
: (d.category ? ('<br/>分类: ' + d.category) : '<br/>分类: 未分类');
|
||||
return rank + (d.name || p.name) + zh + cat + '<br/>次数: ' + cnt;
|
||||
? ('<br/>\\u5206\\u7c7b: ' + d.categories.join('\\u3001'))
|
||||
: (d.category ? ('<br/>\\u5206\\u7c7b: ' + d.category) : '<br/>\\u5206\\u7c7b: \\u672a\\u5206\\u7c7b');
|
||||
return rank + (d.name || p.name) + zh + cat + '<br/>\\u6b21\\u6570: ' + cnt;
|
||||
}}
|
||||
}},
|
||||
series: [{{
|
||||
type: 'wordCloud',
|
||||
shape: 'circle',
|
||||
width: '95%',
|
||||
height: '95%',
|
||||
type: 'wordCloud', shape: 'circle', width: '95%', height: '95%',
|
||||
sizeRange: [{WORDCLOUD_SIZE_MIN}, {WORDCLOUD_SIZE_MAX}],
|
||||
rotationRange: [-15, 15],
|
||||
rotationStep: 15,
|
||||
gridSize: 8,
|
||||
drawOutOfBound: false,
|
||||
layoutAnimation: true,
|
||||
rotationRange: [-15, 15], rotationStep: 15, gridSize: 8,
|
||||
drawOutOfBound: false, layoutAnimation: true,
|
||||
textStyle: {{ fontFamily: 'system-ui, -apple-system, sans-serif' }},
|
||||
emphasis: {{ focus: 'self', textStyle: {{ shadowBlur: 6, shadowColor: '#333' }} }},
|
||||
data: {data_json}
|
||||
data: wcData
|
||||
}}]
|
||||
}});
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}}
|
||||
const rows = Array.from(document.querySelectorAll('.wf-row'));
|
||||
const tabs = Array.from(document.querySelectorAll('.wf-tab'));
|
||||
const label = document.getElementById('wf-page-label');
|
||||
const pageCount = tabs.length || 1;
|
||||
let cur = 1;
|
||||
|
||||
/* --- 词频分页 --- */
|
||||
var rows = Array.from(document.querySelectorAll('.wf-row'));
|
||||
var tabs = Array.from(document.querySelectorAll('.wf-tab'));
|
||||
var label = document.getElementById('wf-page-label');
|
||||
var pageCount = tabs.length || 1;
|
||||
var cur = 1;
|
||||
function showPage(n) {{
|
||||
cur = Math.max(1, Math.min(n, pageCount));
|
||||
rows.forEach(function(r) {{
|
||||
|
|
@ -2305,20 +2623,35 @@ def _assemble_html(
|
|||
tabs.forEach(function(t) {{
|
||||
t.classList.toggle('active', parseInt(t.dataset.page, 10) === cur);
|
||||
}});
|
||||
if (label) label.textContent = '第 ' + cur + ' / ' + pageCount + ' 页';
|
||||
if (label) label.textContent = '\\u7b2c ' + cur + ' / ' + pageCount + ' \\u9875';
|
||||
}}
|
||||
tabs.forEach(function(t) {{
|
||||
t.addEventListener('click', function() {{
|
||||
showPage(parseInt(t.dataset.page, 10));
|
||||
}});
|
||||
t.addEventListener('click', function() {{ showPage(parseInt(t.dataset.page, 10)); }});
|
||||
}});
|
||||
var prev = document.getElementById('wf-prev');
|
||||
var next = document.getElementById('wf-next');
|
||||
if (prev) prev.addEventListener('click', function() {{ showPage(cur - 1); }});
|
||||
if (next) next.addEventListener('click', function() {{ showPage(cur + 1); }});
|
||||
showPage(1);
|
||||
}})();
|
||||
</script>
|
||||
|
||||
/* --- 导航高亮 --- */
|
||||
var navLinks = document.querySelectorAll('.report-nav a');
|
||||
var sectionIds = Array.from(navLinks).map(function(a) {{ return a.getAttribute('href').slice(1); }});
|
||||
function updateNav() {{
|
||||
var scrollY = window.scrollY + 80;
|
||||
var active = sectionIds[0];
|
||||
sectionIds.forEach(function(id) {{
|
||||
var el = document.getElementById(id);
|
||||
if (el && el.offsetTop <= scrollY) active = id;
|
||||
}});
|
||||
navLinks.forEach(function(a) {{
|
||||
a.classList.toggle('active', a.getAttribute('href') === '#' + active);
|
||||
}});
|
||||
}}
|
||||
window.addEventListener('scroll', updateNav);
|
||||
updateNav();
|
||||
}})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
|
@ -2416,6 +2749,14 @@ def generate_report(
|
|||
cleaned_csv=cleaned_csv,
|
||||
)
|
||||
structured_sec = _render_structured_validation_section(struct_samples)
|
||||
|
||||
aud_pie = _audience_pie_data(bundles)
|
||||
sent_bar = _sentiment_bar_data(bundles)
|
||||
pain_top = _pain_top_chart_data(bundles, top_n=10)
|
||||
neg_top = _negative_top_chart_data(bundles, top_n=10)
|
||||
radar = _radar_chart_data(category_data, word_freq)
|
||||
kpis = _compute_dashboard_kpis(total_reviews, bundles)
|
||||
|
||||
html = _assemble_html(
|
||||
product_name=product_name,
|
||||
wordfreq_html=wordfreq_sec,
|
||||
|
|
@ -2423,6 +2764,12 @@ def generate_report(
|
|||
phrases_html=phrases_sec,
|
||||
structured_html=structured_sec,
|
||||
wordcloud_data=wc_data,
|
||||
audience_pie_data=aud_pie,
|
||||
sentiment_data=sent_bar,
|
||||
pain_top_data=pain_top,
|
||||
negative_top_data=neg_top,
|
||||
radar_data=radar,
|
||||
kpis=kpis,
|
||||
)
|
||||
output_html.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_html.write_text(html, encoding="utf-8")
|
||||
|
|
|
|||
797
voc_业务_2/build_report.py
Normal file
797
voc_业务_2/build_report.py
Normal file
|
|
@ -0,0 +1,797 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
步骤 2：主报告构建脚本。读取数据库 → LLM 分析 → 渲染 HTML。
|
||||
|
||||
用法::
|
||||
|
||||
# 全流程（config.yaml 留空则 LLM 自动识别产品名和行业）
|
||||
../310py/bin/python build_report.py
|
||||
|
||||
# 指定产品名（覆盖 config.yaml），自定义输出路径
|
||||
../310py/bin/python build_report.py --product "Bikini Trimmer" --output output/bikini-trimmer.html
|
||||
|
||||
# 跳过 LLM 调用（仅渲染模板骨架，用于验证模板和数据管道）
|
||||
../310py/bin/python build_report.py --no-llm
|
||||
|
||||
# 调试：将 LLM 分析结果保存为 JSON（方便人工检查/修正后重新渲染）
|
||||
../310py/bin/python build_report.py --save-data
|
||||
|
||||
依赖：需先运行 run_pipeline.py 产出 SQLite 数据库。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import yaml
|
||||
|
||||
from data_loader import DataLoader, asin_link_html, build_amazon_url, MarketStats
|
||||
from llm_analyzer import (
|
||||
discover_personas, discover_themes_both, analyze_kano_jtbd_keywords_parallel,
|
||||
analyze_matrix, analyze_all_rootcauses, get_llm_workers, configure_report_llm,
|
||||
)
|
||||
from report_utils import (
|
||||
enrich_theme_keywords, recalc_neg_priorities, compute_persona_pcts,
|
||||
assign_persona_clusters, validate_persona_physiological_labels,
|
||||
prepare_persona_catalog_for_llm,
|
||||
pick_persona_quotes, fix_jtbd_fields, filter_matrix_rows,
|
||||
enrich_matrix_scene_evidence, build_matrix_table_rows_html,
|
||||
enrich_rootcause_quotes, build_executive_summary, enhanced_market_judgment,
|
||||
fix_kano_items, normalize_kano_reverse_items, normalize_persona_dimension,
|
||||
normalize_rootcauses,
|
||||
build_asin_labels, build_asin_short_codes, build_asin_tables_html,
|
||||
build_footer_asin_links, get_layout_config,
|
||||
persona_display_meta,
|
||||
build_asin_theme_insights, asin_link_with_label, quote_cn_summary,
|
||||
sort_personas_by_evidence, sort_rootcauses_by_evidence, jtbd_cell_html,
|
||||
build_product_stopwords, build_kano_grid_html, build_neg_theme_summary_note,
|
||||
build_neg_theme_table_rows, build_pos_theme_table_rows,
|
||||
build_sentiment_keyword_groups, prepare_keyword_display, build_keyword_tables_html,
|
||||
build_quote_blocks_html,
|
||||
)
|
||||
from echarts_builder import (
|
||||
get_echarts_script, build_all_charts, calc_theme_freq, calc_per_asin_theme_freq,
|
||||
get_chart_layout, build_theme_mini_charts_html,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("voc.build_report")
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
|
||||
_PLACEHOLDER_RE = re.compile(r"\{\{[A-Z0-9_]+\}\}")
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = SCRIPT_DIR.parent
|
||||
CONFIG_FILE = SCRIPT_DIR / "config.yaml"
|
||||
TEMPLATE_FILE = SCRIPT_DIR / "template.html"
|
||||
|
||||
|
||||
def product_file_slug(product: str, industry: str = "") -> str:
|
||||
"""产品名 → 安全文件名 slug（仅产品名，不含行业/副标题）。"""
|
||||
s = (product or "voc-report").strip()
|
||||
if industry:
|
||||
ind = industry.strip()
|
||||
if ind and ind in s:
|
||||
for pat in (
|
||||
f"（{ind}）", f"({ind})", f" - {ind}", f" · {ind}",
|
||||
f"｜{ind}", f"|{ind}", f"/ {ind}",
|
||||
):
|
||||
if pat in s:
|
||||
s = s.split(pat, 1)[0].strip()
|
||||
if s.endswith(ind):
|
||||
s = s[: -len(ind)].strip(" -/|·（）()")
|
||||
# 「产品A / 产品B」类双描述只取主产品名（第一段）
|
||||
for sep in ("/", "｜", "|"):
|
||||
if sep in s:
|
||||
s = s.split(sep, 1)[0].strip()
|
||||
break
|
||||
s = s.lower()
|
||||
s = re.sub(r'[/\\:*?"<>|]+', "-", s)
|
||||
s = re.sub(r"\s+", "-", s)
|
||||
s = re.sub(r"-+", "-", s).strip("-")
|
||||
return s or "voc-report"
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
with CONFIG_FILE.open(encoding="utf-8") as f:
|
||||
return yaml.safe_load(f) or {}
|
||||
|
||||
|
||||
def finalize_template(html: str) -> str:
|
||||
"""检查并清理未替换的 {{PLACEHOLDER}}，避免报告页面露出模板标签。"""
|
||||
remaining = sorted(set(_PLACEHOLDER_RE.findall(html)))
|
||||
if remaining:
|
||||
logger.error(
|
||||
"报告模板存在未替换占位符 (%s 个): %s — 请更新 render_html 或重新渲染",
|
||||
len(remaining), ", ".join(remaining[:12]),
|
||||
)
|
||||
html = _PLACEHOLDER_RE.sub("", html)
|
||||
return html
|
||||
|
||||
|
||||
def build_report_data(loader: DataLoader, cfg: dict) -> Dict[str, Any]:
|
||||
"""构建所有报告数据。"""
|
||||
configure_report_llm(cfg)
|
||||
data: Dict[str, Any] = {}
|
||||
llm_workers = get_llm_workers(cfg)
|
||||
product = cfg.get("product_name", "主产品")
|
||||
industry = cfg.get("industry", "当前品类")
|
||||
logger.info("LLM 并发 workers=%s | 报告模式: max thinking", llm_workers)
|
||||
|
||||
# ── 基础统计 ──
|
||||
stats = loader.load_basic_stats()
|
||||
data["stats"] = stats
|
||||
|
||||
# ── 聚类数据 ──
|
||||
persona_sample_reviews = int(cfg.get("persona_sample_reviews", 5))
|
||||
cluster_data = loader.build_cluster_prompt_data(
|
||||
persona_sample_reviews=persona_sample_reviews,
|
||||
)
|
||||
data["cluster_data"] = cluster_data
|
||||
all_clusters = loader.load_cluster_data()
|
||||
reviews = loader.load_reviews()
|
||||
extractions = loader.load_comment_extractions()
|
||||
data["extractions"] = extractions
|
||||
persona_quote_max = int(cfg.get("persona_quote_max", 3))
|
||||
rootcause_quote_max = int(cfg.get("rootcause_quote_max", 4))
|
||||
physio_min = int(cfg.get("persona_physio_min_reviews", 5))
|
||||
min_hit_count = int(cfg.get("persona_min_hit_count", 5))
|
||||
data["persona_min_hit_count"] = min_hit_count
|
||||
prepare_persona_catalog_for_llm(
|
||||
cluster_data["persona_cluster_catalog"], all_clusters, reviews,
|
||||
min_physio_reviews=physio_min,
|
||||
)
|
||||
|
||||
# ── Persona ──
|
||||
personas = discover_personas(cluster_data, product_name=product, industry=industry)
|
||||
personas = assign_persona_clusters(personas, all_clusters)
|
||||
personas = validate_persona_physiological_labels(
|
||||
personas, cluster_data, reviews, all_clusters, min_review_count=physio_min,
|
||||
)
|
||||
data["personas"] = personas
|
||||
|
||||
# ── 主题（差评+好评并发） ──
|
||||
neg_themes, pos_themes = discover_themes_both(
|
||||
cluster_data, stats.neg_review_count, stats.pos_review_count,
|
||||
)
|
||||
neg_themes = enrich_theme_keywords(neg_themes, cluster_data.get("global_negative", []))
|
||||
pos_themes = enrich_theme_keywords(pos_themes, cluster_data.get("global_positive", []))
|
||||
data["neg_themes"] = neg_themes
|
||||
data["pos_themes"] = pos_themes
|
||||
|
||||
# ── 主题频次统计（结构化 category+aspect 优先） ──
|
||||
neg_freq = calc_theme_freq(neg_themes, reviews, is_neg=True, extractions=extractions)
|
||||
pos_freq = calc_theme_freq(pos_themes, reviews, is_neg=False, extractions=extractions)
|
||||
per_asin_neg = calc_per_asin_theme_freq(
|
||||
neg_themes, reviews, is_neg=True, extractions=extractions,
|
||||
)
|
||||
per_asin_pos = calc_per_asin_theme_freq(
|
||||
pos_themes, reviews, is_neg=False, extractions=extractions,
|
||||
)
|
||||
neg_themes = recalc_neg_priorities(
|
||||
neg_themes, neg_freq, stats.neg_review_count, len(stats.asins), per_asin_neg,
|
||||
)
|
||||
data["neg_themes"] = neg_themes
|
||||
data["neg_freq"] = neg_freq
|
||||
data["pos_freq"] = pos_freq
|
||||
data["per_asin_neg"] = per_asin_neg
|
||||
data["per_asin_pos"] = per_asin_pos
|
||||
|
||||
# ── Persona 统计与引用 ──
|
||||
personas = compute_persona_pcts(personas, reviews, all_clusters)
|
||||
personas = sort_personas_by_evidence(personas)
|
||||
data["personas"] = personas
|
||||
data["persona_quotes"] = pick_persona_quotes(
|
||||
personas, reviews, extractions,
|
||||
max_quotes=persona_quote_max,
|
||||
)
|
||||
|
||||
# ── KANO + JTBD + 情感关键词（三者并发） ──
|
||||
neg_kw_groups = build_sentiment_keyword_groups(neg_themes, reviews, limit=6, is_neg=True)
|
||||
pos_kw_groups = build_sentiment_keyword_groups(pos_themes, reviews, limit=6, is_neg=False)
|
||||
kano_raw, jtbd_raw, keywords_raw = analyze_kano_jtbd_keywords_parallel(
|
||||
neg_themes, pos_themes, personas,
|
||||
neg_kw_groups, pos_kw_groups,
|
||||
product_name=product,
|
||||
)
|
||||
kano = normalize_kano_reverse_items(fix_kano_items(kano_raw))
|
||||
data["kano"] = kano
|
||||
jtbd = fix_jtbd_fields(jtbd_raw, personas)
|
||||
data["jtbd"] = jtbd
|
||||
data["keywords_llm"] = keywords_raw
|
||||
_, _, keywords_neg, keywords_pos = prepare_keyword_display(
|
||||
neg_themes, pos_themes, reviews, personas, keywords_raw, limit=6,
|
||||
)
|
||||
data["keywords_neg"] = keywords_neg
|
||||
data["keywords_pos"] = keywords_pos
|
||||
data["keywords"] = keywords_neg + keywords_pos
|
||||
|
||||
# ── 矩阵（依赖 KANO） ──
|
||||
matrix = enrich_matrix_scene_evidence(
|
||||
analyze_matrix(
|
||||
personas, kano, cluster_data, stats.total_reviews,
|
||||
market_avg=stats.weighted_avg_rating,
|
||||
),
|
||||
personas,
|
||||
reviews,
|
||||
)
|
||||
matrix = filter_matrix_rows(
|
||||
matrix,
|
||||
market_avg=stats.weighted_avg_rating,
|
||||
)
|
||||
data["matrix"] = matrix
|
||||
|
||||
# ── 根因分析（各 Persona 并发） ──
|
||||
rootcauses = analyze_all_rootcauses(
|
||||
personas, neg_themes, all_clusters=all_clusters,
|
||||
max_workers=llm_workers,
|
||||
product_name=product, industry=industry, min_hit_count=min_hit_count,
|
||||
)
|
||||
rootcauses = enrich_rootcause_quotes(
|
||||
rootcauses, personas, loader, reviews,
|
||||
persona_quotes=data["persona_quotes"],
|
||||
extractions=extractions,
|
||||
max_quotes=rootcause_quote_max,
|
||||
)
|
||||
rootcauses = normalize_rootcauses(rootcauses, neg_themes)
|
||||
rootcauses = sort_rootcauses_by_evidence(rootcauses, personas)
|
||||
data["rootcauses"] = rootcauses
|
||||
|
||||
# ── 市场竞争 + 决策摘要 + ASIN 标签 ──
|
||||
asin_labels = build_asin_labels(stats)
|
||||
data["asin_labels"] = asin_labels
|
||||
data["asin_short_codes"] = build_asin_short_codes(stats)
|
||||
market_title, market_desc = enhanced_market_judgment(stats)
|
||||
data["market_title"] = market_title
|
||||
data["market_desc"] = market_desc
|
||||
data["executive_summary"] = build_executive_summary(
|
||||
stats, neg_freq, pos_freq, neg_themes, asin_labels=asin_labels,
|
||||
pos_themes=pos_themes,
|
||||
)
|
||||
data["asin_theme_insights"] = build_asin_theme_insights(
|
||||
stats, neg_themes, per_asin_neg, asin_labels,
|
||||
)
|
||||
|
||||
return data
|
||||
|
||||
|
||||
def render_html(template_path: Path, data: Dict[str, Any], cfg: dict) -> str:
|
||||
"""用数据填充模板并返回完整 HTML。"""
|
||||
if not template_path.is_file():
|
||||
raise FileNotFoundError(f"模板文件不存在: {template_path}")
|
||||
html = template_path.read_text(encoding="utf-8")
|
||||
|
||||
# ── 简单占位符 ──
|
||||
product = cfg.get("product_name", "Product")
|
||||
analysis_date = cfg.get("analysis_date", "2026-06-12")
|
||||
data_source = cfg.get("data_source", "卖家精灵 realtime CSV")
|
||||
version = cfg.get("report_version", "v1")
|
||||
|
||||
html = html.replace("{{PRODUCT_NAME}}", product)
|
||||
html = html.replace("{{ANALYSIS_DATE}}", analysis_date)
|
||||
html = html.replace("{{DATA_SOURCE}}", data_source)
|
||||
html = html.replace("{{VERSION}}", version)
|
||||
|
||||
# ── ECharts ──
|
||||
echarts_inline = cfg.get("echarts_inline", True)
|
||||
html = html.replace("{{ECHARTS_SCRIPT}}", get_echarts_script(inline=echarts_inline))
|
||||
|
||||
# ── 导航 ASIN 列表 ──
|
||||
stats: MarketStats = data["stats"]
|
||||
asins = [a.asin for a in stats.asins]
|
||||
html = html.replace("{{ASIN_COUNT}}", str(len(asins)))
|
||||
|
||||
# ── KPI ──
|
||||
html = html.replace("{{TOTAL_REVIEWS}}", f"{stats.total_reviews:,}")
|
||||
html = html.replace("{{WEIGHTED_AVG}}", f"{stats.weighted_avg_rating}")
|
||||
html = html.replace("{{POS_RATE}}", f"{int(stats.pos_rate * 100)}%")
|
||||
html = html.replace("{{NEG_RATE}}", f"{int(stats.neg_rate * 100)}%")
|
||||
neutral_count = stats.total_reviews - stats.neg_review_count - stats.pos_review_count
|
||||
neutral_pct = round(neutral_count / max(stats.total_reviews, 1) * 100)
|
||||
html = html.replace("{{NEUTRAL_RATE}}", f"{neutral_pct}%")
|
||||
html = html.replace("{{NEUTRAL_COUNT}}", str(neutral_count))
|
||||
|
||||
# KPI 颜色
|
||||
avg_color = "danger" if stats.weighted_avg_rating < 3.5 else ("warn" if stats.weighted_avg_rating <= 4.0 else "success")
|
||||
html = html.replace("{{AVG_COLOR}}", avg_color)
|
||||
|
||||
# ── 市场竞争 callout ──
|
||||
callout_type = "danger" if stats.weighted_avg_rating < 3.5 else ("warn" if stats.weighted_avg_rating <= 4.0 else "success")
|
||||
html = html.replace("{{MARKET_CALLOUT_TYPE}}", callout_type)
|
||||
html = html.replace("{{MARKET_TITLE}}", data["market_title"])
|
||||
html = html.replace("{{MARKET_DESC}}", data["market_desc"])
|
||||
|
||||
# ── 决策摘要 ──
|
||||
es = data.get("executive_summary") or {}
|
||||
bullets = es.get("bullets", [])
|
||||
opps = es.get("opportunities", [])
|
||||
conclusion = es.get("conclusion", "")
|
||||
insights = es.get("insights", [])
|
||||
summary_html = ""
|
||||
if conclusion:
|
||||
summary_html += (
|
||||
f'<div class="callout callout-success" style="margin-bottom:10px;padding:10px 14px">'
|
||||
f'<div class="callout-title">产品定义结论</div>'
|
||||
f'<p style="font-size:13px;margin:0">{conclusion}</p></div>'
|
||||
)
|
||||
summary_html += '<ul style="margin:8px 0 0 18px;font-size:13px;color:#444">'
|
||||
for b in bullets:
|
||||
summary_html += f"<li style=\"margin-bottom:6px\">{b}</li>"
|
||||
summary_html += "</ul>"
|
||||
if opps:
|
||||
summary_html += '<div style="margin-top:12px;font-size:12px;font-weight:600;color:#374151">产品机会清单（按差评频次排序）</div><ol style="margin:6px 0 0 18px;font-size:13px;color:#444">'
|
||||
for o in opps:
|
||||
summary_html += f"<li style=\"margin-bottom:4px\">{o}</li>"
|
||||
summary_html += "</ol>"
|
||||
html = html.replace("{{EXEC_SUMMARY_HTML}}", summary_html)
|
||||
|
||||
if insights:
|
||||
ins_html = '<div class="callout callout-warn" style="margin-bottom:18px"><div class="callout-title">关键不对称洞察</div><ul style="margin:6px 0 0 18px;font-size:13px;color:#444">'
|
||||
for ins in insights:
|
||||
ins_html += f"<li style=\"margin-bottom:4px\">{ins}</li>"
|
||||
ins_html += "</ul></div>"
|
||||
else:
|
||||
ins_html = ""
|
||||
html = html.replace("{{INSIGHTS_CALLOUT}}", ins_html)
|
||||
|
||||
asin_theme_ins = data.get("asin_theme_insights") or []
|
||||
if asin_theme_ins:
|
||||
ath = '<div class="callout callout-info" style="margin-bottom:14px"><div class="callout-title">ASIN 主题对比结论</div><ul style="margin:6px 0 0 18px;font-size:13px;color:#444">'
|
||||
for line in asin_theme_ins:
|
||||
ath += f"<li style=\"margin-bottom:4px\">{line}</li>"
|
||||
ath += "</ul></div>"
|
||||
else:
|
||||
ath = ""
|
||||
html = html.replace("{{ASIN_THEME_INSIGHTS_HTML}}", ath)
|
||||
|
||||
asin_labels = data.get("asin_labels") or build_asin_labels(stats)
|
||||
asin_short = data.get("asin_short_codes") or build_asin_short_codes(stats)
|
||||
layout_cfg = get_layout_config(cfg)
|
||||
chart_layout = get_chart_layout(stats, cfg)
|
||||
|
||||
# ── ASIN 表格（大品类摘要 + 附录）──
|
||||
summary_table, appendix_table, table_note = build_asin_tables_html(
|
||||
stats, asin_labels, asin_link_html,
|
||||
top_n=layout_cfg["asin_table_top_n"],
|
||||
large_threshold=layout_cfg["large_asin_threshold"],
|
||||
)
|
||||
html = html.replace("{{ASIN_TABLE_NOTE}}", table_note)
|
||||
html = html.replace("{{ASIN_TABLE_SUMMARY}}", summary_table)
|
||||
html = html.replace("{{ASIN_TABLE_APPENDIX}}", appendix_table)
|
||||
|
||||
# ── 评分分布图布局 ──
|
||||
if chart_layout["large_market"]:
|
||||
star_note = (
|
||||
f"共 {chart_layout['asin_count']} 个竞品：横向堆叠图按评论量排序，"
|
||||
f"可在下方滚动查看全部；轴标签为短码（A/B/…）。"
|
||||
)
|
||||
star_scroll_max = min(720, chart_layout["star_chart_height"])
|
||||
else:
|
||||
star_note = "按竞品展示 5★–1★ 评论堆叠分布。"
|
||||
star_scroll_max = chart_layout["star_chart_height"]
|
||||
html = html.replace("{{STAR_CHART_NOTE}}", star_note)
|
||||
html = html.replace("{{STAR_CHART_HEIGHT}}", str(chart_layout["star_chart_height"]))
|
||||
html = html.replace("{{STAR_SCROLL_MAX}}", str(star_scroll_max))
|
||||
if chart_layout.get("show_star_summary"):
|
||||
html = html.replace(
|
||||
"{{STAR_SUMMARY_HTML}}",
|
||||
'<div style="margin-bottom:14px"><h3 style="margin-bottom:6px">评分分布摘要（Top 15 + 其余聚合）</h3>'
|
||||
f'<div id="starDistSummaryChart" style="width:100%;height:{chart_layout["star_summary_height"]}px"></div></div>',
|
||||
)
|
||||
else:
|
||||
html = html.replace("{{STAR_SUMMARY_HTML}}", "")
|
||||
|
||||
neg_top6 = sorted(data["neg_freq"].items(), key=lambda x: x[1], reverse=True)[:6]
|
||||
pos_top6 = sorted(data["pos_freq"].items(), key=lambda x: x[1], reverse=True)[:6]
|
||||
neg_theme_names = [n for n, _ in neg_top6]
|
||||
pos_theme_names = [n for n, _ in pos_top6]
|
||||
hm_th = layout_cfg["heatmap_asin_threshold"]
|
||||
if chart_layout["use_heatmap"]:
|
||||
neg_theme_note = f"热力图：行=主题、列=竞品短码；颜色越深命中越多。下方为各主题 Top ASIN 明细。竞品 >{hm_th}，请用滑块横向浏览。"
|
||||
pos_theme_note = neg_theme_note.replace("差评", "好评")
|
||||
elif chart_layout["large_market"]:
|
||||
neg_theme_note = "分组柱：X 轴=竞品短码，图例=6 个差评主题；竞品较多时请拖动下方滑块。"
|
||||
pos_theme_note = "分组柱：X 轴=竞品短码，图例=6 个好评主题；竞品较多时请拖动下方滑块。"
|
||||
else:
|
||||
neg_theme_note = "分组柱：X 轴=竞品，图例=Top 6 差评主题。"
|
||||
pos_theme_note = "分组柱：X 轴=竞品，图例=Top 6 好评主题。"
|
||||
html = html.replace("{{NEG_THEME_CHART_NOTE}}", neg_theme_note)
|
||||
html = html.replace("{{POS_THEME_CHART_NOTE}}", pos_theme_note)
|
||||
html = html.replace("{{NEG_THEME_CHART_HEIGHT}}", str(chart_layout["neg_theme_height"]))
|
||||
html = html.replace("{{POS_THEME_CHART_HEIGHT}}", str(chart_layout["pos_theme_height"]))
|
||||
html = html.replace(
|
||||
"{{NEG_THEME_MINI_HTML}}",
|
||||
build_theme_mini_charts_html("neg", neg_theme_names, chart_layout["asin_count"], large_threshold=hm_th),
|
||||
)
|
||||
html = html.replace(
|
||||
"{{POS_THEME_MINI_HTML}}",
|
||||
build_theme_mini_charts_html("pos", pos_theme_names, chart_layout["asin_count"], large_threshold=hm_th),
|
||||
)
|
||||
|
||||
# 移除旧占位符兼容
|
||||
html = html.replace("{{ASIN_TABLE_ROWS}}", "")
|
||||
|
||||
# ── Persona 卡片 ──
|
||||
persona_map = {p.get("name"): p for p in data.get("personas", [])}
|
||||
persona_cards = []
|
||||
for i, p in enumerate(data["personas"]):
|
||||
pq = next((q for q in data["persona_quotes"] if q["persona"] == p["name"]), None)
|
||||
quote_items = (pq.get("quotes") or []) if pq else []
|
||||
if not quote_items and pq and pq.get("quote"):
|
||||
quote_items = [pq]
|
||||
quote_html = build_quote_blocks_html(
|
||||
quote_items,
|
||||
asin_labels,
|
||||
empty_msg="暂无命中池内的代表性评论",
|
||||
)
|
||||
normalize_persona_dimension(p)
|
||||
min_hit = int(
|
||||
data.get("persona_min_hit_count")
|
||||
or cfg.get("persona_min_hit_count", 5)
|
||||
)
|
||||
meta = persona_display_meta(p, stats.total_reviews, min_hit_count=min_hit)
|
||||
persona_cards.append(f"""<div class="persona">
|
||||
<div class="p-name">{p.get("name", "?")}</div>
|
||||
<div class="p-meta">{meta}</div>
|
||||
<div class="p-row"><span class="p-label">核心痛点：</span>{p.get("core_pain", "")}</div>
|
||||
<div class="p-row"><span class="p-label">核心需求：</span>{p.get("core_need", "")}</div>
|
||||
<div class="p-row"><span class="p-label">购买动机：</span>{p.get("purchase_motivation", "")}</div>
|
||||
{quote_html}
|
||||
</div>""")
|
||||
html = html.replace("{{PERSONA_CARDS}}", "\n".join(persona_cards))
|
||||
|
||||
# ── 差评主题表格 ──
|
||||
neg_total = stats.neg_review_count
|
||||
neg_rows_html = build_neg_theme_table_rows(
|
||||
data["neg_freq"],
|
||||
data["neg_themes"],
|
||||
neg_total,
|
||||
len(stats.asins),
|
||||
data["per_asin_neg"],
|
||||
asin_labels,
|
||||
)
|
||||
html = html.replace("{{NEG_THEME_TABLE_ROWS}}", neg_rows_html)
|
||||
|
||||
neg_summary_note = build_neg_theme_summary_note(
|
||||
stats, data["neg_freq"], data["neg_themes"], data.get("per_asin_neg"),
|
||||
)
|
||||
html = html.replace("{{NEG_THEME_SUMMARY_NOTE}}", neg_summary_note)
|
||||
|
||||
# ── 好评主题表格 ──
|
||||
pos_total = stats.pos_review_count
|
||||
pos_rows_html = build_pos_theme_table_rows(
|
||||
data["pos_freq"],
|
||||
data.get("pos_themes") or [],
|
||||
pos_total,
|
||||
data["neg_themes"],
|
||||
data["neg_freq"],
|
||||
neg_total,
|
||||
data.get("kano") or [],
|
||||
)
|
||||
html = html.replace("{{POS_THEME_TABLE_ROWS}}", pos_rows_html)
|
||||
html = html.replace("{{POS_REVIEW_COUNT}}", str(pos_total))
|
||||
pos_pct_sum = sum(
|
||||
round(c / max(pos_total, 1) * 100) for c in data["pos_freq"].values() if c > 0
|
||||
)
|
||||
html = html.replace("{{POS_PCT_SUM}}", str(pos_pct_sum))
|
||||
|
||||
# ── KANO 四象限卡片 ──
|
||||
kano_display = normalize_kano_reverse_items(data.get("kano") or [])
|
||||
html = html.replace("{{KANO_GRID_HTML}}", build_kano_grid_html(kano_display))
|
||||
html = html.replace("{{KANO_TABLE_ROWS}}", "")
|
||||
|
||||
# ── JTBD 表格 ──
|
||||
jtbd_rows = []
|
||||
jtbd_items = fix_jtbd_fields(data.get("jtbd") or [], data.get("personas") or [])
|
||||
for item in jtbd_items:
|
||||
jtbd_rows.append(
|
||||
f'<tr><td>{item.get("persona", "?")}</td>'
|
||||
f'<td class="jtbd-cell">{jtbd_cell_html(item.get("core_job", ""))}</td>'
|
||||
f'<td class="jtbd-cell">{jtbd_cell_html(item.get("functional_motivation", ""))}</td>'
|
||||
f'<td class="jtbd-cell">{jtbd_cell_html(item.get("emotional_motivation", ""))}</td>'
|
||||
f'<td class="jtbd-cell">{jtbd_cell_html(item.get("social_motivation", ""))}</td>'
|
||||
f'<td class="jtbd-cell">{jtbd_cell_html(item.get("trigger", ""))}</td></tr>'
|
||||
)
|
||||
html = html.replace("{{JTBD_TABLE_ROWS}}", "\n".join(jtbd_rows))
|
||||
|
||||
# ── 矩阵（Persona 分组卡片）──
|
||||
matrix_html = build_matrix_table_rows_html(data["matrix"], persona_map)
|
||||
html = html.replace("{{MATRIX_TABLE_ROWS}}", matrix_html)
|
||||
html = html.replace("{{MATRIX_HTML}}", "")
|
||||
|
||||
# ── 根因卡片 ──
|
||||
rc_cards = []
|
||||
rc_idx = 0
|
||||
for i, rc in enumerate(data["rootcauses"]):
|
||||
pname = rc.get("persona_name", f"Persona {i}")
|
||||
if rc.get("skipped"):
|
||||
rc_cards.append(f"""<div class="card collapsed">
|
||||
<div class="card-header" onclick="toggleCard(this)">
|
||||
<span>👤 {pname} — 核心痛点根因</span>
|
||||
<span><span class="pill pill-gray">样本不足</span> <span class="toggle-icon"></span></span>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<p class="rc-text">{rc.get("skip_reason", "聚类命中不足，未生成根因分析")}</p>
|
||||
</div>
|
||||
</div>""")
|
||||
continue
|
||||
causes = rc.get("root_causes", [])
|
||||
affected = ", ".join(rc.get("affected_themes", []))
|
||||
rc_html_parts = []
|
||||
for cause in causes:
|
||||
quotes_html = ""
|
||||
for q in cause.get("quotes", []):
|
||||
asin = q.get("asin", "?")
|
||||
if asin == "?" or not re.match(r"^B[A-Z0-9]{9}$", asin):
|
||||
continue
|
||||
valid_asins = {a.asin for a in stats.asins}
|
||||
if asin not in valid_asins:
|
||||
continue
|
||||
amazon_url = build_amazon_url(asin)
|
||||
cn = q.get("cn_summary") or quote_cn_summary(q.get("text", ""))
|
||||
alabel = asin_labels.get(asin, asin)
|
||||
quotes_html += (
|
||||
f'<div class="quote neg">"{q.get("text", "")}" '
|
||||
f'— <a href="{amazon_url}" target="_blank" rel="noopener">{alabel}</a>'
|
||||
f'<div class="quote-cn">摘要：{cn}</div></div>\n'
|
||||
)
|
||||
rc_html_parts.append(f"""<div class="rc-section">
|
||||
<div class="rc-label">{cause.get("title", "根因")}</div>
|
||||
<p class="rc-text">{cause.get("mechanism", "")}</p>
|
||||
{quotes_html}
|
||||
<div class="rc-label" style="margin-top:4px">→ 产品开发方向</div>
|
||||
<p class="rc-text">{cause.get("dev_direction", "")}</p>
|
||||
</div>""")
|
||||
|
||||
collapsed = " collapsed" if rc_idx >= 2 else ""
|
||||
rc_idx += 1
|
||||
rc_cards.append(f"""<div class="card{collapsed}">
|
||||
<div class="card-header" onclick="toggleCard(this)">
|
||||
<span>👤 {pname} — 核心痛点根因</span>
|
||||
<span><span class="pill pill-warn">{affected}</span> <span class="toggle-icon"></span></span>
|
||||
</div>
|
||||
<div class="card-body">
|
||||
<div class="rc-section">
|
||||
<div class="rc-label">差评主题归因（该群体命中）</div>
|
||||
<p class="rc-text">{affected}</p>
|
||||
</div>
|
||||
{"".join(rc_html_parts)}
|
||||
</div>
|
||||
</div>""")
|
||||
html = html.replace("{{ROOTCAUSE_CARDS}}", "\n".join(rc_cards))
|
||||
|
||||
# ── 情感关键词双表 ──
|
||||
neg_kw = data.get("keywords_neg")
|
||||
pos_kw = data.get("keywords_pos")
|
||||
if neg_kw is None or pos_kw is None:
|
||||
neg_kw, pos_kw = [], []
|
||||
neg_kw_rows, pos_kw_rows = build_keyword_tables_html(neg_kw or [], pos_kw or [])
|
||||
html = html.replace("{{KEYWORD_NEG_TABLE_ROWS}}", neg_kw_rows)
|
||||
html = html.replace("{{KEYWORD_POS_TABLE_ROWS}}", pos_kw_rows)
|
||||
|
||||
# ── 图表 JS ──
|
||||
charts_js = build_all_charts(
|
||||
stats, data["neg_themes"], data["pos_themes"],
|
||||
data["neg_freq"], data["pos_freq"],
|
||||
data["per_asin_neg"], data["per_asin_pos"],
|
||||
asin_labels=asin_labels,
|
||||
asin_short_codes=asin_short,
|
||||
layout=chart_layout,
|
||||
cfg=cfg,
|
||||
)
|
||||
html = html.replace("{{CHART_JS}}", charts_js)
|
||||
|
||||
html = html.replace(
|
||||
"{{FOOTER_ASIN_LINKS}}",
|
||||
build_footer_asin_links(stats, asin_labels),
|
||||
)
|
||||
|
||||
return finalize_template(html)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="构建 VOC 分析报告 HTML")
|
||||
parser.add_argument("--product", help="产品名（覆盖 config.yaml）")
|
||||
parser.add_argument("--industry", help="行业名")
|
||||
parser.add_argument("--output", default=None, help="输出 HTML 路径")
|
||||
parser.add_argument("--no-llm", action="store_true", help="跳过 LLM 调用（仅渲染模板，用于测试）")
|
||||
parser.add_argument("--save-data", action="store_true", help="将 LLM 分析结果保存为 JSON")
|
||||
parser.add_argument("--render-from-json", help="从已保存的 JSON 渲染 HTML（跳过 LLM）")
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg = load_config()
|
||||
if args.product:
|
||||
cfg["product_name"] = args.product
|
||||
if args.industry:
|
||||
cfg["industry"] = args.industry
|
||||
|
||||
product = cfg.get("product_name", "").strip()
|
||||
industry = cfg.get("industry", "").strip()
|
||||
|
||||
# 如果产品名或行业为空，用 LLM 从原始评论中自动识别
|
||||
need_detect = (not product or product == "亚马逊商品" or not industry or industry == "亚马逊电商")
|
||||
if need_detect:
|
||||
input_dir_raw = cfg.get("input_dir", "")
|
||||
input_dir_path_raw = (SCRIPT_DIR / input_dir_raw).resolve() if input_dir_raw else None
|
||||
if input_dir_path_raw and input_dir_path_raw.is_dir():
|
||||
samples, dir_name = DataLoader.load_raw_review_samples(input_dir_path_raw, max_samples=50)
|
||||
if samples:
|
||||
from llm_analyzer import detect_product_and_industry
|
||||
detected_product, detected_industry = detect_product_and_industry(samples, dir_name)
|
||||
if not product or product == "亚马逊商品":
|
||||
product = detected_product
|
||||
cfg["product_name"] = product
|
||||
logger.info("LLM 自动识别产品名: %s", product)
|
||||
if not industry or industry == "亚马逊电商":
|
||||
industry = detected_industry
|
||||
cfg["industry"] = industry
|
||||
logger.info("LLM 自动识别行业: %s", industry)
|
||||
else:
|
||||
product = product or "亚马逊商品"
|
||||
industry = industry or "亚马逊电商"
|
||||
else:
|
||||
product = product or "亚马逊商品"
|
||||
industry = industry or "亚马逊电商"
|
||||
output_dir = SCRIPT_DIR / cfg.get("output_dir", "./output")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
slug = product_file_slug(product, industry)
|
||||
output_html = Path(args.output) if args.output else (output_dir / f"{slug}.html")
|
||||
|
||||
logger.info("=== VOC 报告构建 ===")
|
||||
logger.info("产品: %s | 行业: %s", product, industry)
|
||||
logger.info("项目根目录: %s", PROJECT_ROOT)
|
||||
|
||||
# 加载数据
|
||||
loader = DataLoader(PROJECT_ROOT, product, industry)
|
||||
|
||||
if args.render_from_json:
|
||||
json_path = Path(args.render_from_json)
|
||||
if not json_path.is_file():
|
||||
slug = product_file_slug(product, industry)
|
||||
json_path = output_dir / f"{slug}-analysis-data.json"
|
||||
logger.info("从 JSON 渲染: %s", json_path)
|
||||
with json_path.open(encoding="utf-8") as f:
|
||||
raw = json.load(f)
|
||||
stats = loader.load_basic_stats()
|
||||
reviews = loader.load_reviews()
|
||||
extractions = loader.load_comment_extractions()
|
||||
persona_quote_max = int(cfg.get("persona_quote_max", 3))
|
||||
rootcause_quote_max = int(cfg.get("rootcause_quote_max", 4))
|
||||
all_clusters = loader.load_cluster_data()
|
||||
personas = assign_persona_clusters(raw.get("personas", []), all_clusters)
|
||||
cluster_data_rr = raw.get("cluster_data") or loader.build_cluster_prompt_data(
|
||||
persona_sample_reviews=int(cfg.get("persona_sample_reviews", 5)),
|
||||
)
|
||||
physio_min = int(cfg.get("persona_physio_min_reviews", 5))
|
||||
prepare_persona_catalog_for_llm(
|
||||
cluster_data_rr.get("persona_cluster_catalog") or [],
|
||||
all_clusters, reviews,
|
||||
min_physio_reviews=physio_min,
|
||||
)
|
||||
personas = validate_persona_physiological_labels(
|
||||
personas, cluster_data_rr, reviews, all_clusters, min_review_count=physio_min,
|
||||
)
|
||||
personas = compute_persona_pcts(personas, reviews, all_clusters)
|
||||
personas = sort_personas_by_evidence(personas)
|
||||
persona_quotes = pick_persona_quotes(
|
||||
personas, reviews, extractions,
|
||||
max_quotes=persona_quote_max,
|
||||
)
|
||||
neg_freq = calc_theme_freq(
|
||||
raw.get("neg_themes", []), reviews, is_neg=True, extractions=extractions,
|
||||
)
|
||||
pos_freq = calc_theme_freq(
|
||||
raw.get("pos_themes", []), reviews, is_neg=False, extractions=extractions,
|
||||
)
|
||||
per_asin_neg = calc_per_asin_theme_freq(
|
||||
raw.get("neg_themes", []), reviews, is_neg=True, extractions=extractions,
|
||||
)
|
||||
per_asin_pos = calc_per_asin_theme_freq(
|
||||
raw.get("pos_themes", []), reviews, is_neg=False, extractions=extractions,
|
||||
)
|
||||
neg_themes = recalc_neg_priorities(
|
||||
raw.get("neg_themes", []), neg_freq, stats.neg_review_count, len(stats.asins), per_asin_neg,
|
||||
)
|
||||
asin_labels = build_asin_labels(stats)
|
||||
data = {
|
||||
**raw,
|
||||
"stats": stats,
|
||||
"personas": personas,
|
||||
"persona_quotes": persona_quotes,
|
||||
"neg_themes": neg_themes,
|
||||
"neg_freq": neg_freq,
|
||||
"pos_freq": pos_freq,
|
||||
"per_asin_neg": per_asin_neg,
|
||||
"per_asin_pos": per_asin_pos,
|
||||
"asin_labels": asin_labels,
|
||||
"asin_short_codes": build_asin_short_codes(stats),
|
||||
"executive_summary": build_executive_summary(
|
||||
stats, neg_freq, pos_freq, neg_themes, asin_labels=asin_labels,
|
||||
pos_themes=raw.get("pos_themes", []),
|
||||
),
|
||||
"asin_theme_insights": build_asin_theme_insights(
|
||||
stats, neg_themes, per_asin_neg, asin_labels,
|
||||
),
|
||||
}
|
||||
if not data.get("keywords_neg") and data.get("neg_themes"):
|
||||
_, _, kw_neg, kw_pos = prepare_keyword_display(
|
||||
data["neg_themes"],
|
||||
data.get("pos_themes") or [],
|
||||
reviews,
|
||||
personas,
|
||||
data.get("keywords_llm") or {},
|
||||
limit=6,
|
||||
)
|
||||
data["keywords_neg"] = kw_neg
|
||||
data["keywords_pos"] = kw_pos
|
||||
data["keywords"] = kw_neg + kw_pos
|
||||
market_title, market_desc = enhanced_market_judgment(stats)
|
||||
data["market_title"] = market_title
|
||||
data["market_desc"] = market_desc
|
||||
data["rootcauses"] = enrich_rootcause_quotes(
|
||||
data.get("rootcauses") or [],
|
||||
personas,
|
||||
loader,
|
||||
reviews,
|
||||
persona_quotes=persona_quotes,
|
||||
extractions=extractions,
|
||||
max_quotes=rootcause_quote_max,
|
||||
)
|
||||
elif args.no_llm:
|
||||
logger.warning("--no-llm 模式：跳过 LLM 调用，仅渲染模板")
|
||||
# 使用空数据渲染（测试模板）
|
||||
data = {
|
||||
"stats": loader.load_basic_stats(),
|
||||
"personas": [],
|
||||
"neg_themes": [], "pos_themes": [],
|
||||
"neg_freq": {}, "pos_freq": {},
|
||||
"per_asin_neg": {}, "per_asin_pos": {},
|
||||
"kano": [], "jtbd": [], "matrix": [], "rootcauses": [],
|
||||
"keywords": [], "keywords_neg": [], "keywords_pos": [], "keywords_llm": {},
|
||||
"persona_quotes": [],
|
||||
"asin_labels": build_asin_labels(loader.load_basic_stats()),
|
||||
"asin_short_codes": build_asin_short_codes(loader.load_basic_stats()),
|
||||
"asin_theme_insights": [],
|
||||
"market_title": "数据待生成", "market_desc": "请运行完整流程",
|
||||
"executive_summary": {
|
||||
"bullets": ["请运行完整流程生成分析"],
|
||||
"opportunities": [],
|
||||
"conclusion": "",
|
||||
"insights": [],
|
||||
},
|
||||
}
|
||||
else:
|
||||
data = build_report_data(loader, cfg)
|
||||
|
||||
# 保存中间数据（调试用）
|
||||
if args.save_data:
|
||||
data_json = output_dir / f"{slug}-analysis-data.json"
|
||||
with data_json.open("w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2, default=str)
|
||||
logger.info("分析数据已保存: %s", data_json)
|
||||
|
||||
# 渲染 HTML
|
||||
logger.info("渲染 HTML...")
|
||||
html_content = render_html(TEMPLATE_FILE, data, cfg)
|
||||
|
||||
output_html.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_html.write_text(html_content, encoding="utf-8")
|
||||
logger.info("报告已生成: %s (%s KB)", output_html, len(html_content) // 1024)
|
||||
|
||||
# 如果 ECharts 内联模式且缓存存在，报告大小
|
||||
if cfg.get("echarts_inline", True):
|
||||
from echarts_builder import CACHE_FILE
|
||||
if CACHE_FILE.is_file():
|
||||
logger.info("ECharts 已内联（缓存: %s KB）", CACHE_FILE.stat().st_size // 1024)
|
||||
|
||||
print(f"\n✅ 报告已生成: {output_html}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
68
voc_业务_2/config.yaml
Normal file
68
voc_业务_2/config.yaml
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# VOC 业务分析配置
|
||||
# 每次分析新品类时，修改以下字段即可
|
||||
|
||||
# ── 产品基础信息 ──
|
||||
# product_name 和 industry 留空则由 LLM 自动识别
|
||||
product_name: ""
|
||||
industry: ""
|
||||
|
||||
# ── 数据路径 ──
|
||||
# input_dir: 原始评论 CSV 目录（相对于 voc_业务_2 目录，也支持绝对路径）
|
||||
input_dir: "../Bikini trimmer voc"
|
||||
|
||||
# ── 父目录脚本路径 ──
|
||||
# main_script: main_voc分析.py 的路径（相对于 voc_业务_2 目录）
|
||||
main_script: "../main_voc分析.py"
|
||||
# python_bin: 310py Python 解释器路径
|
||||
python_bin: "../310py/bin/python"
|
||||
|
||||
# ── 报告元数据 ──
|
||||
analysis_date: "2026-06-12"
|
||||
data_source: "卖家精灵 realtime CSV"
|
||||
|
||||
# ── 报告版本 ──
|
||||
report_version: "v1"
|
||||
|
||||
# ── LLM 提示词（业务员可编辑，见 报告LLM提示词.md）──
|
||||
prompts_file: "./prompts.yaml"
|
||||
|
||||
# Persona 发现：每个聚类簇注入 LLM 的代表性评论条数（溯源原文）
|
||||
persona_sample_reviews: 5
|
||||
# Persona 生理标签：绑定簇内至少 N 条评论含对应英文词方可保留（如 pregnant≥5 才可用「孕妇」）
|
||||
persona_physio_min_reviews: 5
|
||||
# Persona 命中门槛：hit_count 低于此值跳过根因 LLM，卡片置信度显示「低样本」
|
||||
persona_min_hit_count: 5
|
||||
# Persona 卡片展示的代表性评论条数（结构化字段匹配 + 原文回退）
|
||||
persona_quote_max: 3
|
||||
# 每条根因展示的代表性评论条数
|
||||
rootcause_quote_max: 4
|
||||
# 进入 step 2 细粒度聚类的 audience 簇最小占比（相对总评论数）
|
||||
audience_coverage_threshold: 0.05
|
||||
|
||||
|
||||
# ── LLM 配置 ──
|
||||
# 复用根目录 voc_llm.py 的配置（DEEPSEEK_API_KEY 环境变量 或 .deepseek_key）
|
||||
# model: LLM 模型名称（留空使用 voc_llm.py 默认值 deepseek-v4-pro）
|
||||
model: ""
|
||||
# report_model: 报告主 LLM（留空使用 deepseek-v4-pro + thinking max）
|
||||
report_model: "deepseek-v4-pro"
|
||||
report_reasoning_effort: "max"
|
||||
# 报告 LLM 最大输出 token（模型上限 384K；建议 64K–128K，根因/KANO 多段 JSON 时适当增大）
|
||||
report_max_tokens: 280000
|
||||
# LLM 并发线程数（根因等多 Persona 任务并行；遇 API 限流可调低至 2）
|
||||
llm_max_workers: 10
|
||||
|
||||
# ── 大品类报告布局（ASIN 数量自适应）──
|
||||
large_asin_threshold: 12 # 超过此数量启用摘要表/横向评分图/主题图 dataZoom
|
||||
heatmap_asin_threshold: 30 # 超过此数量主题对比改用热力图 + small multiples
|
||||
asin_table_top_n: 10 # 摘要表展示 Top N
|
||||
star_summary_top_n: 15 # 评分分布摘要图 Top N + 其余聚合
|
||||
theme_mini_top_n: 8 # 每主题 small multiple 展示 Top N ASIN
|
||||
|
||||
# ── 输出配置 ──
|
||||
output_dir: "./output"
|
||||
# ECharts 是否内联到 HTML（true=下载并嵌入，false=使用 CDN，文件更小）
|
||||
echarts_inline: false
|
||||
# ECharts CDN 地址（echarts_inline=false 时使用，内联模式下也用于下载）
|
||||
echarts_cdn: "https://cdn.jsdelivr.net/npm/echarts@5.5.0/dist/echarts.min.js"
|
||||
|
||||
548
voc_业务_2/data_loader.py
Normal file
548
voc_业务_2/data_loader.py
Normal file
|
|
@ -0,0 +1,548 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
数据加载模块：从 SQLite 数据库和 CSV 中提取结构化数据。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import sqlite3
|
||||
from collections import Counter, defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple
|
||||
|
||||
logger = logging.getLogger("voc.data_loader")
|
||||
|
||||
STRUCTURED_DB_NAME = "voc_structured.sqlite"
|
||||
CLUSTER_DB_NAME = "voc_clustering.sqlite"
|
||||
EMBED_DB_NAME = "voc_embeddings.sqlite"
|
||||
CLEANED_CSV_NAME = "merged_reviews_cleaned.csv"
|
||||
WORD_FREQ_CSV_NAME = "output/word_freq.csv"
|
||||
TERMS_JSON_NAME = "output/voc_terms.json"
|
||||
|
||||
# 与 report_utils._PERSONA_PHRASE_STOPWORDS 保持一致（避免循环 import）
|
||||
_PERSONA_PHRASE_STOPWORDS = frozenset({
|
||||
"self", "unknown", "user", "customer", "buyer", "myself", "the user",
|
||||
"a user", "consumer", "reviewer", "amazon", "product", "item",
|
||||
})
|
||||
|
||||
|
||||
def _phrase_segments_local(phrases: List[str]) -> List[str]:
|
||||
"""将聚类短语拆成可匹配的英文片段。"""
|
||||
segs: List[str] = []
|
||||
for p in phrases:
|
||||
p = (p or "").lower().strip()
|
||||
if not p:
|
||||
continue
|
||||
segs.append(p)
|
||||
for part in p.split(","):
|
||||
part = part.strip()
|
||||
if len(part) >= 3:
|
||||
segs.append(part)
|
||||
return list(dict.fromkeys(segs))
|
||||
|
||||
|
||||
def _cluster_semantic_label(
|
||||
cluster: "ClusterData",
|
||||
phrase_doc_freq: Dict[str, int],
|
||||
n_clusters_in_stage: int,
|
||||
) -> str:
|
||||
"""从 top_phrases 生成簇短语义标签（频次 × IDF，同类 stage 内去重）。"""
|
||||
scored: List[Tuple[float, str]] = []
|
||||
for rank, phrase in enumerate(cluster.top_phrases):
|
||||
tf = 1.0 / (1 + rank)
|
||||
for seg in _phrase_segments_local([phrase]):
|
||||
if seg in _PERSONA_PHRASE_STOPWORDS or len(seg) < 4:
|
||||
continue
|
||||
df = phrase_doc_freq.get(seg, 1)
|
||||
idf = math.log((n_clusters_in_stage + 1) / (df + 0.5))
|
||||
scored.append((tf * idf, seg))
|
||||
if scored:
|
||||
scored.sort(key=lambda x: (-x[0], -len(x[1])))
|
||||
return scored[0][1][:48]
|
||||
for phrase in cluster.top_phrases:
|
||||
p = (phrase or "").strip()
|
||||
if len(p) >= 4:
|
||||
return p[:48]
|
||||
return f"cluster_{cluster.cluster_label}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewRecord:
|
||||
asin: str
|
||||
rating: float
|
||||
title: str
|
||||
content: str
|
||||
verified: bool
|
||||
vine: bool
|
||||
review_date: str
|
||||
source_row: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class ASINStats:
|
||||
asin: str
|
||||
total: int
|
||||
avg_rating: float
|
||||
pos_rate: float
|
||||
neg_rate: float
|
||||
star_dist: Dict[int, int] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClusterData:
|
||||
stage: str
|
||||
cluster_label: int
|
||||
review_count: int
|
||||
phrase_count: int
|
||||
top_phrases: List[str] = field(default_factory=list)
|
||||
source_rows: List[int] = field(default_factory=list)
|
||||
source_asins: List[str] = field(default_factory=list)
|
||||
sentiment: str = ""
|
||||
audience_label: int = -1
|
||||
|
||||
|
||||
@dataclass
|
||||
class MarketStats:
|
||||
total_reviews: int
|
||||
weighted_avg_rating: float
|
||||
pos_rate: float
|
||||
neg_rate: float
|
||||
neg_review_count: int
|
||||
pos_review_count: int
|
||||
asins: List[ASINStats] = field(default_factory=list)
|
||||
|
||||
|
||||
# Persona A/B/C 维度信号词（用于运行时判定簇的 suggested_dimension）
|
||||
_A_DIM_SIGNALS = frozenset({
|
||||
# 生理/物理特征 — 跨品类通用
|
||||
"sensitive skin", "sensitive", "allergy", "allergic",
|
||||
"elderly", "senior", "older",
|
||||
"kid", "child", "children", "toddler", "infant", "baby", "newborn",
|
||||
"pregnant", "pregnancy", "nursing", "breastfeeding",
|
||||
"diabetic", "diabetes",
|
||||
"my dog", "my cat", "my pet", "puppy", "kitten", "pet owner",
|
||||
"professional", "beginner", "first time user",
|
||||
"large breed", "small breed", "small dog", "large dog",
|
||||
"oily skin", "dry skin", "acne", "eczema", "psoriasis",
|
||||
"curly hair", "fine hair", "color treated",
|
||||
})
|
||||
_B_DIM_SIGNALS = frozenset({
|
||||
# 行为/使用场景 — 跨品类通用
|
||||
"travel", "on the go", "portable", "lightweight", "compact",
|
||||
"daily", "everyday", "first time", "first-time", "beginner",
|
||||
"outdoor", "indoor", "home", "office", "kitchen", "car",
|
||||
"quick", "easy to use", "simple", "convenient",
|
||||
"maintenance", "cleaning", "storage", "organization",
|
||||
"camping", "hiking", "gym", "workout",
|
||||
"cooking", "baking", "cleaning house",
|
||||
})
|
||||
_C_DIM_SIGNALS = frozenset({
|
||||
# 购买动机/背景 — 跨品类通用
|
||||
"gift", "present", "birthday", "christmas", "holiday",
|
||||
"for my wife", "for my husband", "for my mom", "for my daughter",
|
||||
"replacement", "replace", "upgrade", "switched from",
|
||||
"price", "cheap", "expensive", "affordable", "worth the money",
|
||||
"waste of money", "not worth", "overpriced", "good value",
|
||||
"recommend", "recommended", "saw on", "social media",
|
||||
"amazon", "online", "review", "reviews", "rating",
|
||||
"bought", "purchased", "ordered", "arrived",
|
||||
})
|
||||
|
||||
|
||||
def _suggest_dimension(stage: str, top_phrases: List[str]) -> str:
|
||||
"""根据簇的 top_phrases 内容判定建议的 Persona 维度（A/B/C）。"""
|
||||
blob = " ".join(top_phrases).lower()
|
||||
a_hits = sum(1 for w in _A_DIM_SIGNALS if w in blob)
|
||||
b_hits = sum(1 for w in _B_DIM_SIGNALS if w in blob)
|
||||
c_hits = sum(1 for w in _C_DIM_SIGNALS if w in blob)
|
||||
if a_hits >= 2:
|
||||
return "A"
|
||||
if a_hits >= 1 and (b_hits + c_hits) == 0:
|
||||
return "A"
|
||||
if b_hits > c_hits and b_hits >= 2:
|
||||
return "B"
|
||||
if c_hits > b_hits and c_hits >= 2:
|
||||
return "C"
|
||||
# 默认按 stage 推理
|
||||
if stage.startswith("3a"):
|
||||
return "C"
|
||||
return "B"
|
||||
|
||||
|
||||
class DataLoader:
|
||||
def __init__(self, project_root: Path, product_name: str = "", industry: str = ""):
|
||||
self.project_root = Path(project_root).resolve()
|
||||
self.product_name = product_name
|
||||
self.industry = industry
|
||||
self.structured_db = self.project_root / STRUCTURED_DB_NAME
|
||||
self.cluster_db = self.project_root / CLUSTER_DB_NAME
|
||||
self.embed_db = self.project_root / EMBED_DB_NAME
|
||||
self.cleaned_csv = self.project_root / CLEANED_CSV_NAME
|
||||
self.word_freq_csv = self.project_root / WORD_FREQ_CSV_NAME
|
||||
self.terms_json = self.project_root / TERMS_JSON_NAME
|
||||
|
||||
@staticmethod
|
||||
def load_raw_review_samples(input_dir: Path, max_samples: int = 50):
|
||||
"""从原始 CSV 目录中加载样本评论内容（仅 content 字段），供 LLM 识别产品/行业。
|
||||
Returns: (samples: list[str], dir_name: str)"""
|
||||
import csv as _csv
|
||||
samples = []
|
||||
dir_path = Path(input_dir)
|
||||
dir_name = dir_path.name if dir_path.is_dir() else ""
|
||||
if not dir_path.is_dir():
|
||||
logger.warning("原始数据目录不存在: %s", dir_path)
|
||||
return samples, dir_name
|
||||
for fname in sorted(dir_path.iterdir()):
|
||||
if not fname.suffix.lower() == ".csv":
|
||||
continue
|
||||
try:
|
||||
with fname.open(encoding="utf-8-sig", newline="") as f:
|
||||
reader = _csv.DictReader(f)
|
||||
if not reader.fieldnames or "content" not in reader.fieldnames:
|
||||
continue
|
||||
for row in reader:
|
||||
text = (row.get("content") or "").strip()
|
||||
if text and len(text) >= 20:
|
||||
samples.append(text)
|
||||
if len(samples) >= max_samples:
|
||||
break
|
||||
if len(samples) >= max_samples:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.debug("跳过文件 %s: %s", fname.name, e)
|
||||
return samples, dir_name
|
||||
|
||||
def load_reviews(self) -> List[ReviewRecord]:
|
||||
if not self.cleaned_csv.is_file():
|
||||
raise FileNotFoundError(f"清洗后评论文件不存在: {self.cleaned_csv}")
|
||||
reviews: List[ReviewRecord] = []
|
||||
with self.cleaned_csv.open(encoding="utf-8-sig", newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for i, row in enumerate(reader, start=1):
|
||||
text = (row.get("content") or "").strip()
|
||||
if not text:
|
||||
continue
|
||||
reviews.append(ReviewRecord(
|
||||
asin=(row.get("asin") or "").strip(),
|
||||
rating=float(row.get("rating") or 0),
|
||||
title=(row.get("title") or "").strip(),
|
||||
content=text,
|
||||
verified=(row.get("verified") or "").strip().lower() == "true",
|
||||
vine=(row.get("vine") or "").strip().lower() == "true",
|
||||
review_date=(row.get("review_date") or "").strip(),
|
||||
source_row=i,
|
||||
))
|
||||
logger.info("已加载 %s 条有效评论", len(reviews))
|
||||
return reviews
|
||||
|
||||
def get_review_by_source_row(self, source_row: int) -> Optional[ReviewRecord]:
|
||||
reviews = self.load_reviews()
|
||||
for r in reviews:
|
||||
if r.source_row == source_row:
|
||||
return r
|
||||
return None
|
||||
|
||||
def load_comment_extractions(
|
||||
self,
|
||||
job_id: Optional[int] = None,
|
||||
) -> Dict[int, Dict[str, Any]]:
|
||||
"""加载 source_row → extraction_json 映射（最新 job 或指定 job_id）。"""
|
||||
if not self.structured_db.is_file():
|
||||
logger.warning("未找到结构化库 %s，引用匹配将仅使用原文关键词", self.structured_db)
|
||||
return {}
|
||||
conn = sqlite3.connect(self.structured_db)
|
||||
try:
|
||||
if job_id is None:
|
||||
row = conn.execute(
|
||||
"SELECT id FROM analysis_jobs ORDER BY id DESC LIMIT 1"
|
||||
).fetchone()
|
||||
if not row:
|
||||
logger.warning("voc_structured.sqlite 中无结构化任务")
|
||||
return {}
|
||||
job_id = int(row[0])
|
||||
cur = conn.execute(
|
||||
"""
|
||||
SELECT source_row, extraction_json
|
||||
FROM comment_extractions
|
||||
WHERE job_id = ?
|
||||
ORDER BY source_row
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
out: Dict[int, Dict[str, Any]] = {}
|
||||
for sr, js in cur.fetchall():
|
||||
try:
|
||||
out[int(sr)] = json.loads(js)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.debug("跳过无效 extraction source_row=%s: %s", sr, e)
|
||||
logger.info("已加载 %s 条结构化提取 (job_id=%s)", len(out), job_id)
|
||||
return out
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def load_basic_stats(self) -> MarketStats:
|
||||
reviews = self.load_reviews()
|
||||
if not reviews:
|
||||
raise ValueError("无有效评论")
|
||||
asin_groups: Dict[str, List[ReviewRecord]] = defaultdict(list)
|
||||
for r in reviews:
|
||||
asin_groups[r.asin].append(r)
|
||||
asin_stats_list: List[ASINStats] = []
|
||||
weighted_sum = 0.0
|
||||
total_count = 0
|
||||
for asin in sorted(asin_groups.keys()):
|
||||
grp = asin_groups[asin]
|
||||
ratings = [r.rating for r in grp if r.rating > 0]
|
||||
n = len(ratings)
|
||||
if n == 0:
|
||||
continue
|
||||
avg = sum(ratings) / n
|
||||
pos = sum(1 for r in ratings if r >= 4) / n
|
||||
neg = sum(1 for r in ratings if r <= 2) / n
|
||||
dist = Counter(int(r) for r in ratings)
|
||||
asin_stats_list.append(ASINStats(
|
||||
asin=asin, total=n, avg_rating=round(avg, 2),
|
||||
pos_rate=round(pos, 3), neg_rate=round(neg, 3),
|
||||
star_dist={i: dist.get(i, 0) for i in range(1, 6)},
|
||||
))
|
||||
weighted_sum += avg * n
|
||||
total_count += n
|
||||
all_ratings = [r.rating for r in reviews if r.rating > 0]
|
||||
wavg = round(weighted_sum / total_count, 2) if total_count else 0.0
|
||||
pos_total = sum(1 for r in all_ratings if r >= 4)
|
||||
neg_total = sum(1 for r in all_ratings if r <= 2)
|
||||
return MarketStats(
|
||||
total_reviews=total_count,
|
||||
weighted_avg_rating=wavg,
|
||||
pos_rate=round(pos_total / total_count, 3) if total_count else 0,
|
||||
neg_rate=round(neg_total / total_count, 3) if total_count else 0,
|
||||
neg_review_count=neg_total,
|
||||
pos_review_count=pos_total,
|
||||
asins=asin_stats_list,
|
||||
)
|
||||
|
||||
def _latest_cluster_run_id(self) -> int:
|
||||
if not self.cluster_db.is_file():
|
||||
raise FileNotFoundError(f"聚类库不存在: {self.cluster_db}")
|
||||
conn = sqlite3.connect(self.cluster_db)
|
||||
try:
|
||||
row = conn.execute("SELECT id FROM cluster_runs ORDER BY id DESC LIMIT 1").fetchone()
|
||||
if not row:
|
||||
raise RuntimeError("voc_clustering.sqlite 中无聚类记录")
|
||||
return int(row[0])
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def load_cluster_data(self) -> Dict[str, List[ClusterData]]:
|
||||
run_id = self._latest_cluster_run_id()
|
||||
conn = sqlite3.connect(self.cluster_db)
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"""SELECT stage, cluster_label, source_row, embed_text,
|
||||
entity_type, audience, sentiment, content
|
||||
FROM cluster_assignments WHERE run_id = ?
|
||||
ORDER BY stage, cluster_label, source_row""",
|
||||
(run_id,),
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
groups: Dict[Tuple[str, int], List[Tuple[int, str, str, str, str]]] = defaultdict(list)
|
||||
for stage, label, src_row, embed_text, etype, audience, sentiment, content in rows:
|
||||
groups[(stage, int(label))].append((
|
||||
int(src_row), embed_text, etype or "", audience or "", sentiment or "",
|
||||
))
|
||||
reviews = self.load_reviews()
|
||||
row_to_asin: Dict[int, str] = {r.source_row: r.asin for r in reviews}
|
||||
result: Dict[str, List[ClusterData]] = defaultdict(list)
|
||||
for (stage, label), items in sorted(groups.items()):
|
||||
unique_src_rows = sorted(set(sr for sr, _, _, _, _ in items))
|
||||
asins = sorted(set(row_to_asin.get(sr, "?") for sr in unique_src_rows))
|
||||
phrases = [et for _, et, _, _, _ in items if et.strip()]
|
||||
phrase_counter = Counter(phrases)
|
||||
top_phrases = [p for p, _ in phrase_counter.most_common(15)]
|
||||
sentiment = items[0][3] if items else ""
|
||||
audience_str = items[0][2] if items else ""
|
||||
aud_label = -1
|
||||
if audience_str:
|
||||
try:
|
||||
aud_label = int(audience_str)
|
||||
except ValueError:
|
||||
aud_label = -1
|
||||
cd = ClusterData(
|
||||
stage=stage, cluster_label=label,
|
||||
review_count=len(unique_src_rows), phrase_count=len(phrases),
|
||||
top_phrases=top_phrases, source_rows=unique_src_rows,
|
||||
source_asins=asins, sentiment=sentiment, audience_label=aud_label,
|
||||
)
|
||||
result[stage].append(cd)
|
||||
logger.info("已加载聚类: %s stages, %s 簇", len(result), sum(len(v) for v in result.values()))
|
||||
return dict(result)
|
||||
|
||||
|
||||
|
||||
def build_persona_cluster_catalog(
|
||||
self,
|
||||
sample_reviews_per_cluster: int = 5,
|
||||
max_review_chars: int = 320,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""供 Persona LLM 绑定的聚类簇目录（含每簇代表性评论和运行时维度建议）。"""
|
||||
all_c = self.load_cluster_data()
|
||||
catalog: List[Dict[str, Any]] = []
|
||||
for stage in (
|
||||
"3a_pain_global",
|
||||
"3b_aspect_opinion_negative",
|
||||
"3b_aspect_opinion_positive",
|
||||
):
|
||||
stage_clusters = sorted(all_c.get(stage, []), key=lambda x: -x.review_count)
|
||||
phrase_doc_freq: Dict[str, int] = defaultdict(int)
|
||||
for c in stage_clusters:
|
||||
segs = set(_phrase_segments_local(c.top_phrases))
|
||||
for seg in segs:
|
||||
if seg not in _PERSONA_PHRASE_STOPWORDS and len(seg) >= 4:
|
||||
phrase_doc_freq[seg] += 1
|
||||
n_stage = max(len(stage_clusters), 1)
|
||||
used_labels: Set[str] = set()
|
||||
for c in stage_clusters:
|
||||
label = _cluster_semantic_label(c, phrase_doc_freq, n_stage)
|
||||
if label in used_labels:
|
||||
for rank, phrase in enumerate(c.top_phrases[1:], start=1):
|
||||
alt = (phrase or "").strip().lower()[:48]
|
||||
if alt and len(alt) >= 4 and alt not in used_labels:
|
||||
label = alt
|
||||
break
|
||||
used_labels.add(label)
|
||||
entry: Dict[str, Any] = {
|
||||
"stage": stage,
|
||||
"label": c.cluster_label,
|
||||
"review_count": c.review_count,
|
||||
"semantic_label": label,
|
||||
"top_phrases": c.top_phrases[:12],
|
||||
"suggested_dimension": _suggest_dimension(stage, c.top_phrases),
|
||||
}
|
||||
if sample_reviews_per_cluster > 0:
|
||||
quotes = self.get_representative_quotes(
|
||||
c.source_rows, max_quotes=sample_reviews_per_cluster,
|
||||
)
|
||||
entry["sample_reviews"] = [
|
||||
{
|
||||
"rating": int(q.get("rating") or 0),
|
||||
"content": (q.get("content") or "")[:max_review_chars],
|
||||
}
|
||||
for q in quotes
|
||||
]
|
||||
catalog.append(entry)
|
||||
return catalog
|
||||
|
||||
def load_audience_clusters(self) -> List[ClusterData]:
|
||||
return self.load_cluster_data().get("1_audience", [])
|
||||
|
||||
def load_global_pain_clusters(self) -> List[ClusterData]:
|
||||
return self.load_cluster_data().get("3a_pain_global", [])
|
||||
|
||||
def load_global_feedback_clusters(self) -> Dict[str, List[ClusterData]]:
|
||||
all_c = self.load_cluster_data()
|
||||
return {
|
||||
"negative": all_c.get("3b_aspect_opinion_negative", []),
|
||||
"positive": all_c.get("3b_aspect_opinion_positive", []),
|
||||
"neutral": all_c.get("3b_aspect_opinion_neutral", []),
|
||||
}
|
||||
|
||||
def load_per_audience_clusters(self) -> Dict[int, Dict[str, List[ClusterData]]]:
|
||||
all_c = self.load_cluster_data()
|
||||
aud_c: Dict[int, Dict[str, List[ClusterData]]] = defaultdict(
|
||||
lambda: {"pain": [], "negative": [], "positive": [], "neutral": []}
|
||||
)
|
||||
for stage, clusters in all_c.items():
|
||||
if stage.startswith("2a_pain_audience_c"):
|
||||
al = int(stage.split("_c")[-1])
|
||||
aud_c[al]["pain"].extend(clusters)
|
||||
elif stage.startswith("2b_aspect_opinion_"):
|
||||
parts = stage.replace("2b_aspect_opinion_", "").split("_audience_c")
|
||||
sentiment = parts[0]
|
||||
al = int(parts[1]) if len(parts) > 1 else -1
|
||||
if sentiment in ("positive", "negative", "neutral"):
|
||||
aud_c[al][sentiment].extend(clusters)
|
||||
return dict(aud_c)
|
||||
|
||||
def build_cluster_prompt_data(
|
||||
self,
|
||||
persona_sample_reviews: int = 5,
|
||||
) -> Dict[str, Any]:
|
||||
global_pain = self.load_global_pain_clusters()
|
||||
global_fb = self.load_global_feedback_clusters()
|
||||
stats = self.load_basic_stats()
|
||||
|
||||
def _sum(clusters: List[ClusterData]) -> List[Dict[str, Any]]:
|
||||
return [
|
||||
{"label": c.cluster_label, "phrase_count": c.phrase_count,
|
||||
"review_count": c.review_count, "asins": c.source_asins,
|
||||
"top_phrases": c.top_phrases}
|
||||
for c in sorted(clusters, key=lambda x: -x.phrase_count)
|
||||
]
|
||||
return {
|
||||
"audience_clusters": [],
|
||||
"global_pains": _sum(global_pain),
|
||||
"global_negative": _sum(global_fb["negative"]),
|
||||
"global_positive": _sum(global_fb["positive"]),
|
||||
"persona_cluster_catalog": self.build_persona_cluster_catalog(
|
||||
sample_reviews_per_cluster=persona_sample_reviews,
|
||||
),
|
||||
"per_audience": {},
|
||||
"total_reviews": stats.total_reviews,
|
||||
"neg_review_count": stats.neg_review_count,
|
||||
"pos_review_count": stats.pos_review_count,
|
||||
"asins": [a.asin for a in stats.asins],
|
||||
}
|
||||
|
||||
def load_word_freq(self, top_n: int = 200) -> List[Tuple[str, int]]:
|
||||
if not self.word_freq_csv.is_file():
|
||||
logger.warning("词频文件不存在: %s", self.word_freq_csv)
|
||||
return []
|
||||
rows: List[Tuple[str, int]] = []
|
||||
with self.word_freq_csv.open(encoding="utf-8", newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
w = (row.get("word") or "").strip()
|
||||
if not w:
|
||||
continue
|
||||
try:
|
||||
c = int(row.get("count") or 0)
|
||||
except ValueError:
|
||||
c = 0
|
||||
rows.append((w, c))
|
||||
rows.sort(key=lambda x: x[1], reverse=True)
|
||||
return rows[:top_n]
|
||||
|
||||
def get_representative_quotes(self, source_rows: List[int], max_quotes: int = 5) -> List[Dict[str, str]]:
|
||||
reviews = self.load_reviews()
|
||||
row_map = {r.source_row: r for r in reviews}
|
||||
candidates = []
|
||||
for sr in source_rows:
|
||||
r = row_map.get(sr)
|
||||
if r and len(r.content) >= 30:
|
||||
candidates.append(r)
|
||||
candidates.sort(key=lambda r: abs(len(r.content) - 200))
|
||||
selected = candidates[:max_quotes]
|
||||
return [
|
||||
{"content": r.content, "asin": r.asin, "rating": str(int(r.rating)),
|
||||
"amazon_url": f"https://www.amazon.com/dp/{r.asin}"}
|
||||
for r in selected
|
||||
]
|
||||
|
||||
def get_quotes_for_cluster(self, cluster: ClusterData, max_quotes: int = 5) -> List[Dict[str, str]]:
|
||||
return self.get_representative_quotes(cluster.source_rows, max_quotes)
|
||||
|
||||
|
||||
def build_amazon_url(asin: str) -> str:
|
||||
return f"https://www.amazon.com/dp/{asin}"
|
||||
|
||||
|
||||
def asin_link_html(asin: str) -> str:
|
||||
url = build_amazon_url(asin)
|
||||
return f'<a href="{url}" target="_blank" rel="noopener">{asin}</a>'
|
||||
|
||||
750
voc_业务_2/echarts_builder.py
Normal file
750
voc_业务_2/echarts_builder.py
Normal file
|
|
@ -0,0 +1,750 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
ECharts 图表构建器：生成内联 ECharts JS 代码和静态库嵌入。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger("voc.echarts")
|
||||
|
||||
# ECharts CDN（用于下载内联）
|
||||
ECHARTS_CDN = "https://cdn.jsdelivr.net/npm/echarts@5.5.0/dist/echarts.min.js"
|
||||
CACHE_FILE = Path(__file__).resolve().parent / ".echarts_cache.js"
|
||||
|
||||
|
||||
def get_echarts_script(inline: bool = True) -> str:
|
||||
"""返回 ECharts 库的 <script> 标签。inline=True 则内联，否则用 CDN。"""
|
||||
if not inline:
|
||||
return f'<script src="{ECHARTS_CDN}"></script>'
|
||||
|
||||
# 内联模式：从缓存或下载
|
||||
if CACHE_FILE.is_file():
|
||||
js = CACHE_FILE.read_text(encoding="utf-8")
|
||||
logger.info("使用缓存的 ECharts (%s KB)", len(js) // 1024)
|
||||
return f"<script>\n{js}\n</script>"
|
||||
|
||||
# 下载
|
||||
logger.info("下载 ECharts 库...")
|
||||
import urllib.request
|
||||
try:
|
||||
with urllib.request.urlopen(ECHARTS_CDN, timeout=30) as resp:
|
||||
js = resp.read().decode("utf-8")
|
||||
CACHE_FILE.write_text(js, encoding="utf-8")
|
||||
logger.info("ECharts 已缓存 (%s KB)", len(js) // 1024)
|
||||
return f"<script>\n{js}\n</script>"
|
||||
except Exception as e:
|
||||
logger.warning("下载 ECharts 失败: %s，降级到 CDN", e)
|
||||
return f'<script src="{ECHARTS_CDN}"></script>'
|
||||
|
||||
|
||||
# ── 图表 JS 生成 ──
|
||||
|
||||
ASIN_COLORS_BAR = ["#ef4444", "#f97316", "#eab308", "#8b5cf6", "#3b82f6"]
|
||||
ASIN_COLORS_POS = ["#22c55e", "#86efac", "#4ade80", "#2dd4bf", "#60a5fa"]
|
||||
|
||||
|
||||
def build_star_dist_chart(asins: List[str], star_data: Dict[str, Dict[int, int]]) -> str:
|
||||
"""评分分布堆叠柱状图。"""
|
||||
datasets = []
|
||||
colors = ["#22c55e", "#86efac", "#d1d5db", "#fbbf24", "#ef4444"]
|
||||
for star in [5, 4, 3, 2, 1]:
|
||||
datasets.append({
|
||||
"name": f"{star}★",
|
||||
"type": "bar",
|
||||
"stack": "total",
|
||||
"data": [star_data.get(a, {}).get(star, 0) for a in asins],
|
||||
"itemStyle": {"color": colors[5 - star]},
|
||||
})
|
||||
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"legend": {"bottom": 0, "textStyle": {"fontSize": 11}},
|
||||
"grid": {"left": 50, "right": 20, "top": 20, "bottom": 40},
|
||||
"xAxis": {"type": "category", "data": asins, "axisLabel": {"fontSize": 10}},
|
||||
"yAxis": {"type": "value"},
|
||||
"series": datasets,
|
||||
}
|
||||
return f"""new Chart(echarts.init(document.getElementById('starDistChart')), {{
|
||||
type: 'bar',
|
||||
data: {json.dumps(asins)},
|
||||
datasets: {json.dumps(datasets, ensure_ascii=False)},
|
||||
options: {{
|
||||
responsive: true, maintainAspectRatio: false,
|
||||
plugins: {{ legend: {{ position: 'bottom', labels: {{ font: {{ size: 11 }} }} }} }},
|
||||
scales: {{ x: {{ stacked: true }}, y: {{ stacked: true, beginAtZero: true }} }}
|
||||
}}
|
||||
}});
|
||||
// ECharts 版本:
|
||||
(function() {{
|
||||
var dom = document.getElementById('starDistChart');
|
||||
var chart = echarts.init(dom);
|
||||
chart.setOption({json.dumps(option, ensure_ascii=False)});
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}})();"""
|
||||
|
||||
|
||||
def build_horizontal_bar_chart(
|
||||
element_id: str,
|
||||
labels: List[str],
|
||||
data: List[int],
|
||||
colors: List[str] | None = None,
|
||||
height: int = 360,
|
||||
*,
|
||||
asin_category_axis: Optional[str] = None,
|
||||
) -> str:
|
||||
"""水平柱状图。asin_category_axis='y' 时 Y 轴短码 tooltip 显示 ASIN 链接。"""
|
||||
if colors is None:
|
||||
colors = ["#ef4444"] * len(labels)
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"grid": {"left": 150, "right": 40, "top": 10, "bottom": 20},
|
||||
"xAxis": {"type": "value"},
|
||||
"yAxis": {"type": "category", "data": labels, "inverse": True,
|
||||
"axisLabel": {"fontSize": 11}},
|
||||
"series": [{
|
||||
"type": "bar",
|
||||
"data": [{"value": v, "itemStyle": {"color": c}} for v, c in zip(data, colors)],
|
||||
"label": {"show": True, "position": "right", "fontSize": 10},
|
||||
}],
|
||||
}
|
||||
return _echarts_init(element_id, option, height=height, asin_category_axis=asin_category_axis)
|
||||
|
||||
|
||||
def build_asin_mini_chart(
|
||||
element_id: str,
|
||||
labels: List[str],
|
||||
data: List[int],
|
||||
color: str = "#ef4444",
|
||||
) -> str:
|
||||
"""单个 ASIN 的迷你柱状图。"""
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"grid": {"left": 100, "right": 30, "top": 5, "bottom": 15},
|
||||
"xAxis": {"type": "value", "axisLabel": {"fontSize": 9}},
|
||||
"yAxis": {"type": "category", "data": labels, "inverse": True,
|
||||
"axisLabel": {"fontSize": 9}},
|
||||
"series": [{
|
||||
"type": "bar",
|
||||
"data": [{"value": v, "itemStyle": {"color": color}} for v in data],
|
||||
}],
|
||||
}
|
||||
return f"""(function() {{
|
||||
var dom = document.getElementById('{element_id}');
|
||||
var chart = echarts.init(dom);
|
||||
chart.setOption({json.dumps(option, ensure_ascii=False)});
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}})();"""
|
||||
|
||||
|
||||
def build_grouped_bar_chart(
|
||||
element_id: str,
|
||||
labels: List[str],
|
||||
datasets: List[Dict[str, Any]],
|
||||
) -> str:
|
||||
"""分组柱状图（多 ASIN 对比）。"""
|
||||
series = [
|
||||
{
|
||||
"name": ds["label"],
|
||||
"type": "bar",
|
||||
"data": ds["data"],
|
||||
"itemStyle": {"color": ds.get("color", "#3b82f6")},
|
||||
}
|
||||
for ds in datasets
|
||||
]
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"legend": {"bottom": 0, "textStyle": {"fontSize": 10}},
|
||||
"grid": {"left": 50, "right": 20, "top": 20, "bottom": 40},
|
||||
"xAxis": {"type": "category", "data": labels, "axisLabel": {"fontSize": 10, "rotate": 20}},
|
||||
"yAxis": {"type": "value"},
|
||||
"series": series,
|
||||
}
|
||||
return f"""(function() {{
|
||||
var dom = document.getElementById('{element_id}');
|
||||
var chart = echarts.init(dom);
|
||||
chart.setOption({json.dumps(option, ensure_ascii=False)});
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}})();"""
|
||||
|
||||
|
||||
# ── 布局与图表类型阈值 ──
|
||||
|
||||
THEME_COLORS_NEG = ["#ef4444", "#f97316", "#eab308", "#dc2626", "#b91c1c", "#fb923c"]
|
||||
THEME_COLORS_POS = ["#22c55e", "#86efac", "#4ade80", "#2dd4bf", "#60a5fa", "#16a34a"]
|
||||
|
||||
|
||||
def get_chart_layout(stats: Any, cfg: Optional[dict] = None) -> Dict[str, Any]:
|
||||
"""根据 ASIN 数量计算图表布局参数。"""
|
||||
cfg = cfg or {}
|
||||
n = len(getattr(stats, "asins", []) or [])
|
||||
large_th = int(cfg.get("large_asin_threshold", 12))
|
||||
heatmap_th = int(cfg.get("heatmap_asin_threshold", 30))
|
||||
large = n > large_th
|
||||
heatmap = n > heatmap_th
|
||||
return {
|
||||
"asin_count": n,
|
||||
"large_market": large,
|
||||
"use_heatmap": heatmap,
|
||||
"star_chart_height": max(320, n * 22) if large else 300,
|
||||
"star_summary_height": 280,
|
||||
"show_star_summary": large and n > 15,
|
||||
"neg_theme_height": 480 if heatmap else (420 if large else 360),
|
||||
"pos_theme_height": 480 if heatmap else (420 if large else 360),
|
||||
}
|
||||
|
||||
|
||||
# ── ASIN 短码 tooltip（悬浮显示代称 + 可点击 ASIN）──
|
||||
|
||||
def build_asin_tooltip_map(asin_short_codes: Dict[str, str]) -> Dict[str, Dict[str, str]]:
|
||||
"""ASIN 短码 A → {asin, url}。"""
|
||||
from data_loader import build_amazon_url
|
||||
return {
|
||||
short: {"asin": asin, "url": build_amazon_url(asin)}
|
||||
for asin, short in asin_short_codes.items()
|
||||
}
|
||||
|
||||
|
||||
def build_asin_tooltip_bootstrap(asin_short_codes: Dict[str, str]) -> str:
|
||||
"""注入全局 VOC_ASIN_MAP 与 axis tooltip formatter。"""
|
||||
map_json = json.dumps(build_asin_tooltip_map(asin_short_codes), ensure_ascii=False)
|
||||
return f"""// ASIN 短码 tooltip（悬浮：A · B0XXX 可点击跳转亚马逊）
|
||||
window.VOC_ASIN_MAP = {map_json};
|
||||
window.vocAsinTooltipHeader = function(code) {{
|
||||
if (!code || String(code).indexOf('其余') === 0) return code;
|
||||
var info = (window.VOC_ASIN_MAP || {{}})[code];
|
||||
if (info && info.asin) {{
|
||||
return code + ' · <a href="' + info.url + '" target="_blank" rel="noopener" style="color:#2563eb;text-decoration:underline">' + info.asin + '</a>';
|
||||
}}
|
||||
return code;
|
||||
}};
|
||||
window.vocApplyAsinAxisTooltip = function(opt, axisKey) {{
|
||||
axisKey = axisKey || 'x';
|
||||
var axis = axisKey === 'y' ? opt.yAxis : opt.xAxis;
|
||||
if (!axis || axis.type !== 'category') return;
|
||||
opt.tooltip = opt.tooltip || {{}};
|
||||
opt.tooltip.trigger = 'axis';
|
||||
opt.tooltip.enterable = true;
|
||||
opt.tooltip.confine = true;
|
||||
opt.tooltip.axisPointer = opt.tooltip.axisPointer || {{type: 'shadow'}};
|
||||
opt.tooltip.formatter = function(params) {{
|
||||
var items = Array.isArray(params) ? params : [params];
|
||||
if (!items.length) return '';
|
||||
var code = items[0].axisValue != null ? items[0].axisValue : (items[0].name || '');
|
||||
var lines = [window.vocAsinTooltipHeader(code)];
|
||||
items.forEach(function(p) {{
|
||||
if (p.seriesName == null || p.value == null) return;
|
||||
var v = p.value;
|
||||
if (Array.isArray(v)) v = v[v.length - 1];
|
||||
if (v && typeof v === 'object' && v.value != null) v = v.value;
|
||||
lines.push((p.marker || '') + p.seriesName + ': ' + v);
|
||||
}});
|
||||
return lines.join('<br/>');
|
||||
}};
|
||||
}};"""
|
||||
|
||||
|
||||
def _echarts_init(
|
||||
element_id: str,
|
||||
option: dict,
|
||||
height: Optional[int] = None,
|
||||
*,
|
||||
asin_category_axis: Optional[str] = None,
|
||||
) -> str:
|
||||
h_js = f"dom.style.height='{height}px';" if height else ""
|
||||
apply_js = ""
|
||||
if asin_category_axis:
|
||||
apply_js = f"if (window.vocApplyAsinAxisTooltip) window.vocApplyAsinAxisTooltip(opt, '{asin_category_axis}');"
|
||||
return f"""(function() {{
|
||||
var dom = document.getElementById('{element_id}');
|
||||
if (!dom) return;
|
||||
{h_js}
|
||||
var opt = {json.dumps(option, ensure_ascii=False)};
|
||||
{apply_js}
|
||||
var chart = echarts.init(dom);
|
||||
chart.setOption(opt);
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}})();"""
|
||||
|
||||
|
||||
def _aggregate_star_dist(asin_list: List[str], star_data: Dict[str, Dict[int, int]]) -> Dict[int, int]:
|
||||
agg: Dict[int, int] = {s: 0 for s in (5, 4, 3, 2, 1)}
|
||||
for asin in asin_list:
|
||||
sd = star_data.get(asin, {})
|
||||
for s in agg:
|
||||
agg[s] += sd.get(s, 0)
|
||||
return agg
|
||||
|
||||
|
||||
def build_overall_charts(
|
||||
neg_freq: dict,
|
||||
pos_freq: dict,
|
||||
neg_themes: list,
|
||||
) -> str:
|
||||
"""生成主题频次画像区的全市场图表（negChartOverall / posChartOverall）。"""
|
||||
priority_color = {"P0": "#ef4444", "P1": "#f97316", "P2": "#eab308"}
|
||||
theme_priority = {t["name"]: t.get("priority", "P2") for t in neg_themes}
|
||||
|
||||
neg_sorted = sorted(neg_freq.items(), key=lambda x: x[1], reverse=True)
|
||||
pos_sorted = sorted(pos_freq.items(), key=lambda x: x[1], reverse=True)
|
||||
neg_labels = [n for n, _ in neg_sorted]
|
||||
neg_data = [c for _, c in neg_sorted]
|
||||
pos_labels = [n for n, _ in pos_sorted]
|
||||
pos_data = [c for _, c in pos_sorted]
|
||||
neg_colors = [priority_color.get(theme_priority.get(n, "P2"), "#94a3b8") for n in neg_labels]
|
||||
|
||||
parts = []
|
||||
parts.append("// 整体主题频次 - 差评")
|
||||
parts.append(build_horizontal_bar_chart("negChartOverall", neg_labels, neg_data, neg_colors))
|
||||
parts.append("// 整体主题频次 - 好评")
|
||||
parts.append(build_horizontal_bar_chart("posChartOverall", pos_labels, pos_data, ["#22c55e"] * len(pos_labels)))
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def build_theme_mini_charts_html(
|
||||
prefix: str,
|
||||
theme_labels: List[str],
|
||||
asin_count: int,
|
||||
*,
|
||||
large_threshold: int = 30,
|
||||
) -> str:
|
||||
"""大品类：每主题 Top N ASIN 小 multiples 的 HTML 容器。"""
|
||||
if not theme_labels or asin_count <= large_threshold:
|
||||
return ""
|
||||
blocks = ['<div class="theme-mini-grid">']
|
||||
for i, theme in enumerate(theme_labels):
|
||||
blocks.append(
|
||||
f'<div class="theme-mini-box"><h4>{theme}</h4>'
|
||||
f'<div id="{prefix}Mini{i}" class="theme-mini-chart"></div></div>'
|
||||
)
|
||||
blocks.append("</div>")
|
||||
return "".join(blocks)
|
||||
|
||||
|
||||
def build_all_charts(
|
||||
stats: Any,
|
||||
neg_themes: List[Dict[str, Any]],
|
||||
pos_themes: List[Dict[str, Any]],
|
||||
neg_freq: Dict[str, int],
|
||||
pos_freq: Dict[str, int],
|
||||
per_asin_neg: Dict[str, Dict[str, int]],
|
||||
per_asin_pos: Dict[str, Dict[str, int]],
|
||||
asin_labels: Optional[Dict[str, str]] = None,
|
||||
asin_short_codes: Optional[Dict[str, str]] = None,
|
||||
layout: Optional[Dict[str, Any]] = None,
|
||||
cfg: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""生成所有图表的 JS 代码。"""
|
||||
asin_labels = asin_labels or {}
|
||||
asin_short_codes = asin_short_codes or asin_labels
|
||||
layout = layout or get_chart_layout(stats, cfg)
|
||||
parts: List[str] = [build_asin_tooltip_bootstrap(asin_short_codes)]
|
||||
|
||||
asins = [a.asin for a in stats.asins]
|
||||
if layout["large_market"]:
|
||||
from report_utils import sorted_asin_stats
|
||||
asins = [a.asin for a in sorted_asin_stats(stats)]
|
||||
else:
|
||||
asins = [a.asin for a in sorted(stats.asins, key=lambda a: (-a.avg_rating, -a.total))]
|
||||
axis_labels = [asin_short_codes.get(a, a) for a in asins]
|
||||
star_data = {a.asin: a.star_dist for a in stats.asins}
|
||||
|
||||
parts.append("// 1. 评分分布堆叠图")
|
||||
parts.append(_build_star_dist_echarts(
|
||||
axis_labels, star_data, asins,
|
||||
horizontal=layout["large_market"],
|
||||
))
|
||||
if layout.get("show_star_summary"):
|
||||
from report_utils import sorted_asin_stats
|
||||
ranked = [a.asin for a in sorted_asin_stats(stats)]
|
||||
top_n = int((cfg or {}).get("star_summary_top_n", 15))
|
||||
top_asins = ranked[:top_n]
|
||||
rest_asins = ranked[top_n:]
|
||||
summary_labels = [asin_short_codes.get(a, a) for a in top_asins]
|
||||
summary_data = {a: star_data[a] for a in top_asins}
|
||||
summary_asins = list(top_asins)
|
||||
if rest_asins:
|
||||
summary_labels.append(f"其余{len(rest_asins)}款")
|
||||
summary_data["__other__"] = _aggregate_star_dist(rest_asins, star_data)
|
||||
summary_asins.append("__other__")
|
||||
parts.append("// 1b. 评分分布摘要")
|
||||
parts.append(_build_star_dist_echarts(
|
||||
summary_labels, summary_data, summary_asins,
|
||||
element_id="starDistSummaryChart", horizontal=False,
|
||||
))
|
||||
|
||||
neg_sorted = sorted(neg_freq.items(), key=lambda x: x[1], reverse=True)
|
||||
pos_sorted = sorted(pos_freq.items(), key=lambda x: x[1], reverse=True)
|
||||
neg_labels = [n for n, _ in neg_sorted]
|
||||
neg_data = [c for _, c in neg_sorted]
|
||||
pos_labels = [n for n, _ in pos_sorted]
|
||||
pos_data = [c for _, c in pos_sorted]
|
||||
|
||||
priority_color = {"P0": "#ef4444", "P1": "#f97316", "P2": "#eab308"}
|
||||
theme_priority = {t["name"]: t.get("priority", "P2") for t in neg_themes}
|
||||
neg_colors = [priority_color.get(theme_priority.get(n, "P2"), "#94a3b8") for n in neg_labels]
|
||||
|
||||
parts.append("// 2. 差评主题")
|
||||
parts.append(build_horizontal_bar_chart("negChart", neg_labels, neg_data, neg_colors))
|
||||
parts.append("// 3. 好评主题")
|
||||
parts.append(build_horizontal_bar_chart("posChart", pos_labels, pos_data, ["#22c55e"] * len(pos_labels)))
|
||||
|
||||
neg_top6 = neg_labels[:6]
|
||||
pos_top6 = pos_labels[:6]
|
||||
mini_top = int((cfg or {}).get("theme_mini_top_n", 8))
|
||||
|
||||
parts.append("// 4. 各 ASIN 差评主题对比")
|
||||
parts.append(_build_per_asin_theme_chart(
|
||||
"negChartPerAsin", neg_top6, per_asin_neg, asins,
|
||||
axis_labels=axis_labels, asin_full_labels=asin_labels,
|
||||
is_neg=True, use_heatmap=layout["use_heatmap"],
|
||||
))
|
||||
if layout["use_heatmap"]:
|
||||
parts.append(_build_theme_mini_charts_js(
|
||||
"neg", neg_top6, per_asin_neg, asins, asin_short_codes, asin_labels,
|
||||
is_neg=True, top_n=mini_top,
|
||||
))
|
||||
|
||||
parts.append("// 5. 各 ASIN 好评主题对比")
|
||||
parts.append(_build_per_asin_theme_chart(
|
||||
"posChartPerAsin", pos_top6, per_asin_pos, asins,
|
||||
axis_labels=axis_labels, asin_full_labels=asin_labels,
|
||||
is_neg=False, use_heatmap=layout["use_heatmap"],
|
||||
))
|
||||
if layout["use_heatmap"]:
|
||||
parts.append(_build_theme_mini_charts_js(
|
||||
"pos", pos_top6, per_asin_pos, asins, asin_short_codes, asin_labels,
|
||||
is_neg=False, top_n=mini_top,
|
||||
))
|
||||
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def _build_per_asin_theme_chart(
|
||||
element_id: str,
|
||||
theme_labels: List[str],
|
||||
per_asin_data: Dict[str, Dict[str, int]],
|
||||
asins: List[str],
|
||||
*,
|
||||
axis_labels: Optional[List[str]] = None,
|
||||
asin_full_labels: Optional[Dict[str, str]] = None,
|
||||
is_neg: bool = True,
|
||||
use_heatmap: bool = False,
|
||||
asin_labels: Optional[Dict[str, str]] = None,
|
||||
) -> str:
|
||||
"""各 ASIN 主题对比：分组柱（X=ASIN，series=主题）或热力图。"""
|
||||
if not theme_labels or not asins:
|
||||
return f"// skip {element_id}: no data"
|
||||
if asin_labels and not axis_labels:
|
||||
axis_labels = [asin_labels.get(a, a) for a in asins]
|
||||
axis_labels = axis_labels or asins
|
||||
asin_full_labels = asin_full_labels or asin_labels or {}
|
||||
|
||||
if use_heatmap:
|
||||
return _build_theme_heatmap(
|
||||
element_id, theme_labels, per_asin_data, asins,
|
||||
axis_labels, asin_full_labels, is_neg=is_neg,
|
||||
)
|
||||
|
||||
colors = THEME_COLORS_NEG if is_neg else THEME_COLORS_POS
|
||||
series = []
|
||||
for i, theme in enumerate(theme_labels):
|
||||
series.append({
|
||||
"name": theme,
|
||||
"type": "bar",
|
||||
"data": [per_asin_data.get(asin, {}).get(theme, 0) for asin in asins],
|
||||
"itemStyle": {"color": colors[i % len(colors)]},
|
||||
})
|
||||
option: Dict[str, Any] = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"legend": {"top": 0, "type": "scroll", "textStyle": {"fontSize": 10}},
|
||||
"grid": {"left": 48, "right": 16, "top": 48, "bottom": 72},
|
||||
"xAxis": {
|
||||
"type": "category",
|
||||
"data": axis_labels,
|
||||
"axisLabel": {"fontSize": 9, "rotate": 45, "interval": 0},
|
||||
},
|
||||
"yAxis": {"type": "value"},
|
||||
"series": series,
|
||||
}
|
||||
if len(asins) > 12:
|
||||
end_pct = min(100, round(12 / len(asins) * 100))
|
||||
option["dataZoom"] = [
|
||||
{"type": "inside", "start": 0, "end": end_pct},
|
||||
{"type": "slider", "start": 0, "end": end_pct, "bottom": 8, "height": 18},
|
||||
]
|
||||
option["grid"]["bottom"] = 96
|
||||
return _echarts_init(element_id, option, asin_category_axis="x")
|
||||
|
||||
|
||||
def _build_theme_heatmap(
|
||||
element_id: str,
|
||||
theme_labels: List[str],
|
||||
per_asin_data: Dict[str, Dict[str, int]],
|
||||
asins: List[str],
|
||||
axis_labels: List[str],
|
||||
asin_full_labels: Dict[str, str],
|
||||
*,
|
||||
is_neg: bool = True,
|
||||
) -> str:
|
||||
data = []
|
||||
max_val = 1
|
||||
for yi, theme in enumerate(theme_labels):
|
||||
for xi, asin in enumerate(asins):
|
||||
v = per_asin_data.get(asin, {}).get(theme, 0)
|
||||
max_val = max(max_val, v)
|
||||
data.append([xi, yi, v])
|
||||
colors = ["#fef2f2", "#fca5a5", "#ef4444", "#b91c1c"] if is_neg else ["#f0fdf4", "#86efac", "#22c55e", "#15803d"]
|
||||
end_pct = min(100, max(15, round(20 / max(len(asins), 1) * 100)))
|
||||
option = {
|
||||
"grid": {"left": 88, "right": 56, "top": 24, "bottom": 72},
|
||||
"xAxis": {
|
||||
"type": "category",
|
||||
"data": axis_labels,
|
||||
"splitArea": {"show": True},
|
||||
"axisLabel": {"fontSize": 9, "rotate": 45, "interval": 0},
|
||||
},
|
||||
"yAxis": {
|
||||
"type": "category",
|
||||
"data": theme_labels,
|
||||
"splitArea": {"show": True},
|
||||
"axisLabel": {"fontSize": 10},
|
||||
},
|
||||
"visualMap": {
|
||||
"min": 0,
|
||||
"max": max_val,
|
||||
"calculable": True,
|
||||
"orient": "vertical",
|
||||
"right": 8,
|
||||
"top": "center",
|
||||
"inRange": {"color": colors},
|
||||
"text": ["高", "低"],
|
||||
},
|
||||
"dataZoom": [
|
||||
{"type": "inside", "xAxisIndex": 0, "start": 0, "end": end_pct},
|
||||
{"type": "slider", "xAxisIndex": 0, "bottom": 8, "height": 18, "start": 0, "end": end_pct},
|
||||
],
|
||||
"series": [{
|
||||
"name": "命中数",
|
||||
"type": "heatmap",
|
||||
"data": data,
|
||||
"label": {"show": False},
|
||||
"emphasis": {"itemStyle": {"shadowBlur": 6, "shadowColor": "rgba(0,0,0,0.2)"}},
|
||||
}],
|
||||
}
|
||||
full_map = {axis_labels[i]: asin_full_labels.get(asins[i], asins[i]) for i in range(len(asins))}
|
||||
js_option = json.dumps(option, ensure_ascii=False)
|
||||
return f"""(function() {{
|
||||
var dom = document.getElementById('{element_id}');
|
||||
if (!dom) return;
|
||||
var opt = {js_option};
|
||||
var fullMap = {json.dumps(full_map, ensure_ascii=False)};
|
||||
opt.tooltip = {{
|
||||
position: 'top',
|
||||
enterable: true,
|
||||
confine: true,
|
||||
formatter: function(p) {{
|
||||
if (!p.data) return '';
|
||||
var code = (opt.xAxis.data[p.data[0]] || '');
|
||||
var theme = (opt.yAxis.data[p.data[1]] || '');
|
||||
var head = window.vocAsinTooltipHeader ? window.vocAsinTooltipHeader(code) : (fullMap[code] || code);
|
||||
return head + '<br/>' + theme + ': ' + p.data[2];
|
||||
}}
|
||||
}};
|
||||
var chart = echarts.init(dom);
|
||||
chart.setOption(opt);
|
||||
window.addEventListener('resize', function() {{ chart.resize(); }});
|
||||
}})();"""
|
||||
|
||||
|
||||
def _build_theme_mini_charts_js(
|
||||
prefix: str,
|
||||
theme_labels: List[str],
|
||||
per_asin_data: Dict[str, Dict[str, int]],
|
||||
asins: List[str],
|
||||
asin_short: Dict[str, str],
|
||||
asin_full: Dict[str, str],
|
||||
*,
|
||||
is_neg: bool = True,
|
||||
top_n: int = 8,
|
||||
) -> str:
|
||||
_ = asin_full
|
||||
parts = []
|
||||
color = "#ef4444" if is_neg else "#22c55e"
|
||||
for i, theme in enumerate(theme_labels):
|
||||
ranked = sorted(
|
||||
((asin, per_asin_data.get(asin, {}).get(theme, 0)) for asin in asins),
|
||||
key=lambda x: (-x[1], x[0]),
|
||||
)
|
||||
top = [(a, c) for a, c in ranked if c > 0][:top_n]
|
||||
if not top:
|
||||
top = ranked[: min(top_n, len(ranked))]
|
||||
labels = [asin_short.get(a, a) for a, _ in top]
|
||||
values = [c for _, c in top]
|
||||
parts.append(build_horizontal_bar_chart(
|
||||
f"{prefix}Mini{i}", labels, values, [color] * len(labels),
|
||||
height=200, asin_category_axis="y",
|
||||
))
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def _build_star_dist_echarts(
|
||||
chart_labels: List[str],
|
||||
star_data: Dict[str, Dict[int, int]],
|
||||
asins: List[str],
|
||||
*,
|
||||
element_id: str = "starDistChart",
|
||||
horizontal: bool = False,
|
||||
) -> str:
|
||||
"""评分分布堆叠图；大品类用横向堆叠。"""
|
||||
colors_5 = ["#22c55e", "#86efac", "#d1d5db", "#fbbf24", "#ef4444"]
|
||||
series = []
|
||||
for star in [5, 4, 3, 2, 1]:
|
||||
series.append({
|
||||
"name": f"{star}★",
|
||||
"type": "bar",
|
||||
"stack": "total",
|
||||
"data": [star_data.get(a, {}).get(star, 0) for a in asins],
|
||||
"itemStyle": {"color": colors_5[5 - star]},
|
||||
})
|
||||
if horizontal:
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"legend": {"top": 0, "textStyle": {"fontSize": 11}},
|
||||
"grid": {"left": 72, "right": 24, "top": 36, "bottom": 24},
|
||||
"xAxis": {"type": "value"},
|
||||
"yAxis": {
|
||||
"type": "category",
|
||||
"data": chart_labels,
|
||||
"inverse": True,
|
||||
"axisLabel": {"fontSize": 10, "width": 64, "overflow": "truncate"},
|
||||
},
|
||||
"series": series,
|
||||
}
|
||||
else:
|
||||
option = {
|
||||
"tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
|
||||
"legend": {"bottom": 0, "textStyle": {"fontSize": 11}},
|
||||
"grid": {"left": 50, "right": 20, "top": 20, "bottom": 40},
|
||||
"xAxis": {
|
||||
"type": "category",
|
||||
"data": chart_labels,
|
||||
"axisLabel": {"fontSize": 10, "rotate": 30 if len(chart_labels) > 8 else 0},
|
||||
},
|
||||
"yAxis": {"type": "value"},
|
||||
"series": series,
|
||||
}
|
||||
axis_key = "y" if horizontal else "x"
|
||||
return _echarts_init(element_id, option, asin_category_axis=axis_key)
|
||||
|
||||
|
||||
# ── 图表数据计算辅助 ──
|
||||
|
||||
_STRONG_HINTS = (
|
||||
"waste", "charge", "broken", "stopped", "doesn't", "does not", "not worth",
|
||||
"terrible", "horrible", "useless", "defect", "return", "refund", "pull",
|
||||
"nick", "burn", "bleed", "cut", "irritat", "bump", "overheat", "loud",
|
||||
"durable", "quality", "shave", "trim", "waterproof", "battery",
|
||||
)
|
||||
|
||||
|
||||
def _is_strong_keyword(kw: str) -> bool:
|
||||
if len(kw) >= 12:
|
||||
return True
|
||||
return any(h in kw for h in _STRONG_HINTS)
|
||||
|
||||
|
||||
def _match_theme_in_text(keywords: List[str], text: str) -> bool:
|
||||
from report_utils import match_text
|
||||
return match_text(keywords, text)
|
||||
|
||||
|
||||
def calc_keyword_group_freq(
|
||||
reviews: List[Any],
|
||||
keywords: List[str],
|
||||
is_neg: bool = True,
|
||||
) -> int:
|
||||
"""词组频次：组内任一词命中即计 1，同一评论只计 1 次；按星级过滤。"""
|
||||
kws = [kw.lower().strip() for kw in keywords if kw and kw.strip()]
|
||||
if not kws:
|
||||
return 0
|
||||
count = 0
|
||||
for r in reviews:
|
||||
if is_neg and r.rating > 2:
|
||||
continue
|
||||
if not is_neg and r.rating < 4:
|
||||
continue
|
||||
text = (r.title + " " + r.content).lower()
|
||||
if _match_theme_in_text(kws, text):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def _theme_matches_review(
|
||||
keywords: List[str],
|
||||
review: Any,
|
||||
extractions: Dict[int, Any],
|
||||
*,
|
||||
is_neg: bool,
|
||||
) -> bool:
|
||||
"""主题命中：优先结构化 category+aspect，回退原文 keyword。"""
|
||||
from report_utils import match_theme_extraction
|
||||
|
||||
kws = [kw.lower().strip() for kw in keywords if kw and kw.strip()]
|
||||
if not kws:
|
||||
return False
|
||||
ext = extractions.get(review.source_row)
|
||||
if ext and match_theme_extraction(kws, ext, is_neg=is_neg):
|
||||
return True
|
||||
text = (review.title + " " + review.content).lower()
|
||||
return _match_theme_in_text(kws, text)
|
||||
|
||||
|
||||
def calc_theme_freq(
|
||||
themes: List[Dict[str, Any]],
|
||||
reviews: List[Any],
|
||||
is_neg: bool = True,
|
||||
extractions: Dict[int, Any] | None = None,
|
||||
) -> Dict[str, int]:
|
||||
"""用主题 keywords 统计频次（结构化 category+aspect 优先，原文回退）。"""
|
||||
ext_map = extractions or {}
|
||||
freq: Dict[str, int] = {}
|
||||
for theme in themes:
|
||||
keywords = [kw.lower().strip() for kw in theme.get("keywords", []) if kw.strip()]
|
||||
count = 0
|
||||
for r in reviews:
|
||||
if is_neg and r.rating > 2:
|
||||
continue
|
||||
if not is_neg and r.rating < 4:
|
||||
continue
|
||||
if _theme_matches_review(keywords, r, ext_map, is_neg=is_neg):
|
||||
count += 1
|
||||
freq[theme["name"]] = count
|
||||
return freq
|
||||
|
||||
|
||||
def calc_per_asin_theme_freq(
|
||||
themes: List[Dict[str, Any]],
|
||||
reviews: List[Any],
|
||||
is_neg: bool = True,
|
||||
extractions: Dict[int, Any] | None = None,
|
||||
) -> Dict[str, Dict[str, int]]:
|
||||
"""按 ASIN 分别统计各主题频次（结构化优先）。"""
|
||||
from collections import defaultdict
|
||||
|
||||
ext_map = extractions or {}
|
||||
result: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int))
|
||||
for theme in themes:
|
||||
keywords = [kw.lower().strip() for kw in theme.get("keywords", []) if kw.strip()]
|
||||
name = theme["name"]
|
||||
for r in reviews:
|
||||
if is_neg and r.rating > 2:
|
||||
continue
|
||||
if not is_neg and r.rating < 4:
|
||||
continue
|
||||
if _theme_matches_review(keywords, r, ext_map, is_neg=is_neg):
|
||||
result[r.asin][name] += 1
|
||||
return {asin: dict(counts) for asin, counts in result.items()}
|
||||
461
voc_业务_2/llm_analyzer.py
Normal file
461
voc_业务_2/llm_analyzer.py
Normal file
|
|
@ -0,0 +1,461 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
LLM 分析模块：调用 DeepSeek LLM 完成 Persona/KANO/JTBD/根因分析。
|
||||
复用父目录 voc_llm.py 的 API Key 和 Client 配置。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple, TypeVar
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
_PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(_PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_PROJECT_ROOT))
|
||||
|
||||
from voc_llm import CHAT_MODEL, chat_extra_body, create_chat_client, require_chat_api_key
|
||||
from prompt_loader import llm_params, optional_block, system_prompt, user_prompt
|
||||
|
||||
logger = logging.getLogger("voc.llm_analyzer")
|
||||
|
||||
DEFAULT_MODEL = CHAT_MODEL
|
||||
REPORT_MODEL = "deepseek-v4-pro"
|
||||
REPORT_REASONING_EFFORT = "max"
|
||||
DEFAULT_LLM_WORKERS = 4
|
||||
|
||||
# 报告构建模式（build_report.py 调用 configure_report_llm 后启用 max thinking）
|
||||
_report_mode = False
|
||||
_report_model = REPORT_MODEL
|
||||
_report_reasoning_effort = REPORT_REASONING_EFFORT
|
||||
REPORT_MAX_TOKENS = 128000
|
||||
REPORT_TIMEOUT = 900.0
|
||||
|
||||
|
||||
def configure_report_llm(cfg: Optional[dict] = None) -> None:
|
||||
"""启用报告 LLM：deepseek-v4-pro + reasoning_effort=max。"""
|
||||
global _report_mode, _report_model, _report_reasoning_effort, REPORT_MAX_TOKENS
|
||||
cfg = cfg or {}
|
||||
_report_mode = True
|
||||
_report_model = (cfg.get("report_model") or REPORT_MODEL).strip() or REPORT_MODEL
|
||||
_report_reasoning_effort = (cfg.get("report_reasoning_effort") or REPORT_REASONING_EFFORT).strip() or "max"
|
||||
if cfg.get("report_max_tokens"):
|
||||
REPORT_MAX_TOKENS = max(8000, int(cfg["report_max_tokens"]))
|
||||
logger.info(
|
||||
"报告 LLM 已配置: model=%s, reasoning_effort=%s, max_tokens=%s",
|
||||
_report_model, _report_reasoning_effort, REPORT_MAX_TOKENS,
|
||||
)
|
||||
|
||||
|
||||
def get_llm_workers(cfg: Optional[dict] = None) -> int:
|
||||
"""LLM 并发线程数（config llm_max_workers 或环境变量 VOC_LLM_WORKERS）。"""
|
||||
if cfg and cfg.get("llm_max_workers"):
|
||||
return max(1, int(cfg["llm_max_workers"]))
|
||||
env = os.environ.get("VOC_LLM_WORKERS")
|
||||
if env:
|
||||
return max(1, int(env))
|
||||
return DEFAULT_LLM_WORKERS
|
||||
|
||||
|
||||
def _strip_think(text: str) -> str:
|
||||
if not text:
|
||||
return text
|
||||
text = re.sub(r"<think>[\s\S]*?</think>", "", text, flags=re.IGNORECASE)
|
||||
text = re.sub(r"</?think>", "", text, flags=re.IGNORECASE)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _call_llm(system, user, *, model=None, temperature=0.3, max_tokens=16000, timeout=300.0, reasoning=None):
|
||||
api_key = require_chat_api_key()
|
||||
use_report = _report_mode if reasoning is None else reasoning
|
||||
use_model = model or (_report_model if use_report else DEFAULT_MODEL)
|
||||
effective_max = max(max_tokens, REPORT_MAX_TOKENS) if use_report else max_tokens
|
||||
effective_timeout = max(timeout, REPORT_TIMEOUT) if use_report else timeout
|
||||
client = create_chat_client(api_key=api_key, timeout=effective_timeout)
|
||||
kwargs = {"model": use_model, "messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
], "max_tokens": effective_max}
|
||||
if use_report:
|
||||
kwargs["reasoning_effort"] = _report_reasoning_effort
|
||||
kwargs["extra_body"] = {"thinking": {"type": "enabled"}}
|
||||
else:
|
||||
kwargs["temperature"] = temperature
|
||||
eb = chat_extra_body(use_model)
|
||||
if eb:
|
||||
kwargs["extra_body"] = eb
|
||||
resp = client.chat.completions.create(**kwargs)
|
||||
msg = resp.choices[0].message
|
||||
content = msg.content or ""
|
||||
if not content.strip():
|
||||
content = getattr(msg, "reasoning_content", None) or ""
|
||||
if use_report and not content.strip():
|
||||
fr = getattr(resp.choices[0], "finish_reason", None)
|
||||
raise RuntimeError(
|
||||
f"报告 LLM content 为空（model={use_model}，finish_reason={fr!r}，max_tokens={effective_max}）；"
|
||||
"请增大 report_max_tokens 或降低 reasoning_effort"
|
||||
)
|
||||
return _strip_think(content)
|
||||
|
||||
|
||||
def _parse_json(text):
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
m = re.search(r"```(?:json)?\s*([\s\S]*?)```", text)
|
||||
if m:
|
||||
try:
|
||||
return json.loads(m.group(1).strip())
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
m = re.search(r"\{[\s\S]*\}", text)
|
||||
if m:
|
||||
try:
|
||||
return json.loads(m.group(0))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
logger.warning("无法解析 JSON: %s...", text[:200])
|
||||
return {}
|
||||
|
||||
|
||||
def _j(obj: Any) -> str:
|
||||
return json.dumps(obj, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
# ── Persona ──
|
||||
|
||||
def build_persona_prompt(cluster_data, product_name: str = "", industry: str = ""):
|
||||
catalog = cluster_data.get("persona_cluster_catalog") or []
|
||||
return user_prompt(
|
||||
"persona",
|
||||
product_name=product_name or "主产品",
|
||||
industry=industry or "当前品类",
|
||||
catalog_json=_j(catalog),
|
||||
audience_clusters_json=_j(cluster_data.get("audience_clusters", [])),
|
||||
global_pains_json=_j(cluster_data.get("global_pains", [])),
|
||||
global_negative_json=_j(cluster_data.get("global_negative", [])),
|
||||
global_positive_json=_j(cluster_data.get("global_positive", [])),
|
||||
)
|
||||
|
||||
|
||||
def discover_personas(
|
||||
cluster_data,
|
||||
product_name: str = "",
|
||||
industry: str = "",
|
||||
):
|
||||
logger.info("LLM: Persona发现（产品=%s）...", product_name or "主产品")
|
||||
params = llm_params("persona")
|
||||
raw = _call_llm(
|
||||
system_prompt("persona"),
|
||||
build_persona_prompt(cluster_data, product_name, industry),
|
||||
**params,
|
||||
)
|
||||
result = _parse_json(raw)
|
||||
personas = result.get("personas", [])
|
||||
logger.info("发现 %s 个Persona", len(personas))
|
||||
return personas
|
||||
|
||||
|
||||
# ── 主题 ──
|
||||
|
||||
def discover_themes(cluster_data, neg_review_count=0, theme_type="negative"):
|
||||
logger.info("LLM: 发现%s主题...", theme_type)
|
||||
data_key = "global_negative" if theme_type == "negative" else "global_positive"
|
||||
extra = ""
|
||||
if theme_type == "negative":
|
||||
extra = optional_block(
|
||||
"theme", "negative_extra",
|
||||
neg_review_count=neg_review_count,
|
||||
p0_threshold=int(neg_review_count * 0.2),
|
||||
)
|
||||
user = user_prompt(
|
||||
"theme",
|
||||
theme_type=theme_type,
|
||||
extra_block=extra,
|
||||
cluster_data_json=_j(cluster_data.get(data_key, [])),
|
||||
)
|
||||
params = llm_params("theme")
|
||||
raw = _call_llm(system_prompt("theme"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
themes = result.get("themes", [])
|
||||
logger.info("发现 %s 个%s主题", len(themes), theme_type)
|
||||
return themes
|
||||
|
||||
|
||||
def discover_themes_both(
|
||||
cluster_data: Dict[str, Any],
|
||||
neg_review_count: int,
|
||||
pos_review_count: int,
|
||||
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
|
||||
"""并发发现差评/好评主题。"""
|
||||
logger.info("LLM: 并发发现差评+好评主题...")
|
||||
with ThreadPoolExecutor(max_workers=2) as ex:
|
||||
f_neg = ex.submit(discover_themes, cluster_data, neg_review_count, "negative")
|
||||
f_pos = ex.submit(discover_themes, cluster_data, pos_review_count, "positive")
|
||||
return f_neg.result(), f_pos.result()
|
||||
|
||||
|
||||
def analyze_kano_jtbd_keywords_parallel(
|
||||
neg_themes: List[Dict[str, Any]],
|
||||
pos_themes: List[Dict[str, Any]],
|
||||
personas: List[Dict[str, Any]],
|
||||
neg_keyword_groups: List[Dict[str, Any]],
|
||||
pos_keyword_groups: List[Dict[str, Any]],
|
||||
product_name: str = "",
|
||||
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]], Dict[str, Any]]:
|
||||
"""并发执行 KANO、JTBD、情感关键词（三者互不依赖）。"""
|
||||
logger.info("LLM: 并发 KANO + JTBD + 情感关键词...")
|
||||
with ThreadPoolExecutor(max_workers=3) as ex:
|
||||
f_kano = ex.submit(analyze_kano, neg_themes, pos_themes, personas)
|
||||
f_jtbd = ex.submit(analyze_jtbd, personas)
|
||||
f_kw = ex.submit(
|
||||
analyze_keywords, neg_keyword_groups, pos_keyword_groups, personas, product_name,
|
||||
)
|
||||
return f_kano.result(), f_jtbd.result(), f_kw.result()
|
||||
|
||||
|
||||
def analyze_kano(neg_themes, pos_themes, personas):
|
||||
logger.info("LLM: KANO分析...")
|
||||
reverse_block = optional_block("kano", "reverse_search_block")
|
||||
user = user_prompt(
|
||||
"kano",
|
||||
reverse_search_block=reverse_block,
|
||||
neg_themes_json=_j(neg_themes),
|
||||
pos_themes_json=_j(pos_themes),
|
||||
personas_json=_j(personas),
|
||||
)
|
||||
params = llm_params("kano")
|
||||
raw = _call_llm(system_prompt("kano"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
kano = result.get("kano", [])
|
||||
logger.info("KANO: %s条", len(kano))
|
||||
return kano
|
||||
|
||||
|
||||
# ── JTBD ──
|
||||
|
||||
def analyze_jtbd(personas):
|
||||
logger.info("LLM: JTBD分析...")
|
||||
user = user_prompt(
|
||||
"jtbd",
|
||||
persona_count=len(personas),
|
||||
personas_json=_j(personas),
|
||||
)
|
||||
params = llm_params("jtbd")
|
||||
raw = _call_llm(system_prompt("jtbd"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
jtbd = result.get("jtbd", [])
|
||||
logger.info("JTBD: %s条", len(jtbd))
|
||||
return jtbd
|
||||
|
||||
|
||||
# ── 矩阵 ──
|
||||
|
||||
def analyze_matrix(personas, kano, cluster_data, total_reviews, market_avg: float = 0.0):
|
||||
_ = cluster_data
|
||||
logger.info("LLM: 矩阵分析...")
|
||||
market_hint = ""
|
||||
if market_avg and market_avg < 3.5:
|
||||
market_hint = optional_block("matrix", "market_low_hint", market_avg=market_avg)
|
||||
top_personas = personas[:4]
|
||||
user = user_prompt(
|
||||
"matrix",
|
||||
market_hint=market_hint,
|
||||
top_personas_json=_j(top_personas),
|
||||
kano_json=_j(kano),
|
||||
total_reviews=total_reviews,
|
||||
market_avg=market_avg,
|
||||
)
|
||||
params = llm_params("matrix")
|
||||
raw = _call_llm(system_prompt("matrix"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
matrix = result.get("matrix", [])
|
||||
logger.info("矩阵: %s行", len(matrix))
|
||||
return matrix
|
||||
|
||||
|
||||
# ── 根因 ──
|
||||
|
||||
def analyze_rootcause_per_persona(
|
||||
persona,
|
||||
neg_themes,
|
||||
bound_clusters: Optional[Dict[str, Any]] = None,
|
||||
product_name: str = "",
|
||||
industry: str = "",
|
||||
):
|
||||
name = persona.get("name", "?")
|
||||
theme_names = [t.get("name") for t in neg_themes if t.get("name")]
|
||||
product_ctx = product_name or "主产品"
|
||||
industry_ctx = industry or "当前品类"
|
||||
logger.info("LLM: 根因-%s...", name)
|
||||
# 构建 Persona 自身绑定簇的摘要数据（替代旧 per_aud_data）
|
||||
bc_data = bound_clusters or {}
|
||||
user = user_prompt(
|
||||
"rootcause",
|
||||
persona_name=name,
|
||||
product_name=product_ctx,
|
||||
industry=industry_ctx,
|
||||
persona_json=_j(persona),
|
||||
bound_clusters_json=_j(bc_data),
|
||||
theme_names_json=_j(theme_names),
|
||||
)
|
||||
params = llm_params("rootcause")
|
||||
raw = _call_llm(system_prompt("rootcause"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
return result
|
||||
|
||||
|
||||
def analyze_all_rootcauses(
|
||||
personas: List[Dict[str, Any]],
|
||||
neg_themes: List[Dict[str, Any]],
|
||||
all_clusters: Optional[Dict[str, List[Any]]] = None,
|
||||
max_workers: int = DEFAULT_LLM_WORKERS,
|
||||
product_name: str = "",
|
||||
industry: str = "",
|
||||
min_hit_count: int = 5,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""并发按 Persona 根因分析；跳过命中不足的 Persona。不再依赖 per_audience。"""
|
||||
|
||||
def _bound_clusters_for(persona: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""从 Persona 自身 cluster_refs 构建绑定簇的摘要数据。"""
|
||||
refs = persona.get("cluster_refs") or {}
|
||||
bc_data: Dict[str, Any] = {}
|
||||
if isinstance(refs, dict):
|
||||
for dim, ref in refs.items():
|
||||
if isinstance(ref, dict) and ref.get("stage") and ref.get("label") is not None:
|
||||
stage = ref["stage"]
|
||||
label = ref["label"]
|
||||
# 尝试从 all_clusters 中查找实际簇数据
|
||||
if all_clusters and stage in all_clusters:
|
||||
for c in all_clusters[stage]:
|
||||
if int(getattr(c, "cluster_label", -1)) == int(label):
|
||||
bc_data[f"{dim}_dim"] = {
|
||||
"stage": stage,
|
||||
"label": label,
|
||||
"top_phrases": getattr(c, "top_phrases", [])[:10],
|
||||
}
|
||||
break
|
||||
return bc_data
|
||||
|
||||
eligible = [
|
||||
(i, p) for i, p in enumerate(personas)
|
||||
if p.get("hit_count", 0) >= min_hit_count
|
||||
]
|
||||
n = len(eligible)
|
||||
if n == 0:
|
||||
logger.warning("无 Persona 达到根因分析命中阈值（min_hit_count=%s）", min_hit_count)
|
||||
return [
|
||||
{"persona_name": p.get("name", f"P{i}"), "persona_index": i,
|
||||
"root_causes": [], "affected_themes": [], "skipped": True,
|
||||
"skip_reason": f"聚类命中 {p.get('hit_count', 0)} 条，低于阈值 {min_hit_count}"}
|
||||
for i, p in enumerate(personas)
|
||||
]
|
||||
|
||||
workers = max(1, min(max_workers, n))
|
||||
logger.info("LLM: 并发根因分析 %s/%s 个 Persona（workers=%s）...", n, len(personas), workers)
|
||||
|
||||
def _one(idx: int, persona: Dict[str, Any]) -> Dict[str, Any]:
|
||||
bc_data = _bound_clusters_for(persona)
|
||||
rc = analyze_rootcause_per_persona(
|
||||
persona, neg_themes,
|
||||
bound_clusters=bc_data,
|
||||
product_name=product_name, industry=industry,
|
||||
)
|
||||
rc["persona_name"] = persona.get("name", f"P{idx}")
|
||||
rc["persona_index"] = idx
|
||||
rc["skipped"] = False
|
||||
return rc
|
||||
|
||||
results_map: Dict[int, Dict[str, Any]] = {}
|
||||
with ThreadPoolExecutor(max_workers=workers) as ex:
|
||||
futures = {ex.submit(_one, i, p): (i, p) for i, p in eligible}
|
||||
for fut in as_completed(futures):
|
||||
i, _ = futures[fut]
|
||||
results_map[i] = fut.result()
|
||||
|
||||
ordered: List[Dict[str, Any]] = []
|
||||
for i, p in enumerate(personas):
|
||||
if i in results_map:
|
||||
ordered.append(results_map[i])
|
||||
else:
|
||||
ordered.append({
|
||||
"persona_name": p.get("name", f"P{i}"),
|
||||
"persona_index": i,
|
||||
"root_causes": [],
|
||||
"affected_themes": [],
|
||||
"skipped": True,
|
||||
"skip_reason": f"聚类命中 {p.get('hit_count', 0)} 条，低于阈值 {min_hit_count}",
|
||||
})
|
||||
return ordered
|
||||
|
||||
|
||||
# ── 情感关键词 ──
|
||||
|
||||
def analyze_keywords(
|
||||
neg_groups: List[Dict[str, Any]],
|
||||
pos_groups: List[Dict[str, Any]],
|
||||
personas,
|
||||
product_name: str = "",
|
||||
):
|
||||
logger.info("LLM: 情感关键词（差评+好评词组）...")
|
||||
skip_hint = "product/item/the/and 及品类核心词"
|
||||
if product_name:
|
||||
skip_hint += f"；当前产品「{product_name}」相关词"
|
||||
user = user_prompt(
|
||||
"keyword",
|
||||
neg_groups_json=_j(neg_groups),
|
||||
pos_groups_json=_j(pos_groups),
|
||||
personas_json=_j(personas),
|
||||
skip_hint=skip_hint,
|
||||
)
|
||||
params = llm_params("keyword")
|
||||
raw = _call_llm(system_prompt("keyword"), user, **params)
|
||||
result = _parse_json(raw)
|
||||
if "negative" in result or "positive" in result:
|
||||
logger.info("情感词: 差评 %s 组 / 好评 %s 组", len(result.get("negative", [])), len(result.get("positive", [])))
|
||||
return result
|
||||
legacy = result.get("keywords", [])
|
||||
logger.info("情感词(legacy): %s条", len(legacy))
|
||||
return {"negative": legacy, "positive": []}
|
||||
|
||||
|
||||
|
||||
# ── 产品/行业自动识别 ──
|
||||
|
||||
def detect_product_and_industry(sample_reviews: list, dir_name: str = "") -> tuple[str, str]:
|
||||
"""从样本评论和目录名中自动识别产品名和行业。返回 (product_name, industry)。"""
|
||||
logger.info("LLM: 识别产品/行业...")
|
||||
sample_text = "\n---\n".join(
|
||||
f"[{i+1}] {r[:300]}" for i, r in enumerate(sample_reviews[:50])
|
||||
)
|
||||
dir_info = ""
|
||||
if dir_name:
|
||||
dir_info = f'\n## 数据来源目录名\n> {dir_name}\n（目录名可能包含产品/品类关键词）\n'
|
||||
user = user_prompt("product_detect", sample_text=sample_text, dir_info=dir_info)
|
||||
params = llm_params("product_detect")
|
||||
raw = _call_llm(system_prompt("product_detect"), user, **params, reasoning=False)
|
||||
result = _parse_json(raw)
|
||||
product = result.get("product_name", "").strip()
|
||||
industry = result.get("industry", "亚马逊电商").strip()
|
||||
logger.info("识别结果: 产品=%s | 行业=%s", product, industry)
|
||||
return product or "亚马逊商品", industry or "亚马逊电商"
|
||||
|
||||
# ── 市场竞争 ──
|
||||
|
||||
def market_competition_judgment(weighted_avg):
|
||||
if weighted_avg < 3.5:
|
||||
return ("系统性缺陷 · 新品进入窗口期",
|
||||
f"加权均分 {weighted_avg}（低于 3.5），按方法论判定为「市场存在严重系统性缺陷，是新品进入的明确窗口期」。")
|
||||
elif weighted_avg <= 4.0:
|
||||
return ("有改进空间",
|
||||
f"加权均分 {weighted_avg}（3.5–4.0 区间），属于「市场有改进空间，部分功能存在普遍短板」。")
|
||||
else:
|
||||
return ("市场成熟",
|
||||
f"加权均分 {weighted_avg}（高于 4.0），属于「市场整体较成熟，需通过差异化或细分切入」。")
|
||||
795
voc_业务_2/output/bikini-trimmer分析结果.html
Normal file
795
voc_业务_2/output/bikini-trimmer分析结果.html
Normal file
File diff suppressed because one or more lines are too long
71
voc_业务_2/prompt_loader.py
Normal file
71
voc_业务_2/prompt_loader.py
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
"""从 prompts.yaml 加载报告 LLM 提示词，供 llm_analyzer 使用。"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
logger = logging.getLogger("voc.prompt_loader")
|
||||
|
||||
PROMPTS_FILE = Path(__file__).resolve().parent / "prompts.yaml"
|
||||
|
||||
|
||||
def _render(template: str, **kwargs: Any) -> str:
|
||||
"""将 {{key}} 替换为值；模板内 JSON 示例的花括号无需转义。"""
|
||||
out = template
|
||||
for key, val in kwargs.items():
|
||||
out = out.replace("{{" + key + "}}", str(val))
|
||||
return out
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def load_prompts(path: Optional[str] = None) -> Dict[str, Any]:
|
||||
fp = Path(path) if path else PROMPTS_FILE
|
||||
if not fp.is_file():
|
||||
raise FileNotFoundError(f"提示词文件不存在: {fp}")
|
||||
with fp.open(encoding="utf-8") as f:
|
||||
data = yaml.safe_load(f) or {}
|
||||
logger.debug("已加载提示词: %s", fp)
|
||||
return data
|
||||
|
||||
|
||||
def reload_prompts() -> None:
|
||||
load_prompts.cache_clear()
|
||||
|
||||
|
||||
def get_section(name: str, path: Optional[str] = None) -> Dict[str, Any]:
|
||||
prompts = load_prompts(path)
|
||||
sec = prompts.get(name)
|
||||
if not sec:
|
||||
raise KeyError(f"prompts.yaml 缺少段落: {name}")
|
||||
return sec
|
||||
|
||||
|
||||
def system_prompt(name: str, path: Optional[str] = None) -> str:
|
||||
return (get_section(name, path).get("system") or "").strip()
|
||||
|
||||
|
||||
def llm_params(name: str, path: Optional[str] = None) -> Dict[str, Any]:
|
||||
sec = get_section(name, path)
|
||||
return {
|
||||
"temperature": float(sec.get("temperature", 0.3)),
|
||||
"max_tokens": int(sec.get("max_tokens", 8000)),
|
||||
}
|
||||
|
||||
|
||||
def user_prompt(name: str, path: Optional[str] = None, **kwargs: Any) -> str:
|
||||
tpl = (get_section(name, path).get("user_template") or "").strip()
|
||||
return _render(tpl, **kwargs)
|
||||
|
||||
|
||||
def optional_block(name: str, block_key: str, path: Optional[str] = None, **kwargs: Any) -> str:
|
||||
"""加载可选子模板（如差评主题 extra 块）。"""
|
||||
sec = get_section(name, path)
|
||||
tpl = (sec.get(block_key) or "").strip()
|
||||
if not tpl:
|
||||
return ""
|
||||
return _render(tpl, **kwargs)
|
||||
364
voc_业务_2/prompts.yaml
Normal file
364
voc_业务_2/prompts.yaml
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
# VOC 报告 LLM 提示词配置
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# 业务员修改指南：
|
||||
# 1. 直接编辑本文件中的 system / user_template / 子模板
|
||||
# 2. 保留 {{占位符}} 不动（运行时自动填入聚类数据等）
|
||||
# 3. 保存后重新运行 build_report.py 或 run_pipeline.py 即可生效
|
||||
# 4. 勿改 JSON 输出字段名（如 personas、themes、kano），否则报告解析失败
|
||||
# 说明文档（业务员改）：提示词编辑稿.md
|
||||
# 方法论依据：VOC分析方法论与报告生成逻辑.md v1.0
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
|
||||
meta:
|
||||
version: "1.2"
|
||||
updated: "2026-06-12"
|
||||
disclaimer: |
|
||||
本文件中的中文/英文示例仅说明输出格式与分析逻辑。
|
||||
实际 Persona、主题名、keywords、场景、KANO 条目、根因须来自当前批次聚类数据与注入 JSON，禁止照搬示例文字。
|
||||
|
||||
# ── 0. 产品/行业自动识别（pipeline 产品名为空时）──
|
||||
product_detect:
|
||||
temperature: 0.2
|
||||
max_tokens: 1000
|
||||
system: |
|
||||
你是亚马逊商品识别专家。根据用户评论内容推断产品名称和所属行业。输出严格JSON，不输出多余文字。
|
||||
user_template: |
|
||||
根据以下亚马逊评论内容和目录名，推断产品名称和所属行业。
|
||||
|
||||
## 评论样本
|
||||
{{sample_text}}
|
||||
{{dir_info}}
|
||||
## 输出JSON
|
||||
{"product_name": "推断的产品名称（中英文均可，简洁描述，如 Kitchen Blender / Pet Repellent Spray）", "industry": "所属行业（中文，如 个人护理/宠物用品/厨房家电/健康补充剂 等）"}
|
||||
|
||||
要求：
|
||||
- product_name 用评论中高频提及的核心产品词命名，不用品牌名或 ASIN；不要包含行业大类词
|
||||
- industry 用一个中文行业大类词
|
||||
- 目录名中的关键词可作为重要参考线索
|
||||
- JSON 示例仅说明格式，product_name 须从上方评论样本归纳
|
||||
|
||||
# ── 1. 用户画像 Persona ──
|
||||
persona:
|
||||
temperature: 0.4
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是资深的消费者洞察专家，严格遵循 VOC 分析方法论 v1.0 第 3.2 节与 4.0 节。
|
||||
Persona 命名用简洁中文（≤6字），避免营销化夸张名称。输出严格JSON。
|
||||
示例仅说明格式；Persona 名与 keywords 须来自当前聚类 top_phrases 与 sample_reviews，禁止照搬示例。
|
||||
user_template: |
|
||||
## 任务：基于聚类数据按三维度发现用户画像（Persona），4-7 个。
|
||||
|
||||
## 分析产品（product_detect / config 已识别，Persona 须针对该品类）
|
||||
产品：{{product_name}} | 行业：{{industry}}
|
||||
purchase_motivation 用「雇佣{{product_name}}完成…」JTBD 句式；Persona 痛点/需求须与该产品使用场景一致，禁止脱离品类写无关人群。
|
||||
|
||||
## 三维度强制覆盖（缺一不可）
|
||||
A. 物理/生理/受众特征 → 从 catalog 中 suggested_dimension="A" 的簇中选择
|
||||
典型信号：特定体质/肤质/年龄/体型/使用对象（如 sensitive skin / coarse hair / pregnant / bikini area）
|
||||
B. 行为/使用场景 / 具体痛点 → 从 catalog 中 suggested_dimension="B" 的簇中选择
|
||||
典型信号：travel / first time / pulls hair / battery dead / durability
|
||||
C. 购买动机/背景 → 从 catalog 中 suggested_dimension="C" 的簇中选择
|
||||
典型信号：switched from / saw on social media / too expensive
|
||||
→ 每个 Persona 必须同时绑 B 或 C 维的具体痛点/需求簇，用于精准计算占比。
|
||||
|
||||
## 自我标注信号强制检查（归纳后必做）
|
||||
在绑定簇的 top_phrases / sample_reviews 中搜索以下模式，出现 ≥5 条且 eligible_physio_labels 达标时才可建对应生理 Persona：
|
||||
- "I have [adj] [noun]"（如 I have sensitive skin）
|
||||
- "My [noun] is/are [adj]"
|
||||
- "As a [noun]"
|
||||
- 若 eligible_physio_labels 为空或不包含对应维度，禁止在名称/core_pain 写生理标签
|
||||
|
||||
## 其他遗漏检查
|
||||
- 长期使用/复购用户：含 after months / second bottle / bought again 的差评是否形成独立群体
|
||||
- 占比低（~5%）但痛点独特、无法被其他 Persona 代表的群体，仍须单独列出
|
||||
|
||||
## 可绑定的聚类簇目录（cluster_refs 必须从中选择）
|
||||
每簇含：semantic_label、suggested_dimension（A/B/C建议维度）、eligible_physio_labels（仅≥5条佐证的生理维度）、top_phrases、sample_reviews
|
||||
{{catalog_json}}
|
||||
|
||||
## 生理标签硬规则（进 LLM 前已过滤，仅 eligible_physio_labels 中的维度可用于命名）
|
||||
- catalog 仅展示 eligible_physio_labels（评论佐证≥5条）；未出现的生理标签禁止写入名称或 core_pain
|
||||
- top_phrases 仅出现 1–4 次不算数；不得凭品类常识脑补
|
||||
- sample_reviews 为溯源原文，core_pain 只能归纳其中明确出现的内容
|
||||
|
||||
## 补充聚类摘要
|
||||
audience_clusters: {{audience_clusters_json}}
|
||||
global_pains: {{global_pains_json}}
|
||||
global_negative: {{global_negative_json}}
|
||||
global_positive: {{global_positive_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"personas":[{"name":"≤6中文字","dimension":"A","cluster_refs":{"A":{"stage":"3b_aspect_opinion_negative","label":2},"B":{"stage":"3b_aspect_opinion_negative","label":5},"C":{"stage":"3a_pain_global","label":1}},"keywords":["从绑定簇 top_phrases 复制"],"core_pain":"核心痛点（≤3条，分号分隔）","core_need":"核心需求（≤3条，分号分隔）","purchase_motivation":"雇佣产品做什么（动词+宾语，JTBD句式）"}]}
|
||||
|
||||
## 硬性要求
|
||||
- 每个 Persona 必须有 cluster_refs；A/B/C 三维均从 suggested_dimension 匹配的簇中选择
|
||||
- A 维 stage 允许: 3a_pain_global / 3b_aspect_opinion_negative / 3b_aspect_opinion_positive（从 suggested_dimension="A" 的簇中选）
|
||||
- B 维 stage 允许: 3a_pain_global / 3b_aspect_opinion_negative / 3b_aspect_opinion_positive（从 suggested_dimension="B" 的簇中选）
|
||||
- C 维 stage 允许: 3a_pain_global / 3b_aspect_opinion_negative / 3b_aspect_opinion_positive（从 suggested_dimension="C" 的簇中选）
|
||||
- 每个 Persona 必须至少绑 1 个 B 或 C 维簇（具体痛点/需求/场景），用于 keyword 过滤占比
|
||||
- cluster_refs 各维 label 必须是上方 catalog 中存在的 label；绑定前先看 semantic_label 与 suggested_dimension
|
||||
- 系统以 A 维锚点簇 + B/C 维 keywords 过滤计算占比
|
||||
- dimension 表示该 Persona 主类型（A/B/C 之一）；cluster_refs.A 始终必填
|
||||
- keywords 至少 5 个，必须从所有已绑定簇的 top_phrases 复制英文片段（≥3字符），禁止编造不在簇中的词
|
||||
- 4-7 个 Persona，三维度 A/B/C 均至少覆盖 1 个；小簇（<15条）仅在有明确短语证据时使用
|
||||
- core_pain 来自该群体差评语义；core_need 来自该群体好评或诉求；purchase_motivation 用「雇佣产品完成…」句式
|
||||
- 命名须从聚类归纳；生理类命名仅当 eligible_physio_labels 含对应维度时才允许
|
||||
- 禁止在 eligible_physio_labels 未列出的生理标签写入名称或 core_pain
|
||||
|
||||
# ── 2. 差评/好评主题 ──
|
||||
theme:
|
||||
temperature: 0.3
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是亚马逊 VOC 分析专家，遵循方法论 3.3 节。
|
||||
从聚类短语归纳主题；根因/失效机制不同的问题必须独立成主题，禁止笼统合并。
|
||||
主题名用简洁中文（≤8字），适用于任意品类。输出严格JSON。
|
||||
示例仅说明拆分逻辑；主题名与 keywords 须来自当前聚类数据，禁止照搬示例主题名。
|
||||
negative_extra: |
|
||||
## 优先级（共 {{neg_review_count}} 条差评，系统会按实际频次与竞品覆盖率重算）
|
||||
- P0：频次 ≥ {{p0_threshold}} 条 且 80%+ 竞品均出现
|
||||
- P1：频次为差评总数 10–20% 且 60%+ 竞品出现
|
||||
- P2：频次为差评总数 3–10% 且 40%+ 竞品出现
|
||||
|
||||
## 拆分红线（方法论 3.3）
|
||||
判断问题：「同一主题下的差评，是否描述同一个物理/工程原因？」若不是，必须拆开。
|
||||
典型拆分（机制/根因不同须拆开，勿照搬下列中文名）：
|
||||
- 核心效果未达预期 → ①效果弱/不明显 ②使用方式与预期不符（机制不同）
|
||||
- 使用过程不适 → ①物理伤害/刺激 ②过热/异味/过敏（根因不同）
|
||||
- 产品失效/损坏 → ①供电/充电问题 ②结构件断裂/脱落（失效环节不同）
|
||||
user_template: |
|
||||
## 任务：归纳 {{theme_type}} 主题（适用于当前品类，勿预设具体产品类型）
|
||||
{{extra_block}}
|
||||
|
||||
## 归纳原则
|
||||
- 从 top_phrases 中归纳 4–8 个主题，差评主题应覆盖 80%+ 差评内容（长尾可合并为「其他」）
|
||||
- keywords 语义相近但失效机制不同 → 必须拆成独立主题
|
||||
- 好评主题可与差评维度对应但用正向表述（如 核心使用体验 ↔ 核心效果未达预期）
|
||||
- 示例主题名仅作格式与拆分参考；实际 name/keywords 必须来自上方 cluster 数据的 top_phrases
|
||||
|
||||
## 数据
|
||||
{{cluster_data_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"themes":[{"name":"≤8中文字","keywords":["english phrase from top_phrases"],"priority":"P0/P1/P2（仅差评）","description":"差评必填：与相近主题的根因拆分理由，格式「与XX根因不同：…」；好评可写满意点一句话"}]}
|
||||
|
||||
## 硬性要求
|
||||
- keywords 必须是英文，从 top_phrases 中复制完整片段或逗号后的子句（≥3字符）
|
||||
- 每个主题至少 5 个 keywords
|
||||
- 差评 priority 按上方门槛初步标注（系统会重算）
|
||||
- 差评示例（格式参考，须从 top_phrases 归纳）：效果未达预期、使用不适、供电失效、结构损坏、性价比低
|
||||
- 好评示例（格式参考）：核心使用体验、产品质量耐用、便携与设计、易用性、超预期惊喜
|
||||
|
||||
# ── 3. KANO 需求分类 ──
|
||||
kano:
|
||||
temperature: 0.3
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是产品需求分析专家，专精 KANO 模型，遵循方法论 4.1 节。
|
||||
四象限分类；每个条目 5 字段缺一不可。输出严格JSON。
|
||||
KANO 示例仅说明四象限判断；item/evidence 须来自当前差评/好评主题 keywords，禁止照搬示例。
|
||||
reverse_search_block: |
|
||||
## 反向型主动搜索（步骤三，不得以「未发现」一笔带过）
|
||||
在差评/好评主题 keywords 中检索以下词组及同义表达，记录出现频次：
|
||||
- 过于复杂：too many parts / too complicated / confusing / hard to use
|
||||
- 过于嘈杂：too loud / so loud / noise / noisy
|
||||
- 功能多余：don't need / unnecessary / didn't ask for / useless feature
|
||||
- 操作繁琐：takes too long / too many steps / annoying to clean
|
||||
处理规则：
|
||||
- 任一词组频次 ≥5 → 输出 reverse 类型条目并附 evidence
|
||||
- 全部词组总频次 <5 → 必须输出 1 条 type=reverse 的占位条目，item 写「本品类暂无明确反向需求」，evidence 写「经主动搜索 [列出搜索词]，共 N 条，低于阈值 5」
|
||||
user_template: |
|
||||
## 任务：KANO 四象限分类
|
||||
|
||||
## 分类标准（含常见误判）
|
||||
- 基本型 Must-be：P0 级差评 + 好评中几乎无人因「做到了 X」而表扬
|
||||
❌ 误判：核心性能指标（好坏都会被提及）→ 期望型
|
||||
✅ 正确：开箱即能用、供电正常、关键部件不脱落
|
||||
- 期望型 Performance：好评差评均出现，做得越好评分越高
|
||||
❌ 误判：续航/容量 → 基本型（超长续航会被特别称赞）
|
||||
✅ 正确：核心效果、续航/容量、易清洁程度
|
||||
- 魅力型 Attractive：好评中出现 love/obsessed/amazing/didn't expect/bonus；差评中几乎不出现
|
||||
❌ 误判：附赠配件 → 期望型（无人因缺该配件差评）
|
||||
✅ 正确：电量/状态显示、超预期配件、意外惊喜功能
|
||||
- 反向型 Reverse：用户主动抱怨某「功能」是负担（功能过载/太吵/太复杂）
|
||||
❌ 误判：产品损坏/充电故障 → 基本型（无人「希望产品损坏」）
|
||||
|
||||
{{reverse_search_block}}
|
||||
|
||||
## 操作步骤
|
||||
1. 基本型：从 P0/P1 差评主题出发，检查好评是否几乎无人表扬该点
|
||||
2. 期望型 vs 魅力型：差评有人因「不够好」→ 期望型；好评有 love/amazing 且竞品普遍缺失 → 魅力型
|
||||
3. 反向型：执行上方主动搜索，按规则输出
|
||||
|
||||
## 差评主题
|
||||
{{neg_themes_json}}
|
||||
## 好评主题
|
||||
{{pos_themes_json}}
|
||||
## Persona
|
||||
{{personas_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"kano":[{"type":"must-be/performance/attractive/reverse","item":"单条需求（动词+名词，禁止用逗号/顿号合并多条）","evidence":"评论频次证据+代表性英文片段（≤80字）","affected_persona":"主要 Persona","reason":"≤60字，解释为何是该类型而非其他类型","competitor_status":"竞品是否满足及满足程度"}]}
|
||||
|
||||
## 硬性要求
|
||||
- 每条 item 只写一条需求，禁止在 item 中用逗号合并
|
||||
- 至少 8 条记录；must-be / performance / attractive 均需覆盖；reverse 按搜索规则输出（含占位条目）
|
||||
- evidence 须含频次级别（如 P0/P1）或条数估计 + 英文原文片段
|
||||
- 质量缺陷、充电故障、配件脱落 → must-be，不是 reverse
|
||||
- reverse 仅限用户主动排斥功能过载（too many parts / too complicated / too loud 等）
|
||||
|
||||
# ── 4. JTBD 动机框架 ──
|
||||
jtbd:
|
||||
temperature: 0.3
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是 JTBD 分析专家，遵循方法论 4.2 节。
|
||||
为每个 Persona 构建 Jobs To Be Done 框架；所有动机字段须能从 Persona 的 keywords/core_pain/core_need/purchase_motivation 中找到语义佐证。
|
||||
无佐证时该字段填 "-"。必须覆盖全部 Persona。输出严格JSON。
|
||||
user_template: |
|
||||
## 任务：为 {{persona_count}} 个 Persona 构建 JTBD
|
||||
{{personas_json}}
|
||||
|
||||
## 填写规则
|
||||
| 字段 | 规则 |
|
||||
| core_job | 动词+宾语，用户想完成的任务；须与 Persona 数据语义一致 |
|
||||
| functional_motivation | 实用层面驱动（效率/效果/成本/便携），禁止写情感词 |
|
||||
| emotional_motivation | 情绪/心理驱动（自信/焦虑/掌控感/安心），禁止写功能词 |
|
||||
| social_motivation | 他人视角/社交驱动（如送礼、伴侣评价、公开场合）；无评论佐证填 "-" |
|
||||
| trigger | 购买触发时机，须来自 Persona 数据中的具体事件描述；无佐证填 "-" |
|
||||
|
||||
## 输出JSON
|
||||
{"jtbd":[{"persona":"名称","core_job":"核心Job","functional_motivation":"功能性动机","emotional_motivation":"情感性动机","social_motivation":"社会性动机或-","trigger":"触发时机或-"}]}
|
||||
|
||||
## 硬性要求
|
||||
- 必须覆盖全部 {{persona_count}} 个 Persona，一人一行
|
||||
- functional 与 emotional 字段内容不可互换
|
||||
- 所有字段（含 core_job / trigger）须与对应 Persona 的 keywords / core_pain / core_need / purchase_motivation 语义一致
|
||||
- 无法从 Persona 数据找到佐证的字段一律填 "-"，禁止编造评论中无依据的内容
|
||||
|
||||
# ── 5. 人群×场景×需求矩阵 ──
|
||||
matrix:
|
||||
temperature: 0.3
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是消费者洞察专家，构建人群×场景×需求矩阵，严格遵循方法论 4.3 节与 5.3 节场景规则。
|
||||
场景只允许填写评论中有原词佐证的描述；找不到佐证则不输出该行。
|
||||
输出严格JSON。全市场均分低时，细分场景「高满意度」须谨慎标注。
|
||||
场景示例仅说明格式；scene 须来自 Persona keywords 中的英文原词佐证，禁止照搬示例。
|
||||
market_low_hint: |
|
||||
## 重要：全市场加权均分 {{market_avg}}（<3.5），多数场景 satisfaction 应为「中等」或「低」，慎用「高」。
|
||||
user_template: |
|
||||
## 任务：为以下 Persona 构建矩阵（仅输出这些 Persona，最多 4 个）
|
||||
|
||||
## 场景规则（方法论 5.3，核心规则）
|
||||
【强制】scene 只允许填写 Persona 的 keywords / 聚类短语中能找到英文原词佐证的场景。
|
||||
禁止基于产品功能、品类常识或逻辑推断填写场景。
|
||||
- ≥5 条评论出现该场景词 → 可填写（如 travel / outdoor / kitchen / office — 以 top_phrases 为准）
|
||||
- 2–4 条 → 不输出该行
|
||||
- <2 条 → 不输出该行（禁止输出 scene 为 — 的行）
|
||||
|
||||
错误示例：
|
||||
- 日常使用（daily 是使用频率非场景）
|
||||
- 节日前突击（评论无对应原词）
|
||||
- 任何场所（泛化代替留空）
|
||||
正确示例：具体地点/情境（须在 keywords 中找到英文原词且 ≥5 条）
|
||||
|
||||
## 需求列映射
|
||||
- must_be_needs：KANO 基本型 + 该群体 P0 差评主题名
|
||||
- performance_needs：KANO 期望型 + 该群体 P1 差评主题名
|
||||
- attractive_needs：KANO 魅力型 + 该群体好评加分点
|
||||
|
||||
## 满意度评级
|
||||
- 高：该群体均分参考 ≥4.0
|
||||
- 中等：3.3–3.9
|
||||
- 低：＜3.3
|
||||
|
||||
## 每个 Persona 最多 2 个有效场景行
|
||||
{{market_hint}}
|
||||
{{top_personas_json}}
|
||||
{{kano_json}}
|
||||
## 总评论数: {{total_reviews}} · 全市场均分: {{market_avg}}
|
||||
|
||||
## 输出JSON
|
||||
{"matrix":[{"persona":"名称","pct":35,"scene":"具体场景（须有评论原词佐证）","must_be_needs":"基本型关键词","performance_needs":"期望型关键词","attractive_needs":"魅力型关键词","satisfaction":"高/中等/低","satisfaction_note":"≤25字"}]}
|
||||
|
||||
## 硬性要求
|
||||
- 禁止输出 scene 为 —、-、N/A 或空的行
|
||||
- 每个 Persona 最多 2 行；全报告最多 4 个 Persona
|
||||
- must_be_needs / performance_needs / attractive_needs 各 ≤12 字，用顿号分隔关键词，禁止完整句子
|
||||
- must_be_needs 须使用差评主题中文名（来自当前批次主题，非示例）
|
||||
- satisfaction_note ≤25 字
|
||||
|
||||
# ── 6. 痛点根因分析 ──
|
||||
rootcause:
|
||||
temperature: 0.4
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是产品工程与消费者洞察专家，遵循方法论 4.4 节。
|
||||
痛点根因分析须从现象到达机制层（非「质量差」类空话）；开发方向须可落地。
|
||||
分析对象是当前品类主产品（见用户消息），禁止把其他品类工具问题当作本产品根因。
|
||||
输出严格JSON，使用中文。
|
||||
根因示例仅说明分析深度；title/mechanism/dev_direction 须针对当前产品与注入主题，禁止照搬示例。
|
||||
user_template: |
|
||||
## 任务：{{persona_name}} 的痛点根因分析（2-3 条，针对 P0/P1 级痛点）
|
||||
|
||||
## 产品范围：{{product_name}}（行业：{{industry}}）
|
||||
## 分析边界：只分析该品类产品本身的结构/功能/体验缺陷
|
||||
|
||||
## 分析框架（每条根因）
|
||||
根因标题 → 导致后果（关联差评主题×频次）→ 失效机制（从结构/材料/工作原理解释）→ 可落地改进
|
||||
|
||||
## 层次要求
|
||||
- ❌ 现象层：「产品质量差」「用户体验不好」
|
||||
- ✅ 机制层：「密封/接口设计不足导致进水腐蚀」「关键部件角度/间距不当导致效果未达预期」
|
||||
|
||||
{{persona_json}}
|
||||
{{bound_clusters_json}}
|
||||
|
||||
## 可用差评主题名（affected_themes 只能从中选择）
|
||||
{{theme_names_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"root_causes":[{"title":"根因标题（≤20字）","mechanism":"失效机制（≤120字，白话，禁止医学/化学术语堆砌）","quote_keywords":["用于匹配评论的英文词"],"dev_direction":"可落地改进（≤80字：结构/材料/工艺/说明/品控等）"}],"affected_themes":["主题名"]}
|
||||
|
||||
## 硬性要求
|
||||
- quotes 字段不要输出（引用由系统从真实评论回填）
|
||||
- quote_keywords 每条根因 2-4 个**买家评论常见英文词/短语**（如 pull, dull, broke, charge, shower, waterproof, loud, trim, smooth, irritation, snag, waste），须与 mechanism 语义相关
|
||||
- quote_keywords 禁止生僻工程术语（如 O-ring、IPX7、DLC、magnetic charging、martensitic）；mechanism 可写工程细节，quote_keywords 必须像亚马逊买家口语
|
||||
- 优先从 bound_clusters 聚类 top_phrases 中选取真实出现的英文片段
|
||||
- affected_themes 只能使用上方差评主题名，优先 P0/P1 主题
|
||||
- dev_direction 须针对 {{product_name}} 可改进点
|
||||
- 每个 Persona 最多 3 条 root_causes
|
||||
|
||||
# ── 7. 情感关键词 ──
|
||||
keyword:
|
||||
temperature: 0.3
|
||||
max_tokens: 8000
|
||||
system: |
|
||||
你是 VOC 文本分析专家，遵循方法论 3.4 节。
|
||||
为已统计好的差评/好评词组补充语境与极性标签；count 由系统提供不可修改。
|
||||
meaning 只写评论中明确出现的语境，禁止推断。输出严格JSON。
|
||||
user_template: |
|
||||
## 任务：为下列词组补充「评论中使用语境」与「情感极性标签」（差评 ≤2★ / 好评 ≥4★，count 已按评论去重）
|
||||
|
||||
## 差评词组（≤2★，系统统计 count）
|
||||
{{neg_groups_json}}
|
||||
|
||||
## 好评词组（≥4★，系统统计 count）
|
||||
{{pos_groups_json}}
|
||||
|
||||
## Persona 列表
|
||||
{{personas_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"negative":[{"id":"g0","words":"可微调英文词组展示","polarity_label":"强负面/负面","meaning":"≤50字语境","related_personas":["Persona名"]}],"positive":[{"id":"g0","polarity_label":"强正面/强正面情感/魅力型信号/正面场景/正面","meaning":"≤50字","related_personas":[]}]}
|
||||
|
||||
## 极性标签规则
|
||||
- 差评：强负面（割伤/拉扯/灼痛等）| 负面（其他抱怨）
|
||||
- 好评：强正面 | 强正面情感（love）| 魅力型信号（amazing/cute/附赠）| 正面场景（shower/travel）| 正面
|
||||
|
||||
## 硬性要求
|
||||
- 必须为输入中每个 id 各输出一条，不得遗漏；不得新增 id
|
||||
- 不得修改 count；words 可微调英文展示，须与 match_keywords 语义一致
|
||||
- related_personas 只能使用上方 Persona 名称；无明确关联时可填「全部群体」
|
||||
- meaning ≤50 字；禁止编造评论中未出现的内容
|
||||
2876
voc_业务_2/report_utils.py
Normal file
2876
voc_业务_2/report_utils.py
Normal file
File diff suppressed because it is too large
Load diff
191
voc_业务_2/run_pipeline.py
Normal file
191
voc_业务_2/run_pipeline.py
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
步骤 1：运行 main_voc分析.py 的 step 1–6（跳过 step 7 报告生成），产出 SQLite 数据库。
|
||||
|
||||
用法::
|
||||
|
||||
# 全流程（config.yaml 留空则 LLM 自动识别产品名和行业）
|
||||
../310py/bin/python run_pipeline.py --input-dir "../Bikini trimmer voc"
|
||||
|
||||
# 手动指定产品/行业（跳过 LLM 识别）
|
||||
../310py/bin/python run_pipeline.py --input-dir "../Bikini trimmer voc" --product "Bikini Trimmer" --industry "个人护理"
|
||||
|
||||
# 断点续跑（从 step 4 向量化开始；前提是 step 1–3 的 sqlite 产物已存在）
|
||||
../310py/bin/python run_pipeline.py --from-step 4
|
||||
|
||||
# 仅重跑词频（step 6）
|
||||
../310py/bin/python run_pipeline.py --from-step 6
|
||||
|
||||
# 仅重跑聚类（step 5）
|
||||
../310py/bin/python run_pipeline.py --from-step 5
|
||||
|
||||
内部调用等价于（以全流程为例）::
|
||||
|
||||
../310py/bin/python ../main_voc分析.py --only-step 1 --input-dir "..." --product "..." --industry "..."
|
||||
../310py/bin/python ../main_voc分析.py --only-step 2 --product "..." --industry "..."
|
||||
...（依此类推到 step 6）
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
logger = logging.getLogger("voc.run_pipeline")
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = SCRIPT_DIR.parent
|
||||
CONFIG_FILE = SCRIPT_DIR / "config.yaml"
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
with CONFIG_FILE.open(encoding="utf-8") as f:
|
||||
return yaml.safe_load(f) or {}
|
||||
|
||||
|
||||
def run_step(python_bin: str, main_script: str, step: int, **kwargs) -> bool:
|
||||
"""运行单个 --only-step。"""
|
||||
cmd = [
|
||||
python_bin, main_script,
|
||||
"--only-step", str(step),
|
||||
"--product", kwargs.get("product", "亚马逊商品"),
|
||||
"--industry", kwargs.get("industry", "亚马逊电商"),
|
||||
]
|
||||
if step == 1 and "input_dir" in kwargs:
|
||||
cmd.extend(["--input-dir", kwargs["input_dir"]])
|
||||
|
||||
logger.info("执行 step %s: %s", step, " ".join(cmd))
|
||||
# 不 capture 输出，子进程日志实时可见，避免长时间步骤看起来像卡死
|
||||
result = subprocess.run(cmd, cwd=str(PROJECT_ROOT))
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error("Step %s 失败 (exit %s)", step, result.returncode)
|
||||
return False
|
||||
logger.info("Step %s 完成", step)
|
||||
return True
|
||||
|
||||
|
||||
def run_build_report(
|
||||
python_bin: str,
|
||||
build_script: str,
|
||||
*,
|
||||
product: str,
|
||||
industry: str,
|
||||
) -> bool:
|
||||
"""运行 build_report.py，子进程日志实时输出到当前终端。"""
|
||||
cmd = [
|
||||
python_bin, "-u", build_script,
|
||||
"--product", product,
|
||||
"--industry", industry,
|
||||
]
|
||||
logger.info("执行报告: %s", " ".join(cmd))
|
||||
env = os.environ.copy()
|
||||
env["PYTHONUNBUFFERED"] = "1"
|
||||
# 显式继承 stdout/stderr，避免长时间 LLM 步骤看起来像卡死
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
cwd=str(SCRIPT_DIR),
|
||||
env=env,
|
||||
stdout=sys.stdout,
|
||||
stderr=sys.stderr,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.error("build_report.py 失败 (exit %s)", result.returncode)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="运行 VOC 分析流水线 step 1-6")
|
||||
parser.add_argument("--input-dir", help="原始 CSV 目录")
|
||||
parser.add_argument("--product", help="产品名")
|
||||
parser.add_argument("--industry", help="行业名")
|
||||
parser.add_argument("--from-step", type=int, default=1, choices=range(1, 7))
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg = load_config()
|
||||
|
||||
product = args.product or cfg.get("product_name", "").strip()
|
||||
industry = args.industry or cfg.get("industry", "").strip()
|
||||
|
||||
# 如果产品名或行业为空，用 LLM 从原始评论中自动识别
|
||||
need_detect = (not product or product == "亚马逊商品" or not industry or industry == "亚马逊电商")
|
||||
if need_detect:
|
||||
input_dir_raw = args.input_dir or cfg.get("input_dir", "")
|
||||
input_dir_path_raw = (SCRIPT_DIR / input_dir_raw).resolve() if input_dir_raw else None
|
||||
if input_dir_path_raw and input_dir_path_raw.is_dir():
|
||||
from data_loader import DataLoader
|
||||
samples, dir_name = DataLoader.load_raw_review_samples(input_dir_path_raw, max_samples=50)
|
||||
if samples:
|
||||
import sys as _sys
|
||||
_sys.path.insert(0, str(SCRIPT_DIR))
|
||||
from llm_analyzer import detect_product_and_industry
|
||||
detected_product, detected_industry = detect_product_and_industry(samples, dir_name)
|
||||
if not product or product == "亚马逊商品":
|
||||
product = detected_product
|
||||
logger.info("LLM 自动识别产品名: %s", product)
|
||||
if not industry or industry == "亚马逊电商":
|
||||
industry = detected_industry
|
||||
logger.info("LLM 自动识别行业: %s", industry)
|
||||
else:
|
||||
if not product:
|
||||
product = "亚马逊商品"
|
||||
if not industry:
|
||||
industry = "亚马逊电商"
|
||||
else:
|
||||
if not product:
|
||||
product = "亚马逊商品"
|
||||
if not industry:
|
||||
industry = "亚马逊电商"
|
||||
input_dir = args.input_dir or cfg.get("input_dir", "")
|
||||
main_script = cfg.get("main_script", "../main_voc分析.py")
|
||||
python_bin = cfg.get("python_bin", "../310py/bin/python")
|
||||
|
||||
# 解析相对路径
|
||||
main_script_path = (SCRIPT_DIR / main_script).resolve()
|
||||
python_bin_path = (SCRIPT_DIR / python_bin) # 不 resolve：保留符号链接以确保 uv venv 正确激活
|
||||
input_dir_path = str((SCRIPT_DIR / input_dir).resolve()) if input_dir else ""
|
||||
|
||||
if not python_bin_path.exists():
|
||||
logger.error("Python 解释器不存在: %s", python_bin_path)
|
||||
sys.exit(1)
|
||||
if not main_script_path.is_file():
|
||||
logger.error("main_voc分析.py 不存在: %s", main_script_path)
|
||||
sys.exit(1)
|
||||
|
||||
logger.info("产品: %s | 行业: %s | 输入目录: %s", product, industry, input_dir_path)
|
||||
logger.info("从 step %s 开始", args.from_step)
|
||||
|
||||
steps = list(range(args.from_step, 7)) # step 1-6 only
|
||||
for step in steps:
|
||||
kwargs = {"product": product, "industry": industry}
|
||||
if step == 1:
|
||||
kwargs["input_dir"] = input_dir_path
|
||||
if not run_step(str(python_bin_path), str(main_script_path), step, **kwargs):
|
||||
logger.error("流水线中止于 step %s", step)
|
||||
sys.exit(1)
|
||||
|
||||
logger.info("流水线 step 1-6 全部完成。数据库已就绪。")
|
||||
|
||||
# 自动运行 build_report.py（日志实时打印到终端）
|
||||
logger.info("自动运行 build_report.py 生成报告...")
|
||||
build_script = str(SCRIPT_DIR / "build_report.py")
|
||||
if not run_build_report(
|
||||
str(python_bin_path),
|
||||
build_script,
|
||||
product=product,
|
||||
industry=industry,
|
||||
):
|
||||
sys.exit(1)
|
||||
logger.info("报告生成完成。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
407
voc_业务_2/template.html
Normal file
407
voc_业务_2/template.html
Normal file
|
|
@ -0,0 +1,407 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{PRODUCT_NAME}} — VOC 深度分析报告 {{VERSION}}</title>
|
||||
{{ECHARTS_SCRIPT}}
|
||||
<style>
|
||||
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'PingFang SC', sans-serif;
|
||||
background: #f8f9fa; color: #1a1a1a; font-size: 14px; line-height: 1.65; }
|
||||
.top-nav { position: sticky; top: 0; z-index: 100; background: rgba(255,255,255,0.97);
|
||||
backdrop-filter: blur(8px); border-bottom: 1px solid #e5e7eb;
|
||||
display: flex; align-items: center; padding: 0 20px; overflow-x: auto; }
|
||||
.top-nav .layer-tag { padding: 4px 10px; font-size: 11px; font-weight: 600; border-radius: 999px;
|
||||
margin: 0 6px 0 2px; white-space: nowrap; }
|
||||
.tag-what { background: #dbeafe; color: #1d4ed8; }
|
||||
.tag-why { background: #fef3c7; color: #92400e; }
|
||||
.top-nav a { padding: 12px 14px; font-size: 12px; font-weight: 500; color: #555;
|
||||
text-decoration: none; white-space: nowrap; border-bottom: 2px solid transparent; transition: color .15s; }
|
||||
.top-nav a:hover { color: #2563eb; }
|
||||
.page { max-width: 1140px; margin: 0 auto; padding: 36px 24px 72px; }
|
||||
.grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 18px; }
|
||||
.grid-3 { display: grid; grid-template-columns: repeat(3,1fr); gap: 14px; }
|
||||
.grid-4 { display: grid; grid-template-columns: repeat(4,1fr); gap: 14px; }
|
||||
.grid-5 { display: grid; grid-template-columns: repeat(5,1fr); gap: 14px; }
|
||||
.exec-summary { background: #fff; border: 1px solid #e5e7eb; border-radius: 8px; padding: 16px; margin-bottom: 18px; }
|
||||
.exec-summary h3 { font-size: 14px; margin-bottom: 8px; color: #0f172a; }
|
||||
h1 { font-size: 24px; font-weight: 700; color: #0f172a; margin-bottom: 4px; }
|
||||
.subtitle { font-size: 12px; color: #888; margin-bottom: 6px; }
|
||||
.layer-header { display: flex; align-items: center; gap: 12px; margin: 40px 0 16px; }
|
||||
.layer-badge { padding: 3px 14px; border-radius: 999px; font-size: 12px; font-weight: 700; letter-spacing: .04em; }
|
||||
.badge-what { background: #2563eb; color: #fff; }
|
||||
.badge-why { background: #d97706; color: #fff; }
|
||||
h2 { font-size: 17px; font-weight: 700; color: #0f172a; }
|
||||
h3 { font-size: 13px; font-weight: 600; color: #374151; margin-bottom: 8px; }
|
||||
.divider { border: none; border-top: 1px solid #e5e7eb; margin: 32px 0; }
|
||||
.note { font-size: 11px; color: #999; margin-bottom: 10px; line-height: 1.5; }
|
||||
.kpi-card { background: #fff; border: 1px solid #e5e7eb; border-radius: 8px; padding: 16px; }
|
||||
.kpi-val { font-size: 28px; font-weight: 700; color: #0f172a; line-height: 1.1; }
|
||||
.kpi-val.blue { color: #2563eb; }
|
||||
.kpi-val.success { color: #16a34a; }
|
||||
.kpi-val.warn { color: #d97706; }
|
||||
.kpi-val.danger { color: #dc2626; }
|
||||
.kpi-lbl { font-size: 12px; color: #666; margin-top: 4px; }
|
||||
.card { background: #fff; border: 1px solid #e5e7eb; border-radius: 8px; overflow: hidden; margin-bottom: 12px; }
|
||||
.card-header { padding: 10px 16px; background: #f9fafb; border-bottom: 1px solid #e5e7eb;
|
||||
font-size: 13px; font-weight: 600; color: #374151;
|
||||
display: flex; justify-content: space-between; align-items: center;
|
||||
cursor: pointer; user-select: none; }
|
||||
.card-header:hover { background: #f3f4f6; }
|
||||
.card-body { padding: 14px 16px; }
|
||||
.card.collapsed .card-body { display: none; }
|
||||
.toggle-icon::after { content: '\25B2'; font-size: 10px; color: #9ca3af; margin-left: 8px; }
|
||||
.card.collapsed .toggle-icon::after { content: '\25BC'; }
|
||||
.chart-box { background: #fff; border: 1px solid #e5e7eb; border-radius: 8px; padding: 16px; margin-bottom: 16px; }
|
||||
.chart-h { height: 240px; }
|
||||
.chart-h-md { height: 300px; }
|
||||
.chart-h-lg { height: 360px; }
|
||||
.chart-scroll { overflow-y: auto; overflow-x: hidden; border-radius: 6px; }
|
||||
.chart-scroll-x { overflow-x: auto; overflow-y: hidden; }
|
||||
.asin-appendix { margin-top: 12px; font-size: 13px; color: #374151; }
|
||||
.asin-appendix summary { cursor: pointer; font-weight: 600; padding: 8px 0; }
|
||||
.star-dist { color: #2563eb; cursor: help; font-size: 11px; }
|
||||
.theme-mini-grid { display: grid; grid-template-columns: repeat(2, 1fr); gap: 12px; margin-top: 14px; }
|
||||
.theme-mini-box { background: #f9fafb; border: 1px solid #e5e7eb; border-radius: 8px; padding: 10px 12px; }
|
||||
.theme-mini-box h4 { font-size: 12px; font-weight: 600; color: #374151; margin-bottom: 6px; }
|
||||
.theme-mini-chart { height: 200px; }
|
||||
.matrix-table td { font-size: 12px; line-height: 1.55; }
|
||||
.matrix-table td small { font-size: 11px; }
|
||||
.footer-asin-details summary { cursor: pointer; color: #2563eb; }
|
||||
.chart-cap { font-size: 11px; color: #999; text-align: center; margin-top: 6px; }
|
||||
.callout { border-radius: 8px; padding: 12px 16px; margin-bottom: 14px; }
|
||||
.callout-warn { background: #fffbeb; border-left: 4px solid #f59e0b; }
|
||||
.callout-danger { background: #fef2f2; border-left: 4px solid #dc2626; }
|
||||
.callout-info { background: #eff6ff; border-left: 4px solid #3b82f6; }
|
||||
.callout-success { background: #f0fdf4; border-left: 4px solid #22c55e; }
|
||||
.callout-title { font-weight: 600; font-size: 13px; margin-bottom: 3px; }
|
||||
.callout-warn .callout-title { color: #92400e; }
|
||||
.callout-danger .callout-title { color: #b91c1c; }
|
||||
.callout-info .callout-title { color: #1d4ed8; }
|
||||
.callout-success .callout-title { color: #15803d; }
|
||||
.callout p { font-size: 13px; color: #444; }
|
||||
.quote { background: #f3f4f6; border-radius: 6px; padding: 8px 12px;
|
||||
font-size: 12px; font-style: italic; color: #555; line-height: 1.55; margin: 6px 0; }
|
||||
.quote.neg { border-left: 3px solid #fca5a5; background: #fff5f5; }
|
||||
.quote.pos { border-left: 3px solid #86efac; background: #f0fff4; }
|
||||
.quote a { color: #2563eb; font-style: normal; text-decoration: none; }
|
||||
.quote a:hover { text-decoration: underline; }
|
||||
.quote-cn { font-size: 11px; color: #666; font-style: normal; margin-top: 4px; }
|
||||
.quote-empty { font-size: 12px; color: #999; font-style: italic; }
|
||||
.pill { display: inline-block; font-size: 11px; font-weight: 600; padding: 2px 8px; border-radius: 999px; }
|
||||
.pill-danger { background: #fee2e2; color: #b91c1c; border: 1px solid #fca5a5; }
|
||||
.pill-warn { background: #fef3c7; color: #92400e; border: 1px solid #fde68a; }
|
||||
.pill-info { background: #dbeafe; color: #1d4ed8; }
|
||||
.pill-success { background: #dcfce7; color: #15803d; }
|
||||
.pill-gray { background: #f3f4f6; color: #6b7280; }
|
||||
.tbl-wrap { overflow-x: auto; margin-bottom: 16px; }
|
||||
table { width: 100%; border-collapse: collapse; background: #fff; border-radius: 8px;
|
||||
overflow: hidden; border: 1px solid #e5e7eb; font-size: 13px; }
|
||||
thead { background: #f3f4f6; }
|
||||
th { padding: 9px 12px; text-align: left; font-size: 11px; font-weight: 600; color: #555;
|
||||
border-bottom: 1px solid #e5e7eb; white-space: nowrap; }
|
||||
td { padding: 9px 12px; border-bottom: 1px solid #f3f4f6; vertical-align: top; }
|
||||
tr:last-child td { border-bottom: none; }
|
||||
td a { color: #2563eb; text-decoration: none; }
|
||||
td a:hover { text-decoration: underline; }
|
||||
tr.row-r td { background: #fff5f5; }
|
||||
tr.row-y td { background: #fffdf0; }
|
||||
tr.row-g td { background: #f0fff4; }
|
||||
tr.row-b td { background: #eff6ff; }
|
||||
.num { text-align: right; }
|
||||
.persona { background: #fff; border: 1px solid #e5e7eb; border-radius: 8px; padding: 16px; }
|
||||
.p-name { font-weight: 700; font-size: 14px; margin-bottom: 2px; }
|
||||
.p-meta { font-size: 11px; color: #888; margin-bottom: 8px; }
|
||||
.p-pct { font-weight: 700; color: #2563eb; }
|
||||
.p-row { font-size: 12px; margin: 4px 0; color: #444; }
|
||||
.p-label { font-weight: 600; color: #374151; }
|
||||
.sat-hi { color: #16a34a; font-weight: 700; }
|
||||
.sat-mid { color: #d97706; font-weight: 700; }
|
||||
.sat-lo { color: #dc2626; font-weight: 700; }
|
||||
.rc-section { margin-bottom: 14px; }
|
||||
.rc-label { font-size: 12px; font-weight: 700; color: #374151; margin: 10px 0 4px; }
|
||||
.rc-text { font-size: 13px; color: #444; line-height: 1.65; }
|
||||
.jtbd-cell { max-width: 180px; font-size: 12px; line-height: 1.5; }
|
||||
.kano-grid { display: grid; grid-template-columns: repeat(2, 1fr); gap: 14px; margin-bottom: 20px; }
|
||||
.kano-card { border-radius: 8px; padding: 14px; border: 1px solid; }
|
||||
.kano-basic { background: #fef2f2; border-color: #fca5a5; }
|
||||
.kano-perf { background: #fffbeb; border-color: #fde68a; }
|
||||
.kano-excite { background: #f0fdf4; border-color: #86efac; }
|
||||
.kano-rev { background: #f5f3ff; border-color: #c4b5fd; }
|
||||
.kano-label { font-size: 11px; font-weight: 700; letter-spacing: .06em; margin-bottom: 6px; text-transform: uppercase; }
|
||||
.kano-basic .kano-label { color: #b91c1c; }
|
||||
.kano-perf .kano-label { color: #92400e; }
|
||||
.kano-excite .kano-label { color: #15803d; }
|
||||
.kano-rev .kano-label { color: #6d28d9; }
|
||||
.kano-title { font-size: 13px; font-weight: 700; margin-bottom: 8px; color: #111; }
|
||||
.kano-item { font-size: 12px; padding: 6px 0; border-bottom: 1px solid rgba(0,0,0,.05); display: grid; grid-template-columns: auto 1fr; gap: 8px; }
|
||||
.kano-item:last-child { border-bottom: none; }
|
||||
.kano-icon { font-size: 14px; margin-top: 1px; line-height: 1.4; }
|
||||
.kano-fields { display: flex; flex-direction: column; gap: 2px; }
|
||||
.kf-name { font-weight: 600; color: #111; font-size: 12px; }
|
||||
.kf-row { font-size: 11px; color: #555; line-height: 1.5; }
|
||||
.kf-key { font-weight: 600; color: #374151; }
|
||||
.kano-empty { font-size: 12px; color: #888; padding: 8px 0; }
|
||||
.danger { color: #dc2626; font-weight: 700; }
|
||||
.warn { color: #d97706; font-weight: 700; }
|
||||
.footer { font-size: 11px; color: #aaa; margin-top: 48px; padding-top: 14px; border-top: 1px solid #e5e7eb; }
|
||||
@media (max-width: 900px) {
|
||||
.grid-2,.grid-3,.grid-4,.grid-5,.kano-grid { grid-template-columns: 1fr; }
|
||||
.theme-mini-grid { grid-template-columns: 1fr; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<nav class="top-nav">
|
||||
<span class="layer-tag tag-what">描述层</span>
|
||||
<a href="#sec-summary">决策摘要</a>
|
||||
<a href="#sec-overview">数据总览</a>
|
||||
<a href="#sec-asin-theme">ASIN主题</a>
|
||||
<a href="#sec-persona">受众画像</a>
|
||||
<a href="#sec-feedback">正负反馈</a>
|
||||
<a href="#sec-keyword">情感词频</a>
|
||||
<span class="layer-tag tag-why">分析层</span>
|
||||
<a href="#sec-kano">KANO 模型</a>
|
||||
<a href="#sec-jtbd">JTBD 动机</a>
|
||||
<a href="#sec-matrix">人群矩阵</a>
|
||||
<a href="#sec-rootcause">痛点根因</a>
|
||||
</nav>
|
||||
|
||||
<div class="page">
|
||||
|
||||
<h1>{{PRODUCT_NAME}} — VOC 深度分析报告</h1>
|
||||
<p class="subtitle">{{ASIN_COUNT}} 个竞品 ASIN · 数据来源：{{DATA_SOURCE}} · 分析日期：{{ANALYSIS_DATE}}</p>
|
||||
|
||||
<div class="callout callout-info" style="margin-top:14px">
|
||||
<div class="callout-title">报告阅读指引</div>
|
||||
<p><strong>描述层 What</strong>：先看「决策摘要」→ 数据总览 → 各 ASIN 主题对比 → 受众画像 → 正负反馈主题 → 情感词频<br>
|
||||
<strong>分析层 Why</strong>：解释动机根因（KANO 需求分层 → JTBD 动机框架 → 人群×场景×需求矩阵 → 痛点根因分析）</p>
|
||||
</div>
|
||||
|
||||
<div class="exec-summary" id="sec-summary">
|
||||
<h3>决策摘要</h3>
|
||||
{{EXEC_SUMMARY_HTML}}
|
||||
</div>
|
||||
|
||||
{{INSIGHTS_CALLOUT}}
|
||||
|
||||
<!-- ══ §1 数据总览 ══ -->
|
||||
<div class="layer-header" id="sec-overview">
|
||||
<span class="layer-badge badge-what">描述层 What</span>
|
||||
<h2>数据总览</h2>
|
||||
</div>
|
||||
|
||||
<div class="callout callout-{{MARKET_CALLOUT_TYPE}}">
|
||||
<div class="callout-title">市场竞争状态：{{MARKET_TITLE}}</div>
|
||||
<p>{{MARKET_DESC}}</p>
|
||||
</div>
|
||||
|
||||
<div class="grid-5" style="margin-bottom:22px">
|
||||
<div class="kpi-card"><div class="kpi-val blue">{{TOTAL_REVIEWS}}</div><div class="kpi-lbl">有效评论总数（Verified+Vine）</div></div>
|
||||
<div class="kpi-card"><div class="kpi-val {{AVG_COLOR}}">{{WEIGHTED_AVG}}</div><div class="kpi-lbl">{{ASIN_COUNT}}款加权平均评分</div></div>
|
||||
<div class="kpi-card"><div class="kpi-val success">{{POS_RATE}}</div><div class="kpi-lbl">正评率（≥4★）</div></div>
|
||||
<div class="kpi-card"><div class="kpi-val warn">{{NEUTRAL_RATE}}</div><div class="kpi-lbl">中评率（3★，{{NEUTRAL_COUNT}}条）</div></div>
|
||||
<div class="kpi-card"><div class="kpi-val danger">{{NEG_RATE}}</div><div class="kpi-lbl">差评率（≤2★）</div></div>
|
||||
</div>
|
||||
|
||||
<h3>各 ASIN 有效评论统计</h3>
|
||||
{{ASIN_TABLE_NOTE}}
|
||||
<div class="tbl-wrap">{{ASIN_TABLE_SUMMARY}}</div>
|
||||
{{ASIN_TABLE_APPENDIX}}
|
||||
|
||||
<div class="chart-box">
|
||||
{{STAR_SUMMARY_HTML}}
|
||||
<h3>各 ASIN 评分分布（堆叠）</h3>
|
||||
<p class="note">{{STAR_CHART_NOTE}}</p>
|
||||
<div class="chart-scroll" style="max-height:{{STAR_SCROLL_MAX}}px">
|
||||
<div id="starDistChart" style="width:100%;height:{{STAR_CHART_HEIGHT}}px"></div>
|
||||
</div>
|
||||
<p class="chart-cap">数据来源：{{DATA_SOURCE}} · 筛选 verified=True 或 vine=True · 轴标签 A/B/… 悬停图表查看完整竞品名</p>
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §1b 各 ASIN 主题分布 ══ -->
|
||||
<div class="layer-header" id="sec-asin-theme">
|
||||
<span class="layer-badge badge-what">描述层 What</span>
|
||||
<h2>各 ASIN 主题分布</h2>
|
||||
</div>
|
||||
<p class="note">横向对比各竞品在 Top 差评/好评主题上的评论命中数，识别弱点集中 ASIN 与卖点组合差异。</p>
|
||||
{{ASIN_THEME_INSIGHTS_HTML}}
|
||||
|
||||
<div class="chart-box">
|
||||
<h3>各 ASIN 差评主题对比（Top 6 主题 × {{ASIN_COUNT}} 竞品）</h3>
|
||||
<p class="note">{{NEG_THEME_CHART_NOTE}}</p>
|
||||
<div id="negChartPerAsin" style="width:100%;height:{{NEG_THEME_CHART_HEIGHT}}px"></div>
|
||||
{{NEG_THEME_MINI_HTML}}
|
||||
</div>
|
||||
<div class="chart-box">
|
||||
<h3>各 ASIN 好评主题对比（Top 6 主题 × {{ASIN_COUNT}} 竞品）</h3>
|
||||
<p class="note">{{POS_THEME_CHART_NOTE}}</p>
|
||||
<div id="posChartPerAsin" style="width:100%;height:{{POS_THEME_CHART_HEIGHT}}px"></div>
|
||||
{{POS_THEME_MINI_HTML}}
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §2 受众画像 ══ -->
|
||||
<div class="layer-header" id="sec-persona">
|
||||
<span class="layer-badge badge-what">描述层 What</span>
|
||||
<h2>用户画像（Persona）</h2>
|
||||
</div>
|
||||
|
||||
<div class="grid-2">{{PERSONA_CARDS}}</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §3 正负反馈 ══ -->
|
||||
<div class="layer-header" id="sec-feedback">
|
||||
<span class="layer-badge badge-what">描述层 What</span>
|
||||
<h2>正负反馈主题统计</h2>
|
||||
</div>
|
||||
|
||||
<p class="note">{{NEG_THEME_SUMMARY_NOTE}}</p>
|
||||
|
||||
<div class="grid-2">
|
||||
<div class="chart-box">
|
||||
<h3>差评主题频次</h3>
|
||||
<div class="chart-h-lg" id="negChart"></div>
|
||||
</div>
|
||||
<div class="chart-box">
|
||||
<h3>好评主题频次（≥4 星，共 {{POS_REVIEW_COUNT}} 条）</h3>
|
||||
<div class="chart-h-lg" id="posChart"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>差评主题明细</h3>
|
||||
<div class="tbl-wrap">
|
||||
<table>
|
||||
<thead><tr><th>差评主题（按根因拆分，不合并）</th><th class="num">频次</th><th class="num">占差评比</th><th>优先级</th><th>涉及范围&最痛 ASIN</th></tr></thead>
|
||||
<tbody>{{NEG_THEME_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h3>好评主题明细</h3>
|
||||
<p class="note">占好评比基于单主题命中计数；同一评论可命中多个主题，各行占比之和可能超过 100%（当前合计约 {{POS_PCT_SUM}}%）。KANO 预判为 What 层摘要，详细分类见下方 KANO 模型章节。</p>
|
||||
<div class="tbl-wrap">
|
||||
<table>
|
||||
<thead><tr><th>好评主题</th><th class="num">频次</th><th class="num">占好评比</th><th>KANO 预判</th></tr></thead>
|
||||
<tbody>{{POS_THEME_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §4 情感词频 ══ -->
|
||||
<div class="layer-header" id="sec-keyword">
|
||||
<span class="layer-badge badge-what">描述层 What</span>
|
||||
<h2>情感关键词分析</h2>
|
||||
</div>
|
||||
<p class="note">规则：同一评论中同一词组多次出现计 1 次；极性根据评论语境判断；含义只写评论中明确出现的内容，不推断。</p>
|
||||
|
||||
<div class="grid-2">
|
||||
<div>
|
||||
<h3 style="font-size:14px;margin:0 0 8px">高频负面情感词（差评，≤2★）</h3>
|
||||
<div class="tbl-wrap">
|
||||
<table>
|
||||
<thead><tr><th>词汇</th><th class="num">频次</th><th>情感极性</th><th>评论中使用语境</th><th>主要关联 Persona</th></tr></thead>
|
||||
<tbody>{{KEYWORD_NEG_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<h3 style="font-size:14px;margin:0 0 8px">高频正面情感词（好评，≥4★）</h3>
|
||||
<div class="tbl-wrap">
|
||||
<table>
|
||||
<thead><tr><th>词汇</th><th class="num">频次</th><th>情感极性</th><th>评论中使用语境</th><th>主要关联 Persona</th></tr></thead>
|
||||
<tbody>{{KEYWORD_POS_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §5 KANO ══ -->
|
||||
<div class="layer-header" id="sec-kano">
|
||||
<span class="layer-badge badge-why">分析层 Why</span>
|
||||
<h2>KANO 模型需求分类</h2>
|
||||
</div>
|
||||
|
||||
<p class="note">每个需求条目包含：需求项 · 频次证据 · 主要影响 Persona · 分类原因 · 竞品现状（5 字段，按 SOP 要求）</p>
|
||||
|
||||
{{KANO_GRID_HTML}}
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §6 JTBD ══ -->
|
||||
<div class="layer-header" id="sec-jtbd">
|
||||
<span class="layer-badge badge-why">分析层 Why</span>
|
||||
<h2>JTBD 动机框架</h2>
|
||||
</div>
|
||||
|
||||
<p class="note">各动机字段须来自 Persona 聚类数据（keywords / core_pain / core_need 等）佐证；无法找到佐证的字段填「-」。</p>
|
||||
|
||||
<div class="tbl-wrap">
|
||||
<table>
|
||||
<thead><tr><th>用户群</th><th>核心 Job</th><th>功能性动机</th><th>情感性动机</th><th>社会性动机</th><th>购买触发时机</th></tr></thead>
|
||||
<tbody>{{JTBD_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §7 人群矩阵 ══ -->
|
||||
<div class="layer-header" id="sec-matrix">
|
||||
<span class="layer-badge badge-why">分析层 Why</span>
|
||||
<h2>人群 × 场景 × 需求矩阵</h2>
|
||||
</div>
|
||||
<p class="note">场景字段来源：评论中出现次数 ≥5 的场景词方可填写；每 Persona 最多 2 个场景、全报告最多 4 个 Persona。全市场均分偏低时，「高满意度」会自动校准为中等并注明原因。</p>
|
||||
|
||||
<div class="tbl-wrap">
|
||||
<table class="matrix-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th style="min-width:130px">用户群</th>
|
||||
<th style="min-width:150px">使用场景（When/Where）<br><small style="font-weight:normal;color:#aaa">仅评论中佐证≥5条的场景</small></th>
|
||||
<th style="min-width:150px">基本型需求</th>
|
||||
<th style="min-width:150px">期望型需求</th>
|
||||
<th style="min-width:120px">魅力型需求</th>
|
||||
<th style="min-width:100px">当前满意度</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>{{MATRIX_TABLE_ROWS}}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<hr class="divider">
|
||||
|
||||
<!-- ══ §8 痛点根因 ══ -->
|
||||
<div class="layer-header" id="sec-rootcause">
|
||||
<span class="layer-badge badge-why">分析层 Why</span>
|
||||
<h2>痛点根因分析（按 Persona 展开）</h2>
|
||||
</div>
|
||||
|
||||
{{ROOTCAUSE_CARDS}}
|
||||
|
||||
<div class="footer">
|
||||
<strong>数据来源</strong>：{{DATA_SOURCE}}（{{ASIN_COUNT}} 个 ASIN）<br>
|
||||
<strong>ASIN</strong>：{{FOOTER_ASIN_LINKS}}<br>
|
||||
<strong>筛选规则</strong>：仅统计 verified=True 或 vine=True 的有效评论，共 {{TOTAL_REVIEWS}} 条<br>
|
||||
<strong>分析框架</strong>：VOC 数据清洗、分析与报告生成通用方法论 v1.0 · <strong>分析截止日期</strong>：{{ANALYSIS_DATE}}
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<script>
|
||||
function toggleCard(header) {
|
||||
header.parentElement.classList.toggle('collapsed');
|
||||
}
|
||||
|
||||
// ── ECharts 图表 ──
|
||||
{{CHART_JS}}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
28
voc_业务_2/报告LLM提示词.md
Normal file
28
voc_业务_2/报告LLM提示词.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# VOC 报告 LLM 提示词 · 说明索引
|
||||
|
||||
## 给谁用哪个文件？
|
||||
|
||||
| 角色 | 文件 | 做什么 |
|
||||
|------|------|--------|
|
||||
| **业务员** | **[提示词编辑稿.md](./提示词编辑稿.md)** | 改中文任务说明、硬性要求、示例（**只改这个**） |
|
||||
| **技术同事** | [prompts.yaml](./prompts.yaml) | 把编辑稿内容同步进来，程序实际读取此文件 |
|
||||
| **开发** | [prompt_loader.py](./prompt_loader.py) · [llm_analyzer.py](./llm_analyzer.py) | 一般不用动 |
|
||||
|
||||
## 工作流程
|
||||
|
||||
```
|
||||
业务员修改 提示词编辑稿.md
|
||||
↓
|
||||
技术同事复制到 prompts.yaml 对应段落（见编辑稿末尾「同步清单」）
|
||||
↓
|
||||
运行 run_pipeline.py 或 build_report.py 生成报告
|
||||
```
|
||||
|
||||
## 模型参数（config.yaml，非提示词正文）
|
||||
|
||||
- `report_model` / `report_reasoning_effort` / `report_max_tokens` — 报告 LLM 模型与思考深度
|
||||
- `llm_max_workers` — 根因等并发数
|
||||
|
||||
## 不走 LLM 的报告内容
|
||||
|
||||
决策摘要、部分市场竞争文案由 `report_utils.py` 规则生成，不在提示词文件内。
|
||||
616
voc_业务_2/提示词编辑稿.md
Normal file
616
voc_业务_2/提示词编辑稿.md
Normal file
|
|
@ -0,0 +1,616 @@
|
|||
# VOC 报告 · 提示词编辑稿（业务员用）
|
||||
|
||||
> **请你只改本文件。** 改完后交给技术同事，他会把内容同步到 `prompts.yaml` 并重新生成报告。
|
||||
> **你不需要打开** `prompts.yaml`（那是程序用的配置文件）。
|
||||
> **方法论依据：** `VOC分析方法论与报告生成逻辑.md` v1.0
|
||||
|
||||
---
|
||||
|
||||
## 使用说明（3 步）
|
||||
|
||||
1. 在下方找到要改的**报告章节**（如「用户画像」「差评主题」）
|
||||
2. 直接修改对应框里的**中文文字**（任务说明、硬性要求、示例等）
|
||||
3. **不要删除** 带 `{{ }}` 的行——那是系统自动填入数据的占位符，例如：
|
||||
- `{{personas_json}}` = 自动填入用户画像列表
|
||||
- `{{catalog_json}}` = 自动填入聚类数据
|
||||
|
||||
**请勿修改：**
|
||||
- JSON 格式示例里的英文字段名（如 `"personas"`、`"themes"`、`"name"`）
|
||||
- 所有 `{{xxx}}` 占位符整行
|
||||
|
||||
**可以修改：**
|
||||
- 「你是…专家」这类角色描述
|
||||
- 「## 任务」「## 硬性要求」下的中文规则和示例
|
||||
- 数量要求（如 4–7 个 Persona 改成 5–8 个）
|
||||
|
||||
**示例边界（v1.2）：**
|
||||
- 文中所有中文/英文示例**仅说明格式与分析逻辑**
|
||||
- 实际 Persona、主题名、keywords、场景、KANO、根因**须来自当前批次聚类数据**，禁止照搬示例文字
|
||||
|
||||
---
|
||||
|
||||
## 章节与报告对照表
|
||||
|
||||
| 本文件章节 | 报告里看到的位置 | 同步到 prompts.yaml 的键名 |
|
||||
|-----------|-----------------|---------------------------|
|
||||
| 0. 产品识别 | （无单独章节，用于自动识别产品名） | `product_detect` |
|
||||
| 1. 用户画像 | 用户画像（Persona） | `persona` |
|
||||
| 2. 正负主题 | 正负反馈主题统计 | `theme` |
|
||||
| 3. KANO | KANO 模型需求分类 | `kano` |
|
||||
| 4. JTBD | JTBD 动机框架 | `jtbd` |
|
||||
| 5. 矩阵 | 人群 × 场景 × 需求矩阵 | `matrix` |
|
||||
| 6. 根因 | 痛点根因分析 | `rootcause` |
|
||||
| 7. 情感词 | 情感关键词分析 | `keyword` |
|
||||
|
||||
---
|
||||
|
||||
# 0. 产品 / 行业自动识别
|
||||
|
||||
**同步位置：** `prompts.yaml` → `product_detect`
|
||||
|
||||
| 参数 | 当前值 | 说明 |
|
||||
|------|--------|------|
|
||||
| temperature | 0.2 | 数值越小输出越稳定，一般不用改 |
|
||||
| max_tokens | 1000 | 最大输出长度，一般不用改 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `product_detect.system`
|
||||
|
||||
```
|
||||
你是亚马逊商品识别专家。根据用户评论内容推断产品名称和所属行业。输出严格JSON，不输出多余文字。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `product_detect.user_template`
|
||||
|
||||
```
|
||||
根据以下亚马逊评论内容和目录名，推断产品名称和所属行业。
|
||||
|
||||
## 评论样本
|
||||
{{sample_text}}
|
||||
{{dir_info}}
|
||||
## 输出JSON
|
||||
{"product_name": "推断的产品名称（中英文均可，简洁描述，如 Kitchen Blender / Pet Repellent Spray）", "industry": "所属行业（中文，如 个人护理/宠物用品/厨房家电/健康补充剂 等）"}
|
||||
|
||||
要求：
|
||||
- product_name 用评论中高频提及的核心产品词命名，不用品牌名或 ASIN；不要包含行业大类词
|
||||
- industry 用一个中文行业大类词
|
||||
- 目录名中的关键词可作为重要参考线索
|
||||
- JSON 示例仅说明格式，product_name 须从上方评论样本归纳
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{sample_text}}` — 系统自动插入评论样本，勿删
|
||||
- `{{dir_info}}` — 系统自动插入文件夹名称，勿删
|
||||
|
||||
---
|
||||
|
||||
# 1. 用户画像 Persona
|
||||
|
||||
**同步位置：** `prompts.yaml` → `persona`
|
||||
**报告章节：** 用户画像（Persona）
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.4 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `persona.system`
|
||||
|
||||
```
|
||||
你是资深的消费者洞察专家，严格遵循 VOC 分析方法论 v1.0 第 3.2 节与 4.0 节。
|
||||
Persona 命名用简洁中文（≤6字），避免营销化夸张名称。输出严格JSON。
|
||||
示例仅说明格式；Persona 名与 keywords 须来自当前聚类 top_phrases，禁止照搬示例。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `persona.user_template`
|
||||
|
||||
```
|
||||
## 任务：基于聚类数据按三维度发现用户画像（Persona），4-7 个。
|
||||
|
||||
## 分析产品（product_detect / config 已识别）
|
||||
产品：{{product_name}} | 行业：{{industry}}
|
||||
purchase_motivation 用「雇佣{{product_name}}完成…」句式；Persona 须与该产品使用场景一致。
|
||||
|
||||
## 三维度强制覆盖（缺一不可）
|
||||
A. 物理/生理/受众特征 → 必须绑定 stage=1_audience 的簇
|
||||
典型信号：特定体质/肤质/年龄/体型/使用对象（如 sensitive / elderly / for kids / large breed）
|
||||
B. 行为/使用场景 → 绑定 3b_aspect_opinion_positive / 3b_aspect_opinion_negative / 3a_pain_global
|
||||
典型信号：travel / daily use / outdoor / first time / gift
|
||||
C. 购买动机/背景 → 绑定 3a_pain_global 或 opinion 簇
|
||||
典型信号：switched from / saw on social media / first time / too expensive
|
||||
→ 不能只有 B 和 C；A 维至少 1 个。若 A 维缺失，说明物理特征群体被遗漏，必须补建。
|
||||
|
||||
## 自我标注信号强制检查（归纳后必做）
|
||||
在绑定簇的 top_phrases 中搜索以下模式，出现 ≥5 条则必须单独建 Persona：
|
||||
- "I have [adj] [noun]"（如 I have sensitive skin / I have a large dog）
|
||||
- "My [noun] is/are [adj]"（如 My pet is very anxious）
|
||||
- "As a [noun]"（如 As a first-time buyer / As a pet owner）
|
||||
- 受众/体质/使用对象相关形容词（elderly / sensitive / indoor / outdoor）
|
||||
|
||||
## 其他遗漏检查
|
||||
- 长期使用/复购用户：含 after months / after a while / second bottle / bought again 的差评是否形成独立群体
|
||||
- 占比低（~5%）但痛点独特、无法被其他 Persona 代表的群体，仍须单独列出
|
||||
|
||||
## 可绑定的聚类簇目录（cluster_ref 必须从中选择；每簇含 top_phrases + sample_reviews 最多 5 条原文）
|
||||
{{catalog_json}}
|
||||
|
||||
## 生理标签硬规则（名称 + core_pain，违反则系统会剔除）
|
||||
- 生理/体质类中文标签须在绑定簇内 ≥5 条评论原文含对应英文词（catalog 字段 physio_review_counts，如 pregnancy: 8）
|
||||
- top_phrases 偶然出现 1–4 次不算；无达标评论禁止写入
|
||||
- core_pain 只能归纳 sample_reviews 中明确出现的内容
|
||||
|
||||
## 补充聚类摘要
|
||||
audience_clusters: {{audience_clusters_json}}
|
||||
global_pains: {{global_pains_json}}
|
||||
global_negative: {{global_negative_json}}
|
||||
global_positive: {{global_positive_json}}
|
||||
|
||||
## 输出JSON
|
||||
{"personas":[{"name":"≤6中文字","dimension":"A","cluster_ref":{"stage":"1_audience","label":0},"keywords":["从绑定簇 top_phrases 复制"],"core_pain":"核心痛点（≤3条，分号分隔）","core_need":"核心需求（≤3条，分号分隔）","purchase_motivation":"雇佣产品做什么（动词+宾语，JTBD句式）"}]}
|
||||
## 硬性要求
|
||||
- 每个 Persona 必须有 cluster_ref；stage 只能是：1_audience / 3a_pain_global / 3b_aspect_opinion_negative / 3b_aspect_opinion_positive
|
||||
- cluster_ref.label 必须是上方目录中存在的 label；系统用该簇的 review_count 作为命中规模（非 keywords 扫全文）
|
||||
- dimension 只能是 A、B 或 C；A 维 Persona 的 cluster_ref.stage 必须是 1_audience
|
||||
- keywords 至少 5 个，必须从绑定簇的 top_phrases 复制英文片段（≥3字符），禁止编造不在簇中的词
|
||||
- 4-7 个 Persona，三维度 A/B/C 均至少覆盖 1 个；优先选 review_count 较大的簇，小簇（<15条）仅在有明确短语证据时使用
|
||||
- core_pain 来自该群体差评语义；core_need 来自该群体好评或诉求；purchase_motivation 用「雇佣产品完成…」句式
|
||||
- 命名示例（格式参考，须从聚类归纳）：敏感体质用户、首次购买用户、粗毛疤痕体质用户、旅行护理用户、蜡脱替代用户（禁止：受害者联盟、体验官、刮刀逃离者等夸张抽象名）
|
||||
- 禁止在 physio_review_counts 未达 ≥5 条时将「孕妇/孕期」写入名称或 core_pain
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{product_name}}` / `{{industry}}` — 产品识别结果（自动填入）
|
||||
- `{{catalog_json}}` — 聚类簇目录（自动填入）
|
||||
- `{{audience_clusters_json}}` 等 — 聚类摘要（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 2. 差评 / 好评主题
|
||||
|
||||
**同步位置：** `prompts.yaml` → `theme`
|
||||
**报告章节：** 正负反馈主题统计
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.3 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `theme.system`
|
||||
|
||||
```
|
||||
你是亚马逊 VOC 分析专家，遵循方法论 3.3 节。
|
||||
从聚类短语归纳主题；根因/失效机制不同的问题必须独立成主题，禁止笼统合并。
|
||||
主题名用简洁中文（≤8字），适用于任意品类。输出严格JSON。
|
||||
示例仅说明拆分逻辑；主题名与 keywords 须来自当前聚类数据，禁止照搬示例主题名。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【差评专用补充】→ 粘贴到 `theme.negative_extra`
|
||||
|
||||
（仅分析差评主题时使用，好评不走这段）
|
||||
|
||||
```
|
||||
## 优先级（共 {{neg_review_count}} 条差评，系统会按实际频次与竞品覆盖率重算）
|
||||
- P0：频次 ≥ {{p0_threshold}} 条 且 80%+ 竞品均出现
|
||||
- P1：频次为差评总数 10–20% 且 60%+ 竞品出现
|
||||
- P2：频次为差评总数 3–10% 且 40%+ 竞品出现
|
||||
|
||||
## 拆分红线（方法论 3.3）
|
||||
判断问题：「同一主题下的差评，是否描述同一个物理/工程原因？」若不是，必须拆开。
|
||||
典型拆分（机制/根因不同须拆开，勿照搬下列中文名）：
|
||||
- 核心效果未达预期 → ①效果弱/不明显 ②使用方式与预期不符（机制不同）
|
||||
- 使用过程不适 → ①物理伤害/刺激 ②过热/异味/过敏（根因不同）
|
||||
- 产品失效/损坏 → ①供电/充电问题 ②结构件断裂/脱落（失效环节不同）
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{neg_review_count}}` — 差评总数（自动填入）
|
||||
- `{{p0_threshold}}` — P0 优先级门槛（自动填入）
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `theme.user_template`
|
||||
|
||||
```
|
||||
## 任务：归纳 {{theme_type}} 主题（适用于当前品类，勿预设具体产品类型）
|
||||
{{extra_block}}
|
||||
|
||||
## 归纳原则
|
||||
- 从 top_phrases 中归纳 4–8 个主题，差评主题应覆盖 80%+ 差评内容（长尾可合并为「其他」）
|
||||
- keywords 语义相近但失效机制不同 → 必须拆成独立主题
|
||||
- 好评主题可与差评维度对应但用正向表述（如 核心使用体验 ↔ 核心效果未达预期）
|
||||
- 示例主题名仅作格式与拆分参考；实际 name/keywords 必须来自上方 cluster 数据的 top_phrases
|
||||
|
||||
## 数据
|
||||
{{cluster_data_json}}
|
||||
## 输出JSON
|
||||
{"themes":[{"name":"≤8中文字","keywords":["english phrase from top_phrases"],"priority":"P0/P1/P2（仅差评）","description":"一句话说明该主题的用户抱怨/满意点"}]}
|
||||
## 硬性要求
|
||||
- keywords 必须是英文，从 top_phrases 中复制完整片段或逗号后的子句（≥3字符）
|
||||
- 每个主题至少 5 个 keywords
|
||||
- 差评 priority 按上方门槛初步标注（系统会重算）
|
||||
- 差评示例（格式参考，须从 top_phrases 归纳）：效果未达预期、使用不适、供电失效、结构损坏、性价比低
|
||||
- 好评示例（格式参考）：核心使用体验、产品质量耐用、便携与设计、易用性、超预期惊喜
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{theme_type}}` — 自动填 `negative` 或 `positive`
|
||||
- `{{extra_block}}` — 差评时自动插入上方「差评专用补充」
|
||||
- `{{cluster_data_json}}` — 聚类短语数据（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 3. KANO 需求分类
|
||||
|
||||
**同步位置：** `prompts.yaml` → `kano`
|
||||
**报告章节：** KANO 模型需求分类
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.3 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `kano.system`
|
||||
|
||||
```
|
||||
你是产品需求分析专家，专精 KANO 模型，遵循方法论 4.1 节。
|
||||
四象限分类；每个条目 5 字段缺一不可。输出严格JSON。
|
||||
KANO 示例仅说明四象限判断；item/evidence 须来自当前差评/好评主题 keywords，禁止照搬示例。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【反向型主动搜索】→ 粘贴到 `kano.reverse_search_block`
|
||||
|
||||
（KANO 分析时自动插入）
|
||||
|
||||
```
|
||||
## 反向型主动搜索（步骤三，不得以「未发现」一笔带过）
|
||||
在差评/好评主题 keywords 中检索以下词组及同义表达，记录出现频次：
|
||||
- 过于复杂：too many parts / too complicated / confusing / hard to use
|
||||
- 过于嘈杂：too loud / so loud / noise / noisy
|
||||
- 功能多余：don't need / unnecessary / didn't ask for / useless feature
|
||||
- 操作繁琐：takes too long / too many steps / annoying to clean
|
||||
处理规则：
|
||||
- 任一词组频次 ≥5 → 输出 reverse 类型条目并附 evidence
|
||||
- 全部词组总频次 <5 → 必须输出 1 条 type=reverse 的占位条目，item 写「本品类暂无明确反向需求」，evidence 写「经主动搜索 [列出搜索词]，共 N 条，低于阈值 5」
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `kano.user_template`
|
||||
|
||||
```
|
||||
## 任务：KANO 四象限分类
|
||||
|
||||
## 分类标准（含常见误判）
|
||||
- 基本型 Must-be：P0 级差评 + 好评中几乎无人因「做到了 X」而表扬
|
||||
❌ 误判：核心性能指标（好坏都会被提及）→ 期望型
|
||||
✅ 正确：开箱即能用、供电正常、关键部件不脱落
|
||||
- 期望型 Performance：好评差评均出现，做得越好评分越高
|
||||
❌ 误判：续航/容量 → 基本型（超长续航会被特别称赞）
|
||||
✅ 正确：核心效果、续航/容量、易清洁程度
|
||||
- 魅力型 Attractive：好评中出现 love/obsessed/amazing/didn't expect/bonus；差评中几乎不出现
|
||||
❌ 误判：附赠配件 → 期望型（无人因缺该配件差评）
|
||||
✅ 正确：电量/状态显示、超预期配件、意外惊喜功能
|
||||
- 反向型 Reverse：用户主动抱怨某「功能」是负担（功能过载/太吵/太复杂）
|
||||
❌ 误判：产品损坏/充电故障 → 基本型（无人「希望产品损坏」）
|
||||
|
||||
{{reverse_search_block}}
|
||||
|
||||
## 操作步骤
|
||||
1. 基本型：从 P0/P1 差评主题出发，检查好评是否几乎无人表扬该点
|
||||
2. 期望型 vs 魅力型：差评有人因「不够好」→ 期望型；好评有 love/amazing 且竞品普遍缺失 → 魅力型
|
||||
3. 反向型：执行上方主动搜索，按规则输出
|
||||
|
||||
## 差评主题
|
||||
{{neg_themes_json}}
|
||||
## 好评主题
|
||||
{{pos_themes_json}}
|
||||
## Persona
|
||||
{{personas_json}}
|
||||
## 输出JSON
|
||||
{"kano":[{"type":"must-be/performance/attractive/reverse","item":"单条需求（动词+名词，禁止用逗号/顿号合并多条）","evidence":"评论频次证据+代表性英文片段（≤80字）","affected_persona":"主要 Persona","reason":"≤60字，解释为何是该类型而非其他类型","competitor_status":"竞品是否满足及满足程度"}]}
|
||||
## 硬性要求
|
||||
- 每条 item 只写一条需求，禁止在 item 中用逗号合并
|
||||
- 至少 8 条记录；must-be / performance / attractive 均需覆盖；reverse 按搜索规则输出（含占位条目）
|
||||
- evidence 须含频次级别（如 P0/P1）或条数估计 + 英文原文片段
|
||||
- 质量缺陷、充电故障、配件脱落 → must-be，不是 reverse
|
||||
- reverse 仅限用户主动排斥功能过载（too many parts / too complicated / too loud 等）
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{reverse_search_block}}` — 自动插入上方「反向型主动搜索」
|
||||
- `{{neg_themes_json}}` / `{{pos_themes_json}}` / `{{personas_json}}` — 前几步分析结果（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 4. JTBD 动机框架
|
||||
|
||||
**同步位置：** `prompts.yaml` → `jtbd`
|
||||
**报告章节：** JTBD 动机框架
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.3 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `jtbd.system`
|
||||
|
||||
```
|
||||
你是 JTBD 分析专家，遵循方法论 4.2 节。
|
||||
为每个 Persona 构建 Jobs To Be Done 框架；所有动机字段须能从 Persona 的 keywords/core_pain/core_need/purchase_motivation 中找到语义佐证。
|
||||
无佐证时该字段填 "-"。必须覆盖全部 Persona。输出严格JSON。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `jtbd.user_template`
|
||||
|
||||
```
|
||||
## 任务：为 {{persona_count}} 个 Persona 构建 JTBD
|
||||
{{personas_json}}
|
||||
|
||||
## 填写规则
|
||||
| 字段 | 规则 |
|
||||
| core_job | 动词+宾语，用户想完成的任务；须与 Persona 数据语义一致 |
|
||||
| functional_motivation | 实用层面驱动（效率/效果/成本/便携），禁止写情感词 |
|
||||
| emotional_motivation | 情绪/心理驱动（自信/焦虑/掌控感/安心），禁止写功能词 |
|
||||
| social_motivation | 他人视角/社交驱动（如送礼、伴侣评价、公开场合）；无评论佐证填 "-" |
|
||||
| trigger | 购买触发时机，须来自 Persona 数据中的具体事件描述；无佐证填 "-" |
|
||||
|
||||
## 输出JSON
|
||||
{"jtbd":[{"persona":"名称","core_job":"核心Job","functional_motivation":"功能性动机","emotional_motivation":"情感性动机","social_motivation":"社会性动机或-","trigger":"触发时机或-"}]}
|
||||
## 硬性要求
|
||||
- 必须覆盖全部 {{persona_count}} 个 Persona，一人一行
|
||||
- functional 与 emotional 字段内容不可互换
|
||||
- 所有字段（含 core_job / trigger）须与对应 Persona 的 keywords / core_pain / core_need / purchase_motivation 语义一致
|
||||
- 无法从 Persona 数据找到佐证的字段一律填 "-"，禁止编造评论中无依据的内容
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{persona_count}}` — Persona 个数（自动填入）
|
||||
- `{{personas_json}}` — Persona 列表（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 5. 人群 × 场景 × 需求矩阵
|
||||
|
||||
**同步位置：** `prompts.yaml` → `matrix`
|
||||
**报告章节：** 人群 × 场景 × 需求矩阵
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.3 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `matrix.system`
|
||||
|
||||
```
|
||||
你是消费者洞察专家，构建人群×场景×需求矩阵，严格遵循方法论 4.3 节与 5.3 节场景规则。
|
||||
场景只允许填写评论中有原词佐证的描述；找不到佐证则不输出该行。
|
||||
输出严格JSON。全市场均分低时，细分场景「高满意度」须谨慎标注。
|
||||
场景示例仅说明格式；scene 须来自 Persona keywords 中的英文原词佐证，禁止照搬示例。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【低分市场补充】→ 粘贴到 `matrix.market_low_hint`
|
||||
|
||||
（仅当全市场加权均分 < 3.5 时自动插入）
|
||||
|
||||
```
|
||||
## 重要：全市场加权均分 {{market_avg}}（<3.5），多数场景 satisfaction 应为「中等」或「低」，慎用「高」。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `matrix.user_template`
|
||||
|
||||
```
|
||||
## 任务：为以下 Persona 构建矩阵（仅输出这些 Persona，最多 4 个）
|
||||
|
||||
## 场景规则（方法论 5.3，核心规则）
|
||||
【强制】scene 只允许填写 Persona 的 keywords / 聚类短语中能找到英文原词佐证的场景。
|
||||
禁止基于产品功能、品类常识或逻辑推断填写场景。
|
||||
- ≥5 条评论出现该场景词 → 可填写（如 travel / outdoor / kitchen / office — 以 top_phrases 为准）
|
||||
- 2–4 条 → 不输出该行
|
||||
- <2 条 → 不输出该行（禁止输出 scene 为 — 的行）
|
||||
|
||||
常见错误（禁止）：
|
||||
- ❌ 日常使用（daily 是使用频率非场景）
|
||||
- ❌ 节日前突击（评论无对应原词）
|
||||
- ❌ 任何场所（泛化代替留空）
|
||||
正确示例：✅ 具体地点/情境（须在 keywords 中找到英文原词且 ≥5 条）
|
||||
|
||||
## 需求列映射
|
||||
- must_be_needs：KANO 基本型 + 该群体 P0 差评主题名
|
||||
- performance_needs：KANO 期望型 + 该群体 P1 差评主题名
|
||||
- attractive_needs：KANO 魅力型 + 该群体好评加分点
|
||||
|
||||
## 满意度评级
|
||||
- 高：该群体均分参考 ≥4.0
|
||||
- 中等：3.3–3.9
|
||||
- 低：＜3.3
|
||||
|
||||
## 每个 Persona 最多 2 个有效场景行
|
||||
{{market_hint}}
|
||||
{{top_personas_json}}
|
||||
{{kano_json}}
|
||||
## 总评论数: {{total_reviews}} · 全市场均分: {{market_avg}}
|
||||
## 输出JSON
|
||||
{"matrix":[{"persona":"名称","pct":35,"scene":"具体场景（须有评论原词佐证）","must_be_needs":"基本型关键词","performance_needs":"期望型关键词","attractive_needs":"魅力型关键词","satisfaction":"高/中等/低","satisfaction_note":"≤25字"}]}
|
||||
## 硬性要求
|
||||
- 禁止输出 scene 为 —、-、N/A 或空的行
|
||||
- 每个 Persona 最多 2 行；全报告最多 4 个 Persona
|
||||
- must_be_needs / performance_needs / attractive_needs 各 ≤12 字，用顿号分隔关键词，禁止完整句子
|
||||
- must_be_needs 须使用差评主题中文名（来自当前批次主题，非示例）
|
||||
- satisfaction_note ≤25 字
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{market_hint}}` — 低分市场时插入上方「低分市场补充」
|
||||
- `{{top_personas_json}}` / `{{kano_json}}` — Persona 与 KANO 数据（自动填入）
|
||||
- `{{total_reviews}}` / `{{market_avg}}` — 评论总数与市场均分（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 6. 痛点根因分析
|
||||
|
||||
**同步位置：** `prompts.yaml` → `rootcause`
|
||||
**报告章节：** 痛点根因分析（按 Persona 展开）
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.4 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `rootcause.system`
|
||||
|
||||
```
|
||||
你是产品工程与消费者洞察专家，遵循方法论 4.4 节。
|
||||
痛点根因分析须从现象到达机制层（非「质量差」类空话）；开发方向须可落地。
|
||||
分析对象是当前品类主产品（见用户消息），禁止把其他品类工具问题当作本产品根因。
|
||||
输出严格JSON，使用中文。
|
||||
根因示例仅说明分析深度；title/mechanism/dev_direction 须针对当前产品与注入主题，禁止照搬示例。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `rootcause.user_template`
|
||||
|
||||
```
|
||||
## 任务：{{persona_name}} 的痛点根因分析（2-3 条，针对 P0/P1 级痛点）
|
||||
|
||||
## 产品范围：{{product_name}}（行业：{{industry}}）
|
||||
## 分析边界：只分析该品类产品本身的结构/功能/体验缺陷
|
||||
|
||||
## 分析框架（每条根因）
|
||||
根因标题 → 导致后果（关联差评主题×频次）→ 失效机制（从结构/材料/工作原理解释）→ 可落地改进
|
||||
|
||||
## 层次要求
|
||||
- ❌ 现象层：「产品质量差」「用户体验不好」
|
||||
- ✅ 机制层：「密封/接口设计不足导致进水腐蚀」「关键部件角度/间距不当导致效果未达预期」
|
||||
|
||||
{{persona_json}}
|
||||
{{per_aud_data_json}}
|
||||
## 可用差评主题名（affected_themes 只能从中选择）
|
||||
{{theme_names_json}}
|
||||
## 输出JSON
|
||||
{"root_causes":[{"title":"根因标题（≤20字）","mechanism":"失效机制（≤120字，白话，禁止医学/化学术语堆砌）","quote_keywords":["用于匹配评论的英文词"],"dev_direction":"可落地改进（≤80字：结构/材料/工艺/说明/品控等）"}],"affected_themes":["主题名"]}
|
||||
## 硬性要求
|
||||
- quotes 字段不要输出（引用由系统从真实评论回填）
|
||||
- quote_keywords 每条根因 2-4 个英文词，须与 mechanism 直接相关
|
||||
- affected_themes 只能使用上方差评主题名，优先 P0/P1 主题
|
||||
- dev_direction 须针对 {{product_name}} 可改进点，禁止：传感器、纳米、AI、蓝牙等专业/科幻表述
|
||||
- 每个 Persona 最多 3 条 root_causes
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{persona_name}}` / `{{product_name}}` / `{{industry}}` — 当前分析对象（自动填入）
|
||||
- `{{persona_json}}` / `{{per_aud_data_json}}` / `{{theme_names_json}}` — 画像与主题数据（自动填入）
|
||||
|
||||
---
|
||||
|
||||
# 7. 情感关键词
|
||||
|
||||
**同步位置：** `prompts.yaml` → `keyword`
|
||||
**报告章节：** 情感关键词分析
|
||||
|
||||
| 参数 | 当前值 |
|
||||
|------|--------|
|
||||
| temperature | 0.3 |
|
||||
| max_tokens | 8000 |
|
||||
|
||||
---
|
||||
|
||||
## 【角色设定】→ 粘贴到 `keyword.system`
|
||||
|
||||
```
|
||||
你是 VOC 文本分析专家，遵循方法论 3.4 节。
|
||||
对差评/好评词组做细粒度极性标注；meaning 只写评论中明确出现的语境，禁止推断。
|
||||
优先标注有决策价值的情感词组。输出严格JSON。
|
||||
示例仅说明标注方式；words/meaning 须来自输入词组与评论语境，禁止照搬示例。
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 【任务说明】→ 粘贴到 `keyword.user_template`
|
||||
|
||||
```
|
||||
## 任务：情感关键词分析（差评≤2★ / 好评≥4★ 双表，词组合并，count 按评论去重）
|
||||
{{neg_groups_json}}
|
||||
{{pos_groups_json}}
|
||||
{{personas_json}}
|
||||
|
||||
## 输入说明
|
||||
- 每条含 id、words（同义/近义词组）、count（至少命中组内一词的评论条数，已去重）
|
||||
- 分别处理 negative_groups 与 positive_groups，输出 id 与输入一一对应
|
||||
|
||||
## 输出字段规则
|
||||
- id：与输入 id 一致
|
||||
- words：沿用输入词组
|
||||
- count：沿用输入 count，禁止改写
|
||||
- polarity：细粒度极性
|
||||
- 差评侧：强负面 / 负面 / 待观察
|
||||
- 好评侧：强正面 / 正面 / 魅力型信号
|
||||
- meaning：该词组在评论中的具体语境，只写评论明确出现的内容
|
||||
- related_personas：关联 Persona 名称，可多个
|
||||
|
||||
## 输出JSON
|
||||
{"negative":[{"id":"neg_1","words":["cut","nick"],"count":42,"polarity":"强负面","meaning":"…","related_personas":[]}],"positive":[{"id":"pos_1","words":["smooth"],"count":30,"polarity":"强正面","meaning":"…","related_personas":[]}]}
|
||||
|
||||
## 硬性要求
|
||||
- 各侧最多输出 15 条；跳过纯功能中性词组
|
||||
- meaning ≤40 字；禁止编造评论中未出现的场景或原因
|
||||
- 必须跳过 {{skip_hint}}
|
||||
```
|
||||
|
||||
**占位符说明：**
|
||||
- `{{neg_groups_json}}` — 差评词组（≤2★，自动填入）
|
||||
- `{{pos_groups_json}}` — 好评词组（≥4★，自动填入）
|
||||
- `{{personas_json}}` — Persona 列表（自动填入）
|
||||
- `{{skip_hint}}` — 需跳过的停用词说明（自动填入，含产品名）
|
||||
|
||||
---
|
||||
|
||||
# 附录：技术同事同步清单
|
||||
|
||||
改完本文件后，请技术同事按章节将内容复制到 `prompts.yaml`：
|
||||
|
||||
| 本文件区块 | prompts.yaml 路径 |
|
||||
|-----------|-------------------|
|
||||
| 【角色设定】 | `xxx.system` |
|
||||
| 【任务说明】 | `xxx.user_template` |
|
||||
| 【差评专用补充】 | `theme.negative_extra` |
|
||||
| 【反向型主动搜索】 | `kano.reverse_search_block` |
|
||||
| 【低分市场补充】 | `matrix.market_low_hint` |
|
||||
|
||||
同步完成后运行：
|
||||
|
||||
```bash
|
||||
cd voc_业务_2
|
||||
../310py/bin/python build_report.py --product "产品名"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*文档版本与 prompts.yaml meta.version 对齐：1.2 · 2026-06-12*
|
||||
79
向量化.py
79
向量化.py
|
|
@ -1,5 +1,5 @@
|
|||
"""
|
||||
从 voc_structured.sqlite 最新 job 展开 audience / pain_point / aspect_opinion
|
||||
从 voc_structured.sqlite 最新 job 展开 persona_signal / pain_point / aspect_opinion
|
||||
（与聚类.py 一致，不向量化单独的 aspect、opinion），使用本地 MLX 写入 voc_embeddings.sqlite。
|
||||
|
||||
溯源：source_row 与结构化时一致（CSV 第 1 条数据行=1）；对应 merged_reviews_cleaned.csv
|
||||
|
|
@ -11,7 +11,7 @@
|
|||
./310py/bin/python 向量化.py --batch-size 16
|
||||
./310py/bin/python 向量化.py --job-id 4 --csv merged_reviews_cleaned.csv
|
||||
|
||||
环境变量：VOC_EMBED_MODEL_PATH、VOC_EMBED_BATCH_SIZE（默认 16）、VOC_EMBED_MAX_TEXT_CHARS（默认 512）。
|
||||
环境变量：VOC_EMBED_MODEL_PATH、VOC_EMBED_BATCH_SIZE（默认 16）、VOC_EMBED_MAX_TEXT_CHARS（默认 10000）。
|
||||
本地 MLX 推理串行执行，--workers 仅保留兼容、固定为 1。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
|
@ -59,11 +59,56 @@ def _resolve_embed_workers(explicit: int | None = None) -> int:
|
|||
logger.warning("本地 embedding 仅支持串行，--workers 已忽略（使用 1）")
|
||||
return 1
|
||||
ENTITY_TYPES = (
|
||||
"audience",
|
||||
"persona_signal",
|
||||
"pain_point",
|
||||
"aspect_opinion",
|
||||
)
|
||||
|
||||
_GENERIC_PERSONA_SIGNALS = frozenset({
|
||||
"self", "unknown", "user", "customer", "buyer", "myself", "the user", "consumer",
|
||||
})
|
||||
|
||||
|
||||
def _normalize_persona_signals(raw: Any) -> List[str]:
|
||||
if raw is None:
|
||||
items: List[Any] = []
|
||||
elif isinstance(raw, list):
|
||||
items = raw
|
||||
elif isinstance(raw, str) and raw.strip():
|
||||
items = [raw.strip()]
|
||||
else:
|
||||
items = []
|
||||
out: List[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in items:
|
||||
text = str(item).strip()
|
||||
if not text or text.lower() in _GENERIC_PERSONA_SIGNALS:
|
||||
continue
|
||||
key = text.lower()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append(text)
|
||||
return out
|
||||
|
||||
|
||||
def _derive_persona_signals_fallback(
|
||||
persona_signals: List[str],
|
||||
pain_points: List[Any],
|
||||
) -> List[str]:
|
||||
"""无 persona_signals 时：用 pain_points 前 2 条作兜底。"""
|
||||
if persona_signals:
|
||||
return persona_signals
|
||||
return [str(p).strip() for p in pain_points if str(p).strip()][:2]
|
||||
|
||||
|
||||
def _should_skip_extraction(
|
||||
persona_signals: List[str],
|
||||
pain_points: List[Any],
|
||||
feedback: List[Any],
|
||||
) -> bool:
|
||||
return not persona_signals and not pain_points and not feedback
|
||||
|
||||
|
||||
@dataclass
|
||||
class EmbedTask:
|
||||
|
|
@ -73,7 +118,7 @@ class EmbedTask:
|
|||
entity_type: str
|
||||
entity_index: int
|
||||
embed_text: str
|
||||
audience: str
|
||||
audience: str | None
|
||||
aspect: str | None
|
||||
opinion: str | None
|
||||
category: str | None
|
||||
|
|
@ -151,11 +196,6 @@ def _init_embed_schema(conn: sqlite3.Connection) -> None:
|
|||
)
|
||||
|
||||
|
||||
def _should_skip_extraction(audience: str, pain_points: List[Any], feedback: List[Any]) -> bool:
|
||||
aud = (audience or "").strip().lower()
|
||||
return aud == "unknown" and not pain_points and not feedback
|
||||
|
||||
|
||||
def _build_tasks(
|
||||
job_id: int,
|
||||
rows: List[sqlite3.Row],
|
||||
|
|
@ -171,7 +211,7 @@ def _build_tasks(
|
|||
logger.warning("跳过无效 JSON extraction_id=%s: %s", ext_id, e)
|
||||
continue
|
||||
|
||||
audience = str(data.get("audience", "unknown")).strip() or "unknown"
|
||||
persona_signals = _normalize_persona_signals(data.get("persona_signals"))
|
||||
pain_points = data.get("pain_points") or []
|
||||
feedback = data.get("product_feedback") or []
|
||||
if not isinstance(pain_points, list):
|
||||
|
|
@ -179,7 +219,7 @@ def _build_tasks(
|
|||
if not isinstance(feedback, list):
|
||||
feedback = []
|
||||
|
||||
if _should_skip_extraction(audience, pain_points, feedback):
|
||||
if _should_skip_extraction(persona_signals, pain_points, feedback):
|
||||
continue
|
||||
|
||||
content = content_map.get(source_row, "")
|
||||
|
|
@ -189,17 +229,18 @@ def _build_tasks(
|
|||
source_row,
|
||||
)
|
||||
|
||||
aud_norm = audience.strip()
|
||||
if aud_norm.lower() != "unknown":
|
||||
signals = _derive_persona_signals_fallback(persona_signals, pain_points)
|
||||
|
||||
for i, sig in enumerate(signals):
|
||||
tasks.append(
|
||||
EmbedTask(
|
||||
job_id=job_id,
|
||||
extraction_id=ext_id,
|
||||
source_row=source_row,
|
||||
entity_type="audience",
|
||||
entity_index=0,
|
||||
embed_text=aud_norm,
|
||||
audience=aud_norm,
|
||||
entity_type="persona_signal",
|
||||
entity_index=i,
|
||||
embed_text=sig,
|
||||
audience=None,
|
||||
aspect=None,
|
||||
opinion=None,
|
||||
category=None,
|
||||
|
|
@ -220,7 +261,7 @@ def _build_tasks(
|
|||
entity_type="pain_point",
|
||||
entity_index=i,
|
||||
embed_text=text,
|
||||
audience=aud_norm,
|
||||
audience=None,
|
||||
aspect=None,
|
||||
opinion=None,
|
||||
category=None,
|
||||
|
|
@ -247,7 +288,7 @@ def _build_tasks(
|
|||
entity_type="aspect_opinion",
|
||||
entity_index=i,
|
||||
embed_text=merged,
|
||||
audience=aud_norm,
|
||||
audience=None,
|
||||
aspect=aspect,
|
||||
opinion=opinion,
|
||||
category=category,
|
||||
|
|
|
|||
|
|
@ -68,17 +68,17 @@ def _get_valid_categories() -> frozenset[str]:
|
|||
return product_feedback_categories()
|
||||
|
||||
# 模型上下文上限；动态分批受 DEFAULT_MAX_BATCH_INPUT_TOKENS 与 DEFAULT_MAX_BATCH_REVIEWS 约束
|
||||
MODEL_MAX_INPUT_TOKENS = 991_800
|
||||
MODEL_MAX_OUTPUT_TOKENS = 65_530
|
||||
MODEL_MAX_INPUT_TOKENS = 1000_800
|
||||
MODEL_MAX_OUTPUT_TOKENS = 200_000
|
||||
CHARS_PER_TOKEN_EST = 3.2
|
||||
DEFAULT_MAX_BATCH_INPUT_TOKENS = 200_000
|
||||
# 仅用于 batch_plan 日志中的输出 token 粗估，不参与分批与 API max_tokens
|
||||
DEFAULT_OUTPUT_TOKENS_PER_REVIEW = 450
|
||||
BATCH_COUNT_MIN = 1
|
||||
# 动态分批时单批评论条数上限（避免单请求过大导致输出截断）
|
||||
DEFAULT_MAX_BATCH_REVIEWS = 100
|
||||
DEFAULT_MAX_BATCH_REVIEWS = 50
|
||||
# Chat 批间并行；与 embedding 共用账号时不宜过高，避免连带 429
|
||||
STRUCT_DEFAULT_WORKERS = 8
|
||||
STRUCT_DEFAULT_WORKERS = 400
|
||||
|
||||
|
||||
def _resolve_max_batch_reviews(explicit: int | None = None) -> int:
|
||||
|
|
@ -492,25 +492,43 @@ _SENTIMENT_ALIASES: Dict[str, str] = {
|
|||
"mildly negative": "negative",
|
||||
}
|
||||
|
||||
# 模型常见误写 -> 合法 category（校验前自动映射，减少无效重试）
|
||||
# 模型常见误写 -> 8 个标准 category（校验前自动映射，减少无效重试）
|
||||
_CATEGORY_ALIASES: Dict[str, str] = {
|
||||
# Price（价格）
|
||||
"value": "Price",
|
||||
"values": "Price",
|
||||
"cost": "Price",
|
||||
"pricing": "Price",
|
||||
"value for money": "Price",
|
||||
"worth": "Price",
|
||||
# Logistics（物流）
|
||||
"packaging": "Logistics",
|
||||
"shipping": "Logistics",
|
||||
"delivery": "Logistics",
|
||||
# Customer Service（客服）
|
||||
"service": "Customer Service",
|
||||
"customer support": "Customer Service",
|
||||
"support": "Customer Service",
|
||||
# Appearance（外观）
|
||||
"design": "Appearance",
|
||||
"look": "Appearance",
|
||||
"performance": "Function",
|
||||
# Function（功效/功能）
|
||||
"efficacy": "Function",
|
||||
"performance": "Function",
|
||||
"effectiveness": "Function",
|
||||
"effect": "Function",
|
||||
# Ingredient（原料）
|
||||
"ingredients": "Ingredient",
|
||||
"material": "Ingredient",
|
||||
"materials": "Ingredient",
|
||||
"formula": "Ingredient",
|
||||
# Trust（信任）
|
||||
"reliability": "Trust",
|
||||
"reliable": "Trust",
|
||||
"brand": "Trust",
|
||||
"authenticity": "Trust",
|
||||
"genuine": "Trust",
|
||||
"trustworthy": "Trust",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -544,7 +562,8 @@ def _normalize_category(raw: Any) -> str | None:
|
|||
for v in valid:
|
||||
if v.lower() == lower:
|
||||
return v
|
||||
return None
|
||||
# 不属于 8 个标准类别：允许模型自行补充新类别，规范为首字母大写后保留
|
||||
return key.title()
|
||||
|
||||
|
||||
def _normalize_product_feedback(items: Any) -> List[Dict[str, str]]:
|
||||
|
|
@ -569,19 +588,55 @@ def _normalize_product_feedback(items: Any) -> List[Dict[str, str]]:
|
|||
return out
|
||||
|
||||
|
||||
_GENERIC_PERSONA_SIGNALS = frozenset({
|
||||
"self", "unknown", "user", "customer", "buyer", "myself", "the user", "consumer",
|
||||
})
|
||||
|
||||
|
||||
def _normalize_persona_signals(raw: Any) -> List[str]:
|
||||
if raw is None:
|
||||
items: List[Any] = []
|
||||
elif isinstance(raw, list):
|
||||
items = raw
|
||||
elif isinstance(raw, str) and raw.strip():
|
||||
items = [raw.strip()]
|
||||
else:
|
||||
items = []
|
||||
out: List[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in items:
|
||||
text = str(item).strip()
|
||||
if not text or text.lower() in _GENERIC_PERSONA_SIGNALS:
|
||||
continue
|
||||
key = text.lower()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append(text)
|
||||
return out
|
||||
|
||||
|
||||
def _validate_extraction_strict(obj: Any, key_label: str = "") -> Dict[str, Any]:
|
||||
"""校验结构化结果；失败时抛出带键名的 ValueError，供回传模型修正。"""
|
||||
prefix = f"{key_label}: " if key_label else ""
|
||||
if not isinstance(obj, dict):
|
||||
raise ValueError(f"{prefix}必须是 JSON 对象")
|
||||
|
||||
for field in ("audience", "pain_points", "product_feedback"):
|
||||
for field in ("persona_signals", "pain_points", "product_feedback"):
|
||||
if field not in obj:
|
||||
raise ValueError(f"{prefix}缺少必填字段 {field}")
|
||||
|
||||
audience = obj.get("audience")
|
||||
if not isinstance(audience, str) or not audience.strip():
|
||||
raise ValueError(f"{prefix}audience 必须为非空字符串")
|
||||
persona_signals = obj.get("persona_signals")
|
||||
if not isinstance(persona_signals, list):
|
||||
raise ValueError(f"{prefix}persona_signals 必须为数组")
|
||||
for i, sig in enumerate(persona_signals):
|
||||
if not isinstance(sig, str) or not str(sig).strip():
|
||||
raise ValueError(f"{prefix}persona_signals[{i}] 必须为非空字符串")
|
||||
if str(sig).strip().lower() in _GENERIC_PERSONA_SIGNALS:
|
||||
raise ValueError(
|
||||
f"{prefix}persona_signals[{i}] 禁止为泛化词 {sig!r}，"
|
||||
"须写具体身份/体质/场景短语"
|
||||
)
|
||||
|
||||
pain_points = obj.get("pain_points")
|
||||
if not isinstance(pain_points, list):
|
||||
|
|
@ -620,11 +675,7 @@ def _normalize_extraction(obj: Any) -> Dict[str, Any]:
|
|||
if not isinstance(obj, dict):
|
||||
raise ValueError("Each extraction must be a JSON object")
|
||||
|
||||
audience = obj.get("audience", "unknown")
|
||||
if not isinstance(audience, str) or not str(audience).strip():
|
||||
audience = "unknown"
|
||||
else:
|
||||
audience = str(audience).strip()
|
||||
persona_signals = _normalize_persona_signals(obj.get("persona_signals", []))
|
||||
|
||||
pain_points = obj.get("pain_points", [])
|
||||
if pain_points is None:
|
||||
|
|
@ -637,14 +688,14 @@ def _normalize_extraction(obj: Any) -> Dict[str, Any]:
|
|||
|
||||
missing = [
|
||||
f
|
||||
for f in ("audience", "pain_points", "product_feedback")
|
||||
for f in ("persona_signals", "pain_points", "product_feedback")
|
||||
if f not in obj
|
||||
]
|
||||
if missing:
|
||||
logger.info("Normalized missing fields %s in extraction", missing)
|
||||
|
||||
return {
|
||||
"audience": audience,
|
||||
"persona_signals": persona_signals,
|
||||
"pain_points": pain_points,
|
||||
"product_feedback": product_feedback,
|
||||
}
|
||||
|
|
@ -658,7 +709,8 @@ def _build_batch_correction_message(keys: List[str], errors: List[str]) -> str:
|
|||
f"校验错误：\n{err_block}\n\n"
|
||||
"要求：\n"
|
||||
f"- 输出一个 JSON 对象，顶层键必须且仅能是：{keys_literal}\n"
|
||||
"- 每个键的值必须包含 audience、pain_points、product_feedback\n"
|
||||
"- 每个键的值必须包含 persona_signals、pain_points、product_feedback\n"
|
||||
"- persona_signals 禁止仅写 self/user/unknown 等泛化词\n"
|
||||
"- product_feedback 每条须含 aspect、opinion、sentiment"
|
||||
"（仅 Positive/Negative/Neutral，禁止 Mixed/Ambiguous；褒贬交织选主倾向或拆条）、"
|
||||
"category（禁止 Value，性价比用 Price）\n"
|
||||
|
|
|
|||
216
聚类.py
216
聚类.py
|
|
@ -3,8 +3,8 @@
|
|||
|
||||
流程：
|
||||
1. audience（LLM 自动调参 n_neighbors）
|
||||
2a. 前两 audience 簇各自独立：簇内 pain_point（LLM 自动调参）
|
||||
2b. 前两 audience 簇各自独立：簇内 aspect_opinion 按 Positive/Negative/Neutral 分桶聚类
|
||||
2a. 占比达阈值的 audience 簇各自独立：簇内 pain_point（LLM 自动调参）
|
||||
2b. 占比达阈值的 audience 簇各自独立：簇内 aspect_opinion 按 Positive/Negative/Neutral 分桶聚类
|
||||
3a. 全量 pain_point（LLM 自动调参）
|
||||
3b. 全量 aspect_opinion 按 Positive/Negative/Neutral 分桶聚类（各自 LLM 自动调参）
|
||||
|
||||
|
|
@ -100,7 +100,9 @@ INITIAL_N_NEIGHBORS = 10
|
|||
MAX_N_NEIGHBORS = 45
|
||||
CROSS_SIMILAR_RATIO_THRESHOLD = 0.10
|
||||
SILHOUETTE_STOP_THRESHOLD = 0.6
|
||||
SILHOUETTE_DECLINE_WINDOW = 8 # 连续 8 个轮廓值：后 7 个均小于第 1 个则停止
|
||||
SILHOUETTE_DECLINE_WINDOW = 10 # 连续 N 个轮廓值：后 N-1 个均小于第 1 个则停止
|
||||
SILHOUETTE_NEIGHBOR_MARGIN = 0.03 # 邻轮轮廓 ≥ 峰值−此值视为「接近」，参与离群数决胜
|
||||
SILHOUETTE_NEIGHBOR_RADIUS = 3 # 峰值轮次前后各 3 轮
|
||||
SAMPLE_CAP = 30
|
||||
SAMPLE_RATIO = 0.6
|
||||
|
||||
|
|
@ -108,7 +110,10 @@ SAMPLE_RATIO = 0.6
|
|||
STEP2_N_NEIGHBORS = 8
|
||||
|
||||
# 簇内去重评论数 / 本步骤参与聚类的去重评论总数 < 该比例则不写入库、不进入报告
|
||||
CLUSTER_MIN_REVIEW_RATIO = 0.10
|
||||
CLUSTER_MIN_REVIEW_RATIO = 0.05
|
||||
# step 2 入选 audience 簇：去重评论数 / 全量清洗评论总数 ≥ 该比例
|
||||
DEFAULT_AUDIENCE_COVERAGE_THRESHOLD = 0.01
|
||||
VOC_BUSINESS_CONFIG = PROJECT_ROOT / "voc_业务_2" / "config.yaml"
|
||||
|
||||
|
||||
@dataclass
|
||||
|
|
@ -546,11 +551,29 @@ def _ai_evaluate_cluster_samples(
|
|||
lines.append(f" {i}. {sent}")
|
||||
total += 1
|
||||
sample_text = "\n".join(lines)
|
||||
prompt = f"""你是 VOC 评论短语聚类质量评估助手。以下是多个聚类类别的抽样短语。
|
||||
prompt = f"""你是 VOC 评论短语聚类质量评估助手。以下是多个聚类类别的抽样短语（英文为主）。
|
||||
|
||||
{sample_text}
|
||||
|
||||
请统计 cross_similar_count：不同聚类类别之间、语义相似的短语条数（每句最多计 1）。
|
||||
任务：统计 cross_similar_count——**不同聚类类别之间**、语义相近的短语条数。
|
||||
|
||||
## 判断标准（从宽，不要漏判）
|
||||
将两条短语判为「跨类相似」，只要它们表达的是**同一类用户意图/问题/反馈**，不要求措辞一致。以下情况**都应计入**：
|
||||
- 同义改写：如 "doesn't work" 与 "not effective"
|
||||
- 同一痛点不同说法：如 "cat pees on bed" 与 "urinates on sofa"
|
||||
- 同一产品缺陷的不同表述：如 "strong smell" 与 "odor too strong"
|
||||
- 核心对象相同、评价方向相同：如 "sprayer broken" 与 "nozzle stopped working"
|
||||
- 一方是另一方的子集或概括：如 "joint pain" 与 "severe joint pain in elderly dog"
|
||||
|
||||
以下情况**不计入**：
|
||||
- 仅在同一聚类类别内部相似（不算跨类）
|
||||
- 明显不同方面：如 "fast shipping" 与 "bad smell"
|
||||
- 褒贬相反：如 "works great" 与 "doesn't work at all"
|
||||
|
||||
## 计数规则
|
||||
1. 逐条短语与其他聚类中的短语比对；只要与**任一**其他类的**任一**短语相近，该条计 1。
|
||||
2. 每条短语最多计 1 次。
|
||||
3. 宁可多计疑似相近，也不要漏掉明显同义/同主题的跨类重复。
|
||||
|
||||
只输出 JSON：
|
||||
{{
|
||||
|
|
@ -562,10 +585,10 @@ def _ai_evaluate_cluster_samples(
|
|||
resp = client.chat.completions.create(
|
||||
model=LLM_MODEL,
|
||||
messages=[
|
||||
{"role": "system", "content": "只输出合法 JSON。"},
|
||||
{"role": "system", "content": "你是聚类质量评估助手。跨类相似判断从宽：同主题、同义改写、同一痛点/反馈的不同说法都应算相似。只输出合法 JSON。"},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
max_tokens=1500,
|
||||
max_tokens=200_000,
|
||||
temperature=0.0,
|
||||
response_format={"type": "json_object"},
|
||||
# 关闭思考，避免 token 耗在 reasoning_content 导致 content 为空且无 JSON
|
||||
|
|
@ -594,8 +617,8 @@ def _silhouette_decline_should_stop(scores: List[float]) -> bool:
|
|||
|
||||
|
||||
def _best_silhouette_in_window(
|
||||
snapshots: List[Tuple[int, np.ndarray, float]],
|
||||
) -> Tuple[int, np.ndarray, float] | None:
|
||||
snapshots: List[Tuple[int, np.ndarray, float, int]],
|
||||
) -> Tuple[int, np.ndarray, float, int] | None:
|
||||
"""在最近轮廓窗口内取轮廓系数最高的一轮；无快照时返回 None。"""
|
||||
if not snapshots:
|
||||
return None
|
||||
|
|
@ -603,6 +626,33 @@ def _best_silhouette_in_window(
|
|||
return max(pool, key=lambda x: x[2])
|
||||
|
||||
|
||||
def _resolve_peak_with_neighbor_noise_tiebreak(
|
||||
snapshots: List[Tuple[int, np.ndarray, float, int]],
|
||||
) -> Tuple[Tuple[int, np.ndarray, float, int], str | None]:
|
||||
"""轮廓窗口早停：以窗口内峰值为中心，±2 邻轮若轮廓 ≥ 峰值−0.03 则与峰值一起按离群数择优。"""
|
||||
pool = snapshots[-SILHOUETTE_DECLINE_WINDOW:]
|
||||
center = max(pool, key=lambda x: x[2])
|
||||
center_nn, _, center_sil, center_noise = center
|
||||
threshold = center_sil - SILHOUETTE_NEIGHBOR_MARGIN
|
||||
by_nn = {s[0]: s for s in snapshots}
|
||||
candidates: Dict[int, Tuple[int, np.ndarray, float, int]] = {center_nn: center}
|
||||
for delta in range(-SILHOUETTE_NEIGHBOR_RADIUS, SILHOUETTE_NEIGHBOR_RADIUS + 1):
|
||||
if delta == 0:
|
||||
continue
|
||||
snap = by_nn.get(center_nn + delta)
|
||||
if snap is not None and snap[2] >= threshold:
|
||||
candidates[snap[0]] = snap
|
||||
if len(candidates) == 1:
|
||||
return center, None
|
||||
chosen = min(candidates.values(), key=lambda s: (s[3], -s[2]))
|
||||
note = (
|
||||
f"邻轮复核：峰值 n_neighbors={center_nn}（轮廓{center_sil:.4f}，离群{center_noise}），"
|
||||
f"候选 {sorted(candidates)} 中择离群最少 → n_neighbors={chosen[0]}"
|
||||
f"（轮廓{chosen[2]:.4f}，离群{chosen[3]}）"
|
||||
)
|
||||
return chosen, note
|
||||
|
||||
|
||||
def _auto_tune_n_neighbors(
|
||||
embeddings: np.ndarray,
|
||||
sentences: List[str],
|
||||
|
|
@ -616,8 +666,8 @@ def _auto_tune_n_neighbors(
|
|||
min_cs = 2
|
||||
silhouette_avg: float | None = None
|
||||
silhouette_history: List[float] = []
|
||||
# (n_neighbors, labels, silhouette) 仅在有有效轮廓时入栈，供早停回退最优轮次
|
||||
silhouette_snapshots: List[Tuple[int, np.ndarray, float]] = []
|
||||
# (n_neighbors, labels, silhouette, n_noise) 仅在有有效轮廓时入栈，供早停回退最优轮次
|
||||
silhouette_snapshots: List[Tuple[int, np.ndarray, float, int]] = []
|
||||
n = len(embeddings)
|
||||
|
||||
while True:
|
||||
|
|
@ -654,34 +704,46 @@ def _auto_tune_n_neighbors(
|
|||
|
||||
if silhouette_avg is not None:
|
||||
silhouette_history.append(silhouette_avg)
|
||||
silhouette_snapshots.append((n_neighbors, labels.copy(), silhouette_avg))
|
||||
silhouette_snapshots.append(
|
||||
(n_neighbors, labels.copy(), silhouette_avg, n_noise)
|
||||
)
|
||||
if _silhouette_decline_should_stop(silhouette_history):
|
||||
window = silhouette_history[-SILHOUETTE_DECLINE_WINDOW:]
|
||||
best = _best_silhouette_in_window(silhouette_snapshots)
|
||||
assert best is not None
|
||||
best_nn, best_labels, best_sil = best
|
||||
best, tiebreak_note = _resolve_peak_with_neighbor_noise_tiebreak(
|
||||
silhouette_snapshots
|
||||
)
|
||||
best_nn, best_labels, best_sil, best_noise = best
|
||||
n_neighbors = best_nn
|
||||
labels = best_labels
|
||||
silhouette_avg = best_sil
|
||||
log_entry["silhouette_window"] = [round(s, 4) for s in window]
|
||||
log_entry["n_noise"] = best_noise
|
||||
stop_tail = (
|
||||
f"，回退至 n_neighbors={best_nn}（轮廓{best_sil:.4f}，离群{best_noise}）"
|
||||
)
|
||||
if tiebreak_note:
|
||||
log_entry["neighbor_tiebreak"] = tiebreak_note
|
||||
stop_tail = f"；{tiebreak_note}"
|
||||
log_entry["stop_reason"] = (
|
||||
f"连续{SILHOUETTE_DECLINE_WINDOW}轮轮廓：后5个均低于窗口首值"
|
||||
f"{window[0]:.4f}，回退至 n_neighbors={best_nn}（轮廓{best_sil:.4f}）"
|
||||
f"连续{SILHOUETTE_DECLINE_WINDOW}轮轮廓：后{SILHOUETTE_DECLINE_WINDOW - 1}个"
|
||||
f"均低于窗口峰值{max(window):.4f}{stop_tail}"
|
||||
)
|
||||
tuning_log.append(log_entry)
|
||||
logger.info(
|
||||
"[%s] 轮廓窗口 %s，早停并回退 n_neighbors=%s 轮廓=%.4f",
|
||||
"[%s] 轮廓窗口 %s，早停并回退 n_neighbors=%s 轮廓=%.4f 离群=%s%s",
|
||||
stage,
|
||||
log_entry["silhouette_window"],
|
||||
best_nn,
|
||||
best_sil,
|
||||
best_noise,
|
||||
f"；{tiebreak_note}" if tiebreak_note else "",
|
||||
)
|
||||
break
|
||||
|
||||
if n_neighbors > MAX_N_NEIGHBORS:
|
||||
best = _best_silhouette_in_window(silhouette_snapshots)
|
||||
if best is not None:
|
||||
best_nn, best_labels, best_sil = best
|
||||
best_nn, best_labels, best_sil, _ = best
|
||||
n_neighbors = best_nn
|
||||
labels = best_labels
|
||||
silhouette_avg = best_sil
|
||||
|
|
@ -1034,10 +1096,24 @@ def _save_tuning_logs(
|
|||
)
|
||||
|
||||
|
||||
def _top2_audience_clusters(
|
||||
def _config_threshold(key: str, default: float) -> float:
|
||||
try:
|
||||
import yaml
|
||||
if VOC_BUSINESS_CONFIG.is_file():
|
||||
with VOC_BUSINESS_CONFIG.open(encoding="utf-8") as f:
|
||||
cfg = yaml.safe_load(f) or {}
|
||||
val = float(cfg.get(key, default))
|
||||
if 0 < val <= 1:
|
||||
return val
|
||||
except (TypeError, ValueError, OSError):
|
||||
pass
|
||||
return default
|
||||
|
||||
|
||||
def _audience_cluster_ranking(
|
||||
conn: sqlite3.Connection, run_id: int
|
||||
) -> Tuple[List[int], Dict[int, int]]:
|
||||
"""返回 (前两簇标签列表, source_row -> audience簇标签)。"""
|
||||
) -> Tuple[Dict[int, set[int]], Dict[int, int]]:
|
||||
"""返回 (cluster_label -> source_rows, source_row -> audience簇标签)。"""
|
||||
cur = conn.execute(
|
||||
"""
|
||||
SELECT cluster_label, source_row
|
||||
|
|
@ -1054,11 +1130,46 @@ def _top2_audience_clusters(
|
|||
sr = int(sr)
|
||||
row_to_cluster[sr] = lab
|
||||
cluster_rows.setdefault(lab, set()).add(sr)
|
||||
ranked = sorted(
|
||||
cluster_rows.items(), key=lambda x: len(x[1]), reverse=True
|
||||
return cluster_rows, row_to_cluster
|
||||
|
||||
|
||||
def _top2_audience_clusters(
|
||||
conn: sqlite3.Connection, run_id: int
|
||||
) -> Tuple[List[int], Dict[int, int]]:
|
||||
"""兼容旧逻辑：返回前两大的 audience 簇。"""
|
||||
cluster_rows, row_to_cluster = _audience_cluster_ranking(conn, run_id)
|
||||
ranked = sorted(cluster_rows.items(), key=lambda x: len(x[1]), reverse=True)
|
||||
return [lab for lab, _ in ranked[:2]], row_to_cluster
|
||||
|
||||
|
||||
def _eligible_audience_clusters(
|
||||
conn: sqlite3.Connection,
|
||||
run_id: int,
|
||||
total_reviews: int,
|
||||
threshold: float = DEFAULT_AUDIENCE_COVERAGE_THRESHOLD,
|
||||
) -> Tuple[List[int], Dict[int, int]]:
|
||||
"""返回 (占比达阈值的 audience 簇列表, source_row -> audience簇标签)。"""
|
||||
cluster_rows, row_to_cluster = _audience_cluster_ranking(conn, run_id)
|
||||
if total_reviews <= 0:
|
||||
return [], row_to_cluster
|
||||
ranked = sorted(cluster_rows.items(), key=lambda x: len(x[1]), reverse=True)
|
||||
eligible = [
|
||||
lab for lab, rows in ranked
|
||||
if len(rows) / total_reviews >= threshold
|
||||
]
|
||||
if len(eligible) > 4:
|
||||
logger.warning(
|
||||
"eligible audience 簇数量=%s（阈值=%.0f%%），可能过度拆分 step 2",
|
||||
len(eligible),
|
||||
threshold * 100,
|
||||
)
|
||||
top2 = [lab for lab, _ in ranked[:2]]
|
||||
return top2, row_to_cluster
|
||||
if not eligible and ranked:
|
||||
logger.warning(
|
||||
"无 audience 簇达到 step2 阈值 %.0f%%（总评论 %s），跳过 step 2",
|
||||
threshold * 100,
|
||||
total_reviews,
|
||||
)
|
||||
return eligible, row_to_cluster
|
||||
|
||||
|
||||
def run_clustering(
|
||||
|
|
@ -1069,12 +1180,14 @@ def run_clustering(
|
|||
cluster_db: Path = CLUSTER_DB,
|
||||
reset_db: bool = True,
|
||||
) -> dict:
|
||||
"""仅运行 3a（全量 pain）和 3b（全量 aspect_opinion × 情感）聚类。
|
||||
已移除步骤1 audience 和步骤2 per-audience 聚类。
|
||||
"""
|
||||
require_chat_api_key()
|
||||
|
||||
econn = sqlite3.connect(embed_db)
|
||||
try:
|
||||
jid = _resolve_job_id(job_id, econn, structured_db)
|
||||
audience_rows = _load_embed_rows(econn, jid, "audience")
|
||||
pain_rows = _load_embed_rows(econn, jid, "pain_point")
|
||||
ao_rows = _load_embed_rows(econn, jid, "aspect_opinion")
|
||||
finally:
|
||||
|
|
@ -1094,50 +1207,6 @@ def run_clustering(
|
|||
total_reviews = _count_csv_reviews(csv_path)
|
||||
logger.info("清洗后评论总数: %s", total_reviews)
|
||||
|
||||
# --- 1 audience ---
|
||||
labels1, meta1 = _cluster_stage(
|
||||
audience_rows, stage="1_audience", use_llm_tune=True, client=client
|
||||
)
|
||||
filt1 = _save_assignments(
|
||||
cconn,
|
||||
run_id,
|
||||
"1_audience",
|
||||
audience_rows,
|
||||
labels1,
|
||||
n_neighbors=meta1.get("n_neighbors"),
|
||||
outlier_participates=False,
|
||||
)
|
||||
meta1["cluster_filter"] = filt1
|
||||
_save_stage_meta(cconn, run_id, "1_audience", meta1)
|
||||
if meta1.get("tuning_log"):
|
||||
_save_tuning_logs(cconn, run_id, "1_audience", meta1["tuning_log"])
|
||||
|
||||
top2, row_to_aud = _top2_audience_clusters(cconn, run_id)
|
||||
meta_top2 = {
|
||||
"top2_audience_clusters": top2,
|
||||
"per_cluster_source_rows": {
|
||||
str(lab): len(_source_rows_for_audience_cluster(row_to_aud, lab))
|
||||
for lab in top2
|
||||
},
|
||||
"step2_mode": "per_audience_cluster_llm_auto_tune",
|
||||
}
|
||||
_save_stage_meta(cconn, run_id, "step2_filter", meta_top2)
|
||||
logger.info("Audience 前两簇（将分别聚类）: %s", top2)
|
||||
|
||||
step2_counts: Dict[str, int] = {}
|
||||
for aud_lab in top2:
|
||||
step2_counts.update(
|
||||
_cluster_step2_for_audience(
|
||||
cconn,
|
||||
run_id,
|
||||
audience_cluster=aud_lab,
|
||||
pain_rows=pain_rows,
|
||||
ao_rows=ao_rows,
|
||||
row_to_aud=row_to_aud,
|
||||
client=client,
|
||||
)
|
||||
)
|
||||
|
||||
# --- 3a pain global ---
|
||||
labels3a, meta3a = _cluster_stage(
|
||||
pain_rows, stage="3a_pain_global", use_llm_tune=True, client=client
|
||||
|
|
@ -1175,10 +1244,7 @@ def run_clustering(
|
|||
"total_reviews": total_reviews,
|
||||
"min_cluster_review_ratio": CLUSTER_MIN_REVIEW_RATIO,
|
||||
"cluster_db": str(cluster_db.resolve()),
|
||||
"top2_audience_clusters": top2,
|
||||
"counts": {
|
||||
"1_audience": len(audience_rows),
|
||||
**step2_counts,
|
||||
"3a_pain_global": len(pain_rows),
|
||||
**step3b_counts,
|
||||
},
|
||||
|
|
|
|||
373
词频.py
373
词频.py
|
|
@ -2,7 +2,8 @@
|
|||
词频统计：从最新结构化任务读取产品与 CSV，两步连跑。
|
||||
|
||||
1. 随机 25 条 content → LLM 归纳「产品专有名词」与「Amazon/产品专属停用词」
|
||||
2. spaCy 全量 content 分词 + 词频 → output/word_freq.csv（专有名词按完整短语统计，不拆词）
|
||||
2. NLTK 全量 content 分词 + stem/lemma 并族 → output/word_freq.csv
|
||||
（专有名词按完整短语统计；每条评论每个词族最多计 1 次）
|
||||
|
||||
用法::
|
||||
|
||||
|
|
@ -15,16 +16,13 @@ import argparse
|
|||
import csv
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import Counter
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Sequence, Set, Tuple
|
||||
|
||||
from spacy.lang.en.stop_words import STOP_WORDS as EN_STOP_WORDS
|
||||
from typing import Dict, Iterable, List, Sequence, Set, Tuple
|
||||
|
||||
from voc_llm import CHAT_MODEL, chat_extra_body, create_chat_client, require_chat_api_key
|
||||
|
||||
|
|
@ -43,9 +41,146 @@ WORD_FREQ_CSV = OUTPUT_DIR / "word_freq.csv"
|
|||
|
||||
MODEL_NAME = CHAT_MODEL
|
||||
|
||||
SAMPLE_SIZE = 25
|
||||
SAMPLE_SIZE = 38
|
||||
SAMPLE_SEED = 42
|
||||
|
||||
# NLTK 英文停用词扩展（与匹配规则参考一致）
|
||||
_EXTRA_STOP_WORDS = frozenset(
|
||||
{
|
||||
"www", "http", "https", "com", "amazon", "asin", "sku",
|
||||
"oz", "lb", "lbs", "inch", "inches", "ft", "mm", "cm", "ml", "kg", "pcs", "pc",
|
||||
}
|
||||
)
|
||||
|
||||
# 产品标题拆词中可视为「不重要」、应参与停用的功能词
|
||||
_PRODUCT_NAME_FILLER = frozenset(
|
||||
{
|
||||
"a", "an", "the", "and", "or", "but", "for", "nor", "so", "yet",
|
||||
"at", "by", "in", "of", "on", "to", "up", "as", "is", "it", "be",
|
||||
"with", "from", "into", "via", "per", "vs", "vs.",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _ensure_nltk():
|
||||
"""加载 NLTK 分词 / 词性 / 词形还原依赖（首次自动下载数据包）。"""
|
||||
try:
|
||||
import nltk # noqa: F401
|
||||
except ImportError as e:
|
||||
raise RuntimeError(
|
||||
"未安装 nltk，请执行: uv pip install --python 310py/bin/python nltk"
|
||||
) from e
|
||||
import nltk
|
||||
from nltk.corpus import wordnet as wn
|
||||
from nltk.stem import PorterStemmer, WordNetLemmatizer
|
||||
from nltk.tag import pos_tag
|
||||
from nltk.tokenize import word_tokenize
|
||||
|
||||
for resource, pkg in (
|
||||
("tokenizers/punkt", "punkt"),
|
||||
("tokenizers/punkt_tab", "punkt_tab"),
|
||||
("corpora/wordnet", "wordnet"),
|
||||
("corpora/omw-1.4", "omw-1.4"),
|
||||
("taggers/averaged_perceptron_tagger", "averaged_perceptron_tagger"),
|
||||
("taggers/averaged_perceptron_tagger_eng", "averaged_perceptron_tagger_eng"),
|
||||
("corpora/stopwords", "stopwords"),
|
||||
):
|
||||
try:
|
||||
nltk.data.find(resource)
|
||||
except LookupError:
|
||||
logger.info("下载 NLTK 数据包: %s", pkg)
|
||||
nltk.download(pkg, quiet=True)
|
||||
|
||||
return word_tokenize, pos_tag, WordNetLemmatizer(), PorterStemmer(), wn
|
||||
|
||||
|
||||
def _penn_to_wn_pos(tag: str, wn) -> str:
|
||||
if tag.startswith("J"):
|
||||
return wn.ADJ
|
||||
if tag.startswith("V"):
|
||||
return wn.VERB
|
||||
if tag.startswith("N"):
|
||||
return wn.NOUN
|
||||
if tag.startswith("R"):
|
||||
return wn.ADV
|
||||
return wn.NOUN
|
||||
|
||||
|
||||
def load_nltk_stop_words() -> Set[str]:
|
||||
_ensure_nltk()
|
||||
import nltk
|
||||
from nltk.corpus import stopwords
|
||||
|
||||
try:
|
||||
words = set(stopwords.words("english"))
|
||||
except LookupError:
|
||||
nltk.download("stopwords", quiet=True)
|
||||
words = set(stopwords.words("english"))
|
||||
words |= _EXTRA_STOP_WORDS
|
||||
return words
|
||||
|
||||
|
||||
class WordVariantEngine:
|
||||
"""NLTK 分词 + stem / POS-lemma 词形变体（不含同义词并族）。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
word_tokenize, pos_tag_fn, lemmatizer, stemmer, wn = _ensure_nltk()
|
||||
self._word_tokenize = word_tokenize
|
||||
self._pos_tag = pos_tag_fn
|
||||
self._lemmatizer = lemmatizer
|
||||
self._stemmer = stemmer
|
||||
self._wn = wn
|
||||
|
||||
def variants_for_word(self, word: str, pos: str | None = None) -> Set[str]:
|
||||
w = (word or "").lower().strip()
|
||||
if not w:
|
||||
return set()
|
||||
out: Set[str] = {w, self._stemmer.stem(w)}
|
||||
if pos is not None:
|
||||
wn_pos = _penn_to_wn_pos(pos, self._wn)
|
||||
out.add(self._lemmatizer.lemmatize(w, pos=wn_pos))
|
||||
for p in (self._wn.NOUN, self._wn.VERB, self._wn.ADJ, self._wn.ADV):
|
||||
out.add(self._lemmatizer.lemmatize(w, pos=p))
|
||||
return {x for x in out if x}
|
||||
|
||||
def variant_pool(self, word: str, pos: str | None = None) -> Set[str]:
|
||||
return self.variants_for_word(word, pos=pos)
|
||||
|
||||
def pos_tag_tokens(self, tokens: Sequence[str]) -> List[Tuple[str, str]]:
|
||||
if not tokens:
|
||||
return []
|
||||
try:
|
||||
return self._pos_tag(list(tokens))
|
||||
except Exception:
|
||||
return [(t, "NN") for t in tokens]
|
||||
|
||||
|
||||
class _UnionFind:
|
||||
def __init__(self) -> None:
|
||||
self._parent: Dict[str, str] = {}
|
||||
|
||||
def add(self, x: str) -> None:
|
||||
if x not in self._parent:
|
||||
self._parent[x] = x
|
||||
|
||||
def find(self, x: str) -> str:
|
||||
self.add(x)
|
||||
while self._parent[x] != x:
|
||||
self._parent[x] = self._parent[self._parent[x]]
|
||||
x = self._parent[x]
|
||||
return x
|
||||
|
||||
def union(self, a: str, b: str) -> None:
|
||||
ra, rb = self.find(a), self.find(b)
|
||||
if ra != rb:
|
||||
self._parent[rb] = ra
|
||||
|
||||
def groups(self) -> Dict[str, List[str]]:
|
||||
out: Dict[str, List[str]] = defaultdict(list)
|
||||
for x in self._parent:
|
||||
out[self.find(x)].append(x)
|
||||
return dict(out)
|
||||
|
||||
|
||||
def _strip_think(text: str) -> str:
|
||||
if not text:
|
||||
|
|
@ -142,7 +277,8 @@ def _build_terms_prompt(
|
|||
|
||||
def _call_llm(system: str, user: str, api_key: str) -> str:
|
||||
_ = api_key
|
||||
client = create_chat_client()
|
||||
# 术语提取只需短 JSON/列表，200k max_tokens 会导致 API 长时间生成或挂起
|
||||
client = create_chat_client(timeout=600.0)
|
||||
extra_body = chat_extra_body(MODEL_NAME)
|
||||
resp = client.chat.completions.create(
|
||||
model=MODEL_NAME,
|
||||
|
|
@ -150,7 +286,8 @@ def _call_llm(system: str, user: str, api_key: str) -> str:
|
|||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
temperature=0.3,
|
||||
temperature=0.6,
|
||||
max_tokens=90000,
|
||||
extra_body=extra_body,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
|
|
@ -275,25 +412,15 @@ def _product_name_tokens(product_name: str) -> Set[str]:
|
|||
}
|
||||
|
||||
|
||||
# 产品标题拆词中可视为「不重要」、应参与停用的功能词（含 spaCy 英文停用词交集）
|
||||
_PRODUCT_NAME_FILLER = frozenset(
|
||||
{
|
||||
"a", "an", "the", "and", "or", "but", "for", "nor", "so", "yet",
|
||||
"at", "by", "in", "of", "on", "to", "up", "as", "is", "it", "be",
|
||||
"with", "from", "into", "via", "per", "vs", "vs.",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _product_name_filler_tokens(product_name: str) -> Set[str]:
|
||||
def _product_name_filler_tokens(product_name: str, nltk_stops: Set[str]) -> Set[str]:
|
||||
"""产品名拆词中的虚词/功能词 → 应停用。"""
|
||||
tokens = _product_name_tokens(product_name)
|
||||
return {t for t in tokens if t in EN_STOP_WORDS or t in _PRODUCT_NAME_FILLER}
|
||||
return {t for t in tokens if t in nltk_stops or t in _PRODUCT_NAME_FILLER}
|
||||
|
||||
|
||||
def _product_name_core_tokens(product_name: str) -> Set[str]:
|
||||
"""产品名中有分析价值的实词 → 不停用。"""
|
||||
fillers = _product_name_filler_tokens(product_name)
|
||||
fillers = _product_name_filler_tokens(product_name, load_nltk_stop_words())
|
||||
return _product_name_tokens(product_name) - fillers
|
||||
|
||||
|
||||
|
|
@ -306,10 +433,9 @@ def _finalize_term_lists(
|
|||
stopwords: List[str],
|
||||
product_name: str,
|
||||
) -> Tuple[List[str], Set[str]]:
|
||||
stop_set = set(EN_STOP_WORDS)
|
||||
stop_set = load_nltk_stop_words()
|
||||
stop_set.update(_dedupe_terms(stopwords))
|
||||
# 产品名虚词（for/the/a 等）纳入停用；实词成分保持可统计
|
||||
stop_set.update(_product_name_filler_tokens(product_name))
|
||||
stop_set.update(_product_name_filler_tokens(product_name, stop_set))
|
||||
for tok in _product_name_core_tokens(product_name):
|
||||
stop_set.discard(tok)
|
||||
pn_lower = _normalize_term(product_name)
|
||||
|
|
@ -321,24 +447,10 @@ def _finalize_term_lists(
|
|||
if t in stop_set:
|
||||
continue
|
||||
cleaned_terms.append(t)
|
||||
# 长短语优先匹配
|
||||
cleaned_terms.sort(key=lambda x: (-len(x.split()), -len(x)))
|
||||
return cleaned_terms, stop_set
|
||||
|
||||
|
||||
def _load_spacy():
|
||||
import spacy
|
||||
|
||||
try:
|
||||
return spacy.load("en_core_web_sm", disable=["ner", "parser"])
|
||||
except OSError as e:
|
||||
raise RuntimeError(
|
||||
"未安装 spaCy 英文模型，请执行: uv pip install --python 310py/bin/python "
|
||||
"'en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/"
|
||||
"en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl'"
|
||||
) from e
|
||||
|
||||
|
||||
def _phrase_pattern(phrase: str) -> re.Pattern[str]:
|
||||
parts = [re.escape(p) for p in phrase.split()]
|
||||
body = r"\s+".join(parts)
|
||||
|
|
@ -366,81 +478,91 @@ def _overlaps_span(char_start: int, char_end: int, spans: Sequence[Tuple[int, in
|
|||
return False
|
||||
|
||||
|
||||
def _apply_product_terms(
|
||||
lower: str,
|
||||
product_terms: List[str],
|
||||
counter: Counter[str],
|
||||
stop_set: Set[str],
|
||||
) -> List[Tuple[int, int]]:
|
||||
"""匹配专有名词短语：只计完整短语频次，并返回需屏蔽拆词统计的字符区间。"""
|
||||
def _collect_protected_spans(lower: str, product_terms: List[str], stop_set: Set[str]) -> List[Tuple[int, int]]:
|
||||
"""返回专有名词短语匹配区间（屏蔽拆词统计，短语单独计数）。"""
|
||||
spans: List[Tuple[int, int]] = []
|
||||
for phrase in product_terms:
|
||||
if not phrase or phrase in stop_set or _is_pure_number(phrase):
|
||||
continue
|
||||
pat = _phrase_pattern(phrase)
|
||||
hits = 0
|
||||
for m in pat.finditer(lower):
|
||||
for m in _phrase_pattern(phrase).finditer(lower):
|
||||
spans.append((m.start(), m.end()))
|
||||
hits += 1
|
||||
if hits:
|
||||
counter[phrase] += hits
|
||||
return _merge_spans(spans)
|
||||
|
||||
|
||||
def _tokenize_doc(
|
||||
nlp, text: str, protected_spans: Sequence[Tuple[int, int]]
|
||||
) -> List[str]:
|
||||
doc = nlp(text)
|
||||
tokens: List[str] = []
|
||||
for tok in doc:
|
||||
if tok.is_space or tok.is_punct:
|
||||
def _phrases_hit_in_review(lower: str, product_terms: List[str], stop_set: Set[str]) -> List[str]:
|
||||
"""本条评论命中的专有名词（每条评论每短语最多计 1 次）。"""
|
||||
hit: List[str] = []
|
||||
for phrase in product_terms:
|
||||
if not phrase or phrase in stop_set or _is_pure_number(phrase):
|
||||
continue
|
||||
char_start = tok.idx
|
||||
char_end = tok.idx + len(tok.text)
|
||||
if _overlaps_span(char_start, char_end, protected_spans):
|
||||
continue
|
||||
lemma = (tok.lemma_ or tok.text).lower().strip()
|
||||
if not lemma or _is_pure_number(lemma):
|
||||
continue
|
||||
if not re.search(r"[a-z]", lemma, re.I):
|
||||
continue
|
||||
tokens.append(lemma)
|
||||
return tokens
|
||||
if _phrase_pattern(phrase).search(lower):
|
||||
hit.append(phrase)
|
||||
return hit
|
||||
|
||||
|
||||
_SINGLE_EN_WORD = re.compile(r"^[a-z]+$")
|
||||
_TOKEN_RE = re.compile(r"[a-z0-9']+")
|
||||
|
||||
|
||||
def _should_lemma_normalize(word: str) -> bool:
|
||||
"""仅对单个英文词做词形还原；多词短语、连字符短语保持原样。"""
|
||||
w = word.strip().lower()
|
||||
if not w or " " in w or "-" in w or "'" in w:
|
||||
def _tokenize_tagged_with_spans(text: str, engine: WordVariantEngine) -> List[Tuple[str, str, int, int]]:
|
||||
"""带字符区间的分词 + 词性标注。"""
|
||||
lower = text.lower()
|
||||
raw: List[Tuple[str, int, int]] = []
|
||||
for m in _TOKEN_RE.finditer(lower):
|
||||
tok = m.group()
|
||||
if tok:
|
||||
raw.append((tok, m.start(), m.end()))
|
||||
if not raw:
|
||||
return []
|
||||
words = [t for t, _, _ in raw]
|
||||
tagged = engine.pos_tag_tokens(words)
|
||||
return [(tagged[i][0], tagged[i][1], raw[i][1], raw[i][2]) for i in range(len(raw))]
|
||||
|
||||
|
||||
def _is_valid_token(tok: str, stop_set: Set[str]) -> bool:
|
||||
if not tok or len(tok) < 1 or tok in stop_set or _is_pure_number(tok):
|
||||
return False
|
||||
return bool(_SINGLE_EN_WORD.match(w))
|
||||
return bool(re.search(r"[a-z]", tok))
|
||||
|
||||
|
||||
def _lemma_form(word: str, nlp) -> str:
|
||||
w = word.strip().lower()
|
||||
if not _should_lemma_normalize(w):
|
||||
return w
|
||||
doc = nlp(w)
|
||||
if not doc:
|
||||
return w
|
||||
tok = doc[0]
|
||||
if tok.is_space or tok.is_punct:
|
||||
return w
|
||||
lemma = (tok.lemma_ or tok.text).lower().strip()
|
||||
if not lemma or lemma == "-":
|
||||
return w
|
||||
return lemma
|
||||
def _register_review_tokens_in_uf(
|
||||
text: str,
|
||||
protected: Sequence[Tuple[int, int]],
|
||||
engine: WordVariantEngine,
|
||||
uf: _UnionFind,
|
||||
variant_index: Dict[str, Set[str]],
|
||||
stop_set: Set[str],
|
||||
) -> None:
|
||||
"""Pass 1：将本条评论 token 注册进 Union-Find（stem/lemma 并族）。"""
|
||||
seen: Set[str] = set()
|
||||
for tok, tag, start, end in _tokenize_tagged_with_spans(text, engine):
|
||||
if _overlaps_span(start, end, protected):
|
||||
continue
|
||||
if not _is_valid_token(tok, stop_set):
|
||||
continue
|
||||
if tok in seen:
|
||||
continue
|
||||
seen.add(tok)
|
||||
uf.add(tok)
|
||||
variants = engine.variant_pool(tok, pos=tag)
|
||||
related: Set[str] = set()
|
||||
for v in variants:
|
||||
related |= variant_index[v]
|
||||
for other in related:
|
||||
uf.union(tok, other)
|
||||
for v in variants:
|
||||
variant_index[v].add(tok)
|
||||
|
||||
|
||||
def _merge_word_forms(counter: Counter[str], nlp) -> Counter[str]:
|
||||
"""写入 CSV 前合并单复数/时态等词形（如 dogs→dog, bought→buy）。"""
|
||||
merged: Counter[str] = Counter()
|
||||
for word, count in counter.items():
|
||||
merged[_lemma_form(word, nlp)] += count
|
||||
return merged
|
||||
def _family_labels(uf: _UnionFind, surface_doc_freq: Counter[str]) -> Dict[str, str]:
|
||||
"""词族代表形：族内 surface 文档频次最高者，并列取最短。"""
|
||||
labels: Dict[str, str] = {}
|
||||
for root, members in uf.groups().items():
|
||||
best = sorted(
|
||||
members,
|
||||
key=lambda m: (-surface_doc_freq.get(m, 0), len(m), m),
|
||||
)[0]
|
||||
labels[root] = best
|
||||
return labels
|
||||
|
||||
|
||||
def _build_word_freq(
|
||||
|
|
@ -448,19 +570,60 @@ def _build_word_freq(
|
|||
product_terms: List[str],
|
||||
stop_set: Set[str],
|
||||
) -> Counter[str]:
|
||||
nlp = _load_spacy()
|
||||
counter: Counter[str] = Counter()
|
||||
"""
|
||||
NLTK 分词 + stem/lemma Union-Find 并族。
|
||||
每条评论：每个词族最多 +1；专有名词短语命中也最多 +1/短语。
|
||||
输出 word 为词族代表形（方案 1）。
|
||||
"""
|
||||
engine = WordVariantEngine()
|
||||
uf = _UnionFind()
|
||||
variant_index: Dict[str, Set[str]] = defaultdict(set)
|
||||
|
||||
# Pass 1：全库注册词形变体并族
|
||||
for _, text in rows:
|
||||
if not text.strip():
|
||||
continue
|
||||
protected = _collect_protected_spans(text.lower(), product_terms, stop_set)
|
||||
_register_review_tokens_in_uf(text, protected, engine, uf, variant_index, stop_set)
|
||||
|
||||
family_counter: Counter[str] = Counter()
|
||||
phrase_counter: Counter[str] = Counter()
|
||||
surface_doc_freq: Counter[str] = Counter()
|
||||
|
||||
# Pass 2：按评论计数（每词族 / 每短语最多 1 次）
|
||||
for _, text in rows:
|
||||
if not text.strip():
|
||||
continue
|
||||
lower = text.lower()
|
||||
protected = _apply_product_terms(lower, product_terms, counter, stop_set)
|
||||
for tok in _tokenize_doc(nlp, text, protected):
|
||||
if tok in stop_set:
|
||||
protected = _collect_protected_spans(lower, product_terms, stop_set)
|
||||
|
||||
for phrase in _phrases_hit_in_review(lower, product_terms, stop_set):
|
||||
phrase_counter[phrase] += 1
|
||||
|
||||
families_seen: Set[str] = set()
|
||||
seen_tok: Set[str] = set()
|
||||
for tok, tag, start, end in _tokenize_tagged_with_spans(text, engine):
|
||||
if _overlaps_span(start, end, protected):
|
||||
continue
|
||||
counter[tok] += 1
|
||||
counter = Counter({k: v for k, v in counter.items() if not _is_pure_number(k)})
|
||||
return _merge_word_forms(counter, nlp)
|
||||
if not _is_valid_token(tok, stop_set):
|
||||
continue
|
||||
if tok in seen_tok:
|
||||
continue
|
||||
seen_tok.add(tok)
|
||||
surface_doc_freq[tok] += 1
|
||||
root = uf.find(tok)
|
||||
if root not in families_seen:
|
||||
families_seen.add(root)
|
||||
family_counter[root] += 1
|
||||
|
||||
labels = _family_labels(uf, surface_doc_freq)
|
||||
merged: Counter[str] = Counter()
|
||||
for root, count in family_counter.items():
|
||||
merged[labels.get(root, root)] += count
|
||||
for phrase, count in phrase_counter.items():
|
||||
merged[phrase] += count
|
||||
|
||||
return Counter({k: v for k, v in merged.items() if v > 0 and not _is_pure_number(k)})
|
||||
|
||||
|
||||
def _save_word_freq(counter: Counter[str], path: Path) -> None:
|
||||
|
|
@ -562,7 +725,7 @@ def run(*, skip_llm: bool = False) -> dict:
|
|||
product_name,
|
||||
)
|
||||
|
||||
logger.info("第 2 步：spaCy 全量分词与词频统计")
|
||||
logger.info("第 2 步：NLTK 分词 + stem/lemma 并族词频统计")
|
||||
counter = _build_word_freq(rows, product_terms, stop_set)
|
||||
_save_word_freq(counter, WORD_FREQ_CSV)
|
||||
logger.info("已写入 %s", WORD_FREQ_CSV)
|
||||
|
|
|
|||
Loading…
Reference in a new issue