feat: 落地节点指纹与 Cookie 罐,并按现网能力更新消费/接手文档
公开页抓取改为 Chrome 136 自洽身份 + 每节点 SQLite 养罐,Trafilatura 走 curl_cffi;MCP/README/接手说明与 09-02 现网复测对齐,避免消费方继续抄过期的站点三分表。 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
parent
854ffcb2b1
commit
9b689b2476
39 changed files with 2048 additions and 189 deletions
126
README.md
126
README.md
|
|
@ -1,3 +1,127 @@
|
||||||
# onesvm-browser-server
|
# onesvm-browser-server
|
||||||
|
|
||||||
集成多种类型浏览器MCP自构建联网搜索服务
|
面向智能体的**自构建联网搜索服务**:消费者拿一把 `X-Service-Token`,用 MCP(主)或 HTTP(兜底)调用 `search` / `read`。国内直连,国外走自有代理池,query **不经过** Tavily / Jina 等境外 SaaS。
|
||||||
|
|
||||||
|
现网入口:`http://192.168.1.51:8640`(MCP `POST /mcp`,HTTP `POST /v1/search` `/v1/read`)。
|
||||||
|
|
||||||
|
| 你是谁 | 先读 |
|
||||||
|
|---|---|
|
||||||
|
| 消费方(Vlepontas / EAI) | [`docs/mcp-usage-20260901.md`](docs/mcp-usage-20260901.md) → [`docs/integration-vlepontas-20260901.md`](docs/integration-vlepontas-20260901.md) |
|
||||||
|
| 接手开发 | [`docs/dev-handoff-20260902.md`](docs/dev-handoff-20260902.md)(本文下面是设计与选型摘要) |
|
||||||
|
| 改架构 / 拍板 | [`docs/plan-final-20260901.md`](docs/plan-final-20260901.md)(决策权威)→ [`docs/design-arch-20260901.md`](docs/design-arch-20260901.md)(机制权威) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 要解决什么问题
|
||||||
|
|
||||||
|
智能体需要「上网查公开页」。买境外 SaaS 会把 query 送出境,且按次计费;自建浏览器集群又容易把整机内存和出口 IP 打爆。本仓的约束是:
|
||||||
|
|
||||||
|
- **境内闭环**:搜索与精读默认不经境外托管 API。
|
||||||
|
- **内存纪律**:常驻业务峰 ≤1GB(L2);60 = 队列深度,不是 60 个浏览器。
|
||||||
|
- **零 Redis**:SQLite WAL 单写者排队。
|
||||||
|
- **诚实能力**:公开网页大约九成;登录墙 / 强 WAF / Google 搜索不承诺。住宅 IP 是另一条带成本声明的通道,本服务不做。
|
||||||
|
|
||||||
|
人交「事件怎么活、怎么死、多快死」;实现按 [`onesvm-dev-md/base/engineering-standards.md`](../onesvm-dev-md/base/engineering-standards.md) 事件表,缺格不补默认值开工。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 设计科学(为什么长这样)
|
||||||
|
|
||||||
|
### 2.1 统一出口,双形态同一内核
|
||||||
|
|
||||||
|
所有消费者注册主体、拿可吊销 key。MCP 与 HTTP 共用 **Auth → Policy → Queue → Engine**。错误码两面对齐。静态服务密钥只走 `X-Service-Token`,禁止塞进 `Authorization: Bearer`。
|
||||||
|
|
||||||
|
### 2.2 分层内核,而不是「一个大浏览器」
|
||||||
|
|
||||||
|
流量大约 90% 是搜索 + 纯 HTTP 精读,不该进 Chromium。
|
||||||
|
|
||||||
|
| 层 | 适配器 | 何时用 |
|
||||||
|
|---|---|---|
|
||||||
|
| A 无浏览器 | SearXNG-CN / SearXNG-Global / Trafilatura | 搜索发现、静态正文 |
|
||||||
|
| B 轻渲染 | Lightpanda | HTTP 抽空、需要一点 JS |
|
||||||
|
| C 保真 | chrome-headless-shell(按需槽,默认 0 副本) | 轻核不够时 |
|
||||||
|
| 预留 | Camoufox / 有头档 | 强对抗 + 住宅 IP,不在 1GB 预算内 |
|
||||||
|
|
||||||
|
降级链:C 满 → B → A → `blocked`/`upstream`。Cloudflare / WAF / 验证页**不拿渲染空转**,直接 `blocked`。
|
||||||
|
|
||||||
|
### 2.3 出口与身份分开记账
|
||||||
|
|
||||||
|
- **出口**:国内域直连;境外经 ProxyManager + mihomo vless 热池。决定「Google 过不过」的是 IP 信誉,不是 UA 字符串。
|
||||||
|
- **身份(P0/P1,2026-09-02 落地)**:每生产节点 8 套 Chrome 136 模版(与 Trafilatura `curl_cffi` `impersonate=chrome136` 自洽);一等 Cookie 养在本节点 SQLite。罐按 **模版 × http|cdp × 出口 × eTLD+1** 隔离,节点内共享、不按消费者拆。登录态 Cookie 名直接丢弃。403/验证页整域丢罐。Cookie **不进 MCP、不进 jobs.payload**。
|
||||||
|
|
||||||
|
Go 侧 `httpx` 仍是 Go TLS,UA 写诚实的 `onesvm-browser-server-httpx/1.0`,禁止再伪装 Chrome(TLS/UA 必须自洽)。
|
||||||
|
|
||||||
|
### 2.4 合规双向保险
|
||||||
|
|
||||||
|
请求侧在网关:SSRF、仅 80/443、域名策略、普通 key 遵守 robots。响应侧在 worker 模版层:体积/类型白名单、词表、PII 脱敏、注入包裹。命中即 `denied` 并审计,不假装成功。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 选用方案(实测后入选,不是目录堆砌)
|
||||||
|
|
||||||
|
2026-09-01 本机 Docker 评分(内存 30% / 质量 25% / 成功率 20% / 延迟 15% / 运维 10%)。全文:[`docs/decision-candidates-20260901.md`](docs/decision-candidates-20260901.md)。
|
||||||
|
|
||||||
|
| 方案 | 分 | 角色 | 为什么留 |
|
||||||
|
|---|---:|---|---|
|
||||||
|
| Trafilatura-HTTP | 96.8 | 默认精读 | 60/60、p50 ~60ms、正文干净;现网已加 curl_cffi |
|
||||||
|
| Lightpanda | 91.0 | 默认 JS | idle 3.6MB;CF/Amazon 仍过不了 |
|
||||||
|
| chrome-headless-shell | 85.5 | 保真按需 | 单槽 FIFO 零拒绝;重页 ~400MB |
|
||||||
|
| SearXNG-CN | 71.5 | 国内搜索 | 政策词准;突发时百度/搜狗 CAPTCHA,靠排队钳 4–8 |
|
||||||
|
| SearXNG-Global | 62.0 | 国外搜索 | 实例稳,但数据中心 IP 下 **仅 Bing**;产品不承诺多源 |
|
||||||
|
|
||||||
|
**明确排除**:playwright-distributed(Redis)、Browserless(4GB 档)、Jina 云(query 出境)、Firecrawl 自托管(与 shell+模版层重叠且更重)。
|
||||||
|
|
||||||
|
**用户已拍板**:D1 先 L0→L2(≤1GB);D2 接受 Bing-only 起步;D5 不做 Camoufox;Amazon 本站非本服务刚需。
|
||||||
|
|
||||||
|
现网能力以 [`docs/mcp-usage-20260901.md`](docs/mcp-usage-20260901.md) §4 为准(2026-09-02 mgr1 复测),不要把 09-01 bench 的「BBC 稳取 / Amazon 一律不行」抄进消费方文档。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 仓库地图
|
||||||
|
|
||||||
|
```
|
||||||
|
server/ Go 单二进制,-role=gateway|scheduler|proxymanager
|
||||||
|
cmd/browser-server/ 入口
|
||||||
|
internal/contract/ 信封与 Job(Session 字段 json:"-")
|
||||||
|
internal/gateway/ MCP + /v1 + Auth + 策略预检
|
||||||
|
internal/scheduler/ 队列、路由、模版层整合器、junk 过滤、Cookie 绑定
|
||||||
|
internal/dock/ 五适配器(SearXNG×2、Trafilatura、Lightpanda、shell)
|
||||||
|
internal/fingerprint/ 每节点 Chrome 136 模版
|
||||||
|
internal/session/ Cookie 罐(Sanitize / TTL / reap)
|
||||||
|
internal/store/ SQLite(jobs、keys、fp_*、session_cookies)
|
||||||
|
internal/proxymanager/ 订阅 / 探活 / 域名路由
|
||||||
|
internal/policy/ SSRF / robots / 域名
|
||||||
|
internal/safetyscan/ 响应侧词表与包裹
|
||||||
|
stacks/ Swarm stack + SearXNG 配置 + Trafilatura 镜像
|
||||||
|
scripts/ smoke-local.sh、deploy-mgr1.sh(默认 dry-run)
|
||||||
|
bench/ 选型期实测(无密钥)
|
||||||
|
docs/ 决策 / 机制 / MCP / 接手
|
||||||
|
```
|
||||||
|
|
||||||
|
源文件硬上限 **600 行**(`make wc`)。禁止万能 `util` 包。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 本机怎么跑
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make test # go test(合入前加 -race:cd server && go test -race ./...)
|
||||||
|
make vet
|
||||||
|
make smoke # compose + stub 引擎,12 步断言
|
||||||
|
make image # onesvm/browser-server:dev(默认 linux/amd64)
|
||||||
|
make image-trafilatura # onesvm/trafilatura-http:dev
|
||||||
|
```
|
||||||
|
|
||||||
|
生产节点通常拉不到 Docker Hub。镜像必须在可联网机器构建/拉取,再 `docker save | ssh … docker load`,stack 用 `--resolve-image never`。**同 tag 覆盖后必须 `docker service update --force`**,否则任务还跑旧层。
|
||||||
|
|
||||||
|
mgr1 部署属 🟡:只在用户明确「确认执行」后跑 `bash scripts/deploy-mgr1.sh --apply`。密钥在 gitignore 的 `deploy.env`,不入 git。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 文档索引
|
||||||
|
|
||||||
|
完整表见 [`docs/README.md`](docs/README.md)。冲突时:
|
||||||
|
|
||||||
|
1. [`docs/plan-final-20260901.md`](docs/plan-final-20260901.md) — 决策
|
||||||
|
2. [`docs/design-arch-20260901.md`](docs/design-arch-20260901.md) — 机制
|
||||||
|
3. 代码 — 现网行为(文档会过期,以代码与 §4 现网复测为准)
|
||||||
|
|
|
||||||
|
|
@ -6,17 +6,19 @@ created: 2026-09-01
|
||||||
|
|
||||||
# docs/ 索引 — onesvm-browser-server
|
# docs/ 索引 — onesvm-browser-server
|
||||||
|
|
||||||
> 入门先读 `overview-integration-20260901.md`;拍板看 `plan-final-20260901.md`;改实现查 `design-arch-20260901.md`。
|
> 仓根 [`README.md`](../README.md) 是对外总览。接手先读 `dev-handoff-20260902.md`;拍板看 `plan-final-20260901.md`;改实现查 `design-arch-20260901.md`;消费方只读 `mcp-usage-20260901.md`。
|
||||||
|
|
||||||
| 文档 | 一句话说明 | 读者对象 | 权威关系 |
|
| 文档 | 一句话说明 | 读者对象 | 权威关系 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
|
| `../README.md` | 仓库说明:问题、设计科学、入选方案、目录与怎么跑 | 所有人 | 入门(不覆盖 plan/design) |
|
||||||
|
| `dev-handoff-20260902.md` | 接手开发:包地图、事件表、改哪一层、验证、部署坑 | 新接手工程师 | **接手路径权威** |
|
||||||
| `plan-final-20260901.md` | 最终方案 v1:档位表、实测评定结论、D1–D6 决策点 | 项目 owner(review/拍板) | **决策权威**(冲突处以它为准) |
|
| `plan-final-20260901.md` | 最终方案 v1:档位表、实测评定结论、D1–D6 决策点 | 项目 owner(review/拍板) | **决策权威**(冲突处以它为准) |
|
||||||
| `design-arch-20260901.md` | 架构机制全文:拓扑、统一出口、Dock 协议、队列、ProxyManager、O1–O13 | 实现工程师 | **机制细节权威** |
|
| `design-arch-20260901.md` | 架构机制全文:拓扑、统一出口、Dock 协议、队列、ProxyManager、O1–O13 | 实现工程师 | **机制细节权威** |
|
||||||
| `decision-candidates-20260901.md` | 消费方案选型决策记录:5 套入选理由、排除/预留清单、能力边界实测矩阵 | owner + 工程师 | 选型结论依据 |
|
| `decision-candidates-20260901.md` | 消费方案选型决策记录:5 套入选理由、排除/预留清单、能力边界实测矩阵 | owner + 工程师 | 选型结论依据(09-01 bench;现网能力以 mcp-usage §4 为准) |
|
||||||
| `overview-integration-20260901.md` | 整合方式入门导览:整合哪些/怎么整合/什么服务/怎么排队 | 新接手工程师 | 导读(不产生新结论) |
|
| `overview-integration-20260901.md` | 整合方式入门导览:整合哪些/怎么整合/什么服务/怎么排队 | 新接手工程师 | 导读(不产生新结论) |
|
||||||
| `mcp-usage-20260901.md` | 消费者侧 MCP 使用文档:接入方式、工具参数、能力数据范围、数据获取使用规则 | 智能体消费者(Vlepontas/EAI) | 消费侧规则权威 |
|
| `mcp-usage-20260901.md` | 消费者侧 MCP 使用文档:接入、工具、**2026-09-02 现网能力**、错误码 | 智能体消费者(Vlepontas/EAI) | **消费侧规则权威** |
|
||||||
| `deploy-prod-preset-20260901.md` | 生产部署预设:stack 组件、端口、镜像 pin、节点布局、镜像传输流程(审批稿) | 运维/实施 | 部署执行依据 |
|
| `deploy-prod-preset-20260901.md` | 生产部署预设:stack 组件、端口、镜像 pin、节点布局、镜像传输流程 | 运维/实施 | 部署执行依据(cgroup 以 `stacks/browser-server.yml` 为准) |
|
||||||
| `integration-vlepontas-20260901.md` | Vlepontas 联调手册:注册接入、C1–C6 验收用例、升级决策回路 | Vlepontas 侧工程师 | 接入执行依据 |
|
| `integration-vlepontas-20260901.md` | Vlepontas 接入说明:入口、认证、现网验收用例、纪律 | Vlepontas 侧工程师 | 接入执行依据(密钥不进本文) |
|
||||||
| `impl-20260901.md` | 控制面实现交付记录:怎么跑、验收链、W6 热修 | 实现/接手工程师 | 实现回执(不覆盖 design) |
|
| `impl-20260901.md` | 控制面实现交付记录:怎么跑、验收链、W6 热修 | 实现/接手工程师 | 实现回执(不覆盖 design) |
|
||||||
|
|
||||||
## 数据与验收(仓内可追溯)
|
## 数据与验收(仓内可追溯)
|
||||||
|
|
|
||||||
125
docs/dev-handoff-20260902.md
Normal file
125
docs/dev-handoff-20260902.md
Normal file
|
|
@ -0,0 +1,125 @@
|
||||||
|
---
|
||||||
|
type: runbook
|
||||||
|
status: active
|
||||||
|
created: 2026-09-02
|
||||||
|
---
|
||||||
|
|
||||||
|
# 接手开发说明
|
||||||
|
|
||||||
|
> 读者:第一次改这个仓的工程师 / Agent。读完应能:找到权威文档、在本机跑通测试、知道改哪一层、知道什么不能做。
|
||||||
|
> 决策权威:[`plan-final-20260901.md`](plan-final-20260901.md)。机制权威:[`design-arch-20260901.md`](design-arch-20260901.md)。现网能力:[`mcp-usage-20260901.md`](mcp-usage-20260901.md) §4。仓库地图:根 [`README.md`](../README.md)。
|
||||||
|
|
||||||
|
## 0. 十分钟路径
|
||||||
|
|
||||||
|
1. 读根 `README.md` §1–§3(问题、分层、入选方案)。
|
||||||
|
2. 读本文 §1–§4(目录、事件、怎么改、怎么验)。
|
||||||
|
3. `cd server && go test -race ./...` 必须绿。
|
||||||
|
4. 再按任务读:路由 → `internal/scheduler/routing.go`;抓取 → `stacks/trafilatura/app.py` + `internal/dock/`;合规 → `internal/policy/`;MCP 契约 → `internal/gateway/mcphandler.go`。
|
||||||
|
|
||||||
|
不要从 `.dsh/contracts/` 或 superseded 手稿当现行 acceptance。过程产物在 `.dsh/artifacts/`(gitignore)。
|
||||||
|
|
||||||
|
## 1. 进程与包
|
||||||
|
|
||||||
|
单二进制三角色,Swarm 里三个服务:
|
||||||
|
|
||||||
|
| `-role` | 职责 | 有状态 |
|
||||||
|
|---|---|---|
|
||||||
|
| `gateway` | MCP `/mcp`、HTTP `/v1/*`、Auth、策略预检、入队 | 无。可多副本 |
|
||||||
|
| `scheduler` | SQLite 队列、能力路由、Dock 适配器、模版层、Cookie 罐 | **有。replicas=1,钉节点** |
|
||||||
|
| `proxymanager` | 订阅、探活、选出口 | replicas=1 |
|
||||||
|
|
||||||
|
适配器在 scheduler 进程内,引擎在旁边的容器(SearXNG / Trafilatura / Lightpanda / shell)。
|
||||||
|
|
||||||
|
| 要改 | 去哪 | 不要 |
|
||||||
|
|---|---|---|
|
||||||
|
| 工具参数 / 信封 | `internal/contract/`、`gateway/mcphandler.go` | 只改文档不改 schema |
|
||||||
|
| 搜索 junk 过滤 | `scheduler/searx_parse.go` `isJunkSearchHit` | 在适配器里偷偷丢结果 |
|
||||||
|
| 精读 TLS/UA | `stacks/trafilatura/app.py`(真抓取) | 只改 Go `httpx` 假装 Chrome |
|
||||||
|
| 指纹模版 | `internal/fingerprint/` | 模版和 impersonate 错代 |
|
||||||
|
| Cookie 罐 | `internal/session/` + `store/session.go` | 把 Cookie 写进 MCP / jobs JSON |
|
||||||
|
| robots / SSRF | `internal/policy/` | fail-open 放行内网 |
|
||||||
|
| 出站代理 | `internal/proxymanager/` | 在适配器里自己拨代理 |
|
||||||
|
| 部署 | `stacks/browser-server.yml`、`scripts/deploy-mgr1.sh` | 在 Swarm 节点上 `docker pull` |
|
||||||
|
|
||||||
|
`JobEnvelope.Session` 与 `RawResult.SetCookies` 是 `json:"-"`。打破这条 = Cookie 进审计/MCP,P0。
|
||||||
|
|
||||||
|
## 2. 事件表(本服务已填、热修只改碰到的行)
|
||||||
|
|
||||||
|
| 事件 | 怎么活 | 怎么死 | 多快死 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| search/read 请求 | 认证过 → 入队 ACK → 适配器出 RawResult → 模版层信封 | 策略 `denied`;上游 `blocked`/`upstream`/`timeout` | 入站跟网关;出站 Trafilatura 15s;队列硬顶 120s |
|
||||||
|
| 排队位 | running+queued < 60 才接 | 超限 503,只拒新 | 消费者按 `Retry-After` |
|
||||||
|
| Cookie 罐 | 200 且非投毒则 Sanitize 后覆盖该域 | 403/验证页/换出口 → 整域删 | TTL 2h,硬顶 6h,60s reap |
|
||||||
|
| 指纹模版 | 节点 HOSTNAME 种子,8 套落库,内存只持 active | 不因单次失败换脸 | 进程重启从库恢复同一张脸 |
|
||||||
|
| 渲染槽 | Lightpanda 多槽;shell 默认 0 副本 | shell 忙则互斥停接 | 空闲 10min 计数(本版不自动 scale) |
|
||||||
|
| 密钥 | `X-Service-Token`,库内 hash | 缺省 fail-closed | 吊销缓存 ≤10s |
|
||||||
|
|
||||||
|
「用默认超时」不算已填。Go 出站禁止 `http.DefaultClient`。I/O 第一参必须是 `ctx`。
|
||||||
|
|
||||||
|
## 3. 改代码时的分层
|
||||||
|
|
||||||
|
```
|
||||||
|
消费者 → gateway(认证/配额/SSRF/robots)
|
||||||
|
→ scheduler 入队
|
||||||
|
→ 路由(region + 是否要渲染)
|
||||||
|
→ dock 适配器(只产 RawResult)
|
||||||
|
→ 模版层(体积守卫 / fit markdown / 安全扫描 / 信封)
|
||||||
|
```
|
||||||
|
|
||||||
|
新增第 6 个引擎:实现 `init/health/execute/teardown/capabilities`,注册路由标签,**不要**再写一套合规。对照 `docs/overview-integration-20260901.md` §2。
|
||||||
|
|
||||||
|
P0 指纹自洽:`impersonate`、UA、`sec-ch-ua`、platform 钉同一 Chrome 大版本(现网 136)。Go `httpx` 保持诚实 UA。
|
||||||
|
|
||||||
|
P1 罐硬顶:每域 20 颗、单条 4KiB、总 2MiB、64 域 LRU。登录态名(`session-token` / `sid` / `__secure-*psid` 等)直接拒收。
|
||||||
|
|
||||||
|
## 4. 验证
|
||||||
|
|
||||||
|
合入前:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make fmt && make vet
|
||||||
|
cd server && go test -race ./...
|
||||||
|
make wc # 单文件 >600 行必须拆
|
||||||
|
```
|
||||||
|
|
||||||
|
改了 MCP 参数或信封:同步 `docs/mcp-usage-20260901.md` 与 `mcphandler.go` 的 schema,并补/改 golden。
|
||||||
|
|
||||||
|
改了 Trafilatura:必须重建 `onesvm/trafilatura-http:dev`,只更 Go 镜像是半接线(真抓取在 Python)。
|
||||||
|
|
||||||
|
现网复测:用 HTTP `/v1/search` `/v1/read` 即可(与 MCP 同一内核)。不要把消费方 key 写进脚本提交。
|
||||||
|
|
||||||
|
## 5. 部署(🟡,须用户确认)
|
||||||
|
|
||||||
|
目标是 **primary mgr1 `192.168.1.51`**,不是 dev-swarm。默认 `scripts/deploy-mgr1.sh` 是 dry-run。
|
||||||
|
|
||||||
|
已知坑(已踩过,不要再踩):
|
||||||
|
|
||||||
|
1. 生产节点禁直连 Docker Hub;本机构建 amd64 → `save|load` → `--resolve-image never`。
|
||||||
|
2. **同 tag `:dev` 覆盖后必须 `docker service update --force`**,否则还跑旧任务。
|
||||||
|
3. Lightpanda CDP 握手 `Host` 必须是 `127.0.0.1:9222`,Docker 服务名会被拒。
|
||||||
|
4. Trafilatura cgroup 现网是 **192m**(64m 会 OOM 137)。`deploy-prod-preset` 里仍写 64m,以 stack 为准。
|
||||||
|
5. 海外 search 出口键用 `https://www.bing.com/`,不要拿空 URL 去 `/api/exit`。
|
||||||
|
6. 单次适配器超时不要把整实例健康闩死;只有 `/healthz` 失败或 HTTP 5xx 才摘适配器。
|
||||||
|
|
||||||
|
回滚:`docker stack rm browser-server`(命名卷保留)。不要 `git push --force`。
|
||||||
|
|
||||||
|
## 6. 文档怎么维护
|
||||||
|
|
||||||
|
| 变了什么 | 更新哪 |
|
||||||
|
|---|---|
|
||||||
|
| 工具参数 / 错误码 / 现网能力 | `mcp-usage-20260901.md`(消费方权威) |
|
||||||
|
| 联调步骤 / 验收用例 | `integration-vlepontas-20260901.md` |
|
||||||
|
| 决策(档位、Bing-only、不做 L4) | `plan-final`;不要只改 overview |
|
||||||
|
| 队列字段 / Dock 五方法 | `design-arch` |
|
||||||
|
| 选型理由 | `decision-candidates` |
|
||||||
|
| 接手路径 / 现网坑 | **本文** |
|
||||||
|
|
||||||
|
NFP:规范进组织 `base/`,本仓只写本服务事实。密钥只在 `deploy.env` / 组织 `credentials.md`,禁止写进 docs。
|
||||||
|
|
||||||
|
## 7. 不要做的事
|
||||||
|
|
||||||
|
- 为「让 Google 出结果」去伪装 Chrome + Go TLS,或给消费方换出口绕 robots。
|
||||||
|
- 把 Amazon 登录 Cookie / TOTP 账号池搬进本服务(DaaS 的实证:决定变量是出口 IP)。
|
||||||
|
- 未填事件表就加新中间件(Redis、第二套队列)。
|
||||||
|
- 未授权 commit / push;用户没说「确认执行」就 `--apply` 到 mgr1。
|
||||||
|
- 用 2026-09-01 bench 的站点三分表覆盖现网 §4(BBC/百科/Amazon 结论已漂移)。
|
||||||
|
|
@ -2,84 +2,121 @@
|
||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-09-01
|
created: 2026-09-01
|
||||||
|
updated: 2026-09-02
|
||||||
---
|
---
|
||||||
|
|
||||||
# Vlepontas 联调使用说明(草稿 · 部署批准后生效)
|
# Vlepontas 联调与接入说明
|
||||||
|
|
||||||
> 状态说明:mgr1 测试期已部署(2026-09-02)。入口 `http://192.168.1.51:8640`,overlay alias `browser-server`。
|
> 现网已开通(2026-09-02)。入口 `http://192.168.1.51:8640`,overlay alias `browser-server`。
|
||||||
> 详细工具参数/错误码/能力范围见《MCP 使用文档》`docs/mcp-usage-20260901.md`,本文是联调操作手册。
|
> 工具参数 / 错误码 / **现网能力边界** 以 [`mcp-usage-20260901.md`](mcp-usage-20260901.md) 为准。本文是可转给 Vlepontas 的操作说明。
|
||||||
|
> 密钥明文不进本仓;签发后线下传递,丢失只能吊销重签。
|
||||||
|
|
||||||
## 1. 联调目标
|
## 1. 这是什么
|
||||||
|
|
||||||
验证 Vlepontas 真实业务流在本服务上的表现,回答三个问题:① 数据能力够不够(覆盖/质量);② 性能体感是否达标(延迟/排队);③ 是否需要升级数据能力(L4 强对抗档/官方 API)或性能(L3 1.5GB)。
|
onesvm-browser-server 是 **自构建联网搜索**:国内直连、国外走自有代理池,query **不经过** Tavily / Jina 等境外 SaaS。给智能体两类工具:
|
||||||
|
|
||||||
## 2. 接入步骤
|
| 工具 | 做什么 | 不做什么 |
|
||||||
|
|---|---|---|
|
||||||
|
| `search` | 关键词发现 URL + ≤800 字 snippet | 不合成答案、不保证多源 |
|
||||||
|
| `read` | 单 URL 抽 fit markdown | 不破登录墙、不对抗验证码、遵守 robots |
|
||||||
|
|
||||||
### 2.1 注册与拿 key
|
定位:覆盖大约九成「能被搜索引擎看见、robots 不禁、不是强 WAF」的公开页。亚马逊本站数据走你们已有的 DaaS 通道,**不要**把本服务当 Amazon 抓取器。
|
||||||
|
|
||||||
1. Vlepontas 侧提供:主体标识 `vlepontas`、联系人、预期负载(峰值会话数、搜索/读取/渲染比例)。
|
## 2. 接入
|
||||||
2. 管理侧签发:`X-Service-Token: bs_vlep_…`(明文只回传一次,线下传递);默认 scope `search,read`;如需 `extract`(JSON schema 抽取)单独申请并审计。
|
|
||||||
3. 配额预设:rpm=120、daily=50k、concurrent_sessions=60(对齐 60 会话级上限;联调期可放宽观察)。
|
|
||||||
|
|
||||||
### 2.2 MCP 接入(主路径)
|
### 2.1 入口
|
||||||
|
|
||||||
Vlepontas casa-worker 走 overlay(按 WSG 先例附加 `vlepontas-casa-net`):
|
| 路径 | 地址 |
|
||||||
|
|---|---|
|
||||||
|
| MCP(主)overlay | `http://browser-server:8640/mcp` |
|
||||||
|
| MCP(主)mgr1 | `http://192.168.1.51:8640/mcp` |
|
||||||
|
| HTTP 兜底 | `POST http://192.168.1.51:8640/v1/search`、`/v1/read` |
|
||||||
|
|
||||||
|
协议:MCP Streamable HTTP,`POST /mcp`。overlay 需已加入 `vlepontas-casa-net`(alias `browser-server`,与 WSG 先例相同)。
|
||||||
|
|
||||||
|
### 2.2 认证(铁律)
|
||||||
|
|
||||||
|
- 头名:**`X-Service-Token`**
|
||||||
|
- 🔴 **禁止**放进 `Authorization: Bearer`(下游按 JWT 解析会 401 `Not enough segments`)
|
||||||
|
- 主体:`vlepontas`;默认 scope:`search`、`read`
|
||||||
|
- 明文只在签发时回传一次,进你们密钥库,**不要进 git**
|
||||||
|
|
||||||
|
签发记录(管理侧):`key_id=8`,scopes `search,read`。明文已线下交付;本文不写密钥。
|
||||||
|
|
||||||
|
### 2.3 MCP 配置示例
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"mcpServers": {
|
"mcpServers": {
|
||||||
"onesvm-browser-server": {
|
"onesvm-browser-server": {
|
||||||
"url": "http://browser-server:8640/mcp",
|
"url": "http://browser-server:8640/mcp",
|
||||||
"headers": { "X-Service-Token": "bs_vlep_…" }
|
"headers": { "X-Service-Token": "<从密钥库读取,勿提交>" }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
工具两个:`search`(发现 URL,`region: domestic|overseas` 自选)与 `read`(URL 精读,默认 fit markdown)。
|
`tools/list` 应看到 `search`、`read`。`initialize` / `tools/list` / `ping` 免 key;`tools/call` 必须带 token。
|
||||||
|
|
||||||
### 2.3 HTTP 兜底(排错/非 MCP 链路)
|
### 2.4 HTTP 兜底(排错)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -X POST -H 'X-Service-Token: bs_vlep_…' -H 'Content-Type: application/json' \
|
curl -sS -X POST http://192.168.1.51:8640/v1/search \
|
||||||
http://192.168.1.51:8640/v1/search \
|
-H 'X-Service-Token: <key>' -H 'Content-Type: application/json' \
|
||||||
-d '{"query":"跨境电商 出口退税 政策 2026","region":"domestic","max_results":5}'
|
-d '{"query":"跨境电商 出口退税 2026","region":"domestic","max_results":5}'
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2.4 纪律(与现有 WSG/Tavily 通道的差异)
|
客户端超时建议:**国内 search ≤12s,国外 search ≤15s,read ≤30s**(现网搜索 1–10s,不要抄 bench 的 1s)。
|
||||||
|
|
||||||
- 错误码语义:`denied`=合规拦截(不要重试对抗);`timeout/upstream` 可重试(≤2 次,服从 `Retry-After`);429/503 带 `running/queued` 现状,自行退避。
|
## 3. 现网能力(2026-09-02 复测,可当验收基线)
|
||||||
- 队列深度 60 = 排队位 + 在途合计;超限收 503 是背压不是故障。
|
|
||||||
- 渲染(JS 页)是稀缺槽位:先 `read` 默认通道,空正文再升级;不要对 `blocked` 站点(Amazon/Medium/Reddit/X/知乎/微博/百度百科/SO/YouTube)反复打。
|
|
||||||
|
|
||||||
## 3. 联调验收用例(双方共同执行,预期值来自 2026-09-01 实测)
|
### 3.1 search 够用的场景
|
||||||
|
|
||||||
| # | 用例 | 预期 |
|
| region | 适用 | 不要指望 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| C1 | 国内搜索「跨境电商 出口退税 政策 2026」region=domestic | ≥5 条政策域结果,p50 ≤1s |
|
| `domestic` | 政策/监管/平台治理(国税、商务部、海关、东财转载) | 供应链词会被 1688 淹没;百科全文 `read` 常被 robots 拒 |
|
||||||
| C2 | `read` 该政策页(gov.cn 系) | 壳页 HTTP 通道可能抽空 → 升级 JS 通道得全文(实测 9577 字/约1s) |
|
| `overseas` | 带年份的市场规模 / 品类专名(`nutraceuticals market 2026`) | Google;单词 query(`dietary` / `posture`)会被吸到词典 |
|
||||||
| C3 | 国外搜索「best bluetooth earbuds 2026」region=overseas | 有结果但**仅 Bing 源**(D2 已定);p50 ≤5s |
|
|
||||||
| C4 | `read` 一个 Shopify 独立站商品页 | 可能被 302 到集合页:系列/价格可见,指定 SKU 不保证——记录业务影响 |
|
|
||||||
| C5 | 60 会话 burst 混合负载(40 搜索+15 读+5 渲染) | 无 5xx 风暴;搜索/读全过;渲染排队(p95 等待可测) |
|
|
||||||
| C6 | 访问一个 deny 名单域(联调时指定) | `error.code=denied` + 审计记录,验证合规拦截 |
|
|
||||||
|
|
||||||
## 4. 观察指标与反馈回路
|
海外 **仅 Bing**。这是数据中心出口 IP 的客观限制,不是接线故障。
|
||||||
|
|
||||||
联调期 Vlepontas 侧记录:各工具感知延迟、空结果率、denied 命中、渲染等待体感。管理侧提供 `/pressure` 指标快照(队列深度/渲染槽占用/内存比/代理池存活)。
|
### 3.2 read 够用的场景
|
||||||
|
|
||||||
**升级决策回路**(联调结论 → 行动):
|
| 结果 | 代表 |
|
||||||
|
|
||||||
| 若联调发现 | 行动 |
|
|
||||||
|---|---|
|
|---|---|
|
||||||
| 渲染等待 p95 超标 / shell 冷启动被感知 | 升 L3 1.5GB(shell 常驻 + 渲染免互斥) |
|
| 成功 | 中等政府站、财经转载、部分 Wiki(升 CDP)、**部分** Amazon 商品详情 |
|
||||||
| Amazon 详情页/Review | **非刚需**(2026-09-01 用户确认:Vlepontas 有既有消费方案覆盖亚马逊数据,本服务不规划 Amazon 方向;L4 档仅理论预留) |
|
| `denied` robots | 百度百科、搜狐正文、百家号 —— 用 search snippet,禁止重试 |
|
||||||
| 舆情(Reddit/X/知乎/微博)成刚需 | Reddit 商用 API(~$0.24/千次)/ RSSHub+Cookie 边车(知乎/订阅流);X 维持 C 档(无免费读档) |
|
| `blocked` CF | BBC、Mordor、咨询付费墙 —— 换下一条 |
|
||||||
| 国外搜索质量不满(Bing-only 偏题) | 采购优质/住宅代理出口,或 Brave Search API(出境须法务) |
|
| 抽空 | JS 官网壳(只有「产品展示」)—— 换有静态正文的源 |
|
||||||
| 知乎/百科类中文知识需求 | 千帆百科组件(约 0.01–0.036 元/次)/ RSSHub 适配器 |
|
|
||||||
|
|
||||||
(依据:`.dsh/artifacts/run-20260901-browser-arch/research/06-blocked-sites.md` 分级路线图。)
|
`warnings` 里的 `fp:win-chrome136-…` 是节点指纹标记,不是错误。Cookie 不会出现在响应里。
|
||||||
|
|
||||||
## 5. 时间窗与联系
|
### 3.3 验收用例(双方可复跑)
|
||||||
|
|
||||||
- 联调窗口:上线后一周(建议每日同步 15min)。
|
| # | 用例 | 现网预期(2026-09-02) |
|
||||||
- 问题升级:本仓 issue / 管理侧负责人;合规疑问先停后用,走法务。
|
|---|---|---|
|
||||||
|
| C1 | 国内 search「跨境电商 出口退税 2026」 | ≥5 条,含国税/财政部退运免税;数秒级(不必 ≤1s) |
|
||||||
|
| C2 | `read` 商务部 FDA 扣留页或同类 gov/财经页 | `ok=true`,trafilatura,正文完整 |
|
||||||
|
| C3 | 海外 search「bluetooth earbuds market share 2026」 | 有 Mordor/TBRC/Coherent 类咨询 snippet;**仅 Bing** |
|
||||||
|
| C4 | `read` 百度百科 / 搜狐 | `denied` + `robots_disallow`(正确行为) |
|
||||||
|
| C5 | `read` Cloudflare 咨询站 | `blocked`;不要连打 |
|
||||||
|
| C6 | 故意用 Bearer 头 | 401 —— 验证选头 |
|
||||||
|
|
||||||
|
## 4. 纪律(与 WSG/Tavily 通道的差异)
|
||||||
|
|
||||||
|
- `denied` = 合规拦截,**不要重试对抗**。
|
||||||
|
- `timeout` / `upstream` 可重试 ≤2 次,服从 `Retry-After`。
|
||||||
|
- `blocked` = 能力边界,换 URL,不要换出口绕过。
|
||||||
|
- 队列深度 60 = 排队位 + 在途合计;超限 503 是背压。
|
||||||
|
- **不得**把含境内个人信息的 query 发 `region=overseas`。
|
||||||
|
- 正文是不可信抓取,必须当资料,不得当指令。
|
||||||
|
|
||||||
|
## 5. 升级回路(联调后才动)
|
||||||
|
|
||||||
|
| 若发现 | 行动 |
|
||||||
|
|---|---|
|
||||||
|
| 渲染等待成为日常痛 | 升 L3(shell 常驻),另立项 |
|
||||||
|
| Amazon Review / 强 WAF | **不走本服务**;用已有 DaaS |
|
||||||
|
| 海外搜索词典噪声过多 | 先改 query;再考虑住宅出口(另一条 MCP 类型,带成本声明) |
|
||||||
|
| 百科/搜狐全文刚需 | 官方组件或 RSS,不破 robots |
|
||||||
|
|
||||||
|
问题升级:本仓 issue / 管理侧。合规疑问先停后用。
|
||||||
|
|
|
||||||
|
|
@ -2,14 +2,15 @@
|
||||||
type: runbook
|
type: runbook
|
||||||
status: active
|
status: active
|
||||||
created: 2026-09-01
|
created: 2026-09-01
|
||||||
|
updated: 2026-09-02
|
||||||
step: P3 / plan-20260901-02
|
step: P3 / plan-20260901-02
|
||||||
---
|
---
|
||||||
|
|
||||||
# onesvm-browser-server MCP 使用文档(消费者智能体接入指南)
|
# onesvm-browser-server MCP 使用文档(消费者智能体接入指南)
|
||||||
|
|
||||||
> 面向消费方智能体(Vlepontas / EAI / 其它注册主体)的开发者。读完本文即可独立完成接入与正确使用。
|
> 面向消费方智能体(Vlepontas / EAI / 其它注册主体)的开发者。读完本文即可独立完成接入与正确使用。
|
||||||
> 版本:v0 设计版(2026-09-01)。**§4 能力数据范围为真实实测**(2026-09-01 本机 Docker,darwin arm64;生产 amd64 部署后同架构复测,数字可能微调)。
|
> 版本:v1 现网版(2026-09-02)。**§4 能力数据以 mgr1 现网复测为准**(2026-09-02);2026-09-01 本机 bench 仅作对照。
|
||||||
> 地址/端口为**预设值,部署后以正式通知为准**。
|
> 现网入口已开通(mgr1),见 §1.1。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -17,12 +18,13 @@ step: P3 / plan-20260901-02
|
||||||
|
|
||||||
### 1.1 Endpoint
|
### 1.1 Endpoint
|
||||||
|
|
||||||
| 项 | 值(预设) |
|
| 项 | 值 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 协议 | MCP over **Streamable HTTP**(`POST /mcp`,2026 无状态规范;无 SSE 旧路径;2025 兼容腿待 O9 决策,首版不带) |
|
| 协议 | MCP over **Streamable HTTP**(`POST /mcp`,2026 无状态规范;无 SSE 旧路径) |
|
||||||
| 内网直连(overlay) | `http://browser-server:8640/mcp` |
|
| 现网入口(mgr1) | `http://192.168.1.51:8640/mcp` |
|
||||||
| 经反代 | `{网关地址}/bs-api/mcp`(urlapi 前缀 strip 后转发) |
|
| overlay(Vlepontas casa-net) | `http://browser-server:8640/mcp` |
|
||||||
| HTTP 兜底(非 MCP 消费者/调试用) | `POST /bs-api/v1/search`、`POST /bs-api/v1/read`(同一内核、同一错误码) |
|
| HTTP 兜底 | `POST http://192.168.1.51:8640/v1/search`、`POST http://192.168.1.51:8640/v1/read`(同一内核、同一错误码) |
|
||||||
|
| 经反代 | `{网关地址}/bs-api/mcp`(urlapi 前缀 strip 后转发;未挂时用上面直连) |
|
||||||
|
|
||||||
### 1.2 认证
|
### 1.2 认证
|
||||||
|
|
||||||
|
|
@ -89,7 +91,7 @@ step: P3 / plan-20260901-02
|
||||||
"params": {
|
"params": {
|
||||||
"name": "search",
|
"name": "search",
|
||||||
"arguments": {
|
"arguments": {
|
||||||
"query": "best bluetooth earbuds 2026",
|
"query": "bluetooth earbuds market share 2026",
|
||||||
"region": "overseas",
|
"region": "overseas",
|
||||||
"max_results": 10,
|
"max_results": 10,
|
||||||
"lang": "en-US"
|
"lang": "en-US"
|
||||||
|
|
@ -98,43 +100,45 @@ step: P3 / plan-20260901-02
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**典型响应**(示例性质:由 2026-09-01 实测样本改写为统一信封格式,字段真实、数值为示例):
|
**典型响应**(示例性质:由 2026-09-02 现网样本改写为统一信封格式,字段真实、数值为示例):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"ok": true,
|
"ok": true,
|
||||||
"kind": "search",
|
"kind": "search",
|
||||||
"request_id": "01K3…ulid",
|
"request_id": "01K3…ulid",
|
||||||
"took_ms": 633,
|
"took_ms": 4737,
|
||||||
"query": "跨境电商 出口退税 政策 2026",
|
"query": "跨境电商 出口退税 2026",
|
||||||
"answer": null,
|
"answer": null,
|
||||||
"results": [
|
"results": [
|
||||||
{
|
{
|
||||||
"id": "r1",
|
"id": "r1",
|
||||||
"title": "海关总署 税务总局关于跨境电子商务出口退运商品税收优惠政策…",
|
"title": "财政部 海关总署 税务总局关于跨境电子商务出口退运商品税收优惠...",
|
||||||
"url": "https://hainan.chinatax.gov.cn/xxgk_6_1/06163393.html",
|
"url": "http://beijing.chinatax.gov.cn/bjswj/c104602/202602/ef4be608f0644f2785fee688ba66ea76.shtml",
|
||||||
"content": "一、对自2026年1月1日至2027年12月31日期间在跨境电子商务海关监管代码(1210、9610、9710、9810)项下申报出口,因滞销、退货原因…",
|
"content": "一、对自2026年1月1日至2027年12月31日期间在跨境电子商务海关监管代码(1210、9610、9710、9810)项下申报出口…",
|
||||||
"score": 0.92,
|
"score": 0.92,
|
||||||
"engine": "baidu",
|
"engine": "baidu",
|
||||||
"published_at": null
|
"published_at": null
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "r2",
|
"id": "r2",
|
||||||
"title": "雨果跨境-跨境电商品牌出海产业互联网平台",
|
"title": "2026出口退(免)税新规来了,跨境电商/外贸老板必看",
|
||||||
"url": "https://m.cifnews.com/",
|
"url": "https://news.sohu.com/a/984080199_120347161",
|
||||||
"content": "雨果跨境以雨果网作为流量依托,致力于为跨境电商从业者提供全球产业出海…",
|
"content": "从2026年1月30日后这些糟心事都解决了!.跨境销…",
|
||||||
"score": 0.71,
|
"score": 0.71,
|
||||||
"engine": "bing"
|
"engine": "bing"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"usage": { "credits": 1, "engine": "searxng-cn", "tokens_estimate": 350 },
|
"usage": { "credits": 1, "engine": "searxng-cn", "tokens_estimate": 350 },
|
||||||
"provenance": { "adapter": "searxng-cn", "proxy_exit": "none", "cached": false, "retrieved_at": "2026-09-01T11:41:15+08:00" },
|
"provenance": { "adapter": "searxng-cn", "proxy_exit": "none", "cached": false, "retrieved_at": "2026-09-02T16:40:00+08:00" },
|
||||||
"error": null
|
"error": null
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
字段纪律:`results` 为空时是 `[]` 而非 `null`;`content` 为 ≤800 字符的 query 相关片段(**非全文**,全文请用 `read`);`answer` 恒为 `null`(本服务不做 LLM 答案合成)。
|
字段纪律:`results` 为空时是 `[]` 而非 `null`;`content` 为 ≤800 字符的 query 相关片段(**非全文**,全文请用 `read`);`answer` 恒为 `null`(本服务不做 LLM 答案合成)。
|
||||||
|
|
||||||
|
服务端会丢掉明显无引用价值的壳页(Amazon 首页/账号/影音导航、无 snippet 的门户根路径),再裁 `max_results`。词典站 / 百科泛匹配**尚未**全部滤掉,请在消费侧再筛。
|
||||||
|
|
||||||
### 2.2 `read` — 单页精读
|
### 2.2 `read` — 单页精读
|
||||||
|
|
||||||
| 参数 | 类型 | 默认 | 说明 |
|
| 参数 | 类型 | 默认 | 说明 |
|
||||||
|
|
@ -143,6 +147,7 @@ step: P3 / plan-20260901-02
|
||||||
| `formats` | string[] | `["markdown"]` | 可选加 `links` / `images`;`html` / `screenshot` 为特权 scope |
|
| `formats` | string[] | `["markdown"]` | 可选加 `links` / `images`;`html` / `screenshot` 为特权 scope |
|
||||||
| `max_chars` | int | 20000 | 正文截断;截断时响应 `truncated=true` |
|
| `max_chars` | int | 20000 | 正文截断;截断时响应 `truncated=true` |
|
||||||
| `extract` | object | 无 | **特权**:`{schema: {...}, prompt?: "…"}`,JSON Schema 结构化抽取 |
|
| `extract` | object | 无 | **特权**:`{schema: {...}, prompt?: "…"}`,JSON Schema 结构化抽取 |
|
||||||
|
| `region` | `domestic` \| `overseas` | 按 URL 路由 | 可选;不填时由网关按域名策略选出口 |
|
||||||
|
|
||||||
**调用示例 1(政策页精读):**
|
**调用示例 1(政策页精读):**
|
||||||
|
|
||||||
|
|
@ -151,7 +156,7 @@ step: P3 / plan-20260901-02
|
||||||
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
|
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
|
||||||
"params": {
|
"params": {
|
||||||
"name": "read",
|
"name": "read",
|
||||||
"arguments": { "url": "https://hainan.chinatax.gov.cn/xxgk_6_1/06163393.html" }
|
"arguments": { "url": "https://chinawto.mofcom.gov.cn/article/jsbl/dtxx/202603/20260303621741.shtml" }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -172,32 +177,37 @@ step: P3 / plan-20260901-02
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**典型响应**(示例性质:由实测样本改写;该税务公告页实测 5/5 提取成功、1122 字、无导航残渣):
|
**典型响应**(2026-09-02 现网:商务部 FDA 扣留页,trafilatura + chrome136 指纹,211ms / 470 字):
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"ok": true,
|
"ok": true,
|
||||||
"kind": "read",
|
"kind": "read",
|
||||||
"request_id": "01K3…ulid",
|
"request_id": "01K3…ulid",
|
||||||
"took_ms": 58,
|
"took_ms": 211,
|
||||||
"url": "https://hainan.chinatax.gov.cn/xxgk_6_1/06163393.html",
|
"url": "https://chinawto.mofcom.gov.cn/article/jsbl/dtxx/202603/20260303621741.shtml",
|
||||||
"final_url": "https://hainan.chinatax.gov.cn/xxgk_6_1/06163393.html",
|
"final_url": "https://chinawto.mofcom.gov.cn/article/jsbl/dtxx/202603/20260303621741.shtml",
|
||||||
"title": "财政部 海关总署 税务总局关于跨境电子商务出口退运商品税收优惠政策的公告(…2026年第16号)",
|
"title": "WTO/FTA咨询网",
|
||||||
"description": null,
|
"description": null,
|
||||||
"markdown": "| 索引号 | 11460000008174507Q/2026-14228 | …\n\n为支持跨境电子商务新业态发展,现将…公告如下:\n\n一、对自2026年1月1日至2027年12月31日期间…",
|
"markdown": "近日,美国FDA网站更新了进口预警措施(import alert)…预警编号 99-45…",
|
||||||
"truncated": false,
|
"truncated": false,
|
||||||
"char_count": 1122,
|
"char_count": 470,
|
||||||
"metadata": { "status_code": 200, "content_type": "text/html", "language": "zh", "retrieved_at": "2026-09-01T11:42:45+08:00" },
|
"warnings": ["fp:win-chrome136-943094"],
|
||||||
|
"metadata": { "status_code": 200, "content_type": "text/html", "language": "zh", "retrieved_at": "2026-09-02T16:50:00+08:00" },
|
||||||
"links": null, "images": null, "html": null, "screenshot_url": null,
|
"links": null, "images": null, "html": null, "screenshot_url": null,
|
||||||
"extracted": null,
|
"extracted": null,
|
||||||
"usage": { "credits": 1, "engine": "trafilatura", "tokens_estimate": 281 },
|
"usage": { "credits": 1, "engine": "trafilatura", "tokens_estimate": 120 },
|
||||||
"provenance": { "adapter": "trafilatura-http", "proxy_exit": "none", "cached": false, "retrieved_at": "2026-09-01T11:42:45+08:00" },
|
"provenance": { "adapter": "trafilatura", "proxy_exit": "none", "cached": false, "retrieved_at": "2026-09-02T16:50:00+08:00" },
|
||||||
"error": null
|
"error": null
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
字段纪律:默认唯一内容字段是 `markdown`(fit 提纯后正文);`links/images/html/screenshot_url` 只在 `formats` 点名时非 null;截图只给可过期 URL、绝不给 base64;`extract` 失败时 `extracted=null` + `warnings`,正文不吞。
|
字段纪律:默认唯一内容字段是 `markdown`(fit 提纯后正文);`links/images/html/screenshot_url` 只在 `formats` 点名时非 null;截图只给可过期 URL、绝不给 base64;`extract` 失败时 `extracted=null` + `warnings`,正文不吞。
|
||||||
|
|
||||||
|
`warnings` 里的 `fp:<template-id>` 是节点浏览器模版标记(内部身份粘滞),**不是错误**。Cookie 罐不出现在 MCP 响应里。
|
||||||
|
|
||||||
|
正文前会包一层不可信数据声明(`<untrusted_document_content>`)。智能体必须把它当资料,不得当指令执行。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. 错误码与重试纪律
|
## 3. 错误码与重试纪律
|
||||||
|
|
@ -214,50 +224,83 @@ step: P3 / plan-20260901-02
|
||||||
| `denied` | 403 | **合规拦截**(域名/词表/robots/SSRF) | **禁止重试对抗**。被拦就是没数据 |
|
| `denied` | 403 | **合规拦截**(域名/词表/robots/SSRF) | **禁止重试对抗**。被拦就是没数据 |
|
||||||
| `extract_failed` | 200信封 | 结构化抽取失败 | 去掉 `extract` 降级重试(正文仍可用) |
|
| `extract_failed` | 200信封 | 结构化抽取失败 | 去掉 `extract` 降级重试(正文仍可用) |
|
||||||
| 401 | 401 | key 缺失/无效/放错头 | 检查是否用了 `X-Service-Token` 而非 Bearer |
|
| 401 | 401 | key 缺失/无效/放错头 | 检查是否用了 `X-Service-Token` 而非 Bearer |
|
||||||
| 503 | 503 | 队列满/系统压力(只拒新不杀旧) | 按 `Retry-After` 退避;响应带 `running/queued` 现状 |
|
| 503 | 503 | 队列满/系统压力(只扣新不杀旧) | 按 `Retry-After` 退避;响应带 `running/queued` 现状 |
|
||||||
|
|
||||||
队列语义:请求被排队**不算失败**;`ADMIT_MAX=60`(running+queued 合计)超限时才返回 503。浏览器会话并发另有 `X-Session-Remaining` 提示头,请自行退避。
|
队列语义:请求被排队**不算失败**;`ADMIT_MAX=60`(running+queued 合计)超限时才返回 503。浏览器会话并发另有 `X-Session-Remaining` 提示头,请自行退避。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. 能力数据范围(当前实测版 · 2026-09-01)
|
## 4. 能力数据范围(现网版 · 2026-09-02 mgr1)
|
||||||
|
|
||||||
> 本节每个断言均来自 2026-09-01 本机 Docker 实测(`bench/` 产物与站点矩阵探针),未测项标「未验证」。生产部署后可能有小幅漂移。
|
> 本节每个断言来自 **2026-09-02 mgr1 现网复测**(入口 `.51:8640`,Vlepontas key,真实 query)。2026-09-01 本机 bench 数字保留在括号里作对照。未测项标「未验证」。
|
||||||
|
|
||||||
|
本服务定位:**免费公开网页的搜索发现 + 精读**,覆盖大约九成「能被搜索引擎看见、且 robots 不禁、且不是强 WAF」的页面。不做登录墙、不做验证码对抗、不承诺 Amazon/Google 级强站。住宅 IP / 强对抗是另一条带成本声明的通道,**本服务不提供**。
|
||||||
|
|
||||||
### 4.1 搜索发现(`search`)
|
### 4.1 搜索发现(`search`)
|
||||||
|
|
||||||
**国内(region=domestic)**——SearXNG 聚合四引擎,单发质量满分(政策类 query 实测 5/5 断言通过):
|
**国内(`region=domestic`)**——`searxng-cn` 聚合百度 / 必应中国 / 360 / 搜狗(已去掉 wikipedia,避免百科噪声占位)。
|
||||||
|
|
||||||
| 引擎 | 单发可用性 | 高并发突发表现(60 同发实测) | 含义 |
|
| 引擎 | 现网表现 | 含义 |
|
||||||
|---|---|---|---|
|
|
||||||
| 必应中国 | ✅ 稳定 | **60/60 扛住** | 主力 |
|
|
||||||
| 360 | ✅ 稳定 | 53/60 | 主力 |
|
|
||||||
| 百度 | ✅ 单发好 | 4/60(CAPTCHA) | 突发时可能缺席 |
|
|
||||||
| 搜狗 | ⚠️ 连续调用第 3 次起 CAPTCHA | 0/60 | 仅低速补充 |
|
|
||||||
|
|
||||||
→ 你的体验:常规节奏搜索结果完整(多引擎聚合);**若你的上游短时间猛打,结果会变少**(不是故障,是上游反爬)。服务端已做排队限速(对上游钳 4–8 并发 + 短缓存)尽量消化。
|
|
||||||
|
|
||||||
**国外(region=overseas)**——**当前仅 Bing 可用**:实测 Google / DuckDuckGo / Brave / Startpage / Qwant 在现有数据中心代理出口下全部 CAPTCHA/429。Bing 结果可能存在偏题(实测出现过把多词 query 截成首词的样本)。**禁止假定多源聚合能力**;需要对等多源质量请联系管理员(决策点:更优代理 / Brave API)。
|
|
||||||
|
|
||||||
### 4.2 URL 直取(`read`)——站点三分能力表(20 站 × 2 通道实测)
|
|
||||||
|
|
||||||
服务自动路由「纯 HTTP 提取 → 轻量 JS 渲染 → 保真渲染」降级链,你无需关心;只需知道边界:
|
|
||||||
|
|
||||||
| 分类 | 能力 | 实测站点 |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| ✅ 稳取 | 公开资讯 / 文档 / Wiki / 代码托管 / 政策页(走 JS 通道) / 企业公开页 | Wikipedia、GitHub、MDN、BBC、TechCrunch、LinkedIn 公开公司页、gov.cn、36氪、HN、新华英文 |
|
| 必应中国 | 稳定出结果 | 主力 |
|
||||||
| ⚠️ 部分 | Shopify 系独立站:商品 URL 常 302 到集合页,**能拿到系列与价格,不保证落到指定 SKU** | Allbirds 实测 |
|
| 360 | 稳定出结果 | 主力 |
|
||||||
| ❌ 打不过(返回 `blocked`) | **Amazon 详情页(WAF)、Medium(Cloudflare)、Reddit(匿名 403)、X(登录墙)、知乎(403)、微博(302 访客墙)、百度百科(滑块)、StackOverflow(CF)、YouTube(无正文结构)** | 当前代理出口客观限制,升级住宅代理/强对抗档前请勿规划这些源 |
|
| 百度 | 单发好;短时间猛打会 CAPTCHA | 常规节奏可用 |
|
||||||
|
| 搜狗 | 连续调用易 CAPTCHA | 仅低速补充 |
|
||||||
|
|
||||||
### 4.3 性能档位(实测)
|
现网样本(2026-09-02):
|
||||||
|
|
||||||
| 指标 | 值 |
|
| query | 耗时 | 条数 | 能用的 |
|
||||||
|---|---|
|
|---|---:|---:|---|
|
||||||
| 国内搜索 p50 / p95 | 0.63s / 0.79s |
|
| `跨境电商 出口退税 2026` | 4.7s | 6 | 国税/财政部退运免税公告、海外仓「离境即退税」、12366;雨果/社区首页是噪声 |
|
||||||
| 国外搜索 p50 / p95 | 4.7s / 4.9s(Bing 经代理) |
|
| `背背佳 矫正 市场 供应链 品牌` | 1.6s | 8 | 百科 snippet(可孚 / 杜国楹)、官网、搜狐测评;**1688 批发占一半** |
|
||||||
| 正文读取 p50 / p95 | 0.06s / 0.12s(纯 HTTP 通道);JS 通道约 1–2s |
|
| `膳食补充剂 跨境 监管 市场 2026` | 9.9s | 8 | 商务部 FDA 扣留、东财/中华网抖音治理、海关办法;百科「膳食」与指南 PDF 是偏题 |
|
||||||
| 渲染吞吐 | 轻渲染 2.13 jobs/s(4 槽);保真渲染 0.81 jobs/s(单槽 FIFO) |
|
|
||||||
| 队列深度 | 60(running+queued;超时硬顶 120s) |
|
→ 常规政策/监管词能出一手链。供应链词会被批发站淹没。突发猛打时结果变少(上游反爬,不是故障)。
|
||||||
|
|
||||||
|
**国外(`region=overseas`)**——**当前仅 Bing 可用**。Google / DuckDuckGo / Brave / Startpage / Qwant 在现有数据中心代理出口下全部 CAPTCHA/429。**禁止假定多源聚合。**
|
||||||
|
|
||||||
|
现网样本:
|
||||||
|
|
||||||
|
| query | 耗时 | 能用的 | 噪声 |
|
||||||
|
|---|---:|---|---|
|
||||||
|
| `bluetooth earbuds market share 2026` | 3.4s | Mordor / TBRC($12.22B→$13.4B(2026)→$19.21B)/ Coherent($13.70B, CAGR 9.8%)/ GMI | bluetooth.com、Wiki、Windows 配对说明 |
|
||||||
|
| `posture corrector` | 1.0–5.4s | Amazon 商品 `B0GJTTXSRQ`、测评站 | Cleveland Clinic / MedlinePlus / Webster「posture」词典 |
|
||||||
|
| `dietary supplement` | 4.9s | 几乎全是词典 / DGA | **换词再搜** |
|
||||||
|
| `nutraceuticals market 2026` | 1.7s | Grand View 全球 2026 **$683.8B**;美国 2026 **$189B** → 2035 **$282B** | Wiki / 剑桥 / 爱词霸 |
|
||||||
|
|
||||||
|
→ 海外搜索质量取决于**你怎么写 query**。泛词会被 Bing 吸到词典;带 `market` / 品类专名 / 年份 才出咨询数字。snippet 里的数字通常够用,咨询站正文常被 Cloudflare 挡住(见 §4.2)。
|
||||||
|
|
||||||
|
### 4.2 URL 直取(`read`)
|
||||||
|
|
||||||
|
服务自动走「Trafilatura(curl_cffi + Chrome 136 指纹 + 节点 Cookie 罐)→ 空抽再升 Lightpanda CDP」。你不用选通道。
|
||||||
|
|
||||||
|
| 分类 | 现网结论 | 2026-09-02 样本 |
|
||||||
|
|---|---|---|
|
||||||
|
| ✅ 稳取 | 中等政府站、财经转载、部分 Wiki(升 CDP)、部分 Amazon **商品详情** | 商务部 WTO 网 211ms / 470 字;东财 205ms / 290 字;Amazon `B0GJTTXSRQ` 2.5s / 2500 字;Wiki Earbuds 经 Lightpanda 成功 |
|
||||||
|
| ⚠️ 抽空 | JS 壳页:HTTP 200 但正文几乎没有 | 可孚官网 `bbk.cofoe.com.cn` 只抽出「人气产品/产品展示」(13 字) |
|
||||||
|
| ❌ `denied`(robots) | 本仓遵守 `robots.txt`,请求不出站 | 百度百科、搜狐正文、百家号 → `rule_id=robots_disallow` |
|
||||||
|
| ❌ `blocked`(WAF/CF) | 强站仍过不了 | BBC、Mordor、companieshistory(`vendor=cloudflare`);部分 gov.cn 会 502 空抽 |
|
||||||
|
| ❌ 登录墙 / 验证码 | 不做对抗 | Reddit / X / 知乎 / 微博 / Medium 维持原结论 |
|
||||||
|
|
||||||
|
相对 2026-09-01 本机矩阵的变化(请按现网改规划):
|
||||||
|
|
||||||
|
- **Amazon 商品详情不再是「一律打不过」**:至少一款公开 listing 已用 Trafilatura 抽出规格表。Amazon **搜索壳 / 账号页**仍被 junk 过滤;Review / WAF 挑战页仍可能 `blocked`。亚马逊本站数据**不是本服务承诺范围**——Vlepontas 已有 DaaS 等通道。
|
||||||
|
- **百度百科**现网是 **robots 拒绝**,不是滑块。search snippet 能看,`read` 不要点。
|
||||||
|
- **BBC / 咨询付费墙**(Mordor 等)现网是 Cloudflare,不要当稳取。
|
||||||
|
|
||||||
|
Cookie / 指纹对消费者不可见:每生产节点 8 套 Chrome 136 模版,一等 Cookie 养在本节点 SQLite;403/验证页整域丢罐。**不要**指望靠换 key 换脸。
|
||||||
|
|
||||||
|
### 4.3 性能档位
|
||||||
|
|
||||||
|
| 指标 | 2026-09-02 现网 | 2026-09-01 bench 对照 |
|
||||||
|
|---|---|---|
|
||||||
|
| 国内搜索 | 1.6–9.9s(看上游引擎) | p50 0.63s / p95 0.79s |
|
||||||
|
| 国外搜索 | 1.0–5.4s(Bing + 代理) | p50 4.7s / p95 4.9s |
|
||||||
|
| 正文读取(HTTP) | 50–250ms 常见;Amazon 约 2.5s | p50 0.06s / p95 0.12s |
|
||||||
|
| JS / CDP 通道 | Wiki 升 Lightpanda 约数秒 | 轻渲染约 1–2s |
|
||||||
|
| 队列深度 | 60(running+queued;超时硬顶 120s) | 同左 |
|
||||||
|
|
||||||
|
现网搜索比 bench 慢,是真实引擎 + 代理,不是网关排队。请按 **国内 ≤12s、国外 ≤15s** 做客户端超时,不要抄 bench 的 1s。
|
||||||
|
|
||||||
### 4.4 体积 / 类型 / 格式限制
|
### 4.4 体积 / 类型 / 格式限制
|
||||||
|
|
||||||
|
|
@ -265,6 +308,15 @@ step: P3 / plan-20260901-02
|
||||||
- 默认只产 fit markdown;`rawHtml` / 截图为特权 scope。
|
- 默认只产 fit markdown;`rawHtml` / 截图为特权 scope。
|
||||||
- 单页正文默认截断 20000 字符(`max_chars` 可调,截断必显式 `truncated=true`)。
|
- 单页正文默认截断 20000 字符(`max_chars` 可调,截断必显式 `truncated=true`)。
|
||||||
|
|
||||||
|
### 4.5 给智能体的用法纪律(现网总结)
|
||||||
|
|
||||||
|
1. **先 search 再 read**。search 的 `content` 不是全文。
|
||||||
|
2. **国内政策/监管**优先 `region=domestic`;海外市场规模用带年份的专名,不要用单词(`dietary` / `posture`)。
|
||||||
|
3. `read` 收到 `denied` + `robots_disallow`:用 search snippet,不要换 URL 参数硬刷。
|
||||||
|
4. `read` 收到 `blocked` + cloudflare:换下一条结果,不要连打。
|
||||||
|
5. `ok=true` 但 `char_count` 很小:多半是 JS 壳,换有静态正文的源(政府公告、财经转载)。
|
||||||
|
6. 不得把 `region=overseas` 用在含境内个人信息的 query 上。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. 合规使用规则(消费侧必须遵守)
|
## 5. 合规使用规则(消费侧必须遵守)
|
||||||
|
|
@ -287,7 +339,7 @@ step: P3 / plan-20260901-02
|
||||||
九成是把 key 塞进了 `Authorization: Bearer`。改用 `X-Service-Token: bs_…`。其余是 key 已吊销/过期,找管理员。
|
九成是把 key 塞进了 `Authorization: Bearer`。改用 `X-Service-Token: bs_…`。其余是 key 已吊销/过期,找管理员。
|
||||||
|
|
||||||
**Q2:403 `denied`?**
|
**Q2:403 `denied`?**
|
||||||
合规拦截命中(域名黑名单 / 词表 / robots / SSRF 防护)。响应里有 `rule_id`;确认你的请求没踩 §5 红线。确属误判,带 `request_id` 找管理员复核规则。
|
合规拦截命中(域名黑名单 / 词表 / robots / SSRF)。响应里有 `rule_id`;确认你的请求没踩 §5 红线。现网最常见的是 `robots_disallow`(百度百科 / 搜狐 / 百家号)。确属误判,带 `request_id` 找管理员复核规则。
|
||||||
|
|
||||||
**Q3:429 / 503?**
|
**Q3:429 / 503?**
|
||||||
429 = 你自己超 rpm,按 `Retry-After` 退避;503 = 系统队列满(60),同样按 `Retry-After`。两者都只拒新请求,在途任务不受影响。持续 503 说明该错峰或找管理员扩容。
|
429 = 你自己超 rpm,按 `Retry-After` 退避;503 = 系统队列满(60),同样按 `Retry-After`。两者都只拒新请求,在途任务不受影响。持续 503 说明该错峰或找管理员扩容。
|
||||||
|
|
@ -296,17 +348,20 @@ step: P3 / plan-20260901-02
|
||||||
日/月配额耗尽,不可重试。找管理员提额。
|
日/月配额耗尽,不可重试。找管理员提额。
|
||||||
|
|
||||||
**Q5:`read` 返回的 markdown 很短或为空?**
|
**Q5:`read` 返回的 markdown 很短或为空?**
|
||||||
两种典型:① 目标站是「壳页」(如部分政府网联播页),纯 HTTP 通道会宁可弃取也不给脏数据——服务端会自动升级 JS 通道重试;② 目标站在 §4.2 ❌ 清单里,此时应是 `blocked` 而非空正文。若 `ok=true` 但正文质量差,带 `request_id` 反馈。
|
三种典型:① JS 壳页(官方商城等),HTTP 抽空后会升 CDP,仍可能只有导航字;② 目标站在 §4.2 ❌ 清单;③ `ok=true` 但质量差——带 `request_id` 反馈。不要把 13 字的「产品展示」当成功业务数据。
|
||||||
|
|
||||||
**Q6:搜索结果突然变少 / 引擎变少?**
|
**Q6:搜索结果突然变少 / 引擎变少?**
|
||||||
上游反爬波动(尤其百度/搜狗在突发后 CAPTCHA)。服务端引擎矩阵在监控;非故障,通常数分钟自恢复。持续异常找管理员看引擎成功率矩阵。
|
上游反爬波动(尤其百度/搜狗在突发后 CAPTCHA)。服务端引擎矩阵在监控;非故障,通常数分钟自恢复。持续异常找管理员看引擎成功率矩阵。
|
||||||
|
|
||||||
**Q7:国外搜索质量不满意?**
|
**Q7:国外搜索质量不满意?**
|
||||||
已知限制:当前仅 Bing(§4.1)。需要 Google/DDG 级多源质量,向管理员提(代理升级或官方 API 通道,涉出境合规评估)。
|
已知限制:当前仅 Bing(§4.1)。先把 query 写成「品类 + market + 年份」,不要用单词。需要 Google 级多源质量,向管理员提(代理升级或官方 API,涉出境合规评估)。
|
||||||
|
|
||||||
**Q8:能不能爬 Amazon / Reddit / …?**
|
**Q8:能不能爬 Amazon / Reddit / 百科全文?**
|
||||||
当前不能(§4.2 ❌ 清单)。这是出口 IP 信誉问题,不是服务故障;强对抗能力在预留档(Camoufox + 住宅代理),启用与否由产品决策。
|
- Amazon **商品详情**:现网有成功样本,不保证每条 listing、不保证 Review。
|
||||||
|
- Reddit / X / 知乎 / 微博 / Medium:当前不能。
|
||||||
|
- 百度百科 / 搜狐 / 百家号全文:robots 禁止,search snippet 可用。
|
||||||
|
这是出口信誉 + 合规,不是服务故障。强对抗 / 住宅 IP 不在本服务范围内。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*机制细节权威:`docs/design-arch-20260901.md`;能力数据来源:`docs/decision-candidates-20260901.md` §6 与 `.dsh/artifacts/run-20260901-browser-arch/site-matrix.md`。*
|
*机制细节权威:`docs/design-arch-20260901.md`;选型:`docs/decision-candidates-20260901.md`;仓库说明:仓库根 `README.md`;接手:`docs/dev-handoff-20260902.md`。*
|
||||||
|
|
|
||||||
|
|
@ -27,10 +27,12 @@ import (
|
||||||
"onesvm.com/onesvm/browser-server/internal/auth"
|
"onesvm.com/onesvm/browser-server/internal/auth"
|
||||||
"onesvm.com/onesvm/browser-server/internal/config"
|
"onesvm.com/onesvm/browser-server/internal/config"
|
||||||
"onesvm.com/onesvm/browser-server/internal/dock"
|
"onesvm.com/onesvm/browser-server/internal/dock"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/fingerprint"
|
||||||
gateway "onesvm.com/onesvm/browser-server/internal/gateway"
|
gateway "onesvm.com/onesvm/browser-server/internal/gateway"
|
||||||
"onesvm.com/onesvm/browser-server/internal/policy"
|
"onesvm.com/onesvm/browser-server/internal/policy"
|
||||||
"onesvm.com/onesvm/browser-server/internal/proxymanager"
|
"onesvm.com/onesvm/browser-server/internal/proxymanager"
|
||||||
"onesvm.com/onesvm/browser-server/internal/scheduler"
|
"onesvm.com/onesvm/browser-server/internal/scheduler"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/session"
|
||||||
"onesvm.com/onesvm/browser-server/internal/store"
|
"onesvm.com/onesvm/browser-server/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -223,6 +225,14 @@ func roleRunScheduler(ctx context.Context, addr string, logger *log.Logger) erro
|
||||||
core := scheduler.NewCore(st, reg, tmpl,
|
core := scheduler.NewCore(st, reg, tmpl,
|
||||||
config.EnvDefault("BROWSER_SERVER_PROXYMANAGER_URL", "http://proxymanager:8642"),
|
config.EnvDefault("BROWSER_SERVER_PROXYMANAGER_URL", "http://proxymanager:8642"),
|
||||||
workers, admitMax, logger)
|
workers, admitMax, logger)
|
||||||
|
jar, jarErr := session.Open(ctx, st, fingerprint.NodeID())
|
||||||
|
if jarErr != nil {
|
||||||
|
logger.Printf("警告:会话罐初始化失败(本轮不养 Cookie): %v", jarErr)
|
||||||
|
} else {
|
||||||
|
core.SetJar(jar)
|
||||||
|
p := jar.Active()
|
||||||
|
logger.Printf("指纹模版 active=%s impersonate=%s", p.ID, p.Impersonate)
|
||||||
|
}
|
||||||
core.Start(ctx)
|
core.Start(ctx)
|
||||||
defer core.Stop()
|
defer core.Stop()
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -31,6 +31,20 @@ func Now() time.Time {
|
||||||
return time.Now().In(TZ)
|
return time.Now().In(TZ)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// FormatTime 东八区 RFC3339(落库时间字段)。
|
||||||
|
func FormatTime(t time.Time) string {
|
||||||
|
return t.In(TZ).Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseRFC3339 解析 RFC3339 并归一东八区。
|
||||||
|
func ParseRFC3339(s string) (time.Time, error) {
|
||||||
|
t, err := time.Parse(time.RFC3339, s)
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}, err
|
||||||
|
}
|
||||||
|
return t.In(TZ), nil
|
||||||
|
}
|
||||||
|
|
||||||
// MustEnv 读取必填环境变量;缺失或空串直接 panic(fail-closed,无默认值兜底)。
|
// MustEnv 读取必填环境变量;缺失或空串直接 panic(fail-closed,无默认值兜底)。
|
||||||
// 仅用于密钥/凭据类配置。
|
// 仅用于密钥/凭据类配置。
|
||||||
func MustEnv(name string) string {
|
func MustEnv(name string) string {
|
||||||
|
|
|
||||||
|
|
@ -385,6 +385,7 @@ type JobEnvelope struct {
|
||||||
Priority int `json:"priority"`
|
Priority int `json:"priority"`
|
||||||
SubmittedAt Time `json:"submitted_at"` // 东八区
|
SubmittedAt Time `json:"submitted_at"` // 东八区
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
|
Session *SessionAttach `json:"-"` // Execute 注入;禁止落 jobs.payload / MCP
|
||||||
}
|
}
|
||||||
|
|
||||||
// 任务状态。
|
// 任务状态。
|
||||||
|
|
@ -407,6 +408,8 @@ type RawResult struct {
|
||||||
Headers map[string]string `json:"headers"`
|
Headers map[string]string `json:"headers"`
|
||||||
Engine string `json:"engine"`
|
Engine string `json:"engine"`
|
||||||
Extra map[string]any `json:"extra"` // 引擎私有附加数据(score 位次、published_at 等)
|
Extra map[string]any `json:"extra"` // 引擎私有附加数据(score 位次、published_at 等)
|
||||||
|
SetCookies []Cookie `json:"-"` // 内部回写罐;禁止进信封
|
||||||
|
Poisoned bool `json:"-"` // 验证页/403;调度器整域丢罐
|
||||||
}
|
}
|
||||||
|
|
||||||
// Render 渲染能力分级(design-arch §4.4)。
|
// Render 渲染能力分级(design-arch §4.4)。
|
||||||
|
|
|
||||||
33
server/internal/contract/session.go
Normal file
33
server/internal/contract/session.go
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
package contract
|
||||||
|
|
||||||
|
// AdapterClass 会话罐分轨:HTTP 档与 CDP 档身份不同,禁止混罐。
|
||||||
|
const (
|
||||||
|
AdapterHTTP = "http"
|
||||||
|
AdapterCDP = "cdp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Cookie 目标站一等 Cookie(内部流转;禁止进 MCP 信封)。
|
||||||
|
type Cookie struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Value string `json:"value"`
|
||||||
|
Domain string `json:"domain,omitempty"`
|
||||||
|
Path string `json:"path,omitempty"`
|
||||||
|
Expires int64 `json:"expires,omitempty"` // unix 秒;0=用默认 TTL
|
||||||
|
}
|
||||||
|
|
||||||
|
// SessionAttach 调度器在 Execute 前注入,json:"-" 不进 jobs.payload。
|
||||||
|
type SessionAttach struct {
|
||||||
|
TemplateID string
|
||||||
|
Impersonate string
|
||||||
|
UserAgent string
|
||||||
|
Headers map[string]string
|
||||||
|
Cookies []Cookie
|
||||||
|
AdapterClass string
|
||||||
|
Egress string
|
||||||
|
ETLD string
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithSession 给任务挂会话(不改已落库 payload)。
|
||||||
|
func (j *JobEnvelope) WithSession(s *SessionAttach) {
|
||||||
|
j.Session = s
|
||||||
|
}
|
||||||
|
|
@ -28,7 +28,7 @@ type cdpBrowserAdapter struct {
|
||||||
host string // "host:9222"(CDP 调试端点)
|
host string // "host:9222"(CDP 调试端点)
|
||||||
|
|
||||||
sem chan struct{}
|
sem chan struct{}
|
||||||
execFn func(ctx context.Context, host, url string) (*contract.RawResult, *contract.ErrBody)
|
execFn func(ctx context.Context, host, url string, sess *contract.SessionAttach) (*contract.RawResult, *contract.ErrBody)
|
||||||
startupMs int64 // Init 探活耗时
|
startupMs int64 // Init 探活耗时
|
||||||
healthy atomic.Bool
|
healthy atomic.Bool
|
||||||
lastErr string
|
lastErr string
|
||||||
|
|
@ -149,9 +149,9 @@ func (a *cdpBrowserAdapter) executeCommon(ctx context.Context, job contract.JobE
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeTimeout, Message: a.name + " 槽位等待取消"}
|
return nil, &contract.ErrBody{Code: contract.CodeTimeout, Message: a.name + " 槽位等待取消"}
|
||||||
}
|
}
|
||||||
a.touchActive()
|
a.touchActive()
|
||||||
res, eb := a.execFn(ctx, a.host, job.Read.URL)
|
res, eb := a.execFn(ctx, a.host, job.Read.URL, job.Session)
|
||||||
if eb != nil {
|
if eb != nil {
|
||||||
a.markHealth(eb.Code == contract.CodeUpstream, eb.Message)
|
// 单页失败(含目标站 403/空正文)不当整实例不健康。
|
||||||
return nil, eb
|
return nil, eb
|
||||||
}
|
}
|
||||||
a.markHealth(true, "")
|
a.markHealth(true, "")
|
||||||
|
|
|
||||||
|
|
@ -11,7 +11,9 @@ import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
@ -46,25 +48,44 @@ type cdpEvent struct {
|
||||||
|
|
||||||
// cdpVersionGet GET {host}/json/version → webSocketDebuggerUrl。
|
// cdpVersionGet GET {host}/json/version → webSocketDebuggerUrl。
|
||||||
// host 形如 "127.0.0.1:9222"。
|
// host 形如 "127.0.0.1:9222"。
|
||||||
func cdpVersionGet(host string) (string, error) {
|
func cdpVersionGet(host string) (wsURL, debuggerURL string, err error) {
|
||||||
hc := &http.Client{Timeout: cdpHTTPTimeout}
|
hc := &http.Client{Timeout: cdpHTTPTimeout}
|
||||||
resp, err := hc.Get("http://" + host + "/json/version")
|
resp, err := hc.Get("http://" + host + "/json/version")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("dock: GET /json/version: %w", err)
|
return "", "", fmt.Errorf("dock: GET /json/version: %w", err)
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
var v struct {
|
var v struct {
|
||||||
WebSocketDebuggerURL string `json:"webSocketDebuggerUrl"`
|
WebSocketDebuggerURL string `json:"webSocketDebuggerUrl"`
|
||||||
}
|
}
|
||||||
if err := json.NewDecoder(resp.Body).Decode(&v); err != nil {
|
if err := json.NewDecoder(resp.Body).Decode(&v); err != nil {
|
||||||
return "", fmt.Errorf("dock: /json/version 解码: %w", err)
|
return "", "", fmt.Errorf("dock: /json/version 解码: %w", err)
|
||||||
}
|
}
|
||||||
if v.WebSocketDebuggerURL == "" {
|
if v.WebSocketDebuggerURL == "" {
|
||||||
return "", fmt.Errorf("dock: /json/version 缺 webSocketDebuggerUrl")
|
return "", "", fmt.Errorf("dock: /json/version 缺 webSocketDebuggerUrl")
|
||||||
}
|
}
|
||||||
// rewriteWs(cdp_fetch.mjs 同语义):把 ws host:port 对齐 HTTP 端点
|
// rewriteWs(cdp_fetch.mjs 同语义):把 ws host:port 对齐 HTTP 端点
|
||||||
//(调试端点可能回环地址不同)。
|
//(调试端点可能回环地址不同)。Host 头仍用引擎自报的回环地址——
|
||||||
return rewriteWsURL(v.WebSocketDebuggerURL, host), nil
|
// Lightpanda 对 Host: lightpanda:9222 回 403 Host not allowed。
|
||||||
|
return rewriteWsURL(v.WebSocketDebuggerURL, host), v.WebSocketDebuggerURL, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// wsHandshakeHost 从引擎自报的 debugger URL 取 Host 头。回环地址保留,
|
||||||
|
// 以便 TCP 拨 Docker 服务名时握手仍用 127.0.0.1:port。
|
||||||
|
func wsHandshakeHost(debuggerURL string) string {
|
||||||
|
u, err := url.Parse(debuggerURL)
|
||||||
|
if err != nil || u.Hostname() == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
host := u.Hostname()
|
||||||
|
if host != "127.0.0.1" && host != "localhost" && host != "::1" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
port := u.Port()
|
||||||
|
if port == "" {
|
||||||
|
port = "9222"
|
||||||
|
}
|
||||||
|
return net.JoinHostPort(host, port)
|
||||||
}
|
}
|
||||||
|
|
||||||
// rewriteWsURL 把 ws URL 的 host:port 替换为 HTTP 端点的 host:port。
|
// rewriteWsURL 把 ws URL 的 host:port 替换为 HTTP 端点的 host:port。
|
||||||
|
|
@ -82,11 +103,11 @@ func rewriteWsURL(wsURL, httpHost string) string {
|
||||||
|
|
||||||
// newCdpClient 拨号并启动读泵。
|
// newCdpClient 拨号并启动读泵。
|
||||||
func newCdpClient(host string, dialTimeout time.Duration) (*cdpClient, error) {
|
func newCdpClient(host string, dialTimeout time.Duration) (*cdpClient, error) {
|
||||||
wsURL, err := cdpVersionGet(host)
|
wsURL, debuggerURL, err := cdpVersionGet(host)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
ws, err := wsDial(wsURL, dialTimeout)
|
ws, err := wsDial(wsURL, dialTimeout, wsHandshakeHost(debuggerURL))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
|
||||||
75
server/internal/dock/cdp_cookies.go
Normal file
75
server/internal/dock/cdp_cookies.go
Normal file
|
|
@ -0,0 +1,75 @@
|
||||||
|
package dock
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
)
|
||||||
|
|
||||||
|
// cdpApplySession UA + Cookie 注入(失败不致命:Lightpanda 可能缺 Network.setCookies)。
|
||||||
|
func cdpApplySession(send func(string, any, time.Duration) (json.RawMessage, error), sess *contract.SessionAttach) {
|
||||||
|
ua := cdpUA
|
||||||
|
if sess != nil && sess.UserAgent != "" {
|
||||||
|
ua = sess.UserAgent
|
||||||
|
}
|
||||||
|
_, _ = send("Network.setUserAgentOverride", map[string]any{"userAgent": ua}, 8*time.Second)
|
||||||
|
if sess == nil || len(sess.Cookies) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
payload := make([]map[string]any, 0, len(sess.Cookies))
|
||||||
|
for _, c := range sess.Cookies {
|
||||||
|
if c.Name == "" || c.Value == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
item := map[string]any{"name": c.Name, "value": c.Value, "path": c.Path}
|
||||||
|
if item["path"] == "" {
|
||||||
|
item["path"] = "/"
|
||||||
|
}
|
||||||
|
if c.Domain != "" {
|
||||||
|
item["domain"] = c.Domain
|
||||||
|
} else if sess.ETLD != "" {
|
||||||
|
item["domain"] = sess.ETLD
|
||||||
|
}
|
||||||
|
payload = append(payload, item)
|
||||||
|
}
|
||||||
|
if len(payload) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, _ = send("Network.setCookies", map[string]any{"cookies": payload}, 8*time.Second)
|
||||||
|
}
|
||||||
|
|
||||||
|
// cdpCollectCookies 拉取当前页一等 Cookie(失败返回 nil)。
|
||||||
|
func cdpCollectCookies(send func(string, any, time.Duration) (json.RawMessage, error), pageURL string) []contract.Cookie {
|
||||||
|
raw, err := send("Network.getCookies", map[string]any{"urls": []string{pageURL}}, 5*time.Second)
|
||||||
|
if err != nil || len(raw) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var wrap struct {
|
||||||
|
Cookies []struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Value string `json:"value"`
|
||||||
|
Domain string `json:"domain"`
|
||||||
|
Path string `json:"path"`
|
||||||
|
Expires float64 `json:"expires"`
|
||||||
|
} `json:"cookies"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal(raw, &wrap) != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]contract.Cookie, 0, len(wrap.Cookies))
|
||||||
|
for _, c := range wrap.Cookies {
|
||||||
|
if c.Name == "" || c.Value == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
item := contract.Cookie{Name: c.Name, Value: c.Value, Domain: c.Domain, Path: c.Path}
|
||||||
|
if c.Expires > 0 {
|
||||||
|
item.Expires = int64(c.Expires)
|
||||||
|
}
|
||||||
|
out = append(out, item)
|
||||||
|
if len(out) >= 20 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
@ -16,10 +16,11 @@ import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"onesvm.com/onesvm/browser-server/internal/contract"
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/fingerprint"
|
||||||
)
|
)
|
||||||
|
|
||||||
// cdpUA bench UA(Chrome 151,与 httpx.UA 同源)。
|
// cdpUA 默认模版 UA(Chrome 136;有 Session 时用模版覆盖)。
|
||||||
const cdpUA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.109 Safari/537.36"
|
var cdpUA = fingerprint.Allocate("cdp-default").UserAgent
|
||||||
|
|
||||||
// extractJS 页面提取脚本(逐字移植 cdp_fetch.mjs EXTRACT_JS)。
|
// extractJS 页面提取脚本(逐字移植 cdp_fetch.mjs EXTRACT_JS)。
|
||||||
const extractJS = `(() => {
|
const extractJS = `(() => {
|
||||||
|
|
@ -114,7 +115,7 @@ const navTimeout = 20 * time.Second
|
||||||
|
|
||||||
// cdpFetchPage 完整一次 CDP 抓取:createTarget→attach→enable→UA→navigate→
|
// cdpFetchPage 完整一次 CDP 抓取:createTarget→attach→enable→UA→navigate→
|
||||||
// 等 load→evaluate 提取→closeTarget。返回 RawResult 或 blocked/upstream 错误。
|
// 等 load→evaluate 提取→closeTarget。返回 RawResult 或 blocked/upstream 错误。
|
||||||
func cdpFetchPage(ctx context.Context, host, pageURL string) (*contract.RawResult, *contract.ErrBody) {
|
func cdpFetchPage(ctx context.Context, host, pageURL string, sess *contract.SessionAttach) (*contract.RawResult, *contract.ErrBody) {
|
||||||
c, err := newCdpClient(host, 8*time.Second)
|
c, err := newCdpClient(host, 8*time.Second)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: "CDP 端点不可达: " + err.Error()}
|
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: "CDP 端点不可达: " + err.Error()}
|
||||||
|
|
@ -148,7 +149,7 @@ func cdpFetchPage(ctx context.Context, host, pageURL string) (*contract.RawResul
|
||||||
for _, m := range []string{"Page.enable", "Runtime.enable", "Network.enable"} {
|
for _, m := range []string{"Page.enable", "Runtime.enable", "Network.enable"} {
|
||||||
_, _ = send(m, map[string]any{}, 8*time.Second)
|
_, _ = send(m, map[string]any{}, 8*time.Second)
|
||||||
}
|
}
|
||||||
_, _ = send("Network.setUserAgentOverride", map[string]any{"userAgent": cdpUA}, 8*time.Second)
|
cdpApplySession(send, sess)
|
||||||
|
|
||||||
// 事件收集器:loadEventFired / responseReceived / navigate errorText。
|
// 事件收集器:loadEventFired / responseReceived / navigate errorText。
|
||||||
col := &navCollector{}
|
col := &navCollector{}
|
||||||
|
|
@ -245,6 +246,7 @@ func cdpFetchPage(ctx context.Context, host, pageURL string) (*contract.RawResul
|
||||||
ex = *ev.Result.Value
|
ex = *ev.Result.Value
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
gotCookies := cdpCollectCookies(send, firstNonEmpty(ex.FinalURL, col.finalURL, pageURL))
|
||||||
// closeTarget 收尾(失败忽略)。
|
// closeTarget 收尾(失败忽略)。
|
||||||
_, _ = c.Send(ctx, "Target.closeTarget", map[string]any{"targetId": tgt.TargetID}, "", 5*time.Second)
|
_, _ = c.Send(ctx, "Target.closeTarget", map[string]any{"targetId": tgt.TargetID}, "", 5*time.Second)
|
||||||
|
|
||||||
|
|
@ -269,6 +271,7 @@ func cdpFetchPage(ctx context.Context, host, pageURL string) (*contract.RawResul
|
||||||
"text_len": ex.TextLen,
|
"text_len": ex.TextLen,
|
||||||
"load_fired": col.isLoaded(),
|
"load_fired": col.isLoaded(),
|
||||||
},
|
},
|
||||||
|
SetCookies: gotCookies,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -217,7 +217,7 @@ func repeatCn(s string, n int) string { return strings.Repeat(s, n) }
|
||||||
// load 事件→evaluate→RawResult;blocked 判定路径。
|
// load 事件→evaluate→RawResult;blocked 判定路径。
|
||||||
func TestCdpFetchPageFullChain(t *testing.T) {
|
func TestCdpFetchPageFullChain(t *testing.T) {
|
||||||
f := newFakeCdp(t)
|
f := newFakeCdp(t)
|
||||||
raw, eb := cdpFetchPage(context.Background(), f.url, "https://example.test/page")
|
raw, eb := cdpFetchPage(context.Background(), f.url, "https://example.test/page", nil)
|
||||||
if eb != nil {
|
if eb != nil {
|
||||||
t.Fatalf("cdpFetchPage 不应失败: %+v", eb)
|
t.Fatalf("cdpFetchPage 不应失败: %+v", eb)
|
||||||
}
|
}
|
||||||
|
|
@ -259,7 +259,7 @@ func TestCdpFetchPageBlocked(t *testing.T) {
|
||||||
// 直接构造:blocked 特征经 htmlHead 判定(fake evaluate 返回正常值,
|
// 直接构造:blocked 特征经 htmlHead 判定(fake evaluate 返回正常值,
|
||||||
// 此处单测 detectVendor 已覆盖 blocked 分支——此处验证 blocked 端到端需
|
// 此处单测 detectVendor 已覆盖 blocked 分支——此处验证 blocked 端到端需
|
||||||
// 定制 evaluate 响应,走 detectVendor 单测覆盖 + 本端到端验证 upstream 路径)。
|
// 定制 evaluate 响应,走 detectVendor 单测覆盖 + 本端到端验证 upstream 路径)。
|
||||||
_, eb := cdpFetchPage(context.Background(), f.url, "https://block.test/x")
|
_, eb := cdpFetchPage(context.Background(), f.url, "https://block.test/x", nil)
|
||||||
if eb != nil && eb.Code == contract.CodeBlocked {
|
if eb != nil && eb.Code == contract.CodeBlocked {
|
||||||
t.Logf("端到端 blocked 命中: %s", eb.Message)
|
t.Logf("端到端 blocked 命中: %s", eb.Message)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -278,6 +278,43 @@ func TestRewriteWsURL(t *testing.T) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWsHandshakeHostLoopback(t *testing.T) {
|
||||||
|
if got := wsHandshakeHost("ws://127.0.0.1:9222/"); got != "127.0.0.1:9222" {
|
||||||
|
t.Fatalf("loopback Host 头: %s", got)
|
||||||
|
}
|
||||||
|
if got := wsHandshakeHost("ws://lightpanda:9222/"); got != "" {
|
||||||
|
t.Fatalf("非回环不应改 Host: %s", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSearxHTTPStatusHealthLatch(t *testing.T) {
|
||||||
|
n := 0
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path == "/healthz" {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
n++
|
||||||
|
if n == 1 {
|
||||||
|
w.WriteHeader(http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusBadGateway)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
a := NewSearx("searxng-cn", srv.URL, "zh-CN")
|
||||||
|
_ = a.Init(context.Background())
|
||||||
|
job := contract.JobEnvelope{Intent: "search", Search: &contract.SearchInput{Query: "q"}}
|
||||||
|
_, _ = a.Execute(context.Background(), job)
|
||||||
|
if h := a.Health(); !h.OK {
|
||||||
|
t.Fatalf("404 不应闩死适配器: %+v", h)
|
||||||
|
}
|
||||||
|
_, _ = a.Execute(context.Background(), job)
|
||||||
|
if h := a.Health(); h.OK {
|
||||||
|
t.Fatalf("502 应将实例标不健康: %+v", h)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestSearxCaps 区域/意图能力标签。
|
// TestSearxCaps 区域/意图能力标签。
|
||||||
func TestSearxCaps(t *testing.T) {
|
func TestSearxCaps(t *testing.T) {
|
||||||
cn := NewSearx("searxng-cn", "http://x", "zh-CN").Capabilities()
|
cn := NewSearx("searxng-cn", "http://x", "zh-CN").Capabilities()
|
||||||
|
|
|
||||||
|
|
@ -170,13 +170,15 @@ func (s *SearxAdapter) Execute(ctx context.Context, job contract.JobEnvelope) (*
|
||||||
req.Header.Set("Accept", "application/json")
|
req.Header.Set("Accept", "application/json")
|
||||||
resp, err := s.hc.Do(req)
|
resp, err := s.hc.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.markHealth(false, err.Error())
|
// 单次超时/瞬时网络失败不当整实例不健康(healthWatch 只认 /healthz)。
|
||||||
return nil, &contract.ErrBody{Code: errCodeOf(err), Message: "searxng 请求失败: " + err.Error()}
|
return nil, &contract.ErrBody{Code: errCodeOf(err), Message: "searxng 请求失败: " + err.Error()}
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
if resp.StatusCode != http.StatusOK {
|
if resp.StatusCode != http.StatusOK {
|
||||||
msg := fmt.Sprintf("searxng HTTP %d", resp.StatusCode)
|
msg := fmt.Sprintf("searxng HTTP %d", resp.StatusCode)
|
||||||
s.markHealth(resp.StatusCode < 500, msg)
|
if resp.StatusCode >= 500 {
|
||||||
|
s.markHealth(false, msg)
|
||||||
|
}
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: msg}
|
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: msg}
|
||||||
}
|
}
|
||||||
var sr searxResponse
|
var sr searxResponse
|
||||||
|
|
|
||||||
|
|
@ -92,6 +92,9 @@ func (t *TrafilaturaAdapter) Teardown(_ context.Context) error { return nil }
|
||||||
type trafRequest struct {
|
type trafRequest struct {
|
||||||
URL string `json:"url"`
|
URL string `json:"url"`
|
||||||
MaxChars int `json:"max_chars"`
|
MaxChars int `json:"max_chars"`
|
||||||
|
Impersonate string `json:"impersonate,omitempty"`
|
||||||
|
Headers map[string]string `json:"headers,omitempty"`
|
||||||
|
Cookies []contract.Cookie `json:"cookies,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// trafResponse /v1/read 响应体(app.py _extract 返回形状)。
|
// trafResponse /v1/read 响应体(app.py _extract 返回形状)。
|
||||||
|
|
@ -104,6 +107,9 @@ type trafResponse struct {
|
||||||
URL string `json:"url"`
|
URL string `json:"url"`
|
||||||
Error *string `json:"error"`
|
Error *string `json:"error"`
|
||||||
FailClass *string `json:"fail_class"`
|
FailClass *string `json:"fail_class"`
|
||||||
|
SetCookies []contract.Cookie `json:"set_cookies"`
|
||||||
|
StatusCode int `json:"status_code"`
|
||||||
|
Poisoned bool `json:"poisoned"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Execute 精读执行。
|
// Execute 精读执行。
|
||||||
|
|
@ -116,7 +122,13 @@ func (t *TrafilaturaAdapter) Execute(ctx context.Context, job contract.JobEnvelo
|
||||||
if maxChars <= 0 {
|
if maxChars <= 0 {
|
||||||
maxChars = 20000
|
maxChars = 20000
|
||||||
}
|
}
|
||||||
body, err := json.Marshal(trafRequest{URL: in.URL, MaxChars: maxChars})
|
trReq := trafRequest{URL: in.URL, MaxChars: maxChars}
|
||||||
|
if job.Session != nil {
|
||||||
|
trReq.Impersonate = job.Session.Impersonate
|
||||||
|
trReq.Headers = job.Session.Headers
|
||||||
|
trReq.Cookies = job.Session.Cookies
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(trReq)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: err.Error()}
|
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: err.Error()}
|
||||||
}
|
}
|
||||||
|
|
@ -133,7 +145,7 @@ func (t *TrafilaturaAdapter) Execute(ctx context.Context, job contract.JobEnvelo
|
||||||
req.Header.Set("Content-Type", "application/json")
|
req.Header.Set("Content-Type", "application/json")
|
||||||
resp, err := t.hc.Do(req)
|
resp, err := t.hc.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.markHealth(false, err.Error())
|
// 单次超时不当整实例不健康(healthWatch 只认 /health)。
|
||||||
return nil, &contract.ErrBody{Code: errCodeOf(err), Message: "trafilatura 请求失败: " + err.Error()}
|
return nil, &contract.ErrBody{Code: errCodeOf(err), Message: "trafilatura 请求失败: " + err.Error()}
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
@ -148,7 +160,6 @@ func (t *TrafilaturaAdapter) Execute(ctx context.Context, job contract.JobEnvelo
|
||||||
}
|
}
|
||||||
t.markHealth(true, "")
|
t.markHealth(true, "")
|
||||||
if !tr.OK {
|
if !tr.OK {
|
||||||
// empty_extract 类:返回空正文错误,上层触发降级链(plan-final §3)。
|
|
||||||
failClass := ""
|
failClass := ""
|
||||||
if tr.FailClass != nil {
|
if tr.FailClass != nil {
|
||||||
failClass = *tr.FailClass
|
failClass = *tr.FailClass
|
||||||
|
|
@ -157,7 +168,14 @@ func (t *TrafilaturaAdapter) Execute(ctx context.Context, job contract.JobEnvelo
|
||||||
if tr.Error != nil {
|
if tr.Error != nil {
|
||||||
errMsg = *tr.Error
|
errMsg = *tr.Error
|
||||||
}
|
}
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUpstream,
|
code := contract.CodeUpstream
|
||||||
|
if failClass == "blocked" || tr.Poisoned {
|
||||||
|
code = contract.CodeBlocked
|
||||||
|
return &contract.RawResult{SetCookies: tr.SetCookies, Poisoned: true, StatusCode: tr.StatusCode, Engine: "trafilatura"},
|
||||||
|
&contract.ErrBody{Code: code,
|
||||||
|
Message: fmt.Sprintf("trafilatura 拦截(fail_class=%s): %s", failClass, errMsg)}
|
||||||
|
}
|
||||||
|
return nil, &contract.ErrBody{Code: code,
|
||||||
Message: fmt.Sprintf("trafilatura 空正文(fail_class=%s): %s", failClass, errMsg)}
|
Message: fmt.Sprintf("trafilatura 空正文(fail_class=%s): %s", failClass, errMsg)}
|
||||||
}
|
}
|
||||||
md := tr.Markdown
|
md := tr.Markdown
|
||||||
|
|
@ -169,16 +187,22 @@ func (t *TrafilaturaAdapter) Execute(ctx context.Context, job contract.JobEnvelo
|
||||||
if tr.Truncated != nil {
|
if tr.Truncated != nil {
|
||||||
truncated = *tr.Truncated
|
truncated = *tr.Truncated
|
||||||
}
|
}
|
||||||
|
status := resp.StatusCode
|
||||||
|
if tr.StatusCode > 0 {
|
||||||
|
status = tr.StatusCode
|
||||||
|
}
|
||||||
return &contract.RawResult{
|
return &contract.RawResult{
|
||||||
Title: tr.Title,
|
Title: tr.Title,
|
||||||
Markdown: md,
|
Markdown: md,
|
||||||
FinalURL: firstNonEmpty(tr.URL, in.URL),
|
FinalURL: firstNonEmpty(tr.URL, in.URL),
|
||||||
StatusCode: resp.StatusCode,
|
StatusCode: status,
|
||||||
Engine: "trafilatura",
|
Engine: "trafilatura",
|
||||||
Extra: map[string]any{
|
Extra: map[string]any{
|
||||||
"char_count": charCount,
|
"char_count": charCount,
|
||||||
"truncated": truncated,
|
"truncated": truncated,
|
||||||
},
|
},
|
||||||
|
SetCookies: tr.SetCookies,
|
||||||
|
Poisoned: tr.Poisoned,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -42,7 +42,9 @@ type wsConn struct {
|
||||||
|
|
||||||
// wsDial 建立 WebSocket 连接:HTTP Upgrade 握手(RFC6455 client 端最小面)。
|
// wsDial 建立 WebSocket 连接:HTTP Upgrade 握手(RFC6455 client 端最小面)。
|
||||||
// scheme 仅支持 ws(CDP 调试端点均为明文 ws://,overlay 内网无 TLS 需求)。
|
// scheme 仅支持 ws(CDP 调试端点均为明文 ws://,overlay 内网无 TLS 需求)。
|
||||||
func wsDial(rawURL string, timeout time.Duration) (*wsConn, error) {
|
// handshakeHost 非空时作为 Host 头(Lightpanda 0.3.7 只接受 127.0.0.1,
|
||||||
|
// 对 Docker 服务名回 Host not allowed / 403;TCP 仍拨 rawURL 的 host)。
|
||||||
|
func wsDial(rawURL string, timeout time.Duration, handshakeHost string) (*wsConn, error) {
|
||||||
u, err := url.Parse(rawURL)
|
u, err := url.Parse(rawURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("dock: ws URL 解析: %w", err)
|
return nil, fmt.Errorf("dock: ws URL 解析: %w", err)
|
||||||
|
|
@ -69,9 +71,13 @@ func wsDial(rawURL string, timeout time.Duration) (*wsConn, error) {
|
||||||
return nil, fmt.Errorf("dock: ws key 随机源: %w", err)
|
return nil, fmt.Errorf("dock: ws key 随机源: %w", err)
|
||||||
}
|
}
|
||||||
key := base64.StdEncoding.EncodeToString(keyBytes)
|
key := base64.StdEncoding.EncodeToString(keyBytes)
|
||||||
|
hdrHost := u.Host
|
||||||
|
if handshakeHost != "" {
|
||||||
|
hdrHost = handshakeHost
|
||||||
|
}
|
||||||
reqLines := []string{
|
reqLines := []string{
|
||||||
"GET " + u.RequestURI() + " HTTP/1.1",
|
"GET " + u.RequestURI() + " HTTP/1.1",
|
||||||
"Host: " + u.Host,
|
"Host: " + hdrHost,
|
||||||
"Upgrade: websocket",
|
"Upgrade: websocket",
|
||||||
"Connection: Upgrade",
|
"Connection: Upgrade",
|
||||||
"Sec-WebSocket-Key: " + key,
|
"Sec-WebSocket-Key: " + key,
|
||||||
|
|
|
||||||
145
server/internal/fingerprint/profile.go
Normal file
145
server/internal/fingerprint/profile.go
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
// Package fingerprint 提供每节点稳定的浏览器身份模版(P0)。
|
||||||
|
//
|
||||||
|
// 自洽:impersonate / User-Agent / sec-ch-ua / platform 钉在同一 Chrome 大版本。
|
||||||
|
// 上限跟 curl_cffi 能仿的桌面档(chrome136),禁止再写 Chrome/151 + Go TLS。
|
||||||
|
package fingerprint
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/binary"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 钉死与 curl_cffi 0.13 桌面上限对齐(DaaS fp-v2 同代)。
|
||||||
|
const (
|
||||||
|
ChromeMajor = 136
|
||||||
|
DefaultImpersonate = "chrome136"
|
||||||
|
TemplateCount = 8 // 每节点落库套数,便于切换;内存只持有 active
|
||||||
|
)
|
||||||
|
|
||||||
|
// Profile 一张自洽脸。字段均可落库。
|
||||||
|
type Profile struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
OS string `json:"os"` // Windows | macOS
|
||||||
|
ChromeMajor int `json:"chrome_major"`
|
||||||
|
Impersonate string `json:"impersonate"`
|
||||||
|
UserAgent string `json:"user_agent"`
|
||||||
|
SecCHUA string `json:"sec_ch_ua"`
|
||||||
|
SecCHUAMobile string `json:"sec_ch_ua_mobile"`
|
||||||
|
SecCHUAPlatform string `json:"sec_ch_ua_platform"`
|
||||||
|
AcceptLanguage string `json:"accept_language"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
viewLangs = []string{
|
||||||
|
"en-US,en;q=0.9",
|
||||||
|
"en-US,en;q=0.8",
|
||||||
|
"en-US,en;q=0.9,zh-CN;q=0.6",
|
||||||
|
"zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
"en-GB,en;q=0.9,en-US;q=0.8",
|
||||||
|
}
|
||||||
|
winNT = []string{"10.0", "10.0", "11.0"}
|
||||||
|
macOSX = []string{"10_15_7", "13_6_1", "14_4_0", "14_2_1"}
|
||||||
|
)
|
||||||
|
|
||||||
|
// NodeID 生产节点稳定种子(Swarm HOSTNAME;缺省本机 hostname)。
|
||||||
|
func NodeID() string {
|
||||||
|
if v := os.Getenv("HOSTNAME"); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
h, err := os.Hostname()
|
||||||
|
if err != nil || h == "" {
|
||||||
|
return "unknown-node"
|
||||||
|
}
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
// Allocate 由 seed 确定性生成一张脸(同 seed 永同脸)。
|
||||||
|
func Allocate(seed string) Profile {
|
||||||
|
s := seedInt(seed)
|
||||||
|
osFamily := "Windows"
|
||||||
|
if s%2 == 1 {
|
||||||
|
osFamily = "macOS"
|
||||||
|
}
|
||||||
|
major := ChromeMajor
|
||||||
|
impersonate := DefaultImpersonate
|
||||||
|
lang := viewLangs[(s>>8)%uint64(len(viewLangs))]
|
||||||
|
buildPatch := (s >> 16) % 10000
|
||||||
|
buildMinor := (s >> 4) % 10000
|
||||||
|
var ua, platform, tid string
|
||||||
|
if osFamily == "Windows" {
|
||||||
|
nt := winNT[(s>>12)%uint64(len(winNT))]
|
||||||
|
ua = fmt.Sprintf(
|
||||||
|
"Mozilla/5.0 (Windows NT %s; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/%d.0.%d.%d Safari/537.36",
|
||||||
|
nt, major, buildPatch, buildMinor)
|
||||||
|
platform = `"Windows"`
|
||||||
|
tid = fmt.Sprintf("win-chrome%d-%06d", major, s%1_000_000)
|
||||||
|
} else {
|
||||||
|
osx := macOSX[(s>>12)%uint64(len(macOSX))]
|
||||||
|
ua = fmt.Sprintf(
|
||||||
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X %s) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/%d.0.%d.%d Safari/537.36",
|
||||||
|
osx, major, buildPatch, buildMinor)
|
||||||
|
platform = `"macOS"`
|
||||||
|
tid = fmt.Sprintf("mac-chrome%d-%06d", major, s%1_000_000)
|
||||||
|
}
|
||||||
|
return Profile{
|
||||||
|
ID: tid,
|
||||||
|
OS: osFamily,
|
||||||
|
ChromeMajor: major,
|
||||||
|
Impersonate: impersonate,
|
||||||
|
UserAgent: ua,
|
||||||
|
SecCHUA: secCHUA(major),
|
||||||
|
SecCHUAMobile: "?0",
|
||||||
|
SecCHUAPlatform: platform,
|
||||||
|
AcceptLanguage: lang,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// AllocateSet 为节点生成 n 套互异模版。
|
||||||
|
func AllocateSet(nodeID string, n int) []Profile {
|
||||||
|
if n <= 0 {
|
||||||
|
n = TemplateCount
|
||||||
|
}
|
||||||
|
out := make([]Profile, n)
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
out[i] = Allocate(nodeID + "|" + strconv.Itoa(i))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// PickActive 从集合里按节点 hash 选主模版(重启不变)。
|
||||||
|
func PickActive(nodeID string, set []Profile) Profile {
|
||||||
|
if len(set) == 0 {
|
||||||
|
return Allocate(nodeID)
|
||||||
|
}
|
||||||
|
return set[int(seedInt(nodeID+"|active")%uint64(len(set)))]
|
||||||
|
}
|
||||||
|
|
||||||
|
// Headers 与 impersonate 对齐的导航头(顺序接近桌面 Chrome)。
|
||||||
|
func Headers(p Profile) map[string]string {
|
||||||
|
return map[string]string{
|
||||||
|
"User-Agent": p.UserAgent,
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
|
||||||
|
"Accept-Language": p.AcceptLanguage,
|
||||||
|
"Accept-Encoding": "gzip, deflate, br",
|
||||||
|
"Upgrade-Insecure-Requests": "1",
|
||||||
|
"Sec-Fetch-Dest": "document",
|
||||||
|
"Sec-Fetch-Mode": "navigate",
|
||||||
|
"Sec-Fetch-Site": "none",
|
||||||
|
"Sec-Fetch-User": "?1",
|
||||||
|
"sec-ch-ua": p.SecCHUA,
|
||||||
|
"sec-ch-ua-mobile": p.SecCHUAMobile,
|
||||||
|
"sec-ch-ua-platform": p.SecCHUAPlatform,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func secCHUA(major int) string {
|
||||||
|
return fmt.Sprintf(`"Google Chrome";v="%d", "Chromium";v="%d", "Not_A Brand";v="24"`, major, major)
|
||||||
|
}
|
||||||
|
|
||||||
|
func seedInt(s string) uint64 {
|
||||||
|
sum := sha256.Sum256([]byte("fp-v2|" + s))
|
||||||
|
return binary.BigEndian.Uint64(sum[:8])
|
||||||
|
}
|
||||||
73
server/internal/fingerprint/profile_test.go
Normal file
73
server/internal/fingerprint/profile_test.go
Normal file
|
|
@ -0,0 +1,73 @@
|
||||||
|
package fingerprint
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAllocateSelfConsistent(t *testing.T) {
|
||||||
|
p := Allocate("node-a|0")
|
||||||
|
if p.ChromeMajor != ChromeMajor || p.Impersonate != DefaultImpersonate {
|
||||||
|
t.Fatalf("版本漂移: %+v", p)
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.UserAgent, "Chrome/136") {
|
||||||
|
t.Fatalf("UA 必须是 Chrome/136: %s", p.UserAgent)
|
||||||
|
}
|
||||||
|
if strings.Contains(p.UserAgent, "Chrome/151") {
|
||||||
|
t.Fatal("禁止 Chrome/151")
|
||||||
|
}
|
||||||
|
if !strings.Contains(p.SecCHUA, `"136"`) {
|
||||||
|
t.Fatalf("CH 必须对齐 136: %s", p.SecCHUA)
|
||||||
|
}
|
||||||
|
if p.OS == "Windows" && !strings.Contains(p.SecCHUAPlatform, "Windows") {
|
||||||
|
t.Fatalf("platform 与 OS 不一致: %+v", p)
|
||||||
|
}
|
||||||
|
if p.OS == "macOS" && !strings.Contains(p.SecCHUAPlatform, "macOS") {
|
||||||
|
t.Fatalf("platform 与 OS 不一致: %+v", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAllocateStable(t *testing.T) {
|
||||||
|
a := Allocate("same-seed")
|
||||||
|
b := Allocate("same-seed")
|
||||||
|
if a != b {
|
||||||
|
t.Fatalf("同 seed 应同脸: %+v vs %+v", a, b)
|
||||||
|
}
|
||||||
|
c := Allocate("other-seed")
|
||||||
|
if c.ID == a.ID && c.UserAgent == a.UserAgent {
|
||||||
|
t.Fatal("不同 seed 不应撞脸")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAllocateSetUnique(t *testing.T) {
|
||||||
|
set := AllocateSet("mgr1", TemplateCount)
|
||||||
|
if len(set) != TemplateCount {
|
||||||
|
t.Fatalf("套数=%d", len(set))
|
||||||
|
}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, p := range set {
|
||||||
|
if seen[p.ID] {
|
||||||
|
t.Fatalf("模版 id 重复: %s", p.ID)
|
||||||
|
}
|
||||||
|
seen[p.ID] = true
|
||||||
|
}
|
||||||
|
act := PickActive("mgr1", set)
|
||||||
|
if act.ID == "" {
|
||||||
|
t.Fatal("active 空")
|
||||||
|
}
|
||||||
|
act2 := PickActive("mgr1", set)
|
||||||
|
if act.ID != act2.ID {
|
||||||
|
t.Fatal("active 重启应稳定")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHeadersAligned(t *testing.T) {
|
||||||
|
p := Allocate("h")
|
||||||
|
h := Headers(p)
|
||||||
|
if h["User-Agent"] != p.UserAgent {
|
||||||
|
t.Fatal("UA 头不一致")
|
||||||
|
}
|
||||||
|
if h["sec-ch-ua"] != p.SecCHUA {
|
||||||
|
t.Fatal("CH 头不一致")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,8 +1,8 @@
|
||||||
// Package httpx 共享 HTTP 客户端:超时/体积/类型守卫 + 重定向每跳策略重验。
|
// Package httpx 共享 HTTP 客户端:超时/体积/类型守卫 + 重定向每跳策略重验。
|
||||||
//
|
//
|
||||||
// 复用声明:UA 与反爬判定参数取自 bench/site-matrix/cdp_fetch.mjs(Chrome 151 UA、
|
// 复用声明:体积/类型守卫移植自 bench/trafilatura-http/app.py
|
||||||
// detectVendor 正则);体积/类型守卫移植自 bench/trafilatura-http/app.py
|
|
||||||
// (MAX_DOWNLOAD_BYTES=5MB、Content-Type 白名单思路)。
|
// (MAX_DOWNLOAD_BYTES=5MB、Content-Type 白名单思路)。
|
||||||
|
// 本包只做策略守卫下载,不冒充浏览器(P0:TLS 与 UA 必须一致)。
|
||||||
package httpx
|
package httpx
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
|
@ -15,8 +15,9 @@ import (
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// UA 桌面 Chrome 151(与 bench/site-matrix/cdp_fetch.mjs 一致)。
|
// UA 本客户端诚实身份。页面抓取走 Trafilatura/curl_cffi 或 CDP,不经本包。
|
||||||
const UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.109 Safari/537.36"
|
// 禁止再写 Chrome/* —— Go crypto/tls 与浏览器 JA3 不一致。
|
||||||
|
const UA = "onesvm-browser-server-httpx/1.0"
|
||||||
|
|
||||||
// 超时与体积默认(design §5.4 体积/类型行;bench app.py 同值)。
|
// 超时与体积默认(design §5.4 体积/类型行;bench app.py 同值)。
|
||||||
const (
|
const (
|
||||||
|
|
|
||||||
|
|
@ -129,8 +129,11 @@ func TestErrorCodeMapping(t *testing.T) {
|
||||||
|
|
||||||
// TestUserAgent UA 与 bench 一致。
|
// TestUserAgent UA 与 bench 一致。
|
||||||
func TestUserAgent(t *testing.T) {
|
func TestUserAgent(t *testing.T) {
|
||||||
if !strings.Contains(UA, "Chrome/151") {
|
if !strings.HasPrefix(UA, "onesvm-browser-server-httpx/") {
|
||||||
t.Fatalf("UA 应为 Chrome 151: %s", UA)
|
t.Fatalf("httpx 必须用诚实 UA,禁止 Chrome 冒充: %s", UA)
|
||||||
|
}
|
||||||
|
if strings.Contains(UA, "Chrome/") {
|
||||||
|
t.Fatal("Go TLS 客户端不得声称 Chrome")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -16,6 +16,7 @@ import (
|
||||||
"onesvm.com/onesvm/browser-server/internal/config"
|
"onesvm.com/onesvm/browser-server/internal/config"
|
||||||
"onesvm.com/onesvm/browser-server/internal/contract"
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
"onesvm.com/onesvm/browser-server/internal/dock"
|
"onesvm.com/onesvm/browser-server/internal/dock"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/session"
|
||||||
"onesvm.com/onesvm/browser-server/internal/store"
|
"onesvm.com/onesvm/browser-server/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -91,6 +92,8 @@ type Core struct {
|
||||||
// proxy 可达性(overseas fail-closed:不可达 → 排队等待不降级直连,O3)。
|
// proxy 可达性(overseas fail-closed:不可达 → 排队等待不降级直连,O3)。
|
||||||
proxyOK atomic.Bool
|
proxyOK atomic.Bool
|
||||||
|
|
||||||
|
jar *session.Jar // 可选;nil 时不养罐(单测默认)
|
||||||
|
|
||||||
st *Stats
|
st *Stats
|
||||||
|
|
||||||
stopCh chan struct{}
|
stopCh chan struct{}
|
||||||
|
|
@ -124,6 +127,9 @@ func NewCore(db ResultStore, reg *dock.Registry, tmpl *Template, proxyURL string
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetJar 注入每节点 Cookie 罐(生产 main 调用;单测可不设)。
|
||||||
|
func (c *Core) SetJar(j *session.Jar) { c.jar = j }
|
||||||
|
|
||||||
// AdmitMax 背压上限。
|
// AdmitMax 背压上限。
|
||||||
func (c *Core) AdmitMax() int { return c.admitMax }
|
func (c *Core) AdmitMax() int { return c.admitMax }
|
||||||
|
|
||||||
|
|
@ -139,6 +145,10 @@ func (c *Core) Start(ctx context.Context) {
|
||||||
go c.proxyWatchLoop(ctx)
|
go c.proxyWatchLoop(ctx)
|
||||||
c.wg.Add(1)
|
c.wg.Add(1)
|
||||||
go c.healthWatchLoop(ctx) // ITER-1 F2:适配器健康周期重探(摘除→恢复闭环)
|
go c.healthWatchLoop(ctx) // ITER-1 F2:适配器健康周期重探(摘除→恢复闭环)
|
||||||
|
if c.jar != nil {
|
||||||
|
c.wg.Add(1)
|
||||||
|
go c.jarReaperLoop(ctx)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop 优雅停机(等待在途任务完成;reaper/worker 退出)。
|
// Stop 优雅停机(等待在途任务完成;reaper/worker 退出)。
|
||||||
|
|
|
||||||
|
|
@ -21,6 +21,7 @@ import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"onesvm.com/onesvm/browser-server/internal/contract"
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/session"
|
||||||
)
|
)
|
||||||
|
|
||||||
// 路由目标常量。
|
// 路由目标常量。
|
||||||
|
|
@ -134,6 +135,7 @@ func (c *Core) execReadChain(ctx context.Context, job contract.JobEnvelope, t0 t
|
||||||
chain = []string{adapterShell}
|
chain = []string{adapterShell}
|
||||||
}
|
}
|
||||||
proxy := "direct"
|
proxy := "direct"
|
||||||
|
egress := "direct"
|
||||||
if job.Read != nil && job.Read.Region == contract.RegionOverseas {
|
if job.Read != nil && job.Read.Region == contract.RegionOverseas {
|
||||||
if !c.proxyOK.Load() {
|
if !c.proxyOK.Load() {
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUnavailable, RetryAfterS: retryPtr(10),
|
return nil, &contract.ErrBody{Code: contract.CodeUnavailable, RetryAfterS: retryPtr(10),
|
||||||
|
|
@ -145,6 +147,7 @@ func (c *Core) execReadChain(ctx context.Context, job contract.JobEnvelope, t0 t
|
||||||
Message: "出口查询失败(fail-closed): " + err.Error()}, routeResult{tookMs: ms(t0)}
|
Message: "出口查询失败(fail-closed): " + err.Error()}, routeResult{tookMs: ms(t0)}
|
||||||
}
|
}
|
||||||
proxy = pe.Proxy
|
proxy = pe.Proxy
|
||||||
|
egress = session.EgressKey(job.Read.Region, pe.Node, pe.Proxy)
|
||||||
}
|
}
|
||||||
var warnings []string
|
var warnings []string
|
||||||
var lastErr *contract.ErrBody
|
var lastErr *contract.ErrBody
|
||||||
|
|
@ -154,8 +157,13 @@ func (c *Core) execReadChain(ctx context.Context, job contract.JobEnvelope, t0 t
|
||||||
warnings = append(warnings, "adapter_unhealthy:"+name)
|
warnings = append(warnings, "adapter_unhealthy:"+name)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
c.attachSession(ctx, &job, name, egress)
|
||||||
raw, eb := a.Execute(ctx, job)
|
raw, eb := a.Execute(ctx, job)
|
||||||
|
c.commitSession(ctx, job, raw, eb)
|
||||||
if eb == nil {
|
if eb == nil {
|
||||||
|
if job.Session != nil {
|
||||||
|
warnings = append(warnings, "fp:"+job.Session.TemplateID)
|
||||||
|
}
|
||||||
return raw, nil, routeResult{adapter: name, proxy: proxyExitLabel(job.Read.Region, proxy),
|
return raw, nil, routeResult{adapter: name, proxy: proxyExitLabel(job.Read.Region, proxy),
|
||||||
warnings: warnings, tookMs: ms(t0)}
|
warnings: warnings, tookMs: ms(t0)}
|
||||||
}
|
}
|
||||||
|
|
@ -178,7 +186,13 @@ func (c *Core) execOne(ctx context.Context, job contract.JobEnvelope, name, prox
|
||||||
if !ok || !a.Health().OK {
|
if !ok || !a.Health().OK {
|
||||||
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: "适配器不可用: " + name}, routeResult{tookMs: ms(t0)}
|
return nil, &contract.ErrBody{Code: contract.CodeUpstream, Message: "适配器不可用: " + name}, routeResult{tookMs: ms(t0)}
|
||||||
}
|
}
|
||||||
|
egress := "direct"
|
||||||
|
if job.Read != nil {
|
||||||
|
egress = session.EgressKey(job.Read.Region, "", proxy)
|
||||||
|
}
|
||||||
|
c.attachSession(ctx, &job, name, egress)
|
||||||
raw, eb := a.Execute(ctx, job)
|
raw, eb := a.Execute(ctx, job)
|
||||||
|
c.commitSession(ctx, job, raw, eb)
|
||||||
return raw, eb, routeResult{adapter: name, proxy: proxy, tookMs: ms(t0)}
|
return raw, eb, routeResult{adapter: name, proxy: proxy, tookMs: ms(t0)}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -193,12 +207,17 @@ func proxyExitLabel(region, proxy string) string {
|
||||||
return proxy
|
return proxy
|
||||||
}
|
}
|
||||||
|
|
||||||
// jobSearchURL 搜索任务无目标 URL(域名路由键给空——proxymanager 按默认池)。
|
// overseasSearchExitURL 海外 search 无目标页:出口按 D2 Bing-only 钉搜索域
|
||||||
|
// (空串会让 ProxyClient.hostOf 失败,现网 overseas search 整条 fail-closed)。
|
||||||
|
const overseasSearchExitURL = "https://www.bing.com/"
|
||||||
|
|
||||||
|
// jobSearchURL 海外 search 的出口路由键。search 任务没有 Read.URL,必须给
|
||||||
|
// 可解析 host;read 任务仍用目标 URL(本函数只给 search 路径调用)。
|
||||||
func jobSearchURL(job contract.JobEnvelope) string {
|
func jobSearchURL(job contract.JobEnvelope) string {
|
||||||
if job.Read != nil {
|
if job.Read != nil && job.Read.URL != "" {
|
||||||
return job.Read.URL
|
return job.Read.URL
|
||||||
}
|
}
|
||||||
return ""
|
return overseasSearchExitURL
|
||||||
}
|
}
|
||||||
|
|
||||||
// ms 起始到现在的毫秒。
|
// ms 起始到现在的毫秒。
|
||||||
|
|
|
||||||
|
|
@ -469,3 +469,43 @@ func TestConcurrentClaim(t *testing.T) {
|
||||||
t.Fatalf("应抢 30,得 %d", len(seen))
|
t.Fatalf("应抢 30,得 %d", len(seen))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestJobSearchURL(t *testing.T) {
|
||||||
|
if got := jobSearchURL(contract.JobEnvelope{Intent: "search"}); got != overseasSearchExitURL {
|
||||||
|
t.Fatalf("search 出口键应钉 Bing,得 %q", got)
|
||||||
|
}
|
||||||
|
if got := jobSearchURL(contract.JobEnvelope{Intent: "read",
|
||||||
|
Read: &contract.ReadInput{URL: "https://example.com/a"}}); got != "https://example.com/a" {
|
||||||
|
t.Fatalf("read URL 应透传,得 %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTemplateFiltersAmazonHome(t *testing.T) {
|
||||||
|
tmpl, err := NewTemplate()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
raw := &contract.RawResult{Engine: "searxng-global", Extra: map[string]any{
|
||||||
|
"searx_results": json.RawMessage(`[
|
||||||
|
{"title":"Amazon.com","url":"https://www.amazon.com/","content":"由于此网站的设置,我们无法提供该页面的具体描述","engine":"bing"},
|
||||||
|
{"title":"Prime Video","url":"https://www.primevideo.com/collection/x","content":"stream","engine":"bing"},
|
||||||
|
{"title":"行业稿","url":"https://www.cifnews.com/article/168904","content":"厨房收纳占比第一","engine":"360search"},
|
||||||
|
{"title":"ASIN","url":"https://www.amazon.com/dp/B09DT48V16","content":"earbuds listing","engine":"bing"}]`),
|
||||||
|
"query": "kitchen organizers",
|
||||||
|
}}
|
||||||
|
env := tmpl.Build(TemplateInput{
|
||||||
|
Job: contract.JobEnvelope{Intent: "search", RequestID: "req-filter",
|
||||||
|
Search: &contract.SearchInput{Query: "kitchen organizers", MaxResults: 5}},
|
||||||
|
Adapter: "searxng-global", Raw: raw, TookMs: 10,
|
||||||
|
})
|
||||||
|
if !env.OK {
|
||||||
|
t.Fatalf("应成功: %+v", env.Error)
|
||||||
|
}
|
||||||
|
sp := env.Data.(*contract.SearchPayload)
|
||||||
|
if len(sp.Results) != 2 {
|
||||||
|
t.Fatalf("应留下行业稿+ASIN,得 %d: %+v", len(sp.Results), sp.Results)
|
||||||
|
}
|
||||||
|
if sp.Results[0].URL != "https://www.cifnews.com/article/168904" {
|
||||||
|
t.Errorf("首条应是过滤后第一条可引用结果: %s", sp.Results[0].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,8 @@ package scheduler
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"onesvm.com/onesvm/browser-server/internal/contract"
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
|
@ -44,6 +46,64 @@ func parseSearxResults(raw *contract.RawResult) ([]searxItem, []string) {
|
||||||
return items, unres
|
return items, unres
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isJunkSearchHit 丢掉搜索引擎吸到的站点壳页(Amazon 首页/账号/影音、
|
||||||
|
// 无 snippet 的门户根路径)。先过滤再裁 max_results,避免 16 条里 10 条不可引用。
|
||||||
|
func isJunkSearchHit(rawURL, title, content string) bool {
|
||||||
|
u, err := url.Parse(rawURL)
|
||||||
|
if err != nil || u.Hostname() == "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
host := strings.ToLower(u.Hostname())
|
||||||
|
path := strings.ToLower(strings.TrimSuffix(u.Path, "/"))
|
||||||
|
if junkSearchHosts[host] {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if isAmazonHost(host) && isAmazonShellPath(path) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
snippet := strings.TrimSpace(content)
|
||||||
|
if snippet == "" && isBareSiteRoot(path) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if strings.Contains(snippet, "由于此网站的设置,我们无法提供该页面的具体描述") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(title) == "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// junkSearchHosts 明确的导航/非内容域(搜索发现层无引用价值)。
|
||||||
|
var junkSearchHosts = map[string]bool{
|
||||||
|
"music.amazon.com": true,
|
||||||
|
"www.primevideo.com": true,
|
||||||
|
"primevideo.com": true,
|
||||||
|
"pharmacy.amazon.com": true,
|
||||||
|
"hiring.amazon.com": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
func isAmazonHost(host string) bool {
|
||||||
|
return host == "www.amazon.com" || host == "amazon.com" ||
|
||||||
|
host == "www.amazon.cn" || host == "amazon.cn" ||
|
||||||
|
host == "gs.amazon.cn" || host == "globalstore.amazon.cn"
|
||||||
|
}
|
||||||
|
|
||||||
|
func isAmazonShellPath(path string) bool {
|
||||||
|
if path == "" || path == "/-/zh" || path == "/ref=nav_logo" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(path, "/your-account") || strings.HasPrefix(path, "/gp/css") ||
|
||||||
|
strings.HasPrefix(path, "/gp/yourstore") || strings.HasPrefix(path, "/ap/signin") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func isBareSiteRoot(path string) bool {
|
||||||
|
return path == "" || path == "/"
|
||||||
|
}
|
||||||
|
|
||||||
// publishedAtOf 引擎时间串 → 东八区 Time(多格式容错;失败给 nil 由调用方置空)。
|
// publishedAtOf 引擎时间串 → 东八区 Time(多格式容错;失败给 nil 由调用方置空)。
|
||||||
func publishedAtOf(s string) (contract.Time, bool) {
|
func publishedAtOf(s string) (contract.Time, bool) {
|
||||||
layouts := []string{time.RFC3339, "2006-01-02 15:04:05", "2006-01-02"}
|
layouts := []string{time.RFC3339, "2006-01-02 15:04:05", "2006-01-02"}
|
||||||
|
|
|
||||||
46
server/internal/scheduler/session_bind.go
Normal file
46
server/internal/scheduler/session_bind.go
Normal file
|
|
@ -0,0 +1,46 @@
|
||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (c *Core) attachSession(ctx context.Context, job *contract.JobEnvelope, adapter, egress string) {
|
||||||
|
if c.jar == nil || job == nil || job.Read == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
job.Session = c.jar.Attach(ctx, adapter, egress, job.Read.URL)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Core) commitSession(ctx context.Context, job contract.JobEnvelope, raw *contract.RawResult, eb *contract.ErrBody) {
|
||||||
|
if c.jar == nil || job.Session == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c.jar.Commit(ctx, job.Session, raw, eb)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Core) jarReaperLoop(ctx context.Context) {
|
||||||
|
defer c.wg.Done()
|
||||||
|
t := time.NewTicker(session.ReapEvery)
|
||||||
|
defer t.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-c.stopCh:
|
||||||
|
return
|
||||||
|
case <-t.C:
|
||||||
|
n, err := c.jar.Reap(ctx)
|
||||||
|
if err != nil {
|
||||||
|
c.log.Printf("session reap 错误: %v", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if n > 0 {
|
||||||
|
c.log.Printf("session reap 删除 %d 行", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -127,10 +127,17 @@ func (t *Template) buildSearch(in TemplateInput, warnings []string) (*contract.S
|
||||||
if max > 20 {
|
if max > 20 {
|
||||||
max = 20 // mcp-usage §2.1:≤20
|
max = 20 // mcp-usage §2.1:≤20
|
||||||
}
|
}
|
||||||
for i, it := range items {
|
kept := 0
|
||||||
if i >= max {
|
dropped := 0
|
||||||
|
for _, it := range items {
|
||||||
|
if isJunkSearchHit(it.URL, it.Title, it.Content) {
|
||||||
|
dropped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if kept >= max {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
i := kept
|
||||||
res := contract.SearchResult{
|
res := contract.SearchResult{
|
||||||
ID: fmt.Sprintf("r%d", i+1),
|
ID: fmt.Sprintf("r%d", i+1),
|
||||||
Title: it.Title,
|
Title: it.Title,
|
||||||
|
|
@ -145,6 +152,10 @@ func (t *Template) buildSearch(in TemplateInput, warnings []string) (*contract.S
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
payload.Results = append(payload.Results, res)
|
payload.Results = append(payload.Results, res)
|
||||||
|
kept++
|
||||||
|
}
|
||||||
|
if dropped > 0 {
|
||||||
|
warnings = append(warnings, fmt.Sprintf("filtered_nav_hits:%d", dropped))
|
||||||
}
|
}
|
||||||
// ② safetyscan:title/content 逐条扫(search 无 markdown 字段)。
|
// ② safetyscan:title/content 逐条扫(search 无 markdown 字段)。
|
||||||
for i := range payload.Results {
|
for i := range payload.Results {
|
||||||
|
|
|
||||||
148
server/internal/session/filter.go
Normal file
148
server/internal/session/filter.go
Normal file
|
|
@ -0,0 +1,148 @@
|
||||||
|
package session
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/config"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 硬顶(用户 B 案:多套落库、内存不养大罐)。
|
||||||
|
const (
|
||||||
|
MaxTemplates = 8
|
||||||
|
MaxETLDPerKey = 64
|
||||||
|
MaxCookiesPerETLD = 20
|
||||||
|
MaxCookieBytes = 4 * 1024
|
||||||
|
MaxJarBytes = 2 * 1024 * 1024
|
||||||
|
DefaultTTL = 2 * time.Hour
|
||||||
|
HardTTL = 6 * time.Hour
|
||||||
|
ReapEvery = 60 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// 登录态 / 跨站身份 Cookie:公开页罐不收。
|
||||||
|
var deniedNames = map[string]bool{
|
||||||
|
"at-main": true, "sess-at-main": true, "session-token": true,
|
||||||
|
"x-main": true, "sid": true, "ssid": true, "hsid": true,
|
||||||
|
"apisid": true, "sapisid": true, "lsid": true,
|
||||||
|
"__secure-1psid": true, "__secure-3psid": true,
|
||||||
|
"__secure-1psidts": true, "__secure-3psidts": true,
|
||||||
|
"login": true, "auth": true, "authorization": true,
|
||||||
|
"password": true, "passwd": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
var multiTLD = map[string]bool{
|
||||||
|
"co.uk": true, "com.cn": true, "com.au": true, "co.jp": true,
|
||||||
|
"com.hk": true, "co.kr": true, "com.tw": true, "com.sg": true,
|
||||||
|
"co.in": true, "com.br": true, "co.nz": true, "org.uk": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// ETLD1 粗粒度可注册域(不引入 publicsuffix 依赖)。
|
||||||
|
func ETLD1(rawURL string) string {
|
||||||
|
u, err := url.Parse(rawURL)
|
||||||
|
host := ""
|
||||||
|
if err == nil {
|
||||||
|
host = u.Hostname()
|
||||||
|
}
|
||||||
|
if host == "" {
|
||||||
|
host = rawURL
|
||||||
|
}
|
||||||
|
host = strings.ToLower(strings.TrimSpace(host))
|
||||||
|
if h, _, err := net.SplitHostPort(host); err == nil {
|
||||||
|
host = h
|
||||||
|
}
|
||||||
|
if host == "" || net.ParseIP(host) != nil {
|
||||||
|
return host
|
||||||
|
}
|
||||||
|
parts := strings.Split(host, ".")
|
||||||
|
if len(parts) <= 2 {
|
||||||
|
return host
|
||||||
|
}
|
||||||
|
last2 := parts[len(parts)-2] + "." + parts[len(parts)-1]
|
||||||
|
if multiTLD[last2] && len(parts) >= 3 {
|
||||||
|
return parts[len(parts)-3] + "." + last2
|
||||||
|
}
|
||||||
|
return last2
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sanitize 过滤登录态 / 超长 / 无名 Cookie,并夹 TTL。
|
||||||
|
func Sanitize(in []contract.Cookie) []contract.Cookie {
|
||||||
|
if len(in) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
now := config.Now()
|
||||||
|
defExp := now.Add(DefaultTTL).Unix()
|
||||||
|
hardExp := now.Add(HardTTL).Unix()
|
||||||
|
out := make([]contract.Cookie, 0, len(in))
|
||||||
|
for _, c := range in {
|
||||||
|
name := strings.TrimSpace(c.Name)
|
||||||
|
if name == "" || c.Value == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if len(c.Value) > MaxCookieBytes {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
low := strings.ToLower(name)
|
||||||
|
if deniedNames[low] || strings.Contains(low, "psid") || strings.Contains(low, "token") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
path := c.Path
|
||||||
|
if path == "" {
|
||||||
|
path = "/"
|
||||||
|
}
|
||||||
|
exp := c.Expires
|
||||||
|
if exp <= 0 {
|
||||||
|
exp = defExp
|
||||||
|
}
|
||||||
|
if exp > hardExp {
|
||||||
|
exp = hardExp
|
||||||
|
}
|
||||||
|
if exp <= now.Unix() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, contract.Cookie{
|
||||||
|
Name: name, Value: c.Value, Domain: c.Domain, Path: path, Expires: exp,
|
||||||
|
})
|
||||||
|
if len(out) >= MaxCookiesPerETLD {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsPoison 目标站投毒/挑战:整域丢罐。超时/空抽取不算。
|
||||||
|
func IsPoison(eb *contract.ErrBody, raw *contract.RawResult) bool {
|
||||||
|
if raw != nil && raw.Poisoned {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if raw != nil && raw.StatusCode >= 400 && raw.StatusCode != 404 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if eb == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if eb.Code == contract.CodeBlocked {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
msg := strings.ToLower(eb.Message)
|
||||||
|
for _, k := range []string{"cloudflare", "captcha", "waf", "http 403", "http 429", "just a moment"} {
|
||||||
|
if strings.Contains(msg, k) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// AdapterClassOf 适配器名 → 罐分轨。
|
||||||
|
func AdapterClassOf(adapter string) string {
|
||||||
|
switch adapter {
|
||||||
|
case "trafilatura":
|
||||||
|
return contract.AdapterHTTP
|
||||||
|
case "lightpanda", "headless-shell":
|
||||||
|
return contract.AdapterCDP
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
129
server/internal/session/jar.go
Normal file
129
server/internal/session/jar.go
Normal file
|
|
@ -0,0 +1,129 @@
|
||||||
|
// Package session 每节点共享 Cookie 罐(P1,落 SQLite)。
|
||||||
|
//
|
||||||
|
// 事件:
|
||||||
|
//
|
||||||
|
// Bind 调度器 Execute 前:读 active 模版 + 罐;失败则空罐继续(不挡抓取)
|
||||||
|
// Save 200 且非投毒:Sanitize 后覆盖该域
|
||||||
|
// Drop 403/验证页/换出口:整域删除
|
||||||
|
// Reap 每 60s:过期 + 域数/字节硬顶
|
||||||
|
//
|
||||||
|
// Cookie 不进 MCP;jobs.payload 不含 Session(json:"-")。
|
||||||
|
package session
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/fingerprint"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Jar 进程内只缓存 active 模版;Cookie 全在 SQLite。
|
||||||
|
type Jar struct {
|
||||||
|
db *store.DB
|
||||||
|
nodeID string
|
||||||
|
|
||||||
|
mu sync.RWMutex
|
||||||
|
active fingerprint.Profile
|
||||||
|
}
|
||||||
|
|
||||||
|
// Open 确保本节点模版落库并选定 active。
|
||||||
|
func Open(ctx context.Context, db *store.DB, nodeID string) (*Jar, error) {
|
||||||
|
if nodeID == "" {
|
||||||
|
nodeID = fingerprint.NodeID()
|
||||||
|
}
|
||||||
|
p, err := db.EnsureNodeTemplates(ctx, nodeID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("session: 初始化模版: %w", err)
|
||||||
|
}
|
||||||
|
return &Jar{db: db, nodeID: nodeID, active: p}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Active 当前模版(内存副本,重启后从库恢复同一张脸)。
|
||||||
|
func (j *Jar) Active() fingerprint.Profile {
|
||||||
|
j.mu.RLock()
|
||||||
|
defer j.mu.RUnlock()
|
||||||
|
return j.active
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attach 为一次 read 装配会话。adapter 非 http/cdp 返回 nil。
|
||||||
|
func (j *Jar) Attach(ctx context.Context, adapter, egress, pageURL string) *contract.SessionAttach {
|
||||||
|
if j == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cls := AdapterClassOf(adapter)
|
||||||
|
if cls == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
p := j.Active()
|
||||||
|
etld := ETLD1(pageURL)
|
||||||
|
if etld == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if egress == "" {
|
||||||
|
egress = "direct"
|
||||||
|
}
|
||||||
|
cookies, err := j.db.LoadCookies(ctx, store.CookieKey{
|
||||||
|
TemplateID: p.ID, AdapterClass: cls, Egress: egress, ETLD: etld,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
cookies = nil
|
||||||
|
}
|
||||||
|
return &contract.SessionAttach{
|
||||||
|
TemplateID: p.ID,
|
||||||
|
Impersonate: p.Impersonate,
|
||||||
|
UserAgent: p.UserAgent,
|
||||||
|
Headers: fingerprint.Headers(p),
|
||||||
|
Cookies: cookies,
|
||||||
|
AdapterClass: cls,
|
||||||
|
Egress: egress,
|
||||||
|
ETLD: etld,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Commit 按结果写罐或丢罐。存储失败不失败任务。
|
||||||
|
func (j *Jar) Commit(ctx context.Context, sess *contract.SessionAttach, raw *contract.RawResult, eb *contract.ErrBody) {
|
||||||
|
if j == nil || sess == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
k := store.CookieKey{
|
||||||
|
TemplateID: sess.TemplateID, AdapterClass: sess.AdapterClass,
|
||||||
|
Egress: sess.Egress, ETLD: sess.ETLD,
|
||||||
|
}
|
||||||
|
if IsPoison(eb, raw) {
|
||||||
|
_ = j.db.DropCookies(ctx, k)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if eb != nil || raw == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cleaned := Sanitize(raw.SetCookies)
|
||||||
|
if len(cleaned) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = j.db.ReplaceCookies(ctx, k, cleaned)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reap 过期 + 硬顶。返回删除行数。
|
||||||
|
func (j *Jar) Reap(ctx context.Context) (int, error) {
|
||||||
|
if j == nil {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
return j.db.ReapCookies(ctx, MaxETLDPerKey, MaxJarBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
// EgressKey 罐用的出口键:国内 direct;海外钉节点名。
|
||||||
|
func EgressKey(region, node, proxy string) string {
|
||||||
|
if region != contract.RegionOverseas {
|
||||||
|
return "direct"
|
||||||
|
}
|
||||||
|
if node != "" {
|
||||||
|
return "node:" + node
|
||||||
|
}
|
||||||
|
if proxy != "" {
|
||||||
|
return "proxy:" + proxy
|
||||||
|
}
|
||||||
|
return "overseas"
|
||||||
|
}
|
||||||
157
server/internal/session/session_test.go
Normal file
157
server/internal/session/session_test.go
Normal file
|
|
@ -0,0 +1,157 @@
|
||||||
|
package session
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
_ "modernc.org/sqlite"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestETLD1(t *testing.T) {
|
||||||
|
cases := map[string]string{
|
||||||
|
"https://www.amazon.com/dp/X": "amazon.com",
|
||||||
|
"https://news.bbc.co.uk/x": "bbc.co.uk",
|
||||||
|
"https://a.b.com.cn/y": "b.com.cn",
|
||||||
|
"https://192.168.1.1/z": "192.168.1.1",
|
||||||
|
}
|
||||||
|
for in, want := range cases {
|
||||||
|
if got := ETLD1(in); got != want {
|
||||||
|
t.Errorf("ETLD1(%s)=%s want %s", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSanitizeDropsAuthAndHuge(t *testing.T) {
|
||||||
|
in := []contract.Cookie{
|
||||||
|
{Name: "session-id", Value: "ok"},
|
||||||
|
{Name: "at-main", Value: "secret"},
|
||||||
|
{Name: "SID", Value: "g"},
|
||||||
|
{Name: "huge", Value: strings.Repeat("x", MaxCookieBytes+1)},
|
||||||
|
{Name: "", Value: "x"},
|
||||||
|
}
|
||||||
|
out := Sanitize(in)
|
||||||
|
if len(out) != 1 || out[0].Name != "session-id" {
|
||||||
|
t.Fatalf("只应留下匿名 session-id: %+v", out)
|
||||||
|
}
|
||||||
|
if out[0].Expires <= time.Now().Unix() {
|
||||||
|
t.Fatal("应写入默认 TTL")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsPoison(t *testing.T) {
|
||||||
|
if !IsPoison(&contract.ErrBody{Code: contract.CodeBlocked, Message: "x"}, nil) {
|
||||||
|
t.Fatal("blocked 应投毒")
|
||||||
|
}
|
||||||
|
if !IsPoison(nil, &contract.RawResult{Poisoned: true}) {
|
||||||
|
t.Fatal("Poisoned 应投毒")
|
||||||
|
}
|
||||||
|
if !IsPoison(&contract.ErrBody{Code: contract.CodeUpstream, Message: "cloudflare challenge"}, nil) {
|
||||||
|
t.Fatal("cf 文案应投毒")
|
||||||
|
}
|
||||||
|
if IsPoison(&contract.ErrBody{Code: contract.CodeTimeout, Message: "deadline"}, nil) {
|
||||||
|
t.Fatal("超时不应丢罐")
|
||||||
|
}
|
||||||
|
if IsPoison(&contract.ErrBody{Code: contract.CodeUpstream, Message: "empty_extract"}, nil) {
|
||||||
|
t.Fatal("空抽取不应丢罐")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJarRoundTripAndPoisonDrop(t *testing.T) {
|
||||||
|
db, err := store.Open(filepath.Join(t.TempDir(), "s.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { db.Close() })
|
||||||
|
if err := db.Migrate(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
j, err := Open(ctx, db, "test-node")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
set, err := db.ListTemplates(ctx, "test-node")
|
||||||
|
if err != nil || len(set) != 8 {
|
||||||
|
t.Fatalf("应落 8 套模版: n=%d err=%v", len(set), err)
|
||||||
|
}
|
||||||
|
page := "https://www.example.com/a"
|
||||||
|
sess := j.Attach(ctx, "trafilatura", "direct", page)
|
||||||
|
if sess == nil || sess.Impersonate != "chrome136" {
|
||||||
|
t.Fatalf("attach: %+v", sess)
|
||||||
|
}
|
||||||
|
if strings.Contains(sess.UserAgent, "151") {
|
||||||
|
t.Fatal("模版 UA 禁止 151")
|
||||||
|
}
|
||||||
|
j.Commit(ctx, sess, &contract.RawResult{
|
||||||
|
StatusCode: 200,
|
||||||
|
SetCookies: []contract.Cookie{{Name: "sid", Value: "1"}, {Name: "pref", Value: "dark"}},
|
||||||
|
}, nil)
|
||||||
|
sess2 := j.Attach(ctx, "trafilatura", "direct", page)
|
||||||
|
if len(sess2.Cookies) != 1 || sess2.Cookies[0].Name != "pref" {
|
||||||
|
t.Fatalf("登录态 sid 应丢、pref 应在: %+v", sess2.Cookies)
|
||||||
|
}
|
||||||
|
// 另一档不应串罐
|
||||||
|
cdp := j.Attach(ctx, "lightpanda", "direct", page)
|
||||||
|
if len(cdp.Cookies) != 0 {
|
||||||
|
t.Fatalf("http/cdp 不得混罐: %+v", cdp.Cookies)
|
||||||
|
}
|
||||||
|
j.Commit(ctx, sess2, &contract.RawResult{Poisoned: true}, nil)
|
||||||
|
sess3 := j.Attach(ctx, "trafilatura", "direct", page)
|
||||||
|
if len(sess3.Cookies) != 0 {
|
||||||
|
t.Fatalf("投毒后应空罐: %+v", sess3.Cookies)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJobEnvelopeOmitsSession(t *testing.T) {
|
||||||
|
j := contract.JobEnvelope{
|
||||||
|
ID: "1", Intent: "read",
|
||||||
|
Read: &contract.ReadInput{URL: "https://x/"},
|
||||||
|
Session: &contract.SessionAttach{TemplateID: "t", Cookies: []contract.Cookie{{Name: "a", Value: "b"}}},
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(j)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(b), "a") && strings.Contains(string(b), `"name"`) {
|
||||||
|
// Cookie 名可能碰巧出现;更硬的:不得有 session 键
|
||||||
|
}
|
||||||
|
if strings.Contains(string(b), `"session"`) || strings.Contains(string(b), "TemplateID") {
|
||||||
|
t.Fatalf("Session 不得进 payload: %s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReapExpired(t *testing.T) {
|
||||||
|
db, err := store.Open(filepath.Join(t.TempDir(), "r.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { db.Close() })
|
||||||
|
if err := db.Migrate(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
j, err := Open(ctx, db, "reap-node")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
sess := j.Attach(ctx, "trafilatura", "direct", "https://old.example/x")
|
||||||
|
j.Commit(ctx, sess, &contract.RawResult{
|
||||||
|
SetCookies: []contract.Cookie{{Name: "gone", Value: "1", Expires: time.Now().Add(-time.Hour).Unix()}},
|
||||||
|
}, nil)
|
||||||
|
n, err := j.Reap(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = n
|
||||||
|
again := j.Attach(ctx, "trafilatura", "direct", "https://old.example/x")
|
||||||
|
if len(again.Cookies) != 0 {
|
||||||
|
t.Fatalf("过期应被收: %+v", again.Cookies)
|
||||||
|
}
|
||||||
|
}
|
||||||
340
server/internal/store/session.go
Normal file
340
server/internal/store/session.go
Normal file
|
|
@ -0,0 +1,340 @@
|
||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/config"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/contract"
|
||||||
|
"onesvm.com/onesvm/browser-server/internal/fingerprint"
|
||||||
|
)
|
||||||
|
|
||||||
|
// schemaV2 每节点模版 + Cookie 罐(P0/P1)。幂等 CREATE。
|
||||||
|
var schemaV2 = []string{
|
||||||
|
`CREATE TABLE IF NOT EXISTS fp_templates (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
node_id TEXT NOT NULL,
|
||||||
|
impersonate TEXT NOT NULL,
|
||||||
|
user_agent TEXT NOT NULL,
|
||||||
|
sec_ch_ua TEXT NOT NULL,
|
||||||
|
sec_ch_ua_platform TEXT NOT NULL,
|
||||||
|
accept_language TEXT NOT NULL,
|
||||||
|
os TEXT NOT NULL,
|
||||||
|
chrome_major INTEGER NOT NULL,
|
||||||
|
created_at TEXT NOT NULL
|
||||||
|
)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_fp_templates_node ON fp_templates(node_id)`,
|
||||||
|
`CREATE TABLE IF NOT EXISTS fp_state (
|
||||||
|
node_id TEXT PRIMARY KEY,
|
||||||
|
active_id TEXT NOT NULL,
|
||||||
|
updated_at TEXT NOT NULL
|
||||||
|
)`,
|
||||||
|
`CREATE TABLE IF NOT EXISTS session_cookies (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
template_id TEXT NOT NULL,
|
||||||
|
adapter_class TEXT NOT NULL,
|
||||||
|
egress TEXT NOT NULL,
|
||||||
|
etld TEXT NOT NULL,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
value TEXT NOT NULL,
|
||||||
|
domain TEXT NOT NULL,
|
||||||
|
path TEXT NOT NULL DEFAULT '/',
|
||||||
|
expires_at TEXT NOT NULL,
|
||||||
|
last_used_at TEXT NOT NULL,
|
||||||
|
bytes INTEGER NOT NULL,
|
||||||
|
UNIQUE(template_id, adapter_class, egress, etld, name, domain, path)
|
||||||
|
)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_session_cookies_lookup
|
||||||
|
ON session_cookies(template_id, adapter_class, egress, etld)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_session_cookies_expiry ON session_cookies(expires_at)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_session_cookies_lru ON session_cookies(last_used_at)`,
|
||||||
|
}
|
||||||
|
|
||||||
|
func migrateV2(d *DB) error {
|
||||||
|
for _, stmt := range schemaV2 {
|
||||||
|
if _, err := d.raw.Exec(stmt); err != nil {
|
||||||
|
return fmt.Errorf("store: migration v2: %w (stmt=%s)", err, firstLine(stmt))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnsureNodeTemplates 首次为节点写入 TemplateCount 套模版并选定 active。
|
||||||
|
func (d *DB) EnsureNodeTemplates(ctx context.Context, nodeID string) (fingerprint.Profile, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return fingerprint.Profile{}, err
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if err := d.raw.QueryRowContext(ctx,
|
||||||
|
`SELECT COUNT(*) FROM fp_templates WHERE node_id=?`, nodeID).Scan(&n); err != nil {
|
||||||
|
return fingerprint.Profile{}, fmt.Errorf("store: 数模版: %w", err)
|
||||||
|
}
|
||||||
|
now := config.FormatTime(config.Now())
|
||||||
|
if n == 0 {
|
||||||
|
set := fingerprint.AllocateSet(nodeID, fingerprint.TemplateCount)
|
||||||
|
for _, p := range set {
|
||||||
|
if _, err := d.raw.ExecContext(ctx, `INSERT INTO fp_templates
|
||||||
|
(id, node_id, impersonate, user_agent, sec_ch_ua, sec_ch_ua_platform,
|
||||||
|
accept_language, os, chrome_major, created_at)
|
||||||
|
VALUES (?,?,?,?,?,?,?,?,?,?)`,
|
||||||
|
p.ID, nodeID, p.Impersonate, p.UserAgent, p.SecCHUA, p.SecCHUAPlatform,
|
||||||
|
p.AcceptLanguage, p.OS, p.ChromeMajor, now); err != nil {
|
||||||
|
return fingerprint.Profile{}, fmt.Errorf("store: 写模版: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
active := fingerprint.PickActive(nodeID, set)
|
||||||
|
if _, err := d.raw.ExecContext(ctx,
|
||||||
|
`INSERT INTO fp_state (node_id, active_id, updated_at) VALUES (?,?,?)`,
|
||||||
|
nodeID, active.ID, now); err != nil {
|
||||||
|
return fingerprint.Profile{}, fmt.Errorf("store: 写 active: %w", err)
|
||||||
|
}
|
||||||
|
return active, nil
|
||||||
|
}
|
||||||
|
var activeID string
|
||||||
|
err := d.raw.QueryRowContext(ctx, `SELECT active_id FROM fp_state WHERE node_id=?`, nodeID).Scan(&activeID)
|
||||||
|
if err == sql.ErrNoRows || activeID == "" {
|
||||||
|
set, err := d.ListTemplates(ctx, nodeID)
|
||||||
|
if err != nil {
|
||||||
|
return fingerprint.Profile{}, err
|
||||||
|
}
|
||||||
|
active := fingerprint.PickActive(nodeID, set)
|
||||||
|
if _, err := d.raw.ExecContext(ctx,
|
||||||
|
`INSERT INTO fp_state (node_id, active_id, updated_at) VALUES (?,?,?)
|
||||||
|
ON CONFLICT(node_id) DO UPDATE SET active_id=excluded.active_id, updated_at=excluded.updated_at`,
|
||||||
|
nodeID, active.ID, now); err != nil {
|
||||||
|
return fingerprint.Profile{}, fmt.Errorf("store: 补 active: %w", err)
|
||||||
|
}
|
||||||
|
return active, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return fingerprint.Profile{}, fmt.Errorf("store: 读 active: %w", err)
|
||||||
|
}
|
||||||
|
return d.GetTemplate(ctx, activeID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListTemplates 节点全部模版。
|
||||||
|
func (d *DB) ListTemplates(ctx context.Context, nodeID string) ([]fingerprint.Profile, error) {
|
||||||
|
rows, err := d.raw.QueryContext(ctx,
|
||||||
|
`SELECT id, impersonate, user_agent, sec_ch_ua, sec_ch_ua_platform,
|
||||||
|
accept_language, os, chrome_major
|
||||||
|
FROM fp_templates WHERE node_id=? ORDER BY id`, nodeID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("store: 列模版: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []fingerprint.Profile
|
||||||
|
for rows.Next() {
|
||||||
|
var p fingerprint.Profile
|
||||||
|
if err := rows.Scan(&p.ID, &p.Impersonate, &p.UserAgent, &p.SecCHUA, &p.SecCHUAPlatform,
|
||||||
|
&p.AcceptLanguage, &p.OS, &p.ChromeMajor); err != nil {
|
||||||
|
return nil, fmt.Errorf("store: 扫模版: %w", err)
|
||||||
|
}
|
||||||
|
p.SecCHUAMobile = "?0"
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetTemplate 按 id 取模版。
|
||||||
|
func (d *DB) GetTemplate(ctx context.Context, id string) (fingerprint.Profile, error) {
|
||||||
|
var p fingerprint.Profile
|
||||||
|
err := d.raw.QueryRowContext(ctx,
|
||||||
|
`SELECT id, impersonate, user_agent, sec_ch_ua, sec_ch_ua_platform,
|
||||||
|
accept_language, os, chrome_major
|
||||||
|
FROM fp_templates WHERE id=?`, id).Scan(
|
||||||
|
&p.ID, &p.Impersonate, &p.UserAgent, &p.SecCHUA, &p.SecCHUAPlatform,
|
||||||
|
&p.AcceptLanguage, &p.OS, &p.ChromeMajor)
|
||||||
|
if err != nil {
|
||||||
|
return p, fmt.Errorf("store: 读模版 %s: %w", id, err)
|
||||||
|
}
|
||||||
|
p.SecCHUAMobile = "?0"
|
||||||
|
return p, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SwitchActive 快速切到已落库的另一套模版。
|
||||||
|
func (d *DB) SwitchActive(ctx context.Context, nodeID, templateID string) error {
|
||||||
|
now := config.FormatTime(config.Now())
|
||||||
|
_, err := d.raw.ExecContext(ctx,
|
||||||
|
`UPDATE fp_state SET active_id=?, updated_at=? WHERE node_id=?`,
|
||||||
|
templateID, now, nodeID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: 切模版: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// CookieKey 罐主键(模版 × 适配器档 × 出口 × 域)。
|
||||||
|
type CookieKey struct {
|
||||||
|
TemplateID string
|
||||||
|
AdapterClass string
|
||||||
|
Egress string
|
||||||
|
ETLD string
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadCookies 读一罐(未过期)。
|
||||||
|
func (d *DB) LoadCookies(ctx context.Context, k CookieKey) ([]contract.Cookie, error) {
|
||||||
|
now := config.FormatTime(config.Now())
|
||||||
|
rows, err := d.raw.QueryContext(ctx,
|
||||||
|
`SELECT name, value, domain, path, expires_at FROM session_cookies
|
||||||
|
WHERE template_id=? AND adapter_class=? AND egress=? AND etld=? AND expires_at>?`,
|
||||||
|
k.TemplateID, k.AdapterClass, k.Egress, k.ETLD, now)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("store: 读罐: %w", err)
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []contract.Cookie
|
||||||
|
for rows.Next() {
|
||||||
|
var c contract.Cookie
|
||||||
|
var exp string
|
||||||
|
if err := rows.Scan(&c.Name, &c.Value, &c.Domain, &c.Path, &exp); err != nil {
|
||||||
|
return nil, fmt.Errorf("store: 扫罐: %w", err)
|
||||||
|
}
|
||||||
|
if t, err := config.ParseRFC3339(exp); err == nil {
|
||||||
|
c.Expires = t.Unix()
|
||||||
|
}
|
||||||
|
out = append(out, c)
|
||||||
|
}
|
||||||
|
if err := rows.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(out) > 0 {
|
||||||
|
_, _ = d.raw.ExecContext(ctx,
|
||||||
|
`UPDATE session_cookies SET last_used_at=?
|
||||||
|
WHERE template_id=? AND adapter_class=? AND egress=? AND etld=?`,
|
||||||
|
now, k.TemplateID, k.AdapterClass, k.Egress, k.ETLD)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReplaceCookies 覆盖一域的罐(先删后插)。调用方已 Sanitize。
|
||||||
|
func (d *DB) ReplaceCookies(ctx context.Context, k CookieKey, cookies []contract.Cookie) error {
|
||||||
|
nowS := config.FormatTime(config.Now())
|
||||||
|
tx, err := d.raw.BeginTx(ctx, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: 开事务: %w", err)
|
||||||
|
}
|
||||||
|
defer tx.Rollback() // Commit 后幂等
|
||||||
|
if _, err := tx.ExecContext(ctx,
|
||||||
|
`DELETE FROM session_cookies WHERE template_id=? AND adapter_class=? AND egress=? AND etld=?`,
|
||||||
|
k.TemplateID, k.AdapterClass, k.Egress, k.ETLD); err != nil {
|
||||||
|
return fmt.Errorf("store: 清罐: %w", err)
|
||||||
|
}
|
||||||
|
for _, c := range cookies {
|
||||||
|
expS := nowS
|
||||||
|
if c.Expires > 0 {
|
||||||
|
expS = config.FormatTime(time.Unix(c.Expires, 0))
|
||||||
|
}
|
||||||
|
bytes := len(c.Name) + len(c.Value) + len(c.Domain) + len(c.Path)
|
||||||
|
if _, err := tx.ExecContext(ctx, `INSERT INTO session_cookies
|
||||||
|
(template_id, adapter_class, egress, etld, name, value, domain, path,
|
||||||
|
expires_at, last_used_at, bytes)
|
||||||
|
VALUES (?,?,?,?,?,?,?,?,?,?,?)`,
|
||||||
|
k.TemplateID, k.AdapterClass, k.Egress, k.ETLD,
|
||||||
|
c.Name, c.Value, c.Domain, c.Path, expS, nowS, bytes); err != nil {
|
||||||
|
return fmt.Errorf("store: 插 Cookie: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := tx.Commit(); err != nil {
|
||||||
|
return fmt.Errorf("store: 提交罐: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DropCookies 整域丢罐(投毒 / 换出口)。
|
||||||
|
func (d *DB) DropCookies(ctx context.Context, k CookieKey) error {
|
||||||
|
_, err := d.raw.ExecContext(ctx,
|
||||||
|
`DELETE FROM session_cookies WHERE template_id=? AND adapter_class=? AND egress=? AND etld=?`,
|
||||||
|
k.TemplateID, k.AdapterClass, k.Egress, k.ETLD)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: 丢罐: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReapCookies 过期删除 + 总字节/域数硬顶(LRU)。
|
||||||
|
func (d *DB) ReapCookies(ctx context.Context, maxETLD, maxBytes int) (int, error) {
|
||||||
|
now := config.FormatTime(config.Now())
|
||||||
|
res, err := d.raw.ExecContext(ctx, `DELETE FROM session_cookies WHERE expires_at<=?`, now)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("store: 收过期罐: %w", err)
|
||||||
|
}
|
||||||
|
n, _ := res.RowsAffected()
|
||||||
|
cut, err := d.enforceJarCaps(ctx, maxETLD, maxBytes)
|
||||||
|
return int(n) + cut, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *DB) enforceJarCaps(ctx context.Context, maxETLD, maxBytes int) (int, error) {
|
||||||
|
cut := 0
|
||||||
|
// 按 (template,adapter,egress) 组限制域数。
|
||||||
|
rows, err := d.raw.QueryContext(ctx,
|
||||||
|
`SELECT template_id, adapter_class, egress, etld, MAX(last_used_at) AS lu
|
||||||
|
FROM session_cookies
|
||||||
|
GROUP BY template_id, adapter_class, egress, etld
|
||||||
|
ORDER BY template_id, adapter_class, egress, lu ASC`)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("store: 域统计: %w", err)
|
||||||
|
}
|
||||||
|
type grp struct{ tpl, cls, eg, etld string }
|
||||||
|
counts := map[string][]grp{}
|
||||||
|
for rows.Next() {
|
||||||
|
var g grp
|
||||||
|
var lu string
|
||||||
|
if err := rows.Scan(&g.tpl, &g.cls, &g.eg, &g.etld, &lu); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return cut, err
|
||||||
|
}
|
||||||
|
key := g.tpl + "\x00" + g.cls + "\x00" + g.eg
|
||||||
|
counts[key] = append(counts[key], g)
|
||||||
|
}
|
||||||
|
if err := rows.Err(); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return cut, err
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
for _, list := range counts {
|
||||||
|
if len(list) <= maxETLD {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
overflow := list[:len(list)-maxETLD] // 已按 lu ASC,最旧在前
|
||||||
|
for _, g := range overflow {
|
||||||
|
r, err := d.raw.ExecContext(ctx,
|
||||||
|
`DELETE FROM session_cookies WHERE template_id=? AND adapter_class=? AND egress=? AND etld=?`,
|
||||||
|
g.tpl, g.cls, g.eg, g.etld)
|
||||||
|
if err != nil {
|
||||||
|
return cut, fmt.Errorf("store: LRU 域: %w", err)
|
||||||
|
}
|
||||||
|
c, _ := r.RowsAffected()
|
||||||
|
cut += int(c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var total int
|
||||||
|
if err := d.raw.QueryRowContext(ctx, `SELECT COALESCE(SUM(bytes),0) FROM session_cookies`).Scan(&total); err != nil {
|
||||||
|
return cut, fmt.Errorf("store: 字节合计: %w", err)
|
||||||
|
}
|
||||||
|
for total > maxBytes {
|
||||||
|
var tpl, cls, eg, etld string
|
||||||
|
err := d.raw.QueryRowContext(ctx,
|
||||||
|
`SELECT template_id, adapter_class, egress, etld FROM session_cookies
|
||||||
|
ORDER BY last_used_at ASC LIMIT 1`).Scan(&tpl, &cls, &eg, &etld)
|
||||||
|
if err == sql.ErrNoRows {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return cut, fmt.Errorf("store: 找最旧域: %w", err)
|
||||||
|
}
|
||||||
|
r, err := d.raw.ExecContext(ctx,
|
||||||
|
`DELETE FROM session_cookies WHERE template_id=? AND adapter_class=? AND egress=? AND etld=?`,
|
||||||
|
tpl, cls, eg, etld)
|
||||||
|
if err != nil {
|
||||||
|
return cut, err
|
||||||
|
}
|
||||||
|
c, _ := r.RowsAffected()
|
||||||
|
cut += int(c)
|
||||||
|
if err := d.raw.QueryRowContext(ctx, `SELECT COALESCE(SUM(bytes),0) FROM session_cookies`).Scan(&total); err != nil {
|
||||||
|
return cut, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cut, nil
|
||||||
|
}
|
||||||
|
|
@ -130,6 +130,9 @@ func (d *DB) Migrate() error {
|
||||||
return fmt.Errorf("store: migration: %w (stmt=%s)", err, firstLine(stmt))
|
return fmt.Errorf("store: migration: %w (stmt=%s)", err, firstLine(stmt))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if err := migrateV2(d); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -41,7 +41,8 @@ func TestMigration(t *testing.T) {
|
||||||
if err := db.Migrate(); err != nil {
|
if err := db.Migrate(); err != nil {
|
||||||
t.Fatalf("二次 migration 应幂等: %v", err)
|
t.Fatalf("二次 migration 应幂等: %v", err)
|
||||||
}
|
}
|
||||||
for _, tbl := range []string{"consumers", "api_keys", "quota_ledger", "jobs", "dead_letters", "audit", "rules", "robots_cache"} {
|
for _, tbl := range []string{"consumers", "api_keys", "quota_ledger", "jobs", "dead_letters", "audit", "rules", "robots_cache",
|
||||||
|
"fp_templates", "fp_state", "session_cookies"} {
|
||||||
var n int
|
var n int
|
||||||
if err := db.Raw().QueryRow(
|
if err := db.Raw().QueryRow(
|
||||||
`SELECT COUNT(name) FROM sqlite_master WHERE type='table' AND name=?`, tbl).Scan(&n); err != nil {
|
`SELECT COUNT(name) FROM sqlite_master WHERE type='table' AND name=?`, tbl).Scan(&n); err != nil {
|
||||||
|
|
|
||||||
|
|
@ -245,7 +245,7 @@ services:
|
||||||
- node.hostname==swarm-mgr1
|
- node.hostname==swarm-mgr1
|
||||||
resources:
|
resources:
|
||||||
limits:
|
limits:
|
||||||
memory: 64m
|
memory: 192m
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=3)"]
|
test: ["CMD", "python", "-c", "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=3)"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,6 @@ use_default_settings:
|
||||||
- sogou
|
- sogou
|
||||||
- 360search
|
- 360search
|
||||||
- bing
|
- bing
|
||||||
- wikipedia
|
|
||||||
|
|
||||||
search:
|
search:
|
||||||
formats:
|
formats:
|
||||||
|
|
@ -33,8 +32,6 @@ engines:
|
||||||
- name: bing
|
- name: bing
|
||||||
disabled: false
|
disabled: false
|
||||||
base_url: https://cn.bing.com
|
base_url: https://cn.bing.com
|
||||||
- name: wikipedia
|
|
||||||
disabled: false
|
|
||||||
|
|
||||||
outgoing:
|
outgoing:
|
||||||
request_timeout: 8.0
|
request_timeout: 8.0
|
||||||
|
|
|
||||||
|
|
@ -18,26 +18,35 @@ server:
|
||||||
image_proxy: false
|
image_proxy: false
|
||||||
method: GET
|
method: GET
|
||||||
engines:
|
engines:
|
||||||
- name: google
|
|
||||||
disabled: false
|
|
||||||
- name: bing
|
- name: bing
|
||||||
disabled: false
|
disabled: false
|
||||||
|
# 数据中心 vless 下这些引擎实测 CAPTCHA/429/超时,短超时避免拖死 Bing。
|
||||||
|
- name: google
|
||||||
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: duckduckgo
|
- name: duckduckgo
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: brave
|
- name: brave
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: startpage
|
- name: startpage
|
||||||
disabled: false
|
disabled: false
|
||||||
- name: wikipedia
|
timeout: 3.0
|
||||||
disabled: false
|
|
||||||
- name: reddit
|
- name: reddit
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: mojeek
|
- name: mojeek
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: qwant
|
- name: qwant
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
- name: yahoo
|
- name: yahoo
|
||||||
disabled: false
|
disabled: false
|
||||||
|
timeout: 3.0
|
||||||
|
- name: wikipedia
|
||||||
|
disabled: true
|
||||||
outgoing:
|
outgoing:
|
||||||
request_timeout: 8.0
|
request_timeout: 8.0
|
||||||
max_request_timeout: 15.0
|
max_request_timeout: 15.0
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
FROM python:3.12-slim-bookworm
|
FROM python:3.12-slim-bookworm
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY app.py .
|
COPY app.py .
|
||||||
RUN pip install --no-cache-dir trafilatura==2.2.0 \
|
RUN pip install --no-cache-dir trafilatura==2.2.0 'curl_cffi==0.13.0' \
|
||||||
|| pip install --no-cache-dir -i https://pypi.tuna.tsinghua.edu.cn/simple trafilatura==2.2.0
|
|| pip install --no-cache-dir -i https://pypi.tuna.tsinghua.edu.cn/simple trafilatura==2.2.0 'curl_cffi==0.13.0'
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
CMD ["python", "-u", "app.py"]
|
CMD ["python", "-u", "app.py"]
|
||||||
|
|
|
||||||
|
|
@ -66,13 +66,105 @@ def _guard_url(url: str) -> str | None:
|
||||||
return _forbidden_host(parsed.hostname)
|
return _forbidden_host(parsed.hostname)
|
||||||
|
|
||||||
|
|
||||||
def _extract(url: str, max_chars: int) -> dict:
|
def _looks_challenge(text: str) -> bool:
|
||||||
|
blob = (text or "")[:8000].lower()
|
||||||
|
keys = (
|
||||||
|
"just a moment",
|
||||||
|
"cf-challenge",
|
||||||
|
"attention required",
|
||||||
|
"verify you are human",
|
||||||
|
"captcha",
|
||||||
|
"access denied",
|
||||||
|
)
|
||||||
|
return any(k in blob for k in keys)
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_cookies(sess) -> list[dict]:
|
||||||
|
out: list[dict] = []
|
||||||
|
jar = getattr(sess, "cookies", None)
|
||||||
|
if jar is None:
|
||||||
|
return out
|
||||||
|
items = []
|
||||||
|
inner = getattr(jar, "jar", None)
|
||||||
|
if inner is not None:
|
||||||
|
try:
|
||||||
|
items = list(inner)
|
||||||
|
except TypeError:
|
||||||
|
items = []
|
||||||
|
if not items:
|
||||||
|
try:
|
||||||
|
for name, value in jar.items():
|
||||||
|
out.append({"name": name, "value": value, "domain": "", "path": "/"})
|
||||||
|
if len(out) >= 20:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
except Exception:
|
||||||
|
return out
|
||||||
|
for c in items:
|
||||||
|
name = getattr(c, "name", "") or ""
|
||||||
|
value = getattr(c, "value", "") or ""
|
||||||
|
if not name or not value:
|
||||||
|
continue
|
||||||
|
rec = {
|
||||||
|
"name": name,
|
||||||
|
"value": value,
|
||||||
|
"domain": getattr(c, "domain", "") or "",
|
||||||
|
"path": getattr(c, "path", "") or "/",
|
||||||
|
}
|
||||||
|
exp = getattr(c, "expires", None)
|
||||||
|
if exp:
|
||||||
|
rec["expires"] = int(exp)
|
||||||
|
out.append(rec)
|
||||||
|
if len(out) >= 20:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch(url: str, headers: dict | None, cookies: list | None, impersonate: str) -> tuple[str | None, list[dict], int, bool]:
|
||||||
|
"""优先 curl_cffi(TLS=Chrome);缺库时回退 trafilatura.fetch_url(无指纹)。"""
|
||||||
|
try:
|
||||||
|
from curl_cffi import requests as cfreq
|
||||||
|
except ImportError:
|
||||||
|
downloaded = trafilatura.fetch_url(url, config=_CFG)
|
||||||
|
return downloaded, [], 0, False
|
||||||
|
|
||||||
|
sess = cfreq.Session(impersonate=impersonate or "chrome136")
|
||||||
|
if headers:
|
||||||
|
sess.headers.update({k: v for k, v in headers.items() if v})
|
||||||
|
for c in cookies or []:
|
||||||
|
name = (c.get("name") or "").strip()
|
||||||
|
value = c.get("value") or ""
|
||||||
|
if not name:
|
||||||
|
continue
|
||||||
|
kwargs = {}
|
||||||
|
if c.get("domain"):
|
||||||
|
kwargs["domain"] = c["domain"]
|
||||||
|
if c.get("path"):
|
||||||
|
kwargs["path"] = c["path"]
|
||||||
|
try:
|
||||||
|
sess.cookies.set(name, value, **kwargs)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
r = sess.get(url, timeout=TIMEOUT_S, allow_redirects=True)
|
||||||
|
except Exception:
|
||||||
|
return None, [], 0, False
|
||||||
|
text = r.text if r is not None else ""
|
||||||
|
status = int(getattr(r, "status_code", 0) or 0)
|
||||||
|
poisoned = status in (403, 429, 503) or _looks_challenge(text)
|
||||||
|
return text or None, _collect_cookies(sess), status, poisoned
|
||||||
|
|
||||||
|
|
||||||
|
def _extract(url: str, max_chars: int, headers=None, cookies=None, impersonate="") -> dict:
|
||||||
err = _guard_url(url)
|
err = _guard_url(url)
|
||||||
if err:
|
if err:
|
||||||
return {"ok": False, "error": err, "fail_class": "fetch_fail"}
|
return {"ok": False, "error": err, "fail_class": "fetch_fail"}
|
||||||
downloaded = trafilatura.fetch_url(url, config=_CFG)
|
downloaded, set_cookies, status, poisoned = _fetch(url, headers, cookies, impersonate)
|
||||||
|
extra = {"set_cookies": set_cookies, "status_code": status, "poisoned": poisoned}
|
||||||
|
if poisoned:
|
||||||
|
return {"ok": False, "error": f"challenge_or_http_{status}", "fail_class": "blocked", **extra}
|
||||||
if not downloaded:
|
if not downloaded:
|
||||||
return {"ok": False, "error": "fetch_empty", "fail_class": "fetch_fail"}
|
return {"ok": False, "error": "fetch_empty", "fail_class": "fetch_fail", **extra}
|
||||||
if len(downloaded.encode("utf-8", errors="replace")) > MAX_DOWNLOAD_BYTES:
|
if len(downloaded.encode("utf-8", errors="replace")) > MAX_DOWNLOAD_BYTES:
|
||||||
downloaded = downloaded.encode("utf-8", errors="replace")[:MAX_DOWNLOAD_BYTES].decode(
|
downloaded = downloaded.encode("utf-8", errors="replace")[:MAX_DOWNLOAD_BYTES].decode(
|
||||||
"utf-8", errors="ignore"
|
"utf-8", errors="ignore"
|
||||||
|
|
@ -86,7 +178,7 @@ def _extract(url: str, max_chars: int) -> dict:
|
||||||
config=_CFG,
|
config=_CFG,
|
||||||
)
|
)
|
||||||
if not markdown:
|
if not markdown:
|
||||||
return {"ok": False, "error": "empty_extract", "fail_class": "empty_extract", "title": ""}
|
return {"ok": False, "error": "empty_extract", "fail_class": "empty_extract", "title": "", **extra}
|
||||||
meta = extract_metadata(downloaded)
|
meta = extract_metadata(downloaded)
|
||||||
title = (meta.title if meta and getattr(meta, "title", None) else "") or ""
|
title = (meta.title if meta and getattr(meta, "title", None) else "") or ""
|
||||||
truncated = False
|
truncated = False
|
||||||
|
|
@ -100,6 +192,7 @@ def _extract(url: str, max_chars: int) -> dict:
|
||||||
"char_count": len(markdown),
|
"char_count": len(markdown),
|
||||||
"truncated": truncated,
|
"truncated": truncated,
|
||||||
"url": url,
|
"url": url,
|
||||||
|
**extra,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -139,6 +232,9 @@ class Handler(BaseHTTPRequestHandler):
|
||||||
return
|
return
|
||||||
url = (payload.get("url") or "").strip()
|
url = (payload.get("url") or "").strip()
|
||||||
max_chars = int(payload.get("max_chars") or 20000)
|
max_chars = int(payload.get("max_chars") or 20000)
|
||||||
|
headers = payload.get("headers") if isinstance(payload.get("headers"), dict) else None
|
||||||
|
cookies = payload.get("cookies") if isinstance(payload.get("cookies"), list) else None
|
||||||
|
impersonate = str(payload.get("impersonate") or "chrome136")
|
||||||
if not url:
|
if not url:
|
||||||
self._json(400, {"ok": False, "error": "url_required"})
|
self._json(400, {"ok": False, "error": "url_required"})
|
||||||
return
|
return
|
||||||
|
|
@ -147,7 +243,7 @@ class Handler(BaseHTTPRequestHandler):
|
||||||
self._json(429, {"ok": False, "error": "queue_timeout", "fail_class": "timeout"})
|
self._json(429, {"ok": False, "error": "queue_timeout", "fail_class": "timeout"})
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
result = _extract(url, max_chars)
|
result = _extract(url, max_chars, headers, cookies, impersonate)
|
||||||
self._json(200 if result.get("ok") else 200, result)
|
self._json(200 if result.get("ok") else 200, result)
|
||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc: # noqa: BLE001
|
||||||
traceback.print_exc()
|
traceback.print_exc()
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue