Compare commits
4 commits
8ec1cc397e
...
9b689b2476
| Author | SHA1 | Date | |
|---|---|---|---|
| 9b689b2476 | |||
| 854ffcb2b1 | |||
| eb972dfa93 | |||
| 983259836d |
722 changed files with 36923 additions and 2 deletions
81
.dsh/casa.yml
Normal file
81
.dsh/casa.yml
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# CASA 项目配置（DSH 版）
|
||||
# 复制到：.dsh/casa.yml
|
||||
# models.families.* 只是偏好声明：DSH 会话模型由部署决定，禁止编造模型名。
|
||||
|
||||
# ---------- 知识库（可选）----------
|
||||
knowledge:
|
||||
mode: off # off | builtin | external；有组织 KB（onesvm-dev-md）→ off
|
||||
before_contract: true
|
||||
builtin_root: casa-commander/kb
|
||||
builtin_entrypoints:
|
||||
- README.md
|
||||
- norms/coding-baseline.md
|
||||
- norms/casa-task.md
|
||||
- procedures/task-flow.md
|
||||
external:
|
||||
root: ""
|
||||
entrypoints: []
|
||||
merge_into_arch_review: true
|
||||
|
||||
# ---------- 模型族（DSH：仅建议；subagent_cursor 可真实解析）----------
|
||||
# family = 角色/复杂度标签。DSH 内置 subagent 继承部署路由（family 仅为偏好）；
|
||||
# 派发 subagent_cursor 时 family 直接映射为 Cursor 模型 spec（brief 的 Required
|
||||
# model 行逐次生效）。spec 语法：{模型id}[?参数=值&...]。
|
||||
# 已实测可用：composer-2.5；grok-4.6 全档（effort low/medium/high/xhigh ±fast）。
|
||||
models:
|
||||
families:
|
||||
simple: grok-4.6?effort=high
|
||||
hard: grok-4.6?effort=xhigh
|
||||
architect: grok-4.6?effort=xhigh
|
||||
reviewer: grok-4.6?effort=xhigh
|
||||
|
||||
routing:
|
||||
default_family: simple
|
||||
complexity:
|
||||
simple: simple
|
||||
hard: hard
|
||||
core_framework: architect
|
||||
|
||||
architecture_review:
|
||||
enabled_for:
|
||||
- core_framework
|
||||
reviewer: subagent_fork # 只读审查者
|
||||
norm_globs:
|
||||
- docs/**/*.md
|
||||
- AGENTS.md
|
||||
- onesvm-dev-md/base/**/*.md
|
||||
require_pass_to_done: true
|
||||
|
||||
# ---------- Gate 运行时（纪律参数）----------
|
||||
runtime:
|
||||
session_ttl_hours: 8
|
||||
worker_ttl_minutes: 30
|
||||
# 缺 Plan/Step/scope 时：warn（默认）= 建议并放行；enforce = 硬拒绝。
|
||||
# 过期 / 暂停 / 错误 Plan ID 仍始终硬拒绝。
|
||||
worker_start: warn
|
||||
# 讨论 CASA 本身不启模式；仅明确「使用/启用」才激活
|
||||
intent_only: true
|
||||
allow_simple_without_gate: true
|
||||
runtime_dir: .dsh/casa-runtime
|
||||
|
||||
policy:
|
||||
max_iterations: 3
|
||||
control_plane_max_file_bytes: 0
|
||||
write_timeline: true
|
||||
|
||||
# ---------- onesvm-dev-md 自动提交（可选，项目级）----------
|
||||
# 仅当工作区含 onesvm-dev-md submodule 且本会话新改时生效；绝不自动 push。
|
||||
dev_md_sync:
|
||||
enabled: true
|
||||
auto_commit: true
|
||||
auto_push: false
|
||||
require_verify_docs: true
|
||||
exclude:
|
||||
- base/credentials.md
|
||||
- base/local-machine-services.md
|
||||
|
||||
preferences_note: |
|
||||
DSH 会话模型由部署决定，本文件的 families 仅是偏好标签，不改变路由。
|
||||
已有组织知识库时保持 knowledge.mode: off。
|
||||
简单任务默认直接执行；仅明确启动 CASA 或已有活动 Plan 时启用门禁。
|
||||
DSH 无 hooks：Gate 是 agent 自律纪律，硬边界为沙箱与审批栈。
|
||||
115
.dsh/contracts/contract-20260901-browser-arch.md
Normal file
115
.dsh/contracts/contract-20260901-browser-arch.md
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
# Contract: onesvm-browser-server 联网搜索服务架构选型与实测评定
|
||||
|
||||
> 状态: done
|
||||
> 创建: 2026-09-01（Asia/Shanghai）
|
||||
> run_id: run-20260901-browser-arch
|
||||
> CASA Contract Gate — 仅 `approved` / `in_progress` 后进入 Execute
|
||||
|
||||
## deliverable_type
|
||||
|
||||
docs + spike（架构方案文档 + 开源方案实测评定；本轮**不**写生产业务代码）
|
||||
|
||||
## complexity
|
||||
|
||||
core_framework（最终方案须经 Architecture Review 通过才可 done）
|
||||
|
||||
## model_family
|
||||
|
||||
architect（最终方案）；实测/调研步骤 hard/simple 见 Plan 步骤表
|
||||
|
||||
## model_override
|
||||
|
||||
（空）
|
||||
|
||||
## 档位
|
||||
|
||||
commander（用户明确「作为开发架构与开发指挥官」）
|
||||
|
||||
## 背景
|
||||
|
||||
新仓 onesvm-browser-server 要构建「支持可扩容、集成多浏览器自构建方案、高并发排队」的联网搜索服务，服务对象为 Vlepontas（60 会话级）与 EAI 等内部 toB 智能体。用户给出 6 项任务：统一出口 / 集群与排队 / 拓展坞协议+整合器 / ProxyManager 合规代理 / 开源方案筛选实测 / 最终方案 review。用户已确认四项关键决策：① 部署目标 dev-swarm 内网集群；② 已有代理订阅（容灾后续补充，明文仅存 `.dsh/artifacts/proxy-subscriptions.local.md`，gitignore）；③ 出口 MCP Server 为主 + HTTP API 兜底；④ 本机 Docker 全量实测。
|
||||
|
||||
## scope
|
||||
|
||||
### in_scope
|
||||
|
||||
```
|
||||
.dsh/contracts/**
|
||||
.dsh/artifacts/**
|
||||
.dsh/casa-runtime/**
|
||||
docs/**
|
||||
research/**
|
||||
bench/**
|
||||
```
|
||||
|
||||
### out_of_scope
|
||||
|
||||
```
|
||||
生产/primary 环境任何变更（本轮仅设计+本机实测）
|
||||
其它业务仓（vlepontas / EAI 等）
|
||||
onesvm-dev-md/base/credentials.md（只读不改）
|
||||
git commit / push（除非用户明确要求）
|
||||
代理订阅明文（禁止写入任何 in-scope 的可提交文件；仅 .dsh/artifacts/*.local.md）
|
||||
生产级源码实现（src/** 本轮不建，方案通过后另开 Contract）
|
||||
```
|
||||
|
||||
## acceptance
|
||||
|
||||
- [ ] 最终方案文档 `docs/` 覆盖用户任务 1–6 全部要点，含 mermaid 拓扑
|
||||
- [ ] 实测报告覆盖**国内外各 ≥2 套**候选方案，每套含：数据质量样本、延迟、内存占用、成功率、反爬表现；搜索需求模板 ≥5 套（国内搜索 / 国内正文提取 / 国外搜索·代理 / 国外 JS 渲染页 / 60 会话 burst 压测）
|
||||
- [ ] 集群规格设计落在 dev-swarm 现实约束（3 节点 × ~6.5GB RAM）内，且**本项目全栈（网关+排队+ProxyManager+无头 worker 集群）总内存 ≤1GB，按需求从低水位渐进递增至 1GB 上限**；给出无 Redis 排队机制与横向扩展公式（含 1GB 内的渐进扩容档位表）
|
||||
- [ ] ProxyManager 设计含：订阅解析容灾、健康检查轮换、出口端合规拦截（违反中国法律的数据获取直接拦截）+ MCP 文档消费侧规则（双向保险）
|
||||
- [ ] 出口形态 = MCP Server 为主 + HTTP API 兜底；HTTP 侧采用组织 urlapi 惯例（独立端口 + `/search-api` 类前缀），认证头遵循 service-secret-protocol（静态 key 走 `X-Service-Token`/`X-API-Key`，禁 Bearer）
|
||||
- [ ] 有头浏览器仅抽象插拔预留，不建具体设施
|
||||
- [ ] 所有实测数据来自本机 Docker 真实运行，禁止纸面编造（每条数据可追溯至 `bench/` 产物）
|
||||
- [ ] Verify + 独立 Reverify 双过；Architecture Review PASS
|
||||
|
||||
## constraints
|
||||
|
||||
- 能不加 Redis 就不加（用户硬性偏好，排队机制须给无 Redis 方案）
|
||||
- **本项目集群总内存硬上限 1GB，且按需求渐进递增（起步低水位 → 逐步增至 1GB）**：内存成为选型的决定性指标，实测必须精确记录每方案 idle/单会话/60 burst 三档内存；超轻量内核（Lightpanda/Bouncy/chrome-headless-shell/PinchTab 类）优先级上调，重型方案（Camoufox 等）须论证单实例内存或降为预留档
|
||||
- **若 bench 数据显示 1GB 无法维持 Vlepontas 60 会话级 + EAI 并发负载**，最终方案必须附「扩容提议」：给出维持该负载所需的具体规格（内存总量/节点分布/各组件副本数）与预期性能（并发能力/延迟/吞吐），每项数字由 bench 实测数据换算支撑，供用户决策
|
||||
- 轻量化/无头优先；有头仅预留插拔
|
||||
- 候选方案最终评定：国内外各 ≥2，总数控制在 4–6 套，稳定质量好优先
|
||||
- 本机 Mac Docker 实测须控制资源（同时运行候选容器 ≤3，实测完即停）
|
||||
- 时间/时区一律 Asia/Shanghai
|
||||
- 源文件行数硬上限 600（本轮主要为文档与 bench 脚本）
|
||||
- 实测执行类委派 `subagent_cursor`（brief 写 `Required model: grok-4.6?effort=high`；复杂实测 `xhigh`）
|
||||
|
||||
## artifact_io
|
||||
|
||||
| stage | 读取 (kinds/paths) | 写入 (kinds/paths) |
|
||||
|-------|--------------------|--------------------|
|
||||
| S1 research | web + 用户附表 | `.dsh/artifacts/run-20260901-browser-arch/research/*.md` |
|
||||
| S2 select | S1 产物 + 附表 | `.dsh/artifacts/run-20260901-browser-arch/select.md` + `bench/plan.md` |
|
||||
| S3a proxy-probe | `.dsh/artifacts/proxy-subscriptions.local.md` | `.dsh/artifacts/run-20260901-browser-arch/proxy-probe.md` |
|
||||
| S3b/S3c bench | S2 选定方案 + 代理 | `bench/**` + `.dsh/artifacts/run-20260901-browser-arch/bench/*.md,*.json` |
|
||||
| S4 arch-design | KB + S1 | `docs/design-arch-*.md` |
|
||||
| S5 final-plan | 全部前置产物 | `docs/plan-final-*.md` |
|
||||
| S6 verify/reverify | S5 文档 + 全部产物 | `.dsh/artifacts/run-20260901-browser-arch/review*.md` |
|
||||
| S7 arch_review | S5 + norms | `.dsh/artifacts/run-20260901-browser-arch/arch-review.md` |
|
||||
|
||||
## stage_dag
|
||||
|
||||
```
|
||||
[S1] ∥ [S3a] → [S2] → [S3b ∥ S3c] → [S5]
|
||||
[S4（依赖 S1，与 S2/S3 并行）] ↗
|
||||
[S5] → [S6 verify+reverify] → [S7 arch_review] → done
|
||||
```
|
||||
|
||||
## revisions
|
||||
|
||||
| 日期 | 变更摘要 | 作废 artifact |
|
||||
|------|----------|---------------|
|
||||
| 2026-09-01 | 初版；用户四问已确认（dev-swarm / 代理订阅已有 / MCP+HTTP / 本机全量实测） | — |
|
||||
| 2026-09-01 | 用户补充硬约束：本项目集群总内存 ≤1GB 且按需求渐进递增；内存升为选型决定性指标 | — |
|
||||
| 2026-09-01 | 用户补充：若 1GB 不够维持 Vlepontas+EAI 负载，须给出扩容提议（规格+预期性能，bench 数据支撑） | — |
|
||||
| 2026-09-01 | S6 ITERATE：Verify PASS + Reverify FAIL（P1×2/P2×4/P3×2）→ 主会话修订 plan-final 与 design-arch（记账口径/入队路径/420×/mihomo 标注/节点布局/扩容换算/端口取代/MCP 表述），送双重复查 | reverify.md 初版结论 FAIL，待复审 |
|
||||
|
||||
## notes
|
||||
|
||||
- 组织 KB 入口：`onesvm-dev-md/base/development-standards.md`、`deployment-rules.md`、`infrastructure-inventory.md`；经验 `experience/projects/vlepontas.md`（既有 WSG/Tavily 先例，本服务为其自构建替代/升级）、`dev-swarm.md`
|
||||
- dev-swarm 事实：3 manager（.61/.62/.63，VM107/108/109），6656MB RAM/节点，overlay 10.60.0.0/16，Engine 29.5.3；镜像传输走 docker save/load + `--resolve-image never`
|
||||
- 已有 central-proxy（.53:7890，订阅 :8080/sub/onesvm.yaml）可作 ProxyManager 参考/复用点
|
||||
- 风险：① 代理订阅节点质量未知（S3a 先探活）；② 本机 Mac 资源有限，全量实测分批；③ 部分候选（Camoufox/Lightpanda）平台兼容性需实测验证
|
||||
- 回滚：本轮仅文档与本机容器，`docker compose down` + 删文档即可
|
||||
137
.dsh/contracts/contract-20260901-browser-impl.md
Normal file
137
.dsh/contracts/contract-20260901-browser-impl.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Contract: onesvm-browser-server 控制面首版实现（到「部署 mgr1 测试」就绪）
|
||||
|
||||
- **contract_id**: contract-20260901-browser-impl
|
||||
- **run_id**: run-20260901-browser-impl
|
||||
- **创建**: 2026-09-01（Asia/Shanghai）
|
||||
- **状态**: approved
|
||||
- **复杂度**: core_framework（触发 Architecture Review）
|
||||
- **用户授权**: 2026-09-01 用户原话确认「按预设（D7=Go、D8=进程内适配器、D9=灰度、D10=附加 casa-net、D1=L0→L2 渐进、D3=:8640//bs-api）开实现 Contract」——D1–D10 全部按预设锁定，不再询问。
|
||||
|
||||
## 1. Goal
|
||||
|
||||
实现 docs 全部设计所定义的**控制面三角色 Go 单仓**（gateway / scheduler / proxymanager，`cmd/browser-server` 单二进制 role 分流），**到部署 mgr1 测试的就绪态**：代码 + 单测全绿 + 本机 docker compose 冒烟通过 + amd64 镜像构建 + stack.yml + 传输/部署脚本。**本 Contract 不执行任何对 mgr1 的实际部署/SSH 操作**（部署轮另走 🟡 确认）。
|
||||
|
||||
## 2. 权威输入（Read，全部必读）
|
||||
|
||||
| 输入 | 路径 | 用途 |
|
||||
|---|---|---|
|
||||
| 设计权威 | `docs/design-arch-20260901.md` | 机制细节（§2 认证/配额、§3 Dock 五方法/信封、§4 队列/背压、§5 ProxyManager/合规、§7 可观测） |
|
||||
| 决策权威 | `docs/plan-final-20260901.md` | 结论与档位（§2 排队/内存、§3 Dock/输出、§5 评分） |
|
||||
| 部署预设 | `docs/deploy-prod-preset-20260901.md` | 服务清单/端口/资源/灰度步骤（D7/D8/D9/D10 已锁） |
|
||||
| MCP 契约 | `docs/mcp-usage-20260901.md` | 工具参数/响应形状/错误码对消费者承诺（T1 契约面） |
|
||||
| 组织规范 | `onesvm-dev-md/base/development-standards.md` | §5 urlapi、§6 镜像 tag、§7 东八区 |
|
||||
| 组织规范 | `onesvm-dev-md/base/deployment-rules.md` | stack 必备字段/secret/命名 |
|
||||
| 实测凭据 | `bench/searxng-{cn,global}/settings.yml`、`bench/{lightpanda,chrome-headless-shell}/compose.yml`、`bench/trafilatura-http/app.py`、`bench/site-matrix/cdp_fetch.mjs`、`bench/proxy/up.sh` | 引擎配置/CDP 最小调用面/适配器接口形状（复用声明见 §6） |
|
||||
|
||||
## 3. 决策锁定（不再开放）
|
||||
|
||||
| # | 决策 | 值 |
|
||||
|---|---|---|
|
||||
| D3 | 端口 | gateway host `:8640`；反代前缀 `/bs-api`；scheduler `:8641`、proxymanager `:8642`、mihomo mixed `:17890` 仅 overlay |
|
||||
| D7 | 技术栈 | Go 单仓单二进制多角色（`go run ./cmd/browser-server -role=…`） |
|
||||
| D8 | 适配器形态 | scheduler 进程内 Go 模块（引擎经 overlay HTTP/CDP 调用；Dock 五方法 = 内部 interface） |
|
||||
| D9 | 上线 | mgr1 单节点起步，灰度扩展三节点另议 |
|
||||
| D10 | 联调网络 | stack 附加 `vlepontas-casa-net`（external，alias `browser-server`） |
|
||||
| D2 | 国外搜索 | Bing-only 姿态 |
|
||||
| O9/O10 | MCP 2025 腿 / 管理面 | 首版仅 2026 无状态 POST /mcp；管理面 CLI + SQL 先行（首版：CLI 子命令写 SQLite） |
|
||||
| 依赖 | 第三方库 | 禁重型依赖；准许 `modernc.org/sqlite`（纯 Go 无 CGO）、`gopkg.in/yaml.v3`；其余 stdlib + 手写（MCP JSON-RPC/CDP WebSocket 自实现，面窄）；网络依赖须经 Go proxy 镜像拉取 |
|
||||
|
||||
## 4. scope（写边界）
|
||||
|
||||
**in-scope（唯一写区）**：`server/`（新建，全部 Go 源码）、`stacks/browser-server.yml`、`scripts/deploy-mgr1.sh`、`scripts/smoke-local.sh`、`docs/impl-20260901.md`（交付记录）、`.dsh/contracts/**`、`.dsh/artifacts/run-20260901-browser-impl/**`。`docs/deploy-prod-preset-20260901.md` 仅允许追加「镜像 tag 约定（development-standards §6）」小节（O13 落地）。
|
||||
|
||||
**out-of-scope**：`docs/` 其余文档、`bench/`、`onesvm-dev-md/`（submodule）、`casa-commander/`（submodule）、`.dsh/casa-runtime/`、`README.md`、`AGENTS.md`、任何 remote/生产操作。**本仓不做 git commit**（除非用户另行指示）。
|
||||
|
||||
## 5. 逐模块 Acceptance（工程级硬门槛）
|
||||
|
||||
### A1 公共包（`server/internal/`）
|
||||
|
||||
- `contract/`：统一信封/JobEnvelope/RawResult/五方法接口/错误码——与 design-arch §3.2/§3.3、mcp-usage §2/§3 字段**一致**；时间字段一律 `+08:00`。
|
||||
- `store/`：SQLite（modernc 纯 Go）open（WAL/busy_timeout=2s/foreign_keys=on）+ migration（schema v1：consumers/api_keys/audit/quota/jobs/dead_letters/rules）+ repository 方法面。SQL 源码内 `?` 占位符、行级错误处理、无 `SELECT *` 糊装。
|
||||
- `auth/`：静态 key 校验 `hmac.Equal`（constant-time）；密钥注入 `${VAR:?required}` fail-closed；**禁 `os.Getenv` 默认值兜底**。
|
||||
- `policy/`：SSRF fail-closed（协议 80/443、私网/元数据 CIDR、CNAME 银行卡号段不重定向重验——**重定向重验按 design §5.4 每跳重验**）、域名规则 trie（direct/pool/deny）、robots（普通 key，TTL 缓存 SQLite）。
|
||||
- `safetyscan/`：词表文件 + 编译正则（NFKC 归一）+ PII redact + 注入包裹 delimiter；命中高危 block；`wordlist_version` 字段。
|
||||
- `httpx/`：共享 HTTP client（大小/类型守卫、超时、重定向重验 hook）。
|
||||
|
||||
### A2 gateway
|
||||
|
||||
- MCP JSON-RPC 2.0 单 handler（`tools/list`、`tools/call`，MCP 头校验/协议版本），HTTP `/v1/search` `/v1/read`，`/healthz` `/readyz`。
|
||||
- 认证 → scope → 429/402/403 → **overlay `POST scheduler:8641/enqueue`**（落盘成功才 200，scheduler 不可达 → 503+Retry-After，gateway 零落盘）。
|
||||
- 认证头仅 `X-Service-Token`（401 时响应头提示禁 Bearer）；admin 面：`X-Service-Token` = admin token，`POST /admin/keys`（签发）、`DELETE /admin/keys/{id}`（吊销）。
|
||||
- 搜索响应走 gateway 进程内短 TTL 缓存（query+region 键，默认 300s，≤50 条 LRU）——消化上游 CAPTCHA（plan-final §2.4 设计结论）。
|
||||
|
||||
### A3 scheduler
|
||||
|
||||
- 队列（SQLite 单写者 + 进程内 bounded queue）：`enqueue` 落 WAL 才 ACK；`ADMIT_MAX=60`（running+queued，env 可调）；429/503 携带 `Retry-After` + `running/queued` 现状。
|
||||
- 派发：`UPDATE … WHERE id=(SELECT …) RETURNING`；租约 `lease_until` + reaper 收割 + `attempts≤2` 仅瞬时错误重试 + 死信表。
|
||||
- Dock 适配器进程内注册表：searxng-cn / searxng-global / trafilatura / lightpanda / headless-shell 五个，各自实现 `contract.DockAdapter` 五方法；capabilities 如 §4.4 标签。
|
||||
- 能力路由：region/render 标签匹配（`region=overseas` 强制走 mihomo 出口；`shell_active ⇒ panda 停新` 互斥，入队侧执行）。
|
||||
- 模版层（模版层整合器，S7 归属）：RawResult → Size/Type guard → fit markdown（trafilatura 引擎侧已产 markdown，CDP 引擎给 text→go 差分算法转 markdown 或直接 bodyText 作正文）→ safetyscan → 信封封装；score 归一 0–1。
|
||||
- 降级链：C 满试 B、B 失败回 A（HTTP+fit），全败 `blocked/upstream` + warnings，不静默降质。
|
||||
- `/pressure` + `/metrics`（文本格式，design §7.2 指标面）。
|
||||
- headless-shell 按需槽：Swarm API（`DOCKER_HOST=tcp://…` 或 unix socket 挂载）scale 0→1，空闲 10min 回收（D8 预设；不可用时适配器返回 unhealthy，不阻塞其它通道）。
|
||||
|
||||
### A4 proxymanager
|
||||
|
||||
- 订阅解析：Clash YAML（复用 bench `lib.py` 逻辑的 Go 版）→ 剔除占位节点 → vless/hysteria2 分池（hy2 不调度）→ 写 mihomo provider 配置 + mihomo 热载。
|
||||
- 探活：`https://www.google.com/generate_204`（禁 cp.cloudflare.com HEAD）；活跃 30s / 全池 5min；连续 2 失败摘除、1 成功回候选；EWMA 延迟按域名分组。
|
||||
- 轮换 P2C + sticky session（TTL map）；域名路由表（direct/deny/pool）与 gateway policy 联动（共享 SQLite rules 表 + 热载）。
|
||||
- HTTP API（仅 overlay）：`/healthz`、`/api/proxies`（探活状态）、`/api/exit?domain=`（sticky 选出口）。
|
||||
- `PROXY_SUB_URLS` 多订阅容灾；订阅 URL 不落盘不入 git（内存传递给 mihomo config 生成）。
|
||||
|
||||
### A5 部署产物
|
||||
|
||||
- `stacks/browser-server.yml`：9 服务（gateway/scheduler/proxymanager/mihomo + searxng-cn/global + trafilatura + lightpanda + backup sidecar）+ shell 按需（`replicas=0`，Swarm 拉起）；deployment-rules §1.3 必备字段全（TZ/placement/healthcheck/limits）；镜像 digest/tag 锁定；`vlepontas-casa-net` external 附加 alias `browser-server`；SQLite named volume 钉 mgr1。
|
||||
- Dockerfile 多阶段（builder golang:1.27-alpine → distroless/base 或 alpine）；Makefile（build/test/lint/fmt/cross-amd64/image）；`scripts/deploy-mgr1.sh`（save|load + stack deploy --resolve-image never + prune dangling 尾步骤）与 `scripts/smoke-local.sh`（compose 冒烟：起本地 compose，签发 key，curl 全链路断言）。
|
||||
- 镜像 tag 约定落地：`onesvm/browser-server:dev` 日常复用（O13）。
|
||||
|
||||
### A6 质量门槛（verify/reverify 共用）
|
||||
|
||||
1. `go build ./...`、`go vet ./...`、`gofmt -l` 空、`go test ./...` **全绿**；关键路径单测覆盖：auth（比对常数时间/过期/吊销）、SSRF 私网判定、队列抢单原子性、词表命中、信封字段与 mcp-usage 一致性（golden test）、路由标签匹配、探活摘挂状态机。
|
||||
2. 行数：`server/` 与脚本每文件 **≤600 行**硬上限（组织 600 行纪律）；>600 须拆分后再验收。
|
||||
3. T 系复核：T1 契约（mock 形状=真实响应形状：searxng json/trafilatura/CDP 均以 bench 实测样本为准）、T2 fail-closed（SSRF/密钥/订阅缺省）、T3 密钥（`${VAR:?}` fail-closed + `hmac` 比对 + 无默认值）、T6（租约 reaper 退避）、T7 入参边界（Pydantic→Go 等价物：入参长度/范围校验）。
|
||||
4. 本机冒烟（smoke-local.sh，可选依赖 Docker 在线）：gateway/scheduler 起 + stub 引擎 → 签发 key → `/v1/search` `/v1/read` MCP+HTTP 全链路 200 + 错误路径（401/403/429/503）断言。
|
||||
5. amd64 交叉构建成功（`GOOS=linux GOARCH=amd64 go build`），镜像 ≤60MB 目标（distroless）。
|
||||
|
||||
## 6. 复用声明（组织纪律：能复用须标明）
|
||||
|
||||
| 复用物 | 来源 | 去向 |
|
||||
|---|---|--- A2 |
|
||||
| searxng 引擎配置（cn 四引擎 / global Bing-only 姿态） | `bench/searxng-{cn,global}/settings.yml` | stack 内嵌同参数 settings（按 Swarm 语法适配） |
|
||||
| trafilatura 服务（`POST /v1/read`、SSRF 守卫、semaphore=8） | `bench/trafilatura-http/app.py`（镜像已构建 `bench-s3b-trafilatura:local`） | stack 的 trafilatura 服务 = **同一代码构建**，镜像 tag `onesvm/trafilatura-http:dev` |
|
||||
| CDP 最小调用面（json/version → ws → Target.create/attach → Page/Runtime/Network enable → navigate → evaluate 提取） | `bench/site-matrix/cdp_fetch.mjs` | lightpanda/headless-shell 适配器 CDP 客户端（Go 重写，面窄） |
|
||||
| Clash 订阅解析/占位节点剔除/区域归类逻辑 | `bench/proxy/lib.py` | proxymanager 订阅解析（Go 移植） |
|
||||
| 反爬特征检测正则（cloudflare/waf/captcha 判定） | `bench/site-matrix/cdp_fetch.mjs detectVendor` | 模版层 blocked 判定 |
|
||||
| compose 参数（镜像 digest pin、shm、proxy env） | `bench/*/compose.yml` | stack yml 引擎服务 |
|
||||
|
||||
## 7. 工件流（artifact_io）
|
||||
|
||||
Worker 交付物一律落 `.dsh/artifacts/run-20260901-browser-impl/`：
|
||||
|
||||
| 产物 | 路径 |
|
||||
|---|---|
|
||||
| 实现回执（builder 自验收） | `impl-receipt.md` |
|
||||
| verify（首轮独立验收） | `verify.md` |
|
||||
| reverify（复核 + 迭代修复记录） | `reverify.md` |
|
||||
| 架构审查（core_framework 强制） | `arch-review.md` |
|
||||
| 冒烟原始输出 | `smoke/` |
|
||||
| 行数检查输出 | `wc.txt` |
|
||||
|
||||
## 7bis. 通信纪律（CASA 铁律）
|
||||
|
||||
- Worker 之间/Worker→指挥官：只传**产物路径 + diff 摘要**，禁「如上所述」传话。
|
||||
- Worker 不得最终自评 PASS——**builder ≠ judge**；verify/reverify 由独立只读 Worker 执行。
|
||||
- 指挥官主会话禁写业务文件（`server/`、`stacks/`、`scripts/`）；仅契约/工件/收口文档。
|
||||
|
||||
## 8. 失败与迭代
|
||||
|
||||
任一 acceptance 不满足 → verify Worker 出 `fail-*.md`（列证据 file:line）→ 指挥官派迭代 Worker **只注入 fail artifact + 本 Contract** 修复；最多 3 轮（policy.max_iterations=3），仍败则升级用户。
|
||||
|
||||
## 9. 部署边界（红线）
|
||||
|
||||
本轮**不 SSH mgr1、不 stack deploy、不 push remote**。`scripts/deploy-mgr1.sh` 只写脚本不执行。部署轮另走 🟡 前置确认（mgr1 free -g / ss -tln / 影响面报告）。
|
||||
|
||||
## 10. 时间与语言
|
||||
|
||||
- 全部时间字段 Asia/Shanghai `+08:00`（development-standards §7）；容器 `TZ=Asia/Shanghai`。
|
||||
- 代码注释/交付文档简体中文，标识符英文。
|
||||
70
.dsh/contracts/plan-20260901-01.md
Normal file
70
.dsh/contracts/plan-20260901-01.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
# Commander Plan: onesvm-browser-server 架构选型与实测评定
|
||||
|
||||
> 状态: done
|
||||
> 创建: 2026-09-01（Asia/Shanghai）
|
||||
> plan_id: plan-20260901-01
|
||||
> Plan Gate — 仅 `approved` / `in_progress` 后 Commander 可派发 Worker
|
||||
|
||||
## 目标
|
||||
|
||||
交付一份用户可 review 的最终方案：统一出口架构 + 无 Redis 高并发排队集群规格（dev-swarm 现实约束）+ 拓展坞协议 Worker 模版/整合器 + ProxyManager 合规代理 + 国内外各 ≥2 套开源方案的实测评定与入选结论。
|
||||
|
||||
## 背景与约束
|
||||
|
||||
- Norm: `onesvm-dev-md/base/development-standards.md`、`deployment-rules.md`、`devops-role-boundaries.md`；全局 service-secret-protocol（静态 key 禁 Bearer）
|
||||
- Fact: dev-swarm 3×~6.5GB RAM；既有 WSG（Tavily）先例 urlapi `/search-api`+`/mcp`；central-proxy `.53:7890`
|
||||
- constraints: 见 Contract（含 2026-09-01 补充：全栈总内存 ≤1GB 渐进递增，内存为选型决定性指标）；执行类一律 `subagent_cursor`（grok）；代理明文仅 `.local.md`
|
||||
|
||||
## 默认 model_family
|
||||
|
||||
hard（实测/调研）；architect（S4/S5 设计）；reviewer（S6/S7）
|
||||
|
||||
## 步骤表（Stage DAG）
|
||||
|
||||
| step_id | goal | worker | model_family | scope (in) | out_of_scope | acceptance | depends_on | status |
|
||||
|---------|------|--------|--------------|------------|--------------|------------|------------|--------|
|
||||
| S1 | 主流联网搜索服务架构调研：统一出口/注册key模式、AI原生输出格式（Firecrawl/Crawl4AI/Jina 等 markdown+schema 实践）、无Redis排队与浏览器集群管理（Browserless/steel 等） | subagent_cursor(explore) | hard | 写入仅 `.dsh/artifacts/run-20260901-browser-arch/research/` | 业务代码 | research/*.md 含可引用的架构要点与来源链接 | — | done |
|
||||
| S3a | 代理解析与探活：解析订阅（Clash/V2Ray），本机起客户端，验证 ≥3 个可用出口节点，测速与目标站连通性（google/bing/reddit/amazon），输出可用节点清单与路由建议 | subagent_cursor(shell) | simple | `.dsh/artifacts/run-20260901-browser-arch/proxy-probe.md` + `bench/proxy/**` | 订阅明文禁入 git 可提交文件 | proxy-probe.md 含实测连通数据 | — | done（52 vless 全活，最优 167ms；hy2 UDP 全灭弃用） |
|
||||
| S2 | 候选筛选 | subagent_cursor(explore) | hard | select.md + bench/plan.md | — | 入选/排除+模板指标 | S1 | done（5套入选） |
|
||||
| S3b | 国内组实测：本机 Docker 部署入选国内方案，跑 5 模板，记录质量/延迟/**idle·单会话·60burst 三档内存**/成功率（内存为决定性指标） | subagent_cursor(implement) | hard | `bench/**` + `.dsh/artifacts/.../bench/` | 不碰代理国外链路 | bench 产物含原始日志+汇总表（含三档内存实测） | S2 | done |
|
||||
| S3c | 国外组实测：同上但走 S3a 代理路由，含 JS 渲染页与反爬观察 | subagent_cursor(implement) | hard | `bench/**` + `.dsh/artifacts/.../bench/` | — | 同上 | S2, S3a | done |
|
||||
| S4 | 任务1–4 架构设计：统一出口+注册/key、拓展坞协议 Worker 模版+整合器、统一 AI 原生输出格式、无 Redis 排队、ProxyManager 合规双向拦截、有头插拔预留、dev-swarm 集群规格与扩展公式 | subagent_fork(implement) | architect | `docs/design-arch-20260901.md` | bench/ | 文档覆盖任务1–4 全部要点，含 mermaid | S1 | done |
|
||||
| S5 | 最终方案整合：S4 设计 + S3 实测结论 + 入选方案与部署规格 + MCP 使用规则草案 → 用户 review 版 | 主会话（指挥官整合） | architect | `docs/plan-final-20260901.md` | — | 覆盖任务 1–6，数据均可追溯 bench 产物 | S3b,S3c,S4 | done |
|
||||
| S6 | Verify + 独立 Reverify | subagent_fork(review) ×2 | reviewer | 只读 S5 + 全部产物 | — | review.md / reverify.md 无未解决 blocker | S5 | done（Verify PASS + Reverify FAIL→修订→复审 PASS；残留 R1/R2 已顺手修） |
|
||||
| S7 | Architecture Review（core_framework 强制） | subagent_fork(arch-review) | reviewer | 只读 S5 + norms | — | arch-review.md PASS | S6 | done（一轮 FAIL P1-AR1 → 修订 → 二轮 PASS） |
|
||||
|
||||
## 并行策略
|
||||
|
||||
- S1 ∥ S3a 同时启动
|
||||
- S4 在 S1 完成后启动，与 S2/S3 实测并行
|
||||
- S3b ∥ S3c 并行（本机资源约束：同时运行候选容器 ≤3，worker 须自纪律分批）
|
||||
- S5 等 S3b/S3c/S4 全部 done 后由主会话整合
|
||||
|
||||
## 全局 acceptance（Plan 级）
|
||||
|
||||
- [ ] Contract acceptance 全过（见 Contract）
|
||||
- [ ] 各 step status = done
|
||||
- [ ] `.dsh/artifacts/run-20260901-browser-arch/review.md` 与 `arch-review.md` 无未解决 blocker
|
||||
|
||||
## artifact 布局
|
||||
|
||||
```
|
||||
.dsh/artifacts/run-20260901-browser-arch/
|
||||
├── commander.log
|
||||
├── research/ # S1
|
||||
├── select.md # S2
|
||||
├── proxy-probe.md # S3a
|
||||
├── bench/ # S3b/S3c 汇总（原始在 bench/）
|
||||
├── review.md / reverify.md
|
||||
└── arch-review.md
|
||||
bench/ # 本机实测 compose/脚本/原始数据（可提交，禁含密钥）
|
||||
docs/ # 设计文档与最终方案（可提交）
|
||||
.dsh/casa-runtime/sessions/session-af595dc1-78f9-4dbd-93ef-6c21e3c1f912.json
|
||||
```
|
||||
|
||||
## 风险与回滚
|
||||
|
||||
- 代理节点不可用 → S3a 报告具体失败，S3c 降级为「仅直连可达国外源」实测 + 方案级评估，并向用户报告
|
||||
- 本机资源不足 → 分批实测，bench/plan.md 记录批次
|
||||
- 候选方案 Mac 不兼容（如 browser-swarm 仅限 Apple Silicon 反而有利；Camoufox 需 xvfb）→ 记录为实测结论之一
|
||||
- 回滚：仅文档与本机容器，无生产影响
|
||||
35
.dsh/contracts/plan-20260901-02.md
Normal file
35
.dsh/contracts/plan-20260901-02.md
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
# Commander Plan: 文档体系 + 部署预设 + 强对抗优化方案（第二轮）
|
||||
|
||||
> 状态: done
|
||||
> 创建: 2026-09-01（Asia/Shanghai）
|
||||
> plan_id: plan-20260901-02
|
||||
> run_id: run-20260901-browser-arch（沿用首轮实测数据）
|
||||
|
||||
## 目标
|
||||
|
||||
交付 5 项：T0 整合架构梳理文档；T1 MCP 使用文档（含能力数据范围）；T2 生产部署预设详情（供用户审批，不实际部署）；T3 打不过的 9 类站点优化方案研究；T4 Vlepontas 联调使用说明草稿。
|
||||
|
||||
## 背景与约束
|
||||
|
||||
- 数据基础：run-20260901-browser-arch 全部产物（bench 实测、site-matrix、三 PASS 验收）
|
||||
- 约束：本轮纯文档/调研，不部署、不改生产；技术栈选型给预设建议并标决策点；文档 ≤600 行/份
|
||||
|
||||
## 步骤表
|
||||
|
||||
| step_id | goal | worker | model_family | scope (in) | acceptance | depends_on | status |
|
||||
|---------|------|--------|--------------|------------|------------|------------|--------|
|
||||
| P1 | T3 强对抗站点优化方案研究：Amazon/Medium/Reddit/X/知乎/微博/百度百科/SO/YouTube 各站的可行获取路径（官方 API/RSSHub/移动端/住宅代理/Camoufox/第三方数据服务），按成本-质量-合规分级 | subagent_cursor(explore) | hard | 写 `.dsh/artifacts/run-20260901-browser-arch/research/06-blocked-sites.md` | 每站给出 ≥1 可行路径 + 成本 + 合规注记 + 来源链接 | — | done |
|
||||
| P2 | T0 整合架构梳理文档：整合哪些自部署服务/怎么整合（Dock 协议+模版层）/提供什么服务/并发排队机制 | subagent_fork(implement) | hard | 写 `docs/overview-integration-20260901.md` + `docs/README.md` 索引 | 覆盖用户四问，引用既有文档不重复造数 | — | done |
|
||||
| P3 | T1 MCP 使用文档：工具清单/参数/认证/错误码/合规使用规则/能力数据范围（引擎矩阵+站点矩阵+性能档位） | subagent_fork(implement) | hard | 写 `docs/mcp-usage-20260901.md` | 智能体读完可正确接入；能力范围与实测一致 | — | done |
|
||||
| P4 | T2 生产部署预设：服务清单（控制层/worker）、技术栈、镜像、资源、节点布局、端口、健康检查、部署流程 | 主会话 | architect | 写 `docs/deploy-prod-preset-20260901.md` | 与档位表/布局一致；标出待用户拍板项 | — | done |
|
||||
| P5 | T4 Vlepontas 联调使用说明草稿（注册/拿 key/接 MCP/验收用例/反馈回路） | 主会话 | hard | 写 `docs/integration-vlepontas-20260901.md` | 批准部署后可直接发对方 | P4 | done |
|
||||
| P6 | 四份文档独立验收 | subagent_fork(review) | reviewer | 只读 | review 无 blocker | P1–P5 | done（PASS；F1-F5 已修） |
|
||||
|
||||
## 并行策略
|
||||
|
||||
P1 ∥ P2 ∥ P3 同时启动；P4/P5 主会话穿插；P6 收尾。
|
||||
|
||||
## 风险
|
||||
|
||||
- T3 部分站点可能无合规可行路径 → 如实写「无可行路径/需住宅代理成本项」
|
||||
- 技术栈预设（Go vs Python）用户可能有偏好 → 标决策点
|
||||
63
.dsh/contracts/plan-20260901-03.md
Normal file
63
.dsh/contracts/plan-20260901-03.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Commander Plan: 控制面 Go 实现（contract-20260901-browser-impl）
|
||||
|
||||
> 状态: done
|
||||
> 创建: 2026-09-01（Asia/Shanghai）· 收口: 2026-09-02
|
||||
> plan_id: plan-20260901-03
|
||||
> run_id: run-20260901-browser-impl
|
||||
> contract: contract-20260901-browser-impl.md
|
||||
> 验收链: W1→W2∥W3∥W4→W5 → ITER-1（冒烟三根因）→ ITER-2（冷启动窗口）→ V1 verify PASS-with-issues → ITER-3（P1 配额结算+三接线）→ V2 reverify PASS → V3 arch-review PASS-with-debt → ITER-4（D-1/D-2 必修债就地修复）。产物：`.dsh/artifacts/run-20260901-browser-impl/{impl-receipt-w1..w5, verify.md, reverify.md, arch-review.md, wc.txt, smoke/smoke-run.log, fail-*.md}`。
|
||||
|
||||
## 目标
|
||||
|
||||
单二进制三角色（gateway/scheduler/proxymanager）+ 五 Dock 适配器 + SQLite 队列 + stack 与部署产物，达到「部署 mgr1 测试」就绪态（详见 Contract §5 A1–A6）。
|
||||
|
||||
## 背景与约束
|
||||
|
||||
- 权威设计：docs/design-arch + plan-final + mcp-usage（契约面）+ deploy-prod-preset（端口/资源/灰度）
|
||||
- 组织纪律：600 行/文件上限、密钥 fail-closed、urlapi、东八区、镜像 save/load
|
||||
- 模型：Worker 用 `impl-glm-5.3-flash`（ZCode 实现型 subagent，含 Bash/文件全工具）；验收用独立 `Explore`（只读）+ 主会话复核
|
||||
|
||||
## 模块化拆解（防上帝文件）
|
||||
|
||||
```
|
||||
server/
|
||||
go.mod # module onesvm.com/onesvm/browser-server
|
||||
cmd/browser-server/main.go # role 分流入口（gateway|scheduler|proxymanager）
|
||||
internal/contract/ # 信封/Job/RawResult/DockAdapter 接口/错误码（A1）
|
||||
internal/store/ # SQLite open+migrations+repos（A1）
|
||||
internal/auth/ # key 校验/签发（A1+A2 admin）
|
||||
internal/policy/ # SSRF/域名规则/robots（A1）
|
||||
internal/safetyscan/ # 词表/PII redact/注入包裹（A1）
|
||||
internal/httpx/ # 共享 client + Size/Type guard（A1）
|
||||
internal/gateway/ # HTTP handler + MCP JSON-RPC + enqueue 转发（A2）
|
||||
internal/scheduler/ # 队列/派发/reaper/路由/模版层/pressure（A3）
|
||||
internal/dock/ # 五适配器：searxng.go trafilatura.go cdp.go（lightpanda+shell 共用）cdp_client.go（A3/D8）
|
||||
internal/proxymanager/ # 订阅/探活/P2C/路由 API（A4）
|
||||
internal/config/ # env 解析（fail-closed 辅助）
|
||||
stacks/browser-server.yml
|
||||
scripts/deploy-mgr1.sh scripts/smoke-local.sh
|
||||
```
|
||||
|
||||
## 步骤表
|
||||
|
||||
| step_id | goal | worker | scope (write) | acceptance | depends_on | status |
|
||||
|---------|------|--------|--------------|------------|------------|--------|
|
||||
| W1 | 骨架门：go.mod+目录+contract/store/config 全量实现+auth/policy/safetyscan/httpx 全量+单测 | impl-glm-5.3-flash | `server/` 全部（本步建立骨架与公共包） | A1 全项 + A6.1/6.2 + go test 全绿 | — | todo |
|
||||
| W2 | gateway：MCP handler + HTTP v1 + admin key 面 + enqueue 转发 + 缓存 | impl-glm-5.3-flash | `server/internal/gateway/` `server/cmd/` | A2 全项 + A6 golden 信封测试 | W1 | todo |
|
||||
| W3 | scheduler：队列/派发/reaper/路由/模版层/pressure + dock 五适配器 | impl-glm-5.3-flash | `server/internal/scheduler/` `server/internal/dock/` | A3 全项 + A6.1/6.2 | W1 | todo |
|
||||
| W4 | proxymanager：订阅解析/探活/P2C/路由 API | impl-glm-5.3-flash | `server/internal/proxymanager/` | A4 全项 | W1 | todo |
|
||||
| W5 | stack/Dockerfile/Makefile/脚本 + 本机冒烟 + amd64 构建 | impl-glm-5.3-flash | `stacks/` `scripts/` `server/Dockerfile` `Makefile` | A5 全项 + A6.4/6.5 | W2,W3,W4 | todo |
|
||||
| V1 | 独立验收 verify（只读）：跑 A6 全部门槛，产 verify.md / fail-*.md | Explore（readonly） | 写 `.dsh/artifacts/run-20260901-browser-impl/verify.md` | A6 六项逐一核对 + T1–T7 复核 | W5 | todo |
|
||||
| V2 | reverify（builder 修复后复核，只读）+ 行数检查落 wc.txt | Explore（readonly） | 写 `reverify.md` `wc.txt` | 修复项复验 + 全绿 | V1（若有 fail） | todo |
|
||||
| V3 | 架构审查（core_framework 强制）：解耦/行数/契约对齐/门闩矩阵 | 主会话指挥（read-only 审查，独立 reviewer subagent） | 写 `arch-review.md` | 无 P0/P1 | V2 | todo |
|
||||
|
||||
## 并行策略
|
||||
|
||||
W1 先行（串行）；W2 ∥ W3 ∥ W4 并行派发；W5 收口；V1→V2→V3 验收链。
|
||||
|
||||
## 风险
|
||||
|
||||
- modernc.org/sqlite 拉取需网络（Go proxy 镜像）；失败则降级 CGO-free 备选或升级用户
|
||||
- 冒烟依赖本机 Docker 与缓存镜像（searxng/trafilatura 已缓存）；境外引擎冒烟可能空结果——用 stub 断言形状即可
|
||||
- 五适配器 + 三角色工程量集中在一个 Worker 会话可能超长 → W1 公共包先行落稳，W2–W4 各自独立目录，边界清晰可各自验收
|
||||
- headless-shell Swarm 按需拉起依赖 socket 权限——冒烟仅验证「不可用时降级」路径，Swarm 拉起留部署轮实机验证
|
||||
39
.dsh/contracts/plan-20260901-04.md
Normal file
39
.dsh/contracts/plan-20260901-04.md
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# Commander Plan: W6 热修 — 订阅 bootstrap 经 central-proxy
|
||||
|
||||
> 状态: done
|
||||
> 创建: 2026-09-02（Asia/Shanghai）
|
||||
> plan_id: plan-20260901-04
|
||||
> run_id: run-20260901-browser-impl
|
||||
> 用户拍板: 2026-09-02 AskQuestion — 路线 A（SUB-1 经 .53:7890）+ 立即更新 mgr1
|
||||
|
||||
## 目标
|
||||
|
||||
mgr1 上 ProxyManager 能拉到机场订阅并探活出 alive>0，海外通道不再因「订阅直连被墙」fail-closed。写 W6 回执。
|
||||
|
||||
## 背景与约束
|
||||
|
||||
- Fact：W6 stack 已在 mgr1（10 服务，9 healthy）；PM 订阅超时/拉取异常 → alive=0
|
||||
- Fact：mgr1 直连订阅域 8s 超时；mgr1→192.168.1.53:7890 github=200
|
||||
- 代码已有 `BROWSER_SERVER_SUB_FETCH_PROXY`（subscription.go）；stack / deploy.env 未接线
|
||||
- 🟡 用户已确认立即 stack 更新；回滚 `docker stack rm browser-server`（卷保留）
|
||||
- 指挥官 fallback：改动面窄（接线+重建+验收），不派 subagent
|
||||
|
||||
## 步骤表
|
||||
|
||||
| step_id | goal | status |
|
||||
|---------|------|--------|
|
||||
| W6a | 接线 stack / .env.example / deploy.env + 单测 | done |
|
||||
| W6b | 重建 amd64 镜像 → save/load → stack deploy | done |
|
||||
| W6c | 验收 PM 订阅成功 + alive>0 + 写回执 | done |
|
||||
|
||||
## 全局 acceptance
|
||||
|
||||
- [x] `BROWSER_SERVER_SUB_FETCH_PROXY` 注入 proxymanager（缺省空=直连，mgr1 显式 `.53:7890`）
|
||||
- [x] `go test ./internal/proxymanager/` 绿
|
||||
- [x] mgr1 PM 日志出现订阅刷新成功或探活 `alive>0`（脱敏，不记 URL）
|
||||
- [x] W6 回执落 `.dsh/artifacts/run-20260901-browser-impl/deploy/w6-receipt.md`
|
||||
|
||||
## 风险与回滚
|
||||
|
||||
- overlay 容器到 `.53:7890` 若不通：回滚 env 为空并 `stack deploy`，海外仍断但不伤国内
|
||||
- 镜像更新只动 `onesvm/browser-server:dev`，不重传第三方
|
||||
11
.gitignore
vendored
11
.gitignore
vendored
|
|
@ -5,7 +5,7 @@
|
|||
.LSOverride
|
||||
|
||||
# Icon must end with two \r
|
||||
Icon
|
||||
Icon
|
||||
|
||||
# Thumbnails
|
||||
._*
|
||||
|
|
@ -190,3 +190,12 @@ cython_debug/
|
|||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
|
||||
.dsh/casa-runtime/
|
||||
.dsh/artifacts/
|
||||
casa-mode.active
|
||||
casa-commander.active
|
||||
deploy.env
|
||||
|
||||
# 本机 / 交叉编译产物（部署走 docker image，不入库）
|
||||
server/browser-server
|
||||
/bin/
|
||||
|
|
|
|||
6
.gitmodules
vendored
Normal file
6
.gitmodules
vendored
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
[submodule "onesvm-dev-md"]
|
||||
path = onesvm-dev-md
|
||||
url = https://git.onesvm.com/1svm/onesvm-dev-md.git
|
||||
[submodule "casa-commander"]
|
||||
path = casa-commander
|
||||
url = https://git.onesvm.com/surmus/casa-commander.git
|
||||
52
AGENTS.md
Normal file
52
AGENTS.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# AGENTS — CASA 指挥官工作区（DSH）
|
||||
|
||||
> DSH 会自动加载本文件为基线指令。会话模型由部署决定，不在本文件指定。
|
||||
|
||||
## 哲学
|
||||
|
||||
让 LLM 做判断，让系统保证秩序。Agent 之间只传**产物**，不传对话。
|
||||
|
||||
## 角色
|
||||
|
||||
### dialogue（主会话）
|
||||
|
||||
- 采集需求 → Contract（含 `complexity` / 档位）→ approved
|
||||
- 禁止跳过 Contract 大改；禁止口头覆盖 acceptance
|
||||
|
||||
### orchestrator（指挥官）
|
||||
|
||||
- Compile + Normalize → 委派 Worker（subagent / subagent_fork）
|
||||
- 编排轮不写大段业务实现；不吞产物全文编拓扑
|
||||
- `core_framework`：验收后主持 Architecture Review（只读审查）
|
||||
|
||||
### worker-explore / worker-shell / worker-implement
|
||||
|
||||
- brief 含 Goal / Read / Write / Scope / Acceptance / Mode
|
||||
- 只在 Contract.scope 与 artifact_io 内工作
|
||||
|
||||
### evaluator-review
|
||||
|
||||
- readonly；builder ≠ judge
|
||||
|
||||
## 默认 Stage
|
||||
|
||||
```
|
||||
explore? → implement → verify → review → arch_review?
|
||||
```
|
||||
|
||||
## Agent Loop
|
||||
|
||||
`PLAN → EXECUTE → VERIFY → REVERIFY → [ARCH_REVIEW] → DONE`
|
||||
失败 ITERATE：只注入 `fail-*.md`（+ Contract）。
|
||||
|
||||
## 工作区约定
|
||||
|
||||
- 契约/产物：`.dsh/contracts/` 与 `.dsh/artifacts/`（不提交）
|
||||
- 会话状态：`.dsh/casa-runtime/`（不提交；agent 自律维护）
|
||||
- 项目配置：`.dsh/casa.yml`；项目技能：`.dsh/skills/`
|
||||
- 组织知识库：`onesvm-dev-md/`（`knowledge.mode: off`，直接按入口读）
|
||||
- 组织规范入口：`onesvm-dev-md/base/development-standards.md`、`deployment-rules.md`、`devops-role-boundaries.md`
|
||||
- 经验库：`onesvm-dev-md/experience/`（`projects/*` 与 `general/lessons-learned.md`）
|
||||
- 凭据：`onesvm-dev-md/base/credentials.md` 只读，不修改、不编造 IP
|
||||
- 绝不 `git push` / force-push；未授权不 commit
|
||||
- CASA 模式细节：加载 `casa-commander` 技能
|
||||
41
Makefile
Normal file
41
Makefile
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Makefile — onesvm-browser-server 仓根（Contract §5-A5）。
|
||||
# 目标覆盖：fmt / vet / test / build-amd64 / image / image-trafilatura / smoke / wc。
|
||||
# 密钥不入本文件（部署 env 走 gitignore 的 deploy.env，见 scripts/deploy-mgr1.sh）。
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
# 生产 amd64（组织部署规范：Mac arm64 构建必须显式 --platform linux/amd64）。
|
||||
PLATFORM ?= linux/amd64
|
||||
|
||||
# 冒烟镜像 tag（本机 arm64；不与生产 :dev 混用）。
|
||||
SMOKE_IMAGE ?= onesvm/browser-server:smoke
|
||||
|
||||
.PHONY: help fmt vet test build-amd64 image image-trafilatura smoke wc
|
||||
|
||||
help: ## 显示本帮助
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-18s\033[0m %s\n", $$1, $$2}'
|
||||
|
||||
fmt: ## gofmt 格式检查（server/；输出为空即通过）
|
||||
cd server && gofmt -l .
|
||||
|
||||
vet: ## go vet 静态检查
|
||||
cd server && go vet ./...
|
||||
|
||||
test: ## 单测全量
|
||||
cd server && go test ./...
|
||||
|
||||
build-amd64: ## amd64 交叉构建产出 bin/（部署轮 save|load 用本机镜像，此 target 供快速验证）
|
||||
mkdir -p bin
|
||||
cd server && GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w" -o ../bin/browser-server-linux-amd64 ./cmd/browser-server
|
||||
|
||||
image: ## 构建生产镜像 onesvm/browser-server:dev（默认 amd64；本机冒烟可 PLATFORM=linux/arm64）
|
||||
docker build --platform $(PLATFORM) -t onesvm/browser-server:dev -f server/Dockerfile .
|
||||
|
||||
image-trafilatura: ## 构建 trafilatura 引擎镜像 onesvm/trafilatura-http:dev（amd64）
|
||||
docker build --platform $(PLATFORM) -t onesvm/trafilatura-http:dev stacks/trafilatura
|
||||
|
||||
smoke: ## 本机 compose 冒烟（起 stub 引擎 + 三角色，12 步断言；证据落 .dsh/artifacts）
|
||||
bash scripts/smoke-local.sh
|
||||
|
||||
wc: ## 源文件行数检查（server *.go 按行数倒序，>600 即违规）
|
||||
find server -name '*.go' | xargs wc -l | sort -rn | head -20
|
||||
126
README.md
126
README.md
|
|
@ -1,3 +1,127 @@
|
|||
# onesvm-browser-server
|
||||
|
||||
集成多种类型浏览器MCP自构建联网搜索服务
|
||||
面向智能体的**自构建联网搜索服务**：消费者拿一把 `X-Service-Token`，用 MCP（主）或 HTTP（兜底）调用 `search` / `read`。国内直连，国外走自有代理池，query **不经过** Tavily / Jina 等境外 SaaS。
|
||||
|
||||
现网入口：`http://192.168.1.51:8640`（MCP `POST /mcp`，HTTP `POST /v1/search` `/v1/read`）。
|
||||
|
||||
| 你是谁 | 先读 |
|
||||
|---|---|
|
||||
| 消费方（Vlepontas / EAI） | [`docs/mcp-usage-20260901.md`](docs/mcp-usage-20260901.md) → [`docs/integration-vlepontas-20260901.md`](docs/integration-vlepontas-20260901.md) |
|
||||
| 接手开发 | [`docs/dev-handoff-20260902.md`](docs/dev-handoff-20260902.md)（本文下面是设计与选型摘要） |
|
||||
| 改架构 / 拍板 | [`docs/plan-final-20260901.md`](docs/plan-final-20260901.md)（决策权威）→ [`docs/design-arch-20260901.md`](docs/design-arch-20260901.md)（机制权威） |
|
||||
|
||||
---
|
||||
|
||||
## 1. 要解决什么问题
|
||||
|
||||
智能体需要「上网查公开页」。买境外 SaaS 会把 query 送出境，且按次计费；自建浏览器集群又容易把整机内存和出口 IP 打爆。本仓的约束是：
|
||||
|
||||
- **境内闭环**：搜索与精读默认不经境外托管 API。
|
||||
- **内存纪律**：常驻业务峰 ≤1GB（L2）；60 = 队列深度，不是 60 个浏览器。
|
||||
- **零 Redis**：SQLite WAL 单写者排队。
|
||||
- **诚实能力**：公开网页大约九成；登录墙 / 强 WAF / Google 搜索不承诺。住宅 IP 是另一条带成本声明的通道，本服务不做。
|
||||
|
||||
人交「事件怎么活、怎么死、多快死」；实现按 [`onesvm-dev-md/base/engineering-standards.md`](../onesvm-dev-md/base/engineering-standards.md) 事件表，缺格不补默认值开工。
|
||||
|
||||
---
|
||||
|
||||
## 2. 设计科学（为什么长这样）
|
||||
|
||||
### 2.1 统一出口，双形态同一内核
|
||||
|
||||
所有消费者注册主体、拿可吊销 key。MCP 与 HTTP 共用 **Auth → Policy → Queue → Engine**。错误码两面对齐。静态服务密钥只走 `X-Service-Token`，禁止塞进 `Authorization: Bearer`。
|
||||
|
||||
### 2.2 分层内核，而不是「一个大浏览器」
|
||||
|
||||
流量大约 90% 是搜索 + 纯 HTTP 精读，不该进 Chromium。
|
||||
|
||||
| 层 | 适配器 | 何时用 |
|
||||
|---|---|---|
|
||||
| A 无浏览器 | SearXNG-CN / SearXNG-Global / Trafilatura | 搜索发现、静态正文 |
|
||||
| B 轻渲染 | Lightpanda | HTTP 抽空、需要一点 JS |
|
||||
| C 保真 | chrome-headless-shell（按需槽，默认 0 副本） | 轻核不够时 |
|
||||
| 预留 | Camoufox / 有头档 | 强对抗 + 住宅 IP，不在 1GB 预算内 |
|
||||
|
||||
降级链：C 满 → B → A → `blocked`/`upstream`。Cloudflare / WAF / 验证页**不拿渲染空转**，直接 `blocked`。
|
||||
|
||||
### 2.3 出口与身份分开记账
|
||||
|
||||
- **出口**：国内域直连；境外经 ProxyManager + mihomo vless 热池。决定「Google 过不过」的是 IP 信誉，不是 UA 字符串。
|
||||
- **身份（P0/P1，2026-09-02 落地）**：每生产节点 8 套 Chrome 136 模版（与 Trafilatura `curl_cffi` `impersonate=chrome136` 自洽）；一等 Cookie 养在本节点 SQLite。罐按 **模版 × http|cdp × 出口 × eTLD+1** 隔离，节点内共享、不按消费者拆。登录态 Cookie 名直接丢弃。403/验证页整域丢罐。Cookie **不进 MCP、不进 jobs.payload**。
|
||||
|
||||
Go 侧 `httpx` 仍是 Go TLS，UA 写诚实的 `onesvm-browser-server-httpx/1.0`，禁止再伪装 Chrome（TLS/UA 必须自洽）。
|
||||
|
||||
### 2.4 合规双向保险
|
||||
|
||||
请求侧在网关：SSRF、仅 80/443、域名策略、普通 key 遵守 robots。响应侧在 worker 模版层：体积/类型白名单、词表、PII 脱敏、注入包裹。命中即 `denied` 并审计，不假装成功。
|
||||
|
||||
---
|
||||
|
||||
## 3. 选用方案（实测后入选，不是目录堆砌）
|
||||
|
||||
2026-09-01 本机 Docker 评分（内存 30% / 质量 25% / 成功率 20% / 延迟 15% / 运维 10%）。全文：[`docs/decision-candidates-20260901.md`](docs/decision-candidates-20260901.md)。
|
||||
|
||||
| 方案 | 分 | 角色 | 为什么留 |
|
||||
|---|---:|---|---|
|
||||
| Trafilatura-HTTP | 96.8 | 默认精读 | 60/60、p50 ~60ms、正文干净；现网已加 curl_cffi |
|
||||
| Lightpanda | 91.0 | 默认 JS | idle 3.6MB；CF/Amazon 仍过不了 |
|
||||
| chrome-headless-shell | 85.5 | 保真按需 | 单槽 FIFO 零拒绝；重页 ~400MB |
|
||||
| SearXNG-CN | 71.5 | 国内搜索 | 政策词准；突发时百度/搜狗 CAPTCHA，靠排队钳 4–8 |
|
||||
| SearXNG-Global | 62.0 | 国外搜索 | 实例稳，但数据中心 IP 下 **仅 Bing**；产品不承诺多源 |
|
||||
|
||||
**明确排除**：playwright-distributed（Redis）、Browserless（4GB 档）、Jina 云（query 出境）、Firecrawl 自托管（与 shell+模版层重叠且更重）。
|
||||
|
||||
**用户已拍板**：D1 先 L0→L2（≤1GB）；D2 接受 Bing-only 起步；D5 不做 Camoufox；Amazon 本站非本服务刚需。
|
||||
|
||||
现网能力以 [`docs/mcp-usage-20260901.md`](docs/mcp-usage-20260901.md) §4 为准（2026-09-02 mgr1 复测），不要把 09-01 bench 的「BBC 稳取 / Amazon 一律不行」抄进消费方文档。
|
||||
|
||||
---
|
||||
|
||||
## 4. 仓库地图
|
||||
|
||||
```
|
||||
server/ Go 单二进制，-role=gateway|scheduler|proxymanager
|
||||
cmd/browser-server/ 入口
|
||||
internal/contract/ 信封与 Job（Session 字段 json:"-"）
|
||||
internal/gateway/ MCP + /v1 + Auth + 策略预检
|
||||
internal/scheduler/ 队列、路由、模版层整合器、junk 过滤、Cookie 绑定
|
||||
internal/dock/ 五适配器（SearXNG×2、Trafilatura、Lightpanda、shell）
|
||||
internal/fingerprint/ 每节点 Chrome 136 模版
|
||||
internal/session/ Cookie 罐（Sanitize / TTL / reap）
|
||||
internal/store/ SQLite（jobs、keys、fp_*、session_cookies）
|
||||
internal/proxymanager/ 订阅 / 探活 / 域名路由
|
||||
internal/policy/ SSRF / robots / 域名
|
||||
internal/safetyscan/ 响应侧词表与包裹
|
||||
stacks/ Swarm stack + SearXNG 配置 + Trafilatura 镜像
|
||||
scripts/ smoke-local.sh、deploy-mgr1.sh（默认 dry-run）
|
||||
bench/ 选型期实测（无密钥）
|
||||
docs/ 决策 / 机制 / MCP / 接手
|
||||
```
|
||||
|
||||
源文件硬上限 **600 行**（`make wc`）。禁止万能 `util` 包。
|
||||
|
||||
---
|
||||
|
||||
## 5. 本机怎么跑
|
||||
|
||||
```bash
|
||||
make test # go test（合入前加 -race：cd server && go test -race ./...）
|
||||
make vet
|
||||
make smoke # compose + stub 引擎，12 步断言
|
||||
make image # onesvm/browser-server:dev（默认 linux/amd64）
|
||||
make image-trafilatura # onesvm/trafilatura-http:dev
|
||||
```
|
||||
|
||||
生产节点通常拉不到 Docker Hub。镜像必须在可联网机器构建/拉取，再 `docker save | ssh … docker load`，stack 用 `--resolve-image never`。**同 tag 覆盖后必须 `docker service update --force`**，否则任务还跑旧层。
|
||||
|
||||
mgr1 部署属 🟡：只在用户明确「确认执行」后跑 `bash scripts/deploy-mgr1.sh --apply`。密钥在 gitignore 的 `deploy.env`，不入 git。
|
||||
|
||||
---
|
||||
|
||||
## 6. 文档索引
|
||||
|
||||
完整表见 [`docs/README.md`](docs/README.md)。冲突时：
|
||||
|
||||
1. [`docs/plan-final-20260901.md`](docs/plan-final-20260901.md) — 决策
|
||||
2. [`docs/design-arch-20260901.md`](docs/design-arch-20260901.md) — 机制
|
||||
3. 代码 — 现网行为（文档会过期，以代码与 §4 现网复测为准）
|
||||
|
|
|
|||
122
bench/chrome-headless-shell/_lib.sh
Normal file
122
bench/chrome-headless-shell/_lib.sh
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
# shared helpers for S3c bench (no secrets)
|
||||
set -euo pipefail
|
||||
|
||||
log() { printf '[%s] %s\n' "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$*"; }
|
||||
|
||||
parse_mem_mib() {
|
||||
python3 - "$1" <<'PY'
|
||||
import re, sys
|
||||
s = sys.argv[1].split("/")[0].strip()
|
||||
m = re.match(r"([0-9.]+)\s*([A-Za-z]+)", s)
|
||||
if not m:
|
||||
print("0")
|
||||
raise SystemExit
|
||||
v, u = float(m.group(1)), m.group(2).lower()
|
||||
mult = {"b":1,"kib":1024,"kb":1000,"mib":1024**2,"mb":1000**2,"gib":1024**3,"gb":1000**3}.get(u)
|
||||
if not mult:
|
||||
print("0"); raise SystemExit
|
||||
print(f"{v * mult / (1024**2):.3f}")
|
||||
PY
|
||||
}
|
||||
|
||||
cgroup_mib() {
|
||||
local c="$1" raw=""
|
||||
raw="$(docker exec "$c" sh -c 'cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null' 2>/dev/null || true)"
|
||||
if [[ -z "$raw" ]]; then
|
||||
echo ""
|
||||
return
|
||||
fi
|
||||
python3 -c "print(f'{int(\"$raw\")/(1024*1024):.3f}')"
|
||||
}
|
||||
|
||||
sample_stats() {
|
||||
local out="$1" interval="$2" duration="$3"
|
||||
shift 3
|
||||
mkdir -p "$(dirname "$out")"
|
||||
echo -e "ts_iso\tname\tmem_usage\tcpu_pct\tmem_mib\tcgroup_mib" > "$out"
|
||||
local end=$(( $(date +%s) + duration ))
|
||||
while [ "$(date +%s)" -lt "$end" ]; do
|
||||
local ts
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
local name mem cpu
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
local mib cg
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
}
|
||||
|
||||
sample_until_file() {
|
||||
local out="$1" interval="$2" sentinel="$3" extra_s="$4"
|
||||
shift 4
|
||||
mkdir -p "$(dirname "$out")"
|
||||
echo -e "ts_iso\tname\tmem_usage\tcpu_pct\tmem_mib\tcgroup_mib" > "$out"
|
||||
while [[ ! -f "$sentinel" ]]; do
|
||||
local ts name mem cpu mib cg
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
local end=$(( $(date +%s) + extra_s ))
|
||||
while [ "$(date +%s)" -lt "$end" ]; do
|
||||
local ts name mem cpu mib cg
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
}
|
||||
|
||||
ensure_image() {
|
||||
local img="$1" note="${2:-}"
|
||||
local platform="linux/arm64"
|
||||
local how="local"
|
||||
if docker image inspect "$img" >/dev/null 2>&1; then
|
||||
local arch
|
||||
arch="$(docker image inspect --format '{{.Architecture}}' "$img")"
|
||||
if [[ "$arch" == "arm64" || "$arch" == "arm64/v8" ]]; then
|
||||
log "image present $img arch=$arch"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
return 0
|
||||
fi
|
||||
log "image $img arch=$arch not arm64; re-pull"
|
||||
fi
|
||||
log "docker pull --platform $platform $img"
|
||||
if docker pull --platform "$platform" "$img"; then
|
||||
how="hub"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
return 0
|
||||
fi
|
||||
local short="${img#docker.io/}"
|
||||
local mirror="docker.m.daocloud.io/${short}"
|
||||
log "primary pull failed; retry $mirror"
|
||||
docker pull --platform "$platform" "$mirror"
|
||||
docker tag "$mirror" "$img"
|
||||
how="daocloud_mirror"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
}
|
||||
|
||||
wait_http() {
|
||||
local url="$1" tries="${2:-40}"
|
||||
local i
|
||||
for i in $(seq 1 "$tries"); do
|
||||
if curl -sf --max-time 2 "$url" >/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
358
bench/chrome-headless-shell/cdp_fetch.mjs
Normal file
358
bench/chrome-headless-shell/cdp_fetch.mjs
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Minimal CDP navigate+extract via native WebSocket (Node 22+).
|
||||
* Usage: node cdp_fetch.mjs <cdp_http> <url> [nav_timeout_s]
|
||||
* Prints one JSON object to stdout. No secrets.
|
||||
*/
|
||||
const cdpHttp = process.argv[2] || "http://127.0.0.1:19222";
|
||||
const targetUrl = process.argv[3];
|
||||
const navTimeoutS = Number(process.argv[4] || 25);
|
||||
if (!targetUrl) {
|
||||
console.error("usage: node cdp_fetch.mjs <cdp_http> <url> [nav_timeout_s]");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const UA =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.109 Safari/537.36";
|
||||
|
||||
const EXTRACT_JS = `(() => {
|
||||
const title = document.title || "";
|
||||
const bodyText = (document.body && document.body.innerText) ? document.body.innerText : "";
|
||||
const html = document.documentElement ? document.documentElement.outerHTML : "";
|
||||
let price = "";
|
||||
const meta = document.querySelector('meta[itemprop="price"], meta[property="product:price:amount"]');
|
||||
if (meta) price = (meta.getAttribute("content") || "").trim();
|
||||
if (!price) {
|
||||
const el = document.querySelector('[class*="price"], [data-testid*="price"], .product-price, #priceblock_ourprice, .a-price .a-offscreen');
|
||||
if (el) price = (el.textContent || "").trim().slice(0, 80);
|
||||
}
|
||||
let jsonldPrice = "";
|
||||
for (const s of document.querySelectorAll('script[type="application/ld+json"]')) {
|
||||
try {
|
||||
const j = JSON.parse(s.textContent || "null");
|
||||
const arr = Array.isArray(j) ? j : [j];
|
||||
const walk = (o) => {
|
||||
if (!o || typeof o !== "object") return;
|
||||
if (o.offers) {
|
||||
const off = Array.isArray(o.offers) ? o.offers[0] : o.offers;
|
||||
if (off && (off.price || off.lowPrice)) jsonldPrice = String(off.price || off.lowPrice);
|
||||
}
|
||||
if (Array.isArray(o["@graph"])) o["@graph"].forEach(walk);
|
||||
};
|
||||
arr.forEach(walk);
|
||||
} catch (e) {}
|
||||
}
|
||||
const quoteCount = document.querySelectorAll(".quote").length;
|
||||
const lc = (title + "\\n" + bodyText).toLowerCase();
|
||||
return {
|
||||
title,
|
||||
text: bodyText.slice(0, 8000),
|
||||
htmlLen: html.length,
|
||||
textLen: bodyText.length,
|
||||
price,
|
||||
jsonldPrice,
|
||||
quoteCount,
|
||||
htmlHead: html.slice(0, 1800),
|
||||
finalUrl: location.href,
|
||||
readyState: document.readyState,
|
||||
looksBlocked: /(robot|captcha|sorry|click the button|continue shopping|just a moment|attention required|verify you are human|are you a human)/i.test(lc),
|
||||
};
|
||||
})()`;
|
||||
|
||||
function rewriteWs(wsUrl, httpEndpoint) {
|
||||
const u = new URL(wsUrl);
|
||||
const h = new URL(httpEndpoint);
|
||||
u.protocol = h.protocol === "https:" ? "wss:" : "ws:";
|
||||
u.hostname = h.hostname;
|
||||
u.port = h.port;
|
||||
return u.toString();
|
||||
}
|
||||
|
||||
async function httpJson(url) {
|
||||
const r = await fetch(url, { signal: AbortSignal.timeout(5000) });
|
||||
if (!r.ok) throw new Error(`GET ${url} ${r.status}`);
|
||||
return r.json();
|
||||
}
|
||||
|
||||
class Cdp {
|
||||
constructor(wsUrl) {
|
||||
this.wsUrl = wsUrl;
|
||||
this.ws = null;
|
||||
this.nextId = 1;
|
||||
this.pending = new Map();
|
||||
this.events = [];
|
||||
this.handlers = new Map();
|
||||
}
|
||||
async open() {
|
||||
this.ws = new WebSocket(this.wsUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => reject(new Error("ws open timeout")), 8000);
|
||||
this.ws.addEventListener("open", () => {
|
||||
clearTimeout(t);
|
||||
resolve();
|
||||
});
|
||||
this.ws.addEventListener("error", (e) => {
|
||||
clearTimeout(t);
|
||||
reject(e);
|
||||
});
|
||||
});
|
||||
this.ws.addEventListener("message", (ev) => {
|
||||
let msg;
|
||||
try {
|
||||
msg = JSON.parse(ev.data);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (msg.id && this.pending.has(msg.id)) {
|
||||
const { resolve, reject } = this.pending.get(msg.id);
|
||||
this.pending.delete(msg.id);
|
||||
if (msg.error) reject(new Error(JSON.stringify(msg.error)));
|
||||
else resolve(msg.result || {});
|
||||
return;
|
||||
}
|
||||
if (msg.method) {
|
||||
this.events.push(msg);
|
||||
const hs = this.handlers.get(msg.method) || [];
|
||||
for (const h of hs) h(msg.params || {}, msg.sessionId);
|
||||
}
|
||||
});
|
||||
}
|
||||
on(method, fn) {
|
||||
const arr = this.handlers.get(method) || [];
|
||||
arr.push(fn);
|
||||
this.handlers.set(method, arr);
|
||||
}
|
||||
send(method, params = {}, sessionId) {
|
||||
const id = this.nextId++;
|
||||
const payload = { id, method, params };
|
||||
if (sessionId) payload.sessionId = sessionId;
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
this.pending.delete(id);
|
||||
reject(new Error(`cdp timeout ${method}`));
|
||||
}, 20000);
|
||||
this.pending.set(id, {
|
||||
resolve: (v) => {
|
||||
clearTimeout(t);
|
||||
resolve(v);
|
||||
},
|
||||
reject: (e) => {
|
||||
clearTimeout(t);
|
||||
reject(e);
|
||||
},
|
||||
});
|
||||
this.ws.send(JSON.stringify(payload));
|
||||
});
|
||||
}
|
||||
close() {
|
||||
try {
|
||||
this.ws && this.ws.close();
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
|
||||
function detectVendor(ex) {
|
||||
const blob = `${ex.title || ""}\n${ex.text || ""}\n${ex.htmlHead || ""}`.toLowerCase();
|
||||
if (/cloudflare|cf-challenge|just a moment|cf-turnstile/.test(blob)) return "cloudflare";
|
||||
if (/datadome|captcha-delivery/.test(blob)) return "datadome";
|
||||
if (/akamai|access denied/.test(blob) && /akamai/.test(blob)) return "akamai";
|
||||
if (/amazon|opfcaptcha|validatecaptcha|enter the characters you see/.test(blob) && /robot|captcha/.test(blob))
|
||||
return "amazon";
|
||||
if (ex.looksBlocked) return "unknown";
|
||||
return "none";
|
||||
}
|
||||
|
||||
async function pickWs() {
|
||||
const notes = [];
|
||||
try {
|
||||
const ver = await httpJson(`${cdpHttp}/json/version`);
|
||||
notes.push({ step: "json/version", keys: Object.keys(ver) });
|
||||
if (ver.webSocketDebuggerUrl) {
|
||||
return { ws: rewriteWs(ver.webSocketDebuggerUrl, cdpHttp), notes, mode: "version" };
|
||||
}
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/version", error: String(e) });
|
||||
}
|
||||
try {
|
||||
const list = await httpJson(`${cdpHttp}/json/list`);
|
||||
notes.push({ step: "json/list", n: Array.isArray(list) ? list.length : 0 });
|
||||
const page = (list || []).find((t) => t.webSocketDebuggerUrl);
|
||||
if (page) return { ws: rewriteWs(page.webSocketDebuggerUrl, cdpHttp), notes, mode: "list" };
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/list", error: String(e) });
|
||||
}
|
||||
try {
|
||||
const created = await httpJson(`${cdpHttp}/json/new?${encodeURIComponent("about:blank")}`);
|
||||
if (created.webSocketDebuggerUrl) {
|
||||
return { ws: rewriteWs(created.webSocketDebuggerUrl, cdpHttp), notes, mode: "new" };
|
||||
}
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/new", error: String(e) });
|
||||
}
|
||||
// last resort: ws on same host/port
|
||||
const u = new URL(cdpHttp);
|
||||
return { ws: `ws://${u.hostname}:${u.port}`, notes, mode: "bare" };
|
||||
}
|
||||
|
||||
async function waitEvent(cdp, method, timeoutMs, pred) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => reject(new Error(`wait ${method} timeout`)), timeoutMs);
|
||||
const fn = (params, sessionId) => {
|
||||
if (pred && !pred(params, sessionId)) return;
|
||||
clearTimeout(t);
|
||||
resolve({ params, sessionId });
|
||||
};
|
||||
cdp.on(method, fn);
|
||||
});
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const t0 = Date.now();
|
||||
const pick = await pickWs();
|
||||
const cdp = new Cdp(pick.ws);
|
||||
await cdp.open();
|
||||
let sessionId;
|
||||
let targetId;
|
||||
const errors = [];
|
||||
try {
|
||||
try {
|
||||
await cdp.send("Browser.getVersion");
|
||||
} catch (e) {
|
||||
errors.push(`Browser.getVersion: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
const created = await cdp.send("Target.createTarget", { url: "about:blank" });
|
||||
targetId = created.targetId;
|
||||
const attached = await cdp.send("Target.attachToTarget", { targetId, flatten: true });
|
||||
sessionId = attached.sessionId;
|
||||
} catch (e) {
|
||||
errors.push(`Target.create/attach: ${e.message}`);
|
||||
sessionId = undefined;
|
||||
}
|
||||
const send = (m, p) => cdp.send(m, p, sessionId);
|
||||
try {
|
||||
await send("Page.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Page.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Runtime.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Runtime.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Network.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Network.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Network.setUserAgentOverride", { userAgent: UA });
|
||||
} catch (e) {
|
||||
errors.push(`UA override: ${e.message}`);
|
||||
}
|
||||
|
||||
let inflight = 0;
|
||||
let lastNet = Date.now();
|
||||
cdp.on("Network.requestWillBeSent", () => {
|
||||
inflight += 1;
|
||||
lastNet = Date.now();
|
||||
});
|
||||
cdp.on("Network.loadingFinished", () => {
|
||||
inflight = Math.max(0, inflight - 1);
|
||||
lastNet = Date.now();
|
||||
});
|
||||
cdp.on("Network.loadingFailed", () => {
|
||||
inflight = Math.max(0, inflight - 1);
|
||||
lastNet = Date.now();
|
||||
});
|
||||
|
||||
let loadFired = false;
|
||||
cdp.on("Page.loadEventFired", () => {
|
||||
loadFired = true;
|
||||
});
|
||||
cdp.on("Page.domContentEventFired", () => {});
|
||||
|
||||
const navStart = Date.now();
|
||||
let navResult;
|
||||
try {
|
||||
navResult = await send("Page.navigate", { url: targetUrl });
|
||||
} catch (e) {
|
||||
throw new Error(`Page.navigate failed: ${e.message}`);
|
||||
}
|
||||
const navDeadline = Date.now() + navTimeoutS * 1000;
|
||||
while (Date.now() < navDeadline && !loadFired) {
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
const loadWaitMs = Date.now() - navStart;
|
||||
// networkidle up to 8s, else 2s settle
|
||||
const idleDeadline = Date.now() + 8000;
|
||||
let usedIdle = false;
|
||||
while (Date.now() < idleDeadline) {
|
||||
if (inflight === 0 && Date.now() - lastNet >= 500 && loadFired) {
|
||||
usedIdle = true;
|
||||
break;
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
if (!usedIdle) {
|
||||
await new Promise((r) => setTimeout(r, 2000));
|
||||
}
|
||||
|
||||
let extract = {};
|
||||
try {
|
||||
const ev = await send("Runtime.evaluate", {
|
||||
expression: EXTRACT_JS,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
});
|
||||
extract = (ev.result && ev.result.value) || {};
|
||||
if (ev.exceptionDetails) {
|
||||
errors.push(`evaluate exception: ${JSON.stringify(ev.exceptionDetails)}`);
|
||||
}
|
||||
} catch (e) {
|
||||
errors.push(`Runtime.evaluate: ${e.message}`);
|
||||
}
|
||||
|
||||
if (targetId) {
|
||||
try {
|
||||
await cdp.send("Target.closeTarget", { targetId });
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const vendor = detectVendor(extract);
|
||||
const out = {
|
||||
ok: true,
|
||||
cdp_mode: pick.mode,
|
||||
t_total: (Date.now() - t0) / 1000,
|
||||
t_nav_s: loadWaitMs / 1000,
|
||||
networkidle: usedIdle,
|
||||
errorText: navResult && navResult.errorText,
|
||||
loaderId: navResult && navResult.loaderId,
|
||||
...extract,
|
||||
challenge_vendor: vendor,
|
||||
blocked: Boolean(extract.looksBlocked) || vendor !== "none",
|
||||
cdp_errors: errors,
|
||||
pick_notes: pick.notes,
|
||||
};
|
||||
process.stdout.write(JSON.stringify(out) + "\n");
|
||||
} finally {
|
||||
cdp.close();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
process.stdout.write(
|
||||
JSON.stringify({
|
||||
ok: false,
|
||||
error: String(e && e.message ? e.message : e),
|
||||
t_total: 0,
|
||||
title: "",
|
||||
text: "",
|
||||
htmlLen: 0,
|
||||
textLen: 0,
|
||||
blocked: true,
|
||||
challenge_vendor: "unknown",
|
||||
}) + "\n"
|
||||
);
|
||||
process.exit(0);
|
||||
});
|
||||
17
bench/chrome-headless-shell/compose.yml
Normal file
17
bench/chrome-headless-shell/compose.yml
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
services:
|
||||
shell:
|
||||
image: docker.io/chromedp/headless-shell:151.0.7922.109
|
||||
platform: linux/arm64
|
||||
container_name: bench-s3c-headless-shell
|
||||
ports:
|
||||
- "19223:9222"
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
# Image ENTRYPOINT /headless-shell/run.sh already sets --no-sandbox,
|
||||
# --remote-debugging-port=9223 and socat 9222->9223. Do not override those ports.
|
||||
command:
|
||||
- --disable-dev-shm-usage
|
||||
- --proxy-server=http://host.docker.internal:17890
|
||||
shm_size: "128m"
|
||||
mem_limit: 400m
|
||||
restart: "no"
|
||||
1
bench/chrome-headless-shell/raw/T4-amz-1.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-amz-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Page Not Found", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 2327, "text_len": 0, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "", "html_head": "<html><head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, shrink-to-fit=no\">\n <title>\n Page Not Found\n </title>\n <style>\n html,body{padding:0;margin:0}img{border:0}#a{background:#232f3e;padding:11px 11px 11px 192px}#b{position:absolute;left:22px;top:12px}#c{position:relative;max-width:800px;padding:0 40px 0 0}#e,#f{height:35px;border:0;font-size:1em}#e{width:100%;margin:0;padding:0 10px;border-radius:4px 0 0 4px}#f{cursor:pointer;background:#febd69;font-weight:bold;border-radius:0 4px 4px 0;-webkit-appearance:none;position:absolute;top:0;right:0;padding:0 12px}@media(max-width:500px){#a{padding:55px 10px 10px}#b{left:6px}}#g{text-align:center;margin:30px 0}#g i"}
|
||||
1
bench/chrome-headless-shell/raw/T4-amz-2.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-amz-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Page Not Found", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 2327, "text_len": 0, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "", "html_head": "<html><head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, shrink-to-fit=no\">\n <title>\n Page Not Found\n </title>\n <style>\n html,body{padding:0;margin:0}img{border:0}#a{background:#232f3e;padding:11px 11px 11px 192px}#b{position:absolute;left:22px;top:12px}#c{position:relative;max-width:800px;padding:0 40px 0 0}#e,#f{height:35px;border:0;font-size:1em}#e{width:100%;margin:0;padding:0 10px;border-radius:4px 0 0 4px}#f{cursor:pointer;background:#febd69;font-weight:bold;border-radius:0 4px 4px 0;-webkit-appearance:none;position:absolute;top:0;right:0;padding:0 12px}@media(max-width:500px){#a{padding:55px 10px 10px}#b{left:6px}}#g{text-align:center;margin:30px 0}#g i"}
|
||||
1
bench/chrome-headless-shell/raw/T4-amz-3.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-amz-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Page Not Found", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 2326, "text_len": 0, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "", "html_head": "<html><head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, shrink-to-fit=no\">\n <title>\n Page Not Found\n </title>\n <style>\n html,body{padding:0;margin:0}img{border:0}#a{background:#232f3e;padding:11px 11px 11px 192px}#b{position:absolute;left:22px;top:12px}#c{position:relative;max-width:800px;padding:0 40px 0 0}#e,#f{height:35px;border:0;font-size:1em}#e{width:100%;margin:0;padding:0 10px;border-radius:4px 0 0 4px}#f{cursor:pointer;background:#febd69;font-weight:bold;border-radius:0 4px 4px 0;-webkit-appearance:none;position:absolute;top:0;right:0;padding:0 12px}@media(max-width:500px){#a{padding:55px 10px 10px}#b{left:6px}}#g{text-align:center;margin:30px 0}#g i"}
|
||||
1
bench/chrome-headless-shell/raw/T4-ctrl-1.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-ctrl-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1499, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”\nby Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\nby Mar", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/chrome-headless-shell/raw/T4-ctrl-2.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-ctrl-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1499, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”\nby Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\nby Mar", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/chrome-headless-shell/raw/T4-ctrl-3.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-ctrl-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1499, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”\nby Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\nby Mar", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/chrome-headless-shell/raw/T4-medium-1.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-medium-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Medium", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 91669, "text_len": 226, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Open in app\n\nSign up\n\nSign in\n\nWrite\nPAGE NOT FOUND\n404\nOut of nothing, something.\nYou can find (just about) anything on Medium — apparently even a page that doesn’t exist. Maybe these stories will take you somewhere new?\nHome", "html_head": "<html lang=\"en\" data-rh-managed=\"lang\"><head><meta http-equiv=\"origin-trial\" content=\"A7vZI3v+Gz7JfuRolKNM4Aff6zaGuT7X0mf3wtoZTnKv6497cVMnhy03KDqX7kBz/q/iidW7srW31oQbBt4VhgoAAACUeyJvcmlnaW4iOiJodHRwczovL3d3dy5nb29nbGUuY29tOjQ0MyIsImZlYXR1cmUiOiJEaXNhYmxlVGhpcmRQYXJ0eVN0b3JhZ2VQYXJ0aXRpb25pbmczIiwiZXhwaXJ5IjoxNzU3OTgwODAwLCJpc1N1YmRvbWFpbiI6dHJ1ZSwiaXNUaGlyZFBhcnR5Ijp0cnVlfQ==\"><meta charset=\"utf-8\"><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><meta name=\"viewport\" content=\"width=device-width,minimum-scale=1,initial-scale=1,maximum-scale=1\"><style data-precedence=\"high\" data-href=\"lite-color-scheme-tokens\">:root{color-scheme:light;--invert-in-dark-mode: 0;--blend-in-dark-mode: normal;--color-bg-neutral-primary: #ffffff;--color-bg-neutral-secondar"}
|
||||
1
bench/chrome-headless-shell/raw/T4-medium-2.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-medium-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 687, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Sorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a341685e3dacf7d5 • Your IP: Click to reveal • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/chrome-headless-shell/raw/T4-medium-3.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-medium-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 687, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Sorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a34168a99e22f7d5 • Your IP: Click to reveal • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/chrome-headless-shell/raw/T4-shop-1.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-shop-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1453525, "text_len": 2886, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium Grey\n\n$100\n\nMEN'S TREE RUNNER\n\nMist\n\n$100\n\nMEN'S TREE RUNNER\n\nWheat\n\n$100\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nAnthracite\n\n$50\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nMushroom\n\n$50\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nAnthracite\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nDark Navy\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nLight", "html_head": "<html class=\"h-full\" lang=\"en-US\" style=\"--global-banner-height: 32px; --global-banner-offset: 32px;\"><head><script async=\"\" src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script type=\"text/javascript\" async=\"\" src=\"https://cdn.attn.tv/allbirds/dtag.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async=\"\" src=\"https://id-shop.govx.com/app/weareallbirds.myshopify.com/govx.js?shop=weareallbirds.myshopify.com\"></script><script src=\"https://connect.facebook.net/signals/config/1689016458013442?v=2.9.390&r=stable&domain=www.allbirds.com&hme=a767ae4c6a92a40ee4d7e358a937ad845c8bdb98a3bbe3412069dded3a7be57d&ex_m=115%2C221%2C166%2C26%2C80%2C81%2C158%2C76%2C75%2C11%2C176%2C100%2C20%2C150%2C138%2C45%2C83%2C88%2C146%2C171%"}
|
||||
1
bench/chrome-headless-shell/raw/T4-shop-2.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-shop-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1459360, "text_len": 2886, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Due to increased demand, orders may take up to 30 days to ship.\n\nFree ground shipping on orders over $100\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium Grey\n\n$100\n\nMEN'S TREE RUNNER\n\nMist\n\n$100\n\nMEN'S TREE RUNNER\n\nWheat\n\n$100\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nAnthracite\n\n$50\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nMushroom\n\n$50\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nAnthracite\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nDark Navy\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nLight", "html_head": "<html class=\"h-full\" lang=\"en-US\" style=\"--global-banner-height: 32px; --global-banner-offset: 32px;\"><head><script async=\"\" src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script type=\"text/javascript\" async=\"\" src=\"https://cdn.attn.tv/allbirds/dtag.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async=\"\" src=\"https://id-shop.govx.com/app/weareallbirds.myshopify.com/govx.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async=\"\" src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script type=\"text/javascript\" async=\"\" src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script async=\"\" src=\"https://s.pini"}
|
||||
1
bench/chrome-headless-shell/raw/T4-shop-3.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-shop-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1462947, "text_len": 2886, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Due to increased demand, orders may take up to 30 days to ship.\n\nFree ground shipping on orders over $100\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium Grey\n\n$100\n\nMEN'S TREE RUNNER\n\nMist\n\n$100\n\nMEN'S TREE RUNNER\n\nWheat\n\n$100\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nAnthracite\n\n$50\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nMushroom\n\n$50\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nAnthracite\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nDark Navy\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nLight", "html_head": "<html class=\"h-full\" lang=\"en-US\" style=\"--global-banner-height: 32px; --global-banner-offset: 32px;\"><head><script type=\"text/javascript\" async=\"\" src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script type=\"text/javascript\" async=\"\" src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script async=\"\" src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script src=\"https://connect.facebook.net/signals/config/1689016458013442?v=2.9.390&r=stable&domain=www.allbirds.com&hme=a767ae4c6a92a40ee4d7e358a937ad845c8bdb98a3bbe3412069dded3a7be57d&ex_m=115%2C221%2C166%2C26%2C80%2C81%2C158%2C76%2C75%2C11%2C176%2C100%2C20%2C150%2C138%2C45%2C83%2C88%2C146%2C171"}
|
||||
1
bench/chrome-headless-shell/raw/T4-t3x-1.excerpt
Normal file
1
bench/chrome-headless-shell/raw/T4-t3x-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-t3x", "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit&ia=web", "html_len": 194292, "text_len": 7985, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "DuckDuckGo\nOpen menu\n All\nImages\nNews\nVideos\nMore\nSearch Assist\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nResults from Reddit\nAll\nr/Earbuds\nr/GoodValue\nr/BuyItForLife\nr/HeadphoneAdvice\n\nr/Earbuds\n\nhttps://www.reddit.com\n › r › Earbuds › comments › 1q7ky9h › best_wireless_earbuds_2026_whats_actually_good_now\n\nbest wireless earbuds 2026. what's actually good now?\n\nPersonal fav: Noble Fokus Rex5. Amazing sound. Second fav: Soundpeats H3. Great price and great sound. I am waiting for the Geekfly Vitas.\n\nPosted 7 months ago\n\nr/Earbuds\n\nhttps://www.reddit.com\n › r › Earbuds › comments › 1utpkfd › best_earbuds_to_buy_in_2026_price_anc_battery_life\n\nBest Earbuds To Buy In 2026? (Price, ANC, Battery Life)\n\nFor me JBl Tune Buds 2 hands down, and. Soundcore P40i and C", "html_head": "<html lang=\"en-US\" class=\"has-zcm is-link-style-exp is-link-order-exp is-link-breadcrumb-exp is-related-search-exp is-vertical-tabs-exp js no-touch opacity csstransforms3d csstransitions svg cssfilters is-not-mobile-device full-urls breadcrumb-urls react has-footer\" style=\"--sds-color-palette-black: #000000; --sds-color-palette-white: #FFFFFF; --sds-color-palette-transparent: transparent; --sds-color-palette-gray-100: #111111; --sds-color-palette-gray-95: #161616; --sds-color-palette-gray-90: #222222; --sds-color-palette-gray-85: #333333; --sds-color-palette-gray-80: #444444; --sds-color-palette-gray-70: #666666; --sds-color-palette-gray-60: #888888; --sds-color-palette-gray-50: #aaaaaa; --sds-color-palette-gray-40: #cccccc; --sds-color-palette-gray-30: #dddddd; --sds-color-palette-gra"}
|
||||
8
bench/chrome-headless-shell/raw/cdp-version.json
Normal file
8
bench/chrome-headless-shell/raw/cdp-version.json
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"Browser": "Chrome/151.0.7922.109",
|
||||
"Protocol-Version": "1.3",
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.109 Safari/537.36",
|
||||
"V8-Version": "15.1.206.16",
|
||||
"WebKit-Version": "537.36 (@28a7a6c409e03c701d3474ef9e3b1f0be6249039)",
|
||||
"webSocketDebuggerUrl": "ws://127.0.0.1:19223/devtools/browser/77e984c4-4007-43b6-a32b-e0d19516d416"
|
||||
}
|
||||
68
bench/chrome-headless-shell/raw/idle.tsv
Normal file
68
bench/chrome-headless-shell/raw/idle.tsv
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T03:56:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.098
|
||||
2026-09-01T03:56:29Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.668
|
||||
2026-09-01T03:56:35Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.246
|
||||
2026-09-01T03:56:35Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.512
|
||||
2026-09-01T03:56:42Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.184
|
||||
2026-09-01T03:56:42Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.184
|
||||
2026-09-01T03:56:48Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 40.285
|
||||
2026-09-01T03:56:48Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.098
|
||||
2026-09-01T03:56:55Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.273
|
||||
2026-09-01T03:56:55Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.844
|
||||
2026-09-01T03:57:02Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.340
|
||||
2026-09-01T03:57:02Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.340
|
||||
2026-09-01T03:57:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.770
|
||||
2026-09-01T03:57:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.762
|
||||
2026-09-01T03:57:15Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.06% 37.460 39.121
|
||||
2026-09-01T03:57:15Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.06% 37.460 39.121
|
||||
2026-09-01T03:57:22Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.113
|
||||
2026-09-01T03:57:22Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.051
|
||||
2026-09-01T03:57:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.03% 37.460 39.590
|
||||
2026-09-01T03:57:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.03% 37.460 38.754
|
||||
2026-09-01T03:57:34Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.02% 37.460 39.633
|
||||
2026-09-01T03:57:34Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.02% 37.460 39.441
|
||||
2026-09-01T03:57:41Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.320
|
||||
2026-09-01T03:57:41Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.168
|
||||
2026-09-01T03:57:47Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.914
|
||||
2026-09-01T03:57:47Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.449
|
||||
2026-09-01T03:57:54Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.04% 37.460 38.980
|
||||
2026-09-01T03:57:54Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.04% 37.460 39.574
|
||||
2026-09-01T03:58:01Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.035
|
||||
2026-09-01T03:58:01Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.027
|
||||
2026-09-01T03:58:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.355
|
||||
2026-09-01T03:58:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.355
|
||||
2026-09-01T03:58:16Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.473
|
||||
2026-09-01T03:58:16Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.277
|
||||
2026-09-01T03:58:23Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.520
|
||||
2026-09-01T03:58:23Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.395
|
||||
2026-09-01T03:58:29Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.703
|
||||
2026-09-01T03:58:29Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.176
|
||||
2026-09-01T03:58:36Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.297
|
||||
2026-09-01T03:58:36Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.812
|
||||
2026-09-01T03:58:43Z bench-s3c-headless-shell 37.46MiB / 400MiB 3.40% 37.460 39.020
|
||||
2026-09-01T03:58:43Z bench-s3c-headless-shell 37.46MiB / 400MiB 3.40% 37.460 39.012
|
||||
2026-09-01T03:58:49Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.871
|
||||
2026-09-01T03:58:56Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.03% 37.460 39.281
|
||||
2026-09-01T03:59:02Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.812
|
||||
2026-09-01T03:59:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.035
|
||||
2026-09-01T03:59:15Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.06% 37.460 39.035
|
||||
2026-09-01T03:59:22Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.777
|
||||
2026-09-01T03:59:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.773
|
||||
2026-09-01T03:59:35Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.15% 37.460 38.574
|
||||
2026-09-01T03:59:42Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.855
|
||||
2026-09-01T03:59:48Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.531
|
||||
2026-09-01T03:59:55Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.10% 37.460 38.781
|
||||
2026-09-01T04:00:02Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.020
|
||||
2026-09-01T04:00:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.918
|
||||
2026-09-01T04:00:15Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.04% 37.460 38.656
|
||||
2026-09-01T04:00:22Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.031
|
||||
2026-09-01T04:00:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.734
|
||||
2026-09-01T04:00:35Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.17% 37.460 38.539
|
||||
2026-09-01T04:00:42Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.031
|
||||
2026-09-01T04:00:48Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.844
|
||||
2026-09-01T04:00:55Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.03% 37.460 38.535
|
||||
2026-09-01T04:01:01Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.406
|
||||
2026-09-01T04:01:08Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.773
|
||||
2026-09-01T04:01:15Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.02% 37.460 38.816
|
||||
2026-09-01T04:01:22Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.523
|
||||
2026-09-01T04:01:28Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 38.863
|
||||
|
35
bench/chrome-headless-shell/raw/image-meta.json
Normal file
35
bench/chrome-headless-shell/raw/image-meta.json
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"ref": "docker.io/chromedp/headless-shell:151.0.7922.109",
|
||||
"digest": null,
|
||||
"id": "sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04",
|
||||
"arch": "arm64",
|
||||
"pull": "local",
|
||||
"probed_at": "2026-09-01T12:10:38+08:00",
|
||||
"oom": false,
|
||||
"container_status": "running",
|
||||
"t5_wall_s": 73.974,
|
||||
"t5_parallelism": 1,
|
||||
"queue_impl": "client_fifo",
|
||||
"queue": {
|
||||
"first_complete_s": 1.9423,
|
||||
"last_complete_s": 73.9736,
|
||||
"queue_wait_p50_s": 0.0899,
|
||||
"queue_wait_p95_s": 0.1035,
|
||||
"rejected": 0
|
||||
},
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "204",
|
||||
"host_docker_internal": "fdc4:f303:9324::254 host.docker.internal",
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway"
|
||||
},
|
||||
"commands": [
|
||||
"docker compose -p bench-s3c-headless-shell up -d",
|
||||
"node cdp_fetch.mjs http://127.0.0.1:19223 <url>",
|
||||
"T5 ThreadPoolExecutor workers=1 n=60"
|
||||
],
|
||||
"notes": [
|
||||
"queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT"
|
||||
]
|
||||
}
|
||||
1
bench/chrome-headless-shell/raw/pull.how
Normal file
1
bench/chrome-headless-shell/raw/pull.how
Normal file
|
|
@ -0,0 +1 @@
|
|||
local
|
||||
0
bench/chrome-headless-shell/raw/session.done
Normal file
0
bench/chrome-headless-shell/raw/session.done
Normal file
52
bench/chrome-headless-shell/raw/session.tsv
Normal file
52
bench/chrome-headless-shell/raw/session.tsv
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T04:06:48Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.06% 37.460 39.031
|
||||
2026-09-01T04:06:50Z bench-s3c-headless-shell 37.47MiB / 400MiB 0.00% 37.470 38.773
|
||||
2026-09-01T04:06:53Z bench-s3c-headless-shell 37.46MiB / 400MiB 0.00% 37.460 39.277
|
||||
2026-09-01T04:06:55Z bench-s3c-headless-shell 37.47MiB / 400MiB 0.08% 37.470 39.027
|
||||
2026-09-01T04:06:58Z bench-s3c-headless-shell 56.43MiB / 400MiB 7.47% 56.430 57.395
|
||||
2026-09-01T04:07:00Z bench-s3c-headless-shell 113.5MiB / 400MiB 5.25% 113.500 114.203
|
||||
2026-09-01T04:07:03Z bench-s3c-headless-shell 114.7MiB / 400MiB 5.30% 114.700 115.641
|
||||
2026-09-01T04:07:05Z bench-s3c-headless-shell 115.8MiB / 400MiB 5.36% 115.800 116.352
|
||||
2026-09-01T04:07:09Z bench-s3c-headless-shell 205.7MiB / 400MiB 47.01% 205.700 206.480
|
||||
2026-09-01T04:07:12Z bench-s3c-headless-shell 257.1MiB / 400MiB 1.33% 257.100 257.645
|
||||
2026-09-01T04:07:15Z bench-s3c-headless-shell 247.5MiB / 400MiB 1.67% 247.500 248.203
|
||||
2026-09-01T04:07:18Z bench-s3c-headless-shell 247MiB / 400MiB 1.24% 247.000 248.352
|
||||
2026-09-01T04:07:20Z bench-s3c-headless-shell 123.8MiB / 400MiB 43.73% 123.800 124.691
|
||||
2026-09-01T04:07:23Z bench-s3c-headless-shell 159.3MiB / 400MiB 31.48% 159.300 160.637
|
||||
2026-09-01T04:07:25Z bench-s3c-headless-shell 156.3MiB / 400MiB 0.26% 156.300 157.262
|
||||
2026-09-01T04:07:28Z bench-s3c-headless-shell 156.3MiB / 400MiB 0.36% 156.300 157.727
|
||||
2026-09-01T04:07:30Z bench-s3c-headless-shell 155.9MiB / 400MiB 0.16% 155.900 157.254
|
||||
2026-09-01T04:07:33Z bench-s3c-headless-shell 134.3MiB / 400MiB 6.44% 134.300 134.812
|
||||
2026-09-01T04:07:35Z bench-s3c-headless-shell 154.7MiB / 400MiB 0.57% 154.700 156.148
|
||||
2026-09-01T04:07:38Z bench-s3c-headless-shell 154.7MiB / 400MiB 0.42% 154.700 156.195
|
||||
2026-09-01T04:07:41Z bench-s3c-headless-shell 154.5MiB / 400MiB 0.04% 154.500 155.852
|
||||
2026-09-01T04:07:44Z bench-s3c-headless-shell 133.5MiB / 400MiB 5.10% 133.500 134.254
|
||||
2026-09-01T04:07:47Z bench-s3c-headless-shell 239MiB / 400MiB 87.90% 239.000 241.680
|
||||
2026-09-01T04:07:50Z bench-s3c-headless-shell 332.2MiB / 400MiB 81.76% 332.200 334.227
|
||||
2026-09-01T04:07:53Z bench-s3c-headless-shell 349.7MiB / 400MiB 37.02% 349.700 350.441
|
||||
2026-09-01T04:07:55Z bench-s3c-headless-shell 390.5MiB / 400MiB 43.56% 390.500 391.207
|
||||
2026-09-01T04:07:58Z bench-s3c-headless-shell 393MiB / 400MiB 21.45% 393.000 392.414
|
||||
2026-09-01T04:08:00Z bench-s3c-headless-shell 393.5MiB / 400MiB 15.33% 393.500 394.273
|
||||
2026-09-01T04:08:03Z bench-s3c-headless-shell 394MiB / 400MiB 11.19% 394.000 395.258
|
||||
2026-09-01T04:08:05Z bench-s3c-headless-shell 150.9MiB / 400MiB 18.52% 150.900 150.859
|
||||
2026-09-01T04:08:08Z bench-s3c-headless-shell 335.1MiB / 400MiB 177.79% 335.100 335.586
|
||||
2026-09-01T04:08:11Z bench-s3c-headless-shell 371.9MiB / 400MiB 32.28% 371.900 373.027
|
||||
2026-09-01T04:08:14Z bench-s3c-headless-shell 373.1MiB / 400MiB 14.79% 373.100 373.648
|
||||
2026-09-01T04:08:17Z bench-s3c-headless-shell 371.9MiB / 400MiB 13.39% 371.900 372.547
|
||||
2026-09-01T04:08:20Z bench-s3c-headless-shell 378.1MiB / 400MiB 13.16% 378.100 378.648
|
||||
2026-09-01T04:08:23Z bench-s3c-headless-shell 378.5MiB / 400MiB 11.82% 378.500 379.273
|
||||
2026-09-01T04:08:25Z bench-s3c-headless-shell 356.3MiB / 400MiB 166.09% 356.300 360.488
|
||||
2026-09-01T04:08:28Z bench-s3c-headless-shell 370.8MiB / 400MiB 25.18% 370.800 371.074
|
||||
2026-09-01T04:08:30Z bench-s3c-headless-shell 377.2MiB / 400MiB 23.43% 377.200 377.801
|
||||
2026-09-01T04:08:33Z bench-s3c-headless-shell 377.2MiB / 400MiB 12.46% 377.200 377.188
|
||||
2026-09-01T04:08:35Z bench-s3c-headless-shell 165.4MiB / 400MiB 28.49% 165.400 165.551
|
||||
2026-09-01T04:08:38Z bench-s3c-headless-shell 172.2MiB / 400MiB 5.89% 172.200 175.695
|
||||
2026-09-01T04:08:40Z bench-s3c-headless-shell 169.1MiB / 400MiB 10.08% 169.100 171.215
|
||||
2026-09-01T04:08:43Z bench-s3c-headless-shell 165MiB / 400MiB 10.34% 165.000 165.492
|
||||
2026-09-01T04:08:45Z bench-s3c-headless-shell 160MiB / 400MiB 3.88% 160.000 160.758
|
||||
2026-09-01T04:08:49Z bench-s3c-headless-shell 245.1MiB / 400MiB 117.18% 245.100 246.645
|
||||
2026-09-01T04:08:52Z bench-s3c-headless-shell 160.4MiB / 400MiB 0.61% 160.400 161.203
|
||||
2026-09-01T04:08:55Z bench-s3c-headless-shell 159.5MiB / 400MiB 0.06% 159.500 160.004
|
||||
2026-09-01T04:08:58Z bench-s3c-headless-shell 158.9MiB / 400MiB 0.02% 158.900 160.461
|
||||
2026-09-01T04:09:00Z bench-s3c-headless-shell 158.9MiB / 400MiB 0.06% 158.900 160.309
|
||||
2026-09-01T04:09:03Z bench-s3c-headless-shell 159.3MiB / 400MiB 0.00% 159.300 160.016
|
||||
|
13
bench/chrome-headless-shell/raw/t4.jsonl
Normal file
13
bench/chrome-headless-shell/raw/t4.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{"ts": "2026-09-01T12:07:01+08:00", "scheme": "chrome-headless-shell", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 3.073, "ttfb": 2.464, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:07:03+08:00", "scheme": "chrome-headless-shell", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.124, "ttfb": 0.545, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:07:05+08:00", "scheme": "chrome-headless-shell", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.129, "ttfb": 0.549, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:07:21+08:00", "scheme": "chrome-headless-shell", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 14.211, "ttfb": 4.059, "t_namelookup": 0, "t_connect": 0, "bytes": 91669, "html_len": 91669, "text_len": 226, "title": "Medium", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Open in app\n\nSign up\n\nSign in\n\nWrite\nPAGE NOT FOUND\n404\nOut of nothing, something.\nYou can find (just about) anything on Medium — apparently even a page that doesn’t exist. Maybe these stories will take you somewhere new?\nHome"}
|
||||
{"ts": "2026-09-01T12:07:33+08:00", "scheme": "chrome-headless-shell", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 10.956, "ttfb": 0.807, "t_namelookup": 0, "t_connect": 0, "bytes": 5193, "html_len": 5193, "text_len": 687, "title": "Attention Required! | Cloudflare", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "cloudflare", "networkidle": false, "assert_ok": false, "note": "medium_wall_or_waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Sorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doin"}
|
||||
{"ts": "2026-09-01T12:07:45+08:00", "scheme": "chrome-headless-shell", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 10.85, "ttfb": 0.704, "t_namelookup": 0, "t_connect": 0, "bytes": 5193, "html_len": 5193, "text_len": 687, "title": "Attention Required! | Cloudflare", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "cloudflare", "networkidle": false, "assert_ok": false, "note": "medium_wall_or_waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Sorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doin"}
|
||||
{"ts": "2026-09-01T12:08:07+08:00", "scheme": "chrome-headless-shell", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 20.495, "ttfb": 10.333, "t_namelookup": 0, "t_connect": 0, "bytes": 1453525, "html_len": 1453525, "text_len": 2886, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium"}
|
||||
{"ts": "2026-09-01T12:08:24+08:00", "scheme": "chrome-headless-shell", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 16.363, "ttfb": 6.212, "t_namelookup": 0, "t_connect": 0, "bytes": 1459360, "html_len": 1459360, "text_len": 2886, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Due to increased demand, orders may take up to 30 days to ship.\n\nFree ground shipping on orders over $100\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium"}
|
||||
{"ts": "2026-09-01T12:08:37+08:00", "scheme": "chrome-headless-shell", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 11.446, "ttfb": 1.298, "t_namelookup": 0, "t_connect": 0, "bytes": 1462947, "html_len": 1462947, "text_len": 2886, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Due to increased demand, orders may take up to 30 days to ship.\n\nFree ground shipping on orders over $100\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium"}
|
||||
{"ts": "2026-09-01T12:08:40+08:00", "scheme": "chrome-headless-shell", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 2.014, "ttfb": 1.537, "t_namelookup": 0, "t_connect": 0, "bytes": 2327, "html_len": 2327, "text_len": 0, "title": "Page Not Found", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": ""}
|
||||
{"ts": "2026-09-01T12:08:42+08:00", "scheme": "chrome-headless-shell", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.305, "ttfb": 0.728, "t_namelookup": 0, "t_connect": 0, "bytes": 2327, "html_len": 2327, "text_len": 0, "title": "Page Not Found", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": ""}
|
||||
{"ts": "2026-09-01T12:08:45+08:00", "scheme": "chrome-headless-shell", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.274, "ttfb": 0.696, "t_namelookup": 0, "t_connect": 0, "bytes": 2326, "html_len": 2326, "text_len": 0, "title": "Page Not Found", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": ""}
|
||||
{"ts": "2026-09-01T12:08:51+08:00", "scheme": "chrome-headless-shell", "template": "T4-t3x", "url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit&ia=web", "http_code": 200, "http_status_in_page": null, "t_total": 5.474, "ttfb": 4.587, "t_namelookup": 0, "t_connect": 0, "bytes": 194292, "html_len": 194292, "text_len": 7985, "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "DuckDuckGo\nOpen menu\n All\nImages\nNews\nVideos\nMore\nSearch Assist\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nResults from Reddit\nAll\nr/Earbuds\nr/GoodValue\nr/BuyItForLife\nr/HeadphoneAdvice\n\nr/Earbuds\n\nhttps://www.reddit.com\n › r › Earbuds › comments › 1q7ky9h › best_wireless_earbuds_2026_whats_actually_good_now\n\nbest wireless earbuds 2026. what's actually good now?\n\nPersonal fav: Noble Fokus Rex5. Amazing sound. Second fav: Soundpeats H3. Great price and great sound."}
|
||||
60
bench/chrome-headless-shell/raw/t5-queue.jsonl
Normal file
60
bench/chrome-headless-shell/raw/t5-queue.jsonl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{"i": 1, "enqueue_s": 0.036948204040527344, "start_s": 0.11634516716003418, "end_s": 1.942345142364502, "queue_wait_s": 0.07939696311950684, "assert_ok": true, "t_total": 1.826}
|
||||
{"i": 2, "enqueue_s": 1.9424331188201904, "start_s": 2.034093141555786, "end_s": 3.155093193054199, "queue_wait_s": 0.0916600227355957, "assert_ok": true, "t_total": 1.121}
|
||||
{"i": 3, "enqueue_s": 3.1551830768585205, "start_s": 3.2411420345306396, "end_s": 4.351141929626465, "queue_wait_s": 0.08595895767211914, "assert_ok": true, "t_total": 1.11}
|
||||
{"i": 4, "enqueue_s": 4.351238012313843, "start_s": 4.442795038223267, "end_s": 5.566795110702515, "queue_wait_s": 0.09155702590942383, "assert_ok": true, "t_total": 1.124}
|
||||
{"i": 5, "enqueue_s": 5.566895961761475, "start_s": 5.673076152801514, "end_s": 6.793076038360596, "queue_wait_s": 0.10618019104003906, "assert_ok": true, "t_total": 1.12}
|
||||
{"i": 6, "enqueue_s": 6.793197870254517, "start_s": 6.883285999298096, "end_s": 8.00528597831726, "queue_wait_s": 0.0900881290435791, "assert_ok": true, "t_total": 1.122}
|
||||
{"i": 7, "enqueue_s": 8.00537896156311, "start_s": 8.103138208389282, "end_s": 9.235138177871704, "queue_wait_s": 0.09775924682617188, "assert_ok": true, "t_total": 1.132}
|
||||
{"i": 8, "enqueue_s": 9.235266923904419, "start_s": 9.343083143234253, "end_s": 10.464083194732666, "queue_wait_s": 0.10781621932983398, "assert_ok": true, "t_total": 1.121}
|
||||
{"i": 9, "enqueue_s": 10.464179039001465, "start_s": 10.55424690246582, "end_s": 11.671247005462646, "queue_wait_s": 0.09006786346435547, "assert_ok": true, "t_total": 1.117}
|
||||
{"i": 10, "enqueue_s": 11.671355962753296, "start_s": 11.761014223098755, "end_s": 12.894014120101929, "queue_wait_s": 0.08965826034545898, "assert_ok": true, "t_total": 1.133}
|
||||
{"i": 11, "enqueue_s": 12.894119024276733, "start_s": 12.98481011390686, "end_s": 14.316810131072998, "queue_wait_s": 0.09069108963012695, "assert_ok": true, "t_total": 1.332}
|
||||
{"i": 12, "enqueue_s": 14.316944122314453, "start_s": 14.396131038665771, "end_s": 15.518131017684937, "queue_wait_s": 0.07918691635131836, "assert_ok": true, "t_total": 1.122}
|
||||
{"i": 13, "enqueue_s": 15.51823902130127, "start_s": 15.611064910888672, "end_s": 16.74106502532959, "queue_wait_s": 0.09282588958740234, "assert_ok": true, "t_total": 1.13}
|
||||
{"i": 14, "enqueue_s": 16.741164207458496, "start_s": 16.82989501953125, "end_s": 17.953895092010498, "queue_wait_s": 0.0887308120727539, "assert_ok": true, "t_total": 1.124}
|
||||
{"i": 15, "enqueue_s": 17.953979969024658, "start_s": 18.042167901992798, "end_s": 19.170167922973633, "queue_wait_s": 0.08818793296813965, "assert_ok": true, "t_total": 1.128}
|
||||
{"i": 16, "enqueue_s": 19.170269012451172, "start_s": 19.267627000808716, "end_s": 20.390626907348633, "queue_wait_s": 0.09735798835754395, "assert_ok": true, "t_total": 1.123}
|
||||
{"i": 17, "enqueue_s": 20.39072299003601, "start_s": 20.472331047058105, "end_s": 21.59333109855652, "queue_wait_s": 0.08160805702209473, "assert_ok": true, "t_total": 1.121}
|
||||
{"i": 18, "enqueue_s": 21.593441247940063, "start_s": 21.68717098236084, "end_s": 22.802170991897583, "queue_wait_s": 0.09372973442077637, "assert_ok": true, "t_total": 1.115}
|
||||
{"i": 19, "enqueue_s": 22.802268028259277, "start_s": 22.906558990478516, "end_s": 24.02855896949768, "queue_wait_s": 0.10429096221923828, "assert_ok": true, "t_total": 1.122}
|
||||
{"i": 20, "enqueue_s": 24.028764009475708, "start_s": 24.12020707130432, "end_s": 25.234207153320312, "queue_wait_s": 0.09144306182861328, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 21, "enqueue_s": 25.23430609703064, "start_s": 25.317145109176636, "end_s": 26.431145191192627, "queue_wait_s": 0.0828390121459961, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 22, "enqueue_s": 26.431237936019897, "start_s": 26.533463954925537, "end_s": 27.651463985443115, "queue_wait_s": 0.10222601890563965, "assert_ok": true, "t_total": 1.118}
|
||||
{"i": 23, "enqueue_s": 27.651615142822266, "start_s": 27.733967065811157, "end_s": 28.855967044830322, "queue_wait_s": 0.0823519229888916, "assert_ok": true, "t_total": 1.122}
|
||||
{"i": 24, "enqueue_s": 28.856059074401855, "start_s": 28.943328142166138, "end_s": 30.05132818222046, "queue_wait_s": 0.08726906776428223, "assert_ok": true, "t_total": 1.108}
|
||||
{"i": 25, "enqueue_s": 30.051440954208374, "start_s": 30.1407208442688, "end_s": 31.261720895767212, "queue_wait_s": 0.0892798900604248, "assert_ok": true, "t_total": 1.121}
|
||||
{"i": 26, "enqueue_s": 31.261831998825073, "start_s": 31.364443063735962, "end_s": 32.48344302177429, "queue_wait_s": 0.10261106491088867, "assert_ok": true, "t_total": 1.119}
|
||||
{"i": 27, "enqueue_s": 32.48355793952942, "start_s": 32.57312297821045, "end_s": 33.69312286376953, "queue_wait_s": 0.08956503868103027, "assert_ok": true, "t_total": 1.12}
|
||||
{"i": 28, "enqueue_s": 33.693238258361816, "start_s": 33.77358317375183, "end_s": 35.103583097457886, "queue_wait_s": 0.08034491539001465, "assert_ok": true, "t_total": 1.33}
|
||||
{"i": 29, "enqueue_s": 35.103683948516846, "start_s": 35.19248604774475, "end_s": 36.308485984802246, "queue_wait_s": 0.08880209922790527, "assert_ok": true, "t_total": 1.116}
|
||||
{"i": 30, "enqueue_s": 36.30860090255737, "start_s": 36.39005517959595, "end_s": 37.50305509567261, "queue_wait_s": 0.08145427703857422, "assert_ok": true, "t_total": 1.113}
|
||||
{"i": 31, "enqueue_s": 37.50316309928894, "start_s": 37.602975368499756, "end_s": 38.72597527503967, "queue_wait_s": 0.09981226921081543, "assert_ok": true, "t_total": 1.123}
|
||||
{"i": 32, "enqueue_s": 38.72608017921448, "start_s": 38.82769417762756, "end_s": 39.94569420814514, "queue_wait_s": 0.10161399841308594, "assert_ok": true, "t_total": 1.118}
|
||||
{"i": 33, "enqueue_s": 39.94578695297241, "start_s": 40.03146290779114, "end_s": 41.148463010787964, "queue_wait_s": 0.08567595481872559, "assert_ok": true, "t_total": 1.117}
|
||||
{"i": 34, "enqueue_s": 41.14858603477478, "start_s": 41.22830533981323, "end_s": 42.354305267333984, "queue_wait_s": 0.07971930503845215, "assert_ok": true, "t_total": 1.126}
|
||||
{"i": 35, "enqueue_s": 42.354403018951416, "start_s": 42.43888783454895, "end_s": 43.55288791656494, "queue_wait_s": 0.08448481559753418, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 36, "enqueue_s": 43.55299425125122, "start_s": 43.65015912055969, "end_s": 44.765159130096436, "queue_wait_s": 0.09716486930847168, "assert_ok": true, "t_total": 1.115}
|
||||
{"i": 37, "enqueue_s": 44.76528596878052, "start_s": 44.86265206336975, "end_s": 45.97665214538574, "queue_wait_s": 0.0973660945892334, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 38, "enqueue_s": 45.97675108909607, "start_s": 46.06692910194397, "end_s": 47.18492913246155, "queue_wait_s": 0.09017801284790039, "assert_ok": true, "t_total": 1.118}
|
||||
{"i": 39, "enqueue_s": 47.18503212928772, "start_s": 47.27218413352966, "end_s": 48.38918423652649, "queue_wait_s": 0.08715200424194336, "assert_ok": true, "t_total": 1.117}
|
||||
{"i": 40, "enqueue_s": 48.389304876327515, "start_s": 48.4699490070343, "end_s": 49.58194899559021, "queue_wait_s": 0.08064413070678711, "assert_ok": true, "t_total": 1.112}
|
||||
{"i": 41, "enqueue_s": 49.58207702636719, "start_s": 49.673800230026245, "end_s": 50.78980016708374, "queue_wait_s": 0.09172320365905762, "assert_ok": true, "t_total": 1.116}
|
||||
{"i": 42, "enqueue_s": 50.78991389274597, "start_s": 50.87759733200073, "end_s": 51.9995973110199, "queue_wait_s": 0.08768343925476074, "assert_ok": true, "t_total": 1.122}
|
||||
{"i": 43, "enqueue_s": 51.99970006942749, "start_s": 52.092223167419434, "end_s": 53.20822310447693, "queue_wait_s": 0.09252309799194336, "assert_ok": true, "t_total": 1.116}
|
||||
{"i": 44, "enqueue_s": 53.2083261013031, "start_s": 53.29989504814148, "end_s": 54.40889501571655, "queue_wait_s": 0.0915689468383789, "assert_ok": true, "t_total": 1.109}
|
||||
{"i": 45, "enqueue_s": 54.4090051651001, "start_s": 54.48979592323303, "end_s": 55.60379600524902, "queue_wait_s": 0.08079075813293457, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 46, "enqueue_s": 55.60388517379761, "start_s": 55.692177057266235, "end_s": 56.80717706680298, "queue_wait_s": 0.08829188346862793, "assert_ok": true, "t_total": 1.115}
|
||||
{"i": 47, "enqueue_s": 56.80728006362915, "start_s": 56.89650011062622, "end_s": 58.02250003814697, "queue_wait_s": 0.08922004699707031, "assert_ok": true, "t_total": 1.126}
|
||||
{"i": 48, "enqueue_s": 58.02260112762451, "start_s": 58.121066093444824, "end_s": 59.23106598854065, "queue_wait_s": 0.0984649658203125, "assert_ok": true, "t_total": 1.11}
|
||||
{"i": 49, "enqueue_s": 59.23120903968811, "start_s": 59.32873010635376, "end_s": 60.4437301158905, "queue_wait_s": 0.09752106666564941, "assert_ok": true, "t_total": 1.115}
|
||||
{"i": 50, "enqueue_s": 60.44385123252869, "start_s": 60.52617692947388, "end_s": 61.64117693901062, "queue_wait_s": 0.08232569694519043, "assert_ok": true, "t_total": 1.115}
|
||||
{"i": 51, "enqueue_s": 61.64126515388489, "start_s": 61.726726055145264, "end_s": 62.83572602272034, "queue_wait_s": 0.08546090126037598, "assert_ok": true, "t_total": 1.109}
|
||||
{"i": 52, "enqueue_s": 62.835824966430664, "start_s": 62.91743326187134, "end_s": 64.02643322944641, "queue_wait_s": 0.08160829544067383, "assert_ok": true, "t_total": 1.109}
|
||||
{"i": 53, "enqueue_s": 64.02653408050537, "start_s": 64.11870408058167, "end_s": 65.22570419311523, "queue_wait_s": 0.09217000007629395, "assert_ok": true, "t_total": 1.107}
|
||||
{"i": 54, "enqueue_s": 65.22580528259277, "start_s": 65.31475305557251, "end_s": 66.73875308036804, "queue_wait_s": 0.08894777297973633, "assert_ok": true, "t_total": 1.424}
|
||||
{"i": 55, "enqueue_s": 66.73885107040405, "start_s": 66.82261180877686, "end_s": 67.93661189079285, "queue_wait_s": 0.08376073837280273, "assert_ok": true, "t_total": 1.114}
|
||||
{"i": 56, "enqueue_s": 67.93670201301575, "start_s": 68.01580023765564, "end_s": 69.12180018424988, "queue_wait_s": 0.07909822463989258, "assert_ok": true, "t_total": 1.106}
|
||||
{"i": 57, "enqueue_s": 69.12189817428589, "start_s": 69.21376180648804, "end_s": 70.3207619190216, "queue_wait_s": 0.09186363220214844, "assert_ok": true, "t_total": 1.107}
|
||||
{"i": 58, "enqueue_s": 70.32087802886963, "start_s": 70.41365098953247, "end_s": 71.52465105056763, "queue_wait_s": 0.0927729606628418, "assert_ok": true, "t_total": 1.111}
|
||||
{"i": 59, "enqueue_s": 71.52476024627686, "start_s": 71.62109017372131, "end_s": 72.76009011268616, "queue_wait_s": 0.09632992744445801, "assert_ok": true, "t_total": 1.139}
|
||||
{"i": 60, "enqueue_s": 72.76019597053528, "start_s": 72.86362600326538, "end_s": 73.9736258983612, "queue_wait_s": 0.10343003273010254, "assert_ok": true, "t_total": 1.11}
|
||||
35
bench/chrome-headless-shell/raw/t5-stats.tsv
Normal file
35
bench/chrome-headless-shell/raw/t5-stats.tsv
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T04:09:05Z bench-s3c-headless-shell 177.3MiB / 400MiB 11.69% 177.300 162.059
|
||||
2026-09-01T04:09:08Z bench-s3c-headless-shell 176.9MiB / 400MiB 24.59% 176.900 177.391
|
||||
2026-09-01T04:09:10Z bench-s3c-headless-shell 170.3MiB / 400MiB 34.24% 170.300 171.977
|
||||
2026-09-01T04:09:13Z bench-s3c-headless-shell 161.9MiB / 400MiB 19.11% 161.900 162.660
|
||||
2026-09-01T04:09:15Z bench-s3c-headless-shell 170.9MiB / 400MiB 18.74% 170.900 176.680
|
||||
2026-09-01T04:09:18Z bench-s3c-headless-shell 176.3MiB / 400MiB 14.50% 176.300 176.949
|
||||
2026-09-01T04:09:21Z bench-s3c-headless-shell 173.2MiB / 400MiB 27.77% 173.200 173.676
|
||||
2026-09-01T04:09:24Z bench-s3c-headless-shell 179MiB / 400MiB 25.77% 179.000 179.383
|
||||
2026-09-01T04:09:27Z bench-s3c-headless-shell 179.2MiB / 400MiB 26.09% 179.200 179.418
|
||||
2026-09-01T04:09:30Z bench-s3c-headless-shell 171.1MiB / 400MiB 26.26% 171.100 172.570
|
||||
2026-09-01T04:09:33Z bench-s3c-headless-shell 179.3MiB / 400MiB 16.52% 179.300 179.531
|
||||
2026-09-01T04:09:35Z bench-s3c-headless-shell 172.8MiB / 400MiB 18.62% 172.800 173.273
|
||||
2026-09-01T04:09:38Z bench-s3c-headless-shell 171.8MiB / 400MiB 24.81% 171.800 173.430
|
||||
2026-09-01T04:09:40Z bench-s3c-headless-shell 178.4MiB / 400MiB 26.57% 178.400 178.359
|
||||
2026-09-01T04:09:43Z bench-s3c-headless-shell 171.8MiB / 400MiB 25.67% 171.800 173.090
|
||||
2026-09-01T04:09:45Z bench-s3c-headless-shell 178.5MiB / 400MiB 26.17% 178.500 178.754
|
||||
2026-09-01T04:09:48Z bench-s3c-headless-shell 178.6MiB / 400MiB 24.31% 178.600 179.500
|
||||
2026-09-01T04:09:50Z bench-s3c-headless-shell 171.2MiB / 400MiB 24.02% 171.200 171.984
|
||||
2026-09-01T04:09:53Z bench-s3c-headless-shell 178.6MiB / 400MiB 11.36% 178.600 179.312
|
||||
2026-09-01T04:09:56Z bench-s3c-headless-shell 171.6MiB / 400MiB 21.23% 171.600 174.141
|
||||
2026-09-01T04:09:59Z bench-s3c-headless-shell 179.3MiB / 400MiB 13.14% 179.300 163.137
|
||||
2026-09-01T04:10:02Z bench-s3c-headless-shell 172.3MiB / 400MiB 16.67% 172.300 175.133
|
||||
2026-09-01T04:10:05Z bench-s3c-headless-shell 179.1MiB / 400MiB 13.05% 179.100 162.582
|
||||
2026-09-01T04:10:08Z bench-s3c-headless-shell 178.5MiB / 400MiB 22.15% 178.500 178.668
|
||||
2026-09-01T04:10:10Z bench-s3c-headless-shell 163MiB / 400MiB 19.19% 163.000 170.336
|
||||
2026-09-01T04:10:13Z bench-s3c-headless-shell 178.6MiB / 400MiB 27.50% 178.600 179.543
|
||||
2026-09-01T04:10:15Z bench-s3c-headless-shell 173MiB / 400MiB 22.18% 173.000 173.691
|
||||
2026-09-01T04:10:18Z bench-s3c-headless-shell 163.5MiB / 400MiB 23.55% 163.500 164.324
|
||||
2026-09-01T04:10:20Z bench-s3c-headless-shell 152.4MiB / 400MiB 0.24% 152.400 153.727
|
||||
2026-09-01T04:10:23Z bench-s3c-headless-shell 152.4MiB / 400MiB 0.13% 152.400 153.738
|
||||
2026-09-01T04:10:25Z bench-s3c-headless-shell 152.6MiB / 400MiB 0.14% 152.600 153.605
|
||||
2026-09-01T04:10:28Z bench-s3c-headless-shell 152.6MiB / 400MiB 0.08% 152.600 154.039
|
||||
2026-09-01T04:10:31Z bench-s3c-headless-shell 152.4MiB / 400MiB 0.18% 152.400 153.559
|
||||
2026-09-01T04:10:34Z bench-s3c-headless-shell 152.4MiB / 400MiB 0.17% 152.400 153.977
|
||||
|
0
bench/chrome-headless-shell/raw/t5.done
Normal file
0
bench/chrome-headless-shell/raw/t5.done
Normal file
60
bench/chrome-headless-shell/raw/t5.jsonl
Normal file
60
bench/chrome-headless-shell/raw/t5.jsonl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{"ts": "2026-09-01T12:09:07+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.826, "ttfb": 1.251, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:08+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.121, "ttfb": 0.555, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:09+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.11, "ttfb": 0.543, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:11+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.124, "ttfb": 0.552, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:12+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.12, "ttfb": 0.545, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:13+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.122, "ttfb": 0.545, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:14+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.132, "ttfb": 0.554, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:15+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.121, "ttfb": 0.547, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:17+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.117, "ttfb": 0.544, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:18+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.133, "ttfb": 0.554, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:19+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.332, "ttfb": 0.748, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:20+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.122, "ttfb": 0.544, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:22+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.13, "ttfb": 0.555, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:23+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.124, "ttfb": 0.548, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:24+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.128, "ttfb": 0.554, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:25+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.123, "ttfb": 0.544, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:27+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.121, "ttfb": 0.558, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:28+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.115, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:29+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.122, "ttfb": 0.54, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:30+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.537, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:31+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:33+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.118, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:34+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.122, "ttfb": 0.538, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:35+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.108, "ttfb": 0.64, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:36+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.121, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:37+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.119, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:39+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.12, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:40+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.33, "ttfb": 0.742, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:41+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.116, "ttfb": 0.536, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:42+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.113, "ttfb": 0.534, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:44+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.123, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:45+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.118, "ttfb": 0.537, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:46+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.117, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:47+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.126, "ttfb": 0.544, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:49+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.536, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:50+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.115, "ttfb": 0.538, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:51+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.541, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:52+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.118, "ttfb": 0.539, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:53+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.117, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:55+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.112, "ttfb": 0.537, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:56+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.116, "ttfb": 0.538, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:57+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.122, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:58+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.116, "ttfb": 0.537, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:09:59+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.109, "ttfb": 0.534, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:01+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.546, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:02+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.115, "ttfb": 0.535, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:03+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.126, "ttfb": 0.542, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:04+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.11, "ttfb": 0.533, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:05+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.115, "ttfb": 0.538, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:07+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.115, "ttfb": 0.534, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:08+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.109, "ttfb": 0.535, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:09+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.109, "ttfb": 0.534, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:10+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.107, "ttfb": 0.533, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:12+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.424, "ttfb": 0.947, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:13+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.114, "ttfb": 0.538, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:14+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.106, "ttfb": 0.534, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:15+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.107, "ttfb": 0.536, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:16+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.111, "ttfb": 0.537, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:18+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.139, "ttfb": 0.662, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
{"ts": "2026-09-01T12:10:19+08:00", "scheme": "chrome-headless-shell", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.11, "ttfb": 0.533, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1499, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"}
|
||||
106
bench/chrome-headless-shell/request.sh
Executable file
106
bench/chrome-headless-shell/request.sh
Executable file
|
|
@ -0,0 +1,106 @@
|
|||
#!/usr/bin/env bash
|
||||
# T4 / T5 CDP request helper. Env: CDP_HTTP SCHEME TIER URL
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
CDP_HTTP="${CDP_HTTP:-http://127.0.0.1:19222}"
|
||||
SCHEME="${SCHEME:-lightpanda}"
|
||||
TIER="${1:?tier}"
|
||||
URL="${2:?url}"
|
||||
TIMEOUT="${3:-25}"
|
||||
OUT="$(node "$ROOT/cdp_fetch.mjs" "$CDP_HTTP" "$URL" "$TIMEOUT")"
|
||||
python3 - "$OUT" "$SCHEME" "$TIER" "$URL" <<'PY'
|
||||
import json, sys, datetime, re
|
||||
raw, scheme, tier, url = sys.argv[1:5]
|
||||
try:
|
||||
d = json.loads(raw)
|
||||
except Exception as e:
|
||||
d = {"ok": False, "error": f"json {e}", "raw": raw[:500]}
|
||||
title = d.get("title") or ""
|
||||
text = d.get("text") or ""
|
||||
price = d.get("price") or ""
|
||||
jprice = d.get("jsonldPrice") or ""
|
||||
quote_n = int(d.get("quoteCount") or 0)
|
||||
blocked = bool(d.get("blocked"))
|
||||
vendor = d.get("challenge_vendor") or "none"
|
||||
lc = (title + "\n" + text).lower()
|
||||
assert_ok = False
|
||||
note = ""
|
||||
if tier == "T4-ctrl":
|
||||
assert_ok = ("quote" in lc) or quote_n >= 3 or bool(re.search(r"albert einstein|jane austen|marilyn monroe|steve jobs", lc))
|
||||
if not assert_ok:
|
||||
note = "js_kernel_or_empty"
|
||||
elif tier == "T4-shop":
|
||||
assert_ok = (("wool runner" in lc) or ("allbirds" in lc)) and (
|
||||
bool(re.search(r"\$[0-9]", price or text)) or bool(re.search(r"[0-9]", jprice))
|
||||
)
|
||||
if not assert_ok:
|
||||
note = "shop_fields_missing"
|
||||
elif tier == "T4-amz":
|
||||
if blocked or re.search(r"robot|captcha|sorry|click the button|continue shopping", lc):
|
||||
assert_ok = True # interception is expected; not a scheme fail
|
||||
note = "waf"
|
||||
blocked = True
|
||||
else:
|
||||
assert_ok = bool(title.strip()) and not blocked
|
||||
elif tier == "T4-medium":
|
||||
assert_ok = bool(title.strip()) and not blocked and len(text) >= 200
|
||||
if blocked:
|
||||
note = "medium_wall_or_waf"
|
||||
elif not assert_ok:
|
||||
note = "medium_thin"
|
||||
elif tier == "T4-t3x":
|
||||
assert_ok = bool(title.strip()) and len(text) >= 80
|
||||
if not assert_ok:
|
||||
note = "serp_thin"
|
||||
elif tier == "T5":
|
||||
assert_ok = ("quote" in lc) or quote_n >= 3
|
||||
if not assert_ok:
|
||||
note = "t5_ctrl_fail"
|
||||
else:
|
||||
assert_ok = bool(d.get("ok")) and bool(title)
|
||||
|
||||
out = {
|
||||
"ts": datetime.datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"scheme": scheme,
|
||||
"template": tier,
|
||||
"url": url,
|
||||
"final_url": d.get("finalUrl") or "",
|
||||
"http_code": 200 if d.get("ok") else 0,
|
||||
"http_status_in_page": None,
|
||||
"t_total": d.get("t_total") or 0,
|
||||
"ttfb": d.get("t_nav_s") or 0,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": d.get("htmlLen") or 0,
|
||||
"html_len": d.get("htmlLen") or 0,
|
||||
"text_len": d.get("textLen") or 0,
|
||||
"title": title[:240],
|
||||
"price": price,
|
||||
"jsonld_price": jprice,
|
||||
"quote_count": quote_n,
|
||||
"blocked": blocked,
|
||||
"challenge_vendor": vendor,
|
||||
"networkidle": d.get("networkidle"),
|
||||
"assert_ok": bool(assert_ok),
|
||||
"note": note or d.get("error") or "",
|
||||
"cdp_mode": d.get("cdp_mode"),
|
||||
"cdp_errors": d.get("cdp_errors") or [],
|
||||
"text_head": text[:500],
|
||||
}
|
||||
print(json.dumps(out, ensure_ascii=False))
|
||||
# excerpt to stderr for samples
|
||||
excerpt = {
|
||||
"tier": tier,
|
||||
"title": title[:200],
|
||||
"final_url": out["final_url"],
|
||||
"html_len": out["html_len"],
|
||||
"text_len": out["text_len"],
|
||||
"price": price,
|
||||
"jsonld_price": jprice,
|
||||
"blocked": blocked,
|
||||
"vendor": vendor,
|
||||
"text_head": text[:800],
|
||||
"html_head": (d.get("htmlHead") or "")[:800],
|
||||
}
|
||||
sys.stderr.write(json.dumps(excerpt, ensure_ascii=False) + "\n")
|
||||
PY
|
||||
323
bench/chrome-headless-shell/results.json
Normal file
323
bench/chrome-headless-shell/results.json
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
{
|
||||
"scheme": "chrome-headless-shell",
|
||||
"group": "foreign_proxy",
|
||||
"image": {
|
||||
"ref": "docker.io/chromedp/headless-shell:151.0.7922.109",
|
||||
"digest": "sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04",
|
||||
"id": "sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04",
|
||||
"arch": "arm64",
|
||||
"pull": "hub"
|
||||
},
|
||||
"probed_at": "2026-09-01T12:10:38+08:00",
|
||||
"commands": [
|
||||
"docker compose -p bench-s3c-headless-shell up -d",
|
||||
"node cdp_fetch.mjs http://127.0.0.1:19223 <url>",
|
||||
"T5 ThreadPoolExecutor workers=1 n=60"
|
||||
],
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "204",
|
||||
"host_docker_internal": "fdc4:f303:9324::254 host.docker.internal",
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway"
|
||||
},
|
||||
"rss_source": "docker_stats+cgroup",
|
||||
"idle_mb_p50": 37.46,
|
||||
"session_delta_mb": 356.54,
|
||||
"burst_peak_mb": 179.3,
|
||||
"session_peak_mb": 394.0,
|
||||
"templates": {
|
||||
"T4-ctrl": {
|
||||
"n": 3,
|
||||
"success": 3,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 1.129,
|
||||
"p95_s": 2.8786,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1499,
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1499,
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1499,
|
||||
"title": "Quotes to Scrape"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-medium": {
|
||||
"n": 3,
|
||||
"success": 1,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 10.956,
|
||||
"p95_s": 13.8855,
|
||||
"assert_ok": true,
|
||||
"note": "medium_wall_or_waf",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 91669,
|
||||
"text_len": 226,
|
||||
"title": "Medium"
|
||||
},
|
||||
{
|
||||
"html_len": 5193,
|
||||
"text_len": 687,
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"html_len": 5193,
|
||||
"text_len": 687,
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-shop": {
|
||||
"n": 3,
|
||||
"success": 3,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 16.363,
|
||||
"p95_s": 20.0818,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 1453525,
|
||||
"text_len": 2886,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"html_len": 1459360,
|
||||
"text_len": 2886,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"html_len": 1462947,
|
||||
"text_len": 2886,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-amz": {
|
||||
"n": 3,
|
||||
"success": 3,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 1.305,
|
||||
"p95_s": 1.9431,
|
||||
"assert_ok": true,
|
||||
"note": "三次均为 title=Page Not Found、text_len=0（软拦截/ASIN 页对无头不可用），未出现明确 captcha 文案",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 2327,
|
||||
"text_len": 0,
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"html_len": 2327,
|
||||
"text_len": 0,
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"html_len": 2326,
|
||||
"text_len": 0,
|
||||
"title": "Page Not Found"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-t3x": {
|
||||
"n": 1,
|
||||
"success": 1,
|
||||
"http_codes": [
|
||||
200
|
||||
],
|
||||
"p50_s": 5.474,
|
||||
"p95_s": 5.474,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 194292,
|
||||
"text_len": 7985,
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"t5": {
|
||||
"n": 60,
|
||||
"parallelism": 1,
|
||||
"queue_impl": "client_fifo",
|
||||
"success": 60,
|
||||
"success_rate": 1.0,
|
||||
"p50_s": 1.1175,
|
||||
"p95_s": 1.3301,
|
||||
"oom": false,
|
||||
"http_429": 0,
|
||||
"http_503": 0,
|
||||
"first_complete_s": 1.9423,
|
||||
"last_complete_s": 73.9736,
|
||||
"queue_wait_p50_s": 36.9965,
|
||||
"queue_wait_p95_s": 69.2738,
|
||||
"rejected": 0,
|
||||
"wall_s": 73.974,
|
||||
"note": "CONCURRENT=1 client_fifo：60 个任务全部入队并顺序执行，无 429/拒绝；墙钟 74s；第 60 个 start_s≈72.9s。生产应是 SQLite WAL + CONCURRENT。"
|
||||
},
|
||||
"t4_waf": [
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Medium"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": true,
|
||||
"vendor": "cloudflare",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": true,
|
||||
"vendor": "cloudflare",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"tier": "T4-t3x",
|
||||
"url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit&ia=web",
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
}
|
||||
],
|
||||
"scorecard": {
|
||||
"mem": 66.7,
|
||||
"quality": 100,
|
||||
"stability": 100,
|
||||
"latency": 100.0,
|
||||
"ops": 55,
|
||||
"total": 85.5
|
||||
},
|
||||
"notes": [
|
||||
"queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT",
|
||||
"image pull: docker.io hub linux/arm64, 无需 daocloud",
|
||||
"compose 不可把带空格的 --user-agent 传给镜像 run.sh（$@ 未加引号）；UA 改由 CDP Network.setUserAgentOverride",
|
||||
"T4-shop: 同样 302 到 /collections/mens，正文可见 $140 等列表价",
|
||||
"T4-medium: 第 1 次 Medium 404 壳页；第 2–3 次 Cloudflare block",
|
||||
"session 峰值 394.0MiB / mem_limit 400m，Allbirds 页，接近 cgroup 上限，未 OOMKilled"
|
||||
]
|
||||
}
|
||||
246
bench/chrome-headless-shell/run.sh
Executable file
246
bench/chrome-headless-shell/run.sh
Executable file
|
|
@ -0,0 +1,246 @@
|
|||
#!/usr/bin/env bash
|
||||
# lightpanda S3c: up → idle 5min → T4 tiers ×3 → T5 (worker=4) → summarize
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$ROOT/_lib.sh"
|
||||
NAME="bench-s3c-headless-shell"
|
||||
IMG="docker.io/chromedp/headless-shell:151.0.7922.109"
|
||||
CDP_HTTP="${CDP_HTTP:-http://127.0.0.1:19223}"
|
||||
SCHEME="${SCHEME:-chrome-headless-shell}"
|
||||
COMPOSE_PROJ="${COMPOSE_PROJ:-bench-s3c-headless-shell}"
|
||||
T5_PAR="${T5_PAR:-1}"
|
||||
RAW="$ROOT/raw"
|
||||
SAMP="$ROOT/samples"
|
||||
mkdir -p "$RAW" "$SAMP"
|
||||
export CDP_HTTP SCHEME
|
||||
exec > >(tee -a "$RAW/run.log") 2>&1
|
||||
|
||||
log "=== $SCHEME S3c start ==="
|
||||
ensure_image "$IMG" "$RAW/pull.how"
|
||||
PULL_HOW="$(cat "$RAW/pull.how" 2>/dev/null || echo unknown)"
|
||||
|
||||
if [[ "${SKIP_UP:-}" != "1" ]]; then
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
log "docker compose up -d project=$COMPOSE_PROJ"
|
||||
(cd "$ROOT" && docker compose -p "$COMPOSE_PROJ" up -d --force-recreate)
|
||||
fi
|
||||
|
||||
log "wait CDP $CDP_HTTP/json/version"
|
||||
if ! wait_http "$CDP_HTTP/json/version" 40; then
|
||||
log "json/version not ready; try /json/list"
|
||||
wait_http "$CDP_HTTP/json/list" 20 || true
|
||||
docker logs --tail 80 "$NAME" | tee "$RAW/container-start.log" || true
|
||||
fi
|
||||
curl -sS --max-time 5 "$CDP_HTTP/json/version" | tee "$RAW/cdp-version.json" || true
|
||||
echo
|
||||
|
||||
ARCH="$(docker image inspect --format '{{.Architecture}}' "$IMG" 2>/dev/null || echo '?')"
|
||||
DIGEST="$(docker inspect --format '{{index .RepoDigests 0}}' "$NAME" 2>/dev/null || true)"
|
||||
IID="$(docker inspect --format '{{.Image}}' "$NAME" 2>/dev/null || true)"
|
||||
log "image arch=$ARCH digest=$DIGEST pull=$PULL_HOW"
|
||||
|
||||
PROXY_CODE="$(curl -sS -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 20 \
|
||||
-x http://127.0.0.1:17890 -A onesvm-bench/s3c \
|
||||
https://www.google.com/generate_204 || true)"
|
||||
log "proxy generate_204 http=$PROXY_CODE"
|
||||
HOST_RES="$(docker exec "$NAME" sh -c 'getent hosts host.docker.internal || true' 2>/dev/null || echo unresolved)"
|
||||
log "host.docker.internal => $HOST_RES"
|
||||
|
||||
if [[ "${SKIP_IDLE:-}" != "1" ]]; then
|
||||
log "idle 300s interval=5"
|
||||
sample_stats "$RAW/idle.tsv" 5 300 "$NAME"
|
||||
log "idle done"
|
||||
else
|
||||
log "SKIP_IDLE=1"
|
||||
fi
|
||||
|
||||
T4_CTRL="https://quotes.toscrape.com/js/"
|
||||
T4_MEDIUM="https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395"
|
||||
T4_SHOP="https://www.allbirds.com/products/mens-wool-runners"
|
||||
T4_AMZ="https://www.amazon.com/dp/B08N5WRWNW"
|
||||
T4_T3X="https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit"
|
||||
|
||||
SENT="$RAW/session.done"
|
||||
rm -f "$SENT"
|
||||
sample_until_file "$RAW/session.tsv" 1 "$SENT" 10 "$NAME" &
|
||||
STAT_PID=$!
|
||||
sleep 10
|
||||
|
||||
: > "$RAW/t4.jsonl"
|
||||
run_tier() {
|
||||
local tier="$1" url="$2" n="$3"
|
||||
local i
|
||||
for i in $(seq 1 "$n"); do
|
||||
log "$tier trial $i url=$url"
|
||||
set +e
|
||||
line="$(bash "$ROOT/request.sh" "$tier" "$url" 25 2>"$RAW/${tier}-${i}.excerpt")"
|
||||
rc=$?
|
||||
set -e
|
||||
echo "$line" | tee -a "$RAW/t4.jsonl"
|
||||
if [[ "$i" -eq 1 ]]; then
|
||||
cp "$RAW/${tier}-${i}.excerpt" "$SAMP/${tier}.json" 2>/dev/null || true
|
||||
fi
|
||||
log "$tier trial $i rc=$rc"
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
run_tier T4-ctrl "$T4_CTRL" 3
|
||||
run_tier T4-medium "$T4_MEDIUM" 3
|
||||
run_tier T4-shop "$T4_SHOP" 3
|
||||
run_tier T4-amz "$T4_AMZ" 3
|
||||
run_tier T4-t3x "$T4_T3X" 1
|
||||
|
||||
touch "$SENT"
|
||||
wait "$STAT_PID" || true
|
||||
|
||||
log "T5 60x T4-ctrl parallelism=$T5_PAR queue=client_fifo"
|
||||
SENT5="$RAW/t5.done"
|
||||
rm -f "$SENT5"
|
||||
sample_until_file "$RAW/t5-stats.tsv" 1 "$SENT5" 15 "$NAME" &
|
||||
STAT5_PID=$!
|
||||
T5_START_MS=$(python3 -c 'import time;print(int(time.time()*1000))')
|
||||
: > "$RAW/t5.jsonl"
|
||||
: > "$RAW/t5-queue.jsonl"
|
||||
|
||||
python3 - "$ROOT" "$T4_CTRL" "$T5_PAR" "$T5_START_MS" "$RAW/t5.jsonl" "$RAW/t5-queue.jsonl" <<'PY'
|
||||
import subprocess, time, json, os, sys
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
root, url, par, t0, outp, qpath = sys.argv[1:7]
|
||||
par = int(par)
|
||||
t0 = int(t0) / 1000.0
|
||||
n = 60
|
||||
req = os.path.join(root, "request.sh")
|
||||
|
||||
def one(i):
|
||||
enq = time.time()
|
||||
p = subprocess.run(["bash", req, "T5", url, "30"], capture_output=True, text=True)
|
||||
end = time.time()
|
||||
line = p.stdout.strip().splitlines()[-1] if p.stdout.strip() else "{}"
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except Exception:
|
||||
row = {"assert_ok": False, "t_total": end - enq, "note": "parse", "http_code": 0}
|
||||
start = end - float(row.get("t_total") or 0)
|
||||
q = {
|
||||
"i": i,
|
||||
"enqueue_s": enq - t0,
|
||||
"start_s": start - t0,
|
||||
"end_s": end - t0,
|
||||
"queue_wait_s": max(0.0, start - enq),
|
||||
"assert_ok": row.get("assert_ok"),
|
||||
"t_total": row.get("t_total"),
|
||||
}
|
||||
return row, q
|
||||
|
||||
rows = []
|
||||
qs = []
|
||||
with ThreadPoolExecutor(max_workers=par) as ex:
|
||||
futs = [ex.submit(one, i) for i in range(1, n + 1)]
|
||||
for f in as_completed(futs):
|
||||
row, q = f.result()
|
||||
rows.append(row)
|
||||
qs.append(q)
|
||||
with open(outp, "w", encoding="utf-8") as f:
|
||||
for r in rows:
|
||||
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
with open(qpath, "w", encoding="utf-8") as f:
|
||||
for q in qs:
|
||||
f.write(json.dumps(q) + "\n")
|
||||
print(f"t5 done n={len(rows)} ok={sum(1 for r in rows if r.get('assert_ok'))}")
|
||||
PY
|
||||
|
||||
T5_END=$(date +%s)
|
||||
# wall from tsv start roughly
|
||||
T5_WALL=$(python3 - "$RAW/t5-queue.jsonl" <<'PY'
|
||||
import json,sys
|
||||
from pathlib import Path
|
||||
qs=[json.loads(l) for l in Path(sys.argv[1]).read_text().splitlines() if l.strip()]
|
||||
if not qs:
|
||||
print(0); raise SystemExit
|
||||
print(round(max(q["end_s"] for q in qs), 3))
|
||||
PY
|
||||
)
|
||||
log "T5 wall=${T5_WALL}s"
|
||||
touch "$SENT5"
|
||||
wait "$STAT5_PID" || true
|
||||
|
||||
python3 - "$RAW/t5.jsonl" "$SAMP" <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
p=Path(sys.argv[1]); d=Path(sys.argv[2])
|
||||
rows=[json.loads(l) for l in p.read_text().splitlines() if l.strip()]
|
||||
for i,r in enumerate(rows[:2],1):
|
||||
(d/f"t5-{i}.json").write_text(json.dumps(r,ensure_ascii=False,indent=2)+"\n")
|
||||
print("saved t5 samples", len(rows))
|
||||
PY
|
||||
|
||||
# html vs text note from first ctrl excerpt
|
||||
python3 - "$SAMP" "$RAW" <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
samp=Path("$SAMP".replace("$SAMP", __import__("os").environ.get("SAMP","")))
|
||||
PY
|
||||
|
||||
OOM="$(docker inspect --format '{{.State.OOMKilled}}' "$NAME" 2>/dev/null || echo false)"
|
||||
STATUS="$(docker inspect --format '{{.State.Status}}' "$NAME" 2>/dev/null || echo unknown)"
|
||||
log "status=$STATUS oom=$OOM"
|
||||
docker logs --tail 50 "$NAME" | tee "$RAW/container-tail.log" || true
|
||||
|
||||
python3 - <<PY
|
||||
import json, datetime, statistics, math
|
||||
from pathlib import Path
|
||||
raw = Path("$RAW")
|
||||
qs = []
|
||||
qp = raw / "t5-queue.jsonl"
|
||||
if qp.exists():
|
||||
qs = [json.loads(l) for l in qp.read_text().splitlines() if l.strip()]
|
||||
def pct(xs, p):
|
||||
if not xs: return None
|
||||
s=sorted(xs)
|
||||
k=(len(s)-1)*p/100
|
||||
f=int(k); c=int(math.ceil(k))
|
||||
if f==c: return round(s[f],4)
|
||||
return round(s[f]+(s[c]-s[f])*(k-f),4)
|
||||
waits=[q.get("queue_wait_s") or 0 for q in qs]
|
||||
ends=[q.get("end_s") or 0 for q in qs]
|
||||
meta = {
|
||||
"ref": "$IMG",
|
||||
"digest": None if "$DIGEST" in ("<no value>", "") else "$DIGEST",
|
||||
"id": "$IID",
|
||||
"arch": "$ARCH",
|
||||
"pull": "$PULL_HOW",
|
||||
"probed_at": datetime.datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"oom": str("$OOM").lower()=="true",
|
||||
"container_status": "$STATUS",
|
||||
"t5_wall_s": float("$T5_WALL" or 0),
|
||||
"t5_parallelism": int("$T5_PAR"),
|
||||
"queue_impl": "client_fifo",
|
||||
"queue": {
|
||||
"first_complete_s": round(min(ends),4) if ends else None,
|
||||
"last_complete_s": round(max(ends),4) if ends else None,
|
||||
"queue_wait_p50_s": pct(waits,50),
|
||||
"queue_wait_p95_s": pct(waits,95),
|
||||
"rejected": 0,
|
||||
},
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "$PROXY_CODE",
|
||||
"host_docker_internal": """$HOST_RES""".strip(),
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway",
|
||||
},
|
||||
"commands": [
|
||||
"docker compose -p $COMPOSE_PROJ up -d",
|
||||
"node cdp_fetch.mjs $CDP_HTTP <url>",
|
||||
"T5 ThreadPoolExecutor workers=$T5_PAR n=60",
|
||||
],
|
||||
"notes": ["queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT"],
|
||||
}
|
||||
(raw/"image-meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2)+"\n")
|
||||
PY
|
||||
|
||||
python3 "$ROOT/summarize.py"
|
||||
log "=== $SCHEME S3c done ==="
|
||||
1
bench/chrome-headless-shell/samples/T4-amz.json
Normal file
1
bench/chrome-headless-shell/samples/T4-amz.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Page Not Found", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 2327, "text_len": 0, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "", "html_head": "<html><head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, shrink-to-fit=no\">\n <title>\n Page Not Found\n </title>\n <style>\n html,body{padding:0;margin:0}img{border:0}#a{background:#232f3e;padding:11px 11px 11px 192px}#b{position:absolute;left:22px;top:12px}#c{position:relative;max-width:800px;padding:0 40px 0 0}#e,#f{height:35px;border:0;font-size:1em}#e{width:100%;margin:0;padding:0 10px;border-radius:4px 0 0 4px}#f{cursor:pointer;background:#febd69;font-weight:bold;border-radius:0 4px 4px 0;-webkit-appearance:none;position:absolute;top:0;right:0;padding:0 12px}@media(max-width:500px){#a{padding:55px 10px 10px}#b{left:6px}}#g{text-align:center;margin:30px 0}#g i"}
|
||||
1
bench/chrome-headless-shell/samples/T4-ctrl.json
Normal file
1
bench/chrome-headless-shell/samples/T4-ctrl.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1499, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”\nby Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”\nby Mar", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/chrome-headless-shell/samples/T4-medium.json
Normal file
1
bench/chrome-headless-shell/samples/T4-medium.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Medium", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 91669, "text_len": 226, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Open in app\n\nSign up\n\nSign in\n\nWrite\nPAGE NOT FOUND\n404\nOut of nothing, something.\nYou can find (just about) anything on Medium — apparently even a page that doesn’t exist. Maybe these stories will take you somewhere new?\nHome", "html_head": "<html lang=\"en\" data-rh-managed=\"lang\"><head><meta http-equiv=\"origin-trial\" content=\"A7vZI3v+Gz7JfuRolKNM4Aff6zaGuT7X0mf3wtoZTnKv6497cVMnhy03KDqX7kBz/q/iidW7srW31oQbBt4VhgoAAACUeyJvcmlnaW4iOiJodHRwczovL3d3dy5nb29nbGUuY29tOjQ0MyIsImZlYXR1cmUiOiJEaXNhYmxlVGhpcmRQYXJ0eVN0b3JhZ2VQYXJ0aXRpb25pbmczIiwiZXhwaXJ5IjoxNzU3OTgwODAwLCJpc1N1YmRvbWFpbiI6dHJ1ZSwiaXNUaGlyZFBhcnR5Ijp0cnVlfQ==\"><meta charset=\"utf-8\"><meta charset=\"utf-8\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><meta name=\"viewport\" content=\"width=device-width,minimum-scale=1,initial-scale=1,maximum-scale=1\"><style data-precedence=\"high\" data-href=\"lite-color-scheme-tokens\">:root{color-scheme:light;--invert-in-dark-mode: 0;--blend-in-dark-mode: normal;--color-bg-neutral-primary: #ffffff;--color-bg-neutral-secondar"}
|
||||
1
bench/chrome-headless-shell/samples/T4-shop.json
Normal file
1
bench/chrome-headless-shell/samples/T4-shop.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1453525, "text_len": 2886, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nCART (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nSHOP WOMENS\nSHOP MENS\nSHOP SOCKS\nSHOP WOMEN'S SALE\nSHOP MEN'S SALE\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFILTER AND SORT\nMEN\nWOMEN\nNEW\n\nMEN'S DASHER NZ\n\nAnthracite\n\n$140\n\nMEN'S TREE RUNNER NZ\n\nMedium Grey\n\n$100\n\nMEN'S TREE RUNNER\n\nMist\n\n$100\n\nMEN'S TREE RUNNER\n\nWheat\n\n$100\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nAnthracite\n\n$50\n\nNEW\n\nMEN'S ALLBIRDS FLIP FLOP\n\nMushroom\n\n$50\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nAnthracite\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nDark Navy\n\n$105\n\nNEW\n\nMEN'S RUNNER NZ SLIP ON\n\nLight", "html_head": "<html class=\"h-full\" lang=\"en-US\" style=\"--global-banner-height: 32px; --global-banner-offset: 32px;\"><head><script async=\"\" src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script type=\"text/javascript\" async=\"\" src=\"https://cdn.attn.tv/allbirds/dtag.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async=\"\" src=\"https://id-shop.govx.com/app/weareallbirds.myshopify.com/govx.js?shop=weareallbirds.myshopify.com\"></script><script src=\"https://connect.facebook.net/signals/config/1689016458013442?v=2.9.390&r=stable&domain=www.allbirds.com&hme=a767ae4c6a92a40ee4d7e358a937ad845c8bdb98a3bbe3412069dded3a7be57d&ex_m=115%2C221%2C166%2C26%2C80%2C81%2C158%2C76%2C75%2C11%2C176%2C100%2C20%2C150%2C138%2C45%2C83%2C88%2C146%2C171%"}
|
||||
1
bench/chrome-headless-shell/samples/T4-t3x.json
Normal file
1
bench/chrome-headless-shell/samples/T4-t3x.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-t3x", "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit&ia=web", "html_len": 194292, "text_len": 7985, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "DuckDuckGo\nOpen menu\n All\nImages\nNews\nVideos\nMore\nSearch Assist\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nResults from Reddit\nAll\nr/Earbuds\nr/GoodValue\nr/BuyItForLife\nr/HeadphoneAdvice\n\nr/Earbuds\n\nhttps://www.reddit.com\n › r › Earbuds › comments › 1q7ky9h › best_wireless_earbuds_2026_whats_actually_good_now\n\nbest wireless earbuds 2026. what's actually good now?\n\nPersonal fav: Noble Fokus Rex5. Amazing sound. Second fav: Soundpeats H3. Great price and great sound. I am waiting for the Geekfly Vitas.\n\nPosted 7 months ago\n\nr/Earbuds\n\nhttps://www.reddit.com\n › r › Earbuds › comments › 1utpkfd › best_earbuds_to_buy_in_2026_price_anc_battery_life\n\nBest Earbuds To Buy In 2026? (Price, ANC, Battery Life)\n\nFor me JBl Tune Buds 2 hands down, and. Soundcore P40i and C", "html_head": "<html lang=\"en-US\" class=\"has-zcm is-link-style-exp is-link-order-exp is-link-breadcrumb-exp is-related-search-exp is-vertical-tabs-exp js no-touch opacity csstransforms3d csstransitions svg cssfilters is-not-mobile-device full-urls breadcrumb-urls react has-footer\" style=\"--sds-color-palette-black: #000000; --sds-color-palette-white: #FFFFFF; --sds-color-palette-transparent: transparent; --sds-color-palette-gray-100: #111111; --sds-color-palette-gray-95: #161616; --sds-color-palette-gray-90: #222222; --sds-color-palette-gray-85: #333333; --sds-color-palette-gray-80: #444444; --sds-color-palette-gray-70: #666666; --sds-color-palette-gray-60: #888888; --sds-color-palette-gray-50: #aaaaaa; --sds-color-palette-gray-40: #cccccc; --sds-color-palette-gray-30: #dddddd; --sds-color-palette-gra"}
|
||||
28
bench/chrome-headless-shell/samples/t5-1.json
Normal file
28
bench/chrome-headless-shell/samples/t5-1.json
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"ts": "2026-09-01T12:09:07+08:00",
|
||||
"scheme": "chrome-headless-shell",
|
||||
"template": "T5",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"http_code": 200,
|
||||
"http_status_in_page": null,
|
||||
"t_total": 1.826,
|
||||
"ttfb": 1.251,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": 8925,
|
||||
"html_len": 8925,
|
||||
"text_len": 1499,
|
||||
"title": "Quotes to Scrape",
|
||||
"price": "",
|
||||
"jsonld_price": "",
|
||||
"quote_count": 10,
|
||||
"blocked": false,
|
||||
"challenge_vendor": "none",
|
||||
"networkidle": true,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"cdp_mode": "version",
|
||||
"cdp_errors": [],
|
||||
"text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"
|
||||
}
|
||||
28
bench/chrome-headless-shell/samples/t5-2.json
Normal file
28
bench/chrome-headless-shell/samples/t5-2.json
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"ts": "2026-09-01T12:09:08+08:00",
|
||||
"scheme": "chrome-headless-shell",
|
||||
"template": "T5",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"http_code": 200,
|
||||
"http_status_in_page": null,
|
||||
"t_total": 1.121,
|
||||
"ttfb": 0.555,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": 8925,
|
||||
"html_len": 8925,
|
||||
"text_len": 1499,
|
||||
"title": "Quotes to Scrape",
|
||||
"price": "",
|
||||
"jsonld_price": "",
|
||||
"quote_count": 10,
|
||||
"blocked": false,
|
||||
"challenge_vendor": "none",
|
||||
"networkidle": true,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"cdp_mode": "version",
|
||||
"cdp_errors": [],
|
||||
"text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”\nby Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”\nby J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”\nby Albert Einstein\nTags: inspirational l"
|
||||
}
|
||||
275
bench/chrome-headless-shell/summarize.py
Normal file
275
bench/chrome-headless-shell/summarize.py
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build lightpanda/results.json from raw artifacts. No secrets."""
|
||||
import json, math, os, statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
RAW = ROOT / "raw"
|
||||
SCHEME = "chrome-headless-shell"
|
||||
|
||||
|
||||
def percentile(xs, p):
|
||||
if not xs:
|
||||
return None
|
||||
s = sorted(xs)
|
||||
if len(s) == 1:
|
||||
return round(s[0], 4)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = math.floor(k)
|
||||
c = math.ceil(k)
|
||||
if f == c:
|
||||
return round(s[int(k)], 4)
|
||||
return round(s[f] + (s[c] - s[f]) * (k - f), 4)
|
||||
|
||||
|
||||
def load_jsonl(p):
|
||||
rows = []
|
||||
if not p.exists():
|
||||
return rows
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
rows.append(json.loads(line))
|
||||
return rows
|
||||
|
||||
|
||||
def tsv_mibs(p):
|
||||
vals = []
|
||||
if not p.exists():
|
||||
return vals
|
||||
lines = p.read_text(encoding="utf-8").splitlines()
|
||||
if len(lines) < 2:
|
||||
return vals
|
||||
hdr = lines[0].split("\t")
|
||||
i = hdr.index("mem_mib") if "mem_mib" in hdr else 4
|
||||
for line in lines[1:]:
|
||||
parts = line.split("\t")
|
||||
if len(parts) <= i:
|
||||
continue
|
||||
try:
|
||||
v = float(parts[i])
|
||||
except ValueError:
|
||||
continue
|
||||
if v > 0:
|
||||
vals.append(v)
|
||||
return vals
|
||||
|
||||
|
||||
def median(xs):
|
||||
return round(statistics.median(xs), 3) if xs else None
|
||||
|
||||
|
||||
def pmax(xs):
|
||||
return round(max(xs), 3) if xs else None
|
||||
|
||||
|
||||
def mem_score(idle, delta, burst):
|
||||
def band(v, cuts):
|
||||
if v is None:
|
||||
return None
|
||||
for score, lim in cuts:
|
||||
if v <= lim:
|
||||
return score
|
||||
return 0
|
||||
parts = [
|
||||
band(idle, [(100, 60), (80, 100), (60, 200), (40, 400)]),
|
||||
band(delta, [(100, 30), (80, 80), (60, 200), (40, 350)]),
|
||||
band(burst, [(100, 200), (80, 400), (60, 700), (40, 1000)]),
|
||||
]
|
||||
parts = [x for x in parts if x is not None]
|
||||
return round(sum(parts) / 3, 1) if parts else None
|
||||
|
||||
|
||||
def quality_js(t4):
|
||||
ctrl = [r for r in t4 if r.get("template") == "T4-ctrl"]
|
||||
shop = [r for r in t4 if r.get("template") == "T4-shop"]
|
||||
if any(r.get("assert_ok") for r in ctrl):
|
||||
if any(r.get("assert_ok") and not r.get("note") == "shop_fields_missing" for r in shop) or any(
|
||||
r.get("assert_ok") for r in shop
|
||||
):
|
||||
return 100
|
||||
# ctrl ok, shop fields missing but text present
|
||||
if shop and any((r.get("text_len") or 0) > 200 for r in shop):
|
||||
return 70
|
||||
return 70
|
||||
if any((r.get("text_len") or 0) > 100 for r in t4):
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def stab_score(rate, oom):
|
||||
if oom:
|
||||
return 0
|
||||
if rate is None:
|
||||
return None
|
||||
if rate >= 0.99:
|
||||
return 100
|
||||
if rate >= 0.95:
|
||||
return 80
|
||||
if rate >= 0.85:
|
||||
return 60
|
||||
if rate >= 0.70:
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def lat_t4(p95):
|
||||
if p95 is None:
|
||||
return None
|
||||
if p95 <= 5:
|
||||
return 100
|
||||
if p95 <= 12:
|
||||
return 70
|
||||
if p95 <= 25:
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def lat_t5(t5_p95, t4_p95):
|
||||
if t5_p95 is None:
|
||||
return None
|
||||
if t4_p95 and t4_p95 > 0:
|
||||
ratio = t5_p95 / t4_p95
|
||||
if ratio <= 3:
|
||||
return 100
|
||||
if ratio <= 5:
|
||||
return 70
|
||||
if ratio <= 8:
|
||||
return 40
|
||||
return 0
|
||||
return 40 if t5_p95 <= 25 else 0
|
||||
|
||||
|
||||
def main():
|
||||
t4 = load_jsonl(RAW / "t4.jsonl")
|
||||
t5 = load_jsonl(RAW / "t5.jsonl")
|
||||
idle = tsv_mibs(RAW / "idle.tsv")
|
||||
sess = tsv_mibs(RAW / "session.tsv")
|
||||
burst = tsv_mibs(RAW / "t5-stats.tsv")
|
||||
meta = {}
|
||||
if (RAW / "image-meta.json").exists():
|
||||
meta = json.loads((RAW / "image-meta.json").read_text(encoding="utf-8"))
|
||||
|
||||
idle_p50 = median(idle)
|
||||
sess_peak = pmax(sess)
|
||||
burst_peak = pmax(burst)
|
||||
delta = None
|
||||
if idle_p50 is not None and sess_peak is not None:
|
||||
delta = round(max(0.0, sess_peak - idle_p50), 3)
|
||||
|
||||
def pack(rows, key):
|
||||
sub = [r for r in rows if r.get("template") == key]
|
||||
times = [r["t_total"] for r in sub if r.get("t_total") is not None]
|
||||
return {
|
||||
"n": len(sub),
|
||||
"success": sum(1 for r in sub if r.get("assert_ok")),
|
||||
"http_codes": [r.get("http_code") for r in sub],
|
||||
"p50_s": percentile(times, 50),
|
||||
"p95_s": percentile(times, 95),
|
||||
"assert_ok": any(r.get("assert_ok") for r in sub),
|
||||
"note": "; ".join(sorted({r.get("note") for r in sub if r.get("note")})),
|
||||
"html_vs_text": [
|
||||
{"html_len": r.get("html_len"), "text_len": r.get("text_len"), "title": r.get("title")}
|
||||
for r in sub[:3]
|
||||
],
|
||||
}
|
||||
|
||||
t4_ctrl_times = [r["t_total"] for r in t4 if r.get("template") == "T4-ctrl" and r.get("t_total") is not None]
|
||||
t5_ok = [r for r in t5 if r.get("assert_ok")]
|
||||
t5_times = [r["t_total"] for r in t5 if r.get("t_total") is not None]
|
||||
t5_n = len(t5)
|
||||
t5_rate = (len(t5_ok) / t5_n) if t5_n else None
|
||||
# T5 assertion uses T4-ctrl quality; scheme stability also considers T4-ctrl
|
||||
ctrl = [r for r in t4 if r.get("template") == "T4-ctrl"]
|
||||
ctrl_rate = (sum(1 for r in ctrl if r.get("assert_ok")) / len(ctrl)) if ctrl else None
|
||||
combined_rate = t5_rate if t5_rate is not None else ctrl_rate
|
||||
|
||||
t4_waf = []
|
||||
for r in t4:
|
||||
t4_waf.append({
|
||||
"tier": r.get("template"),
|
||||
"url": r.get("url"),
|
||||
"blocked": r.get("blocked"),
|
||||
"vendor": r.get("challenge_vendor"),
|
||||
"http_status_in_page": r.get("http_status_in_page"),
|
||||
"final_url": r.get("final_url"),
|
||||
"title": r.get("title"),
|
||||
})
|
||||
|
||||
oom = bool(meta.get("oom"))
|
||||
q = quality_js(t4)
|
||||
t4_p95 = percentile(t4_ctrl_times, 95)
|
||||
t5_p95 = percentile(t5_times, 95)
|
||||
l4 = lat_t4(t4_p95)
|
||||
l5 = lat_t5(t5_p95, t4_p95)
|
||||
lat_parts = [x for x in (l4, l5) if x is not None]
|
||||
lat = round(sum(lat_parts) / len(lat_parts), 1) if lat_parts else None
|
||||
mem = mem_score(idle_p50, delta, burst_peak)
|
||||
stab = stab_score(combined_rate, oom)
|
||||
ops = 70 if SCHEME == "lightpanda" else 55
|
||||
total = None
|
||||
if None not in (mem, q, stab, lat, ops):
|
||||
total = round(mem * 0.30 + q * 0.25 + stab * 0.20 + lat * 0.15 + ops * 0.10, 1)
|
||||
|
||||
queue = meta.get("queue") or {}
|
||||
out = {
|
||||
"scheme": SCHEME,
|
||||
"group": "foreign_proxy",
|
||||
"image": {
|
||||
"ref": meta.get("ref"),
|
||||
"digest": meta.get("digest"),
|
||||
"id": meta.get("id"),
|
||||
"arch": meta.get("arch"),
|
||||
"pull": meta.get("pull"),
|
||||
},
|
||||
"probed_at": meta.get("probed_at"),
|
||||
"commands": meta.get("commands") or [],
|
||||
"proxy": meta.get("proxy") or {},
|
||||
"rss_source": "docker_stats+cgroup",
|
||||
"idle_mb_p50": idle_p50,
|
||||
"session_delta_mb": delta,
|
||||
"burst_peak_mb": burst_peak,
|
||||
"session_peak_mb": sess_peak,
|
||||
"templates": {
|
||||
"T4-ctrl": pack(t4, "T4-ctrl"),
|
||||
"T4-medium": pack(t4, "T4-medium"),
|
||||
"T4-shop": pack(t4, "T4-shop"),
|
||||
"T4-amz": pack(t4, "T4-amz"),
|
||||
"T4-t3x": pack(t4, "T4-t3x"),
|
||||
},
|
||||
"t5": {
|
||||
"n": t5_n,
|
||||
"parallelism": meta.get("t5_parallelism") or 4,
|
||||
"queue_impl": meta.get("queue_impl") or "client_fifo",
|
||||
"success": len(t5_ok),
|
||||
"success_rate": t5_rate,
|
||||
"p50_s": percentile(t5_times, 50),
|
||||
"p95_s": t5_p95,
|
||||
"oom": oom,
|
||||
"http_429": 0,
|
||||
"http_503": 0,
|
||||
"first_complete_s": queue.get("first_complete_s"),
|
||||
"last_complete_s": queue.get("last_complete_s"),
|
||||
"queue_wait_p50_s": queue.get("queue_wait_p50_s"),
|
||||
"queue_wait_p95_s": queue.get("queue_wait_p95_s"),
|
||||
"rejected": queue.get("rejected") or 0,
|
||||
"wall_s": meta.get("t5_wall_s"),
|
||||
"note": "生产应是 SQLite WAL + CONCURRENT；本轮用脚本侧 worker 池代替",
|
||||
},
|
||||
"t4_waf": t4_waf,
|
||||
"scorecard": {
|
||||
"mem": mem,
|
||||
"quality": q,
|
||||
"stability": stab,
|
||||
"latency": lat,
|
||||
"ops": ops,
|
||||
"total": total,
|
||||
},
|
||||
"notes": meta.get("notes") or [],
|
||||
}
|
||||
(ROOT / "results.json").write_text(json.dumps(out, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps({"wrote": str(ROOT / "results.json"), "scorecard": out["scorecard"]}, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
122
bench/lightpanda/_lib.sh
Normal file
122
bench/lightpanda/_lib.sh
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
# shared helpers for S3c bench (no secrets)
|
||||
set -euo pipefail
|
||||
|
||||
log() { printf '[%s] %s\n' "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$*"; }
|
||||
|
||||
parse_mem_mib() {
|
||||
python3 - "$1" <<'PY'
|
||||
import re, sys
|
||||
s = sys.argv[1].split("/")[0].strip()
|
||||
m = re.match(r"([0-9.]+)\s*([A-Za-z]+)", s)
|
||||
if not m:
|
||||
print("0")
|
||||
raise SystemExit
|
||||
v, u = float(m.group(1)), m.group(2).lower()
|
||||
mult = {"b":1,"kib":1024,"kb":1000,"mib":1024**2,"mb":1000**2,"gib":1024**3,"gb":1000**3}.get(u)
|
||||
if not mult:
|
||||
print("0"); raise SystemExit
|
||||
print(f"{v * mult / (1024**2):.3f}")
|
||||
PY
|
||||
}
|
||||
|
||||
cgroup_mib() {
|
||||
local c="$1" raw=""
|
||||
raw="$(docker exec "$c" sh -c 'cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null' 2>/dev/null || true)"
|
||||
if [[ -z "$raw" ]]; then
|
||||
echo ""
|
||||
return
|
||||
fi
|
||||
python3 -c "print(f'{int(\"$raw\")/(1024*1024):.3f}')"
|
||||
}
|
||||
|
||||
sample_stats() {
|
||||
local out="$1" interval="$2" duration="$3"
|
||||
shift 3
|
||||
mkdir -p "$(dirname "$out")"
|
||||
echo -e "ts_iso\tname\tmem_usage\tcpu_pct\tmem_mib\tcgroup_mib" > "$out"
|
||||
local end=$(( $(date +%s) + duration ))
|
||||
while [ "$(date +%s)" -lt "$end" ]; do
|
||||
local ts
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
local name mem cpu
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
local mib cg
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
}
|
||||
|
||||
sample_until_file() {
|
||||
local out="$1" interval="$2" sentinel="$3" extra_s="$4"
|
||||
shift 4
|
||||
mkdir -p "$(dirname "$out")"
|
||||
echo -e "ts_iso\tname\tmem_usage\tcpu_pct\tmem_mib\tcgroup_mib" > "$out"
|
||||
while [[ ! -f "$sentinel" ]]; do
|
||||
local ts name mem cpu mib cg
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
local end=$(( $(date +%s) + extra_s ))
|
||||
while [ "$(date +%s)" -lt "$end" ]; do
|
||||
local ts name mem cpu mib cg
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
while IFS='|' read -r name mem cpu; do
|
||||
[[ -z "${name:-}" ]] && continue
|
||||
mib="$(parse_mem_mib "$mem")"
|
||||
cg="$(cgroup_mib "$name")"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" "$mib" "${cg:-}" >> "$out"
|
||||
done < <(docker stats --no-stream --format '{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}' "$@" 2>/dev/null || true)
|
||||
sleep "$interval"
|
||||
done
|
||||
}
|
||||
|
||||
ensure_image() {
|
||||
local img="$1" note="${2:-}"
|
||||
local platform="linux/arm64"
|
||||
local how="local"
|
||||
if docker image inspect "$img" >/dev/null 2>&1; then
|
||||
local arch
|
||||
arch="$(docker image inspect --format '{{.Architecture}}' "$img")"
|
||||
if [[ "$arch" == "arm64" || "$arch" == "arm64/v8" ]]; then
|
||||
log "image present $img arch=$arch"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
return 0
|
||||
fi
|
||||
log "image $img arch=$arch not arm64; re-pull"
|
||||
fi
|
||||
log "docker pull --platform $platform $img"
|
||||
if docker pull --platform "$platform" "$img"; then
|
||||
how="hub"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
return 0
|
||||
fi
|
||||
local short="${img#docker.io/}"
|
||||
local mirror="docker.m.daocloud.io/${short}"
|
||||
log "primary pull failed; retry $mirror"
|
||||
docker pull --platform "$platform" "$mirror"
|
||||
docker tag "$mirror" "$img"
|
||||
how="daocloud_mirror"
|
||||
[[ -n "$note" ]] && echo "$how" > "$note"
|
||||
}
|
||||
|
||||
wait_http() {
|
||||
local url="$1" tries="${2:-40}"
|
||||
local i
|
||||
for i in $(seq 1 "$tries"); do
|
||||
if curl -sf --max-time 2 "$url" >/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
358
bench/lightpanda/cdp_fetch.mjs
Normal file
358
bench/lightpanda/cdp_fetch.mjs
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Minimal CDP navigate+extract via native WebSocket (Node 22+).
|
||||
* Usage: node cdp_fetch.mjs <cdp_http> <url> [nav_timeout_s]
|
||||
* Prints one JSON object to stdout. No secrets.
|
||||
*/
|
||||
const cdpHttp = process.argv[2] || "http://127.0.0.1:19222";
|
||||
const targetUrl = process.argv[3];
|
||||
const navTimeoutS = Number(process.argv[4] || 25);
|
||||
if (!targetUrl) {
|
||||
console.error("usage: node cdp_fetch.mjs <cdp_http> <url> [nav_timeout_s]");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const UA =
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.109 Safari/537.36";
|
||||
|
||||
const EXTRACT_JS = `(() => {
|
||||
const title = document.title || "";
|
||||
const bodyText = (document.body && document.body.innerText) ? document.body.innerText : "";
|
||||
const html = document.documentElement ? document.documentElement.outerHTML : "";
|
||||
let price = "";
|
||||
const meta = document.querySelector('meta[itemprop="price"], meta[property="product:price:amount"]');
|
||||
if (meta) price = (meta.getAttribute("content") || "").trim();
|
||||
if (!price) {
|
||||
const el = document.querySelector('[class*="price"], [data-testid*="price"], .product-price, #priceblock_ourprice, .a-price .a-offscreen');
|
||||
if (el) price = (el.textContent || "").trim().slice(0, 80);
|
||||
}
|
||||
let jsonldPrice = "";
|
||||
for (const s of document.querySelectorAll('script[type="application/ld+json"]')) {
|
||||
try {
|
||||
const j = JSON.parse(s.textContent || "null");
|
||||
const arr = Array.isArray(j) ? j : [j];
|
||||
const walk = (o) => {
|
||||
if (!o || typeof o !== "object") return;
|
||||
if (o.offers) {
|
||||
const off = Array.isArray(o.offers) ? o.offers[0] : o.offers;
|
||||
if (off && (off.price || off.lowPrice)) jsonldPrice = String(off.price || off.lowPrice);
|
||||
}
|
||||
if (Array.isArray(o["@graph"])) o["@graph"].forEach(walk);
|
||||
};
|
||||
arr.forEach(walk);
|
||||
} catch (e) {}
|
||||
}
|
||||
const quoteCount = document.querySelectorAll(".quote").length;
|
||||
const lc = (title + "\\n" + bodyText).toLowerCase();
|
||||
return {
|
||||
title,
|
||||
text: bodyText.slice(0, 8000),
|
||||
htmlLen: html.length,
|
||||
textLen: bodyText.length,
|
||||
price,
|
||||
jsonldPrice,
|
||||
quoteCount,
|
||||
htmlHead: html.slice(0, 1800),
|
||||
finalUrl: location.href,
|
||||
readyState: document.readyState,
|
||||
looksBlocked: /(robot|captcha|sorry|click the button|continue shopping|just a moment|attention required|verify you are human|are you a human)/i.test(lc),
|
||||
};
|
||||
})()`;
|
||||
|
||||
function rewriteWs(wsUrl, httpEndpoint) {
|
||||
const u = new URL(wsUrl);
|
||||
const h = new URL(httpEndpoint);
|
||||
u.protocol = h.protocol === "https:" ? "wss:" : "ws:";
|
||||
u.hostname = h.hostname;
|
||||
u.port = h.port;
|
||||
return u.toString();
|
||||
}
|
||||
|
||||
async function httpJson(url) {
|
||||
const r = await fetch(url, { signal: AbortSignal.timeout(5000) });
|
||||
if (!r.ok) throw new Error(`GET ${url} ${r.status}`);
|
||||
return r.json();
|
||||
}
|
||||
|
||||
class Cdp {
|
||||
constructor(wsUrl) {
|
||||
this.wsUrl = wsUrl;
|
||||
this.ws = null;
|
||||
this.nextId = 1;
|
||||
this.pending = new Map();
|
||||
this.events = [];
|
||||
this.handlers = new Map();
|
||||
}
|
||||
async open() {
|
||||
this.ws = new WebSocket(this.wsUrl);
|
||||
await new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => reject(new Error("ws open timeout")), 8000);
|
||||
this.ws.addEventListener("open", () => {
|
||||
clearTimeout(t);
|
||||
resolve();
|
||||
});
|
||||
this.ws.addEventListener("error", (e) => {
|
||||
clearTimeout(t);
|
||||
reject(e);
|
||||
});
|
||||
});
|
||||
this.ws.addEventListener("message", (ev) => {
|
||||
let msg;
|
||||
try {
|
||||
msg = JSON.parse(ev.data);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (msg.id && this.pending.has(msg.id)) {
|
||||
const { resolve, reject } = this.pending.get(msg.id);
|
||||
this.pending.delete(msg.id);
|
||||
if (msg.error) reject(new Error(JSON.stringify(msg.error)));
|
||||
else resolve(msg.result || {});
|
||||
return;
|
||||
}
|
||||
if (msg.method) {
|
||||
this.events.push(msg);
|
||||
const hs = this.handlers.get(msg.method) || [];
|
||||
for (const h of hs) h(msg.params || {}, msg.sessionId);
|
||||
}
|
||||
});
|
||||
}
|
||||
on(method, fn) {
|
||||
const arr = this.handlers.get(method) || [];
|
||||
arr.push(fn);
|
||||
this.handlers.set(method, arr);
|
||||
}
|
||||
send(method, params = {}, sessionId) {
|
||||
const id = this.nextId++;
|
||||
const payload = { id, method, params };
|
||||
if (sessionId) payload.sessionId = sessionId;
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
this.pending.delete(id);
|
||||
reject(new Error(`cdp timeout ${method}`));
|
||||
}, 20000);
|
||||
this.pending.set(id, {
|
||||
resolve: (v) => {
|
||||
clearTimeout(t);
|
||||
resolve(v);
|
||||
},
|
||||
reject: (e) => {
|
||||
clearTimeout(t);
|
||||
reject(e);
|
||||
},
|
||||
});
|
||||
this.ws.send(JSON.stringify(payload));
|
||||
});
|
||||
}
|
||||
close() {
|
||||
try {
|
||||
this.ws && this.ws.close();
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
|
||||
function detectVendor(ex) {
|
||||
const blob = `${ex.title || ""}\n${ex.text || ""}\n${ex.htmlHead || ""}`.toLowerCase();
|
||||
if (/cloudflare|cf-challenge|just a moment|cf-turnstile/.test(blob)) return "cloudflare";
|
||||
if (/datadome|captcha-delivery/.test(blob)) return "datadome";
|
||||
if (/akamai|access denied/.test(blob) && /akamai/.test(blob)) return "akamai";
|
||||
if (/amazon|opfcaptcha|validatecaptcha|enter the characters you see/.test(blob) && /robot|captcha/.test(blob))
|
||||
return "amazon";
|
||||
if (ex.looksBlocked) return "unknown";
|
||||
return "none";
|
||||
}
|
||||
|
||||
async function pickWs() {
|
||||
const notes = [];
|
||||
try {
|
||||
const ver = await httpJson(`${cdpHttp}/json/version`);
|
||||
notes.push({ step: "json/version", keys: Object.keys(ver) });
|
||||
if (ver.webSocketDebuggerUrl) {
|
||||
return { ws: rewriteWs(ver.webSocketDebuggerUrl, cdpHttp), notes, mode: "version" };
|
||||
}
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/version", error: String(e) });
|
||||
}
|
||||
try {
|
||||
const list = await httpJson(`${cdpHttp}/json/list`);
|
||||
notes.push({ step: "json/list", n: Array.isArray(list) ? list.length : 0 });
|
||||
const page = (list || []).find((t) => t.webSocketDebuggerUrl);
|
||||
if (page) return { ws: rewriteWs(page.webSocketDebuggerUrl, cdpHttp), notes, mode: "list" };
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/list", error: String(e) });
|
||||
}
|
||||
try {
|
||||
const created = await httpJson(`${cdpHttp}/json/new?${encodeURIComponent("about:blank")}`);
|
||||
if (created.webSocketDebuggerUrl) {
|
||||
return { ws: rewriteWs(created.webSocketDebuggerUrl, cdpHttp), notes, mode: "new" };
|
||||
}
|
||||
} catch (e) {
|
||||
notes.push({ step: "json/new", error: String(e) });
|
||||
}
|
||||
// last resort: ws on same host/port
|
||||
const u = new URL(cdpHttp);
|
||||
return { ws: `ws://${u.hostname}:${u.port}`, notes, mode: "bare" };
|
||||
}
|
||||
|
||||
async function waitEvent(cdp, method, timeoutMs, pred) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => reject(new Error(`wait ${method} timeout`)), timeoutMs);
|
||||
const fn = (params, sessionId) => {
|
||||
if (pred && !pred(params, sessionId)) return;
|
||||
clearTimeout(t);
|
||||
resolve({ params, sessionId });
|
||||
};
|
||||
cdp.on(method, fn);
|
||||
});
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const t0 = Date.now();
|
||||
const pick = await pickWs();
|
||||
const cdp = new Cdp(pick.ws);
|
||||
await cdp.open();
|
||||
let sessionId;
|
||||
let targetId;
|
||||
const errors = [];
|
||||
try {
|
||||
try {
|
||||
await cdp.send("Browser.getVersion");
|
||||
} catch (e) {
|
||||
errors.push(`Browser.getVersion: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
const created = await cdp.send("Target.createTarget", { url: "about:blank" });
|
||||
targetId = created.targetId;
|
||||
const attached = await cdp.send("Target.attachToTarget", { targetId, flatten: true });
|
||||
sessionId = attached.sessionId;
|
||||
} catch (e) {
|
||||
errors.push(`Target.create/attach: ${e.message}`);
|
||||
sessionId = undefined;
|
||||
}
|
||||
const send = (m, p) => cdp.send(m, p, sessionId);
|
||||
try {
|
||||
await send("Page.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Page.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Runtime.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Runtime.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Network.enable");
|
||||
} catch (e) {
|
||||
errors.push(`Network.enable: ${e.message}`);
|
||||
}
|
||||
try {
|
||||
await send("Network.setUserAgentOverride", { userAgent: UA });
|
||||
} catch (e) {
|
||||
errors.push(`UA override: ${e.message}`);
|
||||
}
|
||||
|
||||
let inflight = 0;
|
||||
let lastNet = Date.now();
|
||||
cdp.on("Network.requestWillBeSent", () => {
|
||||
inflight += 1;
|
||||
lastNet = Date.now();
|
||||
});
|
||||
cdp.on("Network.loadingFinished", () => {
|
||||
inflight = Math.max(0, inflight - 1);
|
||||
lastNet = Date.now();
|
||||
});
|
||||
cdp.on("Network.loadingFailed", () => {
|
||||
inflight = Math.max(0, inflight - 1);
|
||||
lastNet = Date.now();
|
||||
});
|
||||
|
||||
let loadFired = false;
|
||||
cdp.on("Page.loadEventFired", () => {
|
||||
loadFired = true;
|
||||
});
|
||||
cdp.on("Page.domContentEventFired", () => {});
|
||||
|
||||
const navStart = Date.now();
|
||||
let navResult;
|
||||
try {
|
||||
navResult = await send("Page.navigate", { url: targetUrl });
|
||||
} catch (e) {
|
||||
throw new Error(`Page.navigate failed: ${e.message}`);
|
||||
}
|
||||
const navDeadline = Date.now() + navTimeoutS * 1000;
|
||||
while (Date.now() < navDeadline && !loadFired) {
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
const loadWaitMs = Date.now() - navStart;
|
||||
// networkidle up to 8s, else 2s settle
|
||||
const idleDeadline = Date.now() + 8000;
|
||||
let usedIdle = false;
|
||||
while (Date.now() < idleDeadline) {
|
||||
if (inflight === 0 && Date.now() - lastNet >= 500 && loadFired) {
|
||||
usedIdle = true;
|
||||
break;
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
if (!usedIdle) {
|
||||
await new Promise((r) => setTimeout(r, 2000));
|
||||
}
|
||||
|
||||
let extract = {};
|
||||
try {
|
||||
const ev = await send("Runtime.evaluate", {
|
||||
expression: EXTRACT_JS,
|
||||
returnByValue: true,
|
||||
awaitPromise: true,
|
||||
});
|
||||
extract = (ev.result && ev.result.value) || {};
|
||||
if (ev.exceptionDetails) {
|
||||
errors.push(`evaluate exception: ${JSON.stringify(ev.exceptionDetails)}`);
|
||||
}
|
||||
} catch (e) {
|
||||
errors.push(`Runtime.evaluate: ${e.message}`);
|
||||
}
|
||||
|
||||
if (targetId) {
|
||||
try {
|
||||
await cdp.send("Target.closeTarget", { targetId });
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const vendor = detectVendor(extract);
|
||||
const out = {
|
||||
ok: true,
|
||||
cdp_mode: pick.mode,
|
||||
t_total: (Date.now() - t0) / 1000,
|
||||
t_nav_s: loadWaitMs / 1000,
|
||||
networkidle: usedIdle,
|
||||
errorText: navResult && navResult.errorText,
|
||||
loaderId: navResult && navResult.loaderId,
|
||||
...extract,
|
||||
challenge_vendor: vendor,
|
||||
blocked: Boolean(extract.looksBlocked) || vendor !== "none",
|
||||
cdp_errors: errors,
|
||||
pick_notes: pick.notes,
|
||||
};
|
||||
process.stdout.write(JSON.stringify(out) + "\n");
|
||||
} finally {
|
||||
cdp.close();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
process.stdout.write(
|
||||
JSON.stringify({
|
||||
ok: false,
|
||||
error: String(e && e.message ? e.message : e),
|
||||
t_total: 0,
|
||||
title: "",
|
||||
text: "",
|
||||
htmlLen: 0,
|
||||
textLen: 0,
|
||||
blocked: true,
|
||||
challenge_vendor: "unknown",
|
||||
}) + "\n"
|
||||
);
|
||||
process.exit(0);
|
||||
});
|
||||
19
bench/lightpanda/compose.yml
Normal file
19
bench/lightpanda/compose.yml
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
services:
|
||||
lightpanda:
|
||||
image: docker.io/lightpanda/browser:0.3.7
|
||||
platform: linux/arm64
|
||||
container_name: bench-s3c-lightpanda
|
||||
command: ["/bin/lightpanda", "serve", "--host", "0.0.0.0", "--port", "9222", "--log_level", "info"]
|
||||
ports:
|
||||
- "19222:9222"
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
environment:
|
||||
HTTP_PROXY: http://host.docker.internal:17890
|
||||
HTTPS_PROXY: http://host.docker.internal:17890
|
||||
http_proxy: http://host.docker.internal:17890
|
||||
https_proxy: http://host.docker.internal:17890
|
||||
NO_PROXY: 127.0.0.1,localhost
|
||||
no_proxy: 127.0.0.1,localhost
|
||||
mem_limit: 256m
|
||||
restart: "no"
|
||||
1
bench/lightpanda/raw/T4-amz-1.excerpt
Normal file
1
bench/lightpanda/raw/T4-amz-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Amazon.com", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 3479, "text_len": 142, "price": "", "jsonld_price": "", "blocked": true, "vendor": "amazon", "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates", "html_head": "<html class=\"a-no-js\" lang=\"en-us\"><!--<![endif]--><head>\n<meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\">\n<meta charset=\"utf-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\n<title dir=\"ltr\">Amazon.com</title>\n<meta name=\"viewport\" content=\"width=device-width\">\n<link rel=\"stylesheet\" href=\"https://images-na.ssl-images-amazon.com/images/G/01/AUIClients/AmazonUI-3c913031596ca78a3768f4e934b1cc02ce238101.secure.min._V1_.css\">\n</head>\n<body>\n\n<!--\n To discuss automated access to Amazon data please contact api-services-support@amazon.com.\n For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_c_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/adverti"}
|
||||
1
bench/lightpanda/raw/T4-amz-2.excerpt
Normal file
1
bench/lightpanda/raw/T4-amz-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Page Not Found", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 2327, "text_len": 0, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "", "html_head": "<html><head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, shrink-to-fit=no\">\n <title>\n Page Not Found\n </title>\n <style>\n html,body{padding:0;margin:0}img{border:0}#a{background:#232f3e;padding:11px 11px 11px 192px}#b{position:absolute;left:22px;top:12px}#c{position:relative;max-width:800px;padding:0 40px 0 0}#e,#f{height:35px;border:0;font-size:1em}#e{width:100%;margin:0;padding:0 10px;border-radius:4px 0 0 4px}#f{cursor:pointer;background:#febd69;font-weight:bold;border-radius:0 4px 4px 0;-webkit-appearance:none;position:absolute;top:0;right:0;padding:0 12px}@media(max-width:500px){#a{padding:55px 10px 10px}#b{left:6px}}#g{text-align:center;margin:30px 0}#g i"}
|
||||
1
bench/lightpanda/raw/T4-amz-3.excerpt
Normal file
1
bench/lightpanda/raw/T4-amz-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Amazon.com", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 3479, "text_len": 142, "price": "", "jsonld_price": "", "blocked": true, "vendor": "amazon", "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates", "html_head": "<html class=\"a-no-js\" lang=\"en-us\"><!--<![endif]--><head>\n<meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\">\n<meta charset=\"utf-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\n<title dir=\"ltr\">Amazon.com</title>\n<meta name=\"viewport\" content=\"width=device-width\">\n<link rel=\"stylesheet\" href=\"https://images-na.ssl-images-amazon.com/images/G/01/AUIClients/AmazonUI-3c913031596ca78a3768f4e934b1cc02ce238101.secure.min._V1_.css\">\n</head>\n<body>\n\n<!--\n To discuss automated access to Amazon data please contact api-services-support@amazon.com.\n For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_c_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/adverti"}
|
||||
1
bench/lightpanda/raw/T4-ctrl-1.excerpt
Normal file
1
bench/lightpanda/raw/T4-ctrl-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1489, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”by Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”by Marilyn ", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/lightpanda/raw/T4-ctrl-2.excerpt
Normal file
1
bench/lightpanda/raw/T4-ctrl-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1489, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”by Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”by Marilyn ", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/lightpanda/raw/T4-ctrl-3.excerpt
Normal file
1
bench/lightpanda/raw/T4-ctrl-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1489, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”by Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”by Marilyn ", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/lightpanda/raw/T4-medium-1.excerpt
Normal file
1
bench/lightpanda/raw/T4-medium-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 725, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a34162ec3ec83f37 • Your IP: Click to reveal 207.32.218.93 • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/lightpanda/raw/T4-medium-2.excerpt
Normal file
1
bench/lightpanda/raw/T4-medium-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 725, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a341633b59fc2f03 • Your IP: Click to reveal 207.32.218.91 • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/lightpanda/raw/T4-medium-3.excerpt
Normal file
1
bench/lightpanda/raw/T4-medium-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 725, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a3416388e9196049 • Your IP: Click to reveal 207.32.218.93 • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/lightpanda/raw/T4-shop-1.excerpt
Normal file
1
bench/lightpanda/raw/T4-shop-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1438467, "text_len": 3426, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatured\nMost relevant\nBest selling\nAlphabetically, A-Z\nAlphabetically, Z-A\nPrice, low to high\nPrice, high to low\nDate, old to new\nDate, new to old\nNEWNEW\n\nMen's Dasher NZ\n\nAnthracite\n\n$140\n\nAdd to Cart\n\nMen's Tree Runner NZ\n\nMedium Grey\n\n$100\n\nAdd to Cart\n\nMen's Tree Runner\n\nMist\n\n$100\n\nAdd to Cart\n\nMen", "html_head": "<html class=\"h-full\" lang=\"en-US\"><head><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script async src=\"https://s.pinimg.com/ct/lib/main.e47a302d.js\"></script><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script src=\"https://connect.facebook.net/signals/config/1689016458013442?v=2.9.390&r=stable&domain=www.allbirds.com&hme=a767ae4c6a92a40ee4d7e358a937ad845c8bdb98a3bbe3412069dded3a7be57d&ex_m=115%2C221%2C166%2C26%2C80%2C81%2C158%2C76%2C75%2C11%2C176%2C100%2C20%2C150%2C138%2C45%2C83%2C88%2C146%2C171%2C178%2C31%2C17%2C30%2C18%2C32%2C33%2C34%2C36%2C54%2C159%2C85%2C123%2C22%2C24%2C50%2C46%2C48%2C47%"}
|
||||
1
bench/lightpanda/raw/T4-shop-2.excerpt
Normal file
1
bench/lightpanda/raw/T4-shop-2.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1439043, "text_len": 3426, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatured\nMost relevant\nBest selling\nAlphabetically, A-Z\nAlphabetically, Z-A\nPrice, low to high\nPrice, high to low\nDate, old to new\nDate, new to old\nNEWNEW\n\nMen's Dasher NZ\n\nAnthracite\n\n$140\n\nAdd to Cart\n\nMen's Tree Runner NZ\n\nMedium Grey\n\n$100\n\nAdd to Cart\n\nMen's Tree Runner\n\nMist\n\n$100\n\nAdd to Cart\n\nMen", "html_head": "<html class=\"h-full\" lang=\"en-US\"><head><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script async src=\"https://s.pinimg.com/ct/lib/main.e47a302d.js\"></script><script async src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script type=\"text/javascript\" async src=\"https://cdn.attn.tv/allbirds/dtag.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async src=\"https://id-shop.govx.com/app/weareallbirds.myshopify.com/govx.js?shop=weareallbirds.myshopify.com\"></script><script src=\"https://connect.facebook.net/sig"}
|
||||
1
bench/lightpanda/raw/T4-shop-3.excerpt
Normal file
1
bench/lightpanda/raw/T4-shop-3.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1439391, "text_len": 3426, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatured\nMost relevant\nBest selling\nAlphabetically, A-Z\nAlphabetically, Z-A\nPrice, low to high\nPrice, high to low\nDate, old to new\nDate, new to old\nNEWNEW\n\nMen's Dasher NZ\n\nAnthracite\n\n$140\n\nAdd to Cart\n\nMen's Tree Runner NZ\n\nMedium Grey\n\n$100\n\nAdd to Cart\n\nMen's Tree Runner\n\nMist\n\n$100\n\nAdd to Cart\n\nMen", "html_head": "<html class=\"h-full\" lang=\"en-US\"><head><script async src=\"https://utt.impactcdn.com/A2840043-f763-41d9-b989-637b09feea141.js\"></script><script type=\"text/javascript\" async src=\"https://cdn.attn.tv/allbirds/dtag.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async src=\"https://id-shop.govx.com/app/weareallbirds.myshopify.com/govx.js?shop=weareallbirds.myshopify.com\"></script><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script async src=\"https://s.pinimg.com/ct/lib/main.e47a302d.js\"></script><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script src=\"https://connect.facebook.net/sig"}
|
||||
1
bench/lightpanda/raw/T4-t3x-1.excerpt
Normal file
1
bench/lightpanda/raw/T4-t3x-1.excerpt
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-t3x", "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "html_len": 82913, "text_len": 1000, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "DuckDuckGo\nProtection. Privacy. Peace of mind.\nOpen menu\nAll\nImages\nVideos\nNews\nMaps\nShopping\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nClose menu\nSearch\nHomepage\nThemes\nSettings\nShare Feedback\nDownloads\niOS Browser\nAndroid Browser\nMac Browser\nWindows Browser\nBrowser Extensions\nMore From DuckDuckGo\nDuckDuckGo Subscription\nDuck.ai\nEmail Protection\nNewsletter\nBlog\nPodcast\nCollaborations\nLearn More\nWhat’s New\nCompare Privacy\nAbout Our Browser\nAbout DuckDuckGo\nOther Resources\nHelp\nCommunity\nCareers\nPrivacy Policy\nTerms of Service\nPress Kit\nAdvertise on Search\nSee What’s DuckDuckNew\n\nExplore DuckDuckGo’s latest product updates.\n\nLearn More\nLearn More\nWhat's New\nCompare Privacy\nSearch Shortcuts\nHelp Pages\nGet More\nSubscription\nEmail Protection\nPodcast & Newslette", "html_head": "<html lang=\"en-US\" class=\"has-zcm is-link-style-exp is-link-order-exp is-link-breadcrumb-exp is-related-search-exp is-vertical-tabs-exp js no-touch opacity csstransforms3d csstransitions svg cssfilters is-not-mobile-device full-urls breadcrumb-urls react has-footer\" style=\"--sds-color-palette-black: #000000; --sds-color-palette-white: #FFFFFF; --sds-color-palette-transparent: transparent; --sds-color-palette-gray-100: #111111; --sds-color-palette-gray-95: #161616; --sds-color-palette-gray-90: #222222; --sds-color-palette-gray-85: #333333; --sds-color-palette-gray-80: #444444; --sds-color-palette-gray-70: #666666; --sds-color-palette-gray-60: #888888; --sds-color-palette-gray-50: #aaaaaa; --sds-color-palette-gray-40: #cccccc; --sds-color-palette-gray-30: #dddddd; --sds-color-palette-gra"}
|
||||
1
bench/lightpanda/raw/cdp-version.json
Normal file
1
bench/lightpanda/raw/cdp-version.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"Browser": "Lightpanda/1.0", "Protocol-Version": "1.3", "User-Agent": "Lightpanda/1.0", "Lightpanda-Version": "0.3.7", "webSocketDebuggerUrl": "ws://127.0.0.1:9222/"}
|
||||
66
bench/lightpanda/raw/idle.tsv
Normal file
66
bench/lightpanda/raw/idle.tsv
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T03:56:29Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.00% 3.570 4.289
|
||||
2026-09-01T03:56:31Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 4.082
|
||||
2026-09-01T03:56:35Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.078
|
||||
2026-09-01T03:56:38Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 4.836
|
||||
2026-09-01T03:56:42Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 5.098
|
||||
2026-09-01T03:56:45Z bench-s3c-lightpanda 3.82MiB / 256MiB 0.00% 3.820 5.074
|
||||
2026-09-01T03:56:49Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.059
|
||||
2026-09-01T03:56:51Z bench-s3c-lightpanda 3.602MiB / 256MiB 0.01% 3.602 4.582
|
||||
2026-09-01T03:56:56Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.828
|
||||
2026-09-01T03:56:58Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 4.828
|
||||
2026-09-01T03:57:03Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 5.070
|
||||
2026-09-01T03:57:05Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.00% 3.570 4.832
|
||||
2026-09-01T03:57:09Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.828
|
||||
2026-09-01T03:57:12Z bench-s3c-lightpanda 3.941MiB / 256MiB 0.01% 3.941 4.656
|
||||
2026-09-01T03:57:16Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.832
|
||||
2026-09-01T03:57:19Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 5.062
|
||||
2026-09-01T03:57:24Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.590
|
||||
2026-09-01T03:57:26Z bench-s3c-lightpanda 3.578MiB / 256MiB 0.01% 3.578 5.324
|
||||
2026-09-01T03:57:30Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.004
|
||||
2026-09-01T03:57:32Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.01% 3.570 4.824
|
||||
2026-09-01T03:57:37Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.824
|
||||
2026-09-01T03:57:39Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 4.840
|
||||
2026-09-01T03:57:43Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.01% 3.570 5.316
|
||||
2026-09-01T03:57:45Z bench-s3c-lightpanda 3.578MiB / 256MiB 0.01% 3.578 5.062
|
||||
2026-09-01T03:57:49Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 4.578
|
||||
2026-09-01T03:57:52Z bench-s3c-lightpanda 3.871MiB / 256MiB 0.01% 3.871 4.859
|
||||
2026-09-01T03:57:56Z bench-s3c-lightpanda 3.562MiB / 256MiB 0.01% 3.562 5.070
|
||||
2026-09-01T03:57:58Z bench-s3c-lightpanda 3.562MiB / 256MiB 0.01% 3.562 4.832
|
||||
2026-09-01T03:58:03Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.812
|
||||
2026-09-01T03:58:05Z bench-s3c-lightpanda 3.844MiB / 256MiB 0.01% 3.844 5.066
|
||||
2026-09-01T03:58:10Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.934
|
||||
2026-09-01T03:58:12Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.00% 3.570 4.840
|
||||
2026-09-01T03:58:18Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 4.906
|
||||
2026-09-01T03:58:20Z bench-s3c-lightpanda 3.676MiB / 256MiB 0.01% 3.676 5.367
|
||||
2026-09-01T03:58:25Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.312
|
||||
2026-09-01T03:58:27Z bench-s3c-lightpanda 4.047MiB / 256MiB 0.01% 4.047 4.836
|
||||
2026-09-01T03:58:31Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.777
|
||||
2026-09-01T03:58:33Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 4.824
|
||||
2026-09-01T03:58:38Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.156
|
||||
2026-09-01T03:58:40Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 5.074
|
||||
2026-09-01T03:58:45Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.824
|
||||
2026-09-01T03:58:51Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.102
|
||||
2026-09-01T03:58:58Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.078
|
||||
2026-09-01T03:59:04Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.934
|
||||
2026-09-01T03:59:10Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.828
|
||||
2026-09-01T03:59:17Z bench-s3c-lightpanda 3.57MiB / 256MiB 0.00% 3.570 4.812
|
||||
2026-09-01T03:59:24Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.070
|
||||
2026-09-01T03:59:30Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.02% 3.559 5.066
|
||||
2026-09-01T03:59:37Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.078
|
||||
2026-09-01T03:59:44Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.590
|
||||
2026-09-01T03:59:51Z bench-s3c-lightpanda 3.566MiB / 256MiB 0.01% 3.566 5.176
|
||||
2026-09-01T03:59:58Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.00% 3.559 4.352
|
||||
2026-09-01T04:00:05Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.824
|
||||
2026-09-01T04:00:12Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.691
|
||||
2026-09-01T04:00:19Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.820
|
||||
2026-09-01T04:00:25Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.848
|
||||
2026-09-01T04:00:32Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.816
|
||||
2026-09-01T04:00:39Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.02% 3.559 5.074
|
||||
2026-09-01T04:00:45Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.152
|
||||
2026-09-01T04:00:52Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.574
|
||||
2026-09-01T04:00:58Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.816
|
||||
2026-09-01T04:01:05Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.816
|
||||
2026-09-01T04:01:12Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.840
|
||||
2026-09-01T04:01:18Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 4.859
|
||||
2026-09-01T04:01:25Z bench-s3c-lightpanda 3.559MiB / 256MiB 0.01% 3.559 5.219
|
||||
|
35
bench/lightpanda/raw/image-meta.json
Normal file
35
bench/lightpanda/raw/image-meta.json
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
{
|
||||
"ref": "docker.io/lightpanda/browser:0.3.7",
|
||||
"digest": null,
|
||||
"id": "sha256:eae2c7f8ff39a7a24540d21c7561ce219fd14b1bf2b92403930c6d140f2d1337",
|
||||
"arch": "arm64",
|
||||
"pull": "local",
|
||||
"probed_at": "2026-09-01T12:06:03+08:00",
|
||||
"oom": false,
|
||||
"container_status": "running",
|
||||
"t5_wall_s": 28.203,
|
||||
"t5_parallelism": 4,
|
||||
"queue_impl": "client_fifo",
|
||||
"queue": {
|
||||
"first_complete_s": 2.0435,
|
||||
"last_complete_s": 28.2032,
|
||||
"queue_wait_p50_s": 0.0857,
|
||||
"queue_wait_p95_s": 0.1007,
|
||||
"rejected": 0
|
||||
},
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "204",
|
||||
"host_docker_internal": "fdc4:f303:9324::254 host.docker.internal",
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway"
|
||||
},
|
||||
"commands": [
|
||||
"docker compose -p bench-s3c-lightpanda up -d",
|
||||
"node cdp_fetch.mjs http://127.0.0.1:19222 <url>",
|
||||
"T5 ThreadPoolExecutor workers=4 n=60"
|
||||
],
|
||||
"notes": [
|
||||
"queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT"
|
||||
]
|
||||
}
|
||||
1
bench/lightpanda/raw/pull.how
Normal file
1
bench/lightpanda/raw/pull.how
Normal file
|
|
@ -0,0 +1 @@
|
|||
local
|
||||
0
bench/lightpanda/raw/session.done
Normal file
0
bench/lightpanda/raw/session.done
Normal file
29
bench/lightpanda/raw/session.tsv
Normal file
29
bench/lightpanda/raw/session.tsv
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T04:03:20Z bench-s3c-lightpanda 144.5MiB / 256MiB 5.75% 144.500 145.848
|
||||
2026-09-01T04:03:20Z bench-s3c-lightpanda 144.5MiB / 256MiB 5.75% 144.500 146.035
|
||||
2026-09-01T04:03:23Z bench-s3c-lightpanda 162.6MiB / 256MiB 31.44% 162.600 173.926
|
||||
2026-09-01T04:03:23Z bench-s3c-lightpanda 162.6MiB / 256MiB 31.44% 162.600 174.023
|
||||
2026-09-01T04:03:25Z bench-s3c-lightpanda 249.9MiB / 256MiB 39.70% 249.900 251.480
|
||||
2026-09-01T04:03:25Z bench-s3c-lightpanda 249.9MiB / 256MiB 39.70% 249.900 251.480
|
||||
2026-09-01T04:03:28Z bench-s3c-lightpanda 251.4MiB / 256MiB 3.07% 251.400 252.656
|
||||
2026-09-01T04:03:28Z bench-s3c-lightpanda 251.4MiB / 256MiB 3.07% 251.400 252.316
|
||||
2026-09-01T04:03:30Z bench-s3c-lightpanda 247MiB / 256MiB 3.70% 247.000 248.508
|
||||
2026-09-01T04:03:30Z bench-s3c-lightpanda 247MiB / 256MiB 3.70% 247.000 248.414
|
||||
2026-09-01T04:03:33Z bench-s3c-lightpanda 102MiB / 256MiB 1.13% 102.000 103.746
|
||||
2026-09-01T04:03:33Z bench-s3c-lightpanda 102MiB / 256MiB 1.13% 102.000 103.738
|
||||
2026-09-01T04:03:35Z bench-s3c-lightpanda 104.1MiB / 256MiB 1.65% 104.100 105.102
|
||||
2026-09-01T04:03:35Z bench-s3c-lightpanda 104.1MiB / 256MiB 1.65% 104.100 105.008
|
||||
2026-09-01T04:03:38Z bench-s3c-lightpanda 106.4MiB / 256MiB 7.40% 106.400 107.207
|
||||
2026-09-01T04:03:38Z bench-s3c-lightpanda 106.4MiB / 256MiB 7.40% 106.400 107.305
|
||||
2026-09-01T04:03:41Z bench-s3c-lightpanda 164.4MiB / 256MiB 25.17% 164.400 166.098
|
||||
2026-09-01T04:03:41Z bench-s3c-lightpanda 164.4MiB / 256MiB 25.17% 164.400 165.906
|
||||
2026-09-01T04:03:44Z bench-s3c-lightpanda 142MiB / 256MiB 0.06% 142.000 143.527
|
||||
2026-09-01T04:03:44Z bench-s3c-lightpanda 142MiB / 256MiB 0.06% 142.000 143.527
|
||||
2026-09-01T04:03:47Z bench-s3c-lightpanda 142MiB / 256MiB 0.09% 142.000 143.891
|
||||
2026-09-01T04:03:47Z bench-s3c-lightpanda 142MiB / 256MiB 0.09% 142.000 144.016
|
||||
2026-09-01T04:03:50Z bench-s3c-lightpanda 138.9MiB / 256MiB 1.05% 138.900 140.781
|
||||
2026-09-01T04:03:50Z bench-s3c-lightpanda 138.9MiB / 256MiB 1.05% 138.900 140.781
|
||||
2026-09-01T04:03:53Z bench-s3c-lightpanda 141.3MiB / 256MiB 0.19% 141.300 142.906
|
||||
2026-09-01T04:03:53Z bench-s3c-lightpanda 141.3MiB / 256MiB 0.19% 141.300 142.898
|
||||
2026-09-01T04:03:55Z bench-s3c-lightpanda 141.6MiB / 256MiB 0.22% 141.600 142.574
|
||||
2026-09-01T04:03:55Z bench-s3c-lightpanda 141.6MiB / 256MiB 0.22% 141.600 142.574
|
||||
|
18
bench/lightpanda/raw/t4.jsonl
Normal file
18
bench/lightpanda/raw/t4.jsonl
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
{"ts": "2026-09-01T12:03:32+08:00", "scheme": "lightpanda", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.607, "ttfb": 1.054, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:03:32+08:00", "scheme": "lightpanda", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 15.021, "ttfb": 4.899, "t_namelookup": 0, "t_connect": 0, "bytes": 1439527, "html_len": 1439527, "text_len": 3426, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatu"}
|
||||
{"ts": "2026-09-01T12:03:35+08:00", "scheme": "lightpanda", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.905, "ttfb": 1.349, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:03:35+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.233, "ttfb": 0.679, "t_namelookup": 0, "t_connect": 0, "bytes": 3479, "html_len": 3479, "text_len": 142, "title": "Amazon.com", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "amazon", "networkidle": true, "assert_ok": true, "note": "waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates"}
|
||||
{"ts": "2026-09-01T12:03:37+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.33, "ttfb": 0.875, "t_namelookup": 0, "t_connect": 0, "bytes": 3479, "html_len": 3479, "text_len": 142, "title": "Amazon.com", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "amazon", "networkidle": true, "assert_ok": true, "note": "waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates"}
|
||||
{"ts": "2026-09-01T12:03:37+08:00", "scheme": "lightpanda", "template": "T4-ctrl", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.727, "ttfb": 1.176, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:03:40+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.368, "ttfb": 0.909, "t_namelookup": 0, "t_connect": 0, "bytes": 3479, "html_len": 3479, "text_len": 142, "title": "Amazon.com", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "amazon", "networkidle": true, "assert_ok": true, "note": "waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates"}
|
||||
{"ts": "2026-09-01T12:03:44+08:00", "scheme": "lightpanda", "template": "T4-t3x", "url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "http_code": 200, "http_status_in_page": null, "t_total": 2.894, "ttfb": 2.337, "t_namelookup": 0, "t_connect": 0, "bytes": 82696, "html_len": 82696, "text_len": 1000, "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "DuckDuckGo\nProtection. Privacy. Peace of mind.\nOpen menu\nAll\nImages\nVideos\nNews\nMaps\nShopping\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nClose menu\nSearch\nHomepage\nThemes\nSettings\nShare Feedback\nDownloads\niOS Browser\nAndroid Browser\nMac Browser\nWindows Browser\nBrowser Extensions\nMore From DuckDuckGo\nDuckDuckGo Subscription\nDuck.ai\nEmail Protection\nNewsletter\nBlog\nPodcast\nCollaborations\nLearn More\nWhat’s New\nCompare Privacy\nAbout Our Browser\nAbout DuckDuckGo\nOther "}
|
||||
{"ts": "2026-09-01T12:03:50+08:00", "scheme": "lightpanda", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 11.521, "ttfb": 1.392, "t_namelookup": 0, "t_connect": 0, "bytes": 5193, "html_len": 5193, "text_len": 725, "title": "Attention Required! | Cloudflare", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "cloudflare", "networkidle": false, "assert_ok": false, "note": "medium_wall_or_waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please inc"}
|
||||
{"ts": "2026-09-01T12:04:02+08:00", "scheme": "lightpanda", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 11.307, "ttfb": 1.182, "t_namelookup": 0, "t_connect": 0, "bytes": 5193, "html_len": 5193, "text_len": 725, "title": "Attention Required! | Cloudflare", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "cloudflare", "networkidle": false, "assert_ok": false, "note": "medium_wall_or_waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please inc"}
|
||||
{"ts": "2026-09-01T12:04:15+08:00", "scheme": "lightpanda", "template": "T4-medium", "url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "http_code": 200, "http_status_in_page": null, "t_total": 11.3, "ttfb": 1.168, "t_namelookup": 0, "t_connect": 0, "bytes": 5193, "html_len": 5193, "text_len": 725, "title": "Attention Required! | Cloudflare", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "cloudflare", "networkidle": false, "assert_ok": false, "note": "medium_wall_or_waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please inc"}
|
||||
{"ts": "2026-09-01T12:04:32+08:00", "scheme": "lightpanda", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 15.496, "ttfb": 5.281, "t_namelookup": 0, "t_connect": 0, "bytes": 1438467, "html_len": 1438467, "text_len": 3426, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatu"}
|
||||
{"ts": "2026-09-01T12:04:47+08:00", "scheme": "lightpanda", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 14.689, "ttfb": 4.561, "t_namelookup": 0, "t_connect": 0, "bytes": 1439043, "html_len": 1439043, "text_len": 3426, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatu"}
|
||||
{"ts": "2026-09-01T12:05:04+08:00", "scheme": "lightpanda", "template": "T4-shop", "url": "https://www.allbirds.com/products/mens-wool-runners", "final_url": "https://www.allbirds.com/collections/mens", "http_code": 200, "http_status_in_page": null, "t_total": 14.881, "ttfb": 4.752, "t_namelookup": 0, "t_connect": 0, "bytes": 1439391, "html_len": 1439391, "text_len": 3426, "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": false, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatu"}
|
||||
{"ts": "2026-09-01T12:05:06+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.31, "ttfb": 0.753, "t_namelookup": 0, "t_connect": 0, "bytes": 3479, "html_len": 3479, "text_len": 142, "title": "Amazon.com", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "amazon", "networkidle": true, "assert_ok": true, "note": "waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates"}
|
||||
{"ts": "2026-09-01T12:05:09+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.627, "ttfb": 1.172, "t_namelookup": 0, "t_connect": 0, "bytes": 2327, "html_len": 2327, "text_len": 0, "title": "Page Not Found", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": ""}
|
||||
{"ts": "2026-09-01T12:05:12+08:00", "scheme": "lightpanda", "template": "T4-amz", "url": "https://www.amazon.com/dp/B08N5WRWNW", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "http_code": 200, "http_status_in_page": null, "t_total": 1.744, "ttfb": 1.287, "t_namelookup": 0, "t_connect": 0, "bytes": 3479, "html_len": 3479, "text_len": 142, "title": "Amazon.com", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": true, "challenge_vendor": "amazon", "networkidle": true, "assert_ok": true, "note": "waf", "cdp_mode": "version", "cdp_errors": [], "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates"}
|
||||
{"ts": "2026-09-01T12:05:17+08:00", "scheme": "lightpanda", "template": "T4-t3x", "url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "http_code": 200, "http_status_in_page": null, "t_total": 3.962, "ttfb": 3.403, "t_namelookup": 0, "t_connect": 0, "bytes": 82913, "html_len": 82913, "text_len": 1000, "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "price": "", "jsonld_price": "", "quote_count": 0, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "DuckDuckGo\nProtection. Privacy. Peace of mind.\nOpen menu\nAll\nImages\nVideos\nNews\nMaps\nShopping\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nClose menu\nSearch\nHomepage\nThemes\nSettings\nShare Feedback\nDownloads\niOS Browser\nAndroid Browser\nMac Browser\nWindows Browser\nBrowser Extensions\nMore From DuckDuckGo\nDuckDuckGo Subscription\nDuck.ai\nEmail Protection\nNewsletter\nBlog\nPodcast\nCollaborations\nLearn More\nWhat’s New\nCompare Privacy\nAbout Our Browser\nAbout DuckDuckGo\nOther "}
|
||||
60
bench/lightpanda/raw/t5-queue.jsonl
Normal file
60
bench/lightpanda/raw/t5-queue.jsonl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{"i": 2, "enqueue_s": 0.03605008125305176, "start_s": 0.1434917449951172, "end_s": 2.043491840362549, "queue_wait_s": 0.10744166374206543, "assert_ok": true, "t_total": 1.9}
|
||||
{"i": 1, "enqueue_s": 0.036004066467285156, "start_s": 0.14379286766052246, "end_s": 2.044792890548706, "queue_wait_s": 0.1077888011932373, "assert_ok": true, "t_total": 1.901}
|
||||
{"i": 3, "enqueue_s": 0.037016868591308594, "start_s": 0.14587664604187012, "end_s": 2.0458767414093018, "queue_wait_s": 0.10885977745056152, "assert_ok": true, "t_total": 1.9}
|
||||
{"i": 4, "enqueue_s": 0.038205862045288086, "start_s": 0.12490296363830566, "end_s": 2.089902877807617, "queue_wait_s": 0.08669710159301758, "assert_ok": true, "t_total": 1.965}
|
||||
{"i": 5, "enqueue_s": 2.043632984161377, "start_s": 2.1439270973205566, "end_s": 4.009927034378052, "queue_wait_s": 0.10029411315917969, "assert_ok": true, "t_total": 1.866}
|
||||
{"i": 6, "enqueue_s": 2.044938802719116, "start_s": 2.135016918182373, "end_s": 4.016016960144043, "queue_wait_s": 0.09007811546325684, "assert_ok": true, "t_total": 1.881}
|
||||
{"i": 7, "enqueue_s": 2.0459978580474854, "start_s": 2.137497901916504, "end_s": 4.017498016357422, "queue_wait_s": 0.09150004386901855, "assert_ok": true, "t_total": 1.88}
|
||||
{"i": 8, "enqueue_s": 2.0900559425354004, "start_s": 2.176635980606079, "end_s": 4.077636003494263, "queue_wait_s": 0.08658003807067871, "assert_ok": true, "t_total": 1.901}
|
||||
{"i": 10, "enqueue_s": 4.0161168575286865, "start_s": 4.104970693588257, "end_s": 5.672970771789551, "queue_wait_s": 0.08885383605957031, "assert_ok": true, "t_total": 1.568}
|
||||
{"i": 9, "enqueue_s": 4.010046720504761, "start_s": 4.096394062042236, "end_s": 5.69339394569397, "queue_wait_s": 0.08634734153747559, "assert_ok": true, "t_total": 1.597}
|
||||
{"i": 12, "enqueue_s": 4.077816009521484, "start_s": 4.163127899169922, "end_s": 5.728127956390381, "queue_wait_s": 0.0853118896484375, "assert_ok": true, "t_total": 1.565}
|
||||
{"i": 11, "enqueue_s": 4.017690896987915, "start_s": 4.096921920776367, "end_s": 5.846921920776367, "queue_wait_s": 0.07923102378845215, "assert_ok": true, "t_total": 1.75}
|
||||
{"i": 15, "enqueue_s": 5.728243827819824, "start_s": 5.816249847412109, "end_s": 7.39424991607666, "queue_wait_s": 0.08800601959228516, "assert_ok": true, "t_total": 1.578}
|
||||
{"i": 14, "enqueue_s": 5.693515777587891, "start_s": 5.773045778274536, "end_s": 7.533045768737793, "queue_wait_s": 0.07953000068664551, "assert_ok": true, "t_total": 1.76}
|
||||
{"i": 16, "enqueue_s": 5.846993923187256, "start_s": 5.920598745346069, "end_s": 7.685598850250244, "queue_wait_s": 0.07360482215881348, "assert_ok": true, "t_total": 1.765}
|
||||
{"i": 13, "enqueue_s": 5.673073053359985, "start_s": 5.751910209655762, "end_s": 7.7039101123809814, "queue_wait_s": 0.07883715629577637, "assert_ok": true, "t_total": 1.952}
|
||||
{"i": 18, "enqueue_s": 7.533138751983643, "start_s": 7.622963905334473, "end_s": 9.24596381187439, "queue_wait_s": 0.08982515335083008, "assert_ok": true, "t_total": 1.623}
|
||||
{"i": 17, "enqueue_s": 7.394368886947632, "start_s": 7.485376834869385, "end_s": 9.450376749038696, "queue_wait_s": 0.09100794792175293, "assert_ok": true, "t_total": 1.965}
|
||||
{"i": 19, "enqueue_s": 7.68567681312561, "start_s": 7.764592885971069, "end_s": 9.718592882156372, "queue_wait_s": 0.07891607284545898, "assert_ok": true, "t_total": 1.954}
|
||||
{"i": 20, "enqueue_s": 7.703984975814819, "start_s": 7.776607036590576, "end_s": 9.794606924057007, "queue_wait_s": 0.07262206077575684, "assert_ok": true, "t_total": 2.018}
|
||||
{"i": 21, "enqueue_s": 9.246098041534424, "start_s": 9.338961839675903, "end_s": 10.889961957931519, "queue_wait_s": 0.09286379814147949, "assert_ok": true, "t_total": 1.551}
|
||||
{"i": 22, "enqueue_s": 9.450468063354492, "start_s": 9.53443169593811, "end_s": 11.289431810379028, "queue_wait_s": 0.08396363258361816, "assert_ok": true, "t_total": 1.755}
|
||||
{"i": 23, "enqueue_s": 9.718717813491821, "start_s": 9.793133020401001, "end_s": 11.35213303565979, "queue_wait_s": 0.07441520690917969, "assert_ok": true, "t_total": 1.559}
|
||||
{"i": 24, "enqueue_s": 9.794688940048218, "start_s": 9.870255947113037, "end_s": 11.421256065368652, "queue_wait_s": 0.07556700706481934, "assert_ok": true, "t_total": 1.551}
|
||||
{"i": 25, "enqueue_s": 10.890063047409058, "start_s": 10.982462882995605, "end_s": 12.712462902069092, "queue_wait_s": 0.09239983558654785, "assert_ok": true, "t_total": 1.73}
|
||||
{"i": 26, "enqueue_s": 11.28954005241394, "start_s": 11.37179183959961, "end_s": 12.940791845321655, "queue_wait_s": 0.08225178718566895, "assert_ok": true, "t_total": 1.569}
|
||||
{"i": 27, "enqueue_s": 11.35223388671875, "start_s": 11.428031206130981, "end_s": 12.985031127929688, "queue_wait_s": 0.07579731941223145, "assert_ok": true, "t_total": 1.557}
|
||||
{"i": 28, "enqueue_s": 11.421355962753296, "start_s": 11.495595693588257, "end_s": 13.044595718383789, "queue_wait_s": 0.07423973083496094, "assert_ok": true, "t_total": 1.549}
|
||||
{"i": 29, "enqueue_s": 12.712561130523682, "start_s": 12.788081884384155, "end_s": 14.542081832885742, "queue_wait_s": 0.07552075386047363, "assert_ok": true, "t_total": 1.754}
|
||||
{"i": 30, "enqueue_s": 12.940971851348877, "start_s": 13.015290021896362, "end_s": 14.625289916992188, "queue_wait_s": 0.07431817054748535, "assert_ok": true, "t_total": 1.61}
|
||||
{"i": 31, "enqueue_s": 12.98521089553833, "start_s": 13.072864055633545, "end_s": 15.049864053726196, "queue_wait_s": 0.08765316009521484, "assert_ok": true, "t_total": 1.977}
|
||||
{"i": 32, "enqueue_s": 13.044671773910522, "start_s": 13.119418144226074, "end_s": 15.068418025970459, "queue_wait_s": 0.07474637031555176, "assert_ok": true, "t_total": 1.949}
|
||||
{"i": 33, "enqueue_s": 14.542194843292236, "start_s": 14.629215955734253, "end_s": 16.426215887069702, "queue_wait_s": 0.0870211124420166, "assert_ok": true, "t_total": 1.797}
|
||||
{"i": 34, "enqueue_s": 14.625378847122192, "start_s": 14.697774887084961, "end_s": 16.457774877548218, "queue_wait_s": 0.07239603996276855, "assert_ok": true, "t_total": 1.76}
|
||||
{"i": 35, "enqueue_s": 15.049958944320679, "start_s": 15.122397661209106, "end_s": 16.674397706985474, "queue_wait_s": 0.07243871688842773, "assert_ok": true, "t_total": 1.552}
|
||||
{"i": 36, "enqueue_s": 15.068493843078613, "start_s": 15.14143991470337, "end_s": 16.69443988800049, "queue_wait_s": 0.07294607162475586, "assert_ok": true, "t_total": 1.553}
|
||||
{"i": 37, "enqueue_s": 16.42633581161499, "start_s": 16.518706798553467, "end_s": 18.069706916809082, "queue_wait_s": 0.09237098693847656, "assert_ok": true, "t_total": 1.551}
|
||||
{"i": 38, "enqueue_s": 16.457868814468384, "start_s": 16.5376398563385, "end_s": 18.08463978767395, "queue_wait_s": 0.07977104187011719, "assert_ok": true, "t_total": 1.547}
|
||||
{"i": 39, "enqueue_s": 16.67448377609253, "start_s": 16.752051830291748, "end_s": 18.304051876068115, "queue_wait_s": 0.07756805419921875, "assert_ok": true, "t_total": 1.552}
|
||||
{"i": 40, "enqueue_s": 16.694534063339233, "start_s": 16.769986152648926, "end_s": 18.315986156463623, "queue_wait_s": 0.07545208930969238, "assert_ok": true, "t_total": 1.546}
|
||||
{"i": 41, "enqueue_s": 18.069817066192627, "start_s": 18.151140928268433, "end_s": 19.696141004562378, "queue_wait_s": 0.08132386207580566, "assert_ok": true, "t_total": 1.545}
|
||||
{"i": 42, "enqueue_s": 18.084723949432373, "start_s": 18.1772038936615, "end_s": 20.108203887939453, "queue_wait_s": 0.09247994422912598, "assert_ok": true, "t_total": 1.931}
|
||||
{"i": 44, "enqueue_s": 18.316064834594727, "start_s": 18.390928030014038, "end_s": 20.15392804145813, "queue_wait_s": 0.07486319541931152, "assert_ok": true, "t_total": 1.763}
|
||||
{"i": 43, "enqueue_s": 18.304176092147827, "start_s": 18.376128911972046, "end_s": 20.292128801345825, "queue_wait_s": 0.07195281982421875, "assert_ok": true, "t_total": 1.916}
|
||||
{"i": 45, "enqueue_s": 19.696249961853027, "start_s": 19.792014837265015, "end_s": 21.3420147895813, "queue_wait_s": 0.0957648754119873, "assert_ok": true, "t_total": 1.55}
|
||||
{"i": 46, "enqueue_s": 20.108312845230103, "start_s": 20.20410394668579, "end_s": 22.059103965759277, "queue_wait_s": 0.09579110145568848, "assert_ok": true, "t_total": 1.855}
|
||||
{"i": 47, "enqueue_s": 20.154053926467896, "start_s": 20.24013590812683, "end_s": 22.06013584136963, "queue_wait_s": 0.08608198165893555, "assert_ok": true, "t_total": 1.82}
|
||||
{"i": 48, "enqueue_s": 20.29221487045288, "start_s": 20.365659713745117, "end_s": 22.143659830093384, "queue_wait_s": 0.07344484329223633, "assert_ok": true, "t_total": 1.778}
|
||||
{"i": 49, "enqueue_s": 21.342138051986694, "start_s": 21.42922592163086, "end_s": 22.98422598838806, "queue_wait_s": 0.08708786964416504, "assert_ok": true, "t_total": 1.555}
|
||||
{"i": 50, "enqueue_s": 22.05921196937561, "start_s": 22.14810585975647, "end_s": 23.752105951309204, "queue_wait_s": 0.08889389038085938, "assert_ok": true, "t_total": 1.604}
|
||||
{"i": 51, "enqueue_s": 22.06025195121765, "start_s": 22.136942863464355, "end_s": 23.833942890167236, "queue_wait_s": 0.0766909122467041, "assert_ok": true, "t_total": 1.697}
|
||||
{"i": 52, "enqueue_s": 22.143775939941406, "start_s": 22.23003077507019, "end_s": 24.156030893325806, "queue_wait_s": 0.08625483512878418, "assert_ok": true, "t_total": 1.926}
|
||||
{"i": 53, "enqueue_s": 22.984335899353027, "start_s": 23.072847843170166, "end_s": 24.64084792137146, "queue_wait_s": 0.08851194381713867, "assert_ok": true, "t_total": 1.568}
|
||||
{"i": 54, "enqueue_s": 23.75220799446106, "start_s": 23.843013048171997, "end_s": 25.400012969970703, "queue_wait_s": 0.0908050537109375, "assert_ok": true, "t_total": 1.557}
|
||||
{"i": 55, "enqueue_s": 23.834028005599976, "start_s": 23.906961679458618, "end_s": 25.46196174621582, "queue_wait_s": 0.07293367385864258, "assert_ok": true, "t_total": 1.555}
|
||||
{"i": 56, "enqueue_s": 24.15613889694214, "start_s": 24.24302387237549, "end_s": 26.18502378463745, "queue_wait_s": 0.08688497543334961, "assert_ok": true, "t_total": 1.942}
|
||||
{"i": 57, "enqueue_s": 24.641007900238037, "start_s": 24.721565008163452, "end_s": 26.270565032958984, "queue_wait_s": 0.08055710792541504, "assert_ok": true, "t_total": 1.549}
|
||||
{"i": 58, "enqueue_s": 25.40013289451599, "start_s": 25.493900775909424, "end_s": 27.041900873184204, "queue_wait_s": 0.09376788139343262, "assert_ok": true, "t_total": 1.548}
|
||||
{"i": 59, "enqueue_s": 25.462048768997192, "start_s": 25.535515785217285, "end_s": 27.077515840530396, "queue_wait_s": 0.07346701622009277, "assert_ok": true, "t_total": 1.542}
|
||||
{"i": 60, "enqueue_s": 26.185149908065796, "start_s": 26.284237146377563, "end_s": 28.203237056732178, "queue_wait_s": 0.09908723831176758, "assert_ok": true, "t_total": 1.919}
|
||||
18
bench/lightpanda/raw/t5-stats.tsv
Normal file
18
bench/lightpanda/raw/t5-stats.tsv
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
ts_iso name mem_usage cpu_pct mem_mib cgroup_mib
|
||||
2026-09-01T04:05:18Z bench-s3c-lightpanda 124.9MiB / 256MiB 4.03% 124.900 125.930
|
||||
2026-09-01T04:05:20Z bench-s3c-lightpanda 132.2MiB / 256MiB 8.18% 132.200 135.078
|
||||
2026-09-01T04:05:23Z bench-s3c-lightpanda 137MiB / 256MiB 5.82% 137.000 138.355
|
||||
2026-09-01T04:05:25Z bench-s3c-lightpanda 138.3MiB / 256MiB 4.90% 138.300 139.402
|
||||
2026-09-01T04:05:28Z bench-s3c-lightpanda 136.7MiB / 256MiB 6.84% 136.700 137.121
|
||||
2026-09-01T04:05:30Z bench-s3c-lightpanda 139MiB / 256MiB 4.21% 139.000 142.672
|
||||
2026-09-01T04:05:33Z bench-s3c-lightpanda 140.8MiB / 256MiB 6.70% 140.800 141.246
|
||||
2026-09-01T04:05:35Z bench-s3c-lightpanda 143.4MiB / 256MiB 4.93% 143.400 143.262
|
||||
2026-09-01T04:05:38Z bench-s3c-lightpanda 142MiB / 256MiB 6.36% 142.000 143.094
|
||||
2026-09-01T04:05:41Z bench-s3c-lightpanda 142.3MiB / 256MiB 7.16% 142.300 143.047
|
||||
2026-09-01T04:05:44Z bench-s3c-lightpanda 136.9MiB / 256MiB 0.68% 136.900 138.492
|
||||
2026-09-01T04:05:48Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.03% 136.600 137.863
|
||||
2026-09-01T04:05:50Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.01% 136.600 138.145
|
||||
2026-09-01T04:05:53Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.01% 136.600 137.398
|
||||
2026-09-01T04:05:55Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.01% 136.600 137.973
|
||||
2026-09-01T04:05:58Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.01% 136.600 138.156
|
||||
2026-09-01T04:06:00Z bench-s3c-lightpanda 136.6MiB / 256MiB 0.00% 136.600 138.137
|
||||
|
0
bench/lightpanda/raw/t5.done
Normal file
0
bench/lightpanda/raw/t5.done
Normal file
60
bench/lightpanda/raw/t5.jsonl
Normal file
60
bench/lightpanda/raw/t5.jsonl
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
{"ts": "2026-09-01T12:05:20+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.9, "ttfb": 1.342, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:20+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.901, "ttfb": 1.341, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:20+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.9, "ttfb": 1.342, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:20+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.965, "ttfb": 1.389, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:22+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.866, "ttfb": 1.306, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:22+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.881, "ttfb": 1.308, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:22+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.88, "ttfb": 1.31, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:22+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.901, "ttfb": 1.339, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:23+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.568, "ttfb": 1.012, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:23+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.597, "ttfb": 1.025, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:23+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.565, "ttfb": 1.016, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:24+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.75, "ttfb": 1.194, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:25+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.578, "ttfb": 1.026, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:25+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.76, "ttfb": 1.206, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:25+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.765, "ttfb": 1.215, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:25+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.952, "ttfb": 1.498, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:27+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.623, "ttfb": 1.068, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:27+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.965, "ttfb": 1.412, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:27+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.954, "ttfb": 1.404, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:28+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 2.018, "ttfb": 1.567, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:29+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.551, "ttfb": 0.999, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:29+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.755, "ttfb": 1.204, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:29+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.559, "ttfb": 1.006, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:29+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.551, "ttfb": 0.999, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:30+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.73, "ttfb": 1.174, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:31+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.569, "ttfb": 1.014, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:31+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.557, "ttfb": 1.001, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:31+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.549, "ttfb": 0.999, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:32+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.754, "ttfb": 1.304, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:32+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.61, "ttfb": 1.054, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:33+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.977, "ttfb": 1.419, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:33+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.949, "ttfb": 1.396, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:34+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.797, "ttfb": 1.241, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:34+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.76, "ttfb": 1.206, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:34+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.552, "ttfb": 0.999, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:34+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.553, "ttfb": 0.997, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:36+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.551, "ttfb": 0.996, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:36+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.547, "ttfb": 0.995, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:36+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.552, "ttfb": 0.998, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:36+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.546, "ttfb": 0.993, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:37+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.545, "ttfb": 0.993, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:38+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.931, "ttfb": 1.374, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:38+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.763, "ttfb": 1.198, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:38+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.916, "ttfb": 1.366, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:39+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.55, "ttfb": 0.996, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:40+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.855, "ttfb": 1.3, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:40+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.82, "ttfb": 1.263, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:40+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.778, "ttfb": 1.218, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:41+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.555, "ttfb": 1.002, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:41+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.604, "ttfb": 1.05, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:42+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.697, "ttfb": 1.144, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:42+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.926, "ttfb": 1.372, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:42+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.568, "ttfb": 1.013, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:43+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.557, "ttfb": 0.999, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:43+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.555, "ttfb": 1, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:44+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.942, "ttfb": 1.391, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:44+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.549, "ttfb": 0.998, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:45+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.548, "ttfb": 0.992, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:45+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.542, "ttfb": 0.988, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
{"ts": "2026-09-01T12:05:46+08:00", "scheme": "lightpanda", "template": "T5", "url": "https://quotes.toscrape.com/js/", "final_url": "https://quotes.toscrape.com/js/", "http_code": 200, "http_status_in_page": null, "t_total": 1.919, "ttfb": 1.365, "t_namelookup": 0, "t_connect": 0, "bytes": 8925, "html_len": 8925, "text_len": 1489, "title": "Quotes to Scrape", "price": "", "jsonld_price": "", "quote_count": 10, "blocked": false, "challenge_vendor": "none", "networkidle": true, "assert_ok": true, "note": "", "cdp_mode": "version", "cdp_errors": [], "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"}
|
||||
106
bench/lightpanda/request.sh
Executable file
106
bench/lightpanda/request.sh
Executable file
|
|
@ -0,0 +1,106 @@
|
|||
#!/usr/bin/env bash
|
||||
# T4 / T5 CDP request helper. Env: CDP_HTTP SCHEME TIER URL
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
CDP_HTTP="${CDP_HTTP:-http://127.0.0.1:19222}"
|
||||
SCHEME="${SCHEME:-lightpanda}"
|
||||
TIER="${1:?tier}"
|
||||
URL="${2:?url}"
|
||||
TIMEOUT="${3:-25}"
|
||||
OUT="$(node "$ROOT/cdp_fetch.mjs" "$CDP_HTTP" "$URL" "$TIMEOUT")"
|
||||
python3 - "$OUT" "$SCHEME" "$TIER" "$URL" <<'PY'
|
||||
import json, sys, datetime, re
|
||||
raw, scheme, tier, url = sys.argv[1:5]
|
||||
try:
|
||||
d = json.loads(raw)
|
||||
except Exception as e:
|
||||
d = {"ok": False, "error": f"json {e}", "raw": raw[:500]}
|
||||
title = d.get("title") or ""
|
||||
text = d.get("text") or ""
|
||||
price = d.get("price") or ""
|
||||
jprice = d.get("jsonldPrice") or ""
|
||||
quote_n = int(d.get("quoteCount") or 0)
|
||||
blocked = bool(d.get("blocked"))
|
||||
vendor = d.get("challenge_vendor") or "none"
|
||||
lc = (title + "\n" + text).lower()
|
||||
assert_ok = False
|
||||
note = ""
|
||||
if tier == "T4-ctrl":
|
||||
assert_ok = ("quote" in lc) or quote_n >= 3 or bool(re.search(r"albert einstein|jane austen|marilyn monroe|steve jobs", lc))
|
||||
if not assert_ok:
|
||||
note = "js_kernel_or_empty"
|
||||
elif tier == "T4-shop":
|
||||
assert_ok = (("wool runner" in lc) or ("allbirds" in lc)) and (
|
||||
bool(re.search(r"\$[0-9]", price or text)) or bool(re.search(r"[0-9]", jprice))
|
||||
)
|
||||
if not assert_ok:
|
||||
note = "shop_fields_missing"
|
||||
elif tier == "T4-amz":
|
||||
if blocked or re.search(r"robot|captcha|sorry|click the button|continue shopping", lc):
|
||||
assert_ok = True # interception is expected; not a scheme fail
|
||||
note = "waf"
|
||||
blocked = True
|
||||
else:
|
||||
assert_ok = bool(title.strip()) and not blocked
|
||||
elif tier == "T4-medium":
|
||||
assert_ok = bool(title.strip()) and not blocked and len(text) >= 200
|
||||
if blocked:
|
||||
note = "medium_wall_or_waf"
|
||||
elif not assert_ok:
|
||||
note = "medium_thin"
|
||||
elif tier == "T4-t3x":
|
||||
assert_ok = bool(title.strip()) and len(text) >= 80
|
||||
if not assert_ok:
|
||||
note = "serp_thin"
|
||||
elif tier == "T5":
|
||||
assert_ok = ("quote" in lc) or quote_n >= 3
|
||||
if not assert_ok:
|
||||
note = "t5_ctrl_fail"
|
||||
else:
|
||||
assert_ok = bool(d.get("ok")) and bool(title)
|
||||
|
||||
out = {
|
||||
"ts": datetime.datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"scheme": scheme,
|
||||
"template": tier,
|
||||
"url": url,
|
||||
"final_url": d.get("finalUrl") or "",
|
||||
"http_code": 200 if d.get("ok") else 0,
|
||||
"http_status_in_page": None,
|
||||
"t_total": d.get("t_total") or 0,
|
||||
"ttfb": d.get("t_nav_s") or 0,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": d.get("htmlLen") or 0,
|
||||
"html_len": d.get("htmlLen") or 0,
|
||||
"text_len": d.get("textLen") or 0,
|
||||
"title": title[:240],
|
||||
"price": price,
|
||||
"jsonld_price": jprice,
|
||||
"quote_count": quote_n,
|
||||
"blocked": blocked,
|
||||
"challenge_vendor": vendor,
|
||||
"networkidle": d.get("networkidle"),
|
||||
"assert_ok": bool(assert_ok),
|
||||
"note": note or d.get("error") or "",
|
||||
"cdp_mode": d.get("cdp_mode"),
|
||||
"cdp_errors": d.get("cdp_errors") or [],
|
||||
"text_head": text[:500],
|
||||
}
|
||||
print(json.dumps(out, ensure_ascii=False))
|
||||
# excerpt to stderr for samples
|
||||
excerpt = {
|
||||
"tier": tier,
|
||||
"title": title[:200],
|
||||
"final_url": out["final_url"],
|
||||
"html_len": out["html_len"],
|
||||
"text_len": out["text_len"],
|
||||
"price": price,
|
||||
"jsonld_price": jprice,
|
||||
"blocked": blocked,
|
||||
"vendor": vendor,
|
||||
"text_head": text[:800],
|
||||
"html_head": (d.get("htmlHead") or "")[:800],
|
||||
}
|
||||
sys.stderr.write(json.dumps(excerpt, ensure_ascii=False) + "\n")
|
||||
PY
|
||||
379
bench/lightpanda/results.json
Normal file
379
bench/lightpanda/results.json
Normal file
|
|
@ -0,0 +1,379 @@
|
|||
{
|
||||
"scheme": "lightpanda",
|
||||
"group": "foreign_proxy",
|
||||
"image": {
|
||||
"ref": "docker.io/lightpanda/browser:0.3.7",
|
||||
"digest": "sha256:eae2c7f8ff39a7a24540d21c7561ce219fd14b1bf2b92403930c6d140f2d1337",
|
||||
"id": "sha256:eae2c7f8ff39a7a24540d21c7561ce219fd14b1bf2b92403930c6d140f2d1337",
|
||||
"arch": "arm64",
|
||||
"pull": "hub"
|
||||
},
|
||||
"probed_at": "2026-09-01T12:06:03+08:00",
|
||||
"commands": [
|
||||
"docker compose -p bench-s3c-lightpanda up -d",
|
||||
"node cdp_fetch.mjs http://127.0.0.1:19222 <url>",
|
||||
"T5 ThreadPoolExecutor workers=4 n=60"
|
||||
],
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "204",
|
||||
"host_docker_internal": "fdc4:f303:9324::254 host.docker.internal",
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway"
|
||||
},
|
||||
"rss_source": "docker_stats+cgroup",
|
||||
"idle_mb_p50": 3.559,
|
||||
"session_delta_mb": 247.841,
|
||||
"burst_peak_mb": 143.4,
|
||||
"session_peak_mb": 251.4,
|
||||
"templates": {
|
||||
"T4-ctrl": {
|
||||
"n": 3,
|
||||
"success": 3,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 1.727,
|
||||
"p95_s": 1.8872,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1489,
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1489,
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"html_len": 8925,
|
||||
"text_len": 1489,
|
||||
"title": "Quotes to Scrape"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-medium": {
|
||||
"n": 3,
|
||||
"success": 0,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 11.307,
|
||||
"p95_s": 11.4996,
|
||||
"assert_ok": false,
|
||||
"note": "medium_wall_or_waf",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 5193,
|
||||
"text_len": 725,
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"html_len": 5193,
|
||||
"text_len": 725,
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"html_len": 5193,
|
||||
"text_len": 725,
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-shop": {
|
||||
"n": 4,
|
||||
"success": 4,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 14.951,
|
||||
"p95_s": 15.4248,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 1439527,
|
||||
"text_len": 3426,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"html_len": 1438467,
|
||||
"text_len": 3426,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"html_len": 1439043,
|
||||
"text_len": 3426,
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-amz": {
|
||||
"n": 6,
|
||||
"success": 6,
|
||||
"http_codes": [
|
||||
200,
|
||||
200,
|
||||
200,
|
||||
200,
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 1.349,
|
||||
"p95_s": 1.7147,
|
||||
"assert_ok": true,
|
||||
"note": "waf",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 3479,
|
||||
"text_len": 142,
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"html_len": 3479,
|
||||
"text_len": 142,
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"html_len": 3479,
|
||||
"text_len": 142,
|
||||
"title": "Amazon.com"
|
||||
}
|
||||
]
|
||||
},
|
||||
"T4-t3x": {
|
||||
"n": 2,
|
||||
"success": 2,
|
||||
"http_codes": [
|
||||
200,
|
||||
200
|
||||
],
|
||||
"p50_s": 3.428,
|
||||
"p95_s": 3.9086,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"html_vs_text": [
|
||||
{
|
||||
"html_len": 82696,
|
||||
"text_len": 1000,
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
},
|
||||
{
|
||||
"html_len": 82913,
|
||||
"text_len": 1000,
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"t5": {
|
||||
"n": 60,
|
||||
"parallelism": 4,
|
||||
"queue_impl": "client_fifo",
|
||||
"success": 60,
|
||||
"success_rate": 1.0,
|
||||
"p50_s": 1.74,
|
||||
"p95_s": 1.965,
|
||||
"oom": false,
|
||||
"http_429": 0,
|
||||
"http_503": 0,
|
||||
"first_complete_s": 2.0435,
|
||||
"last_complete_s": 28.2032,
|
||||
"queue_wait_p50_s": 13.0441,
|
||||
"queue_wait_p95_s": 24.7602,
|
||||
"rejected": 0,
|
||||
"wall_s": 28.203,
|
||||
"note": "生产应是 SQLite WAL + CONCURRENT；本轮用脚本侧 worker 池代替",
|
||||
"queue_wait_note": "queue_wait = 任务开始时刻相对 burst t=0（FIFO 排队）；request.sh 内记录的 enqueue 在 worker 启动时取值，已弃用"
|
||||
},
|
||||
"t4_waf": [
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": true,
|
||||
"vendor": "amazon",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": true,
|
||||
"vendor": "amazon",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"tier": "T4-ctrl",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"title": "Quotes to Scrape"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": true,
|
||||
"vendor": "amazon",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"tier": "T4-t3x",
|
||||
"url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit",
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": true,
|
||||
"vendor": "cloudflare",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": true,
|
||||
"vendor": "cloudflare",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"tier": "T4-medium",
|
||||
"url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"blocked": true,
|
||||
"vendor": "cloudflare",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395",
|
||||
"title": "Attention Required! | Cloudflare"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-shop",
|
||||
"url": "https://www.allbirds.com/products/mens-wool-runners",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.allbirds.com/collections/mens",
|
||||
"title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": true,
|
||||
"vendor": "amazon",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Page Not Found"
|
||||
},
|
||||
{
|
||||
"tier": "T4-amz",
|
||||
"url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"blocked": true,
|
||||
"vendor": "amazon",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://www.amazon.com/dp/B08N5WRWNW",
|
||||
"title": "Amazon.com"
|
||||
},
|
||||
{
|
||||
"tier": "T4-t3x",
|
||||
"url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit",
|
||||
"blocked": false,
|
||||
"vendor": "none",
|
||||
"http_status_in_page": null,
|
||||
"final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit",
|
||||
"title": "best bluetooth earbuds 2026 reddit at DuckDuckGo"
|
||||
}
|
||||
],
|
||||
"scorecard": {
|
||||
"mem": 80.0,
|
||||
"quality": 100,
|
||||
"stability": 100,
|
||||
"latency": 100.0,
|
||||
"ops": 70,
|
||||
"total": 91.0
|
||||
},
|
||||
"notes": [
|
||||
"queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT",
|
||||
"image pull: docker.io hub linux/arm64, 无需 daocloud",
|
||||
"T4-shop: 商品 URL 被站点 302 到 /collections/mens，标题含 Allbirds，列表页可见 $ 价，无单品 jsonld",
|
||||
"T4-amz: Amazon 机器人验证页 title=Amazon.com html~3.5KB text~142，vendor=amazon",
|
||||
"T4-medium: Cloudflare Attention Required，全程 blocked",
|
||||
"session 峰值 251.4MiB / mem_limit 256m，出现在 Allbirds 列表页（html ~1.4MB）"
|
||||
]
|
||||
}
|
||||
246
bench/lightpanda/run.sh
Executable file
246
bench/lightpanda/run.sh
Executable file
|
|
@ -0,0 +1,246 @@
|
|||
#!/usr/bin/env bash
|
||||
# lightpanda S3c: up → idle 5min → T4 tiers ×3 → T5 (worker=4) → summarize
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$ROOT/_lib.sh"
|
||||
NAME="bench-s3c-lightpanda"
|
||||
IMG="docker.io/lightpanda/browser:0.3.7"
|
||||
CDP_HTTP="${CDP_HTTP:-http://127.0.0.1:19222}"
|
||||
SCHEME="${SCHEME:-lightpanda}"
|
||||
COMPOSE_PROJ="${COMPOSE_PROJ:-bench-s3c-lightpanda}"
|
||||
T5_PAR="${T5_PAR:-4}"
|
||||
RAW="$ROOT/raw"
|
||||
SAMP="$ROOT/samples"
|
||||
mkdir -p "$RAW" "$SAMP"
|
||||
export CDP_HTTP SCHEME
|
||||
exec > >(tee -a "$RAW/run.log") 2>&1
|
||||
|
||||
log "=== $SCHEME S3c start ==="
|
||||
ensure_image "$IMG" "$RAW/pull.how"
|
||||
PULL_HOW="$(cat "$RAW/pull.how" 2>/dev/null || echo unknown)"
|
||||
|
||||
if [[ "${SKIP_UP:-}" != "1" ]]; then
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
log "docker compose up -d project=$COMPOSE_PROJ"
|
||||
(cd "$ROOT" && docker compose -p "$COMPOSE_PROJ" up -d --force-recreate)
|
||||
fi
|
||||
|
||||
log "wait CDP $CDP_HTTP/json/version"
|
||||
if ! wait_http "$CDP_HTTP/json/version" 40; then
|
||||
log "json/version not ready; try /json/list"
|
||||
wait_http "$CDP_HTTP/json/list" 20 || true
|
||||
docker logs --tail 80 "$NAME" | tee "$RAW/container-start.log" || true
|
||||
fi
|
||||
curl -sS --max-time 5 "$CDP_HTTP/json/version" | tee "$RAW/cdp-version.json" || true
|
||||
echo
|
||||
|
||||
ARCH="$(docker image inspect --format '{{.Architecture}}' "$IMG" 2>/dev/null || echo '?')"
|
||||
DIGEST="$(docker inspect --format '{{index .RepoDigests 0}}' "$NAME" 2>/dev/null || true)"
|
||||
IID="$(docker inspect --format '{{.Image}}' "$NAME" 2>/dev/null || true)"
|
||||
log "image arch=$ARCH digest=$DIGEST pull=$PULL_HOW"
|
||||
|
||||
PROXY_CODE="$(curl -sS -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 20 \
|
||||
-x http://127.0.0.1:17890 -A onesvm-bench/s3c \
|
||||
https://www.google.com/generate_204 || true)"
|
||||
log "proxy generate_204 http=$PROXY_CODE"
|
||||
HOST_RES="$(docker exec "$NAME" sh -c 'getent hosts host.docker.internal || true' 2>/dev/null || echo unresolved)"
|
||||
log "host.docker.internal => $HOST_RES"
|
||||
|
||||
if [[ "${SKIP_IDLE:-}" != "1" ]]; then
|
||||
log "idle 300s interval=5"
|
||||
sample_stats "$RAW/idle.tsv" 5 300 "$NAME"
|
||||
log "idle done"
|
||||
else
|
||||
log "SKIP_IDLE=1"
|
||||
fi
|
||||
|
||||
T4_CTRL="https://quotes.toscrape.com/js/"
|
||||
T4_MEDIUM="https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395"
|
||||
T4_SHOP="https://www.allbirds.com/products/mens-wool-runners"
|
||||
T4_AMZ="https://www.amazon.com/dp/B08N5WRWNW"
|
||||
T4_T3X="https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit"
|
||||
|
||||
SENT="$RAW/session.done"
|
||||
rm -f "$SENT"
|
||||
sample_until_file "$RAW/session.tsv" 1 "$SENT" 10 "$NAME" &
|
||||
STAT_PID=$!
|
||||
sleep 10
|
||||
|
||||
: > "$RAW/t4.jsonl"
|
||||
run_tier() {
|
||||
local tier="$1" url="$2" n="$3"
|
||||
local i
|
||||
for i in $(seq 1 "$n"); do
|
||||
log "$tier trial $i url=$url"
|
||||
set +e
|
||||
line="$(bash "$ROOT/request.sh" "$tier" "$url" 25 2>"$RAW/${tier}-${i}.excerpt")"
|
||||
rc=$?
|
||||
set -e
|
||||
echo "$line" | tee -a "$RAW/t4.jsonl"
|
||||
if [[ "$i" -eq 1 ]]; then
|
||||
cp "$RAW/${tier}-${i}.excerpt" "$SAMP/${tier}.json" 2>/dev/null || true
|
||||
fi
|
||||
log "$tier trial $i rc=$rc"
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
run_tier T4-ctrl "$T4_CTRL" 3
|
||||
run_tier T4-medium "$T4_MEDIUM" 3
|
||||
run_tier T4-shop "$T4_SHOP" 3
|
||||
run_tier T4-amz "$T4_AMZ" 3
|
||||
run_tier T4-t3x "$T4_T3X" 1
|
||||
|
||||
touch "$SENT"
|
||||
wait "$STAT_PID" || true
|
||||
|
||||
log "T5 60x T4-ctrl parallelism=$T5_PAR queue=client_fifo"
|
||||
SENT5="$RAW/t5.done"
|
||||
rm -f "$SENT5"
|
||||
sample_until_file "$RAW/t5-stats.tsv" 1 "$SENT5" 15 "$NAME" &
|
||||
STAT5_PID=$!
|
||||
T5_START_MS=$(python3 -c 'import time;print(int(time.time()*1000))')
|
||||
: > "$RAW/t5.jsonl"
|
||||
: > "$RAW/t5-queue.jsonl"
|
||||
|
||||
python3 - "$ROOT" "$T4_CTRL" "$T5_PAR" "$T5_START_MS" "$RAW/t5.jsonl" "$RAW/t5-queue.jsonl" <<'PY'
|
||||
import subprocess, time, json, os, sys
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
root, url, par, t0, outp, qpath = sys.argv[1:7]
|
||||
par = int(par)
|
||||
t0 = int(t0) / 1000.0
|
||||
n = 60
|
||||
req = os.path.join(root, "request.sh")
|
||||
|
||||
def one(i):
|
||||
enq = time.time()
|
||||
p = subprocess.run(["bash", req, "T5", url, "30"], capture_output=True, text=True)
|
||||
end = time.time()
|
||||
line = p.stdout.strip().splitlines()[-1] if p.stdout.strip() else "{}"
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except Exception:
|
||||
row = {"assert_ok": False, "t_total": end - enq, "note": "parse", "http_code": 0}
|
||||
start = end - float(row.get("t_total") or 0)
|
||||
q = {
|
||||
"i": i,
|
||||
"enqueue_s": enq - t0,
|
||||
"start_s": start - t0,
|
||||
"end_s": end - t0,
|
||||
"queue_wait_s": max(0.0, start - enq),
|
||||
"assert_ok": row.get("assert_ok"),
|
||||
"t_total": row.get("t_total"),
|
||||
}
|
||||
return row, q
|
||||
|
||||
rows = []
|
||||
qs = []
|
||||
with ThreadPoolExecutor(max_workers=par) as ex:
|
||||
futs = [ex.submit(one, i) for i in range(1, n + 1)]
|
||||
for f in as_completed(futs):
|
||||
row, q = f.result()
|
||||
rows.append(row)
|
||||
qs.append(q)
|
||||
with open(outp, "w", encoding="utf-8") as f:
|
||||
for r in rows:
|
||||
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
||||
with open(qpath, "w", encoding="utf-8") as f:
|
||||
for q in qs:
|
||||
f.write(json.dumps(q) + "\n")
|
||||
print(f"t5 done n={len(rows)} ok={sum(1 for r in rows if r.get('assert_ok'))}")
|
||||
PY
|
||||
|
||||
T5_END=$(date +%s)
|
||||
# wall from tsv start roughly
|
||||
T5_WALL=$(python3 - "$RAW/t5-queue.jsonl" <<'PY'
|
||||
import json,sys
|
||||
from pathlib import Path
|
||||
qs=[json.loads(l) for l in Path(sys.argv[1]).read_text().splitlines() if l.strip()]
|
||||
if not qs:
|
||||
print(0); raise SystemExit
|
||||
print(round(max(q["end_s"] for q in qs), 3))
|
||||
PY
|
||||
)
|
||||
log "T5 wall=${T5_WALL}s"
|
||||
touch "$SENT5"
|
||||
wait "$STAT5_PID" || true
|
||||
|
||||
python3 - "$RAW/t5.jsonl" "$SAMP" <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
p=Path(sys.argv[1]); d=Path(sys.argv[2])
|
||||
rows=[json.loads(l) for l in p.read_text().splitlines() if l.strip()]
|
||||
for i,r in enumerate(rows[:2],1):
|
||||
(d/f"t5-{i}.json").write_text(json.dumps(r,ensure_ascii=False,indent=2)+"\n")
|
||||
print("saved t5 samples", len(rows))
|
||||
PY
|
||||
|
||||
# html vs text note from first ctrl excerpt
|
||||
python3 - "$SAMP" "$RAW" <<'PY'
|
||||
import json
|
||||
from pathlib import Path
|
||||
samp=Path("$SAMP".replace("$SAMP", __import__("os").environ.get("SAMP","")))
|
||||
PY
|
||||
|
||||
OOM="$(docker inspect --format '{{.State.OOMKilled}}' "$NAME" 2>/dev/null || echo false)"
|
||||
STATUS="$(docker inspect --format '{{.State.Status}}' "$NAME" 2>/dev/null || echo unknown)"
|
||||
log "status=$STATUS oom=$OOM"
|
||||
docker logs --tail 50 "$NAME" | tee "$RAW/container-tail.log" || true
|
||||
|
||||
python3 - <<PY
|
||||
import json, datetime, statistics, math
|
||||
from pathlib import Path
|
||||
raw = Path("$RAW")
|
||||
qs = []
|
||||
qp = raw / "t5-queue.jsonl"
|
||||
if qp.exists():
|
||||
qs = [json.loads(l) for l in qp.read_text().splitlines() if l.strip()]
|
||||
def pct(xs, p):
|
||||
if not xs: return None
|
||||
s=sorted(xs)
|
||||
k=(len(s)-1)*p/100
|
||||
f=int(k); c=int(math.ceil(k))
|
||||
if f==c: return round(s[f],4)
|
||||
return round(s[f]+(s[c]-s[f])*(k-f),4)
|
||||
waits=[q.get("queue_wait_s") or 0 for q in qs]
|
||||
ends=[q.get("end_s") or 0 for q in qs]
|
||||
meta = {
|
||||
"ref": "$IMG",
|
||||
"digest": None if "$DIGEST" in ("<no value>", "") else "$DIGEST",
|
||||
"id": "$IID",
|
||||
"arch": "$ARCH",
|
||||
"pull": "$PULL_HOW",
|
||||
"probed_at": datetime.datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"oom": str("$OOM").lower()=="true",
|
||||
"container_status": "$STATUS",
|
||||
"t5_wall_s": float("$T5_WALL" or 0),
|
||||
"t5_parallelism": int("$T5_PAR"),
|
||||
"queue_impl": "client_fifo",
|
||||
"queue": {
|
||||
"first_complete_s": round(min(ends),4) if ends else None,
|
||||
"last_complete_s": round(max(ends),4) if ends else None,
|
||||
"queue_wait_p50_s": pct(waits,50),
|
||||
"queue_wait_p95_s": pct(waits,95),
|
||||
"rejected": 0,
|
||||
},
|
||||
"proxy": {
|
||||
"mixed": "http://127.0.0.1:17890",
|
||||
"container_via": "http://host.docker.internal:17890",
|
||||
"generate_204_http": "$PROXY_CODE",
|
||||
"host_docker_internal": """$HOST_RES""".strip(),
|
||||
"egress_note": "HTTP(S)_PROXY + extra_hosts host-gateway",
|
||||
},
|
||||
"commands": [
|
||||
"docker compose -p $COMPOSE_PROJ up -d",
|
||||
"node cdp_fetch.mjs $CDP_HTTP <url>",
|
||||
"T5 ThreadPoolExecutor workers=$T5_PAR n=60",
|
||||
],
|
||||
"notes": ["queue_impl=client_fifo; 生产应是 SQLite WAL + CONCURRENT"],
|
||||
}
|
||||
(raw/"image-meta.json").write_text(json.dumps(meta, ensure_ascii=False, indent=2)+"\n")
|
||||
PY
|
||||
|
||||
python3 "$ROOT/summarize.py"
|
||||
log "=== $SCHEME S3c done ==="
|
||||
1
bench/lightpanda/samples/T4-amz.json
Normal file
1
bench/lightpanda/samples/T4-amz.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-amz", "title": "Amazon.com", "final_url": "https://www.amazon.com/dp/B08N5WRWNW", "html_len": 3479, "text_len": 142, "price": "", "jsonld_price": "", "blocked": true, "vendor": "amazon", "text_head": "Click the button below to continue shopping\nContinue shopping\nConditions of Use Privacy Policy\n© 1996-2025, Amazon.com, Inc. or its affiliates", "html_head": "<html class=\"a-no-js\" lang=\"en-us\"><!--<![endif]--><head>\n<meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\">\n<meta charset=\"utf-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\">\n<title dir=\"ltr\">Amazon.com</title>\n<meta name=\"viewport\" content=\"width=device-width\">\n<link rel=\"stylesheet\" href=\"https://images-na.ssl-images-amazon.com/images/G/01/AUIClients/AmazonUI-3c913031596ca78a3768f4e934b1cc02ce238101.secure.min._V1_.css\">\n</head>\n<body>\n\n<!--\n To discuss automated access to Amazon data please contact api-services-support@amazon.com.\n For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_c_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/adverti"}
|
||||
1
bench/lightpanda/samples/T4-ctrl.json
Normal file
1
bench/lightpanda/samples/T4-ctrl.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-ctrl", "title": "Quotes to Scrape", "final_url": "https://quotes.toscrape.com/js/", "html_len": 8925, "text_len": 1489, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life live miracle miracles\n“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”by Jane Austen\nTags: aliteracy books classic humor\n“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”by Marilyn ", "html_head": "<html lang=\"en\"><head>\n\t<meta charset=\"UTF-8\">\n\t<title>Quotes to Scrape</title>\n <link rel=\"stylesheet\" href=\"/static/bootstrap.min.css\">\n <link rel=\"stylesheet\" href=\"/static/main.css\">\n \n \n</head>\n<body>\n <div class=\"container\">\n <div class=\"row header-box\">\n <div class=\"col-md-8\">\n <h1>\n <a href=\"/\" style=\"text-decoration: none\">Quotes to Scrape</a>\n </h1>\n </div>\n <div class=\"col-md-4\">\n <p>\n \n <a href=\"/login\">Login</a>\n \n </p>\n </div>\n </div>\n \n<script src=\"/static/jquery.js\"></script>\n<script>\n var data = [\n {\n \"tags\": [\n \"change\",\n \"deep-thoughts\""}
|
||||
1
bench/lightpanda/samples/T4-medium.json
Normal file
1
bench/lightpanda/samples/T4-medium.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-medium", "title": "Attention Required! | Cloudflare", "final_url": "https://medium.com/airbnb-engineering/rearchitecting-airbnbs-frontend-b5da7e3e8395", "html_len": 5193, "text_len": 725, "price": "", "jsonld_price": "", "blocked": true, "vendor": "cloudflare", "text_head": "Please enable cookies.\nSorry, you have been blocked\nYou are unable to access medium.com\nWhy have I been blocked?\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.\n\nWhat can I do to resolve this?\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.\n\nCloudflare Ray ID: a34162ec3ec83f37 • Your IP: Click to reveal 207.32.218.93 • Performance & security by Cloudflare", "html_head": "<html class=\"no-js\" lang=\"en-US\"><!--<![endif]--><head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\">\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\">\n<meta name=\"robots\" content=\"noindex, nofollow\">\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\">\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\">\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n "}
|
||||
1
bench/lightpanda/samples/T4-shop.json
Normal file
1
bench/lightpanda/samples/T4-shop.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-shop", "title": "Allbirds: Sustainable Men’s Shoes, Wear-All-Day Comfort", "final_url": "https://www.allbirds.com/collections/mens", "html_len": 1438467, "text_len": 3426, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "Free ground shipping on orders over $100\n\nDue to increased demand, orders may take up to 30 days to ship.\n\nSkip to Content\n0\nNew Arrivals Shop All Men Women\n0\nCart (0)\n\nSpend $100 more to earn free shipping!\n\nYour cart is empty. Start shopping!\n\nShop Womens\nShop Mens\nShop Socks\nShop Women's Sale\nShop Men's Sale\nHome\n/ Men's Shoes\nMen's Shoes\n\nSustainable, supportive, and wildly comfortable, our sneakers are always ready when you are. \n\nFilter and Sort\nmenwomen\nFilter (19 products)\nmenwomen\nFeatured\nMost relevant\nBest selling\nAlphabetically, A-Z\nAlphabetically, Z-A\nPrice, low to high\nPrice, high to low\nDate, old to new\nDate, new to old\nNEWNEW\n\nMen's Dasher NZ\n\nAnthracite\n\n$140\n\nAdd to Cart\n\nMen's Tree Runner NZ\n\nMedium Grey\n\n$100\n\nAdd to Cart\n\nMen's Tree Runner\n\nMist\n\n$100\n\nAdd to Cart\n\nMen", "html_head": "<html class=\"h-full\" lang=\"en-US\"><head><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/identify_ea043f1c.js\"></script><script async src=\"https://s.pinimg.com/ct/lib/main.e47a302d.js\"></script><script type=\"text/javascript\" async src=\"https://analytics.tiktok.com/i18n/pixel/static/main.MWFiZGNkNWVmMQ.js\" data-id=\"C605T0O68TKST8M2ROJ0\"></script><script src=\"https://connect.facebook.net/signals/config/1689016458013442?v=2.9.390&r=stable&domain=www.allbirds.com&hme=a767ae4c6a92a40ee4d7e358a937ad845c8bdb98a3bbe3412069dded3a7be57d&ex_m=115%2C221%2C166%2C26%2C80%2C81%2C158%2C76%2C75%2C11%2C176%2C100%2C20%2C150%2C138%2C45%2C83%2C88%2C146%2C171%2C178%2C31%2C17%2C30%2C18%2C32%2C33%2C34%2C36%2C54%2C159%2C85%2C123%2C22%2C24%2C50%2C46%2C48%2C47%"}
|
||||
1
bench/lightpanda/samples/T4-t3x.json
Normal file
1
bench/lightpanda/samples/T4-t3x.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"tier": "T4-t3x", "title": "best bluetooth earbuds 2026 reddit at DuckDuckGo", "final_url": "https://duckduckgo.com/?q=best+bluetooth+earbuds+2026+reddit", "html_len": 82913, "text_len": 1000, "price": "", "jsonld_price": "", "blocked": false, "vendor": "none", "text_head": "DuckDuckGo\nProtection. Privacy. Peace of mind.\nOpen menu\nAll\nImages\nVideos\nNews\nMaps\nShopping\nDuck.ai\nSearch Settings\nPrivate\nAll regions\nSafe search: moderate\nAny time\nClose menu\nSearch\nHomepage\nThemes\nSettings\nShare Feedback\nDownloads\niOS Browser\nAndroid Browser\nMac Browser\nWindows Browser\nBrowser Extensions\nMore From DuckDuckGo\nDuckDuckGo Subscription\nDuck.ai\nEmail Protection\nNewsletter\nBlog\nPodcast\nCollaborations\nLearn More\nWhat’s New\nCompare Privacy\nAbout Our Browser\nAbout DuckDuckGo\nOther Resources\nHelp\nCommunity\nCareers\nPrivacy Policy\nTerms of Service\nPress Kit\nAdvertise on Search\nSee What’s DuckDuckNew\n\nExplore DuckDuckGo’s latest product updates.\n\nLearn More\nLearn More\nWhat's New\nCompare Privacy\nSearch Shortcuts\nHelp Pages\nGet More\nSubscription\nEmail Protection\nPodcast & Newslette", "html_head": "<html lang=\"en-US\" class=\"has-zcm is-link-style-exp is-link-order-exp is-link-breadcrumb-exp is-related-search-exp is-vertical-tabs-exp js no-touch opacity csstransforms3d csstransitions svg cssfilters is-not-mobile-device full-urls breadcrumb-urls react has-footer\" style=\"--sds-color-palette-black: #000000; --sds-color-palette-white: #FFFFFF; --sds-color-palette-transparent: transparent; --sds-color-palette-gray-100: #111111; --sds-color-palette-gray-95: #161616; --sds-color-palette-gray-90: #222222; --sds-color-palette-gray-85: #333333; --sds-color-palette-gray-80: #444444; --sds-color-palette-gray-70: #666666; --sds-color-palette-gray-60: #888888; --sds-color-palette-gray-50: #aaaaaa; --sds-color-palette-gray-40: #cccccc; --sds-color-palette-gray-30: #dddddd; --sds-color-palette-gra"}
|
||||
28
bench/lightpanda/samples/t5-1.json
Normal file
28
bench/lightpanda/samples/t5-1.json
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"ts": "2026-09-01T12:05:20+08:00",
|
||||
"scheme": "lightpanda",
|
||||
"template": "T5",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"http_code": 200,
|
||||
"http_status_in_page": null,
|
||||
"t_total": 1.9,
|
||||
"ttfb": 1.342,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": 8925,
|
||||
"html_len": 8925,
|
||||
"text_len": 1489,
|
||||
"title": "Quotes to Scrape",
|
||||
"price": "",
|
||||
"jsonld_price": "",
|
||||
"quote_count": 10,
|
||||
"blocked": false,
|
||||
"challenge_vendor": "none",
|
||||
"networkidle": true,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"cdp_mode": "version",
|
||||
"cdp_errors": [],
|
||||
"text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"
|
||||
}
|
||||
28
bench/lightpanda/samples/t5-2.json
Normal file
28
bench/lightpanda/samples/t5-2.json
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"ts": "2026-09-01T12:05:20+08:00",
|
||||
"scheme": "lightpanda",
|
||||
"template": "T5",
|
||||
"url": "https://quotes.toscrape.com/js/",
|
||||
"final_url": "https://quotes.toscrape.com/js/",
|
||||
"http_code": 200,
|
||||
"http_status_in_page": null,
|
||||
"t_total": 1.901,
|
||||
"ttfb": 1.341,
|
||||
"t_namelookup": 0,
|
||||
"t_connect": 0,
|
||||
"bytes": 8925,
|
||||
"html_len": 8925,
|
||||
"text_len": 1489,
|
||||
"title": "Quotes to Scrape",
|
||||
"price": "",
|
||||
"jsonld_price": "",
|
||||
"quote_count": 10,
|
||||
"blocked": false,
|
||||
"challenge_vendor": "none",
|
||||
"networkidle": true,
|
||||
"assert_ok": true,
|
||||
"note": "",
|
||||
"cdp_mode": "version",
|
||||
"cdp_errors": [],
|
||||
"text_head": "Quotes to Scrape\n\nLogin\n\n“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”by Albert Einstein\nTags: change deep-thoughts thinking world\n“It is our choices, Harry, that show what we truly are, far more than our abilities.”by J.K. Rowling\nTags: abilities choices\n“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein\nTags: inspirational life"
|
||||
}
|
||||
275
bench/lightpanda/summarize.py
Normal file
275
bench/lightpanda/summarize.py
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Build lightpanda/results.json from raw artifacts. No secrets."""
|
||||
import json, math, os, statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
RAW = ROOT / "raw"
|
||||
SCHEME = "lightpanda"
|
||||
|
||||
|
||||
def percentile(xs, p):
|
||||
if not xs:
|
||||
return None
|
||||
s = sorted(xs)
|
||||
if len(s) == 1:
|
||||
return round(s[0], 4)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = math.floor(k)
|
||||
c = math.ceil(k)
|
||||
if f == c:
|
||||
return round(s[int(k)], 4)
|
||||
return round(s[f] + (s[c] - s[f]) * (k - f), 4)
|
||||
|
||||
|
||||
def load_jsonl(p):
|
||||
rows = []
|
||||
if not p.exists():
|
||||
return rows
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
rows.append(json.loads(line))
|
||||
return rows
|
||||
|
||||
|
||||
def tsv_mibs(p):
|
||||
vals = []
|
||||
if not p.exists():
|
||||
return vals
|
||||
lines = p.read_text(encoding="utf-8").splitlines()
|
||||
if len(lines) < 2:
|
||||
return vals
|
||||
hdr = lines[0].split("\t")
|
||||
i = hdr.index("mem_mib") if "mem_mib" in hdr else 4
|
||||
for line in lines[1:]:
|
||||
parts = line.split("\t")
|
||||
if len(parts) <= i:
|
||||
continue
|
||||
try:
|
||||
v = float(parts[i])
|
||||
except ValueError:
|
||||
continue
|
||||
if v > 0:
|
||||
vals.append(v)
|
||||
return vals
|
||||
|
||||
|
||||
def median(xs):
|
||||
return round(statistics.median(xs), 3) if xs else None
|
||||
|
||||
|
||||
def pmax(xs):
|
||||
return round(max(xs), 3) if xs else None
|
||||
|
||||
|
||||
def mem_score(idle, delta, burst):
|
||||
def band(v, cuts):
|
||||
if v is None:
|
||||
return None
|
||||
for score, lim in cuts:
|
||||
if v <= lim:
|
||||
return score
|
||||
return 0
|
||||
parts = [
|
||||
band(idle, [(100, 60), (80, 100), (60, 200), (40, 400)]),
|
||||
band(delta, [(100, 30), (80, 80), (60, 200), (40, 350)]),
|
||||
band(burst, [(100, 200), (80, 400), (60, 700), (40, 1000)]),
|
||||
]
|
||||
parts = [x for x in parts if x is not None]
|
||||
return round(sum(parts) / 3, 1) if parts else None
|
||||
|
||||
|
||||
def quality_js(t4):
|
||||
ctrl = [r for r in t4 if r.get("template") == "T4-ctrl"]
|
||||
shop = [r for r in t4 if r.get("template") == "T4-shop"]
|
||||
if any(r.get("assert_ok") for r in ctrl):
|
||||
if any(r.get("assert_ok") and not r.get("note") == "shop_fields_missing" for r in shop) or any(
|
||||
r.get("assert_ok") for r in shop
|
||||
):
|
||||
return 100
|
||||
# ctrl ok, shop fields missing but text present
|
||||
if shop and any((r.get("text_len") or 0) > 200 for r in shop):
|
||||
return 70
|
||||
return 70
|
||||
if any((r.get("text_len") or 0) > 100 for r in t4):
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def stab_score(rate, oom):
|
||||
if oom:
|
||||
return 0
|
||||
if rate is None:
|
||||
return None
|
||||
if rate >= 0.99:
|
||||
return 100
|
||||
if rate >= 0.95:
|
||||
return 80
|
||||
if rate >= 0.85:
|
||||
return 60
|
||||
if rate >= 0.70:
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def lat_t4(p95):
|
||||
if p95 is None:
|
||||
return None
|
||||
if p95 <= 5:
|
||||
return 100
|
||||
if p95 <= 12:
|
||||
return 70
|
||||
if p95 <= 25:
|
||||
return 40
|
||||
return 0
|
||||
|
||||
|
||||
def lat_t5(t5_p95, t4_p95):
|
||||
if t5_p95 is None:
|
||||
return None
|
||||
if t4_p95 and t4_p95 > 0:
|
||||
ratio = t5_p95 / t4_p95
|
||||
if ratio <= 3:
|
||||
return 100
|
||||
if ratio <= 5:
|
||||
return 70
|
||||
if ratio <= 8:
|
||||
return 40
|
||||
return 0
|
||||
return 40 if t5_p95 <= 25 else 0
|
||||
|
||||
|
||||
def main():
|
||||
t4 = load_jsonl(RAW / "t4.jsonl")
|
||||
t5 = load_jsonl(RAW / "t5.jsonl")
|
||||
idle = tsv_mibs(RAW / "idle.tsv")
|
||||
sess = tsv_mibs(RAW / "session.tsv")
|
||||
burst = tsv_mibs(RAW / "t5-stats.tsv")
|
||||
meta = {}
|
||||
if (RAW / "image-meta.json").exists():
|
||||
meta = json.loads((RAW / "image-meta.json").read_text(encoding="utf-8"))
|
||||
|
||||
idle_p50 = median(idle)
|
||||
sess_peak = pmax(sess)
|
||||
burst_peak = pmax(burst)
|
||||
delta = None
|
||||
if idle_p50 is not None and sess_peak is not None:
|
||||
delta = round(max(0.0, sess_peak - idle_p50), 3)
|
||||
|
||||
def pack(rows, key):
|
||||
sub = [r for r in rows if r.get("template") == key]
|
||||
times = [r["t_total"] for r in sub if r.get("t_total") is not None]
|
||||
return {
|
||||
"n": len(sub),
|
||||
"success": sum(1 for r in sub if r.get("assert_ok")),
|
||||
"http_codes": [r.get("http_code") for r in sub],
|
||||
"p50_s": percentile(times, 50),
|
||||
"p95_s": percentile(times, 95),
|
||||
"assert_ok": any(r.get("assert_ok") for r in sub),
|
||||
"note": "; ".join(sorted({r.get("note") for r in sub if r.get("note")})),
|
||||
"html_vs_text": [
|
||||
{"html_len": r.get("html_len"), "text_len": r.get("text_len"), "title": r.get("title")}
|
||||
for r in sub[:3]
|
||||
],
|
||||
}
|
||||
|
||||
t4_ctrl_times = [r["t_total"] for r in t4 if r.get("template") == "T4-ctrl" and r.get("t_total") is not None]
|
||||
t5_ok = [r for r in t5 if r.get("assert_ok")]
|
||||
t5_times = [r["t_total"] for r in t5 if r.get("t_total") is not None]
|
||||
t5_n = len(t5)
|
||||
t5_rate = (len(t5_ok) / t5_n) if t5_n else None
|
||||
# T5 assertion uses T4-ctrl quality; scheme stability also considers T4-ctrl
|
||||
ctrl = [r for r in t4 if r.get("template") == "T4-ctrl"]
|
||||
ctrl_rate = (sum(1 for r in ctrl if r.get("assert_ok")) / len(ctrl)) if ctrl else None
|
||||
combined_rate = t5_rate if t5_rate is not None else ctrl_rate
|
||||
|
||||
t4_waf = []
|
||||
for r in t4:
|
||||
t4_waf.append({
|
||||
"tier": r.get("template"),
|
||||
"url": r.get("url"),
|
||||
"blocked": r.get("blocked"),
|
||||
"vendor": r.get("challenge_vendor"),
|
||||
"http_status_in_page": r.get("http_status_in_page"),
|
||||
"final_url": r.get("final_url"),
|
||||
"title": r.get("title"),
|
||||
})
|
||||
|
||||
oom = bool(meta.get("oom"))
|
||||
q = quality_js(t4)
|
||||
t4_p95 = percentile(t4_ctrl_times, 95)
|
||||
t5_p95 = percentile(t5_times, 95)
|
||||
l4 = lat_t4(t4_p95)
|
||||
l5 = lat_t5(t5_p95, t4_p95)
|
||||
lat_parts = [x for x in (l4, l5) if x is not None]
|
||||
lat = round(sum(lat_parts) / len(lat_parts), 1) if lat_parts else None
|
||||
mem = mem_score(idle_p50, delta, burst_peak)
|
||||
stab = stab_score(combined_rate, oom)
|
||||
ops = 70 if SCHEME == "lightpanda" else 55
|
||||
total = None
|
||||
if None not in (mem, q, stab, lat, ops):
|
||||
total = round(mem * 0.30 + q * 0.25 + stab * 0.20 + lat * 0.15 + ops * 0.10, 1)
|
||||
|
||||
queue = meta.get("queue") or {}
|
||||
out = {
|
||||
"scheme": SCHEME,
|
||||
"group": "foreign_proxy",
|
||||
"image": {
|
||||
"ref": meta.get("ref"),
|
||||
"digest": meta.get("digest"),
|
||||
"id": meta.get("id"),
|
||||
"arch": meta.get("arch"),
|
||||
"pull": meta.get("pull"),
|
||||
},
|
||||
"probed_at": meta.get("probed_at"),
|
||||
"commands": meta.get("commands") or [],
|
||||
"proxy": meta.get("proxy") or {},
|
||||
"rss_source": "docker_stats+cgroup",
|
||||
"idle_mb_p50": idle_p50,
|
||||
"session_delta_mb": delta,
|
||||
"burst_peak_mb": burst_peak,
|
||||
"session_peak_mb": sess_peak,
|
||||
"templates": {
|
||||
"T4-ctrl": pack(t4, "T4-ctrl"),
|
||||
"T4-medium": pack(t4, "T4-medium"),
|
||||
"T4-shop": pack(t4, "T4-shop"),
|
||||
"T4-amz": pack(t4, "T4-amz"),
|
||||
"T4-t3x": pack(t4, "T4-t3x"),
|
||||
},
|
||||
"t5": {
|
||||
"n": t5_n,
|
||||
"parallelism": meta.get("t5_parallelism") or 4,
|
||||
"queue_impl": meta.get("queue_impl") or "client_fifo",
|
||||
"success": len(t5_ok),
|
||||
"success_rate": t5_rate,
|
||||
"p50_s": percentile(t5_times, 50),
|
||||
"p95_s": t5_p95,
|
||||
"oom": oom,
|
||||
"http_429": 0,
|
||||
"http_503": 0,
|
||||
"first_complete_s": queue.get("first_complete_s"),
|
||||
"last_complete_s": queue.get("last_complete_s"),
|
||||
"queue_wait_p50_s": queue.get("queue_wait_p50_s"),
|
||||
"queue_wait_p95_s": queue.get("queue_wait_p95_s"),
|
||||
"rejected": queue.get("rejected") or 0,
|
||||
"wall_s": meta.get("t5_wall_s"),
|
||||
"note": "生产应是 SQLite WAL + CONCURRENT；本轮用脚本侧 worker 池代替",
|
||||
},
|
||||
"t4_waf": t4_waf,
|
||||
"scorecard": {
|
||||
"mem": mem,
|
||||
"quality": q,
|
||||
"stability": stab,
|
||||
"latency": lat,
|
||||
"ops": ops,
|
||||
"total": total,
|
||||
},
|
||||
"notes": meta.get("notes") or [],
|
||||
}
|
||||
(ROOT / "results.json").write_text(json.dumps(out, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps({"wrote": str(ROOT / "results.json"), "scorecard": out["scorecard"]}, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
488
bench/plan.md
Normal file
488
bench/plan.md
Normal file
|
|
@ -0,0 +1,488 @@
|
|||
# S3b / S3c 本机 Docker 实测计划
|
||||
|
||||
- 对应筛选：`.dsh/artifacts/run-20260901-browser-arch/select.md`
|
||||
- 主机假设：darwin arm64 / Docker Desktop 29.4.0（与 S3a 相同）
|
||||
- 代理：先 `export PROXY_SUB_URL`（只从 `.dsh/artifacts/proxy-subscriptions.local.md` SUB-1 注入），再 `bash bench/proxy/up.sh`。国外组复用 `http://127.0.0.1:17890`。**脚本与 JSON 禁止出现订阅 URL / uuid / password。**
|
||||
- 同时运行 **候选容器 ≤3**。`bench-proxy-mihomo` 算基建，不占 3 席。
|
||||
- 每方案第一次发业务请求前：**idle 采样 5 分钟**。
|
||||
|
||||
## 0. 目录骨架（S3b 建，本文件只约定）
|
||||
|
||||
```
|
||||
bench/
|
||||
lib/
|
||||
sample_stats.sh # docker stats 循环
|
||||
curl_time.sh # 单次计时
|
||||
summarize.py # 读 raw → results.json
|
||||
searxng-cn/ compose.yml settings.yml request.sh results.json
|
||||
trafilatura-http/ compose.yml Dockerfile app.py request.sh results.json
|
||||
searxng-global/ compose.yml settings.yml request.sh results.json
|
||||
lightpanda/ compose.yml request.sh results.json
|
||||
chrome-headless-shell/ compose.yml request.sh results.json
|
||||
raw/ # 各批次 stats-*.tsv 与 curl-*.jsonl（可选）
|
||||
```
|
||||
|
||||
`results.json` 顶层必须能被评分卡直接引用，字段见 §4。
|
||||
|
||||
---
|
||||
|
||||
## 1. 统一采集规范
|
||||
|
||||
### 1.1 容器命名
|
||||
|
||||
| 方案 | 容器名 | 宿主机端口 |
|
||||
|---|---|---|
|
||||
| searxng-cn | `bench-searxng-cn` | 18880 → 8080 |
|
||||
| trafilatura-http | `bench-trafilatura` | 18881 → 8080 |
|
||||
| searxng-global | `bench-searxng-global` | 18882 → 8080 |
|
||||
| lightpanda | `bench-lightpanda` | 19222 → 9222 |
|
||||
| chrome-headless-shell | `bench-headless-shell` | 19223 → 9222 |
|
||||
| 代理（基建） | `bench-proxy-mihomo` | 17890 / 19090（已有） |
|
||||
|
||||
### 1.2 docker stats
|
||||
|
||||
`bench/lib/sample_stats.sh` 约定（S3b 实现时按此抄）：
|
||||
|
||||
```bash
|
||||
# 用法: sample_stats.sh <label> <interval_s> <duration_s> <container...>
|
||||
# 例: bash bench/lib/sample_stats.sh searxng-cn-idle 5 300 bench-searxng-cn
|
||||
label="$1"; interval="$2"; duration="$3"; shift 3
|
||||
out="bench/raw/${label}.tsv"
|
||||
echo -e "ts_iso\tname\tmem_bytes\tmem_limit_bytes\tcpu_pct" > "$out"
|
||||
end=$(( $(date +%s) + duration ))
|
||||
while [ "$(date +%s)" -lt "$end" ]; do
|
||||
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}' "$@" \
|
||||
| while IFS=$'\t' read -r name mem cpu; do
|
||||
# MemUsage 形如 "85.2MiB / 256MiB" —— summarize.py 解析
|
||||
printf '%s\t%s\t%s\t%s\n' "$ts" "$name" "$mem" "$cpu" >> "$out"
|
||||
done
|
||||
sleep "$interval"
|
||||
done
|
||||
```
|
||||
|
||||
- idle：`interval=5` `duration=300`（5 分钟），**期间禁止发业务请求**。
|
||||
- 单会话：发请求前 10s 起采到请求结束后 10s，`interval=1`。
|
||||
- T5 burst：从 t=0 到全部结束 + 15s，`interval=1`。
|
||||
- 指标：`idle_mb` = 5 min 中位；`session_delta_mb` = 单会话峰值 − idle 中位；`burst_peak_mb` = burst 窗口最大。
|
||||
|
||||
辅助（可选，更准 RSS）：
|
||||
|
||||
```bash
|
||||
docker exec "$c" sh -c 'cat /sys/fs/cgroup/memory.current 2>/dev/null || cat /sys/fs/cgroup/memory/memory.usage_in_bytes'
|
||||
```
|
||||
|
||||
cgroup 读不到就只信 `docker stats`，在 `results.json` 标 `rss_source=docker_stats`。
|
||||
|
||||
### 1.3 curl 计时
|
||||
|
||||
```bash
|
||||
# bench/lib/curl_time.sh
|
||||
# 环境: URL METHOD DATA PROXY(可选)
|
||||
extra=()
|
||||
[ -n "${PROXY:-}" ] && extra+=(-x "$PROXY")
|
||||
code_file=$(mktemp)
|
||||
body_file=$(mktemp)
|
||||
metrics=$(curl -sS "${extra[@]}" -X "${METHOD:-GET}" \
|
||||
-o "$body_file" \
|
||||
-w '%{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{size_download}' \
|
||||
--connect-timeout 10 --max-time "${MAX_TIME:-30}" \
|
||||
-A 'onesvm-bench/s3b' \
|
||||
${DATA:+-H 'Content-Type: application/json' -d "$DATA"} \
|
||||
"$URL")
|
||||
# 输出一行 JSON 到 stdout，body 路径给断言脚本
|
||||
```
|
||||
|
||||
写入 jsonl 的字段：`ts, scheme, template, http_code, t_namelookup, t_connect, ttfb, t_total, bytes, assert_ok, note`。
|
||||
|
||||
p50 / p95：对该模板全部 `t_total` 排序后取分位（线性插值即可）。**不要**用 Excel 手估。
|
||||
|
||||
### 1.4 镜像 pin 与 digest
|
||||
|
||||
每个 `compose.yml` 写死 `select.md` 的 tag，启动后：
|
||||
|
||||
```bash
|
||||
docker inspect --format '{{.Name}} {{.Image}} {{index .RepoDigests 0}}' "$c"
|
||||
```
|
||||
|
||||
写入 `results.json.image`。拉不到 Hub 时用 `docker load`（见 select.md §5），不要改 pin 成 `latest`。
|
||||
|
||||
### 1.5 清理
|
||||
|
||||
每批次结束：
|
||||
|
||||
```bash
|
||||
docker rm -f bench-searxng-cn bench-trafilatura bench-searxng-global bench-lightpanda bench-headless-shell || true
|
||||
# 代理只在当天国外批次全部结束时 down
|
||||
# bash bench/proxy/down.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 五套需求模板
|
||||
|
||||
### T1 国内关键词搜索 — 方案 `searxng-cn`
|
||||
|
||||
**请求（可执行）：**
|
||||
|
||||
```bash
|
||||
curl -sS -G 'http://127.0.0.1:18880/search' \
|
||||
--data-urlencode 'q=跨境电商 出口退税 政策 2026' \
|
||||
--data-urlencode 'format=json' \
|
||||
--data-urlencode 'language=zh-CN' \
|
||||
--data-urlencode 'categories=general'
|
||||
```
|
||||
|
||||
**断言（脚本硬判）：**
|
||||
|
||||
1. HTTP 200。
|
||||
2. `results` 为数组且 `length >= 5`。
|
||||
3. 前 10 条中，至少 5 条同时具备非空 `title`、非空 `url`（`http` 开头）、非空 `content`（strip 后 ≥ 8 字符）。
|
||||
4. `results[*].engine` 去重后 **≥ 2**（允许百度+必应，或搜狗+360）。若 `unresponsive_engines` 含某引擎，记入 `note`，不单独失败。
|
||||
5. 至少 1 条 `url` host 匹配 `gov.cn|customs.gov.cn|chinatax.gov.cn|mofcom.gov.cn|yidaiyilu.gov.cn|news.cn|people.com.cn|xinhuanet.com`（政策向）。不满足则 `assert_ok=false` 但把实际 host 列表留下，供人工改阈值。
|
||||
|
||||
**失败分类：** `json_403`（忘开 formats）/ `engine_outage` / `thin_snippets` / `timeout`。
|
||||
|
||||
**质量抽检（人工 2 分钟）：** 打开第 1 条 URL，标题是否与结果 `title` 同主题。
|
||||
|
||||
---
|
||||
|
||||
### T2 国内页面正文提取 — 方案 `trafilatura-http`
|
||||
|
||||
**URL 选择顺序：**
|
||||
|
||||
1. 优先用 T1 结果里第一条 host ∈ `gov.cn|news.cn|people.com.cn` 且 GET 200 的文章 URL。
|
||||
2. 回退固定页（公开、长期稳定、有导航壳）：
|
||||
|
||||
```
|
||||
https://www.gov.cn/zhengce/2021-03/13/content_5592681.htm
|
||||
```
|
||||
|
||||
（《国民经济和社会发展第十四个五年规划和2035年远景目标纲要》。若 2026 年该 URL 变更，改用 T1 第一条 gov.cn。）
|
||||
|
||||
**请求：**
|
||||
|
||||
```bash
|
||||
# 约定包装 API（S3b 实现 app.py 时遵守）
|
||||
curl -sS -X POST 'http://127.0.0.1:18881/v1/read' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://www.gov.cn/zhengce/2021-03/13/content_5592681.htm","formats":["markdown"],"max_chars":20000}'
|
||||
```
|
||||
|
||||
包装器内部：`trafilatura.fetch_url` + `trafilatura.extract(..., output_format="markdown", include_comments=False, include_tables=True, favor_precision=True)`。
|
||||
|
||||
**断言：**
|
||||
|
||||
1. HTTP 200，`ok=true`，`markdown` 长度 ≥ 800。
|
||||
2. `title` 非空。
|
||||
3. **无导航/广告残渣（规则）：** `markdown` 不得匹配下列任一条（忽略大小写）：
|
||||
- `网站地图|设为首页|加入收藏|政务微信|客户端下载|无障碍浏览`
|
||||
- 连续导航口吻：`首页\s*[>|/]\s*政策` 且出现 ≥ 3 次面包屑分隔
|
||||
4. **正文在：** 须同时命中 `十四五` 与 `2035`（回退 URL）；若改用 T1 URL，改为命中该条 T1 `title` 的前 4 个汉字。
|
||||
5. 不得把整页当 HTML 倾倒：`markdown` 中 `<div` / `<script` 出现次数 = 0。
|
||||
6. 若 `truncated=true`，`char_count` 必须 = `max_chars`。
|
||||
|
||||
**失败分类：** `fetch_fail` / `empty_extract` / `nav_residue` / `html_dump`。
|
||||
|
||||
---
|
||||
|
||||
### T3 国外关键词搜索 — 方案 `searxng-global`（经代理）
|
||||
|
||||
**前置：** `bench-proxy-mihomo` 已 up；SearXNG 容器 `HTTP_PROXY=http://host.docker.internal:17890`（或 settings `outgoing.proxies`）。
|
||||
|
||||
**请求：**
|
||||
|
||||
```bash
|
||||
curl -sS -G 'http://127.0.0.1:18882/search' \
|
||||
--data-urlencode 'q=best bluetooth earbuds 2026 reddit' \
|
||||
--data-urlencode 'format=json' \
|
||||
--data-urlencode 'language=en-US' \
|
||||
--data-urlencode 'categories=general'
|
||||
```
|
||||
|
||||
**断言：**
|
||||
|
||||
1. HTTP 200，`results.length >= 5`。
|
||||
2. `results[*].engine` 去重 ≥ 2（多源聚合）。
|
||||
3. `results[*].url` host 去重 ≥ 3。
|
||||
4. 至少 1 条 URL host 为 `reddit.com` 或 `redd.it`，**或** `title|content` 含 `reddit`（小写）。
|
||||
说明：S3a 对 `https://www.reddit.com/` 裸 GET 为 **403**，本模板走元搜索，不直接打 reddit 首页。
|
||||
5. 记录 `unresponsive_engines`。Google 单独 403/验证码 → `note=google_captcha`，只要其余引擎撑住断言 1–4 仍算通过。
|
||||
|
||||
**失败分类：** `proxy_down`（先 `curl -x http://127.0.0.1:17890 https://www.google.com/generate_204` 期望 204）/ `single_engine` / `no_reddit_ref`。
|
||||
|
||||
---
|
||||
|
||||
### T4 国外 JS 渲染页提取 — `lightpanda` 主测，`chrome-headless-shell` 对照
|
||||
|
||||
**页面（按序，记录每一档）：**
|
||||
|
||||
| 档 | URL | 目的 |
|
||||
|---|---|---|
|
||||
| T4-ctrl | `https://quotes.toscrape.com/js/` | 无 WAF 的 JS 对照：至少 3 条 quote 文本 |
|
||||
| T4-shop | `https://www.allbirds.com/products/mens-wool-runners` | Shopify 公开商品：标题 + 价格字段 |
|
||||
| T4-amz | `https://www.amazon.com/dp/B08N5WRWNW` | Amazon 公开商品页（Echo Dot 常见 ASIN）。**预期反爬，必须记。** |
|
||||
|
||||
不要用 Medium 登录墙当主断言。若 Allbirds 404，改 `https://www.allbirds.com/` 首页抽首个商品价。
|
||||
|
||||
**驱动约定（两方案同一脚本，只换 CDP 端点）：**
|
||||
|
||||
- Lightpanda：`http://127.0.0.1:19222`（CDP）。
|
||||
- shell：`http://127.0.0.1:19223`。
|
||||
- 实现可用 **Playwright** `connectOverCDP` 或 **chromedp**；不要再在客户端拉起本地 Chrome。
|
||||
- 导航超时 25s；`waitUntil=networkidle` 最多 8s，超时则退 `domcontentloaded` + 固定 sleep 2s。
|
||||
- 代理：容器启动已带 `--proxy-server` / `HTTP_PROXY`，客户端不要套两层。
|
||||
|
||||
**抽取：**
|
||||
|
||||
```
|
||||
title = document.title
|
||||
text = document.body.innerText.slice(0, 8000)
|
||||
# T4-shop / T4-amz:
|
||||
price = meta[itemprop=price] || [class*="price"] 首个匹配的 innerText
|
||||
jsonld = script[type="application/ld+json"] 里 Offers.price（有则优先）
|
||||
```
|
||||
|
||||
**断言：**
|
||||
|
||||
| 档 | 通过条件 |
|
||||
|---|---|
|
||||
| T4-ctrl | `text` 含 `quote` 或作者名；或 ≥3 个 `.quote` 节点。失败 = 内核 JS 基本不可用。 |
|
||||
| T4-shop | `title` 含 `Wool Runner` 或 `Allbirds`；`price` 匹配 `\$[0-9]` 或 `jsonld` 有数字价。 |
|
||||
| T4-amz | 能解析 `title` 且不像验证码页；`price` 可选。若 title/text 匹配 `(?i)robot|captcha|sorry|click the button|continue shopping` → `blocked=true`，**不记方案失败**，记 `waf`。 |
|
||||
|
||||
**反爬记录（强制字段）：** `http_status_in_page`, `blocked`, `challenge_vendor`（`cloudflare|datadome|akamai|amazon|unknown|none`）, `final_url`。
|
||||
|
||||
评分：T4-ctrl 失败重创质量分；T4-amz 拦截只进 `note`，不把成功率打到 0。
|
||||
|
||||
---
|
||||
|
||||
### T5 60 会话 burst
|
||||
|
||||
**语义：** 60 = **同时涌入的请求数**（队列深度），不是 60 个 Chromium。浏览器方案 `CONCURRENT=1`（shell）或 Lightpanda 单进程多客户端；超槽位必须排队或 429，禁止把 60 个 Chrome 拉起来。
|
||||
|
||||
| 方案 | 涌入内容 | 并发模型 |
|
||||
|---|---|---|
|
||||
| searxng-cn | 60× T1 同 query | 直接打 SearXNG，无自研队列 |
|
||||
| trafilatura-http | 60× T2 同 URL | 包装器内 semaphore=8（避免 60 条 TCP 打爆目标站） |
|
||||
| searxng-global | 60× T3 同 query | 同 CN |
|
||||
| lightpanda | 60× T4-ctrl | 客户端最多 4 个并行 CDP 导航，其余等（模拟网关 QUEUED） |
|
||||
| chrome-headless-shell | 60× T4-ctrl | **并行度 1**，其余 FIFO；超时 30s/个则记 fail 继续 |
|
||||
|
||||
**发射：**
|
||||
|
||||
```bash
|
||||
# 60 个后台 curl 同一秒涌入（搜索类）
|
||||
seq 60 | xargs -P 60 -I{} bash bench/lib/curl_time.sh
|
||||
```
|
||||
|
||||
Trafilatura / 浏览器不要用 `-P 60` 打目标站；用本机队列脚本：60 个任务入队，worker 池 = 上表并行度。
|
||||
|
||||
**记录：**
|
||||
|
||||
- 排队：首个完成 ttfb、最后完成 `t_total`、是否出现 429/503。
|
||||
- 成功率：断言通过 / 60。
|
||||
- p50 / p95：60 个 `t_total`。
|
||||
- `burst_peak_mb`、是否 OOM（`docker inspect .State.OOMKilled`）。
|
||||
- 浏览器方案若未实现网关，用脚本侧 FIFO 代替，`results.json.queue_impl=client_fifo`，并在 note 写「生产应是 SQLite WAL + CONCURRENT」。
|
||||
|
||||
**断言（方案级）：**
|
||||
|
||||
1. 容器未退出、`OOMKilled=false`。
|
||||
2. 搜索类成功率 ≥ 85%；读取/JS ≥ 70%（T4-ctrl）。
|
||||
3. 峰值内存：搜索/trafilatura < 400 MB；lightpanda < 256 MB cgroup；shell < 400 MB cgroup。超限记 `mem_over`，综合评分按评分卡 0 档。
|
||||
|
||||
---
|
||||
|
||||
## 3. 批次安排（本机 ≤3 候选容器）
|
||||
|
||||
每方案：**up → idle 5 min → 单模板 5 次（取中位）→ 该方案 T5 → down**。不要跨方案复用未清的浏览器进程。
|
||||
|
||||
| 批次 | 候选容器 | 基建 | 做的模板 | 预计墙钟 |
|
||||
|---|---|---|---|---|
|
||||
| **A 国内** | `searxng-cn` + `trafilatura-http` | 无 | T1（cn）→ T2（用 T1 URL）→ T5 cn → T5 trafilatura | 40–50 min |
|
||||
| **B 国外搜索+轻渲染** | `searxng-global` + `lightpanda` | `bench-proxy-mihomo` | T3 → T4 三档（panda）→ T5 global → T5 panda | 50–70 min |
|
||||
| **C 保真对照** | `chrome-headless-shell` + `lightpanda`（已有数据可不再 idle 5 min，若刚 down 过则重做 idle） | 同一代理 | T4 三档（shell）→ T5 shell | 40–60 min |
|
||||
| **D（可选）合成冒烟** | `searxng-cn` + `searxng-global` + `trafilatura-http` | 代理 | 各 1 次 T1/T2/T3，只看三容器 RSS 之和 | 15 min |
|
||||
|
||||
批次 A 与 B 不可与 C 重叠。D 不得加入 shell。
|
||||
|
||||
**代理：** 批次 B 开始时 `up.sh`，C 结束或当天收工时 `down.sh`。切换节点（可选）：
|
||||
|
||||
```bash
|
||||
curl -sS -X PUT "http://127.0.0.1:19090/proxies/%F0%9F%9A%80%E8%8A%82%E7%82%B9%E9%80%89%E6%8B%A9" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"🇺🇸美国圣何塞02 | 三网推荐"}'
|
||||
```
|
||||
|
||||
节点名以 `bench/proxy/probe-results.json` 为准。
|
||||
|
||||
---
|
||||
|
||||
## 4. 各方案 compose / 请求要点
|
||||
|
||||
### 4.1 `bench/searxng-cn/compose.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
searxng:
|
||||
image: docker.io/searxng/searxng:2026.8.29-d226b78bc
|
||||
container_name: bench-searxng-cn
|
||||
ports: ["18880:8080"]
|
||||
volumes:
|
||||
- ./settings.yml:/etc/searxng/settings.yml:ro
|
||||
environment:
|
||||
- SEARXNG_BASE_URL=http://localhost:18880/
|
||||
mem_limit: 192m
|
||||
restart: "no"
|
||||
```
|
||||
|
||||
`request.sh t1|t5`。settings 见 `select.md` §1.1。
|
||||
|
||||
### 4.2 `bench/trafilatura-http/`
|
||||
|
||||
`Dockerfile`：
|
||||
|
||||
```
|
||||
FROM python:3.12-slim-bookworm
|
||||
RUN pip install --no-cache-dir trafilatura==2.2.0
|
||||
WORKDIR /app
|
||||
COPY app.py .
|
||||
EXPOSE 8080
|
||||
CMD ["python", "-u", "app.py"]
|
||||
```
|
||||
|
||||
`app.py`：stdlib `http.server` 即可，避免再拉 Flask。只实现 `POST /v1/read`。`mem_limit: 128m`。
|
||||
|
||||
国外复用时加：
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
HTTP_PROXY: http://host.docker.internal:17890
|
||||
HTTPS_PROXY: http://host.docker.internal:17890
|
||||
NO_PROXY: 127.0.0.1,localhost
|
||||
```
|
||||
|
||||
本轮 T2 **不要**开代理。
|
||||
|
||||
### 4.3 `bench/searxng-global/compose.yml`
|
||||
|
||||
同 4.1，端口 18882，容器名 `bench-searxng-global`，`extra_hosts: ["host.docker.internal:host-gateway"]`，环境或 settings 走 17890。`mem_limit: 192m`。
|
||||
|
||||
### 4.4 `bench/lightpanda/compose.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
lightpanda:
|
||||
image: docker.io/lightpanda/browser:0.3.7
|
||||
container_name: bench-lightpanda
|
||||
command: ["lightpanda", "serve", "--host", "0.0.0.0", "--port", "9222"]
|
||||
ports: ["19222:9222"]
|
||||
extra_hosts: ["host.docker.internal:host-gateway"]
|
||||
environment:
|
||||
HTTP_PROXY: http://host.docker.internal:17890
|
||||
HTTPS_PROXY: http://host.docker.internal:17890
|
||||
mem_limit: 256m
|
||||
```
|
||||
|
||||
`request.sh`：Node 或 Python Playwright `connectOverCDP('http://127.0.0.1:19222')`。
|
||||
|
||||
### 4.5 `bench/chrome-headless-shell/compose.yml`
|
||||
|
||||
```yaml
|
||||
services:
|
||||
shell:
|
||||
image: docker.io/chromedp/headless-shell:151.0.7922.109
|
||||
container_name: bench-headless-shell
|
||||
ports: ["19223:9222"]
|
||||
extra_hosts: ["host.docker.internal:host-gateway"]
|
||||
command:
|
||||
- --no-sandbox
|
||||
- --remote-debugging-address=0.0.0.0
|
||||
- --remote-debugging-port=9222
|
||||
- --proxy-server=http://host.docker.internal:17890
|
||||
shm_size: "128m"
|
||||
mem_limit: 400m
|
||||
```
|
||||
|
||||
镜像 ENTRYPOINT 已是 headless-shell 时，`command` 只追加 flag。S3b 若启动即退，把 `docker logs` 贴进 `results.json.note`（**未验证** Desktop 对 151.x arm64）。
|
||||
|
||||
---
|
||||
|
||||
## 5. `results.json` schema
|
||||
|
||||
每个方案一份：
|
||||
|
||||
```json
|
||||
{
|
||||
"scheme": "searxng-cn",
|
||||
"group": "domestic",
|
||||
"image": {"ref": "docker.io/searxng/searxng:2026.8.29-d226b78bc", "digest": "sha256:…"},
|
||||
"probed_at": "2026-09-01T00:00:00+08:00",
|
||||
"rss_source": "docker_stats",
|
||||
"idle_mb_p50": null,
|
||||
"session_delta_mb": null,
|
||||
"burst_peak_mb": null,
|
||||
"templates": {
|
||||
"T1": {
|
||||
"n": 5,
|
||||
"success": 0,
|
||||
"http_codes": [],
|
||||
"p50_s": null,
|
||||
"p95_s": null,
|
||||
"assert_ok": false,
|
||||
"note": ""
|
||||
}
|
||||
},
|
||||
"t5": {
|
||||
"n": 60,
|
||||
"parallelism": 60,
|
||||
"queue_impl": "none",
|
||||
"success": 0,
|
||||
"p50_s": null,
|
||||
"p95_s": null,
|
||||
"oom": false,
|
||||
"http_429": 0,
|
||||
"http_503": 0
|
||||
},
|
||||
"t4_waf": [],
|
||||
"scorecard": {
|
||||
"mem": null, "quality": null, "stability": null, "latency": null, "ops": null, "total": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
T4 的 `t4_waf` 元素：`{tier, url, blocked, vendor, http_status_in_page}`。
|
||||
|
||||
填完后把数字抄回 `select.md` §4.7。本文件不改 select 以外的架构文档。
|
||||
|
||||
---
|
||||
|
||||
## 6. S3b / S3c 执行顺序（可当 checklist）
|
||||
|
||||
**S3b（国内，可无代理）**
|
||||
|
||||
1. `docker pull` 或 `load`：`searxng/searxng:2026.8.29-d226b78bc`、`python:3.12-slim-bookworm`。
|
||||
2. 批次 A：建 `searxng-cn` + `trafilatura-http` 目录与 compose。
|
||||
3. 只起 `bench-searxng-cn` → idle 5 min → T1 ×5 → T5。
|
||||
4. 保持 cn 或 down 后起 `bench-trafilatura`（同时 ≤2）→ idle 5 min → T2 ×5（URL 来自 T1）→ T5。
|
||||
5. 写两份 `results.json`。两容器同时存在时再采 30s，看 RSS 之和（P0 冒烟）。
|
||||
|
||||
**S3c（国外，必须代理）**
|
||||
|
||||
1. `bash bench/proxy/up.sh`；`curl -x http://127.0.0.1:17890 -o /dev/null -w '%{http_code}\n' https://www.google.com/generate_204` 期望 204。
|
||||
2. pull/load：`lightpanda/browser:0.3.7`、`chromedp/headless-shell:151.0.7922.109`。
|
||||
3. 批次 B：global idle → T3 ×5 → T5；panda idle → T4 三档各 3 次 → T5-ctrl。
|
||||
4. 批次 C：down panda（或留着对照但 **不要** 同时 T5）→ shell idle → T4 三档 → T5-ctrl。
|
||||
5. `down.sh`。写 `results.json` + 回填评分卡。
|
||||
|
||||
密钥扫描（收工）：`rg -n 'uuid:|password:|PROXY_SUB_URL=http' bench` 应无节点凭据。
|
||||
|
||||
---
|
||||
|
||||
## 7. 本机注意
|
||||
|
||||
- Apple Silicon 必须拉 **arm64** digest。`docker image inspect --format '{{.Architecture}}'` 若为 `amd64` 会走 qemu，内存与延迟作废，重拉 `--platform linux/arm64`。
|
||||
- `host.docker.internal` 在 Docker Desktop 可用；若不通，改 `172.17.0.1` 或 `host-gateway`，记入 note。
|
||||
- 目标站限流：T2/T4 单 URL 5 次 + T5 会打同一 host。Trafilatura T5 并行度 ≤8；Amazon T4 每方案最多 3 次，禁止 T5 打 Amazon。
|
||||
- 合规：只打公开页；不登录、不绕验证码、不采集个人数据。拦截页当 `blocked`，不当成功正文。
|
||||
12
bench/proxy/down.sh
Executable file
12
bench/proxy/down.sh
Executable file
|
|
@ -0,0 +1,12 @@
|
|||
#!/usr/bin/env bash
|
||||
# Tear down all bench-proxy-* containers.
|
||||
set -euo pipefail
|
||||
ids="$(docker ps -aq --filter 'name=^bench-proxy-' || true)"
|
||||
if [[ -z "$ids" ]]; then
|
||||
echo "[down] no bench-proxy-* containers"
|
||||
exit 0
|
||||
fi
|
||||
# shellcheck disable=SC2086
|
||||
docker rm -f $ids
|
||||
echo "[down] removed: $ids"
|
||||
rm -rf /tmp/bench-proxy-runtime
|
||||
148
bench/proxy/lib.py
Executable file
148
bench/proxy/lib.py
Executable file
|
|
@ -0,0 +1,148 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Sanitized Clash-sub helpers. Never print or persist credentials."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import urllib.parse
|
||||
from typing import Any
|
||||
|
||||
INFO_NAME_RE = re.compile(r"^(剩余流量|套餐到期|到期|流量)")
|
||||
|
||||
REGION_RULES: list[tuple[str, list[str]]] = [
|
||||
("香港", ["香港", "HK", "🇭🇰"]),
|
||||
("台湾", ["台湾", "臺灣", "🇹🇼"]),
|
||||
("日本", ["日本", "东京", "大阪", "🇯🇵"]),
|
||||
("韩国", ["韩国", "韓國", "首尔", "🇰🇷"]),
|
||||
("新加坡", ["新加坡", "狮城", "🇸🇬"]),
|
||||
("美国", ["美国", "美國", "圣何塞", "洛杉矶", "硅谷", "西雅图", "芝加哥", "纽约", "🇺🇸"]),
|
||||
("英国", ["英国", "英國", "伦敦", "🇬🇧"]),
|
||||
("德国", ["德国", "德國", "法兰克福", "🇩🇪"]),
|
||||
("法国", ["法国", "法國", "巴黎", "🇫🇷"]),
|
||||
("加拿大", ["加拿大", "🇨🇦"]),
|
||||
("澳大利亚", ["澳大利亚", "澳洲", "🇦🇺"]),
|
||||
("澳门", ["澳门", "澳門", "🇲🇴"]),
|
||||
("中国", ["中国", "回国"]),
|
||||
]
|
||||
|
||||
|
||||
def infer_region(name: str) -> str:
|
||||
for label, keys in REGION_RULES:
|
||||
for key in keys:
|
||||
if key in name:
|
||||
return label
|
||||
return "其他/未知"
|
||||
|
||||
|
||||
def is_info_node(name: str) -> bool:
|
||||
return bool(INFO_NAME_RE.search(name))
|
||||
|
||||
|
||||
def parse_flow_proxies(text: str) -> list[dict[str, str]]:
|
||||
proxies: list[dict[str, str]] = []
|
||||
for line in text.splitlines():
|
||||
if "name:" not in line or "type:" not in line or "server:" not in line:
|
||||
continue
|
||||
name_m = re.search(r"name:\s*'([^']+)'|name:\s*([^,}]+)", line)
|
||||
type_m = re.search(r"type:\s*([a-zA-Z0-9-]+)", line)
|
||||
if not name_m or not type_m:
|
||||
continue
|
||||
name = (name_m.group(1) or name_m.group(2)).strip()
|
||||
proxies.append(
|
||||
{
|
||||
"name": name,
|
||||
"type": type_m.group(1),
|
||||
"region": infer_region(name),
|
||||
}
|
||||
)
|
||||
return proxies
|
||||
|
||||
|
||||
def detect_format(text: str) -> str:
|
||||
head = text.lstrip()[:400]
|
||||
if "proxies:" in text[:4000] and (
|
||||
"mixed-port:" in head or "port:" in head or head.startswith("proxies:")
|
||||
):
|
||||
return "clash-yaml"
|
||||
if any(head.startswith(s) for s in ("ss://", "vmess://", "trojan://", "vless://")):
|
||||
return "uri-list"
|
||||
compact = re.sub(r"\s+", "", text)
|
||||
if compact and re.fullmatch(r"[A-Za-z0-9+/=_-]+", compact[:80] or ""):
|
||||
return "base64-maybe"
|
||||
return "other"
|
||||
|
||||
|
||||
def rewrite_runtime_config(src: str, mixed_port: int, controller: str) -> str:
|
||||
"""Rewrite listen ports / controller; strip original rules to MATCH selector."""
|
||||
lines = src.splitlines()
|
||||
out: list[str] = []
|
||||
skipping_rules = False
|
||||
selector = "🚀节点选择"
|
||||
for line in lines:
|
||||
if line.startswith("proxy-groups:"):
|
||||
skipping_rules = False
|
||||
if line.startswith("rules:"):
|
||||
skipping_rules = True
|
||||
continue
|
||||
if skipping_rules:
|
||||
continue
|
||||
if line.startswith("mixed-port:"):
|
||||
out.append(f"mixed-port: {mixed_port}")
|
||||
continue
|
||||
if line.startswith("allow-lan:"):
|
||||
out.append("allow-lan: true")
|
||||
continue
|
||||
if line.startswith("external-controller:"):
|
||||
out.append(f"external-controller: '{controller}'")
|
||||
continue
|
||||
if line.startswith("bind-address:"):
|
||||
out.append("bind-address: '*'")
|
||||
continue
|
||||
out.append(line)
|
||||
out.append("rules:")
|
||||
out.append(f" - MATCH,{selector}")
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
def mask_secret(value: str, keep: int = 4) -> str:
|
||||
if not value:
|
||||
return ""
|
||||
return value[:keep] + "***"
|
||||
|
||||
|
||||
def summarize_subscription(text: str) -> dict[str, Any]:
|
||||
fmt = detect_format(text)
|
||||
proxies = parse_flow_proxies(text)
|
||||
real = [p for p in proxies if not is_info_node(p["name"])]
|
||||
info = [p for p in proxies if is_info_node(p["name"])]
|
||||
return {
|
||||
"format": fmt,
|
||||
"proxy_count_raw": len(proxies),
|
||||
"proxy_count_real": len(real),
|
||||
"info_nodes": [p["name"] for p in info],
|
||||
"types": dict(collections.Counter(p["type"] for p in real)),
|
||||
"regions": dict(collections.Counter(p["region"] for p in real)),
|
||||
"proxies": real,
|
||||
}
|
||||
|
||||
|
||||
def api_quote(name: str) -> str:
|
||||
return urllib.parse.quote(name, safe="")
|
||||
|
||||
|
||||
def require_sub_url() -> str:
|
||||
url = os.environ.get("PROXY_SUB_URL", "").strip()
|
||||
if not url:
|
||||
raise SystemExit("PROXY_SUB_URL is required (do not hardcode the URL)")
|
||||
if not url.startswith(("http://", "https://")):
|
||||
raise SystemExit("PROXY_SUB_URL must be an http(s) URL")
|
||||
return url
|
||||
|
||||
|
||||
def dump_json(path: str, obj: Any) -> None:
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
json.dump(obj, fh, ensure_ascii=False, indent=2)
|
||||
fh.write("\n")
|
||||
958
bench/proxy/probe-results.json
Normal file
958
bench/proxy/probe-results.json
Normal file
|
|
@ -0,0 +1,958 @@
|
|||
{
|
||||
"probed_at": "2026-09-01T11:16:35+08:00",
|
||||
"mihomo": {
|
||||
"meta": true,
|
||||
"version": "v1.19.24"
|
||||
},
|
||||
"api": "http://127.0.0.1:19090",
|
||||
"mixed_proxy": "http://127.0.0.1:17890",
|
||||
"delay_url": "https://www.google.com/generate_204",
|
||||
"timeout_ms": 8000,
|
||||
"leaf_count": 69,
|
||||
"tested_count": 69,
|
||||
"alive_count": 52,
|
||||
"alive_ge_3": true,
|
||||
"delay": [
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞02 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 167,
|
||||
"error": null,
|
||||
"elapsed_s": 0.708,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞03 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 168,
|
||||
"error": null,
|
||||
"elapsed_s": 0.748,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞04 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 168,
|
||||
"error": null,
|
||||
"elapsed_s": 0.777,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞03-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 172,
|
||||
"error": null,
|
||||
"elapsed_s": 1.161,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国洛杉矶08 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 174,
|
||||
"error": null,
|
||||
"elapsed_s": 0.748,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国04 | 合适下载使用-0.01倍",
|
||||
"ok": true,
|
||||
"delay_ms": 179,
|
||||
"error": null,
|
||||
"elapsed_s": 0.955,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞02-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 179,
|
||||
"error": null,
|
||||
"elapsed_s": 1.227,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞04-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 179,
|
||||
"error": null,
|
||||
"elapsed_s": 1.158,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国05 | 合适下载使用-0.01倍",
|
||||
"ok": true,
|
||||
"delay_ms": 180,
|
||||
"error": null,
|
||||
"elapsed_s": 1.132,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国01 | 合适下载使用-0.01倍",
|
||||
"ok": true,
|
||||
"delay_ms": 186,
|
||||
"error": null,
|
||||
"elapsed_s": 0.998,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国02 | 合适下载使用-0.01倍",
|
||||
"ok": true,
|
||||
"delay_ms": 186,
|
||||
"error": null,
|
||||
"elapsed_s": 0.975,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞05-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 190,
|
||||
"error": null,
|
||||
"elapsed_s": 1.034,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞01 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 192,
|
||||
"error": null,
|
||||
"elapsed_s": 0.744,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞07 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 194,
|
||||
"error": null,
|
||||
"elapsed_s": 0.926,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞05 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 195,
|
||||
"error": null,
|
||||
"elapsed_s": 1.055,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国洛杉矶08-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 195,
|
||||
"error": null,
|
||||
"elapsed_s": 1.036,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞07-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 198,
|
||||
"error": null,
|
||||
"elapsed_s": 1.016,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞06 | 三网推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 199,
|
||||
"error": null,
|
||||
"elapsed_s": 0.971,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞06-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 199,
|
||||
"error": null,
|
||||
"elapsed_s": 1.019,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京06-0.1倍 | 高速专线推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 280,
|
||||
"error": null,
|
||||
"elapsed_s": 1.704,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京07-0.1倍 | 高速专线推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 280,
|
||||
"error": null,
|
||||
"elapsed_s": 1.443,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本10 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 289,
|
||||
"error": null,
|
||||
"elapsed_s": 1.419,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京08-0.1倍 | 高速专线推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 289,
|
||||
"error": null,
|
||||
"elapsed_s": 1.571,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本08 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 290,
|
||||
"error": null,
|
||||
"elapsed_s": 1.343,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京09-0.1倍 | 高速专线推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 290,
|
||||
"error": null,
|
||||
"elapsed_s": 1.547,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本05 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 292,
|
||||
"error": null,
|
||||
"elapsed_s": 1.797,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本09 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 293,
|
||||
"error": null,
|
||||
"elapsed_s": 1.423,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本07 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 301,
|
||||
"error": null,
|
||||
"elapsed_s": 1.374,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本06 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 303,
|
||||
"error": null,
|
||||
"elapsed_s": 1.4,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本04 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 305,
|
||||
"error": null,
|
||||
"elapsed_s": 1.646,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京02-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 313,
|
||||
"error": null,
|
||||
"elapsed_s": 1.679,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京05-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 316,
|
||||
"error": null,
|
||||
"elapsed_s": 1.912,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京01-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 317,
|
||||
"error": null,
|
||||
"elapsed_s": 1.853,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京03-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 317,
|
||||
"error": null,
|
||||
"elapsed_s": 1.669,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵日本东京04-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 323,
|
||||
"error": null,
|
||||
"elapsed_s": 1.732,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬AWS新加坡01 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 354,
|
||||
"error": null,
|
||||
"elapsed_s": 1.624,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡01-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 359,
|
||||
"error": null,
|
||||
"elapsed_s": 1.855,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡02 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 360,
|
||||
"error": null,
|
||||
"elapsed_s": 1.823,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬AWS新加坡05 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 361,
|
||||
"error": null,
|
||||
"elapsed_s": 1.632,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬AWS新加坡04 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 362,
|
||||
"error": null,
|
||||
"elapsed_s": 1.643,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡02-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 362,
|
||||
"error": null,
|
||||
"elapsed_s": 1.878,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡03-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 362,
|
||||
"error": null,
|
||||
"elapsed_s": 1.879,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡01 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 369,
|
||||
"error": null,
|
||||
"elapsed_s": 1.922,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本01 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 370,
|
||||
"error": null,
|
||||
"elapsed_s": 2.711,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬AWS新加坡02 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 371,
|
||||
"error": null,
|
||||
"elapsed_s": 1.73,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬AWS新加坡03 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 371,
|
||||
"error": null,
|
||||
"elapsed_s": 1.898,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国03 | 合适下载使用-0.01倍",
|
||||
"ok": true,
|
||||
"delay_ms": 383,
|
||||
"error": null,
|
||||
"elapsed_s": 1.168,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡05-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 396,
|
||||
"error": null,
|
||||
"elapsed_s": 1.877,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞01-0.1倍",
|
||||
"ok": true,
|
||||
"delay_ms": 543,
|
||||
"error": null,
|
||||
"elapsed_s": 1.37,
|
||||
"type": "vless",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡04-0.1倍 | 电信联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 579,
|
||||
"error": null,
|
||||
"elapsed_s": 2.248,
|
||||
"type": "vless",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本03 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 620,
|
||||
"error": null,
|
||||
"elapsed_s": 2.008,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇯🇵AWS日本02 | 电信移动联通推荐",
|
||||
"ok": true,
|
||||
"delay_ms": 718,
|
||||
"error": null,
|
||||
"elapsed_s": 2.068,
|
||||
"type": "vless",
|
||||
"region": "日本"
|
||||
},
|
||||
{
|
||||
"name": "🇹🇼台湾 | 避免晚高峰使用",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.014,
|
||||
"type": "hysteria2",
|
||||
"region": "台湾"
|
||||
},
|
||||
{
|
||||
"name": "🇹🇼台湾2 | 避免晚高峰使用",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.014,
|
||||
"type": "hysteria2",
|
||||
"region": "台湾"
|
||||
},
|
||||
{
|
||||
"name": "🇹🇼台湾3 | 避免晚高峰使用",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.015,
|
||||
"type": "hysteria2",
|
||||
"region": "台湾"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡 | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 504 {\"message\":\"Timeout\"}\n",
|
||||
"elapsed_s": 8.031,
|
||||
"type": "hysteria2",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡2 | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 504 {\"message\":\"Timeout\"}\n",
|
||||
"elapsed_s": 8.008,
|
||||
"type": "hysteria2",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡3 | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 504 {\"message\":\"Timeout\"}\n",
|
||||
"elapsed_s": 8.008,
|
||||
"type": "hysteria2",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡4 | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 504 {\"message\":\"Timeout\"}\n",
|
||||
"elapsed_s": 8.007,
|
||||
"type": "hysteria2",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇸🇬新加坡5 | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 504 {\"message\":\"Timeout\"}\n",
|
||||
"elapsed_s": 8.262,
|
||||
"type": "hysteria2",
|
||||
"region": "新加坡"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国01-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.006,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国02-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.006,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国03-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.009,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国04-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.007,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国05-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.004,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国06-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.005,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国07-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.007,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国08-0.1倍 | 电信联通移动推荐",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.006,
|
||||
"type": "hysteria2",
|
||||
"region": "美国"
|
||||
},
|
||||
{
|
||||
"name": "🇭🇰香港HKT | 高速专线-hy2",
|
||||
"ok": false,
|
||||
"delay_ms": null,
|
||||
"error": "HTTP 503 {\"message\":\"An error occurred in the delay test\"}\n",
|
||||
"elapsed_s": 5.004,
|
||||
"type": "hysteria2",
|
||||
"region": "香港"
|
||||
}
|
||||
],
|
||||
"top3": [
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞02 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 167
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞03 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 168
|
||||
},
|
||||
{
|
||||
"name": "🇺🇸美国圣何塞04 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 168
|
||||
}
|
||||
],
|
||||
"matrix": [
|
||||
{
|
||||
"node": "🇺🇸美国圣何塞02 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 167,
|
||||
"sites": [
|
||||
{
|
||||
"url": "https://www.google.com/generate_204",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "204",
|
||||
"ttfb_s": 0.553046,
|
||||
"total_s": 0.553134,
|
||||
"error": null,
|
||||
"elapsed_s": 0.578,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.571639,
|
||||
"total_s": 0.757081,
|
||||
"error": null,
|
||||
"elapsed_s": 0.771,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.bing.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.689005,
|
||||
"total_s": 0.767551,
|
||||
"error": null,
|
||||
"elapsed_s": 0.778,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.reddit.com/",
|
||||
"method": "GET",
|
||||
"ok": false,
|
||||
"http_code": "403",
|
||||
"ttfb_s": 0.548932,
|
||||
"total_s": 0.575258,
|
||||
"error": null,
|
||||
"elapsed_s": 0.589,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.amazon.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.681933,
|
||||
"total_s": 1.028478,
|
||||
"error": null,
|
||||
"elapsed_s": 1.039,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://r.jina.ai/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.791257,
|
||||
"total_s": 0.791369,
|
||||
"error": null,
|
||||
"elapsed_s": 0.813,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://duckduckgo.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.689025,
|
||||
"total_s": 0.859293,
|
||||
"error": null,
|
||||
"elapsed_s": 0.868,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"node": "🇺🇸美国圣何塞03 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 168,
|
||||
"sites": [
|
||||
{
|
||||
"url": "https://www.google.com/generate_204",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "204",
|
||||
"ttfb_s": 0.701933,
|
||||
"total_s": 0.701979,
|
||||
"error": null,
|
||||
"elapsed_s": 0.717,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 1.511273,
|
||||
"total_s": 1.80708,
|
||||
"error": null,
|
||||
"elapsed_s": 1.83,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.bing.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.673573,
|
||||
"total_s": 0.915727,
|
||||
"error": null,
|
||||
"elapsed_s": 0.984,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.reddit.com/",
|
||||
"method": "GET",
|
||||
"ok": false,
|
||||
"http_code": "403",
|
||||
"ttfb_s": 0.750979,
|
||||
"total_s": 0.870084,
|
||||
"error": null,
|
||||
"elapsed_s": 0.891,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.amazon.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 1.011413,
|
||||
"total_s": 1.652145,
|
||||
"error": null,
|
||||
"elapsed_s": 1.674,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://r.jina.ai/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.606865,
|
||||
"total_s": 0.607068,
|
||||
"error": null,
|
||||
"elapsed_s": 0.624,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://duckduckgo.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 1.276337,
|
||||
"total_s": 2.271431,
|
||||
"error": null,
|
||||
"elapsed_s": 2.284,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"node": "🇺🇸美国圣何塞04 | 三网推荐",
|
||||
"type": "vless",
|
||||
"region": "美国",
|
||||
"delay_ms": 168,
|
||||
"sites": [
|
||||
{
|
||||
"url": "https://www.google.com/generate_204",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "204",
|
||||
"ttfb_s": 0.535985,
|
||||
"total_s": 0.536027,
|
||||
"error": null,
|
||||
"elapsed_s": 0.552,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 1.653779,
|
||||
"total_s": 1.663649,
|
||||
"error": null,
|
||||
"elapsed_s": 1.676,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.bing.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.686513,
|
||||
"total_s": 0.771807,
|
||||
"error": null,
|
||||
"elapsed_s": 0.787,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.reddit.com/",
|
||||
"method": "GET",
|
||||
"ok": false,
|
||||
"http_code": "403",
|
||||
"ttfb_s": 0.545687,
|
||||
"total_s": 0.57248,
|
||||
"error": null,
|
||||
"elapsed_s": 0.581,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://www.amazon.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.709847,
|
||||
"total_s": 1.082345,
|
||||
"error": null,
|
||||
"elapsed_s": 1.094,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://r.jina.ai/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.809628,
|
||||
"total_s": 0.80979,
|
||||
"error": null,
|
||||
"elapsed_s": 0.82,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
},
|
||||
{
|
||||
"url": "https://duckduckgo.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "200",
|
||||
"ttfb_s": 0.708627,
|
||||
"total_s": 0.864429,
|
||||
"error": null,
|
||||
"elapsed_s": 0.886,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"commands": {
|
||||
"up": "PROXY_SUB_URL=... bash bench/proxy/up.sh",
|
||||
"probe": "python3 bench/proxy/probe.py --all",
|
||||
"down": "bash bench/proxy/down.sh"
|
||||
},
|
||||
"matrix_extra_google_apex": [
|
||||
{
|
||||
"url": "https://google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "301",
|
||||
"ttfb_s": 0.765955,
|
||||
"total_s": 0.766187,
|
||||
"error": null,
|
||||
"elapsed_s": 0.786,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>",
|
||||
"node": "🇺🇸美国圣何塞02 | 三网推荐"
|
||||
},
|
||||
{
|
||||
"url": "https://google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "301",
|
||||
"ttfb_s": 0.573401,
|
||||
"total_s": 0.573646,
|
||||
"error": null,
|
||||
"elapsed_s": 0.592,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>",
|
||||
"node": "🇺🇸美国圣何塞03 | 三网推荐"
|
||||
},
|
||||
{
|
||||
"url": "https://google.com/",
|
||||
"method": "GET",
|
||||
"ok": true,
|
||||
"http_code": "301",
|
||||
"ttfb_s": 0.572572,
|
||||
"total_s": 0.572753,
|
||||
"error": null,
|
||||
"elapsed_s": 0.589,
|
||||
"cmd": "curl -sS -o /dev/null -X GET -w %{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg} --connect-timeout 10 --max-time 20 -x http://127.0.0.1:17890 -A bench-proxy-probe/1.0 <url>",
|
||||
"node": "🇺🇸美国圣何塞04 | 三网推荐"
|
||||
}
|
||||
]
|
||||
}
|
||||
276
bench/proxy/probe.py
Executable file
276
bench/proxy/probe.py
Executable file
|
|
@ -0,0 +1,276 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Probe mihomo proxies. Credentials never written to output."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from lib import api_quote, infer_region, is_info_node
|
||||
|
||||
DEFAULT_DELAY_URL = "https://www.google.com/generate_204"
|
||||
TARGETS = [
|
||||
"https://www.google.com/generate_204",
|
||||
"https://www.google.com/",
|
||||
"https://www.bing.com/",
|
||||
"https://www.reddit.com/",
|
||||
"https://www.amazon.com/",
|
||||
"https://r.jina.ai/",
|
||||
"https://duckduckgo.com/",
|
||||
]
|
||||
SELECTOR = "🚀节点选择"
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return dt.datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def api(base: str, path: str, method: str = "GET", body: Any | None = None, timeout: float = 15) -> Any:
|
||||
data = None
|
||||
headers = {}
|
||||
if body is not None:
|
||||
data = json.dumps(body).encode("utf-8")
|
||||
headers["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(base + path, data=data, method=method, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read()
|
||||
if not raw:
|
||||
return None
|
||||
return json.loads(raw.decode("utf-8"))
|
||||
|
||||
|
||||
def list_leaf_proxies(base: str) -> list[dict[str, str]]:
|
||||
data = api(base, "/proxies")
|
||||
out: list[dict[str, str]] = []
|
||||
for name, meta in (data or {}).get("proxies", {}).items():
|
||||
ptype = meta.get("type") or ""
|
||||
if ptype in {"Selector", "URLTest", "Fallback", "LoadBalance", "Relay", "Compatible", "Direct", "Reject", "Pass", "RejectDrop"}:
|
||||
continue
|
||||
if name in {"DIRECT", "REJECT", "GLOBAL", "COMPATIBLE", "PASS"}:
|
||||
continue
|
||||
if is_info_node(name):
|
||||
continue
|
||||
out.append({"name": name, "type": ptype.lower(), "region": infer_region(name)})
|
||||
out.sort(key=lambda x: (x["region"], x["name"]))
|
||||
return out
|
||||
|
||||
|
||||
def delay_one(base: str, name: str, url: str, timeout_ms: int) -> dict[str, Any]:
|
||||
q = api_quote(name)
|
||||
path = f"/proxies/{q}/delay?timeout={timeout_ms}&url={urllib.parse.quote(url, safe='')}"
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
data = api(base, path, timeout=timeout_ms / 1000 + 3)
|
||||
delay = data.get("delay") if isinstance(data, dict) else None
|
||||
return {
|
||||
"name": name,
|
||||
"ok": isinstance(delay, int) and delay > 0,
|
||||
"delay_ms": delay,
|
||||
"error": None,
|
||||
"elapsed_s": round(time.perf_counter() - started, 3),
|
||||
}
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = exc.read().decode("utf-8", errors="replace")[:200]
|
||||
return {
|
||||
"name": name,
|
||||
"ok": False,
|
||||
"delay_ms": None,
|
||||
"error": f"HTTP {exc.code} {body}",
|
||||
"elapsed_s": round(time.perf_counter() - started, 3),
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {
|
||||
"name": name,
|
||||
"ok": False,
|
||||
"delay_ms": None,
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
"elapsed_s": round(time.perf_counter() - started, 3),
|
||||
}
|
||||
|
||||
|
||||
def switch_selector(base: str, node: str, group: str = SELECTOR) -> None:
|
||||
api(base, f"/proxies/{api_quote(group)}", method="PUT", body={"name": node})
|
||||
|
||||
|
||||
def curl_via_proxy(proxy: str, url: str, method: str = "GET", max_time: int = 20) -> dict[str, Any]:
|
||||
cmd = [
|
||||
"curl", "-sS", "-o", "/dev/null",
|
||||
"-X", method,
|
||||
"-w", "%{http_code} %{time_namelookup} %{time_connect} %{time_starttransfer} %{time_total} %{errormsg}",
|
||||
"--connect-timeout", "10",
|
||||
"--max-time", str(max_time),
|
||||
"-x", proxy,
|
||||
"-A", "bench-proxy-probe/1.0",
|
||||
url,
|
||||
]
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
parts = (proc.stdout or "").strip().split(" ", 5)
|
||||
http_code = parts[0] if parts else ""
|
||||
ttfb = float(parts[3]) if len(parts) > 3 and parts[3] else None
|
||||
total = float(parts[4]) if len(parts) > 4 and parts[4] else None
|
||||
err = (parts[5] if len(parts) > 5 else "") or (proc.stderr or "").strip()
|
||||
ok = http_code.isdigit() and http_code[0] in {"2", "3"}
|
||||
return {
|
||||
"url": url,
|
||||
"method": method,
|
||||
"ok": ok,
|
||||
"http_code": http_code,
|
||||
"ttfb_s": ttfb,
|
||||
"total_s": total,
|
||||
"error": err or None,
|
||||
"elapsed_s": round(time.perf_counter() - started, 3),
|
||||
"cmd": " ".join(cmd[:-1] + ["<url>"]),
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {
|
||||
"url": url,
|
||||
"method": method,
|
||||
"ok": False,
|
||||
"http_code": None,
|
||||
"ttfb_s": None,
|
||||
"total_s": None,
|
||||
"error": f"{type(exc).__name__}: {exc}",
|
||||
"elapsed_s": round(time.perf_counter() - started, 3),
|
||||
"cmd": "curl -x <proxy> <url>",
|
||||
}
|
||||
|
||||
|
||||
def sample_nodes(nodes: list[dict[str, str]], min_n: int = 10) -> list[dict[str, str]]:
|
||||
if len(nodes) <= min_n:
|
||||
return nodes
|
||||
by_region: dict[str, list[dict[str, str]]] = {}
|
||||
for n in nodes:
|
||||
by_region.setdefault(n["region"], []).append(n)
|
||||
picked: list[dict[str, str]] = []
|
||||
# Prefer at least one per region, then fill by type diversity.
|
||||
for region, items in by_region.items():
|
||||
hy2 = [x for x in items if x["type"] in {"hysteria2", "hysteria"}]
|
||||
rest = [x for x in items if x not in hy2]
|
||||
if hy2:
|
||||
picked.append(hy2[0])
|
||||
if rest:
|
||||
picked.append(rest[0])
|
||||
if len(rest) > 3:
|
||||
picked.append(rest[len(rest) // 2])
|
||||
# unique preserve
|
||||
seen = set()
|
||||
uniq = []
|
||||
for n in picked:
|
||||
if n["name"] in seen:
|
||||
continue
|
||||
seen.add(n["name"])
|
||||
uniq.append(n)
|
||||
i = 0
|
||||
while len(uniq) < min_n and i < len(nodes):
|
||||
if nodes[i]["name"] not in seen:
|
||||
uniq.append(nodes[i])
|
||||
seen.add(nodes[i]["name"])
|
||||
i += 1
|
||||
return uniq
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--api", default=os.environ.get("MIHOMO_API", "http://127.0.0.1:19090"))
|
||||
ap.add_argument("--proxy", default=os.environ.get("MIXED_PROXY", "http://127.0.0.1:17890"))
|
||||
ap.add_argument("--out", default=os.path.join(os.path.dirname(__file__), "probe-results.json"))
|
||||
ap.add_argument("--delay-url", default=DEFAULT_DELAY_URL)
|
||||
ap.add_argument("--timeout-ms", type=int, default=8000)
|
||||
ap.add_argument("--all", action="store_true", help="probe every real node (default: sample >=10)")
|
||||
ap.add_argument("--min-sample", type=int, default=12)
|
||||
ap.add_argument("--workers", type=int, default=6)
|
||||
args = ap.parse_args()
|
||||
|
||||
version = api(args.api, "/version")
|
||||
leaves = list_leaf_proxies(args.api)
|
||||
targets = leaves if args.all else sample_nodes(leaves, args.min_sample)
|
||||
|
||||
print(f"[probe] mihomo={version} leaves={len(leaves)} testing={len(targets)}", flush=True)
|
||||
delay_rows: list[dict[str, Any]] = []
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
|
||||
futs = [
|
||||
pool.submit(delay_one, args.api, n["name"], args.delay_url, args.timeout_ms)
|
||||
for n in targets
|
||||
]
|
||||
for fut in concurrent.futures.as_completed(futs):
|
||||
row = fut.result()
|
||||
meta = next(x for x in targets if x["name"] == row["name"])
|
||||
row.update({"type": meta["type"], "region": meta["region"]})
|
||||
delay_rows.append(row)
|
||||
status = f"{row['delay_ms']}ms" if row["ok"] else row["error"]
|
||||
print(f" delay {row['region']:6} {row['type']:10} {row['name']} -> {status}", flush=True)
|
||||
|
||||
delay_rows.sort(key=lambda r: (not r["ok"], r["delay_ms"] is None, r["delay_ms"] or 10**9))
|
||||
alive = [r for r in delay_rows if r["ok"]]
|
||||
top3 = alive[:3]
|
||||
|
||||
matrix: list[dict[str, Any]] = []
|
||||
for node in top3:
|
||||
print(f"[probe] matrix via {node['name']}", flush=True)
|
||||
switch_selector(args.api, node["name"])
|
||||
time.sleep(0.4)
|
||||
site_rows = []
|
||||
for url in TARGETS:
|
||||
method = "GET"
|
||||
row = curl_via_proxy(args.proxy, url, method=method)
|
||||
# some sites dislike HEAD; GET homepage / 204 only
|
||||
site_rows.append(row)
|
||||
print(
|
||||
f" {row['http_code']} ttfb={row['ttfb_s']} {url} err={row['error']}",
|
||||
flush=True,
|
||||
)
|
||||
matrix.append(
|
||||
{
|
||||
"node": node["name"],
|
||||
"type": node["type"],
|
||||
"region": node["region"],
|
||||
"delay_ms": node["delay_ms"],
|
||||
"sites": site_rows,
|
||||
}
|
||||
)
|
||||
|
||||
result = {
|
||||
"probed_at": now_iso(),
|
||||
"mihomo": version,
|
||||
"api": args.api,
|
||||
"mixed_proxy": args.proxy,
|
||||
"delay_url": args.delay_url,
|
||||
"timeout_ms": args.timeout_ms,
|
||||
"leaf_count": len(leaves),
|
||||
"tested_count": len(targets),
|
||||
"alive_count": len(alive),
|
||||
"alive_ge_3": len(alive) >= 3,
|
||||
"delay": delay_rows,
|
||||
"top3": [
|
||||
{"name": r["name"], "type": r["type"], "region": r["region"], "delay_ms": r["delay_ms"]}
|
||||
for r in top3
|
||||
],
|
||||
"matrix": matrix,
|
||||
"commands": {
|
||||
"up": "PROXY_SUB_URL=... bash bench/proxy/up.sh",
|
||||
"probe": "python3 bench/proxy/probe.py --all",
|
||||
"down": "bash bench/proxy/down.sh",
|
||||
},
|
||||
}
|
||||
os.makedirs(os.path.dirname(args.out) or ".", exist_ok=True)
|
||||
with open(args.out, "w", encoding="utf-8") as fh:
|
||||
json.dump(result, fh, ensure_ascii=False, indent=2)
|
||||
fh.write("\n")
|
||||
print(f"[probe] wrote {args.out} alive={len(alive)}/{len(targets)} ge3={len(alive) >= 3}")
|
||||
return 0 if alive else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
12
bench/proxy/probe.sh
Executable file
12
bench/proxy/probe.sh
Executable file
|
|
@ -0,0 +1,12 @@
|
|||
#!/usr/bin/env bash
|
||||
# Run node delay + target-site matrix against a live bench-proxy-mihomo.
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
export PYTHONPATH="$ROOT"
|
||||
export MIHOMO_API="${MIHOMO_API:-http://127.0.0.1:19090}"
|
||||
export MIXED_PROXY="${MIXED_PROXY:-http://127.0.0.1:17890}"
|
||||
if ! curl -sf --max-time 2 "${MIHOMO_API}/version" >/dev/null; then
|
||||
echo "error: mihomo API not reachable at $MIHOMO_API (run up.sh first)" >&2
|
||||
exit 1
|
||||
fi
|
||||
exec python3 "$ROOT/probe.py" --all --out "$ROOT/probe-results.json" "$@"
|
||||
20
bench/proxy/sub-summary.json
Normal file
20
bench/proxy/sub-summary.json
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"format": "clash-yaml",
|
||||
"proxy_count_raw": 71,
|
||||
"proxy_count_real": 69,
|
||||
"info_nodes": [
|
||||
"剩余流量：3.61 TB",
|
||||
"套餐到期：长期有效"
|
||||
],
|
||||
"types": {
|
||||
"vless": 52,
|
||||
"hysteria2": 17
|
||||
},
|
||||
"regions": {
|
||||
"日本": 19,
|
||||
"新加坡": 17,
|
||||
"美国": 29,
|
||||
"香港": 1,
|
||||
"台湾": 3
|
||||
}
|
||||
}
|
||||
95
bench/proxy/up.sh
Executable file
95
bench/proxy/up.sh
Executable file
|
|
@ -0,0 +1,95 @@
|
|||
#!/usr/bin/env bash
|
||||
# Fetch PROXY_SUB_URL into a temp runtime dir and start bench-proxy-mihomo.
|
||||
# Subscription bytes never land in the git tree.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
export PYTHONPATH="$ROOT"
|
||||
|
||||
MIXED_PORT="${MIXED_PORT:-17890}"
|
||||
API_PORT="${API_PORT:-19090}"
|
||||
CONTAINER="${CONTAINER:-bench-proxy-mihomo}"
|
||||
IMAGE_PRIMARY="${MIHOMO_IMAGE:-metacubex/mihomo:latest}"
|
||||
IMAGE_MIRROR="${MIHOMO_IMAGE_MIRROR:-docker.m.daocloud.io/metacubex/mihomo:latest}"
|
||||
RUNTIME_DIR="${RUNTIME_DIR:-/tmp/bench-proxy-runtime}"
|
||||
|
||||
if [[ -z "${PROXY_SUB_URL:-}" ]]; then
|
||||
echo "error: PROXY_SUB_URL is required" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
mkdir -p "$RUNTIME_DIR"
|
||||
chmod 700 "$RUNTIME_DIR"
|
||||
SUB_RAW="$RUNTIME_DIR/sub.yaml"
|
||||
CFG="$RUNTIME_DIR/config.yaml"
|
||||
|
||||
echo "[up] fetching subscription via PROXY_SUB_URL"
|
||||
HTTP_CODE="$(curl -sS -L --max-time 45 -A "clash-meta/bench-proxy" \
|
||||
-o "$SUB_RAW" -w "%{http_code}" "$PROXY_SUB_URL")"
|
||||
echo "[up] HTTP $HTTP_CODE size=$(wc -c < "$SUB_RAW") bytes"
|
||||
|
||||
if [[ "$HTTP_CODE" != "200" ]]; then
|
||||
echo "error: subscription fetch failed HTTP $HTTP_CODE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 - <<PY
|
||||
import sys
|
||||
sys.path.insert(0, "$ROOT")
|
||||
from lib import rewrite_runtime_config, summarize_subscription
|
||||
src = open("$SUB_RAW", encoding="utf-8", errors="replace").read()
|
||||
summary = summarize_subscription(src)
|
||||
print("[up] format=%s real_nodes=%s types=%s regions=%s" % (
|
||||
summary["format"], summary["proxy_count_real"], summary["types"], summary["regions"]))
|
||||
safe = {k: summary[k] for k in ("format", "proxy_count_raw", "proxy_count_real", "info_nodes", "types", "regions")}
|
||||
open("$ROOT/sub-summary.json", "w", encoding="utf-8").write(
|
||||
__import__("json").dumps(safe, ensure_ascii=False, indent=2) + "\n")
|
||||
# container always listens on 17890/9090; host ports are mapped separately
|
||||
cfg = rewrite_runtime_config(src, mixed_port=17890, controller="0.0.0.0:9090")
|
||||
open("$CFG", "w", encoding="utf-8").write(cfg)
|
||||
PY
|
||||
|
||||
ensure_image() {
|
||||
if docker image inspect "$IMAGE_PRIMARY" >/dev/null 2>&1; then
|
||||
echo "[up] using local image $IMAGE_PRIMARY" >&2
|
||||
printf '%s' "$IMAGE_PRIMARY"
|
||||
return
|
||||
fi
|
||||
echo "[up] pulling $IMAGE_PRIMARY" >&2
|
||||
if docker pull "$IMAGE_PRIMARY" >&2; then
|
||||
printf '%s' "$IMAGE_PRIMARY"
|
||||
return
|
||||
fi
|
||||
echo "[up] primary pull failed; trying mirror $IMAGE_MIRROR" >&2
|
||||
docker pull "$IMAGE_MIRROR" >&2
|
||||
printf '%s' "$IMAGE_MIRROR"
|
||||
}
|
||||
|
||||
IMAGE="$(ensure_image)"
|
||||
docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
|
||||
|
||||
echo "[up] starting $CONTAINER on mixed=$MIXED_PORT api=$API_PORT"
|
||||
docker run -d --name "$CONTAINER" --restart=no \
|
||||
-p "${MIXED_PORT}:17890" \
|
||||
-p "${API_PORT}:9090" \
|
||||
-v "${CFG}:/root/.config/mihomo/config.yaml:ro" \
|
||||
"$IMAGE" -d /root/.config/mihomo >/dev/null
|
||||
|
||||
echo "[up] waiting for external-controller"
|
||||
ok=0
|
||||
for i in $(seq 1 40); do
|
||||
if curl -sf --max-time 2 "http://127.0.0.1:${API_PORT}/version" >/dev/null; then
|
||||
ok=1
|
||||
break
|
||||
fi
|
||||
sleep 0.5
|
||||
done
|
||||
if [[ "$ok" != "1" ]]; then
|
||||
echo "[up] controller not ready; last logs:" >&2
|
||||
docker logs --tail 80 "$CONTAINER" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
curl -sS "http://127.0.0.1:${API_PORT}/version"
|
||||
echo
|
||||
echo "[up] ready container=$CONTAINER mixed=http://127.0.0.1:${MIXED_PORT} api=http://127.0.0.1:${API_PORT}"
|
||||
14
bench/searxng-cn/compose.yml
Normal file
14
bench/searxng-cn/compose.yml
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
services:
|
||||
searxng:
|
||||
image: docker.io/searxng/searxng:2026.8.29-d226b78bc
|
||||
platform: linux/arm64
|
||||
container_name: bench-s3b-searxng-cn
|
||||
ports:
|
||||
- "18880:8080"
|
||||
volumes:
|
||||
- ./settings.yml:/etc/searxng/settings.yml:ro
|
||||
environment:
|
||||
- SEARXNG_BASE_URL=http://localhost:18880/
|
||||
- SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
|
||||
mem_limit: 192m
|
||||
restart: "no"
|
||||
144
bench/searxng-cn/eval_t1.py
Executable file
144
bench/searxng-cn/eval_t1.py
Executable file
|
|
@ -0,0 +1,144 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Assert one SearXNG T1 JSON body. stdout: JSON verdict."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from urllib.parse import urlparse
|
||||
|
||||
POLICY = re.compile(
|
||||
r"(gov\.cn|customs\.gov\.cn|chinatax\.gov\.cn|mofcom\.gov\.cn|"
|
||||
r"yidaiyilu\.gov\.cn|news\.cn|people\.com\.cn|xinhuanet\.com)$",
|
||||
re.I,
|
||||
)
|
||||
T2_PREF = re.compile(r"(gov\.cn|news\.cn|people\.com\.cn)$", re.I)
|
||||
|
||||
|
||||
def first_cjk4(text: str) -> str:
|
||||
chars = re.findall(r"[\u4e00-\u9fff]", text or "")
|
||||
return "".join(chars[:4])
|
||||
|
||||
|
||||
def eval_body(raw: str) -> dict:
|
||||
out = {
|
||||
"assert_ok": False,
|
||||
"fail_class": "",
|
||||
"note": "",
|
||||
"n_results": 0,
|
||||
"n_rich": 0,
|
||||
"engines": [],
|
||||
"engine_counts": {},
|
||||
"unresponsive_engines": [],
|
||||
"policy_hosts": [],
|
||||
"hosts": [],
|
||||
"t2_url": "",
|
||||
"t2_title": "",
|
||||
"t2_title_cjk4": "",
|
||||
"sample_titles": [],
|
||||
}
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
out["fail_class"] = "json_403"
|
||||
out["note"] = "body_not_json"
|
||||
return out
|
||||
if isinstance(data, dict) and data.get("error"):
|
||||
out["fail_class"] = "json_403"
|
||||
out["note"] = str(data.get("error"))[:200]
|
||||
return out
|
||||
results = data.get("results") if isinstance(data, dict) else None
|
||||
if not isinstance(results, list):
|
||||
out["fail_class"] = "json_403"
|
||||
out["note"] = "no_results_array"
|
||||
return out
|
||||
out["n_results"] = len(results)
|
||||
engines = []
|
||||
counts = {}
|
||||
hosts = []
|
||||
policy = []
|
||||
rich = 0
|
||||
t2_url = ""
|
||||
t2_title = ""
|
||||
for i, item in enumerate(results):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
title = (item.get("title") or "").strip()
|
||||
url = (item.get("url") or "").strip()
|
||||
content = (item.get("content") or "").strip()
|
||||
eng = item.get("engine") or ""
|
||||
if isinstance(item.get("engines"), list) and not eng:
|
||||
eng = item["engines"][0] if item["engines"] else ""
|
||||
if eng:
|
||||
engines.append(str(eng))
|
||||
counts[str(eng)] = counts.get(str(eng), 0) + 1
|
||||
host = urlparse(url).hostname or ""
|
||||
if host:
|
||||
hosts.append(host.lower())
|
||||
if POLICY.search(host.lower()):
|
||||
policy.append(host.lower())
|
||||
if title and url.startswith("http") and len(content) >= 8:
|
||||
rich += 1
|
||||
if i < 3:
|
||||
out["sample_titles"].append({"title": title[:120], "url": url[:200], "engine": eng, "host": host})
|
||||
if not t2_url and url.startswith("http") and T2_PREF.search(host.lower()):
|
||||
t2_url = url
|
||||
t2_title = title
|
||||
out["n_rich"] = rich
|
||||
out["engines"] = sorted(set(engines))
|
||||
out["engine_counts"] = counts
|
||||
out["hosts"] = hosts[:20]
|
||||
out["policy_hosts"] = sorted(set(policy))
|
||||
out["t2_url"] = t2_url
|
||||
out["t2_title"] = t2_title
|
||||
out["t2_title_cjk4"] = first_cjk4(t2_title)
|
||||
unresp = data.get("unresponsive_engines") or []
|
||||
out["unresponsive_engines"] = unresp
|
||||
notes = []
|
||||
if unresp:
|
||||
notes.append(f"unresponsive={unresp}")
|
||||
|
||||
ok = True
|
||||
if out["n_results"] < 5:
|
||||
ok = False
|
||||
out["fail_class"] = "engine_outage"
|
||||
notes.append("results<5")
|
||||
if rich < 5:
|
||||
# first 10 rich count — we counted all; restrict to first 10
|
||||
rich10 = 0
|
||||
for item in results[:10]:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
title = (item.get("title") or "").strip()
|
||||
url = (item.get("url") or "").strip()
|
||||
content = (item.get("content") or "").strip()
|
||||
if title and url.startswith("http") and len(content) >= 8:
|
||||
rich10 += 1
|
||||
out["n_rich"] = rich10
|
||||
if rich10 < 5:
|
||||
ok = False
|
||||
out["fail_class"] = out["fail_class"] or "thin_snippets"
|
||||
notes.append("rich<5_in_top10")
|
||||
if len(out["engines"]) < 2:
|
||||
# not a hard fail per select.md: 单引擎全灭记 engine_outage，不直接判方案失败
|
||||
# but plan T1 assert 4 says engines >= 2
|
||||
ok = False
|
||||
out["fail_class"] = out["fail_class"] or "engine_outage"
|
||||
notes.append("engines<2")
|
||||
if not out["policy_hosts"]:
|
||||
ok = False
|
||||
notes.append("no_policy_host")
|
||||
# plan: assert_ok=false but keep host list
|
||||
out["assert_ok"] = ok
|
||||
out["note"] = "; ".join(notes)
|
||||
return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
path = sys.argv[1]
|
||||
raw = open(path, "r", encoding="utf-8", errors="replace").read()
|
||||
print(json.dumps(eval_body(raw), ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
1
bench/searxng-cn/raw/bodies/t1-1.json
Normal file
1
bench/searxng-cn/raw/bodies/t1-1.json
Normal file
File diff suppressed because one or more lines are too long
1
bench/searxng-cn/raw/bodies/t1-2.json
Normal file
1
bench/searxng-cn/raw/bodies/t1-2.json
Normal file
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue