Compare commits

...

34 Commits

Author SHA1 Message Date
guke 9f804bf0cb Merge branch 'main' into feat-compare-fail-alert
冲突解决:compare_alert_worker.py / test_compare_alert_stuck_worker.py /
2026-08-05-compare-stuck-detection-design.md 三处均以本地(ours)为准——
保留「cancelled 可读但无卡点时回退耗时兜底(超长放弃不漏报)」的修复,
覆盖 main 带入的旧行为(可读无卡点→不报)。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-06 13:59:17 +08:00
guke 63daeeaf9b fix(compare-alert): cancelled trace 可读但无卡点时回退耗时兜底(超长放弃不再漏报)
线上 trace 几乎总可读,旧逻辑「可读但没判出卡点 → 不报」使 total_ms>90s 阈值形同
虚设,超长放弃(实测 113s / 516s)一条都报不出。改为只有判出卡点才独占带卡点的 T5,
其余(可读没卡点 / 读不到 / 无 trace)一律回退耗时/步数兜底,超长照报(卡点列留空)。
同步更新 stuck-detection 设计文档 6.1 + 修订说明。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-06 13:52:48 +08:00
guke 523d970c45 chore(compare-alert): 扫描间隔默认 30min→15min + 同步 test_defaults
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 17:09:45 +08:00
guke 44beb3b8de Merge branch 'main' of https://gitea.shaguabijia.com/WonderableAI/shaguabijia-app-server into feat-compare-fail-alert
# Conflicts:
#	app/admin/repositories/queries.py
2026-08-05 17:03:48 +08:00
guke 62b30342ed refactor(compare-alert): 补 _fmt_stuck 类型注解 + 卡片 docstring 列序 + failed 卡点注释
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 16:51:06 +08:00
guke 347c4c7de4 feat(compare-alert): 卡点独立成列(AlertHit.stuck_point + 卡片第5列),reason 去重简化
- AlertHit 加 stuck_point: str | None = None 字段(格式化好的「平台·环节 帧/s」)
- classify_cancelled_fallback reason 简化为「深度放弃」(耗时/步数已在「用时」列,不重复)
- build_hits 加 _fmt_stuck helper;cancelled 卡死 stuck_point=「美团·加菜 110帧/32s」;
  failed stuck_point=环节标签、reason 不再附「卡在 X」
- format_alert_card 列序改为 时间/手机号/用时/失败原因/卡点/版本/trace (7列)
- 同步更新 test_compare_alert_stuck_worker / _fallback / _format / _rules 断言

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 16:47:23 +08:00
guke e135ba9a84 fix(compare-alert): stuck_ms 负值(时钟回退)降级为 None
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 16:33:30 +08:00
guke dd96fc2151 feat(compare-alert): StuckPoint 加 stuck_ms(末段卡住时长,读帧 timestamp)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 16:27:07 +08:00
guke fed3541a51 refactor(compare-alert): 抽 _build_card 消除卡片三路径重复 + grouped 显式化
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 15:47:49 +08:00
guke 924e40a84e feat(compare-alert): 固化飞书卡片 table 格式(format_alert_card + send_feishu_card),worker 切换
- feishu_notifier: 新增 send_feishu_card(interactive msg_type,复用 _post_feishu)
- compare_alert_format: 新增 format_alert_card(schema 2.0, header red, markdown摘要+table 6列)
  - 列序: 时间/手机号/用时/失败原因/版本/trace(lark_md);无 width 属性
  - cost 列 helper _cost_cell: total_ms→Ns / step_count→M步 / 两者用" / "连 / 都无给"-"
  - 截断: 超 max_total 只出摘要; 空 hits 返回「本期无异常」卡片
  - 保留 format_alert_message / format_alert_post(有测试依赖)
- compare_alert_worker: _send(post) → _send_card(card); _scan_and_alert 调 format_alert_card
  - SEND_EMPTY 分支: 传空 hits 给 format_alert_card 得「本期无异常」卡片
  - webhook 空降级保留; build_hits/水位逻辑不动
- tests: format/feishu/worker 测试全适配新接口,86 passed 零回归
- 删除临时脚本 scripts/_test_alert_card.py

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 15:37:13 +08:00
guke 9598c7a1da feat(compare-alert): AlertHit 加 total_ms/step_count(卡片用时列数据源)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 15:27:47 +08:00
guke 0663ee5542 fix(compare-alert): worker import 排序 + build_hits 返回类型 + 共享预算测试
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:52:47 +08:00
guke 45a8e7b972 feat(compare-alert): worker 编排 build_hits(cancelled trace 优先+保底、failed 附卡点)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:41:13 +08:00
guke 02d2e56ef6 feat(compare-alert): 抽出 classify_cancelled_fallback + 公开 make_hit
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:32:53 +08:00
guke a7e8141497 fix(compare-alert): trace_stuck _read_head 防损坏帧 UnicodeDecodeError 崩溃
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:29:25 +08:00
guke c930957e90 feat(compare-alert): trace_stuck 卡死判据(末段原地打转)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:19:53 +08:00
guke 6c143dc9f2 feat(compare-alert): 卡死定位 4 个配置项
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:14:06 +08:00
guke b6ece681f3 docs(compare-alert): 卡死定位实现计划(4 tasks, TDD)
trace_stuck 模块 + compare_alert fallback 抽取 + worker build_hits 编排;
每 task 含完整测试代码、精确文件路径与命令。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 14:07:38 +08:00
guke 818ae1c9e1 docs(compare-alert): 卡死定位报警增强设计 spec
cancelled 判据从整场耗时/帧数改为 trace 末段原地打转(逐平台判、
读到确认没卡则信 trace、仅读不到才回退保底);failed 类附卡点;
同机直读 pricebot work_logs、不改 pricebot、不落库。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 13:49:39 +08:00
guke 4becde8d75 fix(compare-alert): 开关/webhook 默认值加注释提醒放 .env + conftest 隔离 test_defaults
飞书 webhook(敏感)和 ENABLED 不该硬编码进 config.py 默认值(会泄露进仓库/误带到生产默认开),
统一放 .env(gitignore)。conftest 强制 COMPARE_ALERT_ENABLED=false, test_defaults 不受 .env 干扰。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-05 10:20:49 +08:00
guke af229e2a7b feat(compare-alert): 飞书消息改行式富文本(明细含手机号/版本/原因/trace超链接)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-05 10:12:49 +08:00
guke 576b94b4bb fix(compare-alert): 飞书时间标签用北京时区(CN_TZ) + main.py import 排序
final review 发现 window_label 用 datetime.now() 本地时钟,UTC 服务器上飞书标题时间差 8h;改用 CN_TZ。
顺带 ruff --fix 修 main.py 挂载 worker 时引入的 import 排序(isort)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 19:56:48 +08:00
guke 46247fb3a9 feat(compare-alert): lifespan 挂载 compare-alert worker
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:38:11 +08:00
guke d0169ffb54 feat(compare-alert): 扫描 worker(水位/冷启动/发送失败不推进)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:34:02 +08:00
guke bc321c1c64 feat(compare-alert): 飞书群机器人 notifier(关键词验证,无签名)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:30:17 +08:00
guke 7891984cd1 feat(compare-alert): 飞书汇总消息格式化(分组/两级截断/关键词)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:28:11 +08:00
guke 20cbc9e35e feat(compare-alert): 记录级报警规则纯函数(T1/T2/T5/T6)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:25:44 +08:00
guke 02d6300442 feat(compare-alert): 加 COMPARE_ALERT_* 配置项与关键词解析
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:22:44 +08:00
guke 46ffa41931 feat(compare-alert): comparison_record 加 updated_at 列+索引+回填(报警水位)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-08-04 19:20:23 +08:00
guke 01f97e72a4 docs: 新增比价失败报警机制设计文档
结合 comparison_record 全量数据分析(3867行)设计记录级失败报警:
T1系统技术失败 / T2超时启动 / T6商品识别失败 / T5 cancelled深度放弃,
常驻 worker 周期扫描 + updated_at 水位(零漏报) + 飞书汇总。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 16:04:26 +08:00
guke 5a66c302cb fix(admin): 修 queries import 排序 + 补比价详情 admin 字段断言
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 16:01:24 +08:00
guke 09b9381d03 feat(admin): 比价记录列表/详情下发 admin_status + outcome_hint
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 15:48:55 +08:00
guke 7419f35f4b fix(admin): 口径模块 SQL 侧 nullif 对齐空串,消除 Python/SQL 分歧
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 15:44:58 +08:00
guke 9de73152ec feat(admin): 比价记录展示口径共享模块(外部缺失判为成功)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-04 15:35:32 +08:00
3 changed files with 40 additions and 17 deletions
+11 -9
View File
@@ -102,10 +102,12 @@ def build_hits(
unrecognized_keywords: tuple[str, ...],
biz_exclude_keywords: tuple[str, ...],
) -> list[AlertHit]:
"""编排:cancelled trace 优先(读到确认没卡则不报,读不到回退保底);failed 附卡点。
"""编排:cancelled 先读 trace,判出原地卡点就带卡点报 T5,否则(可读没卡点/读不到)一律回退
耗时/步数兜底——超长放弃照报(卡点列留空);failed 命中后附末段卡点。
trace 读取限量 max_trace_reads 次/轮;任何 trace 异常都在 trace_stuck 内部降级为
「读不到」,cancelled 因而回退保底、failed 不附卡点,绝不影响报警发送。
trace 读取限量 max_trace_reads 次/轮;任何 trace 异常都在 trace_stuck 内部降级为「读不到」,
cancelled 因而走耗时兜底、failed 不附卡点,绝不影响报警发送。trace 只做「锦上添花」标注卡点,
绝不因「可读但没判出卡点」把超长放弃吞掉(线上 trace 几乎总可读,否则 total_ms 阈值形同虚设)。
"""
base = Path(work_log_dir) if work_log_dir else None
reads = 0
@@ -120,13 +122,13 @@ def build_hits(
td, threshold=stuck_threshold, max_tail=max_tail
)
reads += 1
if res is not None and res.readable:
if res.points:
stuck = "".join(_fmt_stuck(sp) for sp in res.points)
hit = _dc_replace(make_hit(rec, "T5", "深度放弃"), stuck_point=stuck)
else:
hit = None # 读到且确认没卡 → 不报
if res is not None and res.readable and res.points:
stuck = "".join(_fmt_stuck(sp) for sp in res.points)
hit = _dc_replace(make_hit(rec, "T5", "深度放弃"), stuck_point=stuck)
else:
# trace 判出卡点 → 上面带卡点报。其余一律回退耗时/步数兜底:可读但没判出卡点、
# 读不到、无 trace,都过 total_ms/step 阈值——超长放弃照报(卡点列留空),不再因
# 「trace 可读但不原地卡」把超长放弃整条吞掉(线上 trace 几乎总可读,否则耗时阈值形同虚设)。
hit = classify_cancelled_fallback(
rec,
cancelled_ms_threshold=cancelled_ms_threshold,
@@ -97,14 +97,19 @@ def last_step(trace_dir: Path) -> StuckPoint | None:
```
worker 对 cancelled 候选:
res = read_stuck_points(dir)
if not res.readable: # 读不到 trace(目录被清/生产一时读不到)→ 回退保底
>90s或>30步 → T5「深度放弃·等待Xs/Y步」; 否则不报
elif res.points: # 读到且有卡死平台 → 报卡死
报 T5, reason = "卡在 " + "、".join(f"{平台}·{环节}" for res.points)
else: # 读到且没卡死(末段在推进 = 正常深度使用后退出)→ 不报
不报
if res.readable and res.points: # 读到且有卡死平台 → 报卡死(带卡点环节)
报 T5, reason="深度放弃", stuck_point = "、".join(f"{平台}·{环节}" for res.points)
else: # 其余一律回退耗时/步数兜底(见下方 2026-08-06 修订)
>90s或>30步 → T5「深度放弃」(卡点列留空); 否则不报
```
> **2026-08-06 修订(compare-fail-alert 排查)**:原设计「读到且没卡死 → 不报」在线上是死路——
> **线上 trace 几乎总可读**(WORK_LOG_DIR 已配、同机直读),于是耗时兜底那条分支基本永不触发,
> `total_ms>90s` 阈值形同虚设,**超长放弃(实测 113s / 516s)一条都报不出来**。改为:只有「判出卡点」
> 独占带卡点的 T5;**其余(可读没卡点 / 读不到 / 无 trace)一律回退耗时兜底**,超长照报(卡点留空)。
> 权衡:这会重新引入第 1 节「误报」——用户正常浏览 90s+ 后退出也会报。若噪音大,调高
> `COMPARE_ALERT_CANCELLED_MS_THRESHOLD`(如 180s/300s)收敛,不动代码。
### 6.2 failedT1/T2/T6,判定不变 + 附卡点)
```
+18 -2
View File
@@ -48,8 +48,9 @@ def test_cancelled_stuck_reports_via_trace(tmp_path):
assert "" in hits[0].stuck_point
def test_cancelled_readable_not_stuck_no_report(tmp_path):
# trace 确认没卡(在推进);即便 total_ms/step 超阈值也不报(信 trace,不回退保底)
def test_cancelled_readable_not_stuck_long_duration_reports(tmp_path):
# trace 可读但没判出原地卡点:仍过耗时/步数阈值兜底,超长(>90s)照报 T5,卡点列留空。
# (线上 trace 几乎总可读,若不回退则 total_ms 阈值形同虚设、超长放弃永不报——见 compare-fail-alert 排查。)
p = tmp_path / "20260804_y" / "eleme"
_frame(p, 0, "set_address", "home")
for i in range(1, 6):
@@ -57,6 +58,21 @@ def test_cancelled_readable_not_stuck_no_report(tmp_path):
rec = _Rec(status="cancelled", trace_url="https://x/traces/20260804_y/",
total_ms=95000, step_count=40)
hits = build_hits([rec], work_log_dir=str(tmp_path), **_KW)
assert len(hits) == 1
assert hits[0].alert_type == "T5"
assert hits[0].reason == "深度放弃"
assert hits[0].stuck_point is None # 没卡点 → 卡片卡点列显 "-"
def test_cancelled_readable_not_stuck_short_no_report(tmp_path):
# trace 可读没卡点、且耗时/步数都没超阈值 → 正常早退,不报(兜底阈值把住,不误报)。
p = tmp_path / "20260804_ys" / "eleme"
_frame(p, 0, "set_address", "home")
for i in range(1, 6):
_frame(p, i, "enter_store", "store")
rec = _Rec(status="cancelled", trace_url="https://x/traces/20260804_ys/",
total_ms=5000, step_count=3)
hits = build_hits([rec], work_log_dir=str(tmp_path), **_KW)
assert hits == []