初始化 dish-image-toolkit:菜品图爬虫 + 图片名称检索平台

- crawler/:豆果(主)/下厨房/Bing 爬虫 + dishes.jsonl(3056 菜→9167 图映射)+ verify_repair 按 URL 重下
- retrieval/:三路检索(BM25 + 本地 BGE-M3 向量 + RRF 融合),FastAPI + 前端;
  写死图片目录(默认 crawler/images,可 IMAGE_DIR 覆盖)、绑 0.0.0.0 局域网访问、
  启动自动建索引、服务器 serve 图片
- 图片(1.7G)与向量模型(2.3G)不进 git

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,20 @@
+# 图片(约 9167 张 / 1.7G)不进 git —— 按 crawler 脚本重爬,或另行拷贝
+crawler/images/
+images/
+*.rar
+
+# 本地向量模型(BGE-M3 ~2.3G,首次运行自动下载)
+retrieval/hf_home/
+hf_home/
+
+# 运行时缓存 / 日志 / 产物
+retrieval/cache/
+cache/
+*.log
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.env

README.md

@@ -0,0 +1,31 @@
+# dish-image-toolkit
+
+菜品图工具集:① **爬取**菜品图 + ② 基于**图片名称**的三路检索平台(BM25 / 本地向量 / RRF 融合)。为"比价后展示/检索菜品图"做支撑。
+
+> ⚠ **图片(约 9167 张 / 1.7G)和向量模型(BGE-M3 ~2.3G)不进 git。** clone 后按下面步骤重爬图、首次运行检索平台会自动下模型。
+
+## 目录
+- **`crawler/`** — 菜品图爬虫 + 菜→图映射 `dishes.jsonl`
+- **`retrieval/`** — 检索平台(FastAPI + 前端),目录写死、局域网可访问
+
+## ① 爬图(crawler/)
+豆果 App 接口为主(质量最好):
+```
+cd crawler
+python scrape_douguo.py --input 菜名.txt --out dishes.jsonl --download --per-dish 3 --workers 8 --replace
+```
+图片下到 **`crawler/images/`**,映射追加到 `dishes.jsonl`(断点续爬)。杀软清图后 `python verify_repair.py --repair` 按 URL 重下。详见 [crawler/README.md](crawler/README.md)。
+
+## ② 检索平台(retrieval/)
+```
+cd retrieval
+run.bat            # 或 pip install -r requirements.txt && python server.py
+```
+- **图片目录 = 本仓库的 `crawler/images`**(即上一步爬的图)。
+  - 具体路径:仓库在 `E:\codes\dish-image-toolkit` 时 → **`E:\codes\dish-image-toolkit\crawler\images`**。
+  - 想指向别处已有的图(不搬文件):启动前 `set IMAGE_DIR=<绝对路径>` 覆盖即可,无需改代码。
+- 启动即扫描该目录、自动建索引(BM25 + 本地 BGE-M3 向量);**首次**下模型 ~2.3G(走 hf-mirror),之后约 25s 加载。
+- 绑 `0.0.0.0`,控制台打印**局域网地址**;同事同局域网打开即直接搜(无需选文件夹)。
+  - 连不上多半是防火墙,管理员执行一次:`netsh advfirewall firewall add rule name="dish-retrieval" dir=in action=allow protocol=TCP localport=8799`
+
+三路检索原理与接口见 [retrieval/README.md](retrieval/README.md)。

crawler/README.md

@@ -0,0 +1,23 @@
+# crawler — 菜品图爬虫
+
+给菜名爬多图,记录"菜→哪些图"到 `dishes.jsonl`。
+
+## 源(优先级从高到低)
+- **`scrape_douguo.py` — 豆果 App 接口(最终采用,质量最好)**:`POST api.douguo.net/recipe/search`,960px 高清,无需签名。⚠ 图床 `i1.douguo.com` 有防盗链(下约 1100 张后返 HTTP 493),脚本已自动改走镜像 `i1.douguo.net`(http)。
+- `scrape_xiachufang.py` — 下厨房(菜谱真实图,质量好;但搜索接口硬限速,备用)。
+- `scrape_bing.py` — Bing 图搜(通用图搜,质量差、很多不是菜品,已弃用)。
+
+## 用法
+```
+python scrape_douguo.py --input 菜名.txt --out dishes.jsonl --download --per-dish 3 --workers 8 --replace
+```
+- 图片 → `images/<菜名>_NN.jpg`;映射 → `dishes.jsonl`,每行:
+  `{dish, found, count, images:[{image_url, image_file, ...}], ...}`
+- **断点续爬**:只跳 `found:true` 的菜,失败的会重试。菜名也可直接作命令行参数传。
+
+## 工具
+- `verify_repair.py [--repair]` — 校验 `dishes.jsonl` 里 `image_file` 的完好率;`--repair` 按 `image_url` 重下缺失/损坏图(应对杀软定期清图)。
+- `probe_status.py` — 统计 `dishes.jsonl` 的成功/失败数。
+
+## 现状
+`dishes.jsonl` 已含 **3056 道**(下厨房 974 + 豆果 2082),对应 **9167 张图**。图不在 git 里 —— `--download` 重爬,或把已有 `images/` 拷进来。

crawler/probe_status.py

@@ -0,0 +1,10 @@
+import requests, sys
+H = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
+     "Accept-Language": "zh-CN,zh;q=0.9", "Referer": "https://www.xiachufang.com/"}
+kw = sys.argv[1] if len(sys.argv) > 1 else "番茄炒蛋"
+try:
+    r = requests.get("https://www.xiachufang.com/search/", params={"keyword": kw, "cat": 1001},
+                     headers=H, timeout=20)
+    print(r.status_code)
+except Exception:
+    print("ERR")

crawler/scrape_bing.py

@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""Bing 图片菜品爬虫(每菜多图版,下厨房限速后切换的备用源)。
+
+每道菜去 Bing 图片搜索,取**前 N 张**结果,下载 Bing CDN 缩略图(turl,ts*.mm.bing.net
+极耐爬/统一可靠;原图 murl 也一并记录备用),每菜一行 JSON 到 jsonl,images:[...] 记录
+该菜对应的所有图。与 scrape_xiachufang.py 共用 dishes.jsonl + images/:
+  - 续抓只跳过 found:true 的菜(失败的会重试)→ 自动只补下厨房没爬到的 ~2082 道
+  - 同样的并行 + 全局自适应限速闸(任一线程被限速→全员冷却,根治退避雪崩)
+
+用法:
+  python scrape_bing.py --out dishes.jsonl --download --per-dish 3 --workers 8 --input todo.txt
+"""
+from __future__ import annotations
+
+import argparse
+import html
+import json
+import os
+import random
+import re
+import sys
+import threading
+import time
+import urllib.parse
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timezone
+
+import requests
+
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception:
+        pass
+
+SEARCH_URL = "https://cn.bing.com/images/search"
+
+UA_POOL = [
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Edg/123.0.0.0",
+]
+ACCEPT_LANGS = ["zh-CN,zh;q=0.9,en;q=0.8", "zh-CN,zh;q=0.9", "zh-CN,zh;q=0.8,en-US;q=0.5"]
+
+
+def build_headers() -> dict:
+    return {
+        "User-Agent": random.choice(UA_POOL),
+        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+        "Accept-Language": random.choice(ACCEPT_LANGS),
+        "Referer": "https://cn.bing.com/images/",
+    }
+
+
+_tls = threading.local()
+
+
+def session() -> requests.Session:
+    s = getattr(_tls, "s", None)
+    if s is None:
+        s = _tls.s = requests.Session()
+    return s
+
+
+# ---------- 全局限速治理(同 scrape_xiachufang:节流 + 全员冷却闸,根治雪崩) ----------
+_pace_lock = threading.Lock()
+_next_search = 0.0
+_resume_at = 0.0
+_penalty = 0
+SEARCH_SPACING = 0.5  # Bing 耐爬,默认间隔小;可被 --spacing 覆盖
+
+
+def search_pace() -> None:
+    global _next_search
+    with _pace_lock:
+        now = time.time()
+        slot = max(now, _next_search)
+        _next_search = slot + SEARCH_SPACING + random.uniform(0, 0.3)
+        wait = slot - now
+    if wait > 0:
+        time.sleep(wait)
+
+
+def gate_wait() -> None:
+    while True:
+        with _pace_lock:
+            wait = _resume_at - time.time()
+        if wait <= 0:
+            return
+        time.sleep(min(wait, 3) + random.uniform(0, 0.4))
+
+
+def trip_limit() -> None:
+    global _resume_at, _penalty
+    with _pace_lock:
+        _penalty += 1
+        cool = min(45 * (1.5 ** (_penalty - 1)), 600)
+        nr = time.time() + cool
+        if nr > _resume_at:
+            _resume_at = nr
+            print(f"  [!] 限速#{_penalty} 全局冷却 {cool:.0f}s", file=sys.stderr, flush=True)
+
+
+def ease_limit() -> None:
+    global _penalty
+    if _penalty:
+        with _pace_lock:
+            _penalty = max(0, _penalty - 1)
+
+
+def fetch_search(dish: str, proxy: str | None, retries: int = 5, timeout: int = 25) -> str | None:
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    params = {"q": dish, "first": 1, "count": 35}
+    url = SEARCH_URL + "?" + urllib.parse.urlencode(params)
+    for attempt in range(retries):
+        gate_wait()
+        search_pace()
+        try:
+            r = session().get(url, headers=build_headers(), proxies=proxies, timeout=timeout)
+            if r.status_code == 200:
+                ease_limit()
+                return r.text
+            if r.status_code in (403, 429, 503):
+                trip_limit()
+            else:
+                print(f"  [!] 状态 {r.status_code}", file=sys.stderr)
+                time.sleep(3 * (attempt + 1))
+        except requests.RequestException as e:
+            print(f"  [!] 请求异常 {e}", file=sys.stderr)
+            time.sleep(5 + random.uniform(0, 5))
+    return None
+
+
+def parse_images(html_text: str, n: int) -> list[dict]:
+    """从 Bing 结果页抽前 n 张图的 {image_url(=turl), murl, turl, title, host_page}。"""
+    out: list[dict] = []
+    seen: set[str] = set()
+    for m in re.finditer(r'class="iusc"[^>]+?\sm="([^"]+)"', html_text):
+        try:
+            d = json.loads(html.unescape(m.group(1)))
+        except Exception:
+            continue
+        url = d.get("turl") or d.get("murl")
+        if not url or url in seen:
+            continue
+        seen.add(url)
+        out.append({
+            "image_url": url,
+            "murl": d.get("murl"),
+            "turl": d.get("turl"),
+            "title": (d.get("t") or "").strip() or None,
+            "host_page": d.get("purl"),
+        })
+        if len(out) >= n:
+            break
+    return out
+
+
+def is_image(b: bytes) -> bool:
+    return (b[:3] == b"\xff\xd8\xff" or b[:8] == b"\x89PNG\r\n\x1a\n"
+            or b[:6] in (b"GIF87a", b"GIF89a")
+            or (b[:4] == b"RIFF" and b[8:12] == b"WEBP") or b[:2] == b"BM")
+
+
+def safe_name(dish: str) -> str:
+    return re.sub(r"[^\w一-鿿]+", "_", dish).strip("_") or "dish"
+
+
+def download(url: str, dish: str, idx: int, img_dir: str,
+             proxy: str | None) -> tuple[str | None, int]:
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    headers = {"User-Agent": random.choice(UA_POOL), "Referer": "https://cn.bing.com/",
+               "Accept": "image/avif,image/webp,image/png,image/*;q=0.8,*/*;q=0.5"}
+    try:
+        r = session().get(url, headers=headers, proxies=proxies, timeout=20)
+        if r.status_code == 200 and len(r.content) >= 800 and is_image(r.content):
+            path = os.path.join(img_dir, f"{safe_name(dish)}_{idx:02d}.jpg")
+            with open(path, "wb") as f:
+                f.write(r.content)
+            rel = os.path.relpath(path, os.path.dirname(img_dir) or ".").replace(os.sep, "/")
+            return rel, len(r.content)
+    except requests.RequestException as e:
+        print(f"  [!] 下图失败 {e}", file=sys.stderr)
+    return None, 0
+
+
+def scrape_one(dish: str, proxy: str | None, do_download: bool, img_dir: str,
+               per_dish: int, jitter: float = 0.0) -> dict:
+    if jitter:
+        time.sleep(random.uniform(0, jitter))
+    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+    base = {"dish": dish, "source": "bing", "fetched_at": now}
+    html_text = fetch_search(dish, proxy)
+    if html_text is None:
+        return {**base, "found": False, "count": 0, "images": [], "reason": "fetch_failed"}
+    recs = parse_images(html_text, per_dish)
+    if not recs:
+        return {**base, "found": False, "count": 0, "images": [], "reason": "no_result"}
+    images = []
+    for i, rec in enumerate(recs, 1):
+        e = dict(rec)
+        if do_download:
+            rel, size = download(rec["image_url"], dish, i, img_dir, proxy)
+            e["image_file"] = rel
+            e["bytes"] = size
+            if i < len(recs):
+                time.sleep(random.uniform(0.05, 0.2))
+        images.append(e)
+    saved = sum(1 for e in images if not do_download or e.get("image_file"))
+    if do_download and saved == 0:
+        return {**base, "found": False, "count": 0, "images": images, "reason": "download_failed"}
+    return {**base, "found": True, "count": saved, "images": images}
+
+
+def load_done(out_path: str) -> set[str]:
+    done = set()
+    if os.path.exists(out_path):
+        with open(out_path, encoding="utf-8") as f:
+            for line in f:
+                try:
+                    rec = json.loads(line)
+                    if rec.get("found"):  # 只跳过成功的;失败行会重试
+                        done.add(rec["dish"])
+                except Exception:
+                    pass
+    return done
+
+
+def main():
+    global SEARCH_SPACING
+    ap = argparse.ArgumentParser()
+    ap.add_argument("dishes", nargs="*")
+    ap.add_argument("--input", help="菜名文件,一行一道")
+    ap.add_argument("--out", default="dishes.jsonl")
+    ap.add_argument("--download", action="store_true")
+    ap.add_argument("--img-dir", default="images")
+    ap.add_argument("--per-dish", type=int, default=3)
+    ap.add_argument("--proxy", default=os.environ.get("CRAWL_PROXY"))
+    ap.add_argument("--workers", type=int, default=8)
+    ap.add_argument("--jitter", type=float, default=1.0)
+    ap.add_argument("--spacing", type=float, default=SEARCH_SPACING)
+    args = ap.parse_args()
+    SEARCH_SPACING = args.spacing
+
+    dishes = list(args.dishes)
+    if args.input:
+        with open(args.input, encoding="utf-8") as f:
+            dishes += [ln.strip() for ln in f if ln.strip()]
+    if not dishes:
+        ap.error("没给菜名")
+    dishes = list(dict.fromkeys(dishes))
+
+    os.makedirs(args.img_dir, exist_ok=True)
+    done = load_done(args.out)
+    todo = [d for d in dishes if d not in done]
+    print(f"共 {len(dishes)} 道,已完成 {len(done)},本次待抓 {len(todo)},并行 {args.workers} 线程,源 Bing",
+          flush=True)
+    if not todo:
+        print("没有待抓的菜。")
+        return
+
+    ok = miss = 0
+    t0 = time.time()
+    lock = threading.Lock()
+    with open(args.out, "a", encoding="utf-8") as fout, \
+            ThreadPoolExecutor(max_workers=args.workers) as ex:
+        futs = {ex.submit(scrape_one, d, args.proxy, args.download,
+                          args.img_dir, args.per_dish, args.jitter): d for d in todo}
+        for i, fut in enumerate(as_completed(futs), 1):
+            dish = futs[fut]
+            try:
+                rec = fut.result()
+            except Exception as e:
+                now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+                rec = {"dish": dish, "source": "bing", "found": False, "count": 0,
+                       "images": [], "reason": f"error:{type(e).__name__}", "fetched_at": now}
+            with lock:
+                fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
+                fout.flush()
+            if rec["found"]:
+                ok += 1
+            else:
+                miss += 1
+            if i % 25 == 0 or i == len(todo):
+                rate = i / max(time.time() - t0, 1e-6)
+                eta = (len(todo) - i) / rate if rate else 0
+                print(f"[{i}/{len(todo)}] ✓{ok} ✗{miss}  {rate*60:.0f}道/分  ETA {eta/60:.0f}分",
+                      flush=True)
+    print(f"\n完成:成功 {ok}  缺图 {miss}  → {args.out}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

crawler/scrape_douguo.py

@@ -0,0 +1,302 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""豆果美食 菜品图爬虫(每菜多图版,最高质量源)。
+
+走豆果 App JSON 接口 POST api.douguo.net/recipe/search/0/20(无需签名),每菜取前 N 个
+菜谱的 photo_path(960px 高清真实菜品图),下载到 images/,每菜一行 JSON 到 jsonl。
+与 scrape_xiachufang/scrape_bing 共用 dishes.jsonl + images/:续抓只跳 found:true。
+--replace: 下载前先删该菜已有的 <safe>_*.jpg(用于把旧 Bing 图替换成豆果图)。
+
+用法:
+  python scrape_douguo.py --out dishes.jsonl --download --per-dish 3 --workers 8 --replace --input todo.txt
+"""
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import os
+import random
+import re
+import sys
+import threading
+import time
+import urllib.parse
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timezone
+
+import requests
+
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception:
+        pass
+
+API = "https://api.douguo.net/recipe/search/0/20"
+
+UA_POOL = [
+    "Mozilla/5.0 (Linux; Android 9; MI 8 Build/PKQ1.181121.001) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.136 Mobile Safari/537.36",
+    "Mozilla/5.0 (Linux; Android 10; Redmi K30) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Mobile Safari/537.36",
+    "Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36",
+]
+
+
+def build_headers() -> dict:
+    return {"User-Agent": random.choice(UA_POOL),
+            "Content-Type": "application/x-www-form-urlencoded; charset=utf-8"}
+
+
+_tls = threading.local()
+
+
+def session() -> requests.Session:
+    s = getattr(_tls, "s", None)
+    if s is None:
+        s = _tls.s = requests.Session()
+    return s
+
+
+# ---------- 全局限速治理(同前:节流 + 全员冷却闸) ----------
+_pace_lock = threading.Lock()
+_next_search = 0.0
+_resume_at = 0.0
+_penalty = 0
+SEARCH_SPACING = 0.3  # 豆果接口快且耐用,默认间隔小
+
+
+def search_pace() -> None:
+    global _next_search
+    with _pace_lock:
+        now = time.time()
+        slot = max(now, _next_search)
+        _next_search = slot + SEARCH_SPACING + random.uniform(0, 0.2)
+        wait = slot - now
+    if wait > 0:
+        time.sleep(wait)
+
+
+def gate_wait() -> None:
+    while True:
+        with _pace_lock:
+            wait = _resume_at - time.time()
+        if wait <= 0:
+            return
+        time.sleep(min(wait, 3) + random.uniform(0, 0.4))
+
+
+def trip_limit() -> None:
+    global _resume_at, _penalty
+    with _pace_lock:
+        _penalty += 1
+        cool = min(45 * (1.5 ** (_penalty - 1)), 600)
+        nr = time.time() + cool
+        if nr > _resume_at:
+            _resume_at = nr
+            print(f"  [!] 限速#{_penalty} 全局冷却 {cool:.0f}s", file=sys.stderr, flush=True)
+
+
+def ease_limit() -> None:
+    global _penalty
+    if _penalty:
+        with _pace_lock:
+            _penalty = max(0, _penalty - 1)
+
+
+def fetch_search(dish: str, proxy: str | None, retries: int = 5) -> dict | None:
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    body = urllib.parse.urlencode({"client": 4, "keyword": dish, "order": 0, "_vs": 400}).encode()
+    for attempt in range(retries):
+        gate_wait()
+        search_pace()
+        try:
+            r = session().post(API, data=body, headers=build_headers(), proxies=proxies, timeout=20)
+            if r.status_code == 200:
+                ease_limit()
+                return r.json()
+            if r.status_code in (403, 429, 503):
+                trip_limit()
+            else:
+                print(f"  [!] 状态 {r.status_code}", file=sys.stderr)
+                time.sleep(3 * (attempt + 1))
+        except (requests.RequestException, ValueError) as e:
+            print(f"  [!] 请求异常 {type(e).__name__}", file=sys.stderr)
+            time.sleep(5 + random.uniform(0, 5))
+    return None
+
+
+def parse_images(data: dict | None, n: int) -> list[dict]:
+    recs = ((data or {}).get("result") or {}).get("recipes") or []
+    out: list[dict] = []
+    seen: set[str] = set()
+    for rc in recs:
+        url = rc.get("photo_path") or rc.get("thumb_path") or rc.get("image")
+        if not url:
+            continue
+        key = str(rc.get("cook_id") or url)
+        if key in seen:
+            continue
+        seen.add(key)
+        # .com 图床有 hotlink/限流(HTTP 493),.net 镜像同图可用,但 .net 证书不匹配→走 http
+        net_url = url.replace("douguo.com", "douguo.net").replace("https://", "http://")
+        out.append({
+            "image_url": net_url,
+            "src_com": url,
+            "title": (rc.get("title") or "").strip() or None,
+            "cook_id": rc.get("cook_id"),
+        })
+        if len(out) >= n:
+            break
+    return out
+
+
+def is_image(b: bytes) -> bool:
+    return (b[:3] == b"\xff\xd8\xff" or b[:8] == b"\x89PNG\r\n\x1a\n"
+            or b[:6] in (b"GIF87a", b"GIF89a")
+            or (b[:4] == b"RIFF" and b[8:12] == b"WEBP") or b[:2] == b"BM")
+
+
+def safe_name(dish: str) -> str:
+    return re.sub(r"[^\w一-鿿]+", "_", dish).strip("_") or "dish"
+
+
+def download(url: str, dish: str, idx: int, img_dir: str,
+             proxy: str | None, retries: int = 4) -> tuple[str | None, int]:
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    headers = {"User-Agent": random.choice(UA_POOL), "Referer": "https://www.douguo.com/",
+               "Accept": "image/avif,image/webp,image/png,image/*;q=0.8,*/*;q=0.5"}
+    for attempt in range(retries):
+        gate_wait()  # 图床被限(493/429)时也走全局冷却
+        try:
+            r = session().get(url, headers=headers, proxies=proxies, timeout=25)
+            if r.status_code == 200 and len(r.content) >= 800 and is_image(r.content):
+                ease_limit()
+                path = os.path.join(img_dir, f"{safe_name(dish)}_{idx:02d}.jpg")
+                with open(path, "wb") as f:
+                    f.write(r.content)
+                rel = os.path.relpath(path, os.path.dirname(img_dir) or ".").replace(os.sep, "/")
+                return rel, len(r.content)
+            if r.status_code in (403, 429, 493, 503):
+                trip_limit()
+            else:
+                time.sleep(1.5 * (attempt + 1))
+        except requests.RequestException as e:
+            print(f"  [!] 下图失败 {e}", file=sys.stderr)
+            time.sleep(2)
+    return None, 0
+
+
+def scrape_one(dish: str, proxy: str | None, do_download: bool, img_dir: str,
+               per_dish: int, jitter: float = 0.0, replace: bool = False) -> dict:
+    if jitter:
+        time.sleep(random.uniform(0, jitter))
+    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+    base = {"dish": dish, "source": "douguo", "fetched_at": now}
+    data = fetch_search(dish, proxy)
+    if data is None:
+        return {**base, "found": False, "count": 0, "images": [], "reason": "fetch_failed"}
+    recs = parse_images(data, per_dish)
+    if not recs:
+        return {**base, "found": False, "count": 0, "images": [], "reason": "no_result"}
+    if do_download and replace:  # 先删旧图(把该菜的 Bing 图替换掉)
+        for old in glob.glob(os.path.join(img_dir, safe_name(dish) + "_*.jpg")):
+            try:
+                os.remove(old)
+            except OSError:
+                pass
+    images = []
+    for i, rec in enumerate(recs, 1):
+        e = dict(rec)
+        if do_download:
+            rel, size = download(rec["image_url"], dish, i, img_dir, proxy)
+            e["image_file"] = rel
+            e["bytes"] = size
+            if i < len(recs):
+                time.sleep(random.uniform(0.05, 0.2))
+        images.append(e)
+    saved = sum(1 for e in images if not do_download or e.get("image_file"))
+    if do_download and saved == 0:
+        return {**base, "found": False, "count": 0, "images": images, "reason": "download_failed"}
+    return {**base, "found": True, "count": saved, "images": images}
+
+
+def load_done(out_path: str) -> set[str]:
+    done = set()
+    if os.path.exists(out_path):
+        with open(out_path, encoding="utf-8") as f:
+            for line in f:
+                try:
+                    rec = json.loads(line)
+                    if rec.get("found"):
+                        done.add(rec["dish"])
+                except Exception:
+                    pass
+    return done
+
+
+def main():
+    global SEARCH_SPACING
+    ap = argparse.ArgumentParser()
+    ap.add_argument("dishes", nargs="*")
+    ap.add_argument("--input")
+    ap.add_argument("--out", default="dishes.jsonl")
+    ap.add_argument("--download", action="store_true")
+    ap.add_argument("--img-dir", default="images")
+    ap.add_argument("--per-dish", type=int, default=3)
+    ap.add_argument("--proxy", default=os.environ.get("CRAWL_PROXY"))
+    ap.add_argument("--workers", type=int, default=8)
+    ap.add_argument("--jitter", type=float, default=0.8)
+    ap.add_argument("--spacing", type=float, default=SEARCH_SPACING)
+    ap.add_argument("--replace", action="store_true", help="下载前删该菜已有图(替换旧源)")
+    args = ap.parse_args()
+    SEARCH_SPACING = args.spacing
+
+    dishes = list(args.dishes)
+    if args.input:
+        with open(args.input, encoding="utf-8") as f:
+            dishes += [ln.strip() for ln in f if ln.strip()]
+    if not dishes:
+        ap.error("没给菜名")
+    dishes = list(dict.fromkeys(dishes))
+
+    os.makedirs(args.img_dir, exist_ok=True)
+    done = load_done(args.out)
+    todo = [d for d in dishes if d not in done]
+    print(f"共 {len(dishes)} 道,已完成 {len(done)},本次待抓 {len(todo)},并行 {args.workers} 线程,源 豆果",
+          flush=True)
+    if not todo:
+        print("没有待抓的菜。")
+        return
+
+    ok = miss = 0
+    t0 = time.time()
+    lock = threading.Lock()
+    with open(args.out, "a", encoding="utf-8") as fout, \
+            ThreadPoolExecutor(max_workers=args.workers) as ex:
+        futs = {ex.submit(scrape_one, d, args.proxy, args.download, args.img_dir,
+                          args.per_dish, args.jitter, args.replace): d for d in todo}
+        for i, fut in enumerate(as_completed(futs), 1):
+            dish = futs[fut]
+            try:
+                rec = fut.result()
+            except Exception as e:
+                now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+                rec = {"dish": dish, "source": "douguo", "found": False, "count": 0,
+                       "images": [], "reason": f"error:{type(e).__name__}", "fetched_at": now}
+            with lock:
+                fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
+                fout.flush()
+            if rec["found"]:
+                ok += 1
+            else:
+                miss += 1
+            if i % 25 == 0 or i == len(todo):
+                rate = i / max(time.time() - t0, 1e-6)
+                eta = (len(todo) - i) / rate if rate else 0
+                print(f"[{i}/{len(todo)}] ✓{ok} ✗{miss}  {rate*60:.0f}道/分  ETA {eta/60:.0f}分",
+                      flush=True)
+    print(f"\n完成:成功 {ok}  缺图 {miss}  → {args.out}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

crawler/scrape_xiachufang.py

@@ -0,0 +1,375 @@
+#!/usr/bin/env python3
+"""下厨房菜品图爬虫(每菜多图版)。
+
+给一批菜名,逐个去下厨房搜索,取**前 N 个 recipe 的首图**(真图在 data-src,不是 src,
+N 由 --per-dish 控制,默认 3 → 即同名菜的不同做法各一张),每道菜输出一行 JSON 到 jsonl,
+行内 images:[...] 记录该菜对应的所有图(文件名 + 原图 URL + 做法标题/链接)。
+
+反爬策略(用户要求"悠着点"):
+  - 每请求随机换 User-Agent + Accept-Language + 轻微变化的 header 组合
+  - 请求间随机延迟(默认 4~9s),每 N 条额外长歇一次
+  - 非 200 / 异常 → 指数退避重试;疑似被封(403/503/429)歇更久
+  - 不持久化 cookie(每次请求像"新访客",避免同 cookie 配不同 UA 的机器人特征)
+  - 代理可选(--proxy / 文件轮换);国内站默认直连
+
+断点续抓:已在输出 jsonl 里的菜名自动跳过。
+
+用法:
+  python3 scrape_xiachufang.py --out dishes.jsonl --download --per-dish 3 鸡尾酒 布朗尼 牛肉拉面
+  python3 scrape_xiachufang.py --out dishes.jsonl --download --per-dish 3 --input dishes.txt
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import re
+import sys
+import threading
+import time
+import urllib.parse
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timezone
+
+import requests
+
+# Windows 控制台默认 GBK,无法输出 ✓/✗/… 等字符 → 强制 UTF-8(长跑任务必须稳)
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception:
+        pass
+
+SEARCH_URL = "https://www.xiachufang.com/search/"
+
+# 真实浏览器 UA 池(桌面 + 移动混合),每请求随机取一条
+UA_POOL = [
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
+    "Mozilla/5.0 (iPhone; CPU iPhone OS 17_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Mobile/15E148 Safari/604.1",
+    "Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Mobile Safari/536.36",
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Edg/123.0.0.0",
+]
+ACCEPT_LANGS = [
+    "zh-CN,zh;q=0.9,en;q=0.8",
+    "zh-CN,zh;q=0.9",
+    "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3",
+]
+
+
+def build_headers(referer: str = "https://www.xiachufang.com/") -> dict:
+    """每次随机组装 header,制造"不同访客"特征。"""
+    h = {
+        "User-Agent": random.choice(UA_POOL),
+        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
+        "Accept-Language": random.choice(ACCEPT_LANGS),
+        "Referer": referer,
+        "Upgrade-Insecure-Requests": "1",
+    }
+    # 随机带或不带这些可选头,进一步打散指纹
+    if random.random() < 0.6:
+        h["DNT"] = "1"
+    if random.random() < 0.5:
+        h["Sec-Fetch-Dest"] = "document"
+        h["Sec-Fetch-Mode"] = "navigate"
+        h["Sec-Fetch-Site"] = "same-origin"
+    return h
+
+
+# 每线程独立 Session(连接复用更快,并行更稳;requests.Session 非线程安全→线程本地)
+_tls = threading.local()
+
+
+def session() -> requests.Session:
+    s = getattr(_tls, "s", None)
+    if s is None:
+        s = _tls.s = requests.Session()
+    return s
+
+
+# ---------- 全局限速治理(根治多线程退避雪崩) ----------
+# 1) 搜索节流:全局把"搜索请求"按最小间隔排队(限速的是搜索域名,不是图片CDN)
+# 2) 冷却闸:任一线程被 429/503 → 设全局冷却,所有线程一起等,避免罚时还有线程乱戳
+_pace_lock = threading.Lock()
+_next_search = 0.0          # 下一次允许搜索的时刻
+_resume_at = 0.0           # 全局冷却到期时刻
+_penalty = 0               # 连续被限速次数(自适应升级/消解)
+SEARCH_SPACING = 2.0       # 全局两次搜索最小间隔(秒),可被 --spacing 覆盖
+
+
+def search_pace() -> None:
+    """全局给搜索请求排队,保证两次搜索至少间隔 SEARCH_SPACING 秒。"""
+    global _next_search
+    with _pace_lock:
+        now = time.time()
+        slot = max(now, _next_search)
+        _next_search = slot + SEARCH_SPACING + random.uniform(0, 0.5)
+        wait = slot - now
+    if wait > 0:
+        time.sleep(wait)
+
+
+def gate_wait() -> None:
+    """被限速期间所有线程在此一起等到冷却结束。"""
+    while True:
+        with _pace_lock:
+            wait = _resume_at - time.time()
+        if wait <= 0:
+            return
+        time.sleep(min(wait, 3) + random.uniform(0, 0.4))
+
+
+def trip_limit() -> None:
+    """被 429/503 时触发全局冷却,连续触发自适应升级(60s→封顶600s)。"""
+    global _resume_at, _penalty
+    with _pace_lock:
+        _penalty += 1
+        cool = min(60 * (1.5 ** (_penalty - 1)), 600)
+        nr = time.time() + cool
+        if nr > _resume_at:
+            _resume_at = nr
+            print(f"  [!] 限速#{_penalty} 全局冷却 {cool:.0f}s", file=sys.stderr, flush=True)
+
+
+def ease_limit() -> None:
+    """成功一次就消解一点惩罚,持续顺利时自动恢复速度。"""
+    global _penalty
+    if _penalty:
+        with _pace_lock:
+            _penalty = max(0, _penalty - 1)
+
+
+def fetch(url: str, proxy: str | None, retries: int = 6, timeout: int = 25) -> str | None:
+    """GET 搜索页:全局节流 + 全局冷却闸 + 自适应重试。"""
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    for attempt in range(retries):
+        gate_wait()       # 冷却期一起等
+        search_pace()     # 全局排队,控制搜索速率
+        try:
+            resp = session().get(
+                url, headers=build_headers(), proxies=proxies, timeout=timeout
+            )
+            if resp.status_code == 200:
+                ease_limit()
+                return resp.text
+            if resp.status_code in (403, 429, 503):
+                trip_limit()  # 设全局冷却,下轮 gate_wait 一起等
+            else:
+                print(f"  [!] 状态 {resp.status_code}", file=sys.stderr)
+                time.sleep(5 * (attempt + 1))
+        except requests.RequestException as e:
+            print(f"  [!] 请求异常 {e}", file=sys.stderr)
+            time.sleep(8 + random.uniform(0, 5))
+    return None
+
+
+def normalize_img(url: str, width: int = 600) -> str:
+    """把 215px 缩略图参数换成更大尺寸。chuimg 走七牛 imageView2:
+    base.jpg?imageView2/2/w/600 = 等比缩到宽 600。"""
+    base = url.split("?", 1)[0]
+    return f"{base}?imageView2/2/w/{width}/interlace/1/q/85"
+
+
+def parse_top_recipes(html: str, n: int) -> list[dict]:
+    """从搜索结果页抽**前 n 个** recipe 的 {image_url, recipe_title, recipe_url}。
+    一个搜索页通常有 ~15 个 recipe,各带一张真图 → 即每菜多图(同名菜的不同做法)。
+    无结果 / 结构变化 → []。"""
+    # 先定位结果列表区,避免误抓页面装饰图(IE 提示图等)
+    start = html.find('class="normal-recipe-list"')
+    if start == -1:
+        return []
+    region = html[start:]
+
+    out: list[dict] = []
+    seen: set[str] = set()
+    # 逐个 recipe 块:<a href="/recipe/ID/">...内含 <img data-src=chuimg alt=标题>...</a>
+    for m in re.finditer(r'<a href="(/recipe/(\d+)/)"[^>]*>(.*?)</a>', region, re.S):
+        block = m.group(0)
+        img = re.search(r'data-src="([^"]+chuimg[^"]*)"', block)
+        if not img:
+            continue
+        rid = m.group(2)
+        if rid in seen:  # 同一 recipe 在页面可能出现两次,去重
+            continue
+        seen.add(rid)
+        image_raw = img.group(1)
+        alt = re.search(r'alt="([^"]*)"', block)
+        out.append({
+            "image_url": normalize_img(image_raw),
+            "image_url_raw": image_raw,
+            "recipe_title": (alt.group(1).strip() if alt else None),
+            "recipe_url": "https://www.xiachufang.com" + m.group(1),
+        })
+        if len(out) >= n:
+            break
+    return out
+
+
+def safe_name(dish: str) -> str:
+    return re.sub(r"[^\w一-鿿]+", "_", dish).strip("_") or "dish"
+
+
+def download_image(url: str, dish: str, idx: int, img_dir: str,
+                   proxy: str | None) -> tuple[str | None, int]:
+    """下图到 img_dir/<safe>_<idx>.jpg。chuimg 可能校验 Referer,带上。
+    返回 (相对路径, 字节数);失败 (None, 0)。"""
+    proxies = {"http": proxy, "https": proxy} if proxy else None
+    headers = build_headers()
+    headers["Referer"] = "https://www.xiachufang.com/"
+    headers["Accept"] = "image/avif,image/webp,image/png,image/*;q=0.8,*/*;q=0.5"
+    try:
+        r = session().get(url, headers=headers, proxies=proxies, timeout=25)
+        if r.status_code == 200 and r.content:
+            path = os.path.join(img_dir, f"{safe_name(dish)}_{idx:02d}.jpg")
+            with open(path, "wb") as f:
+                f.write(r.content)
+            rel = os.path.relpath(path, os.path.dirname(img_dir) or ".").replace(os.sep, "/")
+            return rel, len(r.content)
+    except requests.RequestException as e:
+        print(f"  [!] 下图失败 {e}", file=sys.stderr)
+    return None, 0
+
+
+def scrape_one(dish: str, proxy: str | None, do_download: bool, img_dir: str,
+               per_dish: int, jitter: float = 0.0) -> dict:
+    if jitter:  # 并行时每个任务起点随机错开,避免同时齐刷刷打过去
+        time.sleep(random.uniform(0, jitter))
+    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+    url = SEARCH_URL + "?" + urllib.parse.urlencode({"keyword": dish, "cat": 1001})
+    html = fetch(url, proxy)
+    if html is None:
+        return {"dish": dish, "found": False, "count": 0, "images": [],
+                "reason": "fetch_failed", "fetched_at": now}
+    recs = parse_top_recipes(html, per_dish)
+    if not recs:
+        return {"dish": dish, "found": False, "count": 0, "images": [],
+                "reason": "no_result", "fetched_at": now}
+
+    images = []
+    for i, rec in enumerate(recs, 1):
+        entry = dict(rec)
+        if do_download:
+            rel, size = download_image(rec["image_url"], dish, i, img_dir, proxy)
+            entry["image_file"] = rel
+            entry["bytes"] = size
+            if i < len(recs):  # 同菜多图之间轻微间隔,别猛打 CDN
+                time.sleep(random.uniform(0.1, 0.3))
+        images.append(entry)
+    saved = sum(1 for e in images if not do_download or e.get("image_file"))
+    return {"dish": dish, "found": True, "count": saved, "images": images,
+            "fetched_at": now}
+
+
+def load_done(out_path: str) -> set[str]:
+    done = set()
+    if os.path.exists(out_path):
+        with open(out_path, encoding="utf-8") as f:
+            for line in f:
+                try:
+                    rec = json.loads(line)
+                    if rec.get("found"):  # 只把成功的算"已完成";失败行会被重试
+                        done.add(rec["dish"])
+                except Exception:
+                    pass
+    return done
+
+
+def main():
+    global SEARCH_SPACING
+    ap = argparse.ArgumentParser()
+    ap.add_argument("dishes", nargs="*", help="菜名(也可用 --input 从文件读)")
+    ap.add_argument("--input", help="菜名文件,一行一道")
+    ap.add_argument("--out", default="dishes.jsonl", help="输出 jsonl")
+    ap.add_argument("--download", action="store_true", help="同时把图片下载到 images/")
+    ap.add_argument("--img-dir", default="images")
+    ap.add_argument("--per-dish", type=int, default=3, help="每道菜抓取的图片张数(取搜索前 N 个做法)")
+    ap.add_argument("--proxy", default=os.environ.get("CRAWL_PROXY"),
+                    help="如 http://127.0.0.1:7897;默认直连(国内站)")
+    ap.add_argument("--min-delay", type=float, default=4.0)
+    ap.add_argument("--max-delay", type=float, default=9.0)
+    ap.add_argument("--rest-every", type=int, default=40, help="(串行)每抓 N 条额外长歇")
+    ap.add_argument("--workers", type=int, default=1,
+                    help="并行线程数;>1 启用并行(自动去掉串行长延迟)")
+    ap.add_argument("--jitter", type=float, default=0.0,
+                    help="每任务起点随机错开秒数(并行未设时默认 1.5)")
+    ap.add_argument("--spacing", type=float, default=SEARCH_SPACING,
+                    help="全局两次搜索最小间隔秒(控制搜索速率,防限速;默认 2.0)")
+    args = ap.parse_args()
+
+    SEARCH_SPACING = args.spacing
+
+    dishes = list(args.dishes)
+    if args.input:
+        with open(args.input, encoding="utf-8") as f:
+            dishes += [ln.strip() for ln in f if ln.strip()]
+    if not dishes:
+        ap.error("没给菜名")
+    dishes = list(dict.fromkeys(dishes))  # 清单内去重(保序)
+
+    os.makedirs(args.img_dir, exist_ok=True)
+    done = load_done(args.out)
+    todo = [d for d in dishes if d not in done]
+    print(f"共 {len(dishes)} 道,已抓 {len(done)},本次待抓 {len(todo)},并行 {args.workers} 线程",
+          flush=True)
+    if not todo:
+        print("没有待抓的菜。")
+        return
+
+    ok = miss = 0
+    t0 = time.time()
+
+    if args.workers > 1:
+        jitter = args.jitter if args.jitter > 0 else 1.5
+        lock = threading.Lock()
+        with open(args.out, "a", encoding="utf-8") as fout, \
+                ThreadPoolExecutor(max_workers=args.workers) as ex:
+            futs = {ex.submit(scrape_one, d, args.proxy, args.download,
+                              args.img_dir, args.per_dish, jitter): d for d in todo}
+            for i, fut in enumerate(as_completed(futs), 1):
+                dish = futs[fut]
+                try:
+                    rec = fut.result()
+                except Exception as e:
+                    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
+                    rec = {"dish": dish, "found": False, "count": 0, "images": [],
+                           "reason": f"error:{type(e).__name__}", "fetched_at": now}
+                with lock:
+                    fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
+                    fout.flush()
+                if rec["found"]:
+                    ok += 1
+                else:
+                    miss += 1
+                if i % 20 == 0 or i == len(todo):
+                    rate = i / max(time.time() - t0, 1e-6)
+                    eta = (len(todo) - i) / rate if rate else 0
+                    print(f"[{i}/{len(todo)}] ✓{ok} ✗{miss}  "
+                          f"{rate*60:.0f}道/分  ETA {eta/60:.0f}分", flush=True)
+    else:
+        with open(args.out, "a", encoding="utf-8") as fout:
+            for i, dish in enumerate(todo, 1):
+                print(f"[{i}/{len(todo)}] {dish} ...", flush=True)
+                rec = scrape_one(dish, args.proxy, args.download, args.img_dir, args.per_dish)
+                fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
+                fout.flush()
+                if rec["found"]:
+                    ok += 1
+                    titles = "、".join(filter(None, (e.get("recipe_title") for e in rec["images"][:3])))
+                    print(f"    ✓ {rec['count']} 张  {titles}")
+                else:
+                    miss += 1
+                    print(f"    ✗ {rec['reason']}")
+                if i < len(todo):
+                    delay = random.uniform(args.min_delay, args.max_delay)
+                    if i % args.rest_every == 0:
+                        delay += random.uniform(20, 40)
+                        print(f"    …长歇 {delay:.0f}s")
+                    time.sleep(delay)
+    print(f"\n完成:成功 {ok}  缺图 {miss}  → {args.out}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

crawler/verify_repair.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""校验 dishes.jsonl 里记录的图是否都在盘上且为有效 JPEG;缺失/损坏的按记录的
+image_url 重新下载(应对杀软清图 / 下载失败)。
+
+  python verify_repair.py                 # 只报告(dry-run)
+  python verify_repair.py --repair        # 重下缺失/损坏的图
+"""
+import argparse, json, os, sys, time, random
+import requests
+
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception:
+        pass
+
+UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+
+
+def valid_jpeg(path: str) -> bool:
+    """接受常见图片格式(jpg/png/gif/webp/bmp);过小多半是错误页/截断。"""
+    try:
+        if os.path.getsize(path) < 800:
+            return False
+        with open(path, "rb") as f:
+            b = f.read(12)
+        return (b[:3] == b"\xff\xd8\xff" or b[:8] == b"\x89PNG\r\n\x1a\n"
+                or b[:6] in (b"GIF87a", b"GIF89a")
+                or (b[:4] == b"RIFF" and b[8:12] == b"WEBP") or b[:2] == b"BM")
+    except OSError:
+        return False
+
+
+def redownload(url: str, path: str) -> bool:
+    h = {"User-Agent": UA, "Referer": "https://www.xiachufang.com/",
+         "Accept": "image/avif,image/webp,image/png,image/*;q=0.8,*/*;q=0.5"}
+    try:
+        r = requests.get(url, headers=h, timeout=25)
+        if r.status_code == 200 and r.content:
+            os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
+            with open(path, "wb") as f:
+                f.write(r.content)
+            return valid_jpeg(path)
+    except requests.RequestException as e:
+        print(f"  [!] 重下失败 {e}", file=sys.stderr)
+    return False
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--jsonl", default="dishes.jsonl")
+    ap.add_argument("--repair", action="store_true", help="重下缺失/损坏的图")
+    args = ap.parse_args()
+
+    dishes = found = no_result = 0
+    total = present = missing = corrupt = repaired = failed = 0
+    bad_dishes = []
+
+    with open(args.jsonl, encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            rec = json.loads(line)
+            dishes += 1
+            if not rec.get("found"):
+                no_result += 1
+                bad_dishes.append(rec["dish"])
+                continue
+            found += 1
+            for e in rec.get("images", []):
+                fp = e.get("image_file")
+                if not fp:
+                    continue
+                total += 1
+                ok = valid_jpeg(fp)
+                if ok:
+                    present += 1
+                    continue
+                if os.path.exists(fp):
+                    corrupt += 1
+                else:
+                    missing += 1
+                if args.repair and e.get("image_url"):
+                    if redownload(e["image_url"], fp):
+                        repaired += 1
+                    else:
+                        failed += 1
+                    time.sleep(random.uniform(0.1, 0.3))
+
+    print(f"菜条目: {dishes}  (有图 {found} / 无结果 {no_result})")
+    print(f"图片记录: {total}  在盘有效: {present}  缺失: {missing}  损坏: {corrupt}")
+    if args.repair:
+        print(f"重下成功: {repaired}  仍失败: {failed}")
+    if no_result:
+        print(f"无搜索结果的菜({no_result}): {'、'.join(bad_dishes[:40])}{' …' if no_result>40 else ''}")
+    health = present + (repaired if args.repair else 0)
+    print(f"\n图片完好率: {health}/{total} = {100*health/total:.1f}%" if total else "无图片记录")
+
+
+if __name__ == "__main__":
+    main()

retrieval/README.md

@@ -0,0 +1,35 @@
+# retrieval — 图片名称检索平台
+
+基于**图片名称**的三路检索并排对比,给对接业务做检索选型测试。
+
+- **① BM25** —— jieba 分词 + rank_bm25(认字面/关键词)
+- **② 向量** —— 本地 `BAAI/bge-m3`(1024 维,认语义;有 CUDA 走 GPU)
+- **③ RRF 融合** —— 按排名融合两路:`Σ 1/(k + rank)`,k 默认 60(只看名次,绕开分数量纲不可比)
+
+> 语义查询是向量的强项:查"碳酸饮料"能召回"可乐/雪碧"(字面零重叠),BM25 则无结果。
+
+## 跑
+```
+run.bat            # 或 pip install -r requirements.txt && python server.py
+```
+- **图片目录**:默认读 **`<仓库>/crawler/images`**;想指别处 → 启动前 `set IMAGE_DIR=<绝对路径>`。
+- 启动即扫描该目录、自动建索引;**首次**下 BGE-M3 ~2.3G(走 hf-mirror 镜像),之后约 25s 加载。
+- 绑 `0.0.0.0`,控制台打印**局域网地址**;同事同网打开直接搜,无需选文件夹。防火墙放行 8799 入站。
+
+## 接口
+- `GET /api/status` — 建索引进度(`phase`: bm25/loadmodel/embedding、`device`、图片/名称数)
+- `GET /api/search?q=&topk=&rrf_k=&w_bm25=&w_vec=` — 返回 `{bm25[], vector[], fusion[]}`,每条含图片文件名 + 在另两路的命中名次
+- `GET /images/<文件名>` — 服务器直出图片(局域网里显示图靠它)
+
+## 文件
+| 文件 | 作用 |
+| --- | --- |
+| `server.py` | FastAPI:扫描目录 / 启动自动建索引 / status / search + serve 图片与前端 |
+| `embed_local.py` | **本地 BGE-M3 向量**(GPU 自适应、落盘缓存;内含 HF 镜像/代理/torch 版本等环境坑处理) |
+| `embed.py` | (备用)千问 DashScope 云端向量,含直连/自适应限速 |
+| `bm25.py` / `fuse.py` | 中文 BM25(jieba + 补单字) / RRF 融合 |
+| `static/` | 前端单页(纯搜索页,轮询 status 显示建索引进度) |
+
+## 依赖 / 环境
+`sentence-transformers`(+ torch)。检测到 CUDA 自动用 GPU;向量缓存落 `cache/`,模型落 `hf_home/`(均不进 git)。
+`embed_local.py` 已处理:走 hf-mirror 镜像、绕系统代理直连、`HF_HUB_DISABLE_XET=1`、绕过 transformers 对 torch<2.6 加载 .bin 的限制。

retrieval/bm25.py

@@ -0,0 +1,37 @@
+# -*- coding: utf-8 -*-
+"""中文 BM25 检索(jieba 分词 + rank_bm25)。文档 = 菜名。
+
+菜名通常很短(2~6 字),除 jieba 词以外再补单字,提升短词召回。
+"""
+from __future__ import annotations
+
+import re
+
+import jieba
+import numpy as np
+from rank_bm25 import BM25Okapi
+
+jieba.initialize()
+
+
+def tokenize(text: str) -> list[str]:
+    text = text.replace("_", " ").strip()
+    words = [t for t in jieba.lcut_for_search(text) if t.strip()]
+    chars = [c for c in re.sub(r"\s+", "", text) if "一" <= c <= "鿿"]
+    toks = words + chars
+    return toks if toks else [t for t in re.sub(r"\s+", "", text)] or ["_"]
+
+
+class BM25Index:
+    def __init__(self, names: list[str]):
+        self.names = list(names)
+        self.tokenized = [tokenize(n) for n in self.names]
+        self.bm25 = BM25Okapi(self.tokenized)
+
+    def search(self, query: str, topk: int = 30) -> list[tuple[str, float]]:
+        q = tokenize(query)
+        if not q:
+            return []
+        scores = self.bm25.get_scores(q)
+        order = np.argsort(scores)[::-1][:topk]
+        return [(self.names[i], float(scores[i])) for i in order if scores[i] > 0]

retrieval/embed.py

@@ -0,0 +1,231 @@
+# -*- coding: utf-8 -*-
+"""通义千问 DashScope 文本向量客户端(text-embedding-v3,1024 维)。
+
+要点:
+- Key 复用 PriceBot 的 QWEN_API_KEY;可用环境变量 DASHSCOPE_API_KEY / QWEN_API_KEY 覆盖。
+- 接口走 OpenAI 兼容端点;单次批量上限 10(硬限),用线程池并行多批。
+- name -> 向量 落盘缓存(cache/emb_<model>.jsonl):重复加载同一批菜名零成本、零计费。
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+import numpy as np
+import requests
+
+DASHSCOPE_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings"
+# 默认复用 PriceBot 的通义千问 Key(本工具在 dish-images 下、不进 git);优先用环境变量覆盖。
+DEFAULT_KEY = "sk-f66e689a9b0c43239e299137f68c453c"
+BATCH = 10          # text-embedding-v3 单次最多 10 条
+# 免费额度对"并发齐发"很敏感(冷却后多线程同时重试会互撞、永远挤不进窗口),默认单线程顺序发,
+# 由下方自适应间隔 _delay 控速最稳。Key 额度充足时可设环境变量 EMBED_WORKERS=4 之类提速。
+MAX_WORKERS = max(1, int(os.environ.get("EMBED_WORKERS", "1")))
+
+
+def get_key(override: str | None = None) -> str:
+    return (override or os.environ.get("DASHSCOPE_API_KEY")
+            or os.environ.get("QWEN_API_KEY") or DEFAULT_KEY)
+
+
+_tls = threading.local()
+
+
+def _sess() -> requests.Session:
+    """线程本地 session,trust_env=False:忽略系统代理(HTTP(S)_PROXY),直连 dashscope。
+    Clash 等代理偶尔把请求甩到海外节点,会被阿里云按来源风控拒(HTTP 400 Access denied);
+    dashscope.aliyuncs.com 是大陆公网端点,直连最稳(与 PriceBot 调千问一致)。"""
+    s = getattr(_tls, "s", None)
+    if s is None:
+        s = _tls.s = requests.Session()
+        s.trust_env = False
+    return s
+
+
+# ---------- 全局自适应限速:稳态间隔 + 冷却闸,收敛到免费额度可持续的速率 ----------
+# dashscope 免费额度是「滚动每分钟预算」式:瞬时小量能过,持续高频会返回 400,且文案写成
+# Arrearage/overdue(其实是"免费窗口用尽且无余额续费",约 1 分钟自动回血)。故这是【可重试】
+# 信号:命中就放大间隔(降速)+ 拉长全员冷却(等回血),成功就缓慢提速,自动逼近可持续速率。
+_pace_lock = threading.Lock()
+_next_at = 0.0
+_resume_at = 0.0
+_delay = 0.6                    # 当前稳态请求间隔(秒),单线程顺序发
+_cool = 5.0                     # 当前冷却时长(秒),命中放大、成功回落
+DELAY_MIN, DELAY_MAX = 0.3, 4.0
+COOL_MIN, COOL_MAX = 5.0, 45.0
+
+
+def _pace() -> None:
+    """按当前稳态间隔 _delay 给每个请求排队发牌(有效速率 ≈ 1/_delay)。"""
+    global _next_at
+    with _pace_lock:
+        now = time.time()
+        slot = max(now, _next_at)
+        _next_at = slot + _delay
+        wait = slot - now
+    if wait > 0:
+        time.sleep(wait)
+
+
+def _gate_wait() -> None:
+    """命中限速后的全员冷却,期间所有线程一起等(等额度回血)。"""
+    while True:
+        with _pace_lock:
+            wait = _resume_at - time.time()
+        if wait <= 0:
+            return
+        time.sleep(min(wait, 2.0) + 0.05)
+
+
+def _rate_hit() -> None:
+    """命中免费额度限速:降速 + 拉长冷却等回血。"""
+    global _delay, _cool, _resume_at
+    with _pace_lock:
+        _delay = min(_delay * 1.5, DELAY_MAX)
+        _cool = min(_cool * 1.4, COOL_MAX)
+        _resume_at = max(_resume_at, time.time() + _cool)
+
+
+def _ok() -> None:
+    """成功:缓慢提速、缩短冷却,收敛到刚好不撞墙。"""
+    global _delay, _cool
+    with _pace_lock:
+        _delay = max(_delay * 0.9, DELAY_MIN)
+        _cool = max(_cool * 0.9, COOL_MIN)
+
+
+class ArrearsError(RuntimeError):
+    """重试多轮仍被免费额度拒(疑似额度耗尽/欠费),供上层提示"稍后重试/换 Key"。"""
+
+
+def _is_rate_signal(status: int, text: str) -> bool:
+    """免费额度限速信号(可重试):含真限速(429/503/throttling)与"伪欠费"(Arrearage/overdue,
+    实为免费窗口用尽、约 1 分钟回血)。两者都靠退避+等待解决。"""
+    if status in (429, 503):
+        return True
+    low = text.lower()
+    return ("throttling" in low or "rate limit" in low or "requests rate" in low
+            or "arrearage" in low or "overdue" in low or "good standing" in low
+            or "欠费" in text or "余额不足" in text)
+
+
+class Embedder:
+    def __init__(self, model: str = "text-embedding-v4", cache_dir: str = "cache",
+                 key: str | None = None):
+        self.model = model
+        self.key = get_key(key)
+        os.makedirs(cache_dir, exist_ok=True)
+        self.cache_path = os.path.join(cache_dir, f"emb_{model}.jsonl")
+        self.cache: dict[str, list[float]] = {}
+        self.last_failed = 0
+        self.last_error = ""
+        self._lock = threading.Lock()
+        self._load_cache()
+
+    # ---------- 缓存 ----------
+    def _load_cache(self) -> None:
+        if not os.path.exists(self.cache_path):
+            return
+        with open(self.cache_path, encoding="utf-8") as f:
+            for line in f:
+                try:
+                    o = json.loads(line)
+                    self.cache[o["t"]] = o["v"]
+                except Exception:
+                    pass
+
+    def _append_cache(self, items: list[tuple[str, list[float]]]) -> None:
+        with self._lock:
+            with open(self.cache_path, "a", encoding="utf-8") as f:
+                for t, v in items:
+                    f.write(json.dumps({"t": t, "v": v}, ensure_ascii=False) + "\n")
+                    self.cache[t] = v
+
+    # ---------- 接口 ----------
+    def _embed_batch(self, texts: list[str]) -> list[list[float]]:
+        body = {"model": self.model, "input": texts, "encoding_format": "float"}
+        last = None
+        rate_hits = 0
+        for _ in range(7):           # 配合冷却升级,总等待可超 1 分钟,足以熬过免费窗口回血
+            _gate_wait()
+            _pace()
+            try:
+                r = _sess().post(
+                    DASHSCOPE_URL,
+                    headers={"Authorization": f"Bearer {self.key}",
+                             "Content-Type": "application/json"},
+                    json=body, timeout=30)
+                if r.status_code == 200:
+                    _ok()
+                    data = sorted(r.json()["data"], key=lambda d: d["index"])
+                    return [d["embedding"] for d in data]
+                txt = r.text[:200]
+                last = f"HTTP {r.status_code}: {txt}"
+                if _is_rate_signal(r.status_code, txt):
+                    rate_hits += 1
+                    _rate_hit()          # 降速 + 拉长冷却等回血后重试
+                else:
+                    time.sleep(1.0)      # 其它错误轻退避后重试
+            except requests.RequestException as e:
+                last = f"{type(e).__name__}: {e}"
+                time.sleep(2)
+        if rate_hits >= 4:               # 多轮仍被额度拒 → 让上层提示换Key/稍后重试
+            raise ArrearsError(f"模型 {self.model} 免费额度反复受限(可能已用尽);"
+                               f"可稍后重试,或在 embed.py 换 DASHSCOPE_API_KEY")
+        raise RuntimeError(f"embed batch failed: {last}")
+
+    def embed_corpus(self, names: list[str], progress=None) -> None:
+        """把 names 全部编码进缓存。progress(done, total) 回调可选。失败的批跳过(不致命)。"""
+        todo = [n for n in dict.fromkeys(names) if n not in self.cache]
+        total = len(todo)
+        self.last_failed = 0
+        self.last_error = ""
+        if progress:
+            progress(0, total)
+        if not total:
+            return
+        batches = [todo[i:i + BATCH] for i in range(0, total, BATCH)]
+        done = 0
+        ok_count = 0
+        with ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
+            futs = {ex.submit(self._embed_batch, b): b for b in batches}
+            for fut in as_completed(futs):
+                b = futs[fut]
+                try:
+                    self._append_cache(list(zip(b, fut.result())))
+                    ok_count += len(b)
+                except Exception as e:  # noqa: BLE001
+                    self.last_failed += len(b)
+                    self.last_error = str(e)[:200]
+                    if self.last_failed <= len(b) * 2:   # 只打前一两次,避免刷屏
+                        print(f"[embed] 批失败({self.last_failed}): {e}", file=sys.stderr, flush=True)
+                    # 一次都没成功 + 已连续失败 ≥2 批 → 判定 Key 不可用,中止省额度;
+                    # 若前面已有成功(只是滚动额度临时受限),则继续磨,失败的批留待下次续抓。
+                    if ok_count == 0 and self.last_failed >= 20:
+                        print("[embed] 持续失败且零成功,中止向量编码(BM25 仍可用)",
+                              file=sys.stderr, flush=True)
+                        ex.shutdown(wait=False, cancel_futures=True)
+                        break
+                done += len(b)
+                if progress:
+                    progress(done, total)
+
+    # ---------- 取用 ----------
+    def matrix(self, names: list[str]) -> tuple[list[str], np.ndarray]:
+        """返回 (有向量的 name 列表, L2 归一化矩阵 N×dim)。"""
+        keep = [n for n in names if n in self.cache]
+        if not keep:
+            return [], np.zeros((0, 0), dtype=np.float32)
+        m = np.array([self.cache[n] for n in keep], dtype=np.float32)
+        norm = np.linalg.norm(m, axis=1, keepdims=True)
+        norm[norm == 0] = 1.0
+        return keep, m / norm
+
+    def embed_query(self, q: str) -> np.ndarray:
+        v = np.array(self._embed_batch([q])[0], dtype=np.float32)
+        n = np.linalg.norm(v)
+        return v / (n if n else 1.0)

retrieval/embed_local.py

@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+"""本地 BGE-M3 向量(BAAI/bge-m3,1024 维 dense,多语言/中文强,离线无限额)。
+
+与 cloud 版 embed.py 同接口(embed_corpus / matrix / embed_query / last_failed / last_error),
+server.py 直接换用。首次会下载约 2.3G 权重(默认走 hf-mirror.com 镜像,国内快);
+检测到 CUDA 自动用 GPU。无任何调用配额/限速问题。
+"""
+from __future__ import annotations
+
+import json
+import os
+import sys
+import threading
+
+import numpy as np
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+# 默认缓存在 C 盘 .cache,但本机 C: 空间不足 → 放到项目所在盘(bge-m3 约 2.3G)。
+# 想放别处:启动前设环境变量 HF_HOME 覆盖即可。
+os.environ.setdefault("HF_HOME", os.path.join(_HERE, "hf_home"))
+# 国内默认走 HF 镜像(已设则不覆盖),下载更稳;并阻止 transformers 误加载 TensorFlow。
+os.environ.setdefault("HF_ENDPOINT", "https://hf-mirror.com")
+# 关掉 hf-xet 传输:它绕过 HF_ENDPOINT 直连 huggingface.co 的 xet 服务器会超时。
+os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
+os.environ.setdefault("USE_TF", "0")
+os.environ.setdefault("TF_CPP_MIN_LOG_LEVEL", "3")
+# 系统代理(Clash 7897)会解压 zstd 并改写 hf-mirror 的 etag,破坏 hf_hub 元数据校验
+# (报"couldn't connect to hf-mirror.com")。hf-mirror 是国内站,直连即可 → 清掉本进程代理。
+for _k in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy", "ALL_PROXY", "all_proxy"):
+    os.environ.pop(_k, None)
+os.environ.setdefault("NO_PROXY", "*")
+
+MODEL_ID = os.environ.get("BGE_MODEL", "BAAI/bge-m3")
+
+
+class LocalEmbedder:
+    def __init__(self, model: str = MODEL_ID, cache_dir: str = "cache",
+                 device: str | None = None):
+        self.model_id = model
+        tag = model.replace("/", "_")
+        os.makedirs(cache_dir, exist_ok=True)
+        self.cache_path = os.path.join(cache_dir, f"emb_{tag}.jsonl")
+        self.cache: dict[str, list[float]] = {}
+        self.last_failed = 0
+        self.last_error = ""
+        self.device = device
+        self._model = None
+        self._lock = threading.Lock()
+        self._load_cache()
+
+    # ---------- 缓存(与 cloud 版一致) ----------
+    def _load_cache(self) -> None:
+        if not os.path.exists(self.cache_path):
+            return
+        with open(self.cache_path, encoding="utf-8") as f:
+            for line in f:
+                try:
+                    o = json.loads(line)
+                    self.cache[o["t"]] = o["v"]
+                except Exception:
+                    pass
+
+    def _append_cache(self, items) -> None:
+        with self._lock:
+            with open(self.cache_path, "a", encoding="utf-8") as f:
+                for t, v in items:
+                    f.write(json.dumps({"t": t, "v": v}, ensure_ascii=False) + "\n")
+                    self.cache[t] = v
+
+    # ---------- 模型 ----------
+    def load_model(self) -> str:
+        """加载模型(首次会下载权重),返回 device 字符串。"""
+        if self._model is None:
+            import torch
+            # bge-m3 仅发布 pytorch_model.bin(无 safetensors);transformers 5.x 禁止
+            # torch<2.6 加载 .bin(CVE-2025-32434,防不可信 pickle 执行代码)。本机 torch 2.5.1,
+            # 模型来自 BAAI 官方(可信)、本地离线加载 → 关掉该检查,免升级 torch / 免下 safetensors。
+            try:
+                import transformers.modeling_utils as _mu
+                _mu.check_torch_load_is_safe = lambda *a, **k: None
+            except Exception:
+                pass
+            from sentence_transformers import SentenceTransformer
+            dev = self.device or ("cuda" if torch.cuda.is_available() else "cpu")
+            self._model = SentenceTransformer(self.model_id, device=dev)
+            self.device = dev
+        return self.device
+
+    def _encode(self, texts: list[str]) -> np.ndarray:
+        return np.asarray(
+            self._model.encode(texts, normalize_embeddings=True, batch_size=64,
+                               convert_to_numpy=True, show_progress_bar=False),
+            dtype=np.float32)
+
+    # ---------- 编码 ----------
+    def embed_corpus(self, names, progress=None) -> None:
+        """把 names 全部编码进缓存。progress(done, total) 回调可选。"""
+        todo = [n for n in dict.fromkeys(names) if n not in self.cache]
+        total = len(todo)
+        self.last_failed = 0
+        self.last_error = ""
+        if progress:
+            progress(0, total)
+        if not total:
+            return
+        try:
+            self.load_model()
+        except Exception as e:  # noqa: BLE001
+            self.last_failed = total
+            self.last_error = f"加载模型失败: {type(e).__name__}: {e}"
+            print(f"[embed_local] {self.last_error}", file=sys.stderr, flush=True)
+            return
+        step = 256
+        done = 0
+        for i in range(0, total, step):
+            chunk = todo[i:i + step]
+            try:
+                vecs = self._encode(chunk)
+                self._append_cache([(t, v.tolist()) for t, v in zip(chunk, vecs)])
+            except Exception as e:  # noqa: BLE001
+                self.last_failed += len(chunk)
+                self.last_error = str(e)[:200]
+                print(f"[embed_local] 编码失败: {e}", file=sys.stderr, flush=True)
+            done += len(chunk)
+            if progress:
+                progress(done, total)
+
+    # ---------- 取用(与 cloud 版一致) ----------
+    def matrix(self, names) -> tuple[list[str], np.ndarray]:
+        keep = [n for n in names if n in self.cache]
+        if not keep:
+            return [], np.zeros((0, 0), dtype=np.float32)
+        m = np.array([self.cache[n] for n in keep], dtype=np.float32)
+        norm = np.linalg.norm(m, axis=1, keepdims=True)
+        norm[norm == 0] = 1.0
+        return keep, m / norm
+
+    def embed_query(self, q: str) -> np.ndarray:
+        self.load_model()
+        v = self._encode([q])[0]
+        n = np.linalg.norm(v)
+        return v / (n if n else 1.0)

retrieval/fuse.py

@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+"""RRF(Reciprocal Rank Fusion)融合多路排名。
+
+RRF 只看「排名」不看「分数量纲」,天然解决 BM25 分数与余弦相似度不可比的问题:
+    fused(d) = Σ_method  w_method * 1 / (k + rank_method(d))
+k 越大,头部名次的权重差异越平缓(经验默认 60)。
+"""
+from __future__ import annotations
+
+
+def rrf(rankings: dict[str, list[str]], weights: dict[str, float],
+        k: int = 60) -> list[tuple[str, float]]:
+    """rankings: {method: [name 按相关性降序]};返回 [(name, fused_score)] 降序。"""
+    score: dict[str, float] = {}
+    for method, names in rankings.items():
+        w = weights.get(method, 1.0)
+        for rank, name in enumerate(names, start=1):
+            score[name] = score.get(name, 0.0) + w * (1.0 / (k + rank))
+    return sorted(score.items(), key=lambda kv: kv[1], reverse=True)

retrieval/requirements.txt

@@ -0,0 +1,9 @@
+fastapi>=0.110
+uvicorn[standard]>=0.27
+jieba>=0.42.1
+rank-bm25>=0.2.2
+numpy>=1.24
+
+# 本地向量 BGE-M3。依赖 torch:本机已装 CUDA 版(2.5.1+cu121),勿被覆盖。
+# 全新机器装 GPU 版 torch 见 https://pytorch.org(CPU 版也能跑,只是慢些)。
+sentence-transformers>=2.7

retrieval/run.bat

@@ -0,0 +1,8 @@
+@echo off
+chcp 65001 >nul
+cd /d %~dp0
+set PYTHONUTF8=1
+echo [1/2] 安装依赖(已装会秒过)...
+python -m pip install -q -r requirements.txt
+echo [2/2] 启动服务,浏览器打开 http://127.0.0.1:8799
+python server.py

retrieval/server.py

@@ -0,0 +1,262 @@
+# -*- coding: utf-8 -*-
+"""图片名称检索测试台 —— 后端(FastAPI)。
+
+三路检索,全部基于「图片名称」:
+  1) BM25     —— 关键词匹配(jieba 分词)
+  2) 向量     —— 本地 BAAI/bge-m3 余弦相似度
+  3) 融合     —— RRF 合并上面两路排名
+
+【写死目录 + 局域网模式】图片目录写死在 IMAGE_DIR,服务器启动即扫描并自动建索引、
+自己通过 /images 把图片发出去。绑 0.0.0.0,mentor 用局域网 IP 打开即可直接检索,
+无需选文件夹、无需建索引。
+
+跑法:python server.py  → 控制台会打印可发给 mentor 的局域网地址。
+"""
+from __future__ import annotations
+
+import os
+import re
+import socket
+import sys
+import threading
+
+import numpy as np
+from fastapi import Body, FastAPI, Query
+from fastapi.responses import JSONResponse
+from fastapi.staticfiles import StaticFiles
+
+from bm25 import BM25Index
+from embed_local import LocalEmbedder
+from fuse import rrf
+
+for _s in (sys.stdout, sys.stderr):
+    try:
+        _s.reconfigure(encoding="utf-8", errors="replace")
+    except Exception:
+        pass
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+# 图片目录:默认 = 本仓库的 crawler/images(爬虫默认就把图下到那里)。
+# 想指向别处已爬好的图,启动前设环境变量 IMAGE_DIR=<绝对路径> 覆盖即可,无需改代码。
+IMAGE_DIR = os.environ.get("IMAGE_DIR") or os.path.abspath(
+    os.path.join(HERE, os.pardir, "crawler", "images"))
+IMG_EXT = (".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp")
+PORT = 8799
+
+app = FastAPI(title="图片名称检索测试台")
+
+
+def scan_images(img_dir: str) -> dict[str, list[str]]:
+    """扫描图片目录 → {菜名: [文件名,...]}。去掉尾部 _NN 把同名多图归并为一条。"""
+    name2files: dict[str, list[str]] = {}
+    if not os.path.isdir(img_dir):
+        return name2files
+    for fn in os.listdir(img_dir):
+        if not fn.lower().endswith(IMG_EXT):
+            continue
+        stem = os.path.splitext(fn)[0]
+        name = re.sub(r"_\d+$", "", stem) or stem
+        name2files.setdefault(name, []).append(fn)
+    for files in name2files.values():
+        files.sort()
+    return name2files
+
+
+class State:
+    """全局索引状态(单实例工具,够用)。"""
+
+    def __init__(self):
+        self.lock = threading.Lock()
+        self.names: list[str] = []
+        self.name2files: dict[str, list[str]] = {}
+        self.bm25: BM25Index | None = None
+        self.embedder: LocalEmbedder | None = None
+        self.model = "BAAI/bge-m3"
+        self.vec_names: list[str] = []
+        self.vec_matrix: np.ndarray | None = None
+        self.state = "idle"      # idle | building | ready | error
+        self.phase = ""          # bm25 | loadmodel | embedding | done
+        self.emb_done = 0
+        self.emb_total = 0
+        self.failed = 0
+        self.message = ""
+        self.device = ""
+
+
+ST = State()
+
+
+def build_index(names: list[str], model: str) -> None:
+    try:
+        with ST.lock:
+            ST.state, ST.phase, ST.message = "building", "bm25", ""
+            ST.names, ST.model = names, model
+            ST.bm25 = None
+            ST.vec_matrix, ST.vec_names = None, []
+            ST.emb_done = ST.emb_total = ST.failed = 0
+
+        # 1) BM25(秒级)
+        bm = BM25Index(names)
+        with ST.lock:
+            ST.bm25, ST.phase = bm, "loadmodel"
+
+        # 2) 本地 BGE-M3:先加载模型(首次会下载约 2.3G),再编码(有 CUDA 走 GPU)
+        emb = LocalEmbedder(model=model, cache_dir=os.path.join(HERE, "cache"))
+        dev = emb.load_model()
+        with ST.lock:
+            ST.device, ST.phase = dev, "embedding"
+
+        def prog(done: int, total: int) -> None:
+            with ST.lock:
+                ST.emb_done, ST.emb_total = done, total
+
+        emb.embed_corpus(names, progress=prog)
+        vnames, mat = emb.matrix(names)
+        with ST.lock:
+            ST.embedder, ST.vec_names, ST.vec_matrix = emb, vnames, mat
+            ST.failed = emb.last_failed
+            ST.message = "" if vnames else ("向量未生成:" + emb.last_error if emb.last_error else "")
+            ST.state, ST.phase = "ready", "done"
+    except Exception as e:  # noqa: BLE001
+        with ST.lock:
+            ST.state, ST.message = "error", f"{type(e).__name__}: {e}"
+
+
+@app.on_event("startup")
+def _startup_autobuild() -> None:
+    """启动即扫描写死目录并后台建索引,mentor 打开就能搜。"""
+    n2f = scan_images(IMAGE_DIR)
+    with ST.lock:
+        ST.name2files = n2f
+    names = sorted(n2f.keys())
+    if not names:
+        with ST.lock:
+            ST.state = "error"
+            ST.message = f"图片目录为空或不存在:{IMAGE_DIR}"
+        return
+    threading.Thread(target=build_index, args=(names, "BAAI/bge-m3"), daemon=True).start()
+
+
+@app.post("/api/index")
+def api_index(payload: dict = Body(...)):
+    """(保留)手动指定名称重建索引;写死目录模式下一般用不到。"""
+    names = [str(n).strip() for n in payload.get("names", []) if str(n).strip()]
+    names = list(dict.fromkeys(names))
+    model = (payload.get("model") or "BAAI/bge-m3").strip()
+    if not names:
+        return JSONResponse({"ok": False, "error": "没有菜名"}, status_code=400)
+    if ST.state == "building":
+        return JSONResponse({"ok": False, "error": "正在建索引,请稍候"}, status_code=409)
+    threading.Thread(target=build_index, args=(names, model), daemon=True).start()
+    return {"ok": True, "total": len(names), "model": model}
+
+
+@app.get("/api/status")
+def api_status():
+    with ST.lock:
+        return {
+            "state": ST.state,
+            "phase": ST.phase,
+            "bm25_ready": ST.bm25 is not None,
+            "vec_ready": ST.vec_matrix is not None and len(ST.vec_names) > 0,
+            "total": len(ST.names),
+            "n_images": sum(len(v) for v in ST.name2files.values()),
+            "emb_done": ST.emb_done,
+            "emb_total": ST.emb_total,
+            "vec_count": len(ST.vec_names),
+            "failed": ST.failed,
+            "model": ST.model,
+            "device": ST.device,
+            "message": ST.message,
+        }
+
+
+def _bm25_search(q: str, topk: int) -> list[tuple[str, float]]:
+    return ST.bm25.search(q, topk=topk) if ST.bm25 else []
+
+
+def _vec_search(q: str, topk: int) -> list[tuple[str, float]]:
+    mat, names, emb = ST.vec_matrix, ST.vec_names, ST.embedder
+    if mat is None or len(names) == 0 or emb is None:
+        return []
+    qv = emb.embed_query(q)
+    sims = mat @ qv
+    order = np.argsort(sims)[::-1][:topk]
+    return [(names[i], float(sims[i])) for i in order]
+
+
+@app.get("/api/search")
+def api_search(q: str = Query(...), topk: int = 30, rrf_k: int = 60,
+               w_bm25: float = 1.0, w_vec: float = 1.0):
+    q = q.strip()
+    if not q:
+        return {"q": q, "bm25": [], "vector": [], "fusion": []}
+
+    bm = _bm25_search(q, topk)
+    ve = _vec_search(q, topk)
+    bm_rank = [n for n, _ in bm]
+    ve_rank = [n for n, _ in ve]
+    fused = rrf({"bm25": bm_rank, "vector": ve_rank},
+                {"bm25": w_bm25, "vector": w_vec}, k=rrf_k)[:topk]
+
+    bm_s, ve_s = dict(bm), dict(ve)
+    bm_pos = {n: i + 1 for i, n in enumerate(bm_rank)}
+    ve_pos = {n: i + 1 for i, n in enumerate(ve_rank)}
+
+    def card(name: str, score: float) -> dict:
+        return {
+            "name": name,
+            "score": round(score, 4),
+            "files": ST.name2files.get(name, []),     # 服务器 serve 的图片文件名
+            "bm25": round(bm_s[name], 3) if name in bm_s else None,
+            "bm25_rank": bm_pos.get(name),
+            "vec": round(ve_s[name], 4) if name in ve_s else None,
+            "vec_rank": ve_pos.get(name),
+        }
+
+    return {
+        "q": q,
+        "bm25": [card(n, s) for n, s in bm],
+        "vector": [card(n, s) for n, s in ve],
+        "fusion": [card(n, s) for n, s in fused],
+    }
+
+
+# 图片由服务器 serve(局域网里 mentor 的浏览器据此显示);必须在 "/" 之前挂载。
+if os.path.isdir(IMAGE_DIR):
+    app.mount("/images", StaticFiles(directory=IMAGE_DIR), name="images")
+# 静态前端(放在所有 /api 与 /images 之后挂载)
+app.mount("/", StaticFiles(directory=os.path.join(HERE, "static"), html=True), name="static")
+
+
+def _lan_ips() -> list[str]:
+    ips: list[str] = []
+    try:
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        s.connect(("8.8.8.8", 80))
+        ips.append(s.getsockname()[0])
+        s.close()
+    except Exception:
+        pass
+    try:
+        for info in socket.getaddrinfo(socket.gethostname(), None, socket.AF_INET):
+            ip = info[4][0]
+            if ip not in ips:
+                ips.append(ip)
+    except Exception:
+        pass
+    return [i for i in ips if not i.startswith("127.")]
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    print("=" * 56, flush=True)
+    print(f"图片目录:{IMAGE_DIR}", flush=True)
+    print("发给 mentor 的局域网地址:", flush=True)
+    for ip in _lan_ips() or ["<查不到,用 ipconfig 看本机 IPv4>"]:
+        print(f"   http://{ip}:{PORT}", flush=True)
+    print(f"本机自测:http://127.0.0.1:{PORT}", flush=True)
+    print("(若 mentor 连不上:Windows 防火墙放行该端口入站)", flush=True)
+    print("=" * 56, flush=True)
+    uvicorn.run(app, host="0.0.0.0", port=PORT, log_level="info")

retrieval/static/app.js

@@ -0,0 +1,120 @@
+'use strict';
+
+const $ = (id) => document.getElementById(id);
+let pollTimer = null;
+
+// ---------- 启动:轮询索引状态(服务器开机自动建索引) ----------
+window.addEventListener('DOMContentLoaded', pollStatus);
+
+function pollStatus() {
+  clearInterval(pollTimer);
+  tick();
+  pollTimer = setInterval(tick, 800);
+}
+
+async function tick() {
+  let s;
+  try { s = await (await fetch('/api/status')).json(); }
+  catch { $('status').textContent = '连接服务失败,确认服务器已启动。'; return; }
+
+  // BM25 就绪即可搜(向量还在编码也能先看关键词路)
+  if (s.bm25_ready) { $('q').disabled = false; $('search').disabled = false; }
+
+  const bar = $('buildBar');
+  if (s.state === 'building') {
+    bar.classList.remove('hidden');
+    if (s.phase === 'loadmodel') {
+      $('status').textContent = `图片 ${s.n_images} 张 / 名称 ${s.total} 个 · 正在加载向量模型…`;
+      setBar('加载本地向量模型 BGE-M3(首次需下载约 2.3G)…', 8);
+    } else if (s.phase === 'embedding') {
+      const pct = s.emb_total ? Math.round((s.emb_done / s.emb_total) * 100) : 0;
+      const dev = s.device ? ` · ${s.device}` : '';
+      $('status').textContent = `图片 ${s.n_images} 张 / 名称 ${s.total} 个 · 向量编码中(BM25 已就绪,可先搜)`;
+      setBar(`向量编码中${dev} ${s.emb_done}/${s.emb_total}`, pct);
+    } else {
+      $('status').textContent = 'BM25 分词建索引中…';
+      setBar('BM25 分词建索引中…', 4);
+    }
+  } else if (s.state === 'ready') {
+    clearInterval(pollTimer);
+    const dev = s.device ? ` (${s.device})` : '';
+    const note = s.failed ? `,${s.failed} 条失败` : '';
+    $('status').innerHTML = `✓ 就绪:<b>${s.total}</b> 个名称 / <b>${s.n_images}</b> 张图${dev}${note} —— 直接搜索吧。`;
+    bar.classList.add('hidden');
+  } else if (s.state === 'error') {
+    clearInterval(pollTimer);
+    $('status').textContent = '❌ ' + (s.message || '建索引失败');
+    bar.classList.add('hidden');
+  }
+}
+
+function setBar(t, pct) {
+  $('buildText').textContent = t;
+  $('barFill').style.width = (pct || 0) + '%';
+}
+
+// ---------- 搜索 ----------
+$('search').addEventListener('click', doSearch);
+$('q').addEventListener('keydown', (e) => { if (e.key === 'Enter') doSearch(); });
+
+async function doSearch() {
+  const q = $('q').value.trim();
+  if (!q) return;
+  const params = new URLSearchParams({
+    q, topk: $('topk').value || 20, rrf_k: $('rrfk').value || 60,
+    w_bm25: $('wbm').value || 1, w_vec: $('wvec').value || 1,
+  });
+  let d;
+  try { d = await (await fetch('/api/search?' + params)).json(); }
+  catch (err) { alert('搜索失败:' + err.message); return; }
+
+  renderCol('bm25', d.bm25);
+  renderCol('vector', d.vector);
+  renderCol('fusion', d.fusion);
+}
+
+function renderCol(kind, items) {
+  $('cnt-' + kind).textContent = items.length ? `共 ${items.length}` : '';
+  const box = $('col-' + kind);
+  box.innerHTML = '';
+  if (!items.length) { box.innerHTML = '<div class="empty">无结果</div>'; return; }
+  items.forEach((it, i) => box.appendChild(card(kind, it, i + 1)));
+}
+
+function card(kind, it, rank) {
+  const el = document.createElement('div');
+  el.className = 'card';
+
+  const primary = kind === 'bm25' ? `BM25 ${fmt(it.bm25)}`
+    : kind === 'vector' ? `余弦 ${fmt(it.vec)}`
+      : `RRF ${fmt(it.score)}`;
+
+  const badges = [];
+  if (kind !== 'bm25') badges.push(rankBadge('BM25', it.bm25_rank));
+  if (kind !== 'vector') badges.push(rankBadge('向量', it.vec_rank));
+
+  const src = imgURL(it.files && it.files[0]);
+  el.innerHTML = `
+    <div class="rk">${rank}</div>
+    ${src ? `<img loading="lazy" src="${src}" title="${esc(it.name)}">` : '<div class="card-img"></div>'}
+    <div class="meta">
+      <div class="nm" title="${esc(it.name)}">${esc(it.name)}</div>
+      <div class="sc">${primary}</div>
+      <div class="badges">${badges.join('')}</div>
+    </div>`;
+  return el;
+}
+
+function rankBadge(label, rank) {
+  return rank
+    ? `<span class="badge">${label}#${rank}</span>`
+    : `<span class="badge miss">${label}未命中</span>`;
+}
+
+function imgURL(file) {
+  return file ? '/images/' + encodeURIComponent(file) : '';
+}
+
+const fmt = (v) => (v === null || v === undefined ? '—' : v);
+const esc = (s) => String(s).replace(/[&<>"]/g, (c) =>
+  ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;' }[c]));

retrieval/static/index.html

@@ -0,0 +1,59 @@
+<!DOCTYPE html>
+<html lang="zh-CN">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>图片名称检索测试台</title>
+  <link rel="stylesheet" href="/style.css">
+</head>
+<body>
+  <header>
+    <h1>图片名称检索测试台</h1>
+    <p class="sub">基于<b>图片名称</b>的三路检索对比:BM25 关键词 / 向量 BGE-M3 / RRF 融合。图片目录已固定,直接搜即可。</p>
+  </header>
+
+  <!-- 索引状态 -->
+  <section class="panel">
+    <div id="status" class="info muted">连接服务中…</div>
+    <div id="buildBar" class="buildbar hidden">
+      <div class="bar"><div id="barFill" class="fill"></div></div>
+      <span id="buildText"></span>
+    </div>
+  </section>
+
+  <!-- 检索 -->
+  <section class="panel">
+    <div class="row">
+      <input id="q" class="query" type="text" placeholder="输入查询词,例如:红烧肉 / 碳酸饮料 / 甜点 / 辣的川菜" disabled>
+      <label class="sel">返回<input id="topk" type="number" value="20" min="1" max="100" style="width:56px"></label>
+      <button id="search" class="btn primary" disabled>搜索</button>
+    </div>
+    <details class="adv">
+      <summary>高级:融合(RRF)参数</summary>
+      <div class="row">
+        <label class="sel">RRF k <input id="rrfk" type="number" value="60" min="1" max="500" style="width:64px"></label>
+        <label class="sel">BM25 权重 <input id="wbm" type="number" value="1.0" step="0.1" min="0" style="width:64px"></label>
+        <label class="sel">向量 权重 <input id="wvec" type="number" value="1.0" step="0.1" min="0" style="width:64px"></label>
+      </div>
+    </details>
+  </section>
+
+  <!-- 结果:三列 -->
+  <section id="results" class="results">
+    <div class="column" data-kind="bm25">
+      <h2>① BM25 关键词 <span class="cnt" id="cnt-bm25"></span></h2>
+      <div class="cards" id="col-bm25"></div>
+    </div>
+    <div class="column" data-kind="vector">
+      <h2>② 向量 BGE-M3 <span class="cnt" id="cnt-vector"></span></h2>
+      <div class="cards" id="col-vector"></div>
+    </div>
+    <div class="column" data-kind="fusion">
+      <h2>③ RRF 融合 <span class="cnt" id="cnt-fusion"></span></h2>
+      <div class="cards" id="col-fusion"></div>
+    </div>
+  </section>
+
+  <script src="/app.js"></script>
+</body>
+</html>

retrieval/static/style.css

@@ -0,0 +1,68 @@
+* { box-sizing: border-box; }
+body {
+  margin: 0; padding: 0 24px 48px;
+  font-family: -apple-system, "Segoe UI", "Microsoft YaHei", Roboto, sans-serif;
+  color: #1f2328; background: #f6f7f9;
+}
+header { padding: 20px 0 8px; }
+h1 { margin: 0; font-size: 22px; }
+.sub { margin: 6px 0 0; color: #57606a; font-size: 14px; }
+.sub b { color: #1f2328; }
+
+.panel {
+  background: #fff; border: 1px solid #e6e8eb; border-radius: 10px;
+  padding: 16px; margin: 14px 0;
+}
+.row { display: flex; flex-wrap: wrap; gap: 12px; align-items: center; }
+
+.btn {
+  display: inline-flex; align-items: center; gap: 6px;
+  padding: 8px 14px; border-radius: 8px; border: 1px solid #d0d7de;
+  background: #fff; cursor: pointer; font-size: 14px; color: #1f2328;
+}
+.btn:hover { background: #f3f4f6; }
+.btn.primary { background: #1f6feb; border-color: #1f6feb; color: #fff; }
+.btn.primary:hover { background: #1a60d0; }
+.btn.primary:disabled, .btn:disabled { opacity: .5; cursor: not-allowed; }
+.file-btn { position: relative; }
+
+.chk, .sel { font-size: 14px; color: #424a53; display: inline-flex; align-items: center; gap: 6px; }
+.sel select, .sel input, .query {
+  font-size: 14px; padding: 7px 9px; border: 1px solid #d0d7de; border-radius: 8px; background: #fff;
+}
+.query { flex: 1; min-width: 240px; }
+code { background: #eef0f2; padding: 1px 5px; border-radius: 4px; font-size: 12px; }
+
+.info { margin-top: 12px; font-size: 13px; }
+.muted { color: #6b7280; }
+
+.buildbar { margin-top: 12px; display: flex; align-items: center; gap: 10px; font-size: 13px; color: #424a53; }
+.bar { flex: 1; height: 8px; background: #eaecef; border-radius: 6px; overflow: hidden; }
+.fill { height: 100%; width: 0; background: #2da44e; transition: width .25s; }
+
+.preview { margin-top: 12px; display: flex; flex-wrap: wrap; gap: 6px; }
+.preview img { width: 64px; height: 64px; object-fit: cover; border-radius: 6px; border: 1px solid #e6e8eb; }
+
+.adv { margin-top: 10px; font-size: 13px; }
+.adv summary { cursor: pointer; color: #57606a; }
+.adv .row { margin-top: 10px; }
+
+.results { display: grid; grid-template-columns: repeat(3, 1fr); gap: 14px; margin-top: 4px; }
+.column { background: #fff; border: 1px solid #e6e8eb; border-radius: 10px; padding: 12px; min-height: 80px; }
+.column h2 { margin: 0 0 10px; font-size: 15px; display: flex; align-items: center; gap: 8px; }
+.cnt { color: #6b7280; font-size: 12px; font-weight: 400; }
+
+.cards { display: flex; flex-direction: column; gap: 8px; }
+.card { display: flex; gap: 10px; padding: 8px; border: 1px solid #eef0f2; border-radius: 8px; align-items: center; }
+.card:hover { border-color: #c8d1da; background: #fafbfc; }
+.card .rk { width: 22px; text-align: center; font-size: 13px; color: #8b949e; flex: none; }
+.card img { width: 56px; height: 56px; object-fit: cover; border-radius: 6px; flex: none; background: #eef0f2; }
+.card .meta { min-width: 0; }
+.card .nm { font-size: 14px; font-weight: 600; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
+.card .sc { font-size: 12px; color: #1f6feb; margin-top: 2px; }
+.card .badges { margin-top: 3px; display: flex; flex-wrap: wrap; gap: 4px; }
+.badge { font-size: 11px; padding: 1px 6px; border-radius: 999px; background: #eef0f2; color: #57606a; }
+.badge.miss { background: #fff1f0; color: #cf222e; }
+.empty { color: #8b949e; font-size: 13px; padding: 8px 4px; }
+
+@media (max-width: 900px) { .results { grid-template-columns: 1fr; } }