chenshuobo
783dfd059d
智能推荐 / 销量最高两 tab 的离线库此前只有北京(ETL 写死 cityId),改为遍历 美团官方城市字典 359 个地级市全量抓取。实测一个地级市 cityId 已覆盖其下辖县级市 (徐州→邳州/新沂/睢宁等),按地级市抓即可,无需区县层级。 - 城市字典:tools/gen_meituan_cities.py 从美团 Excel 生成随仓库 JSON (app/integrations/data/meituan_cities.json,359 城),app/integrations/cities.py 读取; - ETL:city_id 参数化 + 城市级并发(默认 12)+ worker 启动错峰削平瞬时峰值 + 主线程逐城串行入库(Session 不跨线程); - 配速实测:单城全量 ~110s/~2300 条;15 并发抓完一轮 ~50-60min,402 占 3% 退避全消化; 每 3h 一轮全量(--interval 10800),窗口充裕; - prune 双护栏:本轮 0 入库 或 失败城占比 >5% 时跳过,防上游故障/大面积限流误删全表; - 仅写入侧;读取侧(rec/top-sales 按城过滤)待后续(依赖用户定位→cityId 映射,字典无经纬度)。 --------- Co-authored-by: chenshuobo <1119780489@qq.com> Reviewed-on: #57 Co-authored-by: chenshuobo <chenshuobo@wonderable.ai> Co-committed-by: chenshuobo <chenshuobo@wonderable.ai>
58 lines
1.9 KiB
Python
58 lines
1.9 KiB
Python
# -*- coding: utf-8 -*-
|
|
"""从美团「城市字典」Excel 生成随仓库的 JSON(app/integrations/data/meituan_cities.json)。
|
|
|
|
本地一次性工具:美团每年更新城市字典时,把新 Excel 放进来重跑即可。
|
|
Excel 不入库、不进仓库(二进制、需 openpyxl);生成的 JSON 随仓库提交,部署到服务器后
|
|
ETL(scripts/pull_meituan_coupons.py)直接读它遍历全部城市。
|
|
|
|
字典口径:359 个地级市,经实测一个地级市 cityId 已覆盖其下辖县级市(如徐州→邳州/新沂),
|
|
故无需区县层级。
|
|
|
|
用法:
|
|
python tools/gen_meituan_cities.py ["城市字典xxx.xlsx" 路径]
|
|
(不传则用默认 e:\\codes\\城市字典2025 (1).xlsx)
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import sys
|
|
|
|
import openpyxl
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
ROOT = os.path.dirname(HERE)
|
|
DEFAULT_EXCEL = r"e:\codes\城市字典2025 (1).xlsx"
|
|
OUT = os.path.join(ROOT, "app", "integrations", "data", "meituan_cities.json")
|
|
|
|
|
|
def main() -> None:
|
|
excel = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_EXCEL
|
|
wb = openpyxl.load_workbook(excel, data_only=True)
|
|
ws = wb.worksheets[0]
|
|
rows = list(ws.iter_rows(values_only=True))
|
|
header = rows[0]
|
|
|
|
cities = []
|
|
seen = set()
|
|
for r in rows[1:]:
|
|
cid = str(r[0]).strip() if r[0] else ""
|
|
name = str(r[1]).strip() if len(r) > 1 and r[1] else ""
|
|
prov = str(r[2]).strip() if len(r) > 2 and r[2] else ""
|
|
if not cid or cid in seen:
|
|
continue
|
|
seen.add(cid)
|
|
cities.append({"city_id": cid, "name": name, "province": prov})
|
|
|
|
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
|
with open(OUT, "w", encoding="utf-8") as f:
|
|
json.dump(cities, f, ensure_ascii=False, indent=1)
|
|
|
|
print(f"表头: {header}")
|
|
print(f"写出 {len(cities)} 城 → {OUT}")
|
|
print("样例:", cities[:3])
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|