feat: add S-52 production pipeline v2
This commit is contained in:
110
docs/CM95_S52_ProductionPipeline_v2.md
Normal file
110
docs/CM95_S52_ProductionPipeline_v2.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# CM95 S-52 Production Pipeline v2
|
||||||
|
|
||||||
|
## 1. 目标
|
||||||
|
|
||||||
|
Production Pipeline v2 将已经通过单 Tile 验证的 S-52 runtime PBF 正式接入现有 CM95 生产编排层,并增加可审计 AOI 计划能力。
|
||||||
|
|
||||||
|
核心要求是:同一个 release identity 不允许混入不同 S-52 Lookup、不同 mariner mode 或不同 AOI 的 Tile。
|
||||||
|
|
||||||
|
## 2. Production Plan v2
|
||||||
|
|
||||||
|
新计划 schema:
|
||||||
|
|
||||||
|
```text
|
||||||
|
navsea.cm95.production-plan/v2
|
||||||
|
```
|
||||||
|
|
||||||
|
Pipeline 版本:
|
||||||
|
|
||||||
|
```text
|
||||||
|
cm95pbf-production/2.0-s52-aoi
|
||||||
|
```
|
||||||
|
|
||||||
|
Plan immutable hash 现在除 Dataset、mapping、policy、FID manifest、Builder Contract、jobs 外,还冻结:
|
||||||
|
|
||||||
|
- S-52 是否启用;
|
||||||
|
- S-52 Lookup SHA256;
|
||||||
|
- Point Table;
|
||||||
|
- Boundary Table;
|
||||||
|
- `runtime/reference` mariner mode;
|
||||||
|
- AOI name;
|
||||||
|
- AOI WGS84 bbox。
|
||||||
|
|
||||||
|
S-52 Lookup 路径本身不参与 identity,内容 SHA256 才参与 identity;同内容换路径不会制造新 release。
|
||||||
|
|
||||||
|
## 3. AOI 枚举
|
||||||
|
|
||||||
|
`plan-production` 新增:
|
||||||
|
|
||||||
|
```text
|
||||||
|
--aoi-name
|
||||||
|
--aoi-bbox west,south,east,north
|
||||||
|
```
|
||||||
|
|
||||||
|
AOI 在 Web Mercator 中与 Dataset coverage 几何相交并裁剪后再枚举 XYZ job。这样不是简单按 Tile 中心筛选,边界 coverage 也不会被漏掉。
|
||||||
|
|
||||||
|
Plan `source_stats` 同时记录 AOI 裁剪前后的 coverage piece 数量。
|
||||||
|
|
||||||
|
## 4. S-52 生产参数
|
||||||
|
|
||||||
|
`plan-production` 新增:
|
||||||
|
|
||||||
|
```text
|
||||||
|
--s52-lookups
|
||||||
|
--s52-point-table Simplified|Paper
|
||||||
|
--s52-boundary-table Plain|Symbolized
|
||||||
|
--s52-mariner-mode runtime|reference
|
||||||
|
```
|
||||||
|
|
||||||
|
`build-production` 不再从命令行临时接收这些值,而是只能读取冻结 plan,随后传给 `Cm95TileBuilder`。
|
||||||
|
|
||||||
|
Build 启动前重新校验 Lookup 文件存在且 SHA256 与 plan 一致;文件发生变化时 fail-closed。
|
||||||
|
|
||||||
|
## 5. Runtime release gate
|
||||||
|
|
||||||
|
`finalize-production` 在 S-52 runtime plan 下增加 release 级 runtime contract gate。
|
||||||
|
|
||||||
|
它会解码所有输出 PBF,并检查:
|
||||||
|
|
||||||
|
- `s52_mariner_dependent` 与 `s52_runtime_csp` 一致;
|
||||||
|
- DEPARE 存在 `depth_min_m`,且没有静态 `s52_area_color` 泄漏;
|
||||||
|
- DEPCNT 存在 `contour_depth_m`;
|
||||||
|
- SOUNDG 存在 `s52_sounding_depth_m`;
|
||||||
|
- OBSTRN/WRECKS 存在 `s52_hazard_depth_m`;
|
||||||
|
- 不允许 `s52_is_safety_contour`、`s52_isolated_danger`、`s52_display_base_promoted` 等动态最终状态泄漏。
|
||||||
|
|
||||||
|
失败会把整个 release 标记为 FAIL。
|
||||||
|
|
||||||
|
## 6. Style contract 过渡
|
||||||
|
|
||||||
|
当前旧 NavSea semantic style/sprite contract 不是 S-52 runtime Adapter 的最终 contract。
|
||||||
|
|
||||||
|
因此 CLI 新增:
|
||||||
|
|
||||||
|
```text
|
||||||
|
--skip-style-contract
|
||||||
|
```
|
||||||
|
|
||||||
|
它只跳过旧 style/sprite 审计,不跳过 PBF 计数、FID、hard gate、encoder、名称冲突和 S-52 runtime contract。待 NavSea `S52RuntimePresentationAdapter` 完成后,再接入正式 S-52 style/runtime contract。
|
||||||
|
|
||||||
|
## 7. 兼容性
|
||||||
|
|
||||||
|
Loader 继续接受历史 `production-plan/v1`,但新建 plan 一律生成 v2。
|
||||||
|
|
||||||
|
旧 release 若没有与新 plan 完全一致的 S-52 production contract,不允许直接通过 `migrate-production-release` 复用 Tile。
|
||||||
|
|
||||||
|
## 8. 九州首次运行
|
||||||
|
|
||||||
|
本次 AOI:
|
||||||
|
|
||||||
|
```text
|
||||||
|
name = kyushu-main-nearby-v1
|
||||||
|
bbox = 128.0,29.5,132.5,34.8
|
||||||
|
zooms = 0-12
|
||||||
|
profile = delivery
|
||||||
|
point_table = Paper
|
||||||
|
boundary_table = Plain
|
||||||
|
mariner_mode = runtime
|
||||||
|
```
|
||||||
|
|
||||||
|
该范围用于本轮工程验证,覆盖九州本岛、对马、五岛、种子岛/屋久岛及近海,不作为日本行政区边界定义。
|
||||||
57
docs/报告/CM95_S52_ProductionPipelineV2与九州运行报告_v1.md
Normal file
57
docs/报告/CM95_S52_ProductionPipelineV2与九州运行报告_v1.md
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
# CM95 S-52 Production Pipeline v2 与九州运行报告 v1
|
||||||
|
|
||||||
|
日期:2026-09-18
|
||||||
|
|
||||||
|
## Pipeline 升级
|
||||||
|
|
||||||
|
已完成 Production Plan v2、S-52 immutable contract、AOI coverage 裁剪、build 参数接线、Lookup SHA fail-closed、release runtime contract gate,以及旧 style contract 的显式过渡开关。
|
||||||
|
|
||||||
|
## 福冈湾端到端 Smoke
|
||||||
|
|
||||||
|
AOI:`130.3418,33.5780,130.4297,33.6512`,z12。
|
||||||
|
|
||||||
|
Plan 枚举 4 个 Tile,`build-production --workers 4` 结果:
|
||||||
|
|
||||||
|
- PASS:4/4
|
||||||
|
- PBF:4
|
||||||
|
- output features:3235
|
||||||
|
- 主 Tile `12/3531/1641`:3164 features / 22 layers
|
||||||
|
|
||||||
|
`finalize-production --skip-style-contract`:PASS。
|
||||||
|
|
||||||
|
S-52 runtime gate:
|
||||||
|
|
||||||
|
- `mariner_dependent_count = 2261`
|
||||||
|
- `DEPARE01 = 173`
|
||||||
|
- `DEPCNT02 = 188`
|
||||||
|
- `OBSTRN04 = 19`
|
||||||
|
- `SOUNDG02 = 1881`
|
||||||
|
- `dynamic_final_leaks = {}`
|
||||||
|
- `failure_count = 0`
|
||||||
|
|
||||||
|
planned/report/decoded feature count 均为 3235。
|
||||||
|
|
||||||
|
## 九州 Production Plan
|
||||||
|
|
||||||
|
工程 AOI:`128.0,29.5,132.5,34.8`,名称 `kyushu-main-nearby-v1`。
|
||||||
|
|
||||||
|
Plan:
|
||||||
|
|
||||||
|
- schema:`navsea.cm95.production-plan/v2`
|
||||||
|
- pipeline:`cm95pbf-production/2.0-s52-aoi`
|
||||||
|
- release:`cm95-d1-62a85bd40d3a-bbaaeafee1b4`
|
||||||
|
- jobs:1109
|
||||||
|
- S-52 mariner mode:runtime
|
||||||
|
- Lookup SHA256:`f63cbc0d3c9317b1873c0b0c470a5e7371d1b6d8d58393da660f70b837a52eb7`
|
||||||
|
|
||||||
|
各级 job 数:z5=2、z6=4、z7=12、z8=24、z9=70、z10=197、z11=503、z12=297。z0-z4 在该 AOI 内没有对应 1:20M coverage,因此不制造空 job。
|
||||||
|
|
||||||
|
## 当前运行
|
||||||
|
|
||||||
|
九州 `build-production` 已使用 4 workers、最多 3 attempts 启动。
|
||||||
|
|
||||||
|
日志:`build/logs/cm95_s52_kyushu_pipeline_v2.log`
|
||||||
|
|
||||||
|
Release root:`build/releases-s52-v2/cm95-d1-62a85bd40d3a-bbaaeafee1b4`
|
||||||
|
|
||||||
|
初始 ledger:1109 jobs,4 RUNNING,1105 PENDING。后续以 `status-production` 和最终 `finalize-production` 为权威。
|
||||||
@@ -28,6 +28,18 @@ def _add_common_db(p):
|
|||||||
p.add_argument("--dataset-id", type=int, default=1)
|
p.add_argument("--dataset-id", type=int, default=1)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_aoi_bbox(value: str | None):
|
||||||
|
if value is None or not str(value).strip():
|
||||||
|
return None
|
||||||
|
parts = [part.strip() for part in str(value).split(",")]
|
||||||
|
if len(parts) != 4:
|
||||||
|
raise argparse.ArgumentTypeError("AOI bbox 必须是 west,south,east,north")
|
||||||
|
try:
|
||||||
|
return tuple(float(part) for part in parts)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise argparse.ArgumentTypeError("AOI bbox 必须是四个数字") from exc
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
p = argparse.ArgumentParser(
|
p = argparse.ArgumentParser(
|
||||||
prog="cm95pbf",
|
prog="cm95pbf",
|
||||||
@@ -121,7 +133,13 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
prod_plan.add_argument("--mapping", default=str(DEFAULT_MAPPING))
|
prod_plan.add_argument("--mapping", default=str(DEFAULT_MAPPING))
|
||||||
prod_plan.add_argument("--policy", default=str(DEFAULT_POLICY))
|
prod_plan.add_argument("--policy", default=str(DEFAULT_POLICY))
|
||||||
prod_plan.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
prod_plan.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
||||||
prod_plan.add_argument("--output", default="build/production/cm95_production_plan_v1.json")
|
prod_plan.add_argument("--output", default="build/production/cm95_production_plan_v2.json")
|
||||||
|
prod_plan.add_argument("--s52-lookups", default=None, help="S-52 lookup_rule_ir.json;提供后冻结 SHA256 并进入 production builder")
|
||||||
|
prod_plan.add_argument("--s52-point-table", choices=["Simplified", "Paper"], default="Paper")
|
||||||
|
prod_plan.add_argument("--s52-boundary-table", choices=["Plain", "Symbolized"], default="Plain")
|
||||||
|
prod_plan.add_argument("--s52-mariner-mode", choices=["runtime", "reference"], default="runtime")
|
||||||
|
prod_plan.add_argument("--aoi-name", default=None, help="可审计 AOI 名称")
|
||||||
|
prod_plan.add_argument("--aoi-bbox", type=_parse_aoi_bbox, default=None, metavar="W,S,E,N", help="只枚举与 WGS84 bbox 相交的 coverage/tile")
|
||||||
|
|
||||||
prod_build = sub.add_parser(
|
prod_build = sub.add_parser(
|
||||||
"build-production",
|
"build-production",
|
||||||
@@ -162,6 +180,7 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
prod_final.add_argument("--style", default="config/style/style.navsea-delivery-full-semantic-unified-20260808-cm95-v1.json")
|
prod_final.add_argument("--style", default="config/style/style.navsea-delivery-full-semantic-unified-20260808-cm95-v1.json")
|
||||||
prod_final.add_argument("--sprite-json", default="config/sprite/semantic-sprite-20260808.json")
|
prod_final.add_argument("--sprite-json", default="config/sprite/semantic-sprite-20260808.json")
|
||||||
prod_final.add_argument("--gap-disposition", default="config/mappings/cm95_gap_disposition_v1.yaml")
|
prod_final.add_argument("--gap-disposition", default="config/mappings/cm95_gap_disposition_v1.yaml")
|
||||||
|
prod_final.add_argument("--skip-style-contract", action="store_true", help="跳过旧 NavSea style/sprite contract;S-52 runtime release 仍执行 runtime contract gate")
|
||||||
|
|
||||||
return p
|
return p
|
||||||
|
|
||||||
@@ -206,11 +225,20 @@ def main() -> None:
|
|||||||
output_path=Path(args.output),
|
output_path=Path(args.output),
|
||||||
zooms=parse_zooms(args.zooms),
|
zooms=parse_zooms(args.zooms),
|
||||||
profile=args.profile,
|
profile=args.profile,
|
||||||
|
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
|
||||||
|
s52_point_table=args.s52_point_table,
|
||||||
|
s52_boundary_table=args.s52_boundary_table,
|
||||||
|
s52_mariner_mode=args.s52_mariner_mode,
|
||||||
|
aoi_name=args.aoi_name,
|
||||||
|
aoi_bbox_wgs84=args.aoi_bbox,
|
||||||
)
|
)
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
"plan_id": result["plan_id"],
|
"plan_id": result["plan_id"],
|
||||||
"release_id": result["release_id"],
|
"release_id": result["release_id"],
|
||||||
"job_count": result["job_count"],
|
"job_count": result["job_count"],
|
||||||
|
"production_pipeline_version": result.get("production_pipeline_version"),
|
||||||
|
"s52": result.get("s52"),
|
||||||
|
"aoi": result.get("aoi"),
|
||||||
"source_stats": result["source_stats"],
|
"source_stats": result["source_stats"],
|
||||||
"output": str(Path(args.output)),
|
"output": str(Path(args.output)),
|
||||||
}, ensure_ascii=False, indent=2))
|
}, ensure_ascii=False, indent=2))
|
||||||
@@ -249,8 +277,8 @@ def main() -> None:
|
|||||||
result = finalize_release(
|
result = finalize_release(
|
||||||
Path(args.plan),
|
Path(args.plan),
|
||||||
Path(args.release_root),
|
Path(args.release_root),
|
||||||
Path(args.style) if args.style else None,
|
None if args.skip_style_contract else (Path(args.style) if args.style else None),
|
||||||
Path(args.sprite_json) if args.sprite_json else None,
|
None if args.skip_style_contract else (Path(args.sprite_json) if args.sprite_json else None),
|
||||||
Path(args.gap_disposition) if args.gap_disposition else None,
|
Path(args.gap_disposition) if args.gap_disposition else None,
|
||||||
)
|
)
|
||||||
print(json.dumps({
|
print(json.dumps({
|
||||||
@@ -269,6 +297,7 @@ def main() -> None:
|
|||||||
"contract_unmatched_count": (result.get("style_sprite_contract") or {}).get("contract_unmatched_count"),
|
"contract_unmatched_count": (result.get("style_sprite_contract") or {}).get("contract_unmatched_count"),
|
||||||
"sprite_missing_count": (result.get("style_sprite_contract") or {}).get("sprite_missing_count"),
|
"sprite_missing_count": (result.get("style_sprite_contract") or {}).get("sprite_missing_count"),
|
||||||
},
|
},
|
||||||
|
"s52_runtime_contract": result.get("s52_runtime_contract"),
|
||||||
"quantization_fallbacks": result.get("quantization_fallbacks"),
|
"quantization_fallbacks": result.get("quantization_fallbacks"),
|
||||||
"quantization_omissions": result.get("quantization_omissions"),
|
"quantization_omissions": result.get("quantization_omissions"),
|
||||||
"nav_name_conflict_audit": {
|
"nav_name_conflict_audit": {
|
||||||
|
|||||||
@@ -32,7 +32,10 @@ from .policy import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
PRODUCTION_PLAN_SCHEMA = "navsea.cm95.production-plan/v1"
|
PRODUCTION_PLAN_SCHEMA_V1 = "navsea.cm95.production-plan/v1"
|
||||||
|
PRODUCTION_PLAN_SCHEMA = "navsea.cm95.production-plan/v2"
|
||||||
|
SUPPORTED_PRODUCTION_PLAN_SCHEMAS = {PRODUCTION_PLAN_SCHEMA_V1, PRODUCTION_PLAN_SCHEMA}
|
||||||
|
PRODUCTION_PIPELINE_VERSION = "cm95pbf-production/2.0-s52-aoi"
|
||||||
PRODUCTION_RUN_SCHEMA = "navsea.cm95.production-run/v1"
|
PRODUCTION_RUN_SCHEMA = "navsea.cm95.production-run/v1"
|
||||||
|
|
||||||
|
|
||||||
@@ -231,6 +234,13 @@ def create_production_plan(
|
|||||||
output_path: Path,
|
output_path: Path,
|
||||||
zooms: list[int],
|
zooms: list[int],
|
||||||
profile: str = "delivery",
|
profile: str = "delivery",
|
||||||
|
*,
|
||||||
|
s52_lookups_path: Path | None = None,
|
||||||
|
s52_point_table: str = "Paper",
|
||||||
|
s52_boundary_table: str = "Plain",
|
||||||
|
s52_mariner_mode: str = "runtime",
|
||||||
|
aoi_name: str | None = None,
|
||||||
|
aoi_bbox_wgs84: tuple[float, float, float, float] | None = None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
if profile not in {"delivery", "engineering"}:
|
if profile not in {"delivery", "engineering"}:
|
||||||
raise ValueError("profile must be delivery or engineering")
|
raise ValueError("profile must be delivery or engineering")
|
||||||
@@ -240,6 +250,29 @@ def create_production_plan(
|
|||||||
output_path = Path(output_path)
|
output_path = Path(output_path)
|
||||||
policy = _load_policy(policy_path)
|
policy = _load_policy(policy_path)
|
||||||
zooms = parse_zooms(zooms)
|
zooms = parse_zooms(zooms)
|
||||||
|
if s52_point_table not in {"Simplified", "Paper"}:
|
||||||
|
raise ValueError("s52_point_table must be Simplified or Paper")
|
||||||
|
if s52_boundary_table not in {"Plain", "Symbolized"}:
|
||||||
|
raise ValueError("s52_boundary_table must be Plain or Symbolized")
|
||||||
|
if s52_mariner_mode not in {"runtime", "reference"}:
|
||||||
|
raise ValueError("s52_mariner_mode must be runtime or reference")
|
||||||
|
|
||||||
|
s52_path = Path(s52_lookups_path) if s52_lookups_path is not None else None
|
||||||
|
if s52_path is not None and not s52_path.exists():
|
||||||
|
raise RuntimeError(f"S-52 lookup file not found: {s52_path}")
|
||||||
|
|
||||||
|
aoi_bbox = None
|
||||||
|
aoi_box_m = None
|
||||||
|
if aoi_bbox_wgs84 is not None:
|
||||||
|
west, south, east, north = (float(v) for v in aoi_bbox_wgs84)
|
||||||
|
if not (-180.0 <= west < east <= 180.0):
|
||||||
|
raise ValueError(f"invalid AOI longitude range: {aoi_bbox_wgs84}")
|
||||||
|
if not (-85.0 <= south < north <= 85.0):
|
||||||
|
raise ValueError(f"invalid AOI latitude range: {aoi_bbox_wgs84}")
|
||||||
|
aoi_bbox = [west, south, east, north]
|
||||||
|
minx, miny = lonlat_to_mercator(west, south)
|
||||||
|
maxx, maxy = lonlat_to_mercator(east, north)
|
||||||
|
aoi_box_m = box(minx, miny, maxx, maxy)
|
||||||
|
|
||||||
all_scales = sorted({s for z in zooms for s in _enumeration_scales(policy, z)})
|
all_scales = sorted({s for z in zooms for s in _enumeration_scales(policy, z)})
|
||||||
|
|
||||||
@@ -259,7 +292,16 @@ def create_production_plan(
|
|||||||
ladder = _scale_ladder(policy, z)
|
ladder = _scale_ladder(policy, z)
|
||||||
enumeration = _enumeration_scales(policy, z)
|
enumeration = _enumeration_scales(policy, z)
|
||||||
enumeration_set = set(enumeration)
|
enumeration_set = set(enumeration)
|
||||||
z_pieces = [(cid, scale, geom, src) for cid, scale, geom, src in pieces if scale in enumeration_set]
|
raw_z_pieces = [(cid, scale, geom, src) for cid, scale, geom, src in pieces if scale in enumeration_set]
|
||||||
|
z_pieces = []
|
||||||
|
for cid, scale, geom, src in raw_z_pieces:
|
||||||
|
if aoi_box_m is not None:
|
||||||
|
if not geom.intersects(aoi_box_m):
|
||||||
|
continue
|
||||||
|
geom = geom.intersection(aoi_box_m)
|
||||||
|
if geom.is_empty:
|
||||||
|
continue
|
||||||
|
z_pieces.append((cid, scale, geom, src))
|
||||||
z_jobs: set[tuple[int, int, int]] = set()
|
z_jobs: set[tuple[int, int, int]] = set()
|
||||||
fallback_pieces = 0
|
fallback_pieces = 0
|
||||||
for _cid, _scale, geom, source in z_pieces:
|
for _cid, _scale, geom, source in z_pieces:
|
||||||
@@ -270,6 +312,7 @@ def create_production_plan(
|
|||||||
source_stats[str(z)] = {
|
source_stats[str(z)] = {
|
||||||
"enumeration_scales": enumeration,
|
"enumeration_scales": enumeration,
|
||||||
"scale_ladder": ladder,
|
"scale_ladder": ladder,
|
||||||
|
"coverage_pieces_before_aoi": len(raw_z_pieces),
|
||||||
"coverage_pieces": len(z_pieces),
|
"coverage_pieces": len(z_pieces),
|
||||||
"cell_bbox_fallback_pieces": fallback_pieces,
|
"cell_bbox_fallback_pieces": fallback_pieces,
|
||||||
"tile_count": len(z_jobs),
|
"tile_count": len(z_jobs),
|
||||||
@@ -280,8 +323,20 @@ def create_production_plan(
|
|||||||
for z, x, y in sorted(jobs)
|
for z, x, y in sorted(jobs)
|
||||||
]
|
]
|
||||||
|
|
||||||
|
s52_contract = {
|
||||||
|
"enabled": s52_path is not None,
|
||||||
|
"lookups_sha256": sha256_file(s52_path) if s52_path is not None else None,
|
||||||
|
"point_table": s52_point_table if s52_path is not None else None,
|
||||||
|
"boundary_table": s52_boundary_table if s52_path is not None else None,
|
||||||
|
"mariner_mode": s52_mariner_mode if s52_path is not None else None,
|
||||||
|
}
|
||||||
|
aoi_contract = {
|
||||||
|
"name": str(aoi_name or "").strip() or None,
|
||||||
|
"bbox_wgs84": aoi_bbox,
|
||||||
|
}
|
||||||
immutable = {
|
immutable = {
|
||||||
"schema": PRODUCTION_PLAN_SCHEMA,
|
"schema": PRODUCTION_PLAN_SCHEMA,
|
||||||
|
"production_pipeline_version": PRODUCTION_PIPELINE_VERSION,
|
||||||
"dataset_id": int(dataset_id),
|
"dataset_id": int(dataset_id),
|
||||||
"dataset_fingerprint": fingerprint,
|
"dataset_fingerprint": fingerprint,
|
||||||
"profile": profile,
|
"profile": profile,
|
||||||
@@ -291,6 +346,8 @@ def create_production_plan(
|
|||||||
"builder_contract_version": BUILDER_CONTRACT_VERSION,
|
"builder_contract_version": BUILDER_CONTRACT_VERSION,
|
||||||
"manifest_dataset_fingerprint": manifest_info["dataset_fingerprint"],
|
"manifest_dataset_fingerprint": manifest_info["dataset_fingerprint"],
|
||||||
"manifest_identity_count": int(manifest_info["identity_count"]),
|
"manifest_identity_count": int(manifest_info["identity_count"]),
|
||||||
|
"s52": s52_contract,
|
||||||
|
"aoi": aoi_contract,
|
||||||
"jobs": job_rows,
|
"jobs": job_rows,
|
||||||
}
|
}
|
||||||
plan_id = canonical_sha256(immutable)
|
plan_id = canonical_sha256(immutable)
|
||||||
@@ -302,6 +359,7 @@ def create_production_plan(
|
|||||||
"mapping_path": str(mapping_path),
|
"mapping_path": str(mapping_path),
|
||||||
"policy_path": str(policy_path),
|
"policy_path": str(policy_path),
|
||||||
"manifest_path": str(manifest_path),
|
"manifest_path": str(manifest_path),
|
||||||
|
"s52_lookups_path": str(s52_path) if s52_path is not None else None,
|
||||||
"source_stats": source_stats,
|
"source_stats": source_stats,
|
||||||
"job_count": len(job_rows),
|
"job_count": len(job_rows),
|
||||||
}
|
}
|
||||||
@@ -312,7 +370,7 @@ def create_production_plan(
|
|||||||
|
|
||||||
def load_plan(path: Path) -> dict[str, Any]:
|
def load_plan(path: Path) -> dict[str, Any]:
|
||||||
obj = json.loads(Path(path).read_text(encoding="utf-8"))
|
obj = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||||
if obj.get("schema") != PRODUCTION_PLAN_SCHEMA:
|
if obj.get("schema") not in SUPPORTED_PRODUCTION_PLAN_SCHEMAS:
|
||||||
raise RuntimeError(f"unsupported plan schema: {obj.get('schema')}")
|
raise RuntimeError(f"unsupported plan schema: {obj.get('schema')}")
|
||||||
return obj
|
return obj
|
||||||
|
|
||||||
@@ -476,6 +534,16 @@ def _validate_plan_inputs(db: DbConfig, plan: dict[str, Any]):
|
|||||||
raise RuntimeError("mapping file changed after plan creation")
|
raise RuntimeError("mapping file changed after plan creation")
|
||||||
if sha256_file(policy_path) != plan["policy_sha256"]:
|
if sha256_file(policy_path) != plan["policy_sha256"]:
|
||||||
raise RuntimeError("policy file changed after plan creation")
|
raise RuntimeError("policy file changed after plan creation")
|
||||||
|
s52 = dict(plan.get("s52") or {})
|
||||||
|
if s52.get("enabled"):
|
||||||
|
s52_path_raw = plan.get("s52_lookups_path")
|
||||||
|
if not s52_path_raw:
|
||||||
|
raise RuntimeError("S-52 enabled plan missing s52_lookups_path")
|
||||||
|
s52_path = Path(s52_path_raw)
|
||||||
|
if not s52_path.exists():
|
||||||
|
raise RuntimeError(f"S-52 lookup file missing: {s52_path}")
|
||||||
|
if sha256_file(s52_path) != str(s52.get("lookups_sha256") or ""):
|
||||||
|
raise RuntimeError("S-52 lookup file changed after plan creation")
|
||||||
with _connect(db) as conn:
|
with _connect(db) as conn:
|
||||||
fp = dataset_fingerprint(conn, int(plan["dataset_id"]))
|
fp = dataset_fingerprint(conn, int(plan["dataset_id"]))
|
||||||
if fp != plan["dataset_fingerprint"]:
|
if fp != plan["dataset_fingerprint"]:
|
||||||
@@ -525,6 +593,10 @@ def run_production_plan(
|
|||||||
output_root=output_root,
|
output_root=output_root,
|
||||||
report_root=report_root,
|
report_root=report_root,
|
||||||
profile=profile,
|
profile=profile,
|
||||||
|
s52_lookups_path=(Path(plan["s52_lookups_path"]) if (plan.get("s52") or {}).get("enabled") else None),
|
||||||
|
s52_point_table=str((plan.get("s52") or {}).get("point_table") or "Paper"),
|
||||||
|
s52_boundary_table=str((plan.get("s52") or {}).get("boundary_table") or "Plain"),
|
||||||
|
s52_mariner_mode=str((plan.get("s52") or {}).get("mariner_mode") or "runtime"),
|
||||||
)
|
)
|
||||||
result = builder.build_tile(TileJob(z, x, y))
|
result = builder.build_tile(TileJob(z, x, y))
|
||||||
if result.get("hard_gate_unmapped"):
|
if result.get("hard_gate_unmapped"):
|
||||||
@@ -586,6 +658,9 @@ def _write_run_summary(plan: dict[str, Any], release_dir: Path, summary: dict[st
|
|||||||
"dataset_fingerprint": plan["dataset_fingerprint"],
|
"dataset_fingerprint": plan["dataset_fingerprint"],
|
||||||
"profile": plan["profile"],
|
"profile": plan["profile"],
|
||||||
"builder_contract_version": plan.get("builder_contract_version"),
|
"builder_contract_version": plan.get("builder_contract_version"),
|
||||||
|
"production_pipeline_version": plan.get("production_pipeline_version"),
|
||||||
|
"s52": plan.get("s52"),
|
||||||
|
"aoi": plan.get("aoi"),
|
||||||
"updated_at": utc_now(),
|
"updated_at": utc_now(),
|
||||||
**summary,
|
**summary,
|
||||||
}
|
}
|
||||||
@@ -602,6 +677,9 @@ def production_status(plan_path: Path, release_root: Path) -> dict[str, Any]:
|
|||||||
return {
|
return {
|
||||||
"release_id": plan["release_id"],
|
"release_id": plan["release_id"],
|
||||||
"release_dir": str(release_dir),
|
"release_dir": str(release_dir),
|
||||||
|
"production_pipeline_version": plan.get("production_pipeline_version"),
|
||||||
|
"s52": plan.get("s52"),
|
||||||
|
"aoi": plan.get("aoi"),
|
||||||
**summary,
|
**summary,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -689,6 +767,10 @@ def migrate_production_release(
|
|||||||
raise RuntimeError("old/new dataset fingerprint mismatch")
|
raise RuntimeError("old/new dataset fingerprint mismatch")
|
||||||
if str(old_summary.get("profile") or "") != str(new_plan["profile"]):
|
if str(old_summary.get("profile") or "") != str(new_plan["profile"]):
|
||||||
raise RuntimeError("old/new profile mismatch")
|
raise RuntimeError("old/new profile mismatch")
|
||||||
|
new_s52 = dict(new_plan.get("s52") or {})
|
||||||
|
old_s52 = dict(old_summary.get("s52") or {})
|
||||||
|
if new_s52.get("enabled") and old_s52 != new_s52:
|
||||||
|
raise RuntimeError("old/new S-52 production contract mismatch; direct tile reuse is unsafe")
|
||||||
|
|
||||||
profile = str(new_plan["profile"])
|
profile = str(new_plan["profile"])
|
||||||
new_release_dir = Path(release_root) / str(new_plan["release_id"])
|
new_release_dir = Path(release_root) / str(new_plan["release_id"])
|
||||||
@@ -805,6 +887,9 @@ def migrate_production_release(
|
|||||||
"new_release_dir": str(new_release_dir),
|
"new_release_dir": str(new_release_dir),
|
||||||
"new_release_id": str(new_plan["release_id"]),
|
"new_release_id": str(new_plan["release_id"]),
|
||||||
"builder_contract_version": new_plan.get("builder_contract_version"),
|
"builder_contract_version": new_plan.get("builder_contract_version"),
|
||||||
|
"production_pipeline_version": new_plan.get("production_pipeline_version"),
|
||||||
|
"s52": new_plan.get("s52"),
|
||||||
|
"aoi": new_plan.get("aoi"),
|
||||||
"changed_delivery_classes": sorted(changed_classes),
|
"changed_delivery_classes": sorted(changed_classes),
|
||||||
"reused_pass_jobs": reused,
|
"reused_pass_jobs": reused,
|
||||||
"reused_pbf_files": reused_pbf,
|
"reused_pbf_files": reused_pbf,
|
||||||
|
|||||||
@@ -336,6 +336,22 @@ def finalize_release(
|
|||||||
decoded_features = 0
|
decoded_features = 0
|
||||||
decoded_layers = Counter()
|
decoded_layers = Counter()
|
||||||
duplicate_feature_ids = []
|
duplicate_feature_ids = []
|
||||||
|
|
||||||
|
s52_plan = dict(plan.get("s52") or {})
|
||||||
|
s52_runtime_enabled = bool(s52_plan.get("enabled")) and s52_plan.get("mariner_mode") == "runtime"
|
||||||
|
s52_runtime_failures: list[str] = []
|
||||||
|
s52_runtime_csp_counts = Counter()
|
||||||
|
s52_runtime_layer_counts = Counter()
|
||||||
|
s52_runtime_dependent_count = 0
|
||||||
|
s52_runtime_leaks = Counter()
|
||||||
|
s52_runtime_fact_counts = Counter()
|
||||||
|
s52_dynamic_final_fields = {
|
||||||
|
"s52_is_safety_contour",
|
||||||
|
"s52_sounding_prefix",
|
||||||
|
"s52_sounding_is_shallow",
|
||||||
|
"s52_isolated_danger",
|
||||||
|
"s52_display_base_promoted",
|
||||||
|
}
|
||||||
for path in pbf_files:
|
for path in pbf_files:
|
||||||
raw = path.read_bytes()
|
raw = path.read_bytes()
|
||||||
decoded = mapbox_vector_tile.decode(raw)
|
decoded = mapbox_vector_tile.decode(raw)
|
||||||
@@ -348,12 +364,45 @@ def finalize_release(
|
|||||||
pbf_rel = str(path.relative_to(pbf_root)).replace("\\", "/")
|
pbf_rel = str(path.relative_to(pbf_root)).replace("\\", "/")
|
||||||
for feat in feats:
|
for feat in feats:
|
||||||
fid = feat.get("id")
|
fid = feat.get("id")
|
||||||
|
props = feat.get("properties") or {}
|
||||||
|
if s52_runtime_enabled:
|
||||||
|
if props.get("s52_mariner_dependent") == 1:
|
||||||
|
s52_runtime_dependent_count += 1
|
||||||
|
s52_runtime_layer_counts[layer] += 1
|
||||||
|
csp = str(props.get("s52_runtime_csp") or "")
|
||||||
|
if not csp:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} missing s52_runtime_csp")
|
||||||
|
else:
|
||||||
|
s52_runtime_csp_counts[csp] += 1
|
||||||
|
for key in s52_dynamic_final_fields:
|
||||||
|
if key in props:
|
||||||
|
s52_runtime_leaks[key] += 1
|
||||||
|
if csp in {"DEPARE01", "DEPARE02"}:
|
||||||
|
if "depth_min_m" not in props:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} {csp} missing depth_min_m")
|
||||||
|
if "s52_area_color" in props:
|
||||||
|
s52_runtime_leaks["DEPARE:s52_area_color"] += 1
|
||||||
|
elif csp == "DEPCNT02" and "contour_depth_m" not in props:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} DEPCNT02 missing contour_depth_m")
|
||||||
|
elif csp == "SOUNDG02" and "s52_sounding_depth_m" not in props:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} SOUNDG02 missing s52_sounding_depth_m")
|
||||||
|
elif csp in {"OBSTRN04", "WRECKS02"} and "s52_hazard_depth_m" not in props:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} {csp} missing s52_hazard_depth_m")
|
||||||
|
elif "s52_runtime_csp" in props:
|
||||||
|
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} runtime CSP without mariner-dependent marker")
|
||||||
|
for key in (
|
||||||
|
"depth_min_m", "depth_max_m", "contour_depth_m",
|
||||||
|
"s52_sounding_depth_m", "s52_hazard_depth_m",
|
||||||
|
"s52_surrounding_max_drval1_m", "s52_surrounding_min_drval2_m",
|
||||||
|
):
|
||||||
|
if key in props:
|
||||||
|
s52_runtime_fact_counts[key] += 1
|
||||||
_observe_nav_name_conflict_feature(
|
_observe_nav_name_conflict_feature(
|
||||||
nav_name_conflict_state,
|
nav_name_conflict_state,
|
||||||
pbf_rel,
|
pbf_rel,
|
||||||
layer,
|
layer,
|
||||||
fid,
|
fid,
|
||||||
feat.get("properties") or {},
|
props,
|
||||||
)
|
)
|
||||||
if fid is None:
|
if fid is None:
|
||||||
continue
|
continue
|
||||||
@@ -402,6 +451,20 @@ def finalize_release(
|
|||||||
nav_name_conflict_state
|
nav_name_conflict_state
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if s52_runtime_leaks:
|
||||||
|
s52_runtime_failures.append(f"dynamic final state leaked into runtime PBF: {dict(s52_runtime_leaks)}")
|
||||||
|
s52_runtime_contract = {
|
||||||
|
"status": ("PASS" if not s52_runtime_failures else "FAIL") if s52_runtime_enabled else "NOT_APPLICABLE",
|
||||||
|
"enabled": s52_runtime_enabled,
|
||||||
|
"mariner_dependent_count": s52_runtime_dependent_count,
|
||||||
|
"runtime_csp_counts": dict(sorted(s52_runtime_csp_counts.items())),
|
||||||
|
"runtime_layer_counts": dict(sorted(s52_runtime_layer_counts.items())),
|
||||||
|
"fact_counts": dict(sorted(s52_runtime_fact_counts.items())),
|
||||||
|
"dynamic_final_leaks": dict(sorted(s52_runtime_leaks.items())),
|
||||||
|
"failure_count": len(s52_runtime_failures),
|
||||||
|
"failure_examples": s52_runtime_failures[:100],
|
||||||
|
}
|
||||||
|
|
||||||
failures = []
|
failures = []
|
||||||
if hard_gate:
|
if hard_gate:
|
||||||
failures.append("hard_gate_unmapped")
|
failures.append("hard_gate_unmapped")
|
||||||
@@ -417,6 +480,8 @@ def finalize_release(
|
|||||||
failures.append("duplicate_feature_ids_within_tile_layer")
|
failures.append("duplicate_feature_ids_within_tile_layer")
|
||||||
if style_contract is not None and style_contract.get("overall") != "PASS":
|
if style_contract is not None and style_contract.get("overall") != "PASS":
|
||||||
failures.append("style_or_sprite_contract")
|
failures.append("style_or_sprite_contract")
|
||||||
|
if s52_runtime_enabled and s52_runtime_contract.get("status") != "PASS":
|
||||||
|
failures.append("s52_runtime_contract")
|
||||||
|
|
||||||
result = {
|
result = {
|
||||||
"schema": RELEASE_SCHEMA,
|
"schema": RELEASE_SCHEMA,
|
||||||
@@ -431,6 +496,10 @@ def finalize_release(
|
|||||||
"mapping_sha256": plan["mapping_sha256"],
|
"mapping_sha256": plan["mapping_sha256"],
|
||||||
"policy_sha256": plan["policy_sha256"],
|
"policy_sha256": plan["policy_sha256"],
|
||||||
"builder_contract_version": plan.get("builder_contract_version"),
|
"builder_contract_version": plan.get("builder_contract_version"),
|
||||||
|
"production_pipeline_version": plan.get("production_pipeline_version"),
|
||||||
|
"s52": plan.get("s52"),
|
||||||
|
"aoi": plan.get("aoi"),
|
||||||
|
"s52_runtime_contract": s52_runtime_contract,
|
||||||
"style_sha256": style_sha,
|
"style_sha256": style_sha,
|
||||||
"sprite_json_sha256": sprite_sha,
|
"sprite_json_sha256": sprite_sha,
|
||||||
"gap_disposition_sha256": gap_disposition_sha,
|
"gap_disposition_sha256": gap_disposition_sha,
|
||||||
|
|||||||
Reference in New Issue
Block a user