feat: save PostgreSQL PostGIS CM95 baseline

This commit is contained in:
OpenAI Codex
2026-10-02 18:03:26 +08:00
commit a6dcd872bc
64 changed files with 189715 additions and 0 deletions

10
.env.example Normal file
View File

@@ -0,0 +1,10 @@
CM95_SOURCE_DIR=/root/sourceserver/cm95-data/CM93JAPAN
CM95_PG_HOST=192.168.x.x
CM95_PG_PORT=5432
CM95_PG_DATABASE=cm95_chart
CM95_PG_USER=cm95
CM95_PG_PASSWORD=change-me
CM95_PG_SSLMODE=prefer
CM95_PG_ADMIN_DATABASE=postgres
CM95_PG_CONNECT_TIMEOUT=20

9
.gitignore vendored Normal file
View File

@@ -0,0 +1,9 @@
.env
.venv/
build/
logs/
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.DS_Store

11
AGENTS.md Normal file
View File

@@ -0,0 +1,11 @@
# CM95 PostgreSQL/PostGIS 项目约束
1. 本项目为独立 PostgreSQL/PostGIS 实现,不读取、不修改、不迁移旧 MySQL 数据库。
2. /root/sourceserver/cm95 仅为只读参考源,不得修改其文件或 Git 状态。
3. 输入以 CM95 Japan / CM93JAPAN 原始数据为唯一事实源。
4. parser、geometry normalization、raw preservation、Cell 事务、SHA-256 skip/reimport 语义保持参考工程冻结行为。
5. Geometry 使用 PostGIS SRID 4326;空间索引使用 GiST。
6. PBF mapping、确定性 FID、tile policy、S-52 保持参考工程行为,数据库方言差异在访问层吸收。
7. 运行配置只使用 CM95_PG_* 与 CM95_SOURCE_DIR,不回退旧 MySQL 环境变量。
8. 全国全量导入前必须先通过单 Cell、小样本、DB audit 和单 tile PBF 验证。
9. 文档与日志不得记录数据库密码。

40
README.md Normal file
View File

@@ -0,0 +1,40 @@
# CM95 PostgreSQL/PostGIS Pipeline
独立的数据链:
CM95/CM93 Japan 原始数据
→ 冻结解析与几何标准化
→ PostgreSQL + PostGIS 母数据库
→ 确定性 FID / mapping / S-52
→ MVT/PBF
本工程不依赖 MySQL,也不会读取或修改旧 MySQL 数据库。
旧工程 /root/sourceserver/cm95 仅作为解析、几何与 PBF 行为参考。
## 环境
- Python 3.12
- PostgreSQL 14+
- PostGIS
- psycopg 3
## 基本流程
1. python3.12 -m venv .venv
2. 激活 .venv
3. pip install -r requirements.txt
4. cp .env.example .env
5. 编辑 CM95_PG_* 指向 NAS PostgreSQL
6. cm95pg env-check
7. cm95pg db-init --create-database
8. cm95pg db-check
9. cm95pg scan
10. cm95pg import
11. cm95pg audit --precision-samples 128
NAS 已预先建立数据库时,第 7 步不需要 --create-database。
## PBF
导入完成后先建立 FID manifest,再创建 production plan 并生成 PBF。
mapping、tile policy、S-52、sprite/style 继续沿用参考工程已经验证的配置。

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,18 @@
schema: navsea.cm95.production-policy/v1
version: v1.4.2-materialization-split
status: PRODUCTION_CANDIDATE
zooms: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
profile: delivery
workers: 4
max_attempts: 3
release_root: build/releases
plan_path: build/production/cm95_production_plan_v1.json
rules:
- 生产目标固定为 z0-z12。
- 高 zoom tile 的物理枚举只由 enumeration_scales 决定。
- scale_ladders 的粗比例尺只允许在已枚举 tile 内回退,不得扩张生产 footprint。
- mapping、tile policy、Dataset fingerprint、FID manifest 在计划创建后必须保持不变。
- PASS tile 自动断点续跑;RUNNING 状态在下次启动时恢复为 PENDING。
- FAILED tile 最多尝试 max_attempts 次;失败不会被静默跳过。
- 空 tile 可为 PASS 且不生成 PBF 文件;release manifest 只列真实产物。
- 发布前 finalize-production 必须满足 planned/report/decoded feature 三方一致。

View File

@@ -0,0 +1,96 @@
schema: navsea.cm95.tile-policy/v1
version: v1.3-japan-full-detail-g7500-at-z12
status: PRODUCTION_CANDIDATE
xyz: true
nominal_tile_pixels: 256
buffer_tile_pixels: 8
default_extent: 4096
extent_by_zoom:
'10': 4096
'11': 4096
'12': 1048576
enumeration_scales:
'0':
- 20000000
'1':
- 20000000
'2':
- 20000000
'3':
- 20000000
'4':
- 20000000
'5':
- 3000000
'6':
- 3000000
'7':
- 3000000
'8':
- 1000000
'9':
- 1000000
'10':
- 200000
'11':
- 50000
- 100000
'12':
- 7500
- 20000
- 50000
scale_ladders:
'0':
- 20000000
'1':
- 20000000
'2':
- 20000000
'3':
- 20000000
'4':
- 20000000
'5':
- 3000000
- 20000000
'6':
- 3000000
- 20000000
'7':
- 3000000
- 20000000
'8':
- 1000000
- 3000000
- 20000000
'9':
- 1000000
- 3000000
- 20000000
'10':
- 200000
- 1000000
- 3000000
- 20000000
'11':
- 50000
- 100000
- 200000
- 1000000
- 3000000
- 20000000
'12':
- 7500
- 20000
- 50000
- 100000
- 200000
- 1000000
- 3000000
- 20000000
notes:
- Japan full-detail production 仍保持 z0-z12;不新增 z13/z14。
- z12 enumeration 加入 1:7,500(G),避免正确 Japan 数据中的 G 级细节被生产策略直接忽略。
- z12 scale ladder 优先 7,500,再回退 20k/50k/100k/...。
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea/MapLibre 契约。
- enumeration_scales 仅决定物化范围;scale_ladders 仅决定 tile 内选择/回退。

View File

@@ -0,0 +1,102 @@
schema: navsea.cm95.tile-policy/v1
version: v1.4-japan-national-lowzoom-fallback
status: PRODUCTION_CANDIDATE
xyz: true
nominal_tile_pixels: 256
buffer_tile_pixels: 8
default_extent: 4096
extent_by_zoom:
'10': 4096
'11': 4096
'12': 1048576
enumeration_scales:
'0':
- 20000000
'1':
- 20000000
'2':
- 20000000
'3':
- 20000000
'4':
- 20000000
'5':
- 3000000
'6':
- 3000000
'7':
- 3000000
'8':
- 1000000
'9':
- 1000000
'10':
- 200000
'11':
- 50000
- 100000
'12':
- 7500
- 20000
- 50000
scale_ladders:
'0':
- 20000000
- 3000000
'1':
- 20000000
- 3000000
'2':
- 20000000
- 3000000
'3':
- 20000000
- 3000000
'4':
- 20000000
- 3000000
'5':
- 3000000
- 20000000
'6':
- 3000000
- 20000000
'7':
- 3000000
- 20000000
'8':
- 1000000
- 3000000
- 20000000
'9':
- 1000000
- 3000000
- 20000000
'10':
- 200000
- 1000000
- 3000000
- 20000000
'11':
- 50000
- 100000
- 200000
- 1000000
- 3000000
- 20000000
'12':
- 7500
- 20000
- 50000
- 100000
- 200000
- 1000000
- 3000000
- 20000000
notes:
- Japan nationwide aoi-complete: z0-z4 在 1:20M 不覆盖日本南部时允许回退 1:3M;不改变 z5-z12 细节策略。
- Japan full-detail production 仍保持 z0-z12;不新增 z13/z14。
- z12 enumeration 加入 1:7,500(G),避免正确 Japan 数据中的 G 级细节被生产策略直接忽略。
- z12 scale ladder 优先 7,500,再回退 20k/50k/100k/...。
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea/MapLibre 契约。
- enumeration_scales 仅决定物化范围;scale_ladders 仅决定 tile 内选择/回退。

View File

@@ -0,0 +1,51 @@
schema: navsea.cm95.tile-policy/v1
version: v1.2-production-materialization-split
status: PRODUCTION_CANDIDATE
xyz: true
nominal_tile_pixels: 256
buffer_tile_pixels: 8
default_extent: 4096
extent_by_zoom:
'10': 4096
'11': 4096
'12': 1048576
# enumeration_scales 决定哪些 coverage 可以创建物理 tile。
# 这里只使用与目标 zoom 匹配的原生/近原生比例尺,避免粗比例尺 coverage
# 把高 zoom 生产范围膨胀成百万级静态文件。
enumeration_scales:
'0': [20000000]
'1': [20000000]
'2': [20000000]
'3': [20000000]
'4': [20000000]
'5': [3000000]
'6': [3000000]
'7': [3000000]
'8': [1000000]
'9': [1000000]
'10': [200000]
'11': [50000, 100000]
'12': [20000, 50000]
# scale_ladders 只控制已经决定生成的 tile 内部如何选择数据。
# 粗比例尺仍可作为 tile 内回退,但不再扩张 tile 枚举范围。
scale_ladders:
'0': [20000000]
'1': [20000000]
'2': [20000000]
'3': [20000000]
'4': [20000000]
'5': [3000000, 20000000]
'6': [3000000, 20000000]
'7': [3000000, 20000000]
'8': [1000000, 3000000, 20000000]
'9': [1000000, 3000000, 20000000]
'10': [200000, 1000000, 3000000, 20000000]
'11': [50000, 100000, 200000, 1000000, 3000000, 20000000]
'12': [20000, 50000, 100000, 200000, 1000000, 3000000, 20000000]
notes:
- 生产目标为 z0-z12;Pilot 的 z10-z12 只用于验证。
- enumeration_scales 与 scale_ladders 必须分离。
- enumeration_scales 只决定是否物化 tile;scale_ladders 只决定 tile 内数据回退。
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea 契约。

View File

@@ -0,0 +1,18 @@
# CM95 PBF v1.4.5 定向重建 tile
# 来源:cm95_style_contract_unmatched_features_v1.json
10/912/366
10/913/366
10/916/327
11/1893/717
11/1893/718
11/1908/643
11/1908/645
11/1909/636
11/1909/639
11/1909/643
11/1910/633
12/3662/1409
12/3737/1466
12/3787/1435
12/3787/1436
12/3825/1376

View File

@@ -0,0 +1,22 @@
{
"schema": "navsea.cm95.s52-poc-config/v1",
"targets": [
"LIGHTS",
"LNDMRK",
"BCNLAT",
"BOYLAT",
"TOPMAR"
],
"palette": "DAY_BRIGHT",
"source_id": "navsea_delivery",
"source_layer": "navigation_marks",
"sprite_url": "./sprite",
"tiles_template": "https://REPLACE-ME/{z}/{x}/{y}.pbf",
"policy": {
"static_lookup": "materialize_in_builder",
"conditional_symbology": "explicit_builder_derived_or_runtime_renderer",
"production_release": "do_not_touch_in_poc"
},
"point_table": "Paper",
"sprite_scope": "all"
}

View File

@@ -0,0 +1,113 @@
{
"schema": "navsea.cm95.s52-runtime-capability-registry/v1",
"target": "kyushu_compare_web_debug",
"description": "Kyushu side-by-side browser target. IMPLEMENTED means the declared runtime kind has concrete target-side code evidence; PREVIEW_ONLY and NOT_IMPLEMENTED are non-compliant for strict portrayal parity.",
"capabilities": {
"RUNTIME_CA": {
"status": "IMPLEMENTED",
"fidelity": "S52_SCREEN_MM_SCAMIN_SCALE_AND_RULE_WIDTH",
"evidence": [
{
"path": "web/kyushu_compare.html",
"contains": [
"left-ca-overlay",
"drawLightArcFeature"
]
},
{
"path": "web/s52_runtime_executor.js",
"contains": [
"caScaleMetrics",
"requestedArcWidthMm",
"metresPerPixel"
]
}
]
},
"RUNTIME_DEPARE": {
"status": "IMPLEMENTED",
"fidelity": "ORDERED_RUNTIME_RULE_IR",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"RuntimeExecutor",
"dynamicRules"
]
}
]
},
"RUNTIME_DEPCNT": {
"status": "IMPLEMENTED",
"fidelity": "SAFE_LOOKUP_QUAPOS_AND_SIDE_EFFECTS",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"RuntimeExecutor",
"s52_safe_dashed_rule_chain",
"scamin:100000001"
]
}
]
},
"RUNTIME_SOUNDG": {
"status": "IMPLEMENTED",
"fidelity": "ORDERED_GLYPH_COMPOSITION_WITH_RESOURCE_PIVOTS",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"soundingRules"
,"runtime_icon_offset"
,"s52_sounding_depth_m"
]
}
]
},
"RUNTIME_OBSTRN": {
"status": "IMPLEMENTED",
"fidelity": "ORDERED_POINT_LINE_AREA_RULE_CHAIN",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"hazardRules",
"operationFeatures",
"OBSTRN04",
"ISODGR51"
]
}
]
},
"RUNTIME_WRECKS": {
"status": "IMPLEMENTED",
"fidelity": "ORDERED_POINT_LINE_AREA_RULE_CHAIN",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"hazardRules",
"operationFeatures",
"WRECKS02"
]
}
]
},
"RUNTIME_LIGHTS_STATE": {
"status": "IMPLEMENTED",
"fidelity": "PREVIOUS_POSITION_AND_DESCRIPTION",
"evidence": [
{
"path": "web/s52_runtime_executor.js",
"contains": [
"LightLabelState",
"lastPosition",
"lastDescription",
"yoff:1"
]
}
]
}
}
}

View File

@@ -0,0 +1,63 @@
{
"schema": "navsea.cm95.s52-style-compiler-config/v1",
"name": "CM95 S-52 Fukuoka compiled style v1",
"palette": "DAY_BRIGHT",
"sprite": "http://192.168.200.184:8877/s52-sprite-lc-ap-v3",
"background_color": "DEPDW",
"fallback_color": "NODTA",
"area_fill_opacity": 0.92,
"area_pattern_opacity": 0.85,
"line_width_fallback": 1.0,
"complex_line_width_px": 5.0,
"display_categories": ["Displaybase", "Standard", "Other"],
"source": {
"id": "cm95",
"type": "vector",
"tiles": ["http://192.168.200.184:8877/tiles-presentation-v10/{z}/{x}/{y}.pbf"],
"bounds": [130.341796875, 33.57801474614399, 130.4296875, 33.65120829920498],
"minzoom": 12,
"maxzoom": 12
},
"resources": {
"colors": "build/s52_resources_v2/base/s52_colors.json",
"symbol_metadata": "build/s52_resources_v2/base/s52_symbol_metadata.json",
"bitmap_pattern_metadata": "build/s52_resources_v2/base/s52_pattern_metadata.json",
"vector_pattern_metadata": "build/s52_resources_v2/ap/s52-sprite-ap-pattern-metadata.json",
"line_metadata": "build/s52_resources_v2/final/s52-sprite-full-line-metadata.json",
"sprite_index": "build/s52_resources_v2/final/s52-sprite-full.json",
"lookup_ir": "build/s52_presentation_ir_v2/lookup_rule_ir.json",
"resource_contract": "build/s52_resources_v2/resource_contract.json"
},
"source_layers": [
{"name": "anchor_caution_hazard_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "anchor_caution_hazard_point", "capabilities": ["symbol", "text"]},
{"name": "anchorage_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "anchorage_outline", "capabilities": ["line", "symbol", "text"]},
{"name": "anchorage_point", "capabilities": ["symbol", "text"]},
{"name": "baseline_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "baseline_line", "capabilities": ["line", "symbol", "text"]},
{"name": "bridge_structure", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "clearance_limit_line", "capabilities": ["line", "symbol", "text"]},
{"name": "clearance_limit_point", "capabilities": ["symbol", "text"]},
{"name": "cm95_unmapped", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "depth_contour", "capabilities": ["line", "text"]},
{"name": "facility_boundary_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "facility_boundary_outline", "capabilities": ["line", "symbol", "text"]},
{"name": "facility_boundary_point", "capabilities": ["symbol", "text"]},
{"name": "fixed_fishing_gear_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "land_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "navigation_hazard_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "navigation_hazard_outline", "capabilities": ["line", "symbol", "text"]},
{"name": "navigation_hazard_point", "capabilities": ["symbol", "text"]},
{"name": "navigation_marks", "capabilities": ["symbol", "text"]},
{"name": "onshore_structure_area", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "onshore_structure_point", "capabilities": ["symbol", "text"]},
{"name": "pilot_station_point", "capabilities": ["symbol", "text"]},
{"name": "place_label_land", "capabilities": ["symbol", "text"]},
{"name": "s52_areas", "capabilities": ["fill", "line", "symbol", "text"]},
{"name": "s52_lines", "capabilities": ["line", "symbol", "text"]},
{"name": "s52_points", "capabilities": ["symbol", "text"]},
{"name": "seabed_line", "capabilities": ["line", "symbol", "text"]},
{"name": "seabed_text_point", "capabilities": ["symbol", "text"]}
]
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,83 @@
# CM95 Japan PostgreSQL/PostGIS 实现说明
## 目标
新工程独立完成:
CM95 Japan 原始数据 → PostgreSQL/PostGIS → NavSea PBF。
旧 MySQL 工程只作为行为参考,不是运行依赖,也不会被新工程读取、修改或迁移。
## 保持不变的业务层
以下文件从当前参考工程复制并保持逐字一致:
- CM93 binary parser
- geometry normalization
- scanner / dictionary loader
- PBF mapping
- tile policy
- S-52 resolver
因此 WGS offset、Area/Coverage normalization、META/raw_geotype=16、raw preservation、确定性 FID 和 PBF mapping 继续沿用当前生产逻辑。
## PostgreSQL 数据层
数据库驱动使用 psycopg 3,空间能力使用 PostGIS。
主要对应关系:
- AUTO_INCREMENT → BIGSERIAL
- JSON → JSONB
- BLOB/MEDIUMBLOB → BYTEA
- TINYINT(1) → BOOLEAN
- Spatial index → PostGIS GiST
- geometry WKB → ST_GeomFromWKB(..., 4326)
- geometry readback → ST_AsBinary(...)
- upsert → ON CONFLICT ... DO UPDATE
- insert id → RETURNING id
所有标准化 Geometry 的 SRID 为 4326。
## PBF 空间查询
POINT / LINE / AREA 候选查询由 geometry 表的 GiST index 驱动,使用 bbox 运算符 && 配合 4326 tile envelope;同时继续保留 chart_feature 的数字 bbox 二次过滤。
SOUNDING 是 Point,因此同样使用 bbox 查询保持矩形 MBR 包含语义。PBF 的 Python 排序仍按 source_relative_path/source_record_index 执行,避免依赖数据库执行计划顺序。
## 当前源数据基线
2026-09-19 在 Linux 当前实际 /root/sourceserver/cm95-data/CM93JAPAN 扫描:
- Cells: 1492
- A: 4
- B: 15
- C: 82
- D: 354
- E: 767
- F: 247
- G: 22
- Z: 1
source-only 全量解析:
- Features: 1,282,431
- Attributes: 1,506,625
- Source parse failed: 0
- Unknown objects: 0
- Unknown attributes: 0
这套数据已不同于旧工程历史报告中的 784 Cell 基线。新 PostgreSQL 库应以当前 1492 Cell 源目录作为事实来源。
## 尚需真实 NAS 验证
代码已完成 PostgreSQL/PostGIS 方言切换,但在未配置 NAS 的 CM95_PG_* 连接参数前,不能声称以下步骤已经通过:
1. schema 在 NAS PostgreSQL/PostGIS 实际创建;
2. 单 Cell / 小样本真实导入;
3. 全量 1492 Cell 导入;
4. PostgreSQL DB audit;
5. 从 PostgreSQL 生成单 tile PBF;
6. 全国 PBF production/finalize。
这些验证必须针对实际 NAS 数据库执行。

27
pyproject.toml Normal file
View File

@@ -0,0 +1,27 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "cm95-postgresql-pipeline"
version = "0.1.0"
description = "CM95 Japan source -> PostgreSQL/PostGIS -> NavSea MVT/PBF pipeline"
readme = "README.md"
requires-python = ">=3.12,<3.13"
dependencies = [
"psycopg[binary]>=3.2,<4",
"python-dotenv>=1.0,<2",
"Shapely>=2.0,<3",
"PyYAML>=6.0,<7",
"mapbox-vector-tile>=2.2,<3",
]
[project.scripts]
cm95pg = "cm93db.cli:main"
cm95pbf = "cm95pbf.cli:main"
[tool.setuptools]
package-dir = {"" = "src"}
[tool.setuptools.packages.find]
where = ["src"]

5
requirements.txt Normal file
View File

@@ -0,0 +1,5 @@
psycopg[binary]>=3.2,<4
python-dotenv>=1.0,<2
Shapely>=2.0,<3
PyYAML>=6.0,<7
mapbox-vector-tile>=2.2,<3

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,84 @@
{
"schema": "navsea.cm95.s52-resource-contract/v1",
"status": "PASS",
"reference": {
"project": "OpenCPN",
"commit": "f7211b4e1874c453e37a7f190aa191c55a25c32e",
"chartsymbols_sha256": "c5678321ab3533605bb51f9376ad1909a50cedb4509afc34852d8d857a109de2",
"csp_source_sha256": "d094d9345fa76acf601ca2c1a48da7d65f06f3086b8f810e6d1ab9f8a5014ea4"
},
"defined_resource_counts": {
"SY": 1018,
"AP": 30,
"LC": 59,
"total": 1107
},
"active_resource_counts": {
"SY": 902,
"AP": 27,
"LC": 48,
"total": 977
},
"built_resource_counts": {
"SY": 1018,
"AP": 30,
"LC": 59,
"total": 1107
},
"defined_missing_from_bundle": {
"SY": [],
"AP": [],
"LC": []
},
"active_missing_from_bundle": {
"SY": [],
"AP": [],
"LC": []
},
"upstream_undefined_references": {
"SY": [
"BCNCON81",
"BOYLAT52",
"BOYLAT53",
"BOYLAT54",
"BOYLAT55",
"BOYLAT56",
"BOYSPH79",
"BOYSPR02",
"BOYSPR03",
"DANGER53",
"DGPS01DRFSTA01",
"ESSARE01",
"FLTHAZ02",
"LOWACC03",
"NEWOBJ 01",
"NEWOBJ01",
"PSSARE01",
"TOPSHP33",
"TOWERS74|",
"VEHTRF01",
"boyspp50",
"rdocal02",
"rdocal03"
],
"AP": [],
"LC": [
"ARCSLN01",
"NEWOBJ01"
]
},
"undefined_reference_fallback": {
"SY": "QUESMRK1",
"LC": "QUESMRK1 symbol fallback",
"AP": "QUESMRK1 pattern fallback",
"source": "OpenCPN S52PLIB StringToRules"
},
"artifact_hashes": {
"symbol_metadata_sha256": "5639de5d512b639c143d2ce88e209135e17640de094fdc63752ee6546806e77a",
"bitmap_pattern_metadata_sha256": "738c4b8b0c387d8d6593d77e79fc8165c0d157a205f6dca197f6f5722155a875",
"vector_pattern_metadata_sha256": "263bae788647b1728534e681769da2d24a4cc5ee086a056c8b1e36a7c4c3e78e",
"line_metadata_sha256": "712e509cd70c89cdbed976797f23b3043590023cf9bece0cba4b900b100fa071",
"sprite_index_sha256": "51519fb9a64f11dc100f894b2b54d14f56770ac9d511f15ae574ff5867af857d"
},
"failures": []
}

17
samples/README.md Normal file
View File

@@ -0,0 +1,17 @@
# 样本数据
本 ZIP 不重新分发用户的 CM93 数据。
建议将已经验证的小样本放在项目外,例如:
`D:\Ship\CM93V2\CM93JAPAN_TEST\04501020\`
然后运行:
`cm93db scan D:\Ship\CM93V2\CM93JAPAN_TEST\04501020`
和:
`cm93db inspect-dicts D:\Ship\CM93V2\CM93JAPAN_TEST`
正式 Codex 开发时,以真实样本作为格式回归基线。

View File

@@ -0,0 +1,124 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$PROJECT_ROOT"
if [[ ! -f .env ]]; then
echo "ERROR: .env 不存在。请从 .env.example 复制并填写 NAS PostgreSQL/PostGIS 连接参数。" >&2
exit 2
fi
if [[ ! -x .venv/bin/python ]]; then
echo "ERROR: .venv 不存在。先执行 python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/pip install -e ." >&2
exit 2
fi
source .venv/bin/activate
set -a
source .env
set +a
export PYTHONUNBUFFERED=1
SOURCE_ROOT="${CM95_SOURCE_DIR:-/root/sourceserver/cm95-data/CM93JAPAN}"
ZOOMS="${CM95_PBF_ZOOMS:-0-12}"
WORKERS="${CM95_PBF_WORKERS:-4}"
MANIFEST="${CM95_FID_MANIFEST:-build/cm95_fid_manifest_v1.sqlite}"
PLAN="${CM95_PRODUCTION_PLAN:-build/production/cm95_postgresql_japan_full_plan.json}"
RELEASE_ROOT="${CM95_RELEASE_ROOT:-build/releases-postgresql-japan}"
MAPPING="${CM95_PBF_MAPPING:-config/mappings/cm95_to_navsea_v1.yaml}"
POLICY="${CM95_PBF_POLICY:-config/pbf/cm95_tile_policy_japan_national_v1.yaml}"
AOI_NAME="${CM95_AOI_NAME:-japan-national-v1}"
AOI_BBOX="${CM95_AOI_BBOX:-122.0,20.0,154.5,46.0}"
TILE_ENUMERATION_MODE="${CM95_TILE_ENUMERATION_MODE:-aoi-complete}"
S52_LOOKUPS="${CM95_S52_LOOKUPS:-resources/s52/lookup_rule_ir.json}"
S52_RESOURCE_CONTRACT="${CM95_S52_RESOURCE_CONTRACT:-resources/s52/resource_contract.json}"
echo "== STEP 1: PostgreSQL/PostGIS environment =="
cm95pg env-check
echo "== STEP 2: schema =="
if [[ "${CM95_CREATE_DATABASE:-0}" == "1" ]]; then
cm95pg db-init --create-database
else
cm95pg db-init
fi
cm95pg db-check
echo "== STEP 3: source scan =="
cm95pg scan "$SOURCE_ROOT"
echo "== STEP 4: import CM95 Japan to PostgreSQL/PostGIS =="
cm95pg import "$SOURCE_ROOT"
echo "== STEP 5: source/database audit =="
cm95pg audit "$SOURCE_ROOT" \
--precision-samples "${CM95_PRECISION_SAMPLES:-1024}" \
--report logs/postgresql_full_audit.json
echo "== STEP 6: resolve dataset id =="
DATASET_ID="$(SOURCE_ROOT="$SOURCE_ROOT" python - <<'PY'
import os
from pathlib import Path
from cm93db.config import load_config
from cm93db.database import connect
root = str(Path(os.environ["SOURCE_ROOT"]).resolve())
with connect(load_config()) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT id FROM chart_dataset "
"WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1",
(root,),
)
row = cur.fetchone()
if row is None:
raise SystemExit("dataset not found after import")
print(int(row["id"]))
PY
)"
echo "Dataset ID: $DATASET_ID"
echo "== STEP 7: deterministic FID manifest =="
cm95pbf build-fid-manifest \
--dataset-id "$DATASET_ID" \
--manifest "$MANIFEST" \
--rebuild-fid-manifest
echo "== STEP 8: production plan =="
cm95pbf plan-production \
--dataset-id "$DATASET_ID" \
--zooms "$ZOOMS" \
--profile delivery \
--mapping "$MAPPING" \
--policy "$POLICY" \
--manifest "$MANIFEST" \
--aoi-name "$AOI_NAME" \
--aoi-bbox "$AOI_BBOX" \
--tile-enumeration-mode "$TILE_ENUMERATION_MODE" \
--s52-lookups "$S52_LOOKUPS" \
--s52-resource-contract "$S52_RESOURCE_CONTRACT" \
--s52-point-table Paper \
--s52-boundary-table Plain \
--s52-mariner-mode runtime \
--tile-enumeration-mode coverage \
--output "$PLAN"
echo "== STEP 9: PBF production =="
cm95pbf build-production \
--plan "$PLAN" \
--release-root "$RELEASE_ROOT" \
--workers "$WORKERS" \
--max-attempts 3
echo "== STEP 10: production status =="
cm95pbf status-production \
--plan "$PLAN" \
--release-root "$RELEASE_ROOT"
echo "== STEP 11: final release audit =="
cm95pbf finalize-production \
--plan "$PLAN" \
--release-root "$RELEASE_ROOT" \
--skip-style-contract
echo "CM95 Japan PostgreSQL/PostGIS -> PBF pipeline: PASS"

View File

@@ -0,0 +1,50 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT"
[[ -f .env ]] || { echo "missing .env" >&2; exit 2; }
source .venv/bin/activate
set -a; source .env; set +a
export PYTHONUNBUFFERED=1
MANIFEST="${CM95_FID_MANIFEST:-build/cm95_fid_manifest_v1.sqlite}"
PLAN="${CM95_COMPARE_PLAN:-build/production/cm95_postgresql_kyushu_compare_plan.json}"
RELEASE_ROOT="${CM95_COMPARE_RELEASE_ROOT:-build/releases-postgresql-compare}"
MYSQL_RELEASE="${CM95_MYSQL_COMPARE_RELEASE:-/root/sourceserver/cm95/build/releases-s52-v2/cm95-d1-62a85bd40d3a-2b2e7d976a43}"
WORKERS="${CM95_PBF_WORKERS:-4}"
S52_LOOKUPS="${CM95_S52_LOOKUPS:-resources/s52/lookup_rule_ir.json}"
S52_CONTRACT="${CM95_S52_RESOURCE_CONTRACT:-resources/s52/resource_contract.json}"
DATASET_ID="$(python - <<'PY'
from cm93db.config import load_config
from cm93db.database import connect
with connect(load_config()) as c:
with c.cursor() as cur:
cur.execute("SELECT id FROM chart_dataset WHERE source_type=%s ORDER BY id LIMIT 1",("CM93",))
row=cur.fetchone()
if not row: raise SystemExit("dataset not found")
print(int(row["id"]))
PY
)"
cm95pbf build-fid-manifest --dataset-id "$DATASET_ID" --manifest "$MANIFEST" --rebuild-fid-manifest
cm95pbf plan-production --dataset-id "$DATASET_ID" --zooms 5-12 --profile delivery \
--manifest "$MANIFEST" --s52-lookups "$S52_LOOKUPS" --s52-resource-contract "$S52_CONTRACT" \
--s52-point-table Paper --s52-boundary-table Plain --s52-mariner-mode runtime \
--aoi-name kyushu-main-nearby-v1 --aoi-bbox 128.0,29.5,132.5,34.8 \
--tile-enumeration-mode aoi-complete --output "$PLAN"
cm95pbf build-production --plan "$PLAN" --release-root "$RELEASE_ROOT" --workers "$WORKERS" --max-attempts 3
cm95pbf status-production --plan "$PLAN" --release-root "$RELEASE_ROOT"
cm95pbf finalize-production --plan "$PLAN" --release-root "$RELEASE_ROOT" --skip-style-contract
RELEASE_ID="$(python - "$PLAN" <<'PY'
import json,sys
print(json.load(open(sys.argv[1],encoding="utf-8"))["release_id"])
PY
)"
PG_PBF="$RELEASE_ROOT/$RELEASE_ID/pbf/delivery"
MYSQL_PBF="$MYSQL_RELEASE/pbf/delivery"
python tools/compare_pbf_trees.py "$MYSQL_PBF" "$PG_PBF" --output logs/mysql_postgresql_kyushu_pbf_compare.json
echo "POSTGRES_RELEASE_ID=$RELEASE_ID"
echo "POSTGRES_PBF=$PG_PBF"
echo "MYSQL_PBF=$MYSQL_PBF"

View File

@@ -0,0 +1,46 @@
#!/usr/bin/env bash
set -euo pipefail
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$PROJECT_ROOT"
if [[ ! -f .env ]]; then
echo "ERROR: .env 不存在;请填写 NAS PostgreSQL/PostGIS 参数。" >&2
exit 2
fi
source .venv/bin/activate
SOURCE_ROOT="${CM95_SOURCE_DIR:-/root/sourceserver/cm95-data/CM93JAPAN}"
LIMIT="${CM95_SMOKE_CELL_LIMIT:-20}"
MANIFEST="build/smoke_fid_manifest.sqlite"
cm95pg env-check
cm95pg db-init
cm95pg db-check
cm95pg scan "$SOURCE_ROOT"
cm95pg import "$SOURCE_ROOT" --limit "$LIMIT"
cm95pg audit "$SOURCE_ROOT" --limit "$LIMIT" --precision-samples 20 --report logs/postgresql_smoke_audit.json
DATASET_ID="$(SOURCE_ROOT="$SOURCE_ROOT" python - <<'PY'
import os
from pathlib import Path
from cm93db.config import load_config
from cm93db.database import connect
root = str(Path(os.environ['SOURCE_ROOT']).resolve())
with connect(load_config()) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT id FROM chart_dataset WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1",
(root,),
)
row = cur.fetchone()
if row is None:
raise SystemExit('dataset not found')
print(int(row['id']))
PY
)"
cm95pbf build-fid-manifest --dataset-id "$DATASET_ID" --manifest "$MANIFEST" --rebuild-fid-manifest
cm95pbf build-tile --dataset-id "$DATASET_ID" --z 10 --x 914 --y 385 --profile delivery --manifest "$MANIFEST" --output-root build/smoke-pbf --report-root build/smoke-reports
echo "PostgreSQL/PostGIS smoke: PASS"

View File

@@ -0,0 +1,226 @@
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE TABLE IF NOT EXISTS chart_dataset (
id BIGSERIAL PRIMARY KEY,
source_type VARCHAR(16) NOT NULL DEFAULT 'CM93',
name VARCHAR(255) NOT NULL,
source_root TEXT NOT NULL,
source_fingerprint CHAR(64),
parser_version VARCHAR(32) NOT NULL,
normalized_srid INTEGER NOT NULL DEFAULT 4326,
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_dataset_source_type ON chart_dataset(source_type);
CREATE TABLE IF NOT EXISTS chart_dictionary_object (
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
object_class_id SMALLINT NOT NULL,
object_class VARCHAR(16) NOT NULL,
geometry_code VARCHAR(8),
display_code VARCHAR(8),
description VARCHAR(255),
raw_line TEXT NOT NULL,
PRIMARY KEY (dataset_id, object_class_id)
);
CREATE INDEX IF NOT EXISTS idx_dict_object_class ON chart_dictionary_object(object_class);
CREATE TABLE IF NOT EXISTS chart_dictionary_attribute (
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
attribute_id SMALLINT NOT NULL,
attribute_name VARCHAR(16) NOT NULL,
source_type_name VARCHAR(32) NOT NULL,
decoder_type CHAR(1) NOT NULL,
min_value DOUBLE PRECISION,
max_value DOUBLE PRECISION,
raw_line TEXT NOT NULL,
PRIMARY KEY (dataset_id, attribute_id)
);
CREATE INDEX IF NOT EXISTS idx_dict_attribute_name ON chart_dictionary_attribute(attribute_name);
CREATE TABLE IF NOT EXISTS chart_cell (
id BIGSERIAL PRIMARY KEY,
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
source_relative_path VARCHAR(1024) NOT NULL,
source_file_name VARCHAR(255) NOT NULL,
source_size BIGINT NOT NULL,
source_sha256 CHAR(64),
scale_code CHAR(1) NOT NULL,
native_scale INTEGER NOT NULL,
cell_name VARCHAR(64) NOT NULL,
prolog_word0 INTEGER,
table1_length BIGINT,
table2_length BIGINT,
easting_min DOUBLE PRECISION,
northing_min DOUBLE PRECISION,
easting_max DOUBLE PRECISION,
northing_max DOUBLE PRECISION,
bbox_min_lon DOUBLE PRECISION,
bbox_min_lat DOUBLE PRECISION,
bbox_max_lon DOUBLE PRECISION,
bbox_max_lat DOUBLE PRECISION,
vector_record_count INTEGER,
vector_point_count BIGINT,
point2d_count INTEGER,
point3d_count INTEGER,
feature_count INTEGER,
attribute_count BIGINT,
import_status VARCHAR(16) NOT NULL DEFAULT 'DISCOVERED'
CHECK (import_status IN ('DISCOVERED','IMPORTING','DONE','FAILED')),
error_message TEXT,
imported_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (dataset_id, source_relative_path)
);
CREATE INDEX IF NOT EXISTS idx_cell_scale ON chart_cell(dataset_id, scale_code);
CREATE INDEX IF NOT EXISTS idx_cell_status ON chart_cell(dataset_id, import_status);
CREATE TABLE IF NOT EXISTS chart_coverage (
id BIGSERIAL PRIMARY KEY,
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
source_record_index INTEGER NOT NULL,
object_class_id SMALLINT NOT NULL,
compilation_scale INTEGER,
horizontal_datum INTEGER,
vertical_datum INTEGER,
wgsox DOUBLE PRECISION,
wgsoy DOUBLE PRECISION,
attributes_json JSONB,
geom geometry(Geometry,4326) NOT NULL,
geometry_repaired BOOLEAN NOT NULL DEFAULT FALSE,
geometry_issue VARCHAR(255),
UNIQUE (cell_id, source_record_index)
);
CREATE INDEX IF NOT EXISTS idx_cov_cell ON chart_coverage(cell_id);
CREATE INDEX IF NOT EXISTS spx_cov_geom ON chart_coverage USING GIST(geom);
CREATE TABLE IF NOT EXISTS chart_feature (
id BIGSERIAL PRIMARY KEY,
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
coverage_id BIGINT REFERENCES chart_coverage(id) ON DELETE SET NULL,
source_record_index INTEGER NOT NULL,
object_class_id SMALLINT,
object_class VARCHAR(16) NOT NULL,
geometry_type VARCHAR(16) NOT NULL
CHECK (geometry_type IN ('POINT','LINE','AREA','SOUNDING','META','UNKNOWN')),
scale_code CHAR(1) NOT NULL,
native_scale INTEGER NOT NULL,
raw_geotype SMALLINT,
normalized_srid INTEGER NOT NULL DEFAULT 4326,
bbox_min_lon DOUBLE PRECISION,
bbox_min_lat DOUBLE PRECISION,
bbox_max_lon DOUBLE PRECISION,
bbox_max_lat DOUBLE PRECISION,
attributes_json JSONB,
raw_feature_blob BYTEA,
raw_attribute_blob BYTEA,
geometry_repaired BOOLEAN NOT NULL DEFAULT FALSE,
geometry_issue VARCHAR(255),
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (cell_id, source_record_index)
);
CREATE INDEX IF NOT EXISTS idx_feature_class ON chart_feature(dataset_id, object_class);
CREATE INDEX IF NOT EXISTS idx_feature_scale ON chart_feature(dataset_id, native_scale);
CREATE INDEX IF NOT EXISTS idx_feature_cell ON chart_feature(cell_id);
CREATE INDEX IF NOT EXISTS idx_feature_coverage ON chart_feature(coverage_id);
CREATE TABLE IF NOT EXISTS chart_feature_attribute (
id BIGSERIAL PRIMARY KEY,
feature_id BIGINT NOT NULL REFERENCES chart_feature(id) ON DELETE CASCADE,
ordinal_no SMALLINT NOT NULL,
attribute_id SMALLINT NOT NULL,
attribute_name VARCHAR(16) NOT NULL,
value_type VARCHAR(16) NOT NULL
CHECK (value_type IN ('BYTE','INT','LONG','REAL','WORD10','STRING','LIST','COMPLEX','UNKNOWN')),
value_int BIGINT,
value_real DOUBLE PRECISION,
value_text TEXT,
value_json JSONB,
raw_value_blob BYTEA,
UNIQUE (feature_id, ordinal_no)
);
CREATE INDEX IF NOT EXISTS idx_feature_attr_name ON chart_feature_attribute(attribute_name);
CREATE INDEX IF NOT EXISTS idx_feature_attr_id ON chart_feature_attribute(attribute_id);
CREATE TABLE IF NOT EXISTS chart_point_geometry (
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
geom geometry(Point,4326) NOT NULL
);
CREATE INDEX IF NOT EXISTS spx_point_geom ON chart_point_geometry USING GIST(geom);
CREATE TABLE IF NOT EXISTS chart_line_geometry (
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
geom geometry(Geometry,4326) NOT NULL
);
CREATE INDEX IF NOT EXISTS spx_line_geom ON chart_line_geometry USING GIST(geom);
CREATE TABLE IF NOT EXISTS chart_area_geometry (
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
geom geometry(Geometry,4326) NOT NULL
);
CREATE INDEX IF NOT EXISTS spx_area_geom ON chart_area_geometry USING GIST(geom);
CREATE TABLE IF NOT EXISTS chart_sounding_point (
id BIGSERIAL PRIMARY KEY,
feature_id BIGINT NOT NULL REFERENCES chart_feature(id) ON DELETE CASCADE,
point_index INTEGER NOT NULL,
raw_x BIGINT NOT NULL,
raw_y BIGINT NOT NULL,
raw_z BIGINT NOT NULL,
depth DOUBLE PRECISION NOT NULL,
geom geometry(Point,4326) NOT NULL,
UNIQUE (feature_id, point_index)
);
CREATE INDEX IF NOT EXISTS idx_sounding_depth ON chart_sounding_point(depth);
CREATE INDEX IF NOT EXISTS spx_sounding_geom ON chart_sounding_point USING GIST(geom);
CREATE TABLE IF NOT EXISTS chart_raw_record (
id BIGSERIAL PRIMARY KEY,
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
record_kind VARCHAR(16) NOT NULL
CHECK (record_kind IN ('HEADER','VECTOR','POINT2D','POINT3D','FEATURE','ATTRIBUTE','OTHER')),
source_record_index INTEGER NOT NULL DEFAULT 0,
raw_blob BYTEA NOT NULL,
raw_metadata_json JSONB
);
CREATE INDEX IF NOT EXISTS idx_raw_cell_kind
ON chart_raw_record(cell_id, record_kind, source_record_index);
CREATE TABLE IF NOT EXISTS chart_import_job (
id BIGSERIAL PRIMARY KEY,
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
status VARCHAR(16) NOT NULL DEFAULT 'RUNNING'
CHECK (status IN ('RUNNING','DONE','FAILED','CANCELLED')),
started_at TIMESTAMPTZ NOT NULL,
finished_at TIMESTAMPTZ,
cells_discovered INTEGER NOT NULL DEFAULT 0,
cells_done INTEGER NOT NULL DEFAULT 0,
cells_failed INTEGER NOT NULL DEFAULT 0,
features_source BIGINT NOT NULL DEFAULT 0,
features_imported BIGINT NOT NULL DEFAULT 0,
attributes_source BIGINT NOT NULL DEFAULT 0,
attributes_imported BIGINT NOT NULL DEFAULT 0,
unknown_object_count BIGINT NOT NULL DEFAULT 0,
unknown_attribute_count BIGINT NOT NULL DEFAULT 0,
geometry_invalid_count BIGINT NOT NULL DEFAULT 0,
geometry_repaired_count BIGINT NOT NULL DEFAULT 0,
summary_json JSONB
);
CREATE INDEX IF NOT EXISTS idx_job_dataset ON chart_import_job(dataset_id, started_at);
CREATE TABLE IF NOT EXISTS chart_import_event (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT REFERENCES chart_import_job(id) ON DELETE CASCADE,
cell_id BIGINT REFERENCES chart_cell(id) ON DELETE CASCADE,
level VARCHAR(16) NOT NULL CHECK (level IN ('DEBUG','INFO','WARNING','ERROR')),
event_code VARCHAR(64) NOT NULL,
message TEXT NOT NULL,
details_json JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_event_job ON chart_import_event(job_id, id);
CREATE INDEX IF NOT EXISTS idx_event_cell ON chart_import_event(cell_id, id);
CREATE INDEX IF NOT EXISTS idx_event_code ON chart_import_event(event_code);

1
src/cm93db/__init__.py Normal file
View File

@@ -0,0 +1 @@
__version__ = "0.1.0"

299
src/cm93db/audit.py Normal file
View File

@@ -0,0 +1,299 @@
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from collections import defaultdict
import math
from typing import Any
from shapely import from_wkb
from .cm93_binary import parse_cell
from .config import DbConfig
from .database import (
connect,
assert_postgresql_compatible,
server_version,
as_wkb_sql,
)
from .dictionaries import load_object_dictionary, load_attribute_dictionary
from .geometry import build_coverages, build_feature
from .scanner import scan_cm93, CellFile
@dataclass(frozen=True)
class SourceCellStat:
rel: str
sha256: str
file_size: int
scale: str
features: int
attributes: int
points: int
lines: int
areas: int
coverages: int
sounding_features: int
sounding_points: int
unknown_objects: int
unknown_attributes: int
@dataclass
class PrecisionSample:
rel: str
record_index: int
geometry_type: str
expected_wkb: bytes
point_index: int | None = None
@dataclass
class AuditResult:
source_cells: int = 0
source_parse_failed: int = 0
db_missing_cells: int = 0
db_not_done_cells: int = 0
count_mismatches: int = 0
hash_mismatches: int = 0
unknown_objects: int = 0
unknown_attributes: int = 0
precision_checked: int = 0
precision_failed: int = 0
max_precision_delta_deg: float = 0.0
raw_preservation_mismatches: int = 0
geometry_validity_issues: int = 0
messages: list[str] = field(default_factory=list)
@property
def passed(self) -> bool:
return not any((self.source_parse_failed,self.db_missing_cells,self.db_not_done_cells,self.count_mismatches,self.hash_mismatches,self.raw_preservation_mismatches,self.unknown_objects,self.unknown_attributes,self.precision_failed))
def as_dict(self) -> dict[str, Any]:
return {
"passed": self.passed,
"source_cells": self.source_cells,
"source_parse_failed": self.source_parse_failed,
"db_missing_cells": self.db_missing_cells,
"db_not_done_cells": self.db_not_done_cells,
"count_mismatches": self.count_mismatches,
"hash_mismatches": self.hash_mismatches,
"raw_preservation_mismatches": self.raw_preservation_mismatches,
"geometry_validity_issues": self.geometry_validity_issues,
"unknown_objects": self.unknown_objects,
"unknown_attributes": self.unknown_attributes,
"precision_checked": self.precision_checked,
"precision_failed": self.precision_failed,
"max_precision_delta_deg": self.max_precision_delta_deg,
"messages": self.messages,
}
def _rel(cf: CellFile) -> str:
return str(cf.relative_path).replace("/", "\\")
def _select_cells(scan, limit: int | None, scales: set[str] | None, cell_name: str | None):
cells=list(scan.cells)
if scales:
cells=[c for c in cells if c.scale_code in scales]
if cell_name:
t=cell_name.lower()
cells=[c for c in cells if c.path.name.lower()==t or c.path.stem.lower()==t]
if limit is not None:
cells=cells[:limit]
return cells
def _source_stat(cell, cf: CellFile) -> SourceCellStat:
p=l=a=cov=sf=sp=0
for f in cell.features:
if f.object_class == "_m_sor":
cov += 1
continue
if f.geometry_primitive == 1: p += 1
elif f.geometry_primitive == 2: l += 1
elif f.geometry_primitive == 4: a += 1
elif f.geometry_primitive == 8:
sf += 1
if f.point3d_index is not None:
sp += len(cell.point3d[f.point3d_index].points)
return SourceCellStat(
rel=_rel(cf),sha256=cell.sha256,file_size=cell.file_size,scale=cf.scale_code,features=len(cell.features),attributes=cell.attribute_count,
points=p,lines=l,areas=a,coverages=cov,sounding_features=sf,sounding_points=sp,
unknown_objects=len(cell.unknown_object_ids),unknown_attributes=len(cell.unknown_attribute_ids),
)
def collect_source(root: Path, limit: int | None = None, scales: set[str] | None = None, cell_name: str | None = None,
precision_samples: int = 0, validate_geometry: bool = False):
scan=scan_cm93(root)
if not scan.object_dictionary or not scan.attribute_dictionary:
raise RuntimeError("CM93 字典未找到")
objs={x.object_id:x for x in load_object_dictionary(scan.object_dictionary)}
attrs={x.attribute_id:x for x in load_attribute_dictionary(scan.attribute_dictionary)}
cells=_select_cells(scan,limit,scales,cell_name)
stats: dict[str,SourceCellStat]={}
failures: list[str]=[]
samples: list[PrecisionSample]=[]
geom_issues=0
quota=max(1, math.ceil(precision_samples / max(1,len({c.scale_code for c in cells})))) if precision_samples else 0
per_scale=defaultdict(int)
for n,cf in enumerate(cells,1):
try:
cell=parse_cell(cf.path,objs,attrs)
st=_source_stat(cell,cf); stats[st.rel]=st
want = precision_samples and per_scale[cf.scale_code] < quota and len(samples) < precision_samples
if want or validate_geometry:
covs=build_coverages(cell)
for f in cell.features:
b=build_feature(cell,f,covs,validate=validate_geometry)
if validate_geometry and b.geometry_issue:
geom_issues += 1
if want and per_scale[cf.scale_code] < quota and len(samples) < precision_samples:
if b.geom is not None and b.geometry_type in {"POINT","LINE","AREA"}:
samples.append(PrecisionSample(st.rel,f.index,b.geometry_type,b.geom.wkb))
per_scale[cf.scale_code]+=1
elif b.geometry_type=="SOUNDING" and b.soundings:
samples.append(PrecisionSample(st.rel,f.index,"SOUNDING",b.soundings[0][4].wkb,0))
per_scale[cf.scale_code]+=1
if n % 250 == 0 or n == len(cells):
print(f"source audit [{n}/{len(cells)}]")
except Exception as e:
failures.append(f"{_rel(cf)}: {e}")
return stats,failures,samples,geom_issues
def _group_counts(conn, sql: str) -> dict[int,int]:
with conn.cursor() as cur:
cur.execute(sql)
return {int(r["cell_id"]):int(r["n"]) for r in cur.fetchall()}
def _dataset_id(conn, root: Path) -> int | None:
with conn.cursor() as cur:
cur.execute("SELECT id FROM chart_dataset WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1", (str(root.resolve()),))
r=cur.fetchone(); return int(r["id"]) if r else None
def _precision_db_geom(conn, cell_id: int, sample: PrecisionSample):
if sample.geometry_type == "SOUNDING":
sql=(f"SELECT {as_wkb_sql('s.geom')} AS wkb FROM chart_feature f JOIN chart_sounding_point s ON s.feature_id=f.id "
"WHERE f.cell_id=%s AND f.source_record_index=%s AND s.point_index=%s")
params=(cell_id,sample.record_index,sample.point_index or 0)
else:
table={"POINT":"chart_point_geometry","LINE":"chart_line_geometry","AREA":"chart_area_geometry"}[sample.geometry_type]
sql=(f"SELECT {as_wkb_sql('g.geom')} AS wkb FROM chart_feature f JOIN {table} g ON g.feature_id=f.id "
"WHERE f.cell_id=%s AND f.source_record_index=%s")
params=(cell_id,sample.record_index)
with conn.cursor() as cur:
cur.execute(sql,params); r=cur.fetchone()
return bytes(r["wkb"]) if r and r["wkb"] is not None else None
def run_audit(cfg: DbConfig, root: Path, limit: int | None = None, scales: set[str] | None = None, cell_name: str | None = None,
precision_samples: int = 128, tolerance_deg: float = 1e-12, validate_geometry: bool = False, source_only: bool = False, require_raw: bool = True) -> AuditResult:
result=AuditResult()
stats,failures,samples,geom_issues=collect_source(root,limit,scales,cell_name,precision_samples if not source_only else 0,validate_geometry)
result.source_cells=len(stats)+len(failures)
result.source_parse_failed=len(failures)
result.messages.extend(f"SOURCE_PARSE_FAILED {m}" for m in failures[:100])
result.unknown_objects=sum(s.unknown_objects for s in stats.values())
result.unknown_attributes=sum(s.unknown_attributes for s in stats.values())
print(f"Source cells parsed: {len(stats)}, failed: {len(failures)}")
print(f"Source features: {sum(s.features for s in stats.values())}")
print(f"Source attributes: {sum(s.attributes for s in stats.values())}")
result.geometry_validity_issues=geom_issues
if validate_geometry:
print(f"Source geometry issues (Shapely validity/structure): {geom_issues}")
if source_only:
return result
conn=connect(cfg)
try:
assert_postgresql_compatible(server_version(conn))
did=_dataset_id(conn,root)
if did is None:
result.db_missing_cells=len(stats)
result.messages.append("数据库中找不到该 source_root 对应 dataset")
return result
with conn.cursor() as cur:
cur.execute("SELECT id,source_relative_path,import_status,source_sha256,feature_count,attribute_count FROM chart_cell WHERE dataset_id=%s", (did,))
rows=cur.fetchall()
db_by_rel={str(r["source_relative_path"]):r for r in rows}
cellids=[int(r["id"]) for r in rows]
fcnt=_group_counts(conn,f"SELECT cell_id,COUNT(*) n FROM chart_feature WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
acnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_feature_attribute a JOIN chart_feature f ON f.id=a.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
pcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_point_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
lcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_line_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
arcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_area_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
ccnt=_group_counts(conn,f"SELECT cell_id,COUNT(*) n FROM chart_coverage WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
scnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_sounding_point s JOIN chart_feature f ON f.id=s.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
rawbytes=_group_counts(conn,f"SELECT cell_id,SUM(OCTET_LENGTH(raw_blob)) n FROM chart_raw_record WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
for rel,s in stats.items():
r=db_by_rel.get(rel)
if not r:
result.db_missing_cells+=1; result.messages.append(f"DB_MISSING {rel}"); continue
cid=int(r["id"])
if r["import_status"]!="DONE":
result.db_not_done_cells+=1; result.messages.append(f"DB_NOT_DONE {rel} status={r['import_status']}")
if r.get("source_sha256") and str(r["source_sha256"]) != s.sha256:
result.hash_mismatches+=1; result.messages.append(f"HASH_MISMATCH {rel}")
actual=(fcnt.get(cid,0),acnt.get(cid,0),pcnt.get(cid,0),lcnt.get(cid,0),arcnt.get(cid,0),ccnt.get(cid,0),scnt.get(cid,0))
expected=(s.features,s.attributes,s.points,s.lines,s.areas,s.coverages,s.sounding_points)
if actual != expected:
result.count_mismatches+=1
result.messages.append(f"COUNT_MISMATCH {rel} expected={expected} actual={actual}")
if require_raw and rawbytes.get(cid,0) != s.file_size:
result.raw_preservation_mismatches += 1
result.messages.append(f"RAW_BYTES_MISMATCH {rel} expected={s.file_size} actual={rawbytes.get(cid,0)}")
# Precision audit: source-derived geometry versus geometry round-tripped through PostGIS.
rel_to_id={k:int(v["id"]) for k,v in db_by_rel.items()}
for sample in samples:
cid=rel_to_id.get(sample.rel)
if not cid:
continue
dbwkb=_precision_db_geom(conn,cid,sample)
result.precision_checked+=1
if dbwkb is None:
result.precision_failed+=1
result.messages.append(f"PRECISION_MISSING {sample.rel} record={sample.record_index}")
continue
expected=from_wkb(sample.expected_wkb); actual=from_wkb(dbwkb)
try:
ok=expected.equals_exact(actual,tolerance_deg)
delta=expected.hausdorff_distance(actual) if not ok else 0.0
except Exception:
ok=False; delta=float("inf")
if math.isfinite(delta):
result.max_precision_delta_deg=max(result.max_precision_delta_deg,float(delta))
if not ok:
result.precision_failed+=1
result.messages.append(f"PRECISION_FAIL {sample.rel} record={sample.record_index} type={sample.geometry_type} delta_deg={delta}")
finally:
conn.close()
return result
def print_result(r: AuditResult) -> None:
print("\n=== CM93 AUDIT ===")
print(f"source_cells : {r.source_cells}")
print(f"source_parse_failed : {r.source_parse_failed}")
print(f"db_missing_cells : {r.db_missing_cells}")
print(f"db_not_done_cells : {r.db_not_done_cells}")
print(f"count_mismatches : {r.count_mismatches}")
print(f"hash_mismatches : {r.hash_mismatches}")
print(f"raw_preservation_bad : {r.raw_preservation_mismatches}")
print(f"geometry_validity : {r.geometry_validity_issues} (informational)")
print(f"unknown_objects : {r.unknown_objects}")
print(f"unknown_attributes : {r.unknown_attributes}")
print(f"precision_checked : {r.precision_checked}")
print(f"precision_failed : {r.precision_failed}")
print(f"max_precision_delta° : {r.max_precision_delta_deg:.16g}")
print(f"RESULT : {'PASS' if r.passed else 'FAIL'}")
if r.messages:
print("\nDetails (max 100 shown):")
for m in r.messages[:100]: print(" -",m)

468
src/cm93db/cell_parser.py Normal file
View File

@@ -0,0 +1,468 @@
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
import hashlib
import struct
from typing import Any
from .decoder import decode_cm93
from .dictionaries import AttributeDef, ObjectDef
from .precision import CM93_COORD_MAX, CM93_SEMIMAJOR_M
class CM93ParseError(RuntimeError):
pass
@dataclass(frozen=True)
class CM93Header:
lon_min: float
lat_min: float
lon_max: float
lat_max: float
easting_min: float
northing_min: float
easting_max: float
northing_max: float
vector_records: int
vector_record_points: int
m_46: int
m_4a: int
point3d_records: int
point3d_points: int
m_54: int
point2d_records: int
m_5a: int
m_5c: int
feature_records: int
m_60: int
m_64: int
m_68: int
m_6a: int
m_6c: int
related_object_pointers: int
m_72: int
m_76: int
attribute_bytes: int
m_7c: int
@property
def transform_x_rate(self) -> float:
delta_x = self.easting_max - self.easting_min
if delta_x < 0.0:
delta_x += CM93_SEMIMAJOR_M * 2.0 * 3.141592653589793
return delta_x / CM93_COORD_MAX
@property
def transform_y_rate(self) -> float:
return (self.northing_max - self.northing_min) / CM93_COORD_MAX
@dataclass(frozen=True)
class EdgeRecord:
index: int
points: tuple[tuple[int, int], ...]
raw: bytes
@dataclass(frozen=True)
class SoundingGroup:
index: int
points: tuple[tuple[int, int, int], ...]
raw: bytes
@dataclass(frozen=True)
class ParsedAttribute:
ordinal: int
attribute_id: int
name: str
source_type: str
decoder_type: str
value_type: str
value: Any
value_int: int | None
value_real: float | None
value_text: str | None
value_json: Any | None
raw_value: bytes
@dataclass(frozen=True)
class GeometryRef:
edge_index: int
segment_usage: int
@dataclass
class ParsedFeature:
record_index: int
object_type: int
object_class: str
raw_geotype: int
primitive: int
geometry_refs: tuple[GeometryRef, ...] = ()
point2d_index: int | None = None
point3d_index: int | None = None
related_indices: tuple[int, ...] = ()
related_count_hint: int | None = None
n_attributes: int = 0
attributes: tuple[ParsedAttribute, ...] = ()
raw_record: bytes = b""
raw_attribute_block: bytes = b""
@dataclass(frozen=True)
class ParsedCell:
path: Path
source_size: int
sha256: str
encoded_bytes: bytes
decoded_bytes: bytes
prolog_word0: int
table1_length: int
table2_length: int
header: CM93Header
edges: tuple[EdgeRecord, ...]
point2d: tuple[tuple[int, int], ...]
point3d: tuple[SoundingGroup, ...]
features: tuple[ParsedFeature, ...]
end_offset: int
table_offsets: dict[str, int]
attribute_count: int
class _Reader:
def __init__(self, data: bytes, offset: int = 0):
self.data = data
self.offset = offset
def _need(self, n: int) -> None:
if self.offset + n > len(self.data):
raise CM93ParseError(
f"unexpected EOF at {self.offset}, need {n}, size={len(self.data)}"
)
def u8(self) -> int:
self._need(1)
v = self.data[self.offset]
self.offset += 1
return v
def u16(self) -> int:
self._need(2)
v = struct.unpack_from("<H", self.data, self.offset)[0]
self.offset += 2
return v
def i32(self) -> int:
self._need(4)
v = struct.unpack_from("<i", self.data, self.offset)[0]
self.offset += 4
return v
def f32(self) -> float:
self._need(4)
v = struct.unpack_from("<f", self.data, self.offset)[0]
self.offset += 4
return float(v)
def f64(self) -> float:
self._need(8)
v = struct.unpack_from("<d", self.data, self.offset)[0]
self.offset += 8
return float(v)
def take(self, n: int) -> bytes:
self._need(n)
b = self.data[self.offset:self.offset+n]
self.offset += n
return b
def cstring(self) -> bytes:
end = self.data.find(b"\x00", self.offset)
if end < 0:
raise CM93ParseError(f"unterminated string at {self.offset}")
b = self.data[self.offset:end]
self.offset = end + 1
return b
def _parse_header(r: _Reader) -> CM93Header:
start = r.offset
vals: list[Any] = [r.f64() for _ in range(8)]
vector_records = r.u16()
vector_record_points = r.i32()
m_46 = r.i32()
m_4a = r.i32()
point3d_records = r.u16()
point3d_points = r.i32()
m_54 = r.i32()
point2d_records = r.u16()
m_5a = r.u16()
m_5c = r.u16()
feature_records = r.u16()
m_60 = r.i32()
m_64 = r.i32()
m_68 = r.u16()
m_6a = r.u16()
m_6c = r.u16()
related = r.i32()
m_72 = r.i32()
m_76 = r.u16()
attr_bytes = r.i32()
m_7c = r.i32()
if r.offset - start != 128:
raise CM93ParseError(f"header size != 128: {r.offset-start}")
if min(vector_records, vector_record_points, point3d_records, point3d_points,
point2d_records, feature_records, related, attr_bytes) < 0:
raise CM93ParseError("negative header count")
return CM93Header(
*vals, vector_records, vector_record_points, m_46, m_4a,
point3d_records, point3d_points, m_54, point2d_records,
m_5a, m_5c, feature_records, m_60, m_64, m_68, m_6a, m_6c,
related, m_72, m_76, attr_bytes, m_7c,
)
def _text(b: bytes) -> str:
# CM93 data in the field is commonly ANSI/Windows text. Preserve bytes separately.
return b.decode("cp1252", errors="replace")
def _parse_attributes(
block: bytes,
nattr: int,
attr_defs: dict[int, AttributeDef],
) -> tuple[ParsedAttribute, ...]:
r = _Reader(block)
out: list[ParsedAttribute] = []
for ordinal in range(nattr):
item_start = r.offset
aid = r.u8()
d = attr_defs.get(aid)
if d is None:
raise CM93ParseError(f"unknown attribute id={aid} at attr {ordinal}")
t = d.decoder_type
value: Any
vi: int | None = None
vr: float | None = None
vt: str | None = None
vj: Any | None = None
value_type = "UNKNOWN"
if t == "B":
vi = r.u8(); value = vi; value_type = "BYTE"
elif t == "S":
raw_s = r.cstring(); vt = _text(raw_s); value = vt; value_type = "STRING"
elif t == "R":
vr = r.f32(); value = vr; value_type = "REAL"
elif t == "W":
vi = r.u16(); vr = vi / 10.0; value = vr; value_type = "WORD10"
elif t == "G":
vi = r.i32(); value = vi; value_type = "LONG"
elif t == "C":
prefix = list(r.take(3)); raw_s = r.cstring(); vt = _text(raw_s)
vj = {"prefix": prefix, "text": vt}; value = vj; value_type = "COMPLEX"
elif t == "L":
n = r.u8(); arr = list(r.take(n)); vj = arr; value = arr; value_type = "LIST"
elif t == "I":
# Kept for compatibility with old CM93 decoders; not present in supplied dictionary.
vi = r.u16(); value = vi; value_type = "INT"
else:
raise CM93ParseError(
f"unsupported attribute type={d.source_type_name}/{t} id={aid}"
)
raw_value = block[item_start+1:r.offset]
out.append(ParsedAttribute(
ordinal=ordinal,
attribute_id=aid,
name=d.name,
source_type=d.source_type_name,
decoder_type=t,
value_type=value_type,
value=value,
value_int=vi,
value_real=vr,
value_text=vt,
value_json=vj,
raw_value=raw_value,
))
if r.offset != len(block):
raise CM93ParseError(
f"attribute block length mismatch: parsed={r.offset} actual={len(block)} nattr={nattr}"
)
return tuple(out)
def parse_cell(
path: Path,
object_defs: dict[int, ObjectDef],
attr_defs: dict[int, AttributeDef],
*,
keep_source_bytes: bool = True,
) -> ParsedCell:
encoded = path.read_bytes()
decoded = decode_cm93(encoded)
r = _Reader(decoded)
word0 = r.u16()
table1 = r.i32()
table2 = r.i32()
if word0 + table1 + table2 != len(decoded):
raise CM93ParseError(
f"cell length check failed: {word0}+{table1}+{table2}!={len(decoded)}"
)
if word0 != 138:
raise CM93ParseError(f"unexpected prolog+header length: {word0}")
header = _parse_header(r)
table_offsets = {"prolog": 0, "header": 10, "vector": r.offset}
edges: list[EdgeRecord] = []
total_vector_points = 0
for i in range(header.vector_records):
st = r.offset
n = r.u16()
pts = tuple((r.u16(), r.u16()) for _ in range(n))
total_vector_points += n
edges.append(EdgeRecord(i, pts, decoded[st:r.offset]))
if total_vector_points != header.vector_record_points:
raise CM93ParseError(
f"vector point count mismatch: parsed={total_vector_points} header={header.vector_record_points}"
)
table_offsets["point3d"] = r.offset
groups: list[SoundingGroup] = []
total_3d = 0
for i in range(header.point3d_records):
st = r.offset
n = r.u16()
pts = tuple((r.u16(), r.u16(), r.u16()) for _ in range(n))
total_3d += n
groups.append(SoundingGroup(i, pts, decoded[st:r.offset]))
if total_3d != header.point3d_points:
raise CM93ParseError(
f"3D point count mismatch: parsed={total_3d} header={header.point3d_points}"
)
table_offsets["point2d"] = r.offset
point2d = tuple((r.u16(), r.u16()) for _ in range(header.point2d_records))
table_offsets["feature"] = r.offset
features: list[ParsedFeature] = []
attr_count = 0
total_attr_bytes = 0
for i in range(header.feature_records):
st = r.offset
object_type = r.u8()
geotype = r.u8()
record_len = r.u16()
if record_len < 4:
raise CM93ParseError(f"feature {i}: record length {record_len} < 4")
record_end = st + record_len
if record_end > len(decoded):
raise CM93ParseError(f"feature {i}: record beyond EOF")
primitive = geotype & 0x0F
refs: list[GeometryRef] = []
p2idx = None
p3idx = None
if primitive in (2, 4):
n = r.u16()
for _ in range(n):
raw_idx = r.u16()
edge_idx = raw_idx & 0x1FFF
usage = raw_idx >> 13
if edge_idx >= len(edges):
raise CM93ParseError(f"feature {i}: edge index {edge_idx} out of range")
refs.append(GeometryRef(edge_idx, usage))
elif primitive == 1:
p2idx = r.u16()
if p2idx >= len(point2d):
raise CM93ParseError(f"feature {i}: point2d index {p2idx} out of range")
elif primitive == 8:
p3idx = r.u16()
if p3idx >= len(groups):
raise CM93ParseError(f"feature {i}: point3d index {p3idx} out of range")
elif primitive == 0:
pass
else:
# Unknown low primitive cannot be skipped safely unless record boundary is honored.
# We preserve the record and mark geometry unknown; remaining bytes are consumed below.
pass
related: list[int] = []
related_hint: int | None = None
if geotype & 0x10:
nr = r.u8()
for _ in range(nr):
idx = r.u16()
if idx >= header.feature_records:
raise CM93ParseError(f"feature {i}: related index {idx} out of range")
related.append(idx)
if geotype & 0x20:
related_hint = r.u16() & 0xFF
nattr = 0
attrs: tuple[ParsedAttribute, ...] = ()
attr_block = b""
if geotype & 0x80:
nattr = r.u8()
attr_len = record_end - r.offset
if attr_len < 0:
raise CM93ParseError(f"feature {i}: negative attribute length")
attr_block = r.take(attr_len)
attrs = _parse_attributes(attr_block, nattr, attr_defs)
attr_count += nattr
total_attr_bytes += len(attr_block)
elif r.offset < record_end:
# Known flags consumed fewer bytes than descriptor length. Preserve but do not guess.
r.offset = record_end
if r.offset != record_end:
raise CM93ParseError(
f"feature {i}: record boundary mismatch parsed={r.offset-st} expected={record_len}"
)
obj = object_defs.get(object_type)
object_class = obj.name if obj else f"UNKNOWN_{object_type}"
features.append(ParsedFeature(
record_index=i,
object_type=object_type,
object_class=object_class,
raw_geotype=geotype,
primitive=primitive,
geometry_refs=tuple(refs),
point2d_index=p2idx,
point3d_index=p3idx,
related_indices=tuple(related),
related_count_hint=related_hint,
n_attributes=nattr,
attributes=attrs,
raw_record=decoded[st:record_end],
raw_attribute_block=attr_block,
))
if total_attr_bytes != header.attribute_bytes:
raise CM93ParseError(
f"attribute byte count mismatch: parsed={total_attr_bytes} header={header.attribute_bytes}"
)
if r.offset != len(decoded):
raise CM93ParseError(f"cell end mismatch: parsed={r.offset} file={len(decoded)}")
sha = hashlib.sha256(encoded).hexdigest()
return ParsedCell(
path=path,
source_size=len(encoded),
sha256=sha,
encoded_bytes=encoded if keep_source_bytes else b"",
decoded_bytes=decoded if keep_source_bytes else b"",
prolog_word0=word0,
table1_length=table1,
table2_length=table2,
header=header,
edges=tuple(edges),
point2d=point2d,
point3d=tuple(groups),
features=tuple(features),
end_offset=r.offset,
table_offsets=table_offsets,
attribute_count=attr_count,
)

203
src/cm93db/cli.py Normal file
View File

@@ -0,0 +1,203 @@
from __future__ import annotations
import argparse
import platform
import sys
import json
from datetime import datetime
from pathlib import Path
from .config import load_config, load_source_dir
from .database import (
connect,
server_version,
assert_postgresql_compatible,
postgis_version,
create_database,
apply_schema,
)
from .scanner import scan_cm93
from .dictionaries import load_object_dictionary, load_attribute_dictionary
from .schema_contract import REQUIRED_TABLES
from .importer import ImportOptions, run_import
from .audit import run_audit, print_result
def project_root() -> Path:
return Path(__file__).resolve().parents[2]
def _root(arg: str | None) -> Path:
return Path(arg) if arg else load_source_dir()
def _scales(values: list[str] | None) -> set[str] | None:
if not values:
return None
out=set()
for v in values:
for ch in v.upper().replace(",",""):
if ch.strip(): out.add(ch)
bad=out-set("ZABCDEFG")
if bad:
raise SystemExit(f"无效比例尺: {sorted(bad)}")
return out
def cmd_env_check(_args):
print(f"Python: {sys.version.split()[0]}")
if sys.version_info[:2] != (3, 12):
raise SystemExit("ERROR: 本项目固定 Python 3.12.x;推荐 3.12.10")
print(f"Platform: {platform.platform()}")
cfg = load_config()
conn = connect(cfg)
try:
v = server_version(conn)
assert_postgresql_compatible(v)
print(f"PostgreSQL: {v}")
print(f"PostGIS: {postgis_version(conn)}")
print(f"Database: {cfg.database}")
print("PostgreSQL/PostGIS compatibility: OK")
finally:
conn.close()
def cmd_db_init(args):
cfg = load_config()
if args.create_database:
create_database(cfg)
print(f"Database ensured: {cfg.database}")
schema = project_root() / "sql" / "001_schema_postgresql.sql"
apply_schema(cfg, schema)
print(f"Schema initialized: {schema.name}")
def cmd_db_check(_args):
cfg = load_config()
conn = connect(cfg)
try:
v = server_version(conn)
assert_postgresql_compatible(v)
with conn.cursor() as cur:
cur.execute(
"SELECT table_name FROM information_schema.tables "
"WHERE table_schema='public' AND table_type='BASE TABLE'"
)
tables = {row["table_name"] for row in cur.fetchall()}
missing = sorted(REQUIRED_TABLES - tables)
present = REQUIRED_TABLES & tables
print(f"PostgreSQL: {v}")
print(f"PostGIS: {postgis_version(conn)}")
print(f"CM95 tables: {len(present)}/{len(REQUIRED_TABLES)}")
if missing:
print("Missing:")
for t in missing: print(f" - {t}")
raise SystemExit(2)
print("Schema contract: OK")
finally:
conn.close()
def cmd_scan(args):
result = scan_cm93(_root(args.root))
by_scale = {}
for c in result.cells: by_scale[c.scale_code] = by_scale.get(c.scale_code, 0) + 1
print(f"Root: {result.root}")
print(f"Cells: {len(result.cells)}")
for scale in sorted(by_scale): print(f" {scale}: {by_scale[scale]}")
print(f"CM93OBJ.DIC: {result.object_dictionary or 'NOT FOUND'}")
print(f"ATTR dictionary: {result.attribute_dictionary or 'NOT FOUND'}")
def cmd_inspect_dicts(args):
result = scan_cm93(_root(args.root))
if not result.object_dictionary: raise SystemExit("CM93OBJ.DIC 未找到")
if not result.attribute_dictionary: raise SystemExit("CM93ATTR.DIC/ATTRLUT.DIC 未找到")
objects = load_object_dictionary(result.object_dictionary)
attrs = load_attribute_dictionary(result.attribute_dictionary)
print(f"Object dictionary: {len(objects)} records")
print(f"Attribute dictionary: {len(attrs)} records")
for key in ("DEPARE", "DEPCNT", "LIGHTS", "SOUNDG", "WRECKS", "_m_sor"):
item = next((x for x in objects if x.name == key), None)
print(f" {key}: {item.object_id if item else 'missing'}")
for key in ("DRVAL1", "VALSOU", "_wgsox", "_wgsoy"):
item = next((x for x in attrs if x.name == key), None)
print(f" {key}: {item.attribute_id if item else 'missing'} / {item.source_type_name if item else '-'}")
def cmd_import(args):
cfg=load_config(); root=_root(args.root)
opt=ImportOptions(root=root,limit=args.limit,scales=_scales(args.scale),cell_name=args.cell,
reimport=args.reimport,retry_failed=not args.skip_failed,fail_fast=args.fail_fast,
store_raw_tables=not args.no_raw_tables,shard_count=args.shard_count,
shard_index=args.shard_index)
s=run_import(cfg,opt)
print("\n=== IMPORT SUMMARY ===")
for k in ("cells_discovered","cells_done","cells_skipped","cells_failed","features_source","features_imported","attributes_source","attributes_imported","unknown_object_count","unknown_attribute_count","geometry_issue_count","sounding_points"):
print(f"{k:24}: {getattr(s,k)}")
if s.cells_failed or s.unknown_object_count or s.unknown_attribute_count:
raise SystemExit(2)
def cmd_audit(args):
cfg=load_config() if not args.source_only else load_config()
r=run_audit(cfg,_root(args.root),limit=args.limit,scales=_scales(args.scale),cell_name=args.cell,
precision_samples=args.precision_samples,tolerance_deg=args.tolerance,
validate_geometry=args.validate_geometry,source_only=args.source_only,require_raw=not args.allow_missing_raw)
print_result(r)
report = Path(args.report) if args.report else (project_root() / "logs" / f"cm93_audit_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json")
report.parent.mkdir(parents=True,exist_ok=True)
report.write_text(json.dumps(r.as_dict(),ensure_ascii=False,indent=2),encoding="utf-8")
print(f"Audit report: {report}")
if not r.passed: raise SystemExit(2)
def _add_source_args(s, optional=True):
s.add_argument(
"root",
nargs="?" if optional else None,
help="CM95/CM93 Japan 根目录;省略时读取 .env 的 CM95_SOURCE_DIR",
)
def build_parser():
p = argparse.ArgumentParser(prog="cm95pg")
sub = p.add_subparsers(dest="cmd", required=True)
s = sub.add_parser("env-check"); s.set_defaults(func=cmd_env_check)
s = sub.add_parser("db-init"); s.add_argument("--create-database", action="store_true"); s.set_defaults(func=cmd_db_init)
s = sub.add_parser("db-check"); s.set_defaults(func=cmd_db_check)
s = sub.add_parser("scan"); _add_source_args(s); s.set_defaults(func=cmd_scan)
s = sub.add_parser("inspect-dicts"); _add_source_args(s); s.set_defaults(func=cmd_inspect_dicts)
s = sub.add_parser("import", help="完整解析 CM95/CM93 Japan 并逐 cell 导入 PostgreSQL/PostGIS")
_add_source_args(s)
s.add_argument("--limit", type=int)
s.add_argument("--scale", action="append", help="限定比例尺,例如 --scale G 或 --scale F,G")
s.add_argument("--cell", help="只导入指定 cell 文件名/不带扩展名")
s.add_argument("--reimport", action="store_true", help="重新导入已 DONE cell")
s.add_argument("--skip-failed", action="store_true", help="不重试之前 FAILED 的 cell")
s.add_argument("--fail-fast", action="store_true")
s.add_argument("--no-raw-tables", action="store_true", help="不保存 decoded 原始 table;不建议正式母库使用")
s.add_argument("--shard-count", type=int, help="可选并行分片总数;默认不分片")
s.add_argument("--shard-index", type=int, help="当前分片编号,从 0 开始;需与 --shard-count 配合")
s.set_defaults(func=cmd_import)
s = sub.add_parser("audit", help="重新解析源 CM93 并与数据库逐 cell 审计")
_add_source_args(s)
s.add_argument("--limit", type=int)
s.add_argument("--scale", action="append")
s.add_argument("--cell")
s.add_argument("--precision-samples", type=int, default=128, help="数据库 WKB 双精度回读样本数")
s.add_argument("--tolerance", type=float, default=1e-12, help="经纬度精度比较容差(degree)")
s.add_argument("--validate-geometry", action="store_true", help="额外执行 Shapely validity 全量检查,较慢")
s.add_argument("--source-only", action="store_true", help="只审计源文件解析,不查询数据库")
s.add_argument("--allow-missing-raw", action="store_true", help="不把 raw table 未保存视为审计失败")
s.add_argument("--report", help="JSON 审计报告路径;默认写入项目 logs/")
s.set_defaults(func=cmd_audit)
return p
def main():
args = build_parser().parse_args()
args.func(args)
if __name__ == "__main__": main()

478
src/cm93db/cm93_binary.py Normal file
View File

@@ -0,0 +1,478 @@
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
import hashlib
import math
import struct
from typing import Any
from .dictionaries import AttributeDef, ObjectDef
from .precision import CM93_SEMIMAJOR_M
TABLE_0 = bytes([
0xCD,0xEA,0xDC,0x48,0x3E,0x6D,0xCA,0x7B,0x52,0xE1,0xA4,0x8E,0xAB,0x05,0xA7,0x97,
0xB9,0x60,0x39,0x85,0x7C,0x56,0x7A,0xBA,0x68,0x6E,0xF5,0x5D,0x02,0x4E,0x0F,0xA1,
0x27,0x24,0x41,0x34,0x00,0x5A,0xFE,0xCB,0xD0,0xFA,0xF8,0x6C,0x74,0x96,0x9E,0x0E,
0xC2,0x49,0xE3,0xE5,0xC0,0x3B,0x59,0x18,0xA9,0x86,0x8F,0x30,0xC3,0xA8,0x22,0x0A,
0x14,0x1A,0xB2,0xC9,0xC7,0xED,0xAA,0x29,0x94,0x75,0x0D,0xAC,0x0C,0xF4,0xBB,0xC5,
0x3F,0xFD,0xD9,0x9C,0x4F,0xD5,0x84,0x1E,0xB1,0x81,0x69,0xB4,0x09,0xB8,0x3C,0xAF,
0xA3,0x08,0xBF,0xE0,0x9A,0xD7,0xF7,0x8C,0x67,0x66,0xAE,0xD4,0x4C,0xA5,0xEC,0xF9,
0xB6,0x64,0x78,0x06,0x5B,0x9B,0xF2,0x99,0xCE,0xDB,0x53,0x55,0x65,0x8D,0x07,0x33,
0x04,0x37,0x92,0x26,0x23,0xB5,0x58,0xDA,0x2F,0xB3,0x40,0x5E,0x7F,0x4B,0x62,0x80,
0xE4,0x6F,0x73,0x1D,0xDF,0x17,0xCC,0x28,0x25,0x2D,0xEE,0x3A,0x98,0xE2,0x01,0xEB,
0xDD,0xBC,0x90,0xB0,0xFC,0x95,0x76,0x93,0x46,0x57,0x2C,0x2B,0x50,0x11,0x0B,0xC1,
0xF0,0xE7,0xD6,0x21,0x31,0xDE,0xFF,0xD8,0x12,0xA6,0x4D,0x8A,0x13,0x43,0x45,0x38,
0xD2,0x87,0xA0,0xEF,0x82,0xF1,0x47,0x89,0x6A,0xC8,0x54,0x1B,0x16,0x7E,0x79,0xBD,
0x6B,0x91,0xA2,0x71,0x36,0xB7,0x03,0x3D,0x72,0xC6,0x44,0x8B,0xCF,0x15,0x9F,0x32,
0xC4,0x77,0x83,0x63,0x20,0x88,0xF6,0xAD,0xF3,0xE8,0x4A,0xE9,0x35,0x1C,0x5F,0x19,
0x1F,0x7D,0x70,0xFB,0xD1,0x51,0x10,0xD3,0x2E,0x61,0x9D,0x5C,0x2A,0x42,0xBE,0xE6,
])
_encode = bytes((v ^ 8) for v in TABLE_0)
_decode_arr = bytearray(256)
for i, v in enumerate(_encode):
_decode_arr[v] = i
DECODE_TABLE = bytes(_decode_arr)
def decode_bytes(data: bytes) -> bytes:
return data.translate(DECODE_TABLE)
class CM93ParseError(RuntimeError):
pass
@dataclass(frozen=True)
class Header:
lon_min: float
lat_min: float
lon_max: float
lat_max: float
easting_min: float
northing_min: float
easting_max: float
northing_max: float
usn_vector_records: int
n_vector_record_points: int
m_46: int
m_4a: int
usn_point3d_records: int
m_50: int
m_54: int
usn_point2d_records: int
m_5a: int
m_5c: int
usn_feature_records: int
m_60: int
m_64: int
m_68: int
m_6a: int
m_6c: int
m_nrelated_object_pointers: int
m_72: int
m_76: int
m_78: int
m_7c: int
@property
def x_rate(self) -> float:
delta = self.easting_max - self.easting_min
if delta < 0:
delta += CM93_SEMIMAJOR_M * 2.0 * math.pi
return delta / 65535.0
@property
def y_rate(self) -> float:
return (self.northing_max - self.northing_min) / 65535.0
def as_dict(self) -> dict[str, Any]:
return {k: getattr(self, k) for k in self.__dataclass_fields__}
@dataclass(frozen=True)
class Edge:
index: int
points: tuple[tuple[int, int], ...]
@dataclass(frozen=True)
class Point3DDescriptor:
index: int
points: tuple[tuple[int, int, int], ...]
@dataclass(frozen=True)
class ParsedAttribute:
ordinal: int
attribute_id: int
name: str
decoder_type: str
source_type_name: str
value: Any
raw_value: bytes
raw_entry: bytes
@dataclass
class FeatureRecord:
index: int
object_type: int
object_class: str
geotype: int
geometry_primitive: int
geometry_refs: list[tuple[int, int]] = field(default_factory=list) # (edge index, usage)
point2d_index: int | None = None
point3d_index: int | None = None
related_indices: list[int] = field(default_factory=list)
relation20_raw: int | None = None
attributes: list[ParsedAttribute] = field(default_factory=list)
raw_record: bytes = b""
raw_attribute_blob: bytes = b""
@dataclass(frozen=True)
class RawTables:
prolog: bytes
header: bytes
vectors: bytes
point3d: bytes
point2d: bytes
features: bytes
@dataclass
class ParsedCell:
path: Path
file_size: int
sha256: str
word0: int
table1_length: int
table2_length: int
header: Header
edges: list[Edge]
point3d: list[Point3DDescriptor]
point2d: list[tuple[int, int]]
features: list[FeatureRecord]
raw_tables: RawTables
unknown_object_ids: set[int]
unknown_attribute_ids: set[int]
attribute_count: int
class Cursor:
def __init__(self, data: bytes, pos: int = 0):
self.data = data
self.pos = pos
def remaining(self) -> int:
return len(self.data) - self.pos
def take(self, n: int) -> bytes:
if n < 0 or self.pos + n > len(self.data):
raise CM93ParseError(f"越界读取: pos={self.pos}, n={n}, size={len(self.data)}")
out = self.data[self.pos:self.pos+n]
self.pos += n
return out
def u8(self) -> int:
return self.take(1)[0]
def u16(self) -> int:
return struct.unpack("<H", self.take(2))[0]
def i32(self) -> int:
return struct.unpack("<i", self.take(4))[0]
def f32(self) -> float:
return struct.unpack("<f", self.take(4))[0]
def f64(self) -> float:
return struct.unpack("<d", self.take(8))[0]
HEADER_FMT = "<8dHiiiHiiHHHHiiHHH i H ii".replace(" ", "")
# Explicit parsing is used instead of struct.unpack on the full header so that
# field order remains auditable against the documented 128-byte CM93 sequence.
def parse_header(decoded_header: bytes) -> Header:
if len(decoded_header) != 128:
raise CM93ParseError(f"CM93 header 长度应为 128,实际 {len(decoded_header)}")
c = Cursor(decoded_header)
vals = [c.f64() for _ in range(8)]
h = Header(
lon_min=vals[0], lat_min=vals[1], lon_max=vals[2], lat_max=vals[3],
easting_min=vals[4], northing_min=vals[5], easting_max=vals[6], northing_max=vals[7],
usn_vector_records=c.u16(),
n_vector_record_points=c.i32(),
m_46=c.i32(),
m_4a=c.i32(),
usn_point3d_records=c.u16(),
m_50=c.i32(),
m_54=c.i32(),
usn_point2d_records=c.u16(),
m_5a=c.u16(),
m_5c=c.u16(),
usn_feature_records=c.u16(),
m_60=c.i32(),
m_64=c.i32(),
m_68=c.u16(),
m_6a=c.u16(),
m_6c=c.u16(),
m_nrelated_object_pointers=c.i32(),
m_72=c.i32(),
m_76=c.u16(),
m_78=c.i32(),
m_7c=c.i32(),
)
if c.remaining() != 0:
raise CM93ParseError(f"header 未完全消费: {c.remaining()} bytes")
return h
def _decode_text(raw: bytes) -> str:
# Byte-preserving fallback is intentional. raw_value is stored separately.
for enc in ("utf-8", "cp1252", "latin1"):
try:
return raw.decode(enc)
except UnicodeDecodeError:
pass
return raw.decode("latin1")
def _parse_attr_entry(c: Cursor, ordinal: int, attr_defs: dict[int, AttributeDef]) -> tuple[ParsedAttribute, bool]:
start = c.pos
aid = c.u8()
d = attr_defs.get(aid)
unknown = d is None
if d is None:
raise CM93ParseError(f"未知 attribute id={aid},无法确定值长度")
vstart = c.pos
t = d.decoder_type
if t == "B":
value = c.u8()
elif t == "W":
value = c.u16() / 10.0
elif t == "G":
value = c.i32()
elif t == "R":
value = c.f32()
elif t == "S":
z = c.data.find(b"\x00", c.pos)
if z < 0:
raise CM93ParseError(f"字符串属性 {d.name} 缺少 NUL 终止符")
raw = c.data[c.pos:z]
c.pos = z + 1
value = _decode_text(raw)
elif t == "C":
prefix = list(c.take(3))
z = c.data.find(b"\x00", c.pos)
if z < 0:
raise CM93ParseError(f"复杂属性 {d.name} 缺少 NUL 终止符")
raw = c.data[c.pos:z]
c.pos = z + 1
value = {"prefix": prefix, "text": _decode_text(raw)}
elif t == "L":
n = c.u8()
value = list(c.take(n))
elif t == "I":
value = c.u16()
else:
raise CM93ParseError(f"属性 {d.name}({aid}) 类型 {d.source_type_name}/{t} 无法解码")
vend = c.pos
return ParsedAttribute(
ordinal=ordinal,
attribute_id=aid,
name=d.name,
decoder_type=t,
source_type_name=d.source_type_name,
value=value,
raw_value=c.data[vstart:vend],
raw_entry=c.data[start:vend],
), unknown
def parse_cell(path: Path, object_defs: dict[int, ObjectDef], attr_defs: dict[int, AttributeDef]) -> ParsedCell:
encoded = path.read_bytes()
decoded = decode_bytes(encoded)
if len(decoded) < 138:
raise CM93ParseError(f"文件过短: {path}")
c = Cursor(decoded)
prolog_start = c.pos
word0 = c.u16()
table1 = c.i32()
table2 = c.i32()
prolog = decoded[prolog_start:c.pos]
if word0 != 138:
raise CM93ParseError(f"prolog/header 长度异常: word0={word0}, expected=138")
if word0 + table1 + table2 != len(decoded):
raise CM93ParseError(
f"文件完整性失败: {word0}+{table1}+{table2}!={len(decoded)}"
)
header_start = c.pos
header = parse_header(c.take(128))
raw_header = decoded[header_start:c.pos]
vector_start = c.pos
edges: list[Edge] = []
total_vector_points = 0
for i in range(header.usn_vector_records):
n = c.u16()
pts = tuple((c.u16(), c.u16()) for _ in range(n))
total_vector_points += n
edges.append(Edge(i, pts))
vector_end = c.pos
if total_vector_points != header.n_vector_record_points:
raise CM93ParseError(
f"vector point count mismatch: header={header.n_vector_record_points}, parsed={total_vector_points}"
)
p3_start = c.pos
point3d: list[Point3DDescriptor] = []
total_p3_points = 0
for i in range(header.usn_point3d_records):
n = c.u16()
pts = tuple((c.u16(), c.u16(), c.u16()) for _ in range(n))
total_p3_points += n
point3d.append(Point3DDescriptor(i, pts))
p3_end = c.pos
if header.m_50 >= 0 and total_p3_points != header.m_50:
raise CM93ParseError(f"3D point count mismatch: header={header.m_50}, parsed={total_p3_points}")
p2_start = c.pos
point2d = [(c.u16(), c.u16()) for _ in range(header.usn_point2d_records)]
p2_end = c.pos
feature_start = c.pos
features: list[FeatureRecord] = []
unknown_obj: set[int] = set()
unknown_attr: set[int] = set()
attr_total = 0
for fi in range(header.usn_feature_records):
rec_start = c.pos
object_type = c.u8()
geotype = c.u8()
desc_len = c.u16()
if desc_len < 4:
raise CM93ParseError(f"feature {fi}: obj_desc_bytes={desc_len}<4")
rec_end = rec_start + desc_len
if rec_end > len(decoded):
raise CM93ParseError(f"feature {fi}: record 越界")
od = object_defs.get(object_type)
if od is None:
unknown_obj.add(object_type)
object_class = f"UNKNOWN_{object_type}"
else:
object_class = od.name
prim = geotype & 0x0F
f = FeatureRecord(fi, object_type, object_class, geotype, prim)
if prim in (2, 4):
n = c.u16()
for _ in range(n):
packed = c.u16()
edge_idx = packed & 0x1FFF
usage = packed >> 13
if edge_idx >= len(edges):
raise CM93ParseError(f"feature {fi}: edge index {edge_idx} >= {len(edges)}")
f.geometry_refs.append((edge_idx, usage))
elif prim == 1:
idx = c.u16()
if idx >= len(point2d):
raise CM93ParseError(f"feature {fi}: point2d index {idx} >= {len(point2d)}")
f.point2d_index = idx
elif prim == 8:
idx = c.u16()
if idx >= len(point3d):
raise CM93ParseError(f"feature {fi}: point3d index {idx} >= {len(point3d)}")
f.point3d_index = idx
if geotype & 0x10:
nrel = c.u8()
f.related_indices = [c.u16() for _ in range(nrel)]
for rel in f.related_indices:
if rel >= header.usn_feature_records:
raise CM93ParseError(f"feature {fi}: related index {rel} 越界")
if geotype & 0x20:
f.relation20_raw = c.u16()
# 0x40 is observed as a flag without a documented payload in OpenCPN.
if geotype & 0x80:
nattr = c.u8()
attr_blob_start = c.pos
# By format definition obj_desc_bytes is the total record length.
# Attribute payload is the remaining bytes in this record.
attr_blob = decoded[attr_blob_start:rec_end]
ac = Cursor(attr_blob)
attrs: list[ParsedAttribute] = []
for ordinal in range(nattr):
try:
a, was_unknown = _parse_attr_entry(ac, ordinal, attr_defs)
except CM93ParseError as e:
# Capture the leading id when possible for audit diagnostics.
if ac.pos < len(attr_blob):
unknown_attr.add(attr_blob[ac.pos])
raise CM93ParseError(f"feature {fi} attribute {ordinal}: {e}") from e
if was_unknown:
unknown_attr.add(a.attribute_id)
attrs.append(a)
if ac.remaining() != 0:
raise CM93ParseError(
f"feature {fi}: attribute blob 剩余 {ac.remaining()} bytes; nattr={nattr}"
)
f.attributes = attrs
f.raw_attribute_blob = attr_blob
attr_total += len(attrs)
c.pos = rec_end
else:
# Preserve any unparsed tail, but do not silently cross the record boundary.
if c.pos > rec_end:
raise CM93ParseError(f"feature {fi}: 已解析字段超过 record 长度")
c.pos = rec_end
f.raw_record = decoded[rec_start:rec_end]
features.append(f)
feature_end = c.pos
if feature_end != len(decoded):
raise CM93ParseError(f"文件尾不匹配: parsed={feature_end}, size={len(decoded)}, remaining={len(decoded)-feature_end}")
parsed_attr_bytes = sum(len(f.raw_attribute_blob) for f in features)
if parsed_attr_bytes != header.m_78:
raise CM93ParseError(f"attribute byte count mismatch: header={header.m_78}, parsed={parsed_attr_bytes}")
parsed_geom_refs = sum(len(f.geometry_refs) for f in features)
if parsed_geom_refs != header.m_46 + header.m_4a:
raise CM93ParseError(f"geometry ref count mismatch: header={header.m_46 + header.m_4a}, parsed={parsed_geom_refs}")
parsed_related = sum(len(f.related_indices) for f in features)
if parsed_related != header.m_nrelated_object_pointers:
raise CM93ParseError(f"related pointer count mismatch: header={header.m_nrelated_object_pointers}, parsed={parsed_related}")
# Table-1/Table-2 lengths are retained and audited at file level. The internal
# semantic subtable boundaries are determined from header record counts.
return ParsedCell(
path=path,
file_size=len(encoded),
sha256=hashlib.sha256(encoded).hexdigest(),
word0=word0,
table1_length=table1,
table2_length=table2,
header=header,
edges=edges,
point3d=point3d,
point2d=point2d,
features=features,
raw_tables=RawTables(
prolog=prolog,
header=raw_header,
vectors=decoded[vector_start:vector_end],
point3d=decoded[p3_start:p3_end],
point2d=decoded[p2_start:p2_end],
features=decoded[feature_start:feature_end],
),
unknown_object_ids=unknown_obj,
unknown_attribute_ids=unknown_attr,
attribute_count=attr_total,
)

39
src/cm93db/config.py Normal file
View File

@@ -0,0 +1,39 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
import os
from dotenv import load_dotenv
@dataclass(frozen=True)
class DbConfig:
host: str
port: int
database: str
user: str
password: str
sslmode: str = "prefer"
admin_database: str = "postgres"
connect_timeout: int = 20
def load_config(require_database: bool = True) -> DbConfig:
load_dotenv()
database = os.getenv("CM95_PG_DATABASE", "cm95_chart").strip()
if require_database and not database:
raise RuntimeError("CM95_PG_DATABASE 未配置")
return DbConfig(
host=os.getenv("CM95_PG_HOST", "127.0.0.1").strip(),
port=int(os.getenv("CM95_PG_PORT", "5432")),
database=database,
user=os.getenv("CM95_PG_USER", "cm95").strip(),
password=os.getenv("CM95_PG_PASSWORD", ""),
sslmode=os.getenv("CM95_PG_SSLMODE", "prefer").strip() or "prefer",
admin_database=os.getenv("CM95_PG_ADMIN_DATABASE", "postgres").strip() or "postgres",
connect_timeout=int(os.getenv("CM95_PG_CONNECT_TIMEOUT", "20")),
)
def load_source_dir() -> Path:
load_dotenv()
raw = os.getenv("CM95_SOURCE_DIR", "").strip()
if not raw:
raise RuntimeError("未传入 CM95/CM93 Japan 路径,且 .env 中 CM95_SOURCE_DIR 未配置")
return Path(raw)

127
src/cm93db/database.py Normal file
View File

@@ -0,0 +1,127 @@
from __future__ import annotations
from pathlib import Path
import re
from psycopg import connect as pg_connect, sql
from psycopg.rows import dict_row
from .config import DbConfig
def connect(cfg: DbConfig, with_database: bool = True):
return pg_connect(
host=cfg.host,
port=cfg.port,
user=cfg.user,
password=cfg.password,
dbname=cfg.database if with_database else cfg.admin_database,
sslmode=cfg.sslmode,
connect_timeout=cfg.connect_timeout,
autocommit=False,
row_factory=dict_row,
)
def server_version(conn) -> str:
with conn.cursor() as cur:
cur.execute("SHOW server_version")
row = cur.fetchone()
return str(row["server_version"])
def postgresql_version_tuple(version: str) -> tuple[int, int]:
m = re.match(r"^(\d+)(?:\.(\d+))?", version)
if not m:
raise RuntimeError(f"无法识别 PostgreSQL 版本: {version}")
return int(m.group(1)), int(m.group(2) or 0)
def assert_postgresql_compatible(version: str) -> None:
major, _minor = postgresql_version_tuple(version)
if major < 14:
raise RuntimeError(
f"CM95 PostgreSQL 版要求 PostgreSQL 14+;当前服务器为 {version}"
)
def postgis_version(conn) -> str:
with conn.cursor() as cur:
cur.execute("SELECT PostGIS_Version() AS v")
row = cur.fetchone()
return str(row["v"])
def assert_postgis_available(conn) -> str:
try:
return postgis_version(conn)
except Exception as exc:
conn.rollback()
raise RuntimeError("目标 PostgreSQL 必须安装并启用 PostGIS extension") from exc
def geom_from_wkb_sql(value_sql: str = "%s", srid: int = 4326) -> str:
return f"ST_GeomFromWKB({value_sql},{int(srid)})"
def geom_from_text_sql(value_sql: str = "%s", srid: int = 4326) -> str:
return f"ST_GeomFromText({value_sql},{int(srid)})"
def as_wkb_sql(geom_sql: str) -> str:
return f"ST_AsBinary({geom_sql})"
def create_database(cfg: DbConfig) -> None:
conn = connect(cfg, with_database=False)
try:
conn.autocommit = True
assert_postgresql_compatible(server_version(conn))
with conn.cursor() as cur:
cur.execute("SELECT 1 FROM pg_database WHERE datname=%s", (cfg.database,))
if cur.fetchone() is None:
cur.execute(
sql.SQL("CREATE DATABASE {}").format(sql.Identifier(cfg.database))
)
finally:
conn.close()
def _split_sql(text: str) -> list[str]:
statements: list[str] = []
buf: list[str] = []
in_single = False
in_double = False
i = 0
while i < len(text):
ch = text[i]
nxt = text[i + 1] if i + 1 < len(text) else ""
if not (in_single or in_double) and ch == "-" and nxt == "-":
while i < len(text) and text[i] != "\n":
i += 1
continue
if ch == "'" and not in_double:
if in_single and nxt == "'":
buf.extend([ch, nxt])
i += 2
continue
in_single = not in_single
elif ch == '"' and not in_single:
in_double = not in_double
if ch == ";" and not (in_single or in_double):
stmt = "".join(buf).strip()
if stmt:
statements.append(stmt)
buf = []
else:
buf.append(ch)
i += 1
tail = "".join(buf).strip()
if tail:
statements.append(tail)
return statements
def apply_schema(cfg: DbConfig, schema_path: Path) -> None:
statements = _split_sql(schema_path.read_text(encoding="utf-8"))
conn = connect(cfg)
try:
assert_postgresql_compatible(server_version(conn))
with conn.cursor() as cur:
for stmt in statements:
cur.execute(stmt)
conn.commit()
assert_postgis_available(conn)
except Exception:
conn.rollback()
raise
finally:
conn.close()

38
src/cm93db/decoder.py Normal file
View File

@@ -0,0 +1,38 @@
from __future__ import annotations
# CM93 byte substitution table. This is a format constant, not executable code.
# Decode table is derived as: encode[i] = TABLE_0[i] ^ 8; decode[encode[i]] = i.
TABLE_0 = bytes([
0xCD,0xEA,0xDC,0x48,0x3E,0x6D,0xCA,0x7B,0x52,0xE1,0xA4,0x8E,0xAB,0x05,0xA7,0x97,
0xB9,0x60,0x39,0x85,0x7C,0x56,0x7A,0xBA,0x68,0x6E,0xF5,0x5D,0x02,0x4E,0x0F,0xA1,
0x27,0x24,0x41,0x34,0x00,0x5A,0xFE,0xCB,0xD0,0xFA,0xF8,0x6C,0x74,0x96,0x9E,0x0E,
0xC2,0x49,0xE3,0xE5,0xC0,0x3B,0x59,0x18,0xA9,0x86,0x8F,0x30,0xC3,0xA8,0x22,0x0A,
0x14,0x1A,0xB2,0xC9,0xC7,0xED,0xAA,0x29,0x94,0x75,0x0D,0xAC,0x0C,0xF4,0xBB,0xC5,
0x3F,0xFD,0xD9,0x9C,0x4F,0xD5,0x84,0x1E,0xB1,0x81,0x69,0xB4,0x09,0xB8,0x3C,0xAF,
0xA3,0x08,0xBF,0xE0,0x9A,0xD7,0xF7,0x8C,0x67,0x66,0xAE,0xD4,0x4C,0xA5,0xEC,0xF9,
0xB6,0x64,0x78,0x06,0x5B,0x9B,0xF2,0x99,0xCE,0xDB,0x53,0x55,0x65,0x8D,0x07,0x33,
0x04,0x37,0x92,0x26,0x23,0xB5,0x58,0xDA,0x2F,0xB3,0x40,0x5E,0x7F,0x4B,0x62,0x80,
0xE4,0x6F,0x73,0x1D,0xDF,0x17,0xCC,0x28,0x25,0x2D,0xEE,0x3A,0x98,0xE2,0x01,0xEB,
0xDD,0xBC,0x90,0xB0,0xFC,0x95,0x76,0x93,0x46,0x57,0x2C,0x2B,0x50,0x11,0x0B,0xC1,
0xF0,0xE7,0xD6,0x21,0x31,0xDE,0xFF,0xD8,0x12,0xA6,0x4D,0x8A,0x13,0x43,0x45,0x38,
0xD2,0x87,0xA0,0xEF,0x82,0xF1,0x47,0x89,0x6A,0xC8,0x54,0x1B,0x16,0x7E,0x79,0xBD,
0x6B,0x91,0xA2,0x71,0x36,0xB7,0x03,0x3D,0x72,0xC6,0x44,0x8B,0xCF,0x15,0x9F,0x32,
0xC4,0x77,0x83,0x63,0x20,0x88,0xF6,0xAD,0xF3,0xE8,0x4A,0xE9,0x35,0x1C,0x5F,0x19,
0x1F,0x7D,0x70,0xFB,0xD1,0x51,0x10,0xD3,0x2E,0x61,0x9D,0x5C,0x2A,0x42,0xBE,0xE6,
])
def _make_decode_table() -> bytes:
decode = bytearray(256)
for plain, base in enumerate(TABLE_0):
encoded = base ^ 0x08
decode[encoded] = plain
return bytes(decode)
DECODE_TABLE = _make_decode_table()
def decode_cm93(data: bytes) -> bytes:
"""Decode a complete CM93 cell. Byte substitution is position-independent."""
return data.translate(DECODE_TABLE)

View File

@@ -0,0 +1,94 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
TYPE_MAP = {
"aFLOAT": "R",
"aBYTE": "B",
"aSTRING": "S",
"aCMPLX": "C",
"aLIST": "L",
"aWORD10": "W",
"aLONG": "G",
"aNotUsed": "?",
}
@dataclass(frozen=True)
class ObjectDef:
name: str
object_id: int
geometry_code: str
display_code: str | None
description: str | None
raw_line: str
@dataclass(frozen=True)
class AttributeDef:
name: str
attribute_id: int
source_type_name: str
decoder_type: str
min_value: float | None
max_value: float | None
raw_line: str
def _read_lines(path: Path):
for enc in ("utf-8-sig", "cp1252", "latin1"):
try:
return path.read_text(encoding=enc).splitlines()
except UnicodeDecodeError:
pass
raise UnicodeDecodeError("unknown", b"", 0, 1, f"无法读取 {path}")
def load_object_dictionary(path: Path) -> list[ObjectDef]:
out = []
for line in _read_lines(path):
if not line.strip():
continue
parts = line.rstrip("\r\n").split("|")
if len(parts) < 3:
continue
try:
object_id = int(parts[1].strip())
except ValueError:
continue
out.append(ObjectDef(
name=parts[0].strip(),
object_id=object_id,
geometry_code=parts[2].strip(),
display_code=parts[3].strip() if len(parts) > 3 else None,
description=parts[4].strip() if len(parts) > 4 else None,
raw_line=line,
))
return out
def load_attribute_dictionary(path: Path) -> list[AttributeDef]:
out = []
for line in _read_lines(path):
if not line.strip():
continue
parts = [p.strip() for p in line.rstrip("\r\n").split("|")]
if len(parts) < 3:
continue
try:
attribute_id = int(parts[1])
except ValueError:
continue
source_type = parts[2]
def f(i):
if len(parts) <= i or parts[i] == "":
return None
try:
return float(parts[i])
except ValueError:
return None
out.append(AttributeDef(
name=parts[0],
attribute_id=attribute_id,
source_type_name=source_type,
decoder_type=TYPE_MAP.get(source_type, "?"),
min_value=f(3),
max_value=f(4),
raw_line=line,
))
return out

870
src/cm93db/geometry.py Normal file
View File

@@ -0,0 +1,870 @@
from __future__ import annotations
from dataclasses import dataclass
import math
from typing import Iterable, Any
try:
from shapely import normalize as _shapely_normalize
except ImportError:
_shapely_normalize = None
from shapely.geometry import Point, LineString, MultiLineString, Polygon, MultiPolygon
from shapely.geometry.base import BaseGeometry
from shapely.ops import unary_union, polygonize_full
from shapely.validation import explain_validity
from .cm93_binary import ParsedCell, FeatureRecord
from .precision import CM93_SEMIMAJOR_M
# 已通过 CM93 实际数据审计的 Area normalization 规则:
# 1. valid Polygon/MultiPolygon 完全原样 PASS-THROUGH;
# 2. invalid Polygon/MultiPolygon 仅使用原始 boundary 做 noding + polygonize_full;
# 3. planar face 采用 odd-even parity 选择;
# 4. 只对修复后的 candidate 做 GEOS canonical normalize;
# 5. 不使用 make_valid() / buffer(0) 覆盖原始 CM93 语义;
# 6. raw_feature_blob / chart_raw_record 仍由 importer 原样保留。
_NORMALIZATION_BOUNDARY_TOLERANCE = 1e-10
_POINT_ON_SEGMENT_EPS = 1e-12
@dataclass(frozen=True)
class CoverageInfo:
source_record_index: int
geom_unshifted: BaseGeometry
wgsox: float
wgsoy: float
compilation_scale: int | None
horizontal_datum: int | None
vertical_datum: int | None
attributes: dict[str, Any]
geometry_repaired: bool
geometry_issue: str | None
@dataclass
class BuiltFeature:
feature: FeatureRecord
geometry_type: str
geom: BaseGeometry | None
coverage_record_index: int | None
geometry_issue: str | None
geometry_repaired: bool
soundings: list[tuple[int, int, int, float, Point, int | None]]
attributes_map: dict[str, Any]
def attr_map(feature: FeatureRecord) -> dict[str, Any]:
out: dict[str, Any] = {}
for a in feature.attributes:
if a.name in out:
prev = out[a.name]
if not isinstance(prev, list) or (prev and not isinstance(prev[0], (dict, list))):
out[a.name] = [prev, a.value]
else:
prev.append(a.value)
else:
out[a.name] = a.value
if feature.related_indices:
out["_cm93_related_indices"] = feature.related_indices
if feature.relation20_raw is not None:
out["_cm93_relation20_raw"] = feature.relation20_raw
return out
def raw_xy_to_mercator(
cell: ParsedCell,
x: int,
y: int,
wgsox: float = 0.0,
wgsoy: float = 0.0,
) -> tuple[float, float]:
h = cell.header
easting = x * h.x_rate + h.easting_min - wgsox
northing = y * h.y_rate + h.northing_min - wgsoy
return easting, northing
def mercator_to_lonlat(easting: float, northing: float) -> tuple[float, float]:
lon = math.degrees(easting / CM93_SEMIMAJOR_M)
lat = math.degrees(
2.0 * math.atan(math.exp(northing / CM93_SEMIMAJOR_M))
- math.pi / 2.0
)
# Normalize only after exact formula. This handles cells crossing the dateline.
while lon > 180.0:
lon -= 360.0
while lon < -180.0:
lon += 360.0
return lon, lat
def raw_xy_to_lonlat(
cell: ParsedCell,
x: int,
y: int,
wgsox: float = 0.0,
wgsoy: float = 0.0,
) -> tuple[float, float]:
return mercator_to_lonlat(
*raw_xy_to_mercator(cell, x, y, wgsox, wgsoy)
)
def _oriented_edge_points(
cell: ParsedCell,
edge_idx: int,
usage: int,
) -> list[tuple[int, int]]:
pts = list(cell.edges[edge_idx].points)
if usage & 4:
pts.reverse()
return pts
def _append_joined(
dst: list[tuple[int, int]],
src: list[tuple[int, int]],
) -> None:
if not src:
return
if not dst:
dst.extend(src)
elif dst[-1] == src[0]:
dst.extend(src[1:])
else:
# Preserve discontinuity; audit will report it through geometry validity/issue.
dst.extend(src)
def raw_line_parts(
cell: ParsedCell,
feature: FeatureRecord,
) -> list[list[tuple[int, int]]]:
parts: list[list[tuple[int, int]]] = []
current: list[tuple[int, int]] = []
for edge_idx, usage in feature.geometry_refs:
seg = _oriented_edge_points(cell, edge_idx, usage)
if not seg:
continue
if not current:
current = list(seg)
elif current[-1] == seg[0]:
current.extend(seg[1:])
else:
if len(current) >= 2:
parts.append(current)
current = list(seg)
if len(current) >= 2:
parts.append(current)
return parts
def raw_line_points(
cell: ParsedCell,
feature: FeatureRecord,
) -> list[tuple[int, int]]:
# Compatibility helper for diagnostics. Disconnected parts are concatenated here;
# production geometry uses raw_line_parts() and preserves disconnection.
out: list[tuple[int, int]] = []
for part in raw_line_parts(cell, feature):
out.extend(part)
return out
def raw_area_rings(
cell: ParsedCell,
feature: FeatureRecord,
) -> list[list[tuple[int, int]]]:
rings: list[list[tuple[int, int]]] = []
current: list[tuple[int, int]] = []
start: tuple[int, int] | None = None
for edge_idx, usage in feature.geometry_refs:
seg = _oriented_edge_points(cell, edge_idx, usage)
if not seg:
continue
if start is None:
start = seg[0]
_append_joined(current, seg)
if current and start is not None and current[-1] == start:
if len(current) >= 4:
rings.append(current)
current = []
start = None
if current:
# Keep an open ring visible to audit. Shapely Polygon will close it, so the issue
# is returned explicitly by the Area builder.
rings.append(current)
return rings
def _transform_points(
cell: ParsedCell,
pts: Iterable[tuple[int, int]],
wgsox: float,
wgsoy: float,
) -> list[tuple[float, float]]:
return [
raw_xy_to_lonlat(cell, x, y, wgsox, wgsoy)
for x, y in pts
]
def _area_from_rings(
coord_rings: list[list[tuple[float, float]]],
) -> BaseGeometry | None:
valid_rings = [r for r in coord_rings if len(r) >= 3]
if not valid_rings:
return None
# CM93 edge chains are emitted as closed contours. Most features are one exterior
# followed by holes. If a later contour is not inside the current exterior, treat it
# as another exterior rather than forcing an invalid hole.
polygons: list[Polygon] = []
current_exterior = valid_rings[0]
holes: list[list[tuple[float, float]]] = []
exterior_poly = Polygon(current_exterior)
for r in valid_rings[1:]:
rp = Polygon(r)
try:
inside = exterior_poly.covers(rp.representative_point())
except Exception:
inside = False
if inside:
holes.append(r)
else:
polygons.append(Polygon(current_exterior, holes))
current_exterior = r
holes = []
exterior_poly = Polygon(current_exterior)
polygons.append(Polygon(current_exterior, holes))
if len(polygons) == 1:
return polygons[0]
return MultiPolygon(polygons)
def _geom_issue(
geom: BaseGeometry | None,
extra: str | None = None,
validate: bool = False,
) -> str | None:
parts = []
if extra:
parts.append(extra)
if geom is None:
parts.append("geometry missing")
elif geom.is_empty:
parts.append("geometry empty")
elif validate and not geom.is_valid:
parts.append(explain_validity(geom))
return "; ".join(parts) if parts else None
def _polygon_rings(
geom: BaseGeometry,
) -> list[list[tuple[float, float]]]:
rings: list[list[tuple[float, float]]] = []
def add_polygon(poly: Polygon) -> None:
rings.append([
(float(c[0]), float(c[1]))
for c in poly.exterior.coords
])
for ring in poly.interiors:
rings.append([
(float(c[0]), float(c[1]))
for c in ring.coords
])
if geom.geom_type == "Polygon":
add_polygon(geom)
elif geom.geom_type == "MultiPolygon":
for poly in geom.geoms:
add_polygon(poly)
return rings
def _point_on_segment(
px: float,
py: float,
ax: float,
ay: float,
bx: float,
by: float,
eps: float = _POINT_ON_SEGMENT_EPS,
) -> bool:
vx = bx - ax
vy = by - ay
wx = px - ax
wy = py - ay
cross_value = abs(vx * wy - vy * wx)
if cross_value > eps * max(1.0, abs(vx), abs(vy)):
return False
dot_value = wx * vx + wy * vy
if dot_value < -eps:
return False
vv = vx * vx + vy * vy
if dot_value - vv > eps:
return False
return True
def _odd_even_inside(
point: tuple[float, float],
rings: list[list[tuple[float, float]]],
) -> bool:
x, y = point
crossings = 0
for ring in rings:
for i in range(len(ring) - 1):
x1, y1 = ring[i]
x2, y2 = ring[i + 1]
if _point_on_segment(x, y, x1, y1, x2, y2):
# polygonize face representative_point should not normally lie
# on the input boundary. Treat a numerical boundary hit as inside.
return True
if (y1 > y) != (y2 > y):
x_intersection = (
x1
+ (y - y1) * (x2 - x1) / (y2 - y1)
)
if x_intersection > x:
crossings += 1
return (crossings % 2) == 1
def _canonicalize_polygonal(
geom: BaseGeometry,
) -> BaseGeometry:
if _shapely_normalize is not None:
return _shapely_normalize(geom)
normalizer = getattr(geom, "normalize", None)
if callable(normalizer):
normalized = normalizer()
if normalized is not None:
return normalized
raise RuntimeError(
"CM93 Area normalization requires Shapely/GEOS normalize support"
)
def _normalize_cm93_area(
geom: BaseGeometry | None,
structural_issue: str | None = None,
) -> tuple[BaseGeometry | None, bool, str | None]:
"""
Normalize an OGC-invalid CM93 Area using the audited CM93 fill semantics.
Valid geometry is returned byte/structure untouched.
Open/discontinuous source rings are not silently repaired.
"""
if geom is None:
return None, False, _geom_issue(None, structural_issue)
if geom.is_empty:
return geom, False, _geom_issue(geom, structural_issue)
if geom.geom_type not in ("Polygon", "MultiPolygon"):
return (
geom,
False,
f"area geometry has unexpected type {geom.geom_type}",
)
# Structural source problems are evidence, not retraced-connector semantics.
# Preserve them for audit instead of silently normalizing them away.
if structural_issue:
issue = _geom_issue(
geom,
structural_issue,
validate=True,
)
return geom, False, issue
# Critical compatibility rule: already-valid geometry stays exactly as built.
if geom.is_valid:
return geom, False, None
original_reason = explain_validity(geom)
try:
original_rings = _polygon_rings(geom)
noded_boundary = unary_union(geom.boundary)
polygonized, _cuts, _dangles, _invalid_rings = polygonize_full(
noded_boundary
)
if polygonized.geom_type == "Polygon":
faces = [polygonized]
elif polygonized.geom_type in ("MultiPolygon", "GeometryCollection"):
faces = [
g
for g in polygonized.geoms
if g.geom_type == "Polygon"
]
else:
faces = []
selected: list[Polygon] = []
for face in faces:
rp = face.representative_point()
if _odd_even_inside(
(float(rp.x), float(rp.y)),
original_rings,
):
selected.append(face)
if not selected:
return (
geom,
False,
f"{original_reason}; odd-even normalization produced no polygon face",
)
candidate = unary_union(selected)
if candidate.geom_type not in ("Polygon", "MultiPolygon"):
return (
geom,
False,
f"{original_reason}; odd-even normalization produced {candidate.geom_type}",
)
if candidate.is_empty:
return (
geom,
False,
f"{original_reason}; odd-even normalization produced empty geometry",
)
if not candidate.is_valid:
return (
geom,
False,
f"{original_reason}; normalized candidate invalid: "
f"{explain_validity(candidate)}",
)
# Safety guard learned from the dry-run audit: normalization may remove
# retraced zero-area connector linework, but must never invent a new
# boundary outside the original CM93 line network.
candidate_extra_boundary = candidate.boundary.difference(
noded_boundary
)
if (
not candidate_extra_boundary.is_empty
and float(candidate_extra_boundary.length)
> _NORMALIZATION_BOUNDARY_TOLERANCE
):
return (
geom,
False,
f"{original_reason}; normalized candidate introduces "
f"new boundary length={candidate_extra_boundary.length}",
)
candidate = _canonicalize_polygonal(candidate)
if candidate.geom_type not in ("Polygon", "MultiPolygon"):
return (
geom,
False,
f"{original_reason}; canonical candidate has type "
f"{candidate.geom_type}",
)
if candidate.is_empty or not candidate.is_valid:
return (
geom,
False,
f"{original_reason}; canonical candidate is not valid",
)
return candidate, True, None
except Exception as exc:
return (
geom,
False,
f"{original_reason}; odd-even normalization failed: "
f"{type(exc).__name__}: {exc}",
)
def build_unshifted_geometry(
cell: ParsedCell,
feature: FeatureRecord,
validate: bool = False,
) -> tuple[BaseGeometry | None, str | None]:
prim = feature.geometry_primitive
if prim == 1 and feature.point2d_index is not None:
x, y = cell.point2d[feature.point2d_index]
return Point(raw_xy_to_lonlat(cell, x, y)), None
if prim == 2:
parts = raw_line_parts(cell, feature)
if not parts:
return None, "line has fewer than 2 points"
coords = [
_transform_points(cell, p, 0.0, 0.0)
for p in parts
]
g = (
LineString(coords[0])
if len(coords) == 1
else MultiLineString(coords)
)
return g, _geom_issue(g, validate=validate)
if prim == 4:
raw_rings = raw_area_rings(cell, feature)
open_count = sum(
1
for r in raw_rings
if r and r[0] != r[-1]
)
rings = [
_transform_points(cell, r, 0.0, 0.0)
for r in raw_rings
]
g = _area_from_rings(rings)
extra = f"{open_count} open ring(s)" if open_count else None
return g, _geom_issue(g, extra, validate=validate)
return None, None
def build_coverages(cell: ParsedCell) -> list[CoverageInfo]:
out: list[CoverageInfo] = []
for f in cell.features:
if f.object_class != "_m_sor":
continue
geom, pre_issue = build_unshifted_geometry(
cell,
f,
validate=False,
)
if geom is None:
continue
raw_rings = raw_area_rings(cell, f) if f.geometry_primitive == 4 else []
open_count = sum(
1
for r in raw_rings
if r and r[0] != r[-1]
)
structural_issue = (
f"{open_count} open ring(s)"
if open_count
else pre_issue
)
if f.geometry_primitive == 4:
geom, repaired, issue = _normalize_cm93_area(
geom,
structural_issue=structural_issue,
)
else:
repaired = False
issue = _geom_issue(
geom,
pre_issue,
validate=False,
)
if geom is None:
continue
a = attr_map(f)
def number(name: str, default=0.0):
v = a.get(name, default)
if isinstance(v, list):
v = v[0] if v else default
try:
return float(v)
except (TypeError, ValueError):
return float(default)
def integer(name: str):
if name not in a:
return None
try:
return int(a[name])
except (TypeError, ValueError):
return None
out.append(CoverageInfo(
source_record_index=f.index,
geom_unshifted=geom,
wgsox=number("_wgsox", 0.0),
wgsoy=number("_wgsoy", 0.0),
compilation_scale=integer("CSCALE"),
horizontal_datum=integer("HORDAT"),
vertical_datum=integer("VERDAT"),
attributes=a,
geometry_repaired=repaired,
geometry_issue=issue,
))
return out
def choose_coverage(
coverages: list[CoverageInfo],
geom_unshifted: BaseGeometry | None,
) -> CoverageInfo | None:
if not coverages:
return None
if len(coverages) == 1:
return coverages[0]
if geom_unshifted is None or geom_unshifted.is_empty:
return coverages[0]
minx, miny, maxx, maxy = geom_unshifted.bounds
probe = Point(
(minx + maxx) / 2.0,
(miny + maxy) / 2.0,
)
for cov in coverages:
try:
if cov.geom_unshifted.covers(probe):
return cov
except Exception:
continue
return coverages[0]
def _depth(raw_z: int) -> float:
return (
float(raw_z - 12000)
if raw_z >= 12000
else raw_z / 10.0
)
def build_feature(
cell: ParsedCell,
feature: FeatureRecord,
coverages: list[CoverageInfo],
validate: bool = False,
) -> BuiltFeature:
amap = attr_map(feature)
prim = feature.geometry_primitive
# _m_sor is stored spatially in chart_coverage. Reuse the already-normalized
# coverage object here so chart_feature bbox/repair metadata matches it.
if feature.object_class == "_m_sor":
cov = next(
(
c
for c in coverages
if c.source_record_index == feature.index
),
None,
)
if cov is not None:
return BuiltFeature(
feature=feature,
geometry_type="META",
geom=cov.geom_unshifted,
coverage_record_index=feature.index,
geometry_issue=cov.geometry_issue,
geometry_repaired=cov.geometry_repaired,
soundings=[],
attributes_map=amap,
)
g, issue = build_unshifted_geometry(
cell,
feature,
validate=validate,
)
return BuiltFeature(
feature=feature,
geometry_type="META",
geom=g,
coverage_record_index=feature.index,
geometry_issue=issue,
geometry_repaired=False,
soundings=[],
attributes_map=amap,
)
# CM93 raw geotype 16 is a parent/no-geometry object. It is expected to
# have no own Point/Line/Area and must not be counted as geometry invalid.
if feature.geotype == 16:
return BuiltFeature(
feature=feature,
geometry_type="META",
geom=None,
coverage_record_index=None,
geometry_issue=None,
geometry_repaired=False,
soundings=[],
attributes_map=amap,
)
if prim == 8 and feature.point3d_index is not None:
desc = cell.point3d[feature.point3d_index]
soundings = []
for x, y, z in desc.points:
unshifted = Point(raw_xy_to_lonlat(cell, x, y))
cov = choose_coverage(coverages, unshifted)
wx = cov.wgsox if cov else 0.0
wy = cov.wgsoy if cov else 0.0
p = Point(
raw_xy_to_lonlat(
cell,
x,
y,
wx,
wy,
)
)
soundings.append((
x,
y,
z,
_depth(z),
p,
cov.source_record_index if cov else None,
))
return BuiltFeature(
feature=feature,
geometry_type="SOUNDING",
geom=None,
coverage_record_index=None,
geometry_issue=None,
geometry_repaired=False,
soundings=soundings,
attributes_map=amap,
)
# Unshifted geometry is used only to choose the WGS offset coverage.
# Do not let optional validation of this intermediate copy create a
# duplicate Area issue; final shifted Area is normalized below.
unshifted, pre_issue = build_unshifted_geometry(
cell,
feature,
validate=False,
)
cov = choose_coverage(coverages, unshifted)
wx = cov.wgsox if cov else 0.0
wy = cov.wgsoy if cov else 0.0
geometry_repaired = False
if prim == 1 and feature.point2d_index is not None:
x, y = cell.point2d[feature.point2d_index]
geom = Point(
raw_xy_to_lonlat(
cell,
x,
y,
wx,
wy,
)
)
gtype = "POINT"
issue = _geom_issue(
geom,
pre_issue,
validate=validate,
)
elif prim == 2:
parts = raw_line_parts(cell, feature)
coords = [
_transform_points(cell, p, wx, wy)
for p in parts
]
geom = (
(
LineString(coords[0])
if len(coords) == 1
else MultiLineString(coords)
)
if coords
else None
)
gtype = "LINE"
issue = _geom_issue(
geom,
pre_issue,
validate=validate,
)
elif prim == 4:
raw_rings = raw_area_rings(cell, feature)
open_count = sum(
1
for r in raw_rings
if r and r[0] != r[-1]
)
rings = [
_transform_points(cell, r, wx, wy)
for r in raw_rings
]
raw_geom = _area_from_rings(rings)
structural_issue = (
f"{open_count} open ring(s)"
if open_count
else None
)
geom, geometry_repaired, issue = _normalize_cm93_area(
raw_geom,
structural_issue=structural_issue,
)
gtype = "AREA"
else:
geom = None
gtype = "META" if prim == 0 else "UNKNOWN"
issue = _geom_issue(
geom,
pre_issue,
validate=validate,
)
return BuiltFeature(
feature=feature,
geometry_type=gtype,
geom=geom,
coverage_record_index=(
cov.source_record_index
if cov
else None
),
geometry_issue=issue,
geometry_repaired=geometry_repaired,
soundings=[],
attributes_map=amap,
)

1125
src/cm93db/importer.py Normal file

File diff suppressed because it is too large Load Diff

39
src/cm93db/precision.py Normal file
View File

@@ -0,0 +1,39 @@
from __future__ import annotations
import math
from dataclasses import dataclass
CM93_COORD_MAX = 65535.0
CM93_SEMIMAJOR_M = 6378388.0
@dataclass(frozen=True)
class CellTransform:
easting_min: float
northing_min: float
easting_max: float
northing_max: float
@property
def x_rate(self) -> float:
delta=self.easting_max-self.easting_min
if delta < 0:
delta += CM93_SEMIMAJOR_M * 2.0 * math.pi
return delta / CM93_COORD_MAX
@property
def y_rate(self) -> float:
return (self.northing_max - self.northing_min) / CM93_COORD_MAX
def raw_xy_to_cm93_mercator(raw_x: int, raw_y: int, t: CellTransform) -> tuple[float, float]:
if not 0 <= raw_x <= 65535 or not 0 <= raw_y <= 65535:
raise ValueError("CM93 raw x/y 必须在 0..65535")
return float(raw_x*t.x_rate+t.easting_min), float(raw_y*t.y_rate+t.northing_min)
def apply_mercator_offset(easting: float, northing: float, wgsox: float, wgsoy: float) -> tuple[float, float]:
# CM93 coverage attributes are metre offsets. CM93 Transform applies them by
# subtraction before the Mercator→lon/lat conversion.
return easting-wgsox, northing-wgsoy
def cm93_mercator_to_lonlat(easting: float, northing: float) -> tuple[float, float]:
lon=math.degrees(easting/CM93_SEMIMAJOR_M)
lat=math.degrees(2.0*math.atan(math.exp(northing/CM93_SEMIMAJOR_M))-math.pi/2.0)
return lon,lat

59
src/cm93db/scanner.py Normal file
View File

@@ -0,0 +1,59 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable
SCALE_NATIVE = {
"Z": 20_000_000,
"A": 3_000_000,
"B": 1_000_000,
"C": 200_000,
"D": 100_000,
"E": 50_000,
"F": 20_000,
"G": 7_500,
}
@dataclass(frozen=True)
class CellFile:
path: Path
relative_path: Path
scale_code: str
native_scale: int
@dataclass(frozen=True)
class ScanResult:
root: Path
cells: tuple[CellFile, ...]
object_dictionary: Path | None
attribute_dictionary: Path | None
def _find_case_insensitive(root: Path, names: set[str]) -> Path | None:
lowers = {n.lower() for n in names}
for p in root.rglob("*"):
if p.is_file() and p.name.lower() in lowers:
return p
return None
def scan_cm93(root: Path) -> ScanResult:
root = root.resolve()
if not root.exists():
raise FileNotFoundError(root)
cells = []
for p in root.rglob("*"):
if not p.is_file():
continue
ext = p.suffix.upper().lstrip(".")
if ext in SCALE_NATIVE:
cells.append(
CellFile(
path=p,
relative_path=p.relative_to(root),
scale_code=ext,
native_scale=SCALE_NATIVE[ext],
)
)
cells.sort(key=lambda c: (c.scale_code, str(c.relative_path).lower()))
obj = _find_case_insensitive(root, {"CM93OBJ.DIC"})
attr = _find_case_insensitive(root, {"ATTRLUT.DIC", "CM93ATTR.DIC"})
return ScanResult(root, tuple(cells), obj, attr)

View File

@@ -0,0 +1,21 @@
SCHEMA_VERSION = 1
TARGET_POSTGRESQL_MIN_MAJOR = 14
POSTGIS_REQUIRED = True
NORMALIZED_SRID = 4326
REQUIRED_TABLES = {
"chart_dataset",
"chart_dictionary_object",
"chart_dictionary_attribute",
"chart_cell",
"chart_coverage",
"chart_feature",
"chart_feature_attribute",
"chart_point_geometry",
"chart_line_geometry",
"chart_area_geometry",
"chart_sounding_point",
"chart_raw_record",
"chart_import_job",
"chart_import_event",
}

2
src/cm95pbf/__init__.py Normal file
View File

@@ -0,0 +1,2 @@
__version__ = "0.4.14-s52-opencpn-lookup-flow"
BUILDER_CONTRACT_VERSION = "cm95pbf-builder/0.4.14-s52-opencpn-lookup-flow"

2178
src/cm95pbf/builder.py Normal file

File diff suppressed because it is too large Load Diff

436
src/cm95pbf/cli.py Normal file
View File

@@ -0,0 +1,436 @@
from __future__ import annotations
import argparse
import json
from pathlib import Path
from cm93db.config import load_config
from cm93db.database import connect
from .builder import Cm95TileBuilder, TileJob
from .identity import build_manifest
from .policy import child_tiles
from .production import (
create_production_plan,
migrate_production_release,
parse_zooms,
production_status,
run_production_plan,
)
from .release import finalize_release
DEFAULT_MAPPING = Path("config/mappings/cm95_to_navsea_v1.yaml")
DEFAULT_POLICY = Path("config/pbf/cm95_tile_policy_v1.yaml")
DEFAULT_MANIFEST = Path("build/cm95_fid_manifest_v1.sqlite")
def _add_common_db(p):
p.add_argument("--dataset-id", type=int, default=1)
def _parse_aoi_bbox(value: str | None):
if value is None or not str(value).strip():
return None
parts = [part.strip() for part in str(value).split(",")]
if len(parts) != 4:
raise argparse.ArgumentTypeError("AOI bbox 必须是 west,south,east,north")
try:
return tuple(float(part) for part in parts)
except ValueError as exc:
raise argparse.ArgumentTypeError("AOI bbox 必须是四个数字") from exc
def build_parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser(
prog="cm95pbf",
description="CM95 PostgreSQL/PostGIS → NavSea-compatible MVT/PBF generator",
)
sub = p.add_subparsers(dest="command", required=True)
manifest = sub.add_parser(
"build-fid-manifest",
help="建立全 Dataset 确定性 int32 MVT Feature ID 清单",
)
_add_common_db(manifest)
manifest.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
manifest.add_argument("--rebuild-fid-manifest", action="store_true")
tile = sub.add_parser(
"build-tile",
help="生成单张 CM95 NavSea-compatible PBF",
)
_add_common_db(tile)
tile.add_argument("--z", type=int, required=True)
tile.add_argument("--x", type=int, required=True)
tile.add_argument("--y", type=int, required=True)
tile.add_argument("--profile", choices=["engineering", "delivery"], default="delivery")
tile.add_argument("--mapping", default=str(DEFAULT_MAPPING))
tile.add_argument("--policy", default=str(DEFAULT_POLICY))
tile.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
tile.add_argument("--output-root", default="build/pbf")
tile.add_argument("--report-root", default="build/pbf-reports")
tile.add_argument("--s52-lookups", default=None)
tile.add_argument(
"--s52-point-table",
choices=["Simplified", "Paper"],
default="Paper",
)
tile.add_argument(
"--s52-boundary-table",
choices=["Plain", "Symbolized"],
default="Plain",
)
tile.add_argument(
"--s52-mariner-mode",
choices=["runtime", "reference"],
default="runtime",
help="runtime=保留船员设置相关事实;reference=按固定 S52MarinerSettings 物化,用于 OpenCPN 对照",
)
sample = sub.add_parser(
"build-sample-aoi",
help="生成 z10 parent + z11 children + z12 grandchildren,共 21 张测试 tile",
)
_add_common_db(sample)
sample.add_argument("--z10-x", type=int, default=914)
sample.add_argument("--z10-y", type=int, default=385)
sample.add_argument(
"--profiles",
default="engineering,delivery",
help="逗号分隔:engineering,delivery",
)
sample.add_argument("--mapping", default=str(DEFAULT_MAPPING))
sample.add_argument("--policy", default=str(DEFAULT_POLICY))
sample.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
sample.add_argument("--output-root", default="build/pbf")
sample.add_argument("--report-root", default="build/pbf-reports")
sample.add_argument("--s52-lookups", default=None)
sample.add_argument(
"--s52-point-table",
choices=["Simplified", "Paper"],
default="Paper",
)
sample.add_argument(
"--s52-boundary-table",
choices=["Plain", "Symbolized"],
default="Plain",
)
sample.add_argument(
"--s52-mariner-mode",
choices=["runtime", "reference"],
default="runtime",
help="runtime=保留船员设置相关事实;reference=按固定 S52MarinerSettings 物化,用于 OpenCPN 对照",
)
prod_plan = sub.add_parser(
"plan-production",
help="按 Dataset coverage 枚举生产 tile,并冻结不可变生成计划",
)
_add_common_db(prod_plan)
prod_plan.add_argument("--zooms", default="0-12", help="缩放级别,支持 0-12 或 0,1,...,12")
prod_plan.add_argument("--profile", choices=["engineering", "delivery"], default="delivery")
prod_plan.add_argument("--mapping", default=str(DEFAULT_MAPPING))
prod_plan.add_argument("--policy", default=str(DEFAULT_POLICY))
prod_plan.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
prod_plan.add_argument("--output", default="build/production/cm95_production_plan_v2.json")
prod_plan.add_argument("--s52-lookups", default=None, help="S-52 lookup_rule_ir.json;提供后冻结 SHA256 并进入 production builder")
prod_plan.add_argument(
"--s52-resource-contract",
default=None,
help="S-52 resource_contract.json;S-52 正式生产必须为 PASS,并冻结 SHA256",
)
prod_plan.add_argument("--s52-point-table", choices=["Simplified", "Paper"], default="Paper")
prod_plan.add_argument("--s52-boundary-table", choices=["Plain", "Symbolized"], default="Plain")
prod_plan.add_argument("--s52-mariner-mode", choices=["runtime", "reference"], default="runtime")
prod_plan.add_argument("--aoi-name", default=None, help="可审计 AOI 名称")
prod_plan.add_argument("--aoi-bbox", type=_parse_aoi_bbox, default=None, metavar="W,S,E,N", help="WGS84 AOI bbox;aoi-complete 模式下枚举 bbox 内完整 XYZ 金字塔")
prod_plan.add_argument(
"--tile-enumeration-mode",
choices=["auto", "coverage", "aoi-complete"],
default="auto",
help="auto=有 AOI 时使用 aoi-complete,否则 coverage;正式区域/全国生产应使用 aoi-complete",
)
prod_build = sub.add_parser(
"build-production",
help="按冻结计划批量生成,支持断点续跑、并发和失败重试",
)
prod_build.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
prod_build.add_argument("--release-root", default="build/releases")
prod_build.add_argument("--workers", type=int, default=4)
prod_build.add_argument("--max-attempts", type=int, default=3)
prod_migrate = sub.add_parser(
"migrate-production-release",
help="把旧 release 的 delivery-equivalent PASS tile 复用到新计划,只重建受影响/失败任务(支持 builder contract 升级)",
)
prod_migrate.add_argument("--old-release-dir", required=True)
prod_migrate.add_argument("--new-plan", default="build/production/cm95_production_plan_v1_4_5.json")
prod_migrate.add_argument("--release-root", default="build/releases")
prod_migrate.add_argument("--copy-mode", choices=["hardlink", "copy"], default="hardlink")
prod_migrate.add_argument(
"--invalidate-tiles-file",
default=None,
help="每行一个 z/x/y;这些旧 PASS tile 强制留为 PENDING,适用于 builder contract 的定向重建",
)
prod_status = sub.add_parser(
"status-production",
help="查看生产计划当前账本状态",
)
prod_status.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
prod_status.add_argument("--release-root", default="build/releases")
prod_final = sub.add_parser(
"finalize-production",
help="对完成的生产产物做三方计数/FID/哈希盘点并生成 release manifest",
)
prod_final.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
prod_final.add_argument("--release-root", default="build/releases")
prod_final.add_argument("--style", default="config/style/style.navsea-delivery-full-semantic-unified-20260808-cm95-v1.json")
prod_final.add_argument("--sprite-json", default="config/sprite/semantic-sprite-20260808.json")
prod_final.add_argument("--gap-disposition", default="config/mappings/cm95_gap_disposition_v1.yaml")
prod_final.add_argument("--presentation-conservation-audit", default=None, help="Source→PBF presentation conservation audit JSON;builder 0.4.11+ 正式 release 必填且必须 PASS")
prod_final.add_argument("--runtime-capability-audit", default=None, help="目标 renderer runtime capability audit JSON;提供后必须 PASS")
prod_final.add_argument("--skip-style-contract", action="store_true", help="跳过旧 NavSea style/sprite contract;S-52 runtime release 仍执行 runtime contract gate")
return p
def ensure_manifest(dataset_id: int, manifest_path: Path) -> dict:
cfg = load_config()
with connect(cfg) as conn:
return build_manifest(
conn,
dataset_id,
manifest_path,
rebuild=False,
)
def main() -> None:
args = build_parser().parse_args()
cfg = load_config()
if args.command == "plan-production":
manifest_path = Path(args.manifest)
if not manifest_path.exists():
print("FID manifest 不存在,先自动建立。")
print(json.dumps(ensure_manifest(args.dataset_id, manifest_path), ensure_ascii=False, indent=2))
result = create_production_plan(
db=cfg,
dataset_id=args.dataset_id,
mapping_path=Path(args.mapping),
policy_path=Path(args.policy),
manifest_path=manifest_path,
output_path=Path(args.output),
zooms=parse_zooms(args.zooms),
profile=args.profile,
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
s52_resource_contract_path=(
Path(args.s52_resource_contract)
if args.s52_resource_contract
else None
),
s52_point_table=args.s52_point_table,
s52_boundary_table=args.s52_boundary_table,
s52_mariner_mode=args.s52_mariner_mode,
aoi_name=args.aoi_name,
aoi_bbox_wgs84=args.aoi_bbox,
tile_enumeration_mode=args.tile_enumeration_mode,
)
print(json.dumps({
"plan_id": result["plan_id"],
"release_id": result["release_id"],
"job_count": result["job_count"],
"production_pipeline_version": result.get("production_pipeline_version"),
"s52": result.get("s52"),
"aoi": result.get("aoi"),
"tile_enumeration_mode": result.get("tile_enumeration_mode"),
"source_stats": result["source_stats"],
"output": str(Path(args.output)),
}, ensure_ascii=False, indent=2))
return
if args.command == "build-production":
result = run_production_plan(
db=cfg,
plan_path=Path(args.plan),
release_root=Path(args.release_root),
workers=args.workers,
max_attempts=args.max_attempts,
)
print(json.dumps(result, ensure_ascii=False, indent=2))
states = result.get("states") or {}
if states.get("FAILED", 0) or states.get("PENDING", 0) or states.get("RUNNING", 0):
raise SystemExit(2)
return
if args.command == "migrate-production-release":
result = migrate_production_release(
old_release_dir=Path(args.old_release_dir),
new_plan_path=Path(args.new_plan),
release_root=Path(args.release_root),
copy_mode=args.copy_mode,
invalidate_tiles_file=(Path(args.invalidate_tiles_file) if args.invalidate_tiles_file else None),
)
print(json.dumps(result, ensure_ascii=False, indent=2))
return
if args.command == "status-production":
print(json.dumps(production_status(Path(args.plan), Path(args.release_root)), ensure_ascii=False, indent=2))
return
if args.command == "finalize-production":
result = finalize_release(
Path(args.plan),
Path(args.release_root),
None if args.skip_style_contract else (Path(args.style) if args.style else None),
None if args.skip_style_contract else (Path(args.sprite_json) if args.sprite_json else None),
Path(args.gap_disposition) if args.gap_disposition else None,
Path(args.presentation_conservation_audit) if args.presentation_conservation_audit else None,
Path(args.runtime_capability_audit) if args.runtime_capability_audit else None,
)
print(json.dumps({
"status": result["status"],
"release_id": result["release_id"],
"planned_tile_jobs": result["planned_tile_jobs"],
"pbf_file_count": result["pbf_file_count"],
"planned_output_features": result["planned_output_features"],
"decoded_pbf_features": result["decoded_pbf_features"],
"semantic_completeness": result.get("semantic_completeness"),
"semantic_backlog": result.get("semantic_backlog"),
"non_renderable_omissions": result.get("non_renderable_omissions"),
"non_renderable_omission_layers": result.get("non_renderable_omission_layers"),
"tile_footprint_contract": result.get("tile_footprint_contract"),
"style_sprite_contract": {
"overall": (result.get("style_sprite_contract") or {}).get("overall"),
"contract_unmatched_count": (result.get("style_sprite_contract") or {}).get("contract_unmatched_count"),
"sprite_missing_count": (result.get("style_sprite_contract") or {}).get("sprite_missing_count"),
},
"s52_resource_contract": result.get("s52_resource_contract"),
"s52_runtime_contract": result.get("s52_runtime_contract"),
"s52_light_presentation_contract": result.get("s52_light_presentation_contract"),
"s52_presentation_conservation_contract": result.get("s52_presentation_conservation_contract"),
"s52_runtime_capability_contract": result.get("s52_runtime_capability_contract"),
"quantization_fallbacks": result.get("quantization_fallbacks"),
"quantization_omissions": result.get("quantization_omissions"),
"nav_name_conflict_audit": {
"overall": (result.get("nav_name_conflict_audit") or {}).get("overall"),
"policy": (result.get("nav_name_conflict_audit") or {}).get("policy"),
"report_occurrences": (result.get("nav_name_conflict_audit") or {}).get("report_occurrences"),
"trace_occurrences": (result.get("nav_name_conflict_audit") or {}).get("trace_occurrences"),
"safe_omission_occurrences": (result.get("nav_name_conflict_audit") or {}).get("safe_omission_occurrences"),
"unique_conflict_outputs": (result.get("nav_name_conflict_audit") or {}).get("unique_conflict_outputs"),
"affected_tiles": (result.get("nav_name_conflict_audit") or {}).get("affected_tiles"),
},
"failures": result["failures"],
}, ensure_ascii=False, indent=2))
if result["status"] != "PASS":
raise SystemExit(2)
return
if args.command == "build-fid-manifest":
with connect(cfg) as conn:
result = build_manifest(
conn,
args.dataset_id,
Path(args.manifest),
rebuild=bool(args.rebuild_fid_manifest),
)
print(json.dumps(result, ensure_ascii=False, indent=2))
return
manifest_path = Path(args.manifest)
if not manifest_path.exists():
print("FID manifest 不存在,先自动建立。")
print(
json.dumps(
ensure_manifest(args.dataset_id, manifest_path),
ensure_ascii=False,
indent=2,
)
)
if args.command == "build-tile":
builder = Cm95TileBuilder(
db=cfg,
dataset_id=args.dataset_id,
mapping_path=Path(args.mapping),
policy_path=Path(args.policy),
fid_manifest_path=manifest_path,
output_root=Path(args.output_root) / args.profile,
report_root=Path(args.report_root),
profile=args.profile,
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
s52_point_table=args.s52_point_table,
s52_boundary_table=args.s52_boundary_table,
s52_mariner_mode=args.s52_mariner_mode,
)
result = builder.build_tile(
TileJob(args.z, args.x, args.y)
)
print(json.dumps(result, ensure_ascii=False, indent=2))
return
if args.command == "build-sample-aoi":
profiles = [
token.strip()
for token in str(args.profiles).split(",")
if token.strip()
]
for profile in profiles:
if profile not in {"engineering", "delivery"}:
raise SystemExit(f"invalid profile: {profile}")
jobs = [(10, args.z10_x, args.z10_y)]
jobs.extend(child_tiles(10, args.z10_x, args.z10_y, 11))
jobs.extend(child_tiles(10, args.z10_x, args.z10_y, 12))
results = []
for profile in profiles:
builder = Cm95TileBuilder(
db=cfg,
dataset_id=args.dataset_id,
mapping_path=Path(args.mapping),
policy_path=Path(args.policy),
fid_manifest_path=manifest_path,
output_root=Path(args.output_root) / profile,
report_root=Path(args.report_root),
profile=profile,
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
s52_point_table=args.s52_point_table,
s52_boundary_table=args.s52_boundary_table,
s52_mariner_mode=args.s52_mariner_mode,
)
for z, x, y in jobs:
result = builder.build_tile(TileJob(z, x, y))
results.append(result)
print(
f"[{profile}] {z}/{x}/{y} "
f"features={result['output_feature_count']} "
f"layers={result['output_layer_count']} "
f"hard_gate_unmapped={len(result['hard_gate_unmapped'])}"
)
summary = {
"dataset_id": args.dataset_id,
"root_tile": f"10/{args.z10_x}/{args.z10_y}",
"profiles": profiles,
"tile_count_per_profile": len(jobs),
"results": results,
}
summary_path = Path(args.report_root) / "sample_aoi_summary_v1.json"
summary_path.parent.mkdir(parents=True, exist_ok=True)
summary_path.write_text(
json.dumps(summary, ensure_ascii=False, indent=2),
encoding="utf-8",
)
print(f"SUMMARY: {summary_path}")
return
if __name__ == "__main__":
main()

1124
src/cm95pbf/csp.py Normal file

File diff suppressed because it is too large Load Diff

329
src/cm95pbf/identity.py Normal file
View File

@@ -0,0 +1,329 @@
from __future__ import annotations
import hashlib
from pathlib import Path
import sqlite3
from typing import Iterable
ROLE_PRIMARY = 0
ROLE_BOUNDARY = 1
ROLE_POINT_ON_SURFACE = 2
ROLE_SECONDARY = 3
ROLE_FACTOR = 16
MAX_SIGNED_INT32 = (1 << 31) - 1
def feature_source_key(
dataset_id: int,
source_relative_path: str,
source_record_index: int,
geometry_type: str,
) -> str:
path = str(source_relative_path).replace("\\", "/")
return (
f"cm95:{int(dataset_id)}:{path}:"
f"{int(source_record_index)}:{str(geometry_type).lower()}"
)
def sounding_source_key(
dataset_id: int,
source_relative_path: str,
source_record_index: int,
point_index: int,
) -> str:
return (
feature_source_key(
dataset_id,
source_relative_path,
source_record_index,
"SOUNDING",
)
+ f":point:{int(point_index)}"
)
def dataset_fingerprint(conn, dataset_id: int) -> str:
h = hashlib.sha256()
with conn.cursor() as cur:
cur.execute(
"""
SELECT
source_relative_path,
COALESCE(source_sha256,'') AS source_sha256,
source_size,
scale_code,
native_scale
FROM chart_cell
WHERE dataset_id=%s
ORDER BY source_relative_path
""",
(dataset_id,),
)
for row in cur.fetchall():
line = (
f"{row['source_relative_path']}|{row['source_sha256']}|"
f"{row['source_size']}|{row['scale_code']}|{row['native_scale']}\n"
)
h.update(line.encode("utf-8"))
return h.hexdigest()
class FidManifest:
def __init__(self, path: Path):
self.path = Path(path)
self.db: sqlite3.Connection | None = None
def open(self) -> "FidManifest":
self.path.parent.mkdir(parents=True, exist_ok=True)
self.db = sqlite3.connect(self.path)
self.db.row_factory = sqlite3.Row
return self
def close(self) -> None:
if self.db is not None:
self.db.close()
self.db = None
def __enter__(self) -> "FidManifest":
return self.open()
def __exit__(self, exc_type, exc, tb) -> None:
self.close()
def _require(self) -> sqlite3.Connection:
if self.db is None:
raise RuntimeError("manifest not open")
return self.db
def init_schema(self) -> None:
db = self._require()
db.executescript(
"""
CREATE TABLE IF NOT EXISTS manifest_meta (
k TEXT PRIMARY KEY,
v TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS manifest_identity (
source_key TEXT PRIMARY KEY,
base_ord INTEGER NOT NULL UNIQUE,
source_kind TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_manifest_identity_ord
ON manifest_identity(base_ord);
"""
)
db.commit()
def clear(self) -> None:
db = self._require()
self.init_schema()
db.execute("DELETE FROM manifest_identity")
db.execute("DELETE FROM manifest_meta")
db.commit()
def set_meta(self, key: str, value: str) -> None:
db = self._require()
db.execute(
"INSERT OR REPLACE INTO manifest_meta(k,v) VALUES(?,?)",
(str(key), str(value)),
)
def get_meta(self, key: str) -> str | None:
db = self._require()
row = db.execute(
"SELECT v FROM manifest_meta WHERE k=?",
(str(key),),
).fetchone()
return None if row is None else str(row["v"])
def insert_many(self, rows: Iterable[tuple[str, int, str]]) -> None:
db = self._require()
db.executemany(
"""
INSERT INTO manifest_identity(source_key,base_ord,source_kind)
VALUES(?,?,?)
""",
rows,
)
def commit(self) -> None:
self._require().commit()
def base_ord(self, source_key: str) -> int:
db = self._require()
row = db.execute(
"SELECT base_ord FROM manifest_identity WHERE source_key=?",
(source_key,),
).fetchone()
if row is None:
raise KeyError(f"source identity missing from manifest: {source_key}")
return int(row["base_ord"])
def mvt_id(self, source_key: str, role_code: int = ROLE_PRIMARY) -> int:
if role_code < 0 or role_code >= ROLE_FACTOR:
raise ValueError(f"invalid role code: {role_code}")
value = self.base_ord(source_key) * ROLE_FACTOR + int(role_code)
if value > MAX_SIGNED_INT32:
raise RuntimeError(
"deterministic fid exceeded signed int32; "
"increase identity design before publishing"
)
return value
def build_manifest(
mysql_conn,
dataset_id: int,
manifest_path: Path,
rebuild: bool = False,
) -> dict:
fingerprint = dataset_fingerprint(mysql_conn, dataset_id)
with FidManifest(manifest_path) as manifest:
manifest.init_schema()
old_dataset = manifest.get_meta("dataset_id")
old_fingerprint = manifest.get_meta("dataset_fingerprint")
if old_dataset is not None and not rebuild:
if old_dataset == str(dataset_id) and old_fingerprint == fingerprint:
count = int(
manifest._require()
.execute("SELECT COUNT(*) FROM manifest_identity")
.fetchone()[0]
)
return {
"status": "REUSED",
"dataset_id": dataset_id,
"dataset_fingerprint": fingerprint,
"identity_count": count,
"manifest": str(manifest_path),
}
raise RuntimeError(
"existing fid manifest belongs to a different dataset state; "
"run with --rebuild-fid-manifest"
)
manifest.clear()
base_ord = 0
batch: list[tuple[str, int, str]] = []
with mysql_conn.cursor() as cur:
cur.execute(
"""
SELECT
c.source_relative_path,
f.source_record_index,
f.geometry_type
FROM chart_feature f
JOIN chart_cell c ON c.id=f.cell_id
WHERE f.dataset_id=%s
ORDER BY
c.source_relative_path,
f.source_record_index,
f.geometry_type
""",
(dataset_id,),
)
while True:
mysql_batch = cur.fetchmany(5000)
if not mysql_batch:
break
for row in mysql_batch:
base_ord += 1
batch.append(
(
feature_source_key(
dataset_id,
row["source_relative_path"],
row["source_record_index"],
row["geometry_type"],
),
base_ord,
"feature",
)
)
manifest.insert_many(batch)
batch.clear()
feature_identity_count = base_ord
cur.execute(
"""
SELECT
c.source_relative_path,
f.source_record_index,
s.point_index
FROM chart_sounding_point s
JOIN chart_feature f ON f.id=s.feature_id
JOIN chart_cell c ON c.id=f.cell_id
WHERE f.dataset_id=%s
ORDER BY
c.source_relative_path,
f.source_record_index,
s.point_index
""",
(dataset_id,),
)
while True:
mysql_batch = cur.fetchmany(5000)
if not mysql_batch:
break
for row in mysql_batch:
base_ord += 1
batch.append(
(
sounding_source_key(
dataset_id,
row["source_relative_path"],
row["source_record_index"],
row["point_index"],
),
base_ord,
"sounding_point",
)
)
manifest.insert_many(batch)
batch.clear()
manifest.set_meta("schema", "navsea.cm95.fid-manifest/v1")
manifest.set_meta("dataset_id", str(dataset_id))
manifest.set_meta("dataset_fingerprint", fingerprint)
manifest.set_meta("role_factor", str(ROLE_FACTOR))
manifest.set_meta("feature_identity_count", str(feature_identity_count))
manifest.set_meta("identity_count", str(base_ord))
manifest.commit()
return {
"status": "BUILT",
"dataset_id": dataset_id,
"dataset_fingerprint": fingerprint,
"feature_identity_count": feature_identity_count,
"identity_count": base_ord,
"manifest": str(manifest_path),
}
def validate_manifest(mysql_conn, dataset_id: int, manifest_path: Path) -> dict:
fingerprint = dataset_fingerprint(mysql_conn, dataset_id)
with FidManifest(manifest_path) as manifest:
manifest.init_schema()
dataset = manifest.get_meta("dataset_id")
stored_fp = manifest.get_meta("dataset_fingerprint")
if dataset != str(dataset_id) or stored_fp != fingerprint:
raise RuntimeError(
"fid manifest does not match current dataset; rebuild required"
)
count = int(
manifest._require()
.execute("SELECT COUNT(*) FROM manifest_identity")
.fetchone()[0]
)
return {
"dataset_id": dataset_id,
"dataset_fingerprint": fingerprint,
"identity_count": count,
"manifest": str(manifest_path),
}

633
src/cm95pbf/mapping.py Normal file
View File

@@ -0,0 +1,633 @@
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
import yaml
DELIVERY_PROPERTY_ALLOWLIST = frozenset(
{
"canonical_object_type",
"class_code",
"chart_fill_pattern",
"chart_fill_style",
"chart_icon_image",
"chart_label_position_code",
"chart_label_subtext",
"chart_label_text",
"chart_line_color",
"chart_line_width",
"chart_symbol_code",
"chart_text_color",
"chart_text_style",
"clearance_height_m",
"contour_depth_m",
"depth_max_m",
"depth_min_m",
"depth_value_m",
"display_code",
"least_depth_m",
"light_color_code",
"light_sector_mode",
"arc_id",
"arc_source_code",
"name_ja",
"place_name_en",
"s52_csp",
"s52_csp_resolved",
"s52_csp_rule_opcodes",
"s52_mariner_dependent",
"s52_runtime_csp",
"s52_is_safety_contour",
"s52_display_category",
"s52_area_color",
"s52_area_pattern_code",
"s52_arc_color",
"s52_arc_end_deg",
"s52_arc_outline_color",
"s52_arc_outline_width",
"s52_arc_radius_mm",
"s52_arc_sector_radius_mm",
"s52_arc_start_deg",
"s52_arc_width",
"s52_has_text",
"s52_display_base_promoted",
"s52_hazard_depth_m",
"s52_hazard_procedure",
"s52_hazard_quality_rule_chain",
"s52_hazard_name_rule_chain",
"s52_valsou_present",
"s52_valsou_m",
"s52_isolated_danger",
"s52_surrounding_max_drval1_m",
"s52_surrounding_min_drval2_m",
"s52_quapos_code",
"s52_quasou_codes",
"s52_tecsou_codes",
"s52_status_codes",
"s52_watlev_code",
"s52_expsou_code",
"s52_catobs_code",
"s52_catwrk_code",
"s52_quapos_code_present",
"s52_watlev_code_present",
"s52_expsou_code_present",
"s52_catobs_code_present",
"s52_catwrk_code_present",
"s52_line_color",
"s52_line_complex_code",
"s52_line_count",
"s52_line_sequence",
"s52_line_style",
"s52_line_width",
"s52_line1_style",
"s52_line1_width",
"s52_line1_color",
"s52_line2_style",
"s52_line2_width",
"s52_line2_color",
"s52_rule_opcodes",
"s52_rule_chain",
"s52_rule_chain_version",
"s52_source_object_class",
"s52_source_geometry",
"s52_scamin_override",
"s52_safe_dashed_rule_chain",
"s52_safe_solid_rule_chain",
"s52_resare_plain_rule_chain",
"s52_resare_symbolized_rule_chain",
"s52_lights_metres_rule_chain",
"s52_lights_feet_rule_chain",
"s52_light_arc_color",
"s52_light_arc_radius_mm",
"s52_light_mode",
"s52_light_nominal_range_nm",
"s52_light_obscured",
"s52_light_label_text",
"s52_light_presentations",
"s52_light_rotation_deg",
"s52_light_sector_radius_mm",
"s52_light_symbol_code",
"s52_sector_display_end_deg",
"s52_sector_display_start_deg",
"s52_sector_end_deg",
"s52_sector_start_deg",
"s52_sector_sweep_deg",
"s52_display_priority",
"s52_priority_override",
"s52_lookup_rcid",
"s52_object_class",
"s52_symbol_code",
"s52_symbol_count",
"s52_symbol_rotation_deg",
"s52_symbol_rotation_source",
"s52_symbol_rule_sequence",
"s52_symbol_sequence",
"s52_sounding_depth_m",
"s52_sounding_display_depth",
"s52_sounding_depth_unit",
"s52_sounding_is_shallow",
"s52_sounding_low_quality",
"s52_sounding_prefix",
"s52_sounding_swept",
"s52_sounding_symbol_sequence",
"s52_soundg_point_csp",
"s52_table_name",
"s52_text_literal",
"s52_text1_value",
"s52_text1_hjust",
"s52_text1_vjust",
"s52_text1_space",
"s52_text1_size",
"s52_text1_body_size",
"s52_text1_charspec",
"s52_text1_xoff",
"s52_text1_yoff",
"s52_text1_color",
"s52_text1_group",
"s52_text2_value",
"s52_text2_hjust",
"s52_text2_vjust",
"s52_text2_space",
"s52_text2_size",
"s52_text2_body_size",
"s52_text2_charspec",
"s52_text2_xoff",
"s52_text2_yoff",
"s52_text2_color",
"s52_text2_group",
"s52_text3_value",
"s52_text3_hjust",
"s52_text3_vjust",
"s52_text3_space",
"s52_text3_size",
"s52_text3_body_size",
"s52_text3_charspec",
"s52_text3_xoff",
"s52_text3_yoff",
"s52_text3_color",
"s52_text3_group",
"s52_text_count",
"shape_class_code",
}
)
DELIVERY_MAIN_STATUSES = frozenset(
{
"MAPPED",
"MAPPED_CONDITIONAL",
"COMPOSITE",
"MAPPED_DERIVED_POINTS",
}
)
DELIVERY_DERIVED_STATUSES = frozenset(
{
"MAPPED",
"MAPPED_CONDITIONAL",
"DERIVED_LABEL_ONLY",
}
)
NAV_COMPONENT_CLASSES = frozenset(
{
"LIGHTS",
"BCNLAT",
"BCNSPP",
"BOYSPP",
"RTPBCN",
"TOWERS",
"_extgn",
}
)
LIGHT_COLOUR = {
1: ("white", "W"),
3: ("red", "R"),
4: ("green", "G"),
6: ("yellow", "Y"),
}
LIGHT_CHARACTER = {
1: "F",
2: "Fl",
3: "LFl",
4: "Q",
5: "VQ",
6: "UQ",
7: "Iso",
8: "Oc",
9: "IQ",
10: "IVQ",
11: "IUQ",
12: "Mo",
13: "F+Fl",
14: "Fl+LFl",
15: "Oc+Fl",
16: "F+LFl",
17: "Al.Oc",
18: "Al.LFl",
19: "Al.Fl",
20: "Al.Gr",
21: "Q+LFl",
22: "VQ+LFl",
23: "UQ+LFl",
}
NATSUR_TEXT = {
1: "M",
2: "Cy",
3: "Si",
4: "S",
5: "St",
6: "G",
7: "P",
8: "Cb",
9: "R",
11: "Lv",
14: "Co",
17: "Sh",
18: "Bo",
}
def scalarize(value: Any) -> Any:
if value is None:
return None
if isinstance(value, bool):
return value
if isinstance(value, (int, float, str)):
return value
if isinstance(value, (list, dict, tuple)):
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
return str(value)
def get_path(data: dict[str, Any], path: str) -> Any:
cur: Any = data
for token in str(path).split("."):
if isinstance(cur, dict) and token in cur:
cur = cur[token]
else:
return None
return cur
def first_direct_name(attrs: dict[str, Any]) -> str | None:
"""Return a direct source-owned navigation name, never a proximity-derived text."""
for key in ("OBJNAM", "NOBJNM"):
value = attrs.get(key)
if isinstance(value, str):
value = value.strip()
if value:
return value
return None
def unique_direct_names(rows: list[dict[str, Any]]) -> list[str]:
out: list[str] = []
for attrs in rows:
value = first_direct_name(attrs)
if value and value not in out:
out.append(value)
return out
def format_number(value: Any) -> str:
if value is None:
return ""
try:
v = float(value)
except Exception:
return str(value)
if abs(v - round(v)) < 1e-9:
return str(int(round(v)))
return f"{v:.2f}".rstrip("0").rstrip(".")
def format_depth(value: Any) -> str | None:
if value is None:
return None
return format_number(value)
def format_clearance(value: Any) -> str | None:
if value is None:
return None
return f"{format_number(value)}m"
def _depth_token(value: Any) -> str:
return format_number(value)
# Current NavSea unified style only consumes these depth-area buckets.
# Keep this list aligned with style.navsea-delivery-full-semantic-unified-20260808.
NAVSEA_DEPTH_ZONE_INTERVALS: tuple[tuple[float, float, str], ...] = (
(0, 2, "depth_zone_0-2m"),
(0, 5, "depth_zone_0-5m"),
(0, 10, "depth_zone_0-10m"),
(0, 20, "depth_zone_0-20m"),
(2, 5, "depth_zone_2-5m"),
(2, 10, "depth_zone_2-10m"),
(2, 20, "depth_zone_2-20m"),
(5, 10, "depth_zone_5-10m"),
(5, 20, "depth_zone_5-20m"),
(10, 20, "depth_zone_10-20m"),
(20, 100, "depth_zone_20-100m"),
(20, 200, "depth_zone_20-200m"),
(100, 200, "depth_zone_100-200m"),
(200, 500, "depth_zone_200-500m"),
(500, 1000, "depth_zone_500-1000m"),
(1000, 2000, "depth_zone_1000-2000m"),
(2000, 3000, "depth_zone_2000-3000m"),
(3000, 4000, "depth_zone_3000-4000m"),
(4000, 5000, "depth_zone_4000-5000m"),
(5000, 6000, "depth_zone_5000-6000m"),
(6000, 7000, "depth_zone_6000-7000m"),
(7000, 8000, "depth_zone_7000-8000m"),
(8000, 9000, "depth_zone_8000-9000m"),
)
def _smallest_supported_depth_bucket(lo: float, hi: float | None) -> str | None:
if lo >= 9000 and (hi is None or hi >= lo):
return "depth_zone_over_9000m"
candidates: list[tuple[float, float, float, str]] = []
for bucket_lo, bucket_hi, token in NAVSEA_DEPTH_ZONE_INTERVALS:
if hi is None:
# A lower-bounded zone uses the tightest bucket containing the lower bound.
if bucket_lo <= lo < bucket_hi:
width = bucket_hi - bucket_lo
edge_distance = abs(bucket_lo - lo)
candidates.append((width, edge_distance, -bucket_lo, token))
else:
# Prefer the smallest supported interval that fully contains the source range.
if bucket_lo <= lo and hi <= bucket_hi:
width = bucket_hi - bucket_lo
edge_distance = abs(bucket_lo - lo) + abs(bucket_hi - hi)
candidates.append((width, edge_distance, -bucket_lo, token))
if not candidates:
return None
candidates.sort()
return candidates[0][3]
def depth_zone_style(lo: Any, hi: Any) -> str:
"""Normalize CM95 depth ranges to the existing NavSea style vocabulary.
Arbitrary CM95 intervals such as 7.5-10m, 13-20m or 20-50m must never
leak into delivery PBF because the current style does not consume them.
The chosen style bucket is the smallest existing NavSea interval that fully
contains the source range. The original DRVAL1/DRVAL2 remain available in
engineering trace/source attributes.
"""
if lo is None and hi is None:
return "water_area"
try:
lv = float(lo) if lo is not None else 0.0
except Exception:
return "water_area"
hv: float | None
if hi is None:
hv = None
else:
try:
hv = float(hi)
except Exception:
hv = None
if hv is not None and hv < lv:
lv, hv = hv, lv
token = _smallest_supported_depth_bucket(lv, hv)
if token is not None:
return token
# No truthful single bucket exists in the current style vocabulary.
# Use neutral water rather than inventing a style value that will not render.
return "water_area"
def format_natsur(natsur: Any, natqua: Any) -> str | None:
values = natsur if isinstance(natsur, list) else [natsur] if natsur is not None else []
parts: list[str] = []
for value in values:
try:
code = int(value)
except Exception:
code = None
text = NATSUR_TEXT.get(code, str(value) if value is not None else "")
if text and text not in parts:
parts.append(text)
if not parts:
return None
return "/".join(parts)
def s57_light_color(colour: Any) -> str | None:
values = colour if isinstance(colour, list) else [colour] if colour is not None else []
for value in values:
try:
code = int(value)
except Exception:
continue
if code in LIGHT_COLOUR:
return LIGHT_COLOUR[code][0]
return None
def light_colour_abbrs(colour: Any) -> list[str]:
values = colour if isinstance(colour, list) else [colour] if colour is not None else []
out: list[str] = []
for value in values:
try:
code = int(value)
except Exception:
continue
item = LIGHT_COLOUR.get(code)
if item and item[1] not in out:
out.append(item[1])
return out
def format_light_abbreviation(attrs: dict[str, Any]) -> str | None:
parts: list[str] = []
litchr = attrs.get("LITCHR")
if litchr is not None:
try:
char = LIGHT_CHARACTER.get(int(litchr), f"L{int(litchr)}")
except Exception:
char = str(litchr)
siggrp = attrs.get("SIGGRP")
if siggrp and str(siggrp) not in {"(1)", "1"}:
char += str(siggrp)
parts.append(char)
colours = light_colour_abbrs(attrs.get("COLOUR"))
if colours:
parts.append(",".join(colours))
if attrs.get("SIGPER") is not None:
parts.append(f"{format_number(attrs['SIGPER'])}s")
if attrs.get("HEIGHT") is not None:
parts.append(f"{format_number(attrs['HEIGHT'])}m")
if attrs.get("VALNMR") is not None:
parts.append(f"{format_number(attrs['VALNMR'])}M")
if not parts:
colours = light_colour_abbrs(attrs.get("COLOUR"))
if colours:
return ",".join(colours)
return None
return " ".join(parts)
def nav_icon_from_bcnlat(attrs: dict[str, Any], lighted: bool = False) -> str:
try:
cat = int(attrs.get("CATLAM"))
except Exception:
cat = 0
suffix = {
1: "port",
2: "starboard",
3: "starboard",
4: "port",
}.get(cat, "special")
if lighted:
return f"beacon_light_mark_{suffix}"
return f"beacon_small_{suffix}"
def nav_icon_from_bcnspp(attrs: dict[str, Any], lighted: bool = False) -> str:
return "beacon_light_mark_special" if lighted else "beacon_small_special"
def nav_icon_from_boyspp(attrs: dict[str, Any], lighted: bool = False) -> str:
try:
shape = int(attrs.get("BOYSHP"))
except Exception:
shape = 0
if lighted:
if shape == 4:
return "buoy_pillar_light_special"
return "buoy_light_special"
return {
1: "buoy_cone",
2: "buoy_cylinder",
3: "buoy_sphere",
4: "buoy_pillar",
6: "buoy_barrel_light",
}.get(shape, "buoy_lattice_special")
def nav_icon_from_rtpbcn(attrs: dict[str, Any], lighted: bool = False) -> str:
return "beacon_light" if lighted else "beacon_large"
class MappingRegistry:
def __init__(self, path: Path):
self.path = Path(path)
self.document = yaml.safe_load(self.path.read_text(encoding="utf-8"))
self.rules: dict[tuple[str, str], dict[str, Any]] = {}
for rule in self.document.get("rules", []):
key = (
str(rule.get("cm95_object_class") or ""),
str(rule.get("geometry_type") or ""),
)
self.rules[key] = rule
def resolve(self, object_class: str, geometry_type: str) -> dict[str, Any] | None:
return self.rules.get((str(object_class), str(geometry_type)))
def eval_condition(self, condition: Any, attrs: dict[str, Any]) -> bool:
if condition is None or condition == "":
return True
text = str(condition).strip()
if text.endswith(" present"):
field = text[:-8].strip()
return get_path(attrs, field) is not None
return False
def evaluate_fields(
self,
field_spec: dict[str, Any],
attrs: dict[str, Any],
context: dict[str, Any] | None = None,
) -> dict[str, Any]:
context = context or {}
merged = dict(attrs)
merged.update(context)
out: dict[str, Any] = {}
for field, spec in (field_spec or {}).items():
value = self._evaluate_spec(spec, merged)
value = scalarize(value)
if value is not None and value != "":
out[str(field)] = value
return out
def _evaluate_spec(self, spec: Any, attrs: dict[str, Any]) -> Any:
if not isinstance(spec, dict):
return spec
if "const" in spec:
return spec["const"]
if "from" in spec:
return get_path(attrs, str(spec["from"]))
derive = spec.get("derive")
if derive is None:
return None
text = str(derive)
if text == "wreck_surveyed if VALSOU present else wreck_submerged_dangerous":
return "wreck_surveyed" if attrs.get("VALSOU") is not None else "wreck_submerged_dangerous"
if text == "rock_awash if WATLEV in [2,4,5] else rock_submerged":
try:
watlev = int(attrs.get("WATLEV"))
except Exception:
watlev = None
return "rock_awash" if watlev in {2, 4, 5} else "rock_submerged"
if text == "format_depth(VALDCO)":
return format_depth(attrs.get("VALDCO"))
if text == "format_depth(chart_sounding_point.depth)":
return format_depth(get_path(attrs, "chart_sounding_point.depth"))
if text == "depth_zone_style(DRVAL1,DRVAL2)":
return depth_zone_style(attrs.get("DRVAL1"), attrs.get("DRVAL2"))
if text == "depth_zone_style(DRVAL1,null)":
return depth_zone_style(attrs.get("DRVAL1"), None)
if text == "format_natsur(NATSUR,NATQUA)":
return format_natsur(attrs.get("NATSUR"), attrs.get("NATQUA"))
if text == "_texta.text":
return get_path(attrs, "_texta.text")
if text == "s57_colour_to_light_color(COLOUR)":
return s57_light_color(attrs.get("COLOUR"))
if text == "format_light_abbreviation(LITCHR,SIGGRP,SIGPER,COLOUR,HEIGHT,VALNMR)":
return format_light_abbreviation(attrs)
if text == "first_present(OBJNAM,NOBJNM)":
return first_direct_name(attrs)
if text == "sector if SECTR1/SECTR2 present else omni":
return "sector" if attrs.get("SECTR1") is not None or attrs.get("SECTR2") is not None else "omni"
if text == "format_clearance(VERCLR)":
return format_clearance(attrs.get("VERCLR"))
if text == "nav_icon_from_bcnlat_attributes":
return nav_icon_from_bcnlat(attrs, lighted=False)
if text == "nav_icon_from_bcnspp_attributes":
return nav_icon_from_bcnspp(attrs, lighted=False)
if text == "nav_icon_from_boyspp_attributes":
return nav_icon_from_boyspp(attrs, lighted=False)
if text == "nav_icon_from_rtpbcn_attributes":
return nav_icon_from_rtpbcn(attrs, lighted=False)
return None
def minimize_delivery_properties(self, properties: dict[str, Any]) -> dict[str, Any]:
return {
key: scalarize(value)
for key, value in properties.items()
if key in DELIVERY_PROPERTY_ALLOWLIST and scalarize(value) is not None
}

98
src/cm95pbf/policy.py Normal file
View File

@@ -0,0 +1,98 @@
from __future__ import annotations
from dataclasses import dataclass
import math
WEB_MERCATOR_HALF_WORLD = 20037508.342789244
MAX_MERCATOR_LAT = 85.0511287798066
@dataclass(frozen=True)
class TileBounds:
z: int
x: int
y: int
west_m: float
south_m: float
east_m: float
north_m: float
@property
def width_m(self) -> float:
return self.east_m - self.west_m
@property
def height_m(self) -> float:
return self.north_m - self.south_m
@property
def mercator_tuple(self) -> tuple[float, float, float, float]:
return self.west_m, self.south_m, self.east_m, self.north_m
@property
def wgs84_tuple(self) -> tuple[float, float, float, float]:
west, south = mercator_to_lonlat(self.west_m, self.south_m)
east, north = mercator_to_lonlat(self.east_m, self.north_m)
return west, south, east, north
def lonlat_to_mercator(lon: float, lat: float) -> tuple[float, float]:
lat = max(-MAX_MERCATOR_LAT, min(MAX_MERCATOR_LAT, float(lat)))
x = WEB_MERCATOR_HALF_WORLD * float(lon) / 180.0
y = math.log(math.tan((90.0 + lat) * math.pi / 360.0)) / (math.pi / 180.0)
y = WEB_MERCATOR_HALF_WORLD * y / 180.0
return x, y
def mercator_to_lonlat(x: float, y: float) -> tuple[float, float]:
lon = float(x) / WEB_MERCATOR_HALF_WORLD * 180.0
lat = float(y) / WEB_MERCATOR_HALF_WORLD * 180.0
lat = 180.0 / math.pi * (
2.0 * math.atan(math.exp(lat * math.pi / 180.0)) - math.pi / 2.0
)
return lon, lat
def xyz_tile_bounds(z: int, x: int, y: int) -> TileBounds:
n = 1 << int(z)
if x < 0 or y < 0 or x >= n or y >= n:
raise ValueError(f"invalid XYZ tile z={z} x={x} y={y}")
tile_size = (WEB_MERCATOR_HALF_WORLD * 2.0) / n
west = -WEB_MERCATOR_HALF_WORLD + x * tile_size
east = west + tile_size
north = WEB_MERCATOR_HALF_WORLD - y * tile_size
south = north - tile_size
return TileBounds(
z=z,
x=x,
y=y,
west_m=west,
south_m=south,
east_m=east,
north_m=north,
)
def expanded_mercator_bounds(
tile: TileBounds,
buffer_tile_pixels: float,
nominal_tile_pixels: float = 256.0,
) -> tuple[float, float, float, float]:
pad = tile.width_m * float(buffer_tile_pixels) / float(nominal_tile_pixels)
return (
tile.west_m - pad,
tile.south_m - pad,
tile.east_m + pad,
tile.north_m + pad,
)
def child_tiles(z: int, x: int, y: int, child_z: int) -> list[tuple[int, int, int]]:
if child_z < z:
raise ValueError("child_z must be >= z")
factor = 1 << (child_z - z)
out: list[tuple[int, int, int]] = []
for cx in range(x * factor, (x + 1) * factor):
for cy in range(y * factor, (y + 1) * factor):
out.append((child_z, cx, cy))
return out

410
src/cm95pbf/presentation.py Normal file
View File

@@ -0,0 +1,410 @@
from __future__ import annotations
from dataclasses import asdict, dataclass
import re
from typing import Any, Iterable
SUPPORTED_OPCODES = frozenset(
{"SY", "LS", "LC", "AC", "AP", "TX", "TE", "CS", "CA", "MP", "OP"}
)
OPENCPN_NATSUR_TEXT = {
1: "mud", 2: "clay", 3: "silt", 4: "sand", 5: "stone",
6: "gravel", 7: "pebbles", 8: "cobbles", 9: "rock", 11: "lava",
14: "coral", 17: "shells", 18: "boulder", 51: "Wd", 56: "Bo",
}
def _strip_quotes(value: str) -> str:
text = str(value or "").strip()
if len(text) >= 2 and text[0] in {"'", '"'} and text[-1] == text[0]:
return text[1:-1]
return text
def _resolve_text_param(value: str, attrs: dict[str, Any]) -> str:
raw = str(value or "").strip()
quoted = len(raw) >= 2 and raw[0] in {"'", '"'} and raw[-1] == raw[0]
if quoted:
return raw[1:-1]
if "=" in raw:
key, default = raw.split("=", 1)
current = attrs.get(key)
if current not in (None, ""):
return str(current)
return default.rstrip(")")
return raw.rstrip(")")
def _text_attr_value(name: str, attrs: dict[str, Any]) -> str | None:
key = _strip_quotes(name)
if key not in attrs or attrs.get(key) in (None, ""):
return None
value = attrs.get(key)
if isinstance(value, dict):
text_value = value.get("text")
if text_value not in (None, ""):
return str(text_value)
return None
if key == "NATSUR":
values = value if isinstance(value, (list, tuple)) else str(value).split(",")
out = []
for item in values:
try:
out.append(OPENCPN_NATSUR_TEXT.get(int(item), "unk"))
except (TypeError, ValueError):
out.append("unk")
return ",".join(out)
if isinstance(value, (list, tuple)):
return ",".join(str(x) for x in value)
return str(value)
def _format_te_value(fmt_raw: str, attr_raw: str, attrs: dict[str, Any]) -> str | None:
value = _text_attr_value(attr_raw, attrs)
if value is None:
return None
fmt = _strip_quotes(fmt_raw).replace("lf", "f")
match = re.search(r"%(?:[-+0 #]*\d*(?:\.\d+)?)?([csfdi])", fmt)
try:
if match is None:
return fmt
conversion = match.group(1)
if conversion == "f":
return fmt % float(value)
if conversion in {"d", "i"}:
return fmt % int(float(value))
if conversion == "c":
return fmt % (int(float(value)) if str(value).lstrip("+-").isdigit() else str(value)[0])
return fmt % value
except (TypeError, ValueError, IndexError):
return None
def materialize_text_hints(instruction: str, attrs: dict[str, Any]) -> dict[str, Any]:
"""Materialize OpenCPN-compatible TX/TE rules into at most three text slots."""
out: dict[str, Any] = {}
slot = 0
for rule in parse_instruction_string(instruction):
if rule.opcode not in {"TX", "TE"}:
continue
args = rule.args
if rule.opcode == "TX":
if len(args) < 9:
continue
first = str(args[0]).strip()
if len(first) >= 2 and first[0] in {"'", '"'} and first[-1] == first[0]:
text = _strip_quotes(first)
else:
text = _text_attr_value(first, attrs)
layout = args[1:9]
else:
if len(args) < 10:
continue
text = _format_te_value(args[0], args[1], attrs)
layout = args[2:10]
if text in (None, ""):
continue
slot += 1
if slot > 3:
break
prefix = f"s52_text{slot}_"
out[prefix + "value"] = str(text)
out[prefix + "hjust"] = _resolve_text_param(layout[0], attrs)
out[prefix + "vjust"] = _resolve_text_param(layout[1], attrs)
out[prefix + "space"] = _resolve_text_param(layout[2], attrs)
charspec = _resolve_text_param(layout[3], attrs)
out[prefix + "charspec"] = charspec
if len(charspec) >= 4 and charspec[3:].isdigit():
body_size = int(charspec[3:])
out[prefix + "body_size"] = body_size
# OpenCPN normalizes S-52 body size to the user ChartText base
# instead of rendering 18/20 as literal pixel sizes. Use a
# deterministic 10px web base; a future NavSea text-scale setting
# can shift this derived size without losing the raw body size.
normalized = int((min(20, max(0, body_size)) / 20.0) * 4.0)
out[prefix + "size"] = max(10, 10 + normalized)
out[prefix + "xoff"] = _resolve_text_param(layout[4], attrs)
out[prefix + "yoff"] = _resolve_text_param(layout[5], attrs)
out[prefix + "color"] = _resolve_text_param(layout[6], attrs)
out[prefix + "group"] = _resolve_text_param(layout[7], attrs)
if slot:
out["s52_text_count"] = slot
return out
def materialize_symbol_hints(instruction: str, attrs: dict[str, Any]) -> dict[str, Any]:
"""Resolve generic SY orientation parameters without changing symbol choice."""
symbols: list[dict[str, Any]] = []
for rule in parse_instruction_string(instruction):
if rule.opcode != "SY" or not rule.args:
continue
entry: dict[str, Any] = {
"sequence": rule.sequence,
"code": rule.args[0].strip().strip("'\"")[:8],
}
if len(rule.args) >= 2:
source = rule.args[1].strip().strip("'\"")
raw_value = attrs.get(source, source)
try:
entry["rotation_deg"] = float(raw_value) % 360.0
entry["rotation_source"] = source
except (TypeError, ValueError):
pass
symbols.append(entry)
if not symbols:
return {}
out: dict[str, Any] = {}
if len(symbols) > 1 or any("rotation_deg" in item for item in symbols):
out["s52_symbol_rule_sequence"] = symbols
first = symbols[0]
if "rotation_deg" in first:
out["s52_symbol_rotation_deg"] = first["rotation_deg"]
out["s52_symbol_rotation_source"] = first.get("rotation_source")
return out
def materialize_rule_ir(
instruction: str,
attrs: dict[str, Any],
*,
source: str,
procedure: str | None = None,
) -> list[dict[str, Any]]:
"""Return an ordered, executable representation of an S-52 rule string."""
out: list[dict[str, Any]] = []
for rule in parse_instruction_string(instruction):
item = rule.to_dict()
item.update({"source": source, "procedure": procedure})
if rule.opcode in {"TX", "TE"}:
text = materialize_text_hints(rule.raw, attrs)
if text.get("s52_text1_value") not in (None, ""):
item["text"] = {
key.removeprefix("s52_text1_"): value
for key, value in text.items()
if key.startswith("s52_text1_")
}
if rule.opcode == "OP":
item["side_effect"] = "upstream_parser_noop"
out.append(item)
return out
class PresentationParseError(ValueError):
pass
@dataclass(frozen=True)
class PresentationRule:
opcode: str
args_raw: str
args: tuple[str, ...]
raw: str
sequence: int
def to_dict(self) -> dict[str, Any]:
data = asdict(self)
data["args"] = list(self.args)
return data
def _split_args(text: str) -> tuple[str, ...]:
if not text.strip():
return ()
out: list[str] = []
buf: list[str] = []
quote: str | None = None
depth = 0
i = 0
while i < len(text):
ch = text[i]
if quote is not None:
buf.append(ch)
if ch == quote:
quote = None
elif ch == "\\" and i + 1 < len(text):
i += 1
buf.append(text[i])
elif ch in {"'", '"'}:
quote = ch
buf.append(ch)
elif ch == "(":
depth += 1
buf.append(ch)
elif ch == ")":
depth = max(0, depth - 1)
buf.append(ch)
elif ch == "," and depth == 0:
out.append("".join(buf).strip())
buf.clear()
else:
buf.append(ch)
i += 1
out.append("".join(buf).strip())
return tuple(out)
def parse_instruction_string(instruction: str) -> list[PresentationRule]:
"""Parse S-52 INST using OpenCPN StringToRules/SCANFWRD semantics.
OpenCPN intentionally scans each command to the next semicolon/record end
instead of balancing parentheses. Some production Presentation Library
records rely on that tolerance, so this parser does the same.
"""
text = str(instruction or "").replace("\x1f", ";").strip()
rules: list[PresentationRule] = []
i = 0
while i < len(text):
while i < len(text) and (text[i].isspace() or text[i] == ";"):
i += 1
if i >= len(text):
break
if i + 2 >= len(text):
raise PresentationParseError(f"truncated opcode at {i}: {text!r}")
opcode = text[i : i + 2]
if opcode not in SUPPORTED_OPCODES:
raise PresentationParseError(
f"unsupported opcode {opcode!r} at {i}: {text!r}"
)
start = i
i += 2
while i < len(text) and text[i].isspace():
i += 1
if i >= len(text) or text[i] != "(":
raise PresentationParseError(
f"missing '(' after {opcode} at {i}: {text!r}"
)
i += 1
args_start = i
while i < len(text) and text[i] != ";":
i += 1
payload = text[args_start:i].strip()
if payload.endswith(")"):
payload = payload[:-1].rstrip()
raw = text[start:i].strip()
rules.append(
PresentationRule(
opcode=opcode,
args_raw=payload,
args=_split_args(payload),
raw=raw,
sequence=len(rules),
)
)
return rules
def compile_lookup_ir(row: dict[str, Any]) -> dict[str, Any]:
rules = parse_instruction_string(str(row.get("instruction") or ""))
return {
"id": row.get("id"),
"rcid": row.get("rcid"),
"object_class": row.get("object_class"),
"geometry_type": row.get("geometry_type"),
"table_name": row.get("table_name"),
"display_priority": row.get("display_priority"),
"display_category": row.get("display_category"),
"radar_priority": row.get("radar_priority"),
"attribute_conditions_raw": list(row.get("attribute_conditions_raw") or []),
"instruction": row.get("instruction") or "",
"rules": [rule.to_dict() for rule in rules],
}
def referenced_resource(rule: PresentationRule) -> tuple[str, str] | None:
if not rule.args:
return None
kind_by_opcode = {
"SY": "symbol",
"LC": "line_style",
"AP": "pattern",
"CS": "conditional",
"AC": "color",
}
kind = kind_by_opcode.get(rule.opcode)
if kind is None:
return None
value = rule.args[0].strip().strip("'\"")
if rule.opcode in {"SY", "LC", "AP", "CS"}:
value = value[:8]
return (kind, value) if value else None
def flatten_rules(rows: Iterable[dict[str, Any]]) -> list[PresentationRule]:
out: list[PresentationRule] = []
for row in rows:
out.extend(parse_instruction_string(str(row.get("instruction") or "")))
return out
def static_rule_hints(instruction: str) -> dict[str, Any]:
"""Materialize compact scalar hints; full rule IR remains lookup-side."""
rules = parse_instruction_string(instruction)
out: dict[str, Any] = {}
opcodes = [rule.opcode for rule in rules]
if opcodes:
out["s52_rule_opcodes"] = ",".join(opcodes)
for rule in rules:
args = rule.args
if rule.opcode == "SY" and args and "s52_symbol_code" not in out:
out["s52_symbol_code"] = args[0].strip().strip("'\"")[:8]
elif rule.opcode == "AC" and args and "s52_area_color" not in out:
out["s52_area_color"] = args[0].strip().strip("'\"")
elif rule.opcode == "AP" and args and "s52_area_pattern_code" not in out:
out["s52_area_pattern_code"] = args[0].strip().strip("'\"")[:8]
elif rule.opcode == "LC" and args and "s52_line_complex_code" not in out:
out["s52_line_complex_code"] = args[0].strip().strip("'\"")[:8]
elif rule.opcode == "LS" and len(args) >= 3 and "s52_line_style" not in out:
out["s52_line_style"] = args[0].strip().strip("'\"")
out["s52_line_width"] = args[1].strip().strip("'\"")
out["s52_line_color"] = args[2].strip().strip("'\"")
elif rule.opcode == "CA" and len(args) >= 8:
out["s52_arc_outline_color"] = args[0].strip().strip("\'\"")
out["s52_arc_outline_width"] = args[1].strip().strip("\'\"")
out["s52_arc_color"] = args[2].strip().strip("\'\"")
out["s52_arc_width"] = args[3].strip().strip("\'\"")
out["s52_arc_start_deg"] = args[4].strip().strip("\'\"")
out["s52_arc_end_deg"] = args[5].strip().strip("\'\"")
out["s52_arc_radius_mm"] = args[6].strip().strip("\'\"")
out["s52_arc_sector_radius_mm"] = args[7].strip().strip("\'\"")
elif rule.opcode == "OP" and args:
out.setdefault("s52_priority_override", args[0].strip().strip("\'\""))
elif rule.opcode in {"TX", "TE"}:
out["s52_has_text"] = 1
if rule.opcode == "TX" and args:
literal = args[0].strip()
if len(literal) >= 2 and literal[0] in {"\'", '"'} and literal[-1] == literal[0]:
out.setdefault("s52_text_literal", literal[1:-1])
elif rule.opcode == "CS" and args:
name = args[0].strip().strip("'\"")[:8]
existing = out.get("s52_csp")
out["s52_csp"] = f"{existing},{name}" if existing else name
line_rules = [
rule for rule in rules if rule.opcode == "LS" and len(rule.args) >= 3
]
for index, rule in enumerate(line_rules, 1):
prefix = f"s52_line{index}_"
out[prefix + "style"] = rule.args[0].strip().strip("'\"")
out[prefix + "width"] = rule.args[1].strip().strip("'\"")
out[prefix + "color"] = rule.args[2].strip().strip("'\"")
if len(line_rules) > 1:
out["s52_line_sequence"] = "|".join(
",".join(arg.strip().strip("'\"") for arg in rule.args[:3])
for rule in line_rules
)
out["s52_line_count"] = len(line_rules)
symbol_codes = [
rule.args[0].strip().strip("'\"")[:8]
for rule in rules
if rule.opcode == "SY" and rule.args
]
if symbol_codes:
out.setdefault("s52_symbol_code", symbol_codes[0])
out["s52_symbol_sequence"] = ",".join(symbol_codes)
if len(symbol_codes) > 1:
out["s52_symbol_count"] = len(symbol_codes)
return out

View File

@@ -0,0 +1,114 @@
from __future__ import annotations
from decimal import Decimal, InvalidOperation
import hashlib
import json
from typing import Any
LIGHT_PRESENTATION_SIGNATURE_FIELDS = (
"source_record_index",
"s52_csp_resolved",
"s52_display_priority",
"s52_display_category",
"s52_light_mode",
"s52_light_symbol_code",
"s52_arc_outline_color",
"s52_arc_outline_width",
"s52_arc_color",
"s52_arc_width",
"s52_arc_start_deg",
"s52_arc_end_deg",
"s52_arc_radius_mm",
"s52_arc_sector_radius_mm",
"s52_sector_start_deg",
"s52_sector_end_deg",
"s52_sector_sweep_deg",
"s52_light_nominal_range_nm",
"s52_light_rotation_deg",
"s52_light_obscured",
"s52_light_label_text",
"s52_text_literal",
)
_NUMERIC_FIELDS = {
"source_record_index",
"s52_arc_outline_width",
"s52_arc_width",
"s52_arc_start_deg",
"s52_arc_end_deg",
"s52_arc_radius_mm",
"s52_arc_sector_radius_mm",
"s52_sector_start_deg",
"s52_sector_end_deg",
"s52_sector_sweep_deg",
"s52_light_nominal_range_nm",
"s52_light_rotation_deg",
}
def _normalize_number(value: Any) -> str | None:
if value in (None, ""):
return None
try:
number = Decimal(str(value))
except (InvalidOperation, ValueError):
return str(value)
if not number.is_finite():
return str(value)
number = number.normalize()
if number == number.to_integral():
return str(number.quantize(Decimal("1")))
text = format(number, "f")
return text.rstrip("0").rstrip(".") if "." in text else text
def canonical_light_presentation(
presentation: dict[str, Any],
) -> dict[str, Any]:
canonical: dict[str, Any] = {}
for field in LIGHT_PRESENTATION_SIGNATURE_FIELDS:
value = presentation.get(field)
if field in _NUMERIC_FIELDS:
value = _normalize_number(value)
elif field == "s52_light_obscured":
if value in (None, ""):
value = None
elif isinstance(value, str):
value = value.strip().lower() in {"1", "true", "yes"}
else:
value = bool(value)
elif value is not None:
value = str(value)
canonical[field] = value
return canonical
def light_presentation_signature(
presentation: dict[str, Any],
) -> str:
canonical = canonical_light_presentation(presentation)
payload = json.dumps(
canonical,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def parse_light_presentations(value: Any) -> list[dict[str, Any]]:
if value in (None, ""):
return []
if isinstance(value, str):
value = json.loads(value)
if not isinstance(value, list):
raise ValueError("s52_light_presentations must decode to a list")
rows: list[dict[str, Any]] = []
for index, item in enumerate(value):
if not isinstance(item, dict):
raise ValueError(
f"s52_light_presentations[{index}] must be an object"
)
rows.append(dict(item))
return rows

1072
src/cm95pbf/production.py Normal file

File diff suppressed because it is too large Load Diff

847
src/cm95pbf/release.py Normal file
View File

@@ -0,0 +1,847 @@
from __future__ import annotations
from collections import Counter
from datetime import datetime, timezone
import hashlib
import json
import re
from pathlib import Path
from typing import Any
import mapbox_vector_tile
from .production import ProductionLedger, load_plan, sha256_file
from .style_contract import audit_pbf_style_and_sprite
RELEASE_SCHEMA = "navsea.cm95.release-manifest/v1"
def _builder_contract_at_least(value: Any, target: tuple[int, int, int]) -> bool:
match = re.match(r"^cm95pbf-builder/(\d+)\.(\d+)\.(\d+)", str(value or ""))
if not match:
return False
return tuple(int(part) for part in match.groups()) >= target
def _utc_now():
return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
def _external_audit_contract(
path: Path | None,
*,
required: bool,
expected_schema: str,
expected_plan_id: str | None = None,
expected_pbf_dir: Path | None = None,
) -> dict[str, Any]:
result: dict[str, Any] = {
"required": required,
"status": "NOT_AUDITED",
"path": None,
"sha256": None,
"schema": None,
"audit_status": None,
"failures": [],
}
if path is None:
if required:
result["status"] = "FAIL"
result["failures"].append("required_audit_missing")
return result
path = Path(path)
result["path"] = str(path)
if not path.is_file():
result["status"] = "FAIL"
result["failures"].append("audit_file_missing")
return result
result["sha256"] = sha256_file(path)
try:
obj = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
result["status"] = "FAIL"
result["failures"].append(
f"audit_parse_error:{type(exc).__name__}"
)
return result
result["schema"] = obj.get("schema")
result["audit_status"] = obj.get("status")
if obj.get("schema") != expected_schema:
result["failures"].append("audit_schema_mismatch")
if obj.get("status") != "PASS":
result["failures"].append("audit_not_pass")
if expected_plan_id is not None and obj.get("plan_id") != expected_plan_id:
result["failures"].append("audit_plan_id_mismatch")
if expected_pbf_dir is not None:
raw_pbf_dir = obj.get("pbf_dir")
if not raw_pbf_dir:
result["failures"].append("audit_pbf_dir_missing")
else:
try:
actual = Path(raw_pbf_dir).resolve()
expected = Path(expected_pbf_dir).resolve()
except Exception:
result["failures"].append("audit_pbf_dir_invalid")
else:
if actual != expected:
result["failures"].append("audit_pbf_dir_mismatch")
for key in (
"expected_multi_light_groups",
"actual_multi_light_groups",
"expected_presentation_records",
"actual_presentation_records",
"signature_mismatch_count",
"occurrence_drift_count",
"occurrence_scope_mismatch_count",
"required_runtime_kinds",
"missing_runtime_kinds",
"noncompliant_runtime_kinds",
):
if key in obj:
result[key] = obj.get(key)
result["status"] = "PASS" if not result["failures"] else "FAIL"
return result
def _collect_nav_name_conflict_expectations(
report_root: Path,
conflict_reports: list[tuple[Path, int]],
) -> dict[str, Any]:
"""Collect ambiguous navigation-name outputs that must remain unlabeled.
This is intentionally fail-closed:
- a report-side conflict must have exactly matching trace rows;
- each trace row must explicitly say nav_name_source=conflict;
- there must be at least two distinct direct-name candidates;
- the corresponding PBF feature is validated later and must omit
chart_label_text rather than guessing one candidate.
"""
expected_refs: dict[tuple[str, str, int], dict[str, Any]] = {}
unique_outputs: dict[tuple[str, int], dict[str, Any]] = {}
issues: list[dict[str, Any]] = []
trace_occurrences = 0
for report_path, report_count in conflict_reports:
rel_report = report_path.relative_to(report_root)
trace_path = report_path.with_name(report_path.stem + ".trace.jsonl")
pbf_rel = rel_report.with_suffix(".pbf").as_posix()
if not trace_path.exists():
issues.append({
"type": "missing_trace_file",
"report": rel_report.as_posix(),
"expected_conflicts": int(report_count),
})
continue
tile_rows: list[dict[str, Any]] = []
with trace_path.open("r", encoding="utf-8") as handle:
for lineno, raw in enumerate(handle, start=1):
raw = raw.strip()
if not raw:
continue
try:
row = json.loads(raw)
except json.JSONDecodeError as exc:
issues.append({
"type": "invalid_trace_json",
"trace": str(trace_path.relative_to(report_root)),
"line": lineno,
"error": str(exc),
})
continue
if isinstance(row, dict) and row.get("nav_name_conflict") is True:
tile_rows.append(row)
trace_occurrences += len(tile_rows)
if len(tile_rows) != int(report_count):
issues.append({
"type": "report_trace_conflict_count_mismatch",
"report": rel_report.as_posix(),
"report_count": int(report_count),
"trace_count": len(tile_rows),
})
for row in tile_rows:
output_id = row.get("output_id")
layer = str(row.get("output_layer") or "")
name_source = str(row.get("nav_name_source") or "")
raw_candidates = row.get("nav_name_candidates") or []
candidates: list[str] = []
for value in raw_candidates:
text = str(value).strip()
if text and text not in candidates:
candidates.append(text)
if output_id is None or not layer:
issues.append({
"type": "invalid_conflict_trace_identity",
"report": rel_report.as_posix(),
"row": row,
})
continue
try:
fid = int(output_id)
except (TypeError, ValueError):
issues.append({
"type": "invalid_conflict_output_id",
"report": rel_report.as_posix(),
"output_id": output_id,
})
continue
if name_source != "conflict":
issues.append({
"type": "conflict_trace_wrong_name_source",
"report": rel_report.as_posix(),
"output_id": fid,
"nav_name_source": name_source,
})
if len(candidates) < 2:
issues.append({
"type": "conflict_trace_insufficient_candidates",
"report": rel_report.as_posix(),
"output_id": fid,
"candidates": candidates,
})
ref_key = (pbf_rel, layer, fid)
if ref_key in expected_refs:
issues.append({
"type": "duplicate_conflict_trace_reference",
"pbf": pbf_rel,
"layer": layer,
"output_id": fid,
})
expected_refs[ref_key] = {
"pbf": pbf_rel,
"layer": layer,
"output_id": fid,
"source_id": row.get("source_id"),
"relation20": row.get("relation20"),
"native_scale": row.get("native_scale"),
"candidates": candidates,
}
unique_key = (layer, fid)
unique = unique_outputs.setdefault(unique_key, {
"output_id": fid,
"output_layer": layer,
"source_id": row.get("source_id"),
"relation20": row.get("relation20"),
"native_scale": row.get("native_scale"),
"nav_name_candidates": list(candidates),
"occurrences": 0,
"tiles": [],
})
unique["occurrences"] += 1
if pbf_rel not in unique["tiles"]:
unique["tiles"].append(pbf_rel)
for candidate in candidates:
if candidate not in unique["nav_name_candidates"]:
unique["nav_name_candidates"].append(candidate)
return {
"report_occurrences": sum(int(count) for _path, count in conflict_reports),
"trace_occurrences": trace_occurrences,
"expected_refs": expected_refs,
"unique_outputs": unique_outputs,
"issues": issues,
"seen_refs": set(),
"labeled_conflicts": [],
}
def _observe_nav_name_conflict_feature(
state: dict[str, Any],
pbf_rel: str,
layer: str,
fid: Any,
properties: dict[str, Any],
) -> None:
if fid is None:
return
try:
key = (pbf_rel, str(layer), int(fid))
except (TypeError, ValueError):
return
expected = state["expected_refs"].get(key)
if expected is None:
return
state["seen_refs"].add(key)
label = properties.get("chart_label_text")
if label is not None and str(label).strip():
state["labeled_conflicts"].append({
**expected,
"chart_label_text": str(label),
})
def _finalize_nav_name_conflict_audit(state: dict[str, Any]) -> dict[str, Any]:
expected_keys = set(state["expected_refs"])
seen_keys = set(state["seen_refs"])
missing = sorted(expected_keys - seen_keys)
issues = list(state["issues"])
if state["report_occurrences"] != state["trace_occurrences"]:
issues.append({
"type": "global_report_trace_conflict_count_mismatch",
"report_occurrences": state["report_occurrences"],
"trace_occurrences": state["trace_occurrences"],
})
for pbf_rel, layer, fid in missing:
issues.append({
"type": "conflict_output_missing_from_pbf",
"pbf": pbf_rel,
"layer": layer,
"output_id": fid,
})
for item in state["labeled_conflicts"]:
issues.append({
"type": "ambiguous_conflict_was_labeled",
**item,
})
unique_details = []
for row in state["unique_outputs"].values():
item = dict(row)
item["tiles"] = sorted(item.get("tiles") or [])
unique_details.append(item)
unique_details.sort(
key=lambda row: (
int(row.get("native_scale") or 0),
str(row.get("source_id") or ""),
int(row.get("output_id") or 0),
)
)
affected_tiles = sorted({
key[0] for key in state["expected_refs"]
})
safe_omissions = len(seen_keys) - len(state["labeled_conflicts"])
return {
"overall": "PASS" if not issues else "FAIL",
"policy": "AMBIGUOUS_DIRECT_NAMES_OMIT_LABEL_FAIL_CLOSED",
"report_occurrences": int(state["report_occurrences"]),
"trace_occurrences": int(state["trace_occurrences"]),
"safe_omission_occurrences": int(safe_omissions),
"unique_conflict_outputs": len(unique_details),
"affected_tiles": len(affected_tiles),
"labeled_conflict_occurrences": len(state["labeled_conflicts"]),
"missing_pbf_occurrences": len(missing),
"issues": issues,
"unique_conflicts": unique_details,
}
def finalize_release(
plan_path: Path,
release_root: Path,
style_path: Path | None = None,
sprite_json_path: Path | None = None,
gap_disposition_path: Path | None = None,
presentation_conservation_audit_path: Path | None = None,
runtime_capability_audit_path: Path | None = None,
) -> dict[str, Any]:
plan = load_plan(plan_path)
release_dir = Path(release_root) / str(plan["release_id"])
ledger = ProductionLedger(release_dir / "production_run_v1.sqlite")
status = ledger.summary(str(plan["profile"]))
if status["states"].get("FAILED", 0) or status["states"].get("PENDING", 0) or status["states"].get("RUNNING", 0):
raise RuntimeError(f"release has unfinished jobs: {status['states']}")
if status["states"].get("PASS", 0) != int(plan["job_count"]):
raise RuntimeError(f"PASS job count mismatch: {status['states']} plan={plan['job_count']}")
pbf_root = release_dir / "pbf" / str(plan["profile"])
report_root = release_dir / "reports" / str(plan["profile"])
pbf_files = sorted(pbf_root.rglob("*.pbf")) if pbf_root.exists() else []
report_files = sorted(
p for p in report_root.rglob("*.json")
if not p.name.endswith(".trace.json")
) if report_root.exists() else []
hard_gate = []
encoder_dropped = []
name_conflicts = 0
nav_name_conflict_reports: list[tuple[Path, int]] = []
non_renderable_omissions = 0
non_renderable_omission_layers = Counter()
quantization_fallbacks = 0
quantization_omissions = 0
quantization_omission_layers = Counter()
encoder_reconciled_candidates = 0
encoder_reconciled_tiles = 0
report_multi_light_groups = 0
report_light_presentation_records = 0
planned = 0
output = 0
report_layer_counts = Counter()
for path in report_files:
obj = json.loads(path.read_text(encoding="utf-8"))
hard_gate += [{"report": str(path.relative_to(release_dir)), **x} for x in obj.get("hard_gate_unmapped") or []]
encoder_dropped += [{"report": str(path.relative_to(release_dir)), **x} for x in obj.get("encoder_dropped_features") or []]
nav_summary = obj.get("navigation_composites") or {}
tile_name_conflicts = int(nav_summary.get("name_conflict_groups") or 0)
report_multi_light_groups += int(nav_summary.get("multi_light_groups") or 0)
report_light_presentation_records += int(nav_summary.get("light_presentation_records") or 0)
name_conflicts += tile_name_conflicts
if tile_name_conflicts:
nav_name_conflict_reports.append((path, tile_name_conflicts))
label_omissions = obj.get("non_renderable_omissions") or []
non_renderable_omissions += len(label_omissions)
for item in label_omissions:
if isinstance(item, dict):
non_renderable_omission_layers[str(item.get("layer") or "UNKNOWN")] += 1
quantization_fallbacks += len(obj.get("quantization_fallbacks") or [])
omissions = obj.get("quantization_omissions") or []
quantization_omissions += len(omissions)
reconcile_candidates = obj.get("encoder_drop_candidates") or []
if reconcile_candidates:
encoder_reconciled_tiles += 1
for candidate in reconcile_candidates:
if isinstance(candidate, dict):
encoder_reconciled_candidates += len(candidate.get("dropped") or [])
for item in omissions:
if isinstance(item, dict):
quantization_omission_layers[str(item.get("layer") or "UNKNOWN")] += 1
planned += int(obj.get("planned_output_feature_count") or 0)
output += int(obj.get("output_feature_count") or 0)
report_layer_counts.update(obj.get("output_layer_counts") or {})
nav_name_conflict_state = _collect_nav_name_conflict_expectations(
report_root,
nav_name_conflict_reports,
)
inventory = []
decoded_features = 0
decoded_layers = Counter()
duplicate_feature_ids = []
s52_plan = dict(plan.get("s52") or {})
s52_runtime_enabled = bool(s52_plan.get("enabled")) and s52_plan.get("mariner_mode") == "runtime"
s52_runtime_failures: list[str] = []
s52_runtime_csp_counts = Counter()
s52_runtime_layer_counts = Counter()
s52_runtime_dependent_count = 0
s52_runtime_leaks = Counter()
s52_runtime_fact_counts = Counter()
s52_runtime_unknown_fact_exceptions = Counter()
light_presentation_required = _builder_contract_at_least(
plan.get("builder_contract_version"), (0, 4, 11)
)
pbf_multi_light_groups = 0
pbf_light_presentation_records = 0
light_presentation_failures: list[str] = []
s52_dynamic_final_fields = {
"s52_is_safety_contour",
"s52_sounding_prefix",
"s52_sounding_is_shallow",
"s52_isolated_danger",
"s52_display_base_promoted",
}
for path in pbf_files:
raw = path.read_bytes()
decoded = mapbox_vector_tile.decode(raw)
seen = set()
tile_features = 0
for layer, payload in decoded.items():
feats = payload.get("features", [])
decoded_layers[layer] += len(feats)
tile_features += len(feats)
pbf_rel = str(path.relative_to(pbf_root)).replace("\\", "/")
for feat in feats:
fid = feat.get("id")
props = feat.get("properties") or {}
raw_light_presentations = props.get("s52_light_presentations")
if raw_light_presentations not in (None, ""):
try:
light_presentations = (
json.loads(raw_light_presentations)
if isinstance(raw_light_presentations, str)
else raw_light_presentations
)
except Exception as exc:
light_presentation_failures.append(
f"{pbf_rel}:{layer}:{fid} invalid s52_light_presentations JSON: {type(exc).__name__}"
)
light_presentations = []
if not isinstance(light_presentations, list) or len(light_presentations) < 2:
light_presentation_failures.append(
f"{pbf_rel}:{layer}:{fid} multi-light presentation array must contain at least 2 items"
)
else:
pbf_multi_light_groups += 1
pbf_light_presentation_records += len(light_presentations)
for index, presentation in enumerate(light_presentations):
if not isinstance(presentation, dict):
light_presentation_failures.append(
f"{pbf_rel}:{layer}:{fid} light presentation {index} is not an object"
)
continue
if presentation.get("s52_csp_resolved") != "LIGHTS05":
light_presentation_failures.append(
f"{pbf_rel}:{layer}:{fid} light presentation {index} is not resolved LIGHTS05"
)
if "source_record_index" not in presentation:
light_presentation_failures.append(
f"{pbf_rel}:{layer}:{fid} light presentation {index} missing source_record_index"
)
if s52_runtime_enabled:
if props.get("s52_mariner_dependent") == 1:
s52_runtime_dependent_count += 1
s52_runtime_layer_counts[layer] += 1
csp = str(props.get("s52_runtime_csp") or "")
if not csp:
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} missing s52_runtime_csp")
else:
s52_runtime_csp_counts[csp] += 1
for key in s52_dynamic_final_fields:
if key in props:
s52_runtime_leaks[key] += 1
if csp in {"DEPARE01", "DEPARE02"}:
if "depth_min_m" not in props:
if props.get("canonical_object_type") == "dredged_area":
s52_runtime_unknown_fact_exceptions[
"DEPARE:dredged_area_missing_depth_min"
] += 1
else:
s52_runtime_failures.append(
f"{pbf_rel}:{layer}:{fid} {csp} missing depth_min_m"
)
if "s52_area_color" in props:
s52_runtime_leaks["DEPARE:s52_area_color"] += 1
elif csp == "DEPCNT02" and "contour_depth_m" not in props:
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} DEPCNT02 missing contour_depth_m")
elif csp == "SOUNDG02" and "s52_sounding_depth_m" not in props:
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} SOUNDG02 missing s52_sounding_depth_m")
elif csp in {"OBSTRN04", "WRECKS02"} and "s52_hazard_depth_m" not in props:
if csp == "OBSTRN04" and props.get("s52_expsou_code") == 1:
s52_runtime_unknown_fact_exceptions[
"OBSTRN04:EXPSOU1_unknown_hazard_depth"
] += 1
else:
s52_runtime_failures.append(
f"{pbf_rel}:{layer}:{fid} {csp} missing s52_hazard_depth_m"
)
elif "s52_runtime_csp" in props:
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} runtime CSP without mariner-dependent marker")
for key in (
"depth_min_m", "depth_max_m", "contour_depth_m",
"s52_sounding_depth_m", "s52_hazard_depth_m",
"s52_surrounding_max_drval1_m", "s52_surrounding_min_drval2_m",
):
if key in props:
s52_runtime_fact_counts[key] += 1
_observe_nav_name_conflict_feature(
nav_name_conflict_state,
pbf_rel,
layer,
fid,
props,
)
if fid is None:
continue
key = (layer, int(fid))
if key in seen:
duplicate_feature_ids.append({"file": str(path.relative_to(release_dir)), "layer": layer, "id": int(fid)})
seen.add(key)
decoded_features += tile_features
inventory.append({
"path": str(path.relative_to(release_dir)).replace("\\", "/"),
"bytes": len(raw),
"sha256": hashlib.sha256(raw).hexdigest(),
"features": tile_features,
"layers": len(decoded),
})
style_sha = None
sprite_sha = None
style_contract = None
if style_path is not None:
style_path = Path(style_path)
style_sha = sha256_file(style_path)
if sprite_json_path is not None:
sprite_json_path = Path(sprite_json_path)
sprite_sha = sha256_file(sprite_json_path)
style_contract = audit_pbf_style_and_sprite(
pbf_root,
style_path,
sprite_json_path,
)
gap_disposition_sha = None
semantic_backlog = None
semantic_completeness = "NOT_AUDITED"
if gap_disposition_path is not None:
import yaml
gap_disposition_path = Path(gap_disposition_path)
if gap_disposition_path.exists():
gap_disposition_sha = sha256_file(gap_disposition_path)
gap_obj = yaml.safe_load(gap_disposition_path.read_text(encoding="utf-8")) or {}
semantic_backlog = dict(gap_obj.get("summary") or {})
pending = int((semantic_backlog or {}).get("pending_semantic_review") or 0)
semantic_completeness = "COMPLETE" if pending == 0 else "PENDING_REVIEW"
nav_name_conflict_audit = _finalize_nav_name_conflict_audit(
nav_name_conflict_state
)
if s52_runtime_leaks:
s52_runtime_failures.append(f"dynamic final state leaked into runtime PBF: {dict(s52_runtime_leaks)}")
if light_presentation_required:
if report_multi_light_groups != pbf_multi_light_groups:
light_presentation_failures.append(
"report/PBF multi-light group count mismatch: "
f"report={report_multi_light_groups} pbf={pbf_multi_light_groups}"
)
if report_light_presentation_records != pbf_light_presentation_records:
light_presentation_failures.append(
"report/PBF light presentation record count mismatch: "
f"report={report_light_presentation_records} pbf={pbf_light_presentation_records}"
)
s52_light_presentation_contract = {
"status": (
"PASS" if not light_presentation_failures else "FAIL"
) if light_presentation_required else "NOT_APPLICABLE",
"required": light_presentation_required,
"report_multi_light_groups": report_multi_light_groups,
"pbf_multi_light_groups": pbf_multi_light_groups,
"report_light_presentation_records": report_light_presentation_records,
"pbf_light_presentation_records": pbf_light_presentation_records,
"failure_count": len(light_presentation_failures),
"failure_examples": light_presentation_failures[:100],
}
presentation_conservation_required = _builder_contract_at_least(
plan.get("builder_contract_version"), (0, 4, 11)
)
presentation_conservation_contract = _external_audit_contract(
presentation_conservation_audit_path,
required=presentation_conservation_required,
expected_schema="navsea.cm95.s52-presentation-conservation-audit/v1",
expected_plan_id=str(plan.get("plan_id") or ""),
expected_pbf_dir=pbf_root,
)
runtime_capability_contract = _external_audit_contract(
runtime_capability_audit_path,
required=False,
expected_schema="navsea.cm95.s52-runtime-capability-audit/v1",
expected_plan_id=str(plan.get("plan_id") or ""),
expected_pbf_dir=None,
)
s52_runtime_contract = {
"status": ("PASS" if not s52_runtime_failures else "FAIL") if s52_runtime_enabled else "NOT_APPLICABLE",
"enabled": s52_runtime_enabled,
"mariner_dependent_count": s52_runtime_dependent_count,
"runtime_csp_counts": dict(sorted(s52_runtime_csp_counts.items())),
"runtime_layer_counts": dict(sorted(s52_runtime_layer_counts.items())),
"fact_counts": dict(sorted(s52_runtime_fact_counts.items())),
"unknown_fact_exceptions": dict(sorted(s52_runtime_unknown_fact_exceptions.items())),
"dynamic_final_leaks": dict(sorted(s52_runtime_leaks.items())),
"failure_count": len(s52_runtime_failures),
"failure_examples": s52_runtime_failures[:100],
}
tile_enumeration_mode = str(plan.get("tile_enumeration_mode") or "coverage")
pbf_rel_paths = {
path.relative_to(pbf_root).as_posix()
for path in pbf_files
}
explicit_empty_tiles: list[str] = []
missing_nonempty_pbf_tiles: list[str] = []
report_tile_paths: set[str] = set()
for path in report_files:
obj = json.loads(path.read_text(encoding="utf-8"))
rel_pbf = path.relative_to(report_root).with_suffix(".pbf").as_posix()
report_tile_paths.add(rel_pbf)
if rel_pbf in pbf_rel_paths:
continue
planned_count = int(obj.get("planned_output_feature_count") or 0)
output_count = int(obj.get("output_feature_count") or 0)
hard_gate_count = len(obj.get("hard_gate_unmapped") or [])
encoder_drop_count = len(obj.get("encoder_dropped_features") or [])
if (
planned_count == 0
and output_count == 0
and hard_gate_count == 0
and encoder_drop_count == 0
):
explicit_empty_tiles.append(rel_pbf)
else:
missing_nonempty_pbf_tiles.append(rel_pbf)
unexpected_pbf_tiles = sorted(pbf_rel_paths - report_tile_paths)
tile_footprint_contract = {
"status": "NOT_APPLICABLE",
"mode": tile_enumeration_mode,
"planned_tile_jobs": int(plan["job_count"]),
"pbf_file_count": len(pbf_files),
"report_file_count": len(report_files),
"explicit_empty_tile_count": len(explicit_empty_tiles),
"materialized_or_empty_tile_count": len(pbf_files) + len(explicit_empty_tiles),
"missing_nonempty_pbf_tiles": missing_nonempty_pbf_tiles,
"unexpected_pbf_tiles": unexpected_pbf_tiles,
}
if tile_enumeration_mode == "aoi-complete":
tile_footprint_contract["status"] = (
"PASS"
if (
len(report_files) == int(plan["job_count"])
and len(pbf_files) + len(explicit_empty_tiles) == int(plan["job_count"])
and not missing_nonempty_pbf_tiles
and not unexpected_pbf_tiles
)
else "FAIL"
)
s52_resource_contract = {
"status": "NOT_APPLICABLE",
"path": plan.get("s52_resource_contract_path"),
"sha256": None,
"expected_sha256": (plan.get("s52") or {}).get("resource_contract_sha256"),
"defined_resource_counts": None,
"active_resource_counts": None,
"failures": [],
}
expected_resource_sha = str(
(plan.get("s52") or {}).get("resource_contract_sha256") or ""
)
if expected_resource_sha:
resource_path_raw = plan.get("s52_resource_contract_path")
resource_failures: list[str] = []
resource_obj: dict[str, Any] = {}
if not resource_path_raw:
resource_failures.append("missing_resource_contract_path")
else:
resource_path = Path(resource_path_raw)
if not resource_path.exists():
resource_failures.append("resource_contract_file_missing")
else:
actual_sha = sha256_file(resource_path)
s52_resource_contract["sha256"] = actual_sha
if actual_sha != expected_resource_sha:
resource_failures.append("resource_contract_sha256_mismatch")
try:
resource_obj = json.loads(
resource_path.read_text(encoding="utf-8")
)
except Exception as exc:
resource_failures.append(
f"resource_contract_parse_error:{type(exc).__name__}"
)
else:
if resource_obj.get("status") != "PASS":
resource_failures.append("resource_contract_not_pass")
s52_resource_contract["defined_resource_counts"] = (
resource_obj.get("defined_resource_counts")
)
s52_resource_contract["active_resource_counts"] = (
resource_obj.get("active_resource_counts")
)
s52_resource_contract["failures"] = resource_failures
s52_resource_contract["status"] = (
"PASS" if not resource_failures else "FAIL"
)
failures = []
if expected_resource_sha and s52_resource_contract["status"] != "PASS":
failures.append("s52_resource_contract")
if tile_enumeration_mode == "aoi-complete":
if len(report_files) != int(plan["job_count"]):
failures.append("aoi_complete_report_count_mismatch")
if tile_footprint_contract["status"] != "PASS":
failures.append("aoi_complete_tile_footprint_mismatch")
if hard_gate:
failures.append("hard_gate_unmapped")
if encoder_dropped:
failures.append("encoder_dropped_features")
if name_conflicts and nav_name_conflict_audit.get("overall") != "PASS":
failures.append("nav_name_conflict_audit")
if planned != output:
failures.append("planned_vs_report_output_mismatch")
if output != decoded_features:
failures.append("report_output_vs_decoded_pbf_mismatch")
if duplicate_feature_ids:
failures.append("duplicate_feature_ids_within_tile_layer")
if style_contract is not None and style_contract.get("overall") != "PASS":
failures.append("style_or_sprite_contract")
if s52_runtime_enabled and s52_runtime_contract.get("status") != "PASS":
failures.append("s52_runtime_contract")
if light_presentation_required and s52_light_presentation_contract.get("status") != "PASS":
failures.append("s52_light_presentation_contract")
if presentation_conservation_required and presentation_conservation_contract.get("status") != "PASS":
failures.append("s52_presentation_conservation_contract")
if runtime_capability_audit_path is not None and runtime_capability_contract.get("status") != "PASS":
failures.append("s52_runtime_capability_contract")
result = {
"schema": RELEASE_SCHEMA,
"created_at": _utc_now(),
"status": "PASS" if not failures else "FAIL",
"release_id": plan["release_id"],
"plan_id": plan["plan_id"],
"dataset_id": plan["dataset_id"],
"dataset_fingerprint": plan["dataset_fingerprint"],
"profile": plan["profile"],
"zooms": plan["zooms"],
"mapping_sha256": plan["mapping_sha256"],
"policy_sha256": plan["policy_sha256"],
"builder_contract_version": plan.get("builder_contract_version"),
"production_pipeline_version": plan.get("production_pipeline_version"),
"tile_enumeration_mode": tile_enumeration_mode,
"tile_footprint_contract": tile_footprint_contract,
"s52": plan.get("s52"),
"aoi": plan.get("aoi"),
"s52_resource_contract": s52_resource_contract,
"s52_runtime_contract": s52_runtime_contract,
"s52_light_presentation_contract": s52_light_presentation_contract,
"s52_presentation_conservation_contract": presentation_conservation_contract,
"s52_runtime_capability_contract": runtime_capability_contract,
"style_sha256": style_sha,
"sprite_json_sha256": sprite_sha,
"gap_disposition_sha256": gap_disposition_sha,
"semantic_completeness": semantic_completeness,
"semantic_backlog": semantic_backlog,
"style_sprite_contract": style_contract,
"planned_tile_jobs": int(plan["job_count"]),
"pbf_file_count": len(pbf_files),
"report_file_count": len(report_files),
"planned_output_features": planned,
"report_output_features": output,
"decoded_pbf_features": decoded_features,
"non_renderable_omissions": non_renderable_omissions,
"non_renderable_omission_layers": dict(sorted(non_renderable_omission_layers.items())),
"quantization_fallbacks": quantization_fallbacks,
"quantization_omissions": quantization_omissions,
"quantization_omission_layers": dict(sorted(quantization_omission_layers.items())),
"encoder_reconciled_tiles": encoder_reconciled_tiles,
"encoder_reconciled_drop_candidates": encoder_reconciled_candidates,
"hard_gate_unmapped": hard_gate,
"encoder_dropped_features": encoder_dropped,
"nav_name_conflicts": name_conflicts,
"nav_name_conflict_omissions": nav_name_conflict_audit.get("safe_omission_occurrences", 0),
"nav_name_conflict_unique_outputs": nav_name_conflict_audit.get("unique_conflict_outputs", 0),
"nav_name_conflict_audit": nav_name_conflict_audit,
"duplicate_feature_ids": duplicate_feature_ids,
"output_layer_counts": dict(sorted(decoded_layers.items())),
"failures": failures,
"inventory": inventory,
}
out = release_dir / "release_manifest_v1.json"
out.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
return result

652
src/cm95pbf/s52.py Normal file
View File

@@ -0,0 +1,652 @@
from __future__ import annotations
import re
import struct
from typing import Any, Iterable
from .presentation import (
materialize_rule_ir,
materialize_symbol_hints,
materialize_text_hints,
parse_instruction_string,
static_rule_hints,
)
from .csp import (
S52MarinerSettings,
is_mariner_dependent_csp,
materialize_mariner_runtime_facts,
resolve_csp_hints,
)
S52_DISPLAY_PRIORITY_BY_CODE = {
"0": "No data",
"1": "Group 1",
"2": "Area 1",
"3": "Area 2",
"4": "Point Symbol",
"5": "Line Symbol",
"6": "Area Symbol",
"7": "Routing",
"8": "Hazards",
"9": "Mariners",
}
def _apply_presentation_side_effects(out: dict[str, Any]) -> None:
if int(out.get("s52_display_base_promoted") or 0) == 1:
out["s52_display_category"] = "Displaybase"
OPENCPN_CM93_CLASS_SUBSTITUTIONS: dict[str, str] = {
"ITDARE": "DEPARE",
"_m_sor": "M_COVR",
"SPOGRD": "DMPGRD",
"FSHHAV": "FSHFAC",
"OFSPRD": "CTNARE",
"BRTFAC": "BERTHS",
"TNKCON": "SILTNK",
"ROADPT": "ROADWY",
"VEGARE": "VEGATN",
"PRDINS": "PRDARE",
"DYKARE": "DYKCON",
"DYKCRW": "DYKCON",
"SLTPAN": "LNDRGN",
"TELPHC": "CONVYR",
"DIFFUS": "OBSTRN",
"SLIPWY": "SLCONS",
"LNDSTS": "SLCONS",
"RMPARE": "SLCONS",
}
# OpenCPN's current Presentation Library contains these extension Lookup names
# in lowercase, while this CM93 dictionary stores the corresponding classes
# in uppercase. Keep the aliases explicit: lookup matching must remain
# case-sensitive because other extension classes (for example TOPMAR/topmar)
# have distinct semantics.
CM93_PRESENTATION_CLASS_ALIASES: dict[str, str] = {
"RIVBNK": "rivbnk",
"CANBNK": "canbnk",
}
CM93_LANDMARK_CLASS_TO_CATLMK: dict[str, int] = {
"CHIMNY": 3,
"DSHAER": 4,
"FLGSTF": 5,
"FLASTK": 6,
"MSTCON": 7,
"WNDMIL": 18,
"WIMCON": 19,
}
CM93_CLASS_SYNTHETIC_ATTRIBUTES: dict[str, dict[str, Any]] = {
"SILBUI": {"CATSIL": 1},
"SLTPAN": {"CATLND": 15},
"TELPHC": {"CATCON": 1},
"DIFFUS": {"CATOBS": 3},
"SLIPWY": {"CATSLC": 13},
"LNDSTS": {"CATSLC": 11},
"RMPARE": {"CATSLC": 12},
}
def normalize_cm93_object_class_for_s52(object_class: str) -> str:
"""Return the OpenCPN-compatible Presentation Library object class."""
name = str(object_class or "")
if name in CM93_LANDMARK_CLASS_TO_CATLMK:
return "LNDMRK"
if name == "SILBUI":
return "SILTNK"
name = OPENCPN_CM93_CLASS_SUBSTITUTIONS.get(name, name)
return CM93_PRESENTATION_CLASS_ALIASES.get(name, name)
def normalize_cm93_object_for_s52(
object_class: str, geometry_type: str, attrs: dict[str, Any]
) -> tuple[str, dict[str, Any]]:
"""Normalize CM93 class/attributes into the fixed OpenCPN S-52 view."""
source_class = str(object_class or "")
lookup_class = normalize_cm93_object_class_for_s52(source_class)
out = normalize_cm93_attributes_for_s52(attrs)
if source_class == "$AREAS" and str(out.get("$SCODE") or "") == "II25":
out["$SCODE"] = "BACKGROUND"
if str(geometry_type).upper() == "POINT":
catlmk = CM93_LANDMARK_CLASS_TO_CATLMK.get(source_class)
if catlmk is not None:
out.setdefault("CATLMK", catlmk)
aton = (
lookup_class.startswith("LIT")
or lookup_class.startswith("LIGHTS")
or lookup_class.startswith("BCN")
or lookup_class.startswith("_slgto")
or lookup_class.startswith("_boygn")
or lookup_class.startswith("_bcngn")
or lookup_class.startswith("_extgn")
or lookup_class.startswith("TOWERS")
or lookup_class.startswith("BOY")
)
if aton and "OBJNAM" not in out and "INFORM" in out:
out["OBJNAM"] = out.pop("INFORM")
for key, value in CM93_CLASS_SYNTHETIC_ATTRIBUTES.get(source_class, {}).items():
out.setdefault(key, value)
return lookup_class, out
CM93_COLMAR_TO_S57_COLOUR: dict[int, str] = {
1: "4", # green
2: "2", # black
3: "3", # red
4: "6", # yellow
5: "1", # white
6: "11", # orange
7: "2,6", # black/yellow
8: "2,6,2", # black/yellow/black
9: "6,2", # yellow/black
10: "6,2,6", # yellow/black/yellow
11: "3,1", # red/white
12: "4,3,4", # green/red/green
13: "3,4,3", # red/green/red
14: "2,3,2", # black/red/black
15: "6,3,6", # yellow/red/yellow
16: "4,3", # green/red
17: "3,4", # red/green
18: "4,1", # green/white
}
TOPMAR01_FLOATING_SYMBOLS: dict[int, str] = {
1: "TOPMAR02", 2: "TOPMAR04", 3: "TOPMAR10", 4: "TOPMAR12",
5: "TOPMAR13", 6: "TOPMAR14", 7: "TOPMAR65", 8: "TOPMAR17",
9: "TOPMAR16", 10: "TOPMAR08", 11: "TOPMAR07", 12: "TOPMAR14",
13: "TOPMAR05", 14: "TOPMAR06", 17: "TMARDEF2", 18: "TOPMAR10",
19: "TOPMAR13", 20: "TOPMAR14", 21: "TOPMAR13", 22: "TOPMAR14",
23: "TOPMAR14", 24: "TOPMAR02", 25: "TOPMAR04", 26: "TOPMAR10",
27: "TOPMAR17", 28: "TOPMAR18", 29: "TOPMAR02", 30: "TOPMAR17",
31: "TOPMAR14", 32: "TOPMAR10", 33: "TMARDEF2",
}
TOPMAR01_RIGID_SYMBOLS: dict[int, str] = {
1: "TOPMAR22", 2: "TOPMAR24", 3: "TOPMAR30", 4: "TOPMAR32",
5: "TOPMAR33", 6: "TOPMAR34", 7: "TOPMAR85", 8: "TOPMAR86",
9: "TOPMAR36", 10: "TOPMAR28", 11: "TOPMAR27", 12: "TOPMAR14",
13: "TOPMAR25", 14: "TOPMAR26", 15: "TOPMAR88", 16: "TOPMAR87",
17: "TMARDEF1", 18: "TOPMAR30", 19: "TOPMAR33", 20: "TOPMAR34",
21: "TOPMAR33", 22: "TOPMAR34", 23: "TOPMAR34", 24: "TOPMAR22",
25: "TOPMAR24", 26: "TOPMAR30", 27: "TOPMAR86", 28: "TOPMAR89",
29: "TOPMAR22", 30: "TOPMAR86", 31: "TOPMAR14", 32: "TOPMAR30",
33: "TMARDEF1",
}
def _s57_list_string(value: Any) -> str | None:
if value is None:
return None
if isinstance(value, (list, tuple)):
return ",".join(str(int(x)) for x in value)
if isinstance(value, bool):
return str(int(value))
if isinstance(value, int):
return str(value)
if isinstance(value, float) and value.is_integer():
return str(int(value))
return str(value)
def normalize_cm93_attributes_for_s52(attrs: dict[str, Any]) -> dict[str, Any]:
"""Return the fixed OpenCPN CM93-to-S57 typed attribute view."""
out = dict(attrs)
for key, value in list(out.items()):
if len(str(key)) == 6 and isinstance(value, (list, tuple)):
normalized = _s57_list_string(value)
if normalized is not None:
out[key] = normalized
if "COLOUR" in out:
value = _s57_list_string(out.get("COLOUR"))
if value is not None:
out["COLOUR"] = value
elif "COLMAR" in out:
try:
code = int(out["COLMAR"])
except (TypeError, ValueError):
code = -1
translated = CM93_COLMAR_TO_S57_COLOUR.get(code)
if translated is not None:
out["COLOUR"] = translated
for key in ("QUASOU", "CATLIT"):
value = out.get(key)
if isinstance(value, int):
out[key] = str(int(value))
return out
def split_lookup_condition(condition: str) -> tuple[str, str]:
text = str(condition)
if len(text) < 6:
raise ValueError(f"invalid S-52 lookup condition: {condition!r}")
return text[:6].upper(), text[6:]
def _c_atoi(value: str) -> int:
"""Minimal C atoi semantics used by fixed OpenCPN FindBestLUP."""
match = re.match(r"^[\s]*([+-]?\d+)", str(value))
return int(match.group(1)) if match else 0
def _c_atof_float32(value: str) -> float:
"""OpenCPN assigns atof() to float before exact REAL comparison."""
try:
parsed = float(str(value).strip() or "0")
except ValueError:
parsed = 0.0
return struct.unpack("f", struct.pack("f", parsed))[0]
def lookup_condition_matches(condition: str, attrs: dict[str, Any]) -> bool:
"""Match one condition as fixed OpenCPN FindBestLUP does."""
acronym, expected = split_lookup_condition(condition)
actual_present = acronym in attrs and attrs.get(acronym) is not None
if expected.startswith(" "):
return actual_present
if expected.startswith("?"):
# Preserve the fixed-source quirk: this path never increments countATT.
return False
if not actual_present:
return False
actual = attrs.get(acronym)
if isinstance(actual, bool):
actual = int(actual)
if isinstance(actual, int):
return actual == _c_atoi(expected)
if isinstance(actual, float):
return actual == _c_atof_float32(expected)
return str(actual) == expected
def _lookup_sequence(row: dict[str, Any]) -> int:
try:
return int(row.get("id") or 0)
except (TypeError, ValueError):
return 0
def sort_lookup_candidates(rows: Iterable[dict[str, Any]]) -> list[dict[str, Any]]:
"""Match fixed OpenCPN CompareLUPObjects."""
return sorted(
rows,
key=lambda row: (
-len(row.get("attribute_conditions_raw") or []),
_lookup_sequence(row),
),
)
def _has_lookup_visible_attributes(attrs: dict[str, Any]) -> bool:
return any(len(str(key)) == 6 and value is not None for key, value in attrs.items())
def find_best_lookup(
rows: Iterable[dict[str, Any]],
attrs: dict[str, Any],
*,
table_name: str | None = None,
geometry_type: str | None = None,
) -> dict[str, Any] | None:
"""Reproduce non-strict fixed OpenCPN FindBestLUP selection."""
normalized = normalize_cm93_attributes_for_s52(attrs)
candidates = []
for row in rows:
if table_name and str(row.get("table_name")) != table_name:
continue
if geometry_type and str(row.get("geometry_type")) != geometry_type:
continue
candidates.append(row)
ordered = sort_lookup_candidates(candidates)
if not ordered:
return None
first = ordered[0]
if not _has_lookup_visible_attributes(normalized):
return first
for row in ordered:
conditions = row.get("attribute_conditions_raw") or []
if not conditions:
continue
count_att = sum(
1 for condition in conditions
if lookup_condition_matches(condition, normalized)
)
if count_att == len(conditions):
return row
for row in ordered:
if not (row.get("attribute_conditions_raw") or []):
return row
return first
def primary_symbol_code(row: dict[str, Any] | None) -> str | None:
if not row:
return None
symbols = row.get("sy") or []
if not symbols:
return None
return str(symbols[0])
def conditional_procedures(row: dict[str, Any] | None) -> tuple[str, ...]:
if not row:
return ()
return tuple(str(x) for x in (row.get("cs") or []))
def _s52_int_list(value: Any) -> list[int]:
if value is None or value == "":
return []
if isinstance(value, (list, tuple)):
raw = value
elif isinstance(value, str):
text = value.strip().strip("[]")
raw = [part.strip() for part in text.split(",") if part.strip()]
else:
raw = [value]
out: list[int] = []
for item in raw:
try:
out.append(int(item))
except (TypeError, ValueError):
continue
return out
def _s52_float(value: Any, default: float | None = None) -> float | None:
try:
return float(value)
except (TypeError, ValueError):
return default
def _lights05_color(colours: list[int]) -> str:
if len(colours) == 1:
if 3 in colours:
return "red"
if 4 in colours:
return "green"
if any(code in colours for code in (1, 6, 11)):
return "yellow_white"
return "magenta"
if len(colours) == 2:
if 1 in colours and 3 in colours:
return "red"
if 1 in colours and 4 in colours:
return "green"
return "magenta"
def _lights05_flare_symbol(colours: list[int]) -> str:
color = _lights05_color(colours)
if color == "red":
return "LIGHTS11"
if color == "green":
return "LIGHTS12"
if color == "yellow_white":
return "LIGHTS13"
return "LITDEF11"
def resolve_lights05(attrs: dict[str, Any]) -> dict[str, Any]:
"""Resolve OpenCPN PLIB LIGHTS05/LIGHTS06 into runtime-friendly fields."""
catlit = _s52_int_list(attrs.get("CATLIT"))
if 8 in catlit or 11 in catlit:
return {"s52_csp_resolved": "LIGHTS05", "s52_light_mode": "special", "s52_light_symbol_code": "LIGHTS82"}
if 9 in catlit:
return {"s52_csp_resolved": "LIGHTS05", "s52_light_mode": "special", "s52_light_symbol_code": "LIGHTS81"}
colours = _s52_int_list(attrs.get("COLOUR")) or [12]
valnmr = _s52_float(attrs.get("VALNMR"), 9.0) or 9.0
sectr1 = _s52_float(attrs.get("SECTR1"))
sectr2 = _s52_float(attrs.get("SECTR2"))
arc_color = _lights05_color(colours)
litvis = _s52_int_list(attrs.get("LITVIS"))
obscured = any(code in litvis for code in (3, 7, 8))
if obscured:
arc_color = "obscured"
base: dict[str, Any] = {
"s52_csp_resolved": "LIGHTS05",
"s52_light_arc_color": arc_color,
"s52_light_nominal_range_nm": float(valnmr),
}
if sectr1 is None or sectr2 is None:
if valnmr < 10.0:
base.update({"s52_light_mode": "flare", "s52_light_symbol_code": _lights05_flare_symbol(colours), "s52_light_rotation_deg": 135.0})
return base
radius = 3.0 if valnmr < 7.0 else 10.0 if valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0
base.update({"s52_light_mode": "all_round", "s52_light_arc_radius_mm": radius})
return base
sweep = (sectr2 - sectr1) % 360.0
if sweep < 1.0:
base.update({"s52_light_mode": "all_round", "s52_light_arc_radius_mm": 10.0 if valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0})
return base
arc_radius = 10.0 if 0 < valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0
base.update({
"s52_light_mode": "sector",
"s52_sector_start_deg": float(sectr1),
"s52_sector_end_deg": float(sectr2),
"s52_sector_sweep_deg": float(sweep),
"s52_sector_display_start_deg": float((sectr1 + 180.0) % 360.0),
"s52_sector_display_end_deg": float((sectr2 + 180.0) % 360.0),
"s52_light_arc_radius_mm": float(arc_radius),
"s52_light_sector_radius_mm": 25.0,
"s52_light_obscured": bool(obscured),
})
return base
class S52LookupRegistry:
"""Read extracted S-52 lookups once and expose PBF-ready properties."""
def __init__(
self,
path,
point_table: str = "Paper",
boundary_table: str = "Plain",
mariner_mode: str = "runtime",
):
import json
from pathlib import Path
self.path = Path(path)
self.point_table = str(point_table)
self.boundary_table = str(boundary_table)
self.mariner_mode = str(mariner_mode)
if self.mariner_mode not in {"runtime", "reference"}:
raise ValueError("mariner_mode must be runtime or reference")
self.settings = S52MarinerSettings(symbolized_boundaries=self.boundary_table == "Symbolized")
rows = json.loads(self.path.read_text(encoding="utf-8"))
self.by_class: dict[str, list[dict[str, Any]]] = {}
for row in rows:
row = dict(row)
row["_static_hints"] = static_rule_hints(
str(row.get("instruction") or "")
)
object_class = str(row.get("object_class") or "")
self.by_class.setdefault(object_class, []).append(row)
def resolve(
self,
object_class: str,
geometry_type: str,
attrs: dict[str, Any],
) -> dict[str, Any] | None:
geometry = str(geometry_type).upper()
table_geometry = {
"POINT": (self.point_table, "Point"),
"SOUNDING": (self.point_table, "Point"),
"LINE": ("Lines", "Line"),
"AREA": (self.boundary_table, "Area"),
}.get(geometry)
if table_geometry is None:
return None
table_name, lookup_geometry = table_geometry
lookup_class, lookup_attrs = normalize_cm93_object_for_s52(
object_class, geometry_type, attrs
)
return find_best_lookup(
self.by_class.get(lookup_class, []),
lookup_attrs,
table_name=table_name,
geometry_type=lookup_geometry,
)
def _plain_area_instruction(self, object_class: str) -> str | None:
row = find_best_lookup(
self.by_class.get(object_class, []),
{},
table_name="Plain",
geometry_type="Area",
)
if row is None:
return None
instruction = str(row.get("instruction") or "").strip()
return instruction or None
def properties(
self,
object_class: str,
geometry_type: str,
attrs: dict[str, Any],
*,
next_safe_contour_m: float | None = None,
sounding_depth_m: float | None = None,
associated_depth_ranges: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
lookup_class, normalized_attrs = normalize_cm93_object_for_s52(
object_class, geometry_type, attrs
)
row = self.resolve(object_class, geometry_type, attrs)
if row is None:
return {}
out: dict[str, Any] = {
"s52_lookup_rcid": str(row.get("rcid") or ""),
"s52_display_category": str(row.get("display_category") or ""),
"s52_display_priority": str(row.get("display_priority") or ""),
"s52_table_name": str(row.get("table_name") or ""),
}
instruction = str(row.get("instruction") or "")
out.update(dict(row.get("_static_hints") or {}))
out.update(materialize_text_hints(instruction, normalized_attrs))
out.update(materialize_symbol_hints(instruction, normalized_attrs))
if lookup_class != str(object_class):
out["s52_object_class"] = lookup_class
procedures = [
token.strip()
for token in str(out.get("s52_csp") or "").split(",")
if token.strip()
]
safe_dashed_instruction = self._plain_area_instruction("SAFECD")
safe_solid_instruction = self._plain_area_instruction("SAFECN")
unresolved: list[str] = []
resolved_by_procedure: dict[str, dict[str, Any]] = {}
for procedure in procedures:
if self.mariner_mode == "runtime" and is_mariner_dependent_csp(procedure):
runtime_facts = materialize_mariner_runtime_facts(
procedure,
lookup_class,
str(geometry_type),
normalized_attrs,
sounding_depth_m=sounding_depth_m,
associated_depth_ranges=associated_depth_ranges,
)
if procedure == "DEPCNT02":
if safe_dashed_instruction:
runtime_facts["s52_safe_dashed_rule_chain"] = materialize_rule_ir(
safe_dashed_instruction,
normalized_attrs,
source="lookup",
procedure="SAFECD",
)
if safe_solid_instruction:
runtime_facts["s52_safe_solid_rule_chain"] = materialize_rule_ir(
safe_solid_instruction,
normalized_attrs,
source="lookup",
procedure="SAFECN",
)
out.update(runtime_facts)
unresolved.append(procedure)
continue
resolved = resolve_csp_hints(
procedure,
lookup_class,
str(geometry_type),
normalized_attrs,
self.settings,
next_safe_contour_m=next_safe_contour_m,
sounding_depth_m=sounding_depth_m,
associated_depth_ranges=associated_depth_ranges,
safe_dashed_instruction=safe_dashed_instruction,
safe_solid_instruction=safe_solid_instruction,
)
if resolved:
out.update(resolved)
resolved_by_procedure[procedure] = resolved
else:
unresolved.append(procedure)
ordered_chain: list[dict[str, Any]] = []
for rule in parse_instruction_string(instruction):
if rule.opcode != "CS" or not rule.args:
ordered_chain.extend(
materialize_rule_ir(rule.raw, normalized_attrs, source="lookup")
)
continue
procedure = rule.args[0].strip().strip("'\"")[:8]
if procedure in unresolved:
dynamic = materialize_rule_ir(
rule.raw, normalized_attrs, source="lookup"
)[0]
dynamic["dynamic"] = True
dynamic["procedure"] = procedure
ordered_chain.append(dynamic)
else:
ordered_chain.extend(
list(resolved_by_procedure.get(procedure, {}).get("s52_rule_chain") or [])
)
out["s52_rule_chain_version"] = 1
out["s52_rule_chain"] = ordered_chain
out["s52_source_object_class"] = lookup_class
out["s52_source_geometry"] = str(geometry_type).upper()
if unresolved:
out["s52_csp"] = ",".join(unresolved)
else:
out.pop("s52_csp", None)
_apply_presentation_side_effects(out)
return {key: value for key, value in out.items() if value != ""}
def resolve_topmar01_symbol(attrs: dict[str, Any], *, floating: bool) -> str:
"""Resolve S-52 TOPMAR01 exactly as OpenCPN PLIB for known TOPSHP values."""
value = attrs.get("TOPSHP")
if value is None or value == "":
return "QUESMRK1"
try:
topshp = int(value)
except (TypeError, ValueError):
return "QUESMRK1"
if floating:
return TOPMAR01_FLOATING_SYMBOLS.get(topshp, "TMARDEF2")
return TOPMAR01_RIGID_SYMBOLS.get(topshp, "TMARDEF1")

View File

@@ -0,0 +1,174 @@
from __future__ import annotations
from collections import Counter, defaultdict
import json
from pathlib import Path
from typing import Any
import mapbox_vector_tile
SPRITE_PROPERTY_FIELDS = (
"chart_icon_image",
"chart_fill_pattern",
"chart_line_pattern",
)
def ev(expr: Any, props: dict[str, Any]):
if not isinstance(expr, list):
return expr
if not expr:
return None
op = expr[0]
if op == "get":
return props.get(expr[1])
if op == "has":
return expr[1] in props
if op == "!":
return not bool(ev(expr[1], props))
if op == "all":
return all(bool(ev(x, props)) for x in expr[1:])
if op == "any":
return any(bool(ev(x, props)) for x in expr[1:])
if op == "==":
return ev(expr[1], props) == ev(expr[2], props)
if op == "!=":
return ev(expr[1], props) != ev(expr[2], props)
if op == "slice":
value = ev(expr[1], props)
if value is None:
return None
start = int(ev(expr[2], props))
end = int(ev(expr[3], props)) if len(expr) > 3 else None
return str(value)[start:end]
if op == "to-number":
value = ev(expr[1], props)
try:
num = float(value)
return int(num) if num.is_integer() else num
except Exception:
return 0
if op == "literal":
return expr[1]
if op == "match":
value = ev(expr[1], props)
pairs = expr[2:-1]
default = expr[-1]
for i in range(0, len(pairs), 2):
label = pairs[i]
output = pairs[i + 1]
if isinstance(label, list) and label and label[0] == "literal":
labels = label[1]
if value in labels:
return ev(output, props)
elif isinstance(label, list):
if value in label:
return ev(output, props)
elif value == label:
return ev(output, props)
return ev(default, props)
raise RuntimeError(f"unsupported style filter op: {op}")
def audit_pbf_style_and_sprite(
pbf_root: Path,
style_path: Path,
sprite_json_path: Path | None = None,
) -> dict[str, Any]:
pbf_root = Path(pbf_root)
style = json.loads(Path(style_path).read_text(encoding="utf-8"))
sprite_keys = None
if sprite_json_path is not None:
sprite_obj = json.loads(Path(sprite_json_path).read_text(encoding="utf-8"))
sprite_keys = set(sprite_obj)
style_by_source_layer = defaultdict(list)
for layer in style.get("layers", []):
if layer.get("source") == "navsea_delivery" and layer.get("source-layer"):
style_by_source_layer[layer["source-layer"]].append(layer)
files = sorted(pbf_root.rglob("*.pbf"))
contract_unmatched = Counter()
zoom_hidden = Counter()
visible = Counter()
sprite_missing = Counter()
sprite_values = Counter()
total_features = 0
for path in files:
rel = path.relative_to(pbf_root)
if len(rel.parts) < 3:
raise RuntimeError(f"PBF path must end with z/x/y.pbf: {path}")
z = int(rel.parts[-3])
decoded = mapbox_vector_tile.decode(path.read_bytes())
for source_layer, layer in decoded.items():
style_layers = style_by_source_layer.get(source_layer, [])
for feature in layer.get("features", []):
total_features += 1
props = feature.get("properties") or {}
filter_matches = []
for sl in style_layers:
flt = sl.get("filter")
if flt is None or bool(ev(flt, props)):
filter_matches.append(sl)
key = (z, source_layer)
if not filter_matches:
contract_unmatched[key] += 1
else:
zoom_matches = []
for sl in filter_matches:
minz = sl.get("minzoom")
maxz = sl.get("maxzoom")
if minz is not None and z < minz:
continue
if maxz is not None and z >= maxz:
continue
zoom_matches.append(sl)
if zoom_matches:
visible[key] += 1
else:
zoom_hidden[key] += 1
if sprite_keys is not None:
for field in SPRITE_PROPERTY_FIELDS:
value = props.get(field)
if isinstance(value, str) and value:
sprite_values[(field, value)] += 1
if value not in sprite_keys:
sprite_missing[(field, value)] += 1
return {
"schema": "navsea.cm95.style-sprite-contract-audit/v1",
"pbf_file_count": len(files),
"feature_count": total_features,
"style_source_layer_count": len(style_by_source_layer),
"contract_unmatched_count": sum(contract_unmatched.values()),
"zoom_hidden_count": sum(zoom_hidden.values()),
"visible_count": sum(visible.values()),
"sprite_key_count": None if sprite_keys is None else len(sprite_keys),
"sprite_missing_count": sum(sprite_missing.values()),
"contract_unmatched": [
{"z": z, "source_layer": sl, "count": count}
for (z, sl), count in sorted(contract_unmatched.items())
],
"zoom_hidden": [
{"z": z, "source_layer": sl, "count": count}
for (z, sl), count in sorted(zoom_hidden.items())
],
"sprite_missing": [
{"field": field, "sprite_key": key, "count": count}
for (field, key), count in sorted(sprite_missing.items())
],
"sprite_values": [
{"field": field, "sprite_key": key, "count": count}
for (field, key), count in sorted(sprite_values.items())
],
"overall": (
"PASS"
if sum(contract_unmatched.values()) == 0 and sum(sprite_missing.values()) == 0
else "FAIL"
),
}

47
tools/compare_pbf_trees.py Executable file
View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse, hashlib, json
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
def digest(p: Path):
h=hashlib.sha256()
with p.open("rb") as f:
for b in iter(lambda:f.read(1024*1024),b""):
h.update(b)
return p.stat().st_size,h.hexdigest()
def inventory(root: Path):
return {str(p.relative_to(root)):p for p in root.rglob("*.pbf") if p.is_file()}
def main():
ap=argparse.ArgumentParser()
ap.add_argument("mysql_root")
ap.add_argument("postgres_root")
ap.add_argument("--output",default="logs/mysql_postgresql_pbf_compare.json")
ap.add_argument("--workers",type=int,default=8)
args=ap.parse_args()
mr,pr=Path(args.mysql_root),Path(args.postgres_root)
mi,pi=inventory(mr),inventory(pr)
common=sorted(set(mi)&set(pi))
def one(rel):
ms,mh=digest(mi[rel]); ps,ph=digest(pi[rel])
return rel,ms,mh,ps,ph
identical=0; diffs=[]
with ThreadPoolExecutor(max_workers=args.workers) as ex:
for rel,ms,mh,ps,ph in ex.map(one,common):
if mh==ph and ms==ps: identical+=1
else: diffs.append({"path":rel,"mysql_bytes":ms,"postgres_bytes":ps,"mysql_sha256":mh,"postgres_sha256":ph})
out={
"mysql_root":str(mr.resolve()),"postgres_root":str(pr.resolve()),
"mysql_files":len(mi),"postgres_files":len(pi),"common_files":len(common),
"identical_files":identical,"different_files":len(diffs),
"mysql_only":sorted(set(mi)-set(pi)),"postgres_only":sorted(set(pi)-set(mi)),
"different_examples":diffs[:100],
"result":"PASS" if len(mi)==len(pi)==identical and not diffs else "DIFF",
}
op=Path(args.output);op.parent.mkdir(parents=True,exist_ok=True)
op.write_text(json.dumps(out,ensure_ascii=False,indent=2),encoding="utf-8")
print(json.dumps({k:v for k,v in out.items() if k not in ("mysql_only","postgres_only","different_examples")},ensure_ascii=False,indent=2))
print(f"report={op}")
if __name__=="__main__": main()

File diff suppressed because one or more lines are too long

59
web/assets/maplibre-gl.js Normal file

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,61 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>CM95 MySQL vs PostgreSQL PBF 对比</title>
<link rel="stylesheet" href="/assets/maplibre-gl.css">
<style>
*{box-sizing:border-box}html,body{height:100%;margin:0;background:#111;color:#eee;font-family:system-ui,-apple-system,"Segoe UI",sans-serif}
body{overflow:hidden}#bar{height:52px;display:flex;align-items:center;gap:12px;padding:0 12px;background:#17191d;border-bottom:1px solid #3b3f46;font-size:12px}
#bar strong{font-size:14px;white-space:nowrap}#bar button{border:1px solid #555;background:#272b31;color:#fff;border-radius:5px;padding:5px 9px;cursor:pointer}
#bar label{display:flex;gap:5px;align-items:center;white-space:nowrap}.muted{color:#aeb7c3}#readout{margin-left:auto;font-variant-numeric:tabular-nums;white-space:nowrap}
#maps{height:calc(100% - 52px);display:grid;grid-template-columns:1fr 1fr;gap:2px;background:#555}
.pane{position:relative;min-width:0;overflow:hidden;background:#d8edf5}.map{position:absolute;inset:0}
.title{position:absolute;z-index:5;left:10px;top:10px;padding:7px 9px;border-radius:6px;background:rgba(18,20,24,.88);font-size:12px;line-height:1.4}
.title b{display:block;font-size:14px}.state{font-size:11px;color:#cbd2dc}.ok{color:#91e39b}.err{color:#ff8c8c}
.inspect{position:absolute;z-index:5;left:10px;right:10px;bottom:10px;max-height:28%;overflow:auto;padding:7px 9px;border-radius:6px;background:rgba(18,20,24,.9);font:11px/1.4 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap}
#stats{position:absolute;z-index:10;top:62px;left:50%;transform:translateX(-50%);max-width:72%;padding:7px 10px;border-radius:6px;background:rgba(18,20,24,.9);font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;pointer-events:none}
@media(max-width:820px){#maps{grid-template-columns:1fr;grid-template-rows:1fr 1fr}#bar .muted{display:none}#stats{max-width:94%}}
</style>
</head>
<body>
<div id="bar"><strong>CM95 PBF:MySQL vs PostgreSQL</strong><span class="muted">同源数据 / 同一 S-52 style / 同步视图</span>
<label><input id="sync" type="checkbox" checked>同步</label><button id="reset">九州</button><button id="info">统计</button><span id="readout">--</span></div>
<div id="stats" hidden></div>
<div id="maps">
<section class="pane"><div id="mysql-map" class="map"></div><div class="title"><b>左:MySQL</b><span id="mysql-state" class="state">加载中…</span></div><div id="mysql-inspect" class="inspect">点击任一侧,同点检查 Feature。</div></section>
<section class="pane"><div id="pg-map" class="map"></div><div class="title"><b>右:PostgreSQL / PostGIS</b><span id="pg-state" class="state">加载中…</span></div><div id="pg-inspect" class="inspect">点击任一侧,同点检查 Feature。</div></section>
</div>
<script src="/assets/maplibre-gl.js"></script>
<script>
const BOUNDS=[[128.0,29.5],[132.5,34.8]], START={center:[130.25,32.15],zoom:6.6,bearing:0,pitch:0};
const sync=document.getElementById("sync"), readout=document.getElementById("readout"), stats=document.getElementById("stats");
let styleTemplate=null,syncing=false,m1=null,m2=null;
function state(id,text,ok=true){const e=document.getElementById(id);e.textContent=text;e.className="state "+(ok?"ok":"err")}
function blank(){return {version:8,sources:{},layers:[{id:"bg",type:"background",paint:{"background-color":"#d8edf5"}}]}}
function styleFor(kind){const s=JSON.parse(JSON.stringify(styleTemplate));const src=s.sources&&s.sources.cm95;if(!src)throw new Error("style 缺少 sources.cm95");
src.type="vector";src.tiles=[location.origin+"/"+kind+"/{z}/{x}/{y}.pbf"];src.minzoom=5;src.maxzoom=12;src.bounds=[128,29.5,132.5,34.8];delete src.url;s.sprite=location.origin+"/sprite";return s}
const mysqlMap=new maplibregl.Map({container:"mysql-map",style:blank(),...START,attributionControl:false});
const pgMap=new maplibregl.Map({container:"pg-map",style:blank(),...START,attributionControl:false});
mysqlMap.addControl(new maplibregl.NavigationControl(),"top-right");pgMap.addControl(new maplibregl.NavigationControl(),"top-right");
function camera(m){const c=m.getCenter();return {center:[c.lng,c.lat],zoom:m.getZoom(),bearing:m.getBearing(),pitch:m.getPitch()}}
function link(a,b){if(syncing||!sync.checked)return;syncing=true;try{b.jumpTo(camera(a))}finally{syncing=false}}
mysqlMap.on("move",()=>{link(mysqlMap,pgMap);showReadout(mysqlMap)});pgMap.on("move",()=>{link(pgMap,mysqlMap);showReadout(pgMap)});
function showReadout(m){const c=m.getCenter();readout.textContent="z="+m.getZoom().toFixed(2)+" "+c.lng.toFixed(5)+", "+c.lat.toFixed(5)}
document.getElementById("reset").onclick=()=>{mysqlMap.fitBounds(BOUNDS,{padding:36,duration:0});link(mysqlMap,pgMap)};
document.getElementById("info").onclick=()=>{stats.hidden=!stats.hidden};
async function load(){try{const [sr,ir]=await Promise.all([fetch("/style.json",{cache:"no-store"}),fetch("/api/info",{cache:"no-store"})]);styleTemplate=await sr.json();const info=await ir.json();
stats.textContent="MySQL: "+info.mysql.release+" | "+info.mysql.files+" tiles\nPostgreSQL: "+info.postgresql.release+" | "+info.postgresql.files+" tiles\nCommon tile hash: "+info.compare.identical+"/"+info.compare.common+" identical";
mysqlMap.setStyle(styleFor("mysql"));pgMap.setStyle(styleFor("postgres"));}catch(e){state("mysql-state",String(e),false);state("pg-state",String(e),false)}}
mysqlMap.on("style.load",()=>state("mysql-state","S-52 / MySQL PBF",true));pgMap.on("style.load",()=>state("pg-state","S-52 / PostgreSQL PBF",true));
function key(f){const p=f.properties||{};return [f.sourceLayer||"",f.id??"",p.canonical_object_type||"",p.source_object_class||""].join("|")}
function summary(f){const p=f.properties||{};return [f.sourceLayer||"-","id="+(f.id??"-"),p.canonical_object_type||"",p.source_object_class||"",p.chart_label_text||"",p.contour_depth_m!==undefined?"depth="+p.contour_depth_m:""].filter(Boolean).join(" ")}
function inspect(map,ll,id){const el=document.getElementById(id);if(!map.loaded()){el.textContent="地图加载中";return}const p=map.project(ll),box=[[p.x-5,p.y-5],[p.x+5,p.y+5]],u=new Map();
for(const f of map.queryRenderedFeatures(box)){if(!u.has(key(f)))u.set(key(f),f)}const a=[...u.values()].slice(0,24);el.textContent=ll.lng.toFixed(6)+", "+ll.lat.toFixed(6)+"\n命中 "+u.size+" 项\n"+a.map((f,i)=>String(i+1).padStart(2,"0")+". "+summary(f)).join("\n")}
function compare(ll){if(m1)m1.remove();if(m2)m2.remove();m1=new maplibregl.Marker().setLngLat(ll).addTo(mysqlMap);m2=new maplibregl.Marker().setLngLat(ll).addTo(pgMap);inspect(mysqlMap,ll,"mysql-inspect");inspect(pgMap,ll,"pg-inspect")}
mysqlMap.on("click",e=>compare(e.lngLat));pgMap.on("click",e=>compare(e.lngLat));
mysqlMap.on("error",e=>{const m=e?.error?.message||"";if(!m.includes("404"))state("mysql-state",m,false)});pgMap.on("error",e=>{const m=e?.error?.message||"";if(!m.includes("404"))state("pg-state",m,false)});
window.addEventListener("resize",()=>{mysqlMap.resize();pgMap.resize()});showReadout(mysqlMap);load();
</script>
</body></html>

View File

@@ -0,0 +1,105 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import hashlib
import json
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse
def parse_args():
p=argparse.ArgumentParser(description="CM95 MySQL vs PostgreSQL PBF comparison server")
p.add_argument("--host",default="0.0.0.0")
p.add_argument("--port",type=int,default=18889)
p.add_argument("--mysql-pbf-root",required=True)
p.add_argument("--postgres-pbf-root",required=True)
p.add_argument("--mysql-release",default="mysql-current")
p.add_argument("--postgres-release",default="postgres-current")
return p.parse_args()
def sha256(path: Path) -> str:
h=hashlib.sha256()
with path.open("rb") as f:
for b in iter(lambda:f.read(1024*1024),b""):
h.update(b)
return h.hexdigest()
def pbf_inventory(root: Path):
return {str(p.relative_to(root)):p for p in root.rglob("*.pbf") if p.is_file()}
def main():
args=parse_args()
web=Path(__file__).resolve().parent
mysql_root=Path(args.mysql_pbf_root).resolve()
pg_root=Path(args.postgres_pbf_root).resolve()
html=web/"mysql_postgresql_compare.html"
style=web/"resources/style.s52.json"
assets=web/"assets"
resources=web/"resources"
for p in [mysql_root,pg_root,html,style,assets/"maplibre-gl.js",assets/"maplibre-gl.css",resources/"sprite.json",resources/"sprite.png"]:
if not p.exists():
raise SystemExit(f"missing: {p}")
mi=pbf_inventory(mysql_root)
pi=pbf_inventory(pg_root)
common=sorted(set(mi)&set(pi))
identical=0
different=[]
for rel in common:
if mi[rel].stat().st_size==pi[rel].stat().st_size and sha256(mi[rel])==sha256(pi[rel]):
identical+=1
elif len(different)<50:
different.append(rel)
info={
"mysql":{"release":args.mysql_release,"files":len(mi),"root":str(mysql_root)},
"postgresql":{"release":args.postgres_release,"files":len(pi),"root":str(pg_root)},
"compare":{"common":len(common),"identical":identical,"different":len(common)-identical,"different_examples":different},
}
class Handler(BaseHTTPRequestHandler):
def send(self,data:bytes,ctype:str,status=200):
self.send_response(status)
self.send_header("Content-Type",ctype)
self.send_header("Content-Length",str(len(data)))
self.send_header("Cache-Control","no-cache")
self.send_header("Access-Control-Allow-Origin","*")
self.end_headers()
self.wfile.write(data)
def serve_file(self,path:Path,ctype:str):
if not path.is_file():
self.send_error(404)
return
self.send(path.read_bytes(),ctype)
def do_GET(self):
path=urlparse(self.path).path
if path in ("/","/index.html","/compare"):
return self.serve_file(html,"text/html; charset=utf-8")
if path=="/api/info":
return self.send(json.dumps(info,ensure_ascii=False,indent=2).encode(),"application/json; charset=utf-8")
if path=="/style.json":
return self.serve_file(style,"application/json; charset=utf-8")
if path in ("/assets/maplibre-gl.js","/assets/maplibre-gl.css"):
fn=path.rsplit("/",1)[-1]
ct="application/javascript; charset=utf-8" if fn.endswith(".js") else "text/css; charset=utf-8"
return self.serve_file(assets/fn,ct)
if path in ("/sprite.json","/sprite.png","/sprite@2x.json","/sprite@2x.png"):
fn=path[1:]
ct="application/json; charset=utf-8" if fn.endswith(".json") else "image/png"
return self.serve_file(resources/fn,ct)
for prefix,root in (("/mysql/",mysql_root),("/postgres/",pg_root)):
if path.startswith(prefix) and path.endswith(".pbf"):
rel=path[len(prefix):]
parts=rel.split("/")
if len(parts)!=3 or not all(x.isdigit() for x in [parts[0],parts[1],parts[2][:-4]]):
self.send_error(404); return
return self.serve_file(root/rel,"application/vnd.mapbox-vector-tile")
self.send_error(404)
def log_message(self,fmt,*args):
pass
print(json.dumps(info,ensure_ascii=False,indent=2))
print(f"Compare: http://{args.host}:{args.port}/")
ThreadingHTTPServer((args.host,args.port),Handler).serve_forever()
if __name__=="__main__":
main()

7338
web/resources/sprite.json Normal file

File diff suppressed because it is too large Load Diff

BIN
web/resources/sprite.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 394 KiB

7338
web/resources/sprite@2x.json Normal file

File diff suppressed because it is too large Load Diff

BIN
web/resources/sprite@2x.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 562 KiB

42381
web/resources/style.s52.json Normal file

File diff suppressed because it is too large Load Diff