feat: save PostgreSQL PostGIS CM95 baseline
This commit is contained in:
10
.env.example
Normal file
10
.env.example
Normal file
@@ -0,0 +1,10 @@
|
||||
CM95_SOURCE_DIR=/root/sourceserver/cm95-data/CM93JAPAN
|
||||
|
||||
CM95_PG_HOST=192.168.x.x
|
||||
CM95_PG_PORT=5432
|
||||
CM95_PG_DATABASE=cm95_chart
|
||||
CM95_PG_USER=cm95
|
||||
CM95_PG_PASSWORD=change-me
|
||||
CM95_PG_SSLMODE=prefer
|
||||
CM95_PG_ADMIN_DATABASE=postgres
|
||||
CM95_PG_CONNECT_TIMEOUT=20
|
||||
9
.gitignore
vendored
Normal file
9
.gitignore
vendored
Normal file
@@ -0,0 +1,9 @@
|
||||
.env
|
||||
.venv/
|
||||
build/
|
||||
logs/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
.DS_Store
|
||||
11
AGENTS.md
Normal file
11
AGENTS.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# CM95 PostgreSQL/PostGIS 项目约束
|
||||
|
||||
1. 本项目为独立 PostgreSQL/PostGIS 实现,不读取、不修改、不迁移旧 MySQL 数据库。
|
||||
2. /root/sourceserver/cm95 仅为只读参考源,不得修改其文件或 Git 状态。
|
||||
3. 输入以 CM95 Japan / CM93JAPAN 原始数据为唯一事实源。
|
||||
4. parser、geometry normalization、raw preservation、Cell 事务、SHA-256 skip/reimport 语义保持参考工程冻结行为。
|
||||
5. Geometry 使用 PostGIS SRID 4326;空间索引使用 GiST。
|
||||
6. PBF mapping、确定性 FID、tile policy、S-52 保持参考工程行为,数据库方言差异在访问层吸收。
|
||||
7. 运行配置只使用 CM95_PG_* 与 CM95_SOURCE_DIR,不回退旧 MySQL 环境变量。
|
||||
8. 全国全量导入前必须先通过单 Cell、小样本、DB audit 和单 tile PBF 验证。
|
||||
9. 文档与日志不得记录数据库密码。
|
||||
40
README.md
Normal file
40
README.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# CM95 PostgreSQL/PostGIS Pipeline
|
||||
|
||||
独立的数据链:
|
||||
|
||||
CM95/CM93 Japan 原始数据
|
||||
→ 冻结解析与几何标准化
|
||||
→ PostgreSQL + PostGIS 母数据库
|
||||
→ 确定性 FID / mapping / S-52
|
||||
→ MVT/PBF
|
||||
|
||||
本工程不依赖 MySQL,也不会读取或修改旧 MySQL 数据库。
|
||||
旧工程 /root/sourceserver/cm95 仅作为解析、几何与 PBF 行为参考。
|
||||
|
||||
## 环境
|
||||
|
||||
- Python 3.12
|
||||
- PostgreSQL 14+
|
||||
- PostGIS
|
||||
- psycopg 3
|
||||
|
||||
## 基本流程
|
||||
|
||||
1. python3.12 -m venv .venv
|
||||
2. 激活 .venv
|
||||
3. pip install -r requirements.txt
|
||||
4. cp .env.example .env
|
||||
5. 编辑 CM95_PG_* 指向 NAS PostgreSQL
|
||||
6. cm95pg env-check
|
||||
7. cm95pg db-init --create-database
|
||||
8. cm95pg db-check
|
||||
9. cm95pg scan
|
||||
10. cm95pg import
|
||||
11. cm95pg audit --precision-samples 128
|
||||
|
||||
NAS 已预先建立数据库时,第 7 步不需要 --create-database。
|
||||
|
||||
## PBF
|
||||
|
||||
导入完成后先建立 FID manifest,再创建 production plan 并生成 PBF。
|
||||
mapping、tile policy、S-52、sprite/style 继续沿用参考工程已经验证的配置。
|
||||
1502
config/mappings/cm95_gap_disposition_japan_dataset_1_v1.yaml
Normal file
1502
config/mappings/cm95_gap_disposition_japan_dataset_1_v1.yaml
Normal file
File diff suppressed because it is too large
Load Diff
1284
config/mappings/cm95_gap_disposition_v1.yaml
Normal file
1284
config/mappings/cm95_gap_disposition_v1.yaml
Normal file
File diff suppressed because it is too large
Load Diff
1502
config/mappings/cm95_to_navsea_v1.yaml
Normal file
1502
config/mappings/cm95_to_navsea_v1.yaml
Normal file
File diff suppressed because it is too large
Load Diff
18
config/pbf/cm95_production_v1.yaml
Normal file
18
config/pbf/cm95_production_v1.yaml
Normal file
@@ -0,0 +1,18 @@
|
||||
schema: navsea.cm95.production-policy/v1
|
||||
version: v1.4.2-materialization-split
|
||||
status: PRODUCTION_CANDIDATE
|
||||
zooms: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
|
||||
profile: delivery
|
||||
workers: 4
|
||||
max_attempts: 3
|
||||
release_root: build/releases
|
||||
plan_path: build/production/cm95_production_plan_v1.json
|
||||
rules:
|
||||
- 生产目标固定为 z0-z12。
|
||||
- 高 zoom tile 的物理枚举只由 enumeration_scales 决定。
|
||||
- scale_ladders 的粗比例尺只允许在已枚举 tile 内回退,不得扩张生产 footprint。
|
||||
- mapping、tile policy、Dataset fingerprint、FID manifest 在计划创建后必须保持不变。
|
||||
- PASS tile 自动断点续跑;RUNNING 状态在下次启动时恢复为 PENDING。
|
||||
- FAILED tile 最多尝试 max_attempts 次;失败不会被静默跳过。
|
||||
- 空 tile 可为 PASS 且不生成 PBF 文件;release manifest 只列真实产物。
|
||||
- 发布前 finalize-production 必须满足 planned/report/decoded feature 三方一致。
|
||||
96
config/pbf/cm95_tile_policy_japan_full_detail_v1.yaml
Normal file
96
config/pbf/cm95_tile_policy_japan_full_detail_v1.yaml
Normal file
@@ -0,0 +1,96 @@
|
||||
schema: navsea.cm95.tile-policy/v1
|
||||
version: v1.3-japan-full-detail-g7500-at-z12
|
||||
status: PRODUCTION_CANDIDATE
|
||||
xyz: true
|
||||
nominal_tile_pixels: 256
|
||||
buffer_tile_pixels: 8
|
||||
default_extent: 4096
|
||||
extent_by_zoom:
|
||||
'10': 4096
|
||||
'11': 4096
|
||||
'12': 1048576
|
||||
enumeration_scales:
|
||||
'0':
|
||||
- 20000000
|
||||
'1':
|
||||
- 20000000
|
||||
'2':
|
||||
- 20000000
|
||||
'3':
|
||||
- 20000000
|
||||
'4':
|
||||
- 20000000
|
||||
'5':
|
||||
- 3000000
|
||||
'6':
|
||||
- 3000000
|
||||
'7':
|
||||
- 3000000
|
||||
'8':
|
||||
- 1000000
|
||||
'9':
|
||||
- 1000000
|
||||
'10':
|
||||
- 200000
|
||||
'11':
|
||||
- 50000
|
||||
- 100000
|
||||
'12':
|
||||
- 7500
|
||||
- 20000
|
||||
- 50000
|
||||
scale_ladders:
|
||||
'0':
|
||||
- 20000000
|
||||
'1':
|
||||
- 20000000
|
||||
'2':
|
||||
- 20000000
|
||||
'3':
|
||||
- 20000000
|
||||
'4':
|
||||
- 20000000
|
||||
'5':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'6':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'7':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'8':
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'9':
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'10':
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'11':
|
||||
- 50000
|
||||
- 100000
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'12':
|
||||
- 7500
|
||||
- 20000
|
||||
- 50000
|
||||
- 100000
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
notes:
|
||||
- Japan full-detail production 仍保持 z0-z12;不新增 z13/z14。
|
||||
- z12 enumeration 加入 1:7,500(G),避免正确 Japan 数据中的 G 级细节被生产策略直接忽略。
|
||||
- z12 scale ladder 优先 7,500,再回退 20k/50k/100k/...。
|
||||
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea/MapLibre 契约。
|
||||
- enumeration_scales 仅决定物化范围;scale_ladders 仅决定 tile 内选择/回退。
|
||||
102
config/pbf/cm95_tile_policy_japan_national_v1.yaml
Normal file
102
config/pbf/cm95_tile_policy_japan_national_v1.yaml
Normal file
@@ -0,0 +1,102 @@
|
||||
schema: navsea.cm95.tile-policy/v1
|
||||
version: v1.4-japan-national-lowzoom-fallback
|
||||
status: PRODUCTION_CANDIDATE
|
||||
xyz: true
|
||||
nominal_tile_pixels: 256
|
||||
buffer_tile_pixels: 8
|
||||
default_extent: 4096
|
||||
extent_by_zoom:
|
||||
'10': 4096
|
||||
'11': 4096
|
||||
'12': 1048576
|
||||
enumeration_scales:
|
||||
'0':
|
||||
- 20000000
|
||||
'1':
|
||||
- 20000000
|
||||
'2':
|
||||
- 20000000
|
||||
'3':
|
||||
- 20000000
|
||||
'4':
|
||||
- 20000000
|
||||
'5':
|
||||
- 3000000
|
||||
'6':
|
||||
- 3000000
|
||||
'7':
|
||||
- 3000000
|
||||
'8':
|
||||
- 1000000
|
||||
'9':
|
||||
- 1000000
|
||||
'10':
|
||||
- 200000
|
||||
'11':
|
||||
- 50000
|
||||
- 100000
|
||||
'12':
|
||||
- 7500
|
||||
- 20000
|
||||
- 50000
|
||||
scale_ladders:
|
||||
'0':
|
||||
- 20000000
|
||||
- 3000000
|
||||
'1':
|
||||
- 20000000
|
||||
- 3000000
|
||||
'2':
|
||||
- 20000000
|
||||
- 3000000
|
||||
'3':
|
||||
- 20000000
|
||||
- 3000000
|
||||
'4':
|
||||
- 20000000
|
||||
- 3000000
|
||||
'5':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'6':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'7':
|
||||
- 3000000
|
||||
- 20000000
|
||||
'8':
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'9':
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'10':
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'11':
|
||||
- 50000
|
||||
- 100000
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
'12':
|
||||
- 7500
|
||||
- 20000
|
||||
- 50000
|
||||
- 100000
|
||||
- 200000
|
||||
- 1000000
|
||||
- 3000000
|
||||
- 20000000
|
||||
notes:
|
||||
- Japan nationwide aoi-complete: z0-z4 在 1:20M 不覆盖日本南部时允许回退 1:3M;不改变 z5-z12 细节策略。
|
||||
- Japan full-detail production 仍保持 z0-z12;不新增 z13/z14。
|
||||
- z12 enumeration 加入 1:7,500(G),避免正确 Japan 数据中的 G 级细节被生产策略直接忽略。
|
||||
- z12 scale ladder 优先 7,500,再回退 20k/50k/100k/...。
|
||||
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea/MapLibre 契约。
|
||||
- enumeration_scales 仅决定物化范围;scale_ladders 仅决定 tile 内选择/回退。
|
||||
51
config/pbf/cm95_tile_policy_v1.yaml
Normal file
51
config/pbf/cm95_tile_policy_v1.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
schema: navsea.cm95.tile-policy/v1
|
||||
version: v1.2-production-materialization-split
|
||||
status: PRODUCTION_CANDIDATE
|
||||
xyz: true
|
||||
nominal_tile_pixels: 256
|
||||
buffer_tile_pixels: 8
|
||||
default_extent: 4096
|
||||
extent_by_zoom:
|
||||
'10': 4096
|
||||
'11': 4096
|
||||
'12': 1048576
|
||||
|
||||
# enumeration_scales 决定哪些 coverage 可以创建物理 tile。
|
||||
# 这里只使用与目标 zoom 匹配的原生/近原生比例尺,避免粗比例尺 coverage
|
||||
# 把高 zoom 生产范围膨胀成百万级静态文件。
|
||||
enumeration_scales:
|
||||
'0': [20000000]
|
||||
'1': [20000000]
|
||||
'2': [20000000]
|
||||
'3': [20000000]
|
||||
'4': [20000000]
|
||||
'5': [3000000]
|
||||
'6': [3000000]
|
||||
'7': [3000000]
|
||||
'8': [1000000]
|
||||
'9': [1000000]
|
||||
'10': [200000]
|
||||
'11': [50000, 100000]
|
||||
'12': [20000, 50000]
|
||||
|
||||
# scale_ladders 只控制已经决定生成的 tile 内部如何选择数据。
|
||||
# 粗比例尺仍可作为 tile 内回退,但不再扩张 tile 枚举范围。
|
||||
scale_ladders:
|
||||
'0': [20000000]
|
||||
'1': [20000000]
|
||||
'2': [20000000]
|
||||
'3': [20000000]
|
||||
'4': [20000000]
|
||||
'5': [3000000, 20000000]
|
||||
'6': [3000000, 20000000]
|
||||
'7': [3000000, 20000000]
|
||||
'8': [1000000, 3000000, 20000000]
|
||||
'9': [1000000, 3000000, 20000000]
|
||||
'10': [200000, 1000000, 3000000, 20000000]
|
||||
'11': [50000, 100000, 200000, 1000000, 3000000, 20000000]
|
||||
'12': [20000, 50000, 100000, 200000, 1000000, 3000000, 20000000]
|
||||
notes:
|
||||
- 生产目标为 z0-z12;Pilot 的 z10-z12 只用于验证。
|
||||
- enumeration_scales 与 scale_ladders 必须分离。
|
||||
- enumeration_scales 只决定是否物化 tile;scale_ladders 只决定 tile 内数据回退。
|
||||
- z0-z11 extent=4096;z12 extent=1048576,保持已验证 NavSea 契约。
|
||||
18
config/pbf/cm95_v1_4_5_nonrenderable_label_tiles.txt
Normal file
18
config/pbf/cm95_v1_4_5_nonrenderable_label_tiles.txt
Normal file
@@ -0,0 +1,18 @@
|
||||
# CM95 PBF v1.4.5 定向重建 tile
|
||||
# 来源:cm95_style_contract_unmatched_features_v1.json
|
||||
10/912/366
|
||||
10/913/366
|
||||
10/916/327
|
||||
11/1893/717
|
||||
11/1893/718
|
||||
11/1908/643
|
||||
11/1908/645
|
||||
11/1909/636
|
||||
11/1909/639
|
||||
11/1909/643
|
||||
11/1910/633
|
||||
12/3662/1409
|
||||
12/3737/1466
|
||||
12/3787/1435
|
||||
12/3787/1436
|
||||
12/3825/1376
|
||||
22
config/s52/cm95_s52_poc_v1.json
Normal file
22
config/s52/cm95_s52_poc_v1.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"schema": "navsea.cm95.s52-poc-config/v1",
|
||||
"targets": [
|
||||
"LIGHTS",
|
||||
"LNDMRK",
|
||||
"BCNLAT",
|
||||
"BOYLAT",
|
||||
"TOPMAR"
|
||||
],
|
||||
"palette": "DAY_BRIGHT",
|
||||
"source_id": "navsea_delivery",
|
||||
"source_layer": "navigation_marks",
|
||||
"sprite_url": "./sprite",
|
||||
"tiles_template": "https://REPLACE-ME/{z}/{x}/{y}.pbf",
|
||||
"policy": {
|
||||
"static_lookup": "materialize_in_builder",
|
||||
"conditional_symbology": "explicit_builder_derived_or_runtime_renderer",
|
||||
"production_release": "do_not_touch_in_poc"
|
||||
},
|
||||
"point_table": "Paper",
|
||||
"sprite_scope": "all"
|
||||
}
|
||||
113
config/s52/cm95_s52_runtime_capabilities_web_compare_v1.json
Normal file
113
config/s52/cm95_s52_runtime_capabilities_web_compare_v1.json
Normal file
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"schema": "navsea.cm95.s52-runtime-capability-registry/v1",
|
||||
"target": "kyushu_compare_web_debug",
|
||||
"description": "Kyushu side-by-side browser target. IMPLEMENTED means the declared runtime kind has concrete target-side code evidence; PREVIEW_ONLY and NOT_IMPLEMENTED are non-compliant for strict portrayal parity.",
|
||||
"capabilities": {
|
||||
"RUNTIME_CA": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "S52_SCREEN_MM_SCAMIN_SCALE_AND_RULE_WIDTH",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/kyushu_compare.html",
|
||||
"contains": [
|
||||
"left-ca-overlay",
|
||||
"drawLightArcFeature"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"caScaleMetrics",
|
||||
"requestedArcWidthMm",
|
||||
"metresPerPixel"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_DEPARE": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "ORDERED_RUNTIME_RULE_IR",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"RuntimeExecutor",
|
||||
"dynamicRules"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_DEPCNT": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "SAFE_LOOKUP_QUAPOS_AND_SIDE_EFFECTS",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"RuntimeExecutor",
|
||||
"s52_safe_dashed_rule_chain",
|
||||
"scamin:100000001"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_SOUNDG": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "ORDERED_GLYPH_COMPOSITION_WITH_RESOURCE_PIVOTS",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"soundingRules"
|
||||
,"runtime_icon_offset"
|
||||
,"s52_sounding_depth_m"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_OBSTRN": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "ORDERED_POINT_LINE_AREA_RULE_CHAIN",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"hazardRules",
|
||||
"operationFeatures",
|
||||
"OBSTRN04",
|
||||
"ISODGR51"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_WRECKS": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "ORDERED_POINT_LINE_AREA_RULE_CHAIN",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"hazardRules",
|
||||
"operationFeatures",
|
||||
"WRECKS02"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"RUNTIME_LIGHTS_STATE": {
|
||||
"status": "IMPLEMENTED",
|
||||
"fidelity": "PREVIOUS_POSITION_AND_DESCRIPTION",
|
||||
"evidence": [
|
||||
{
|
||||
"path": "web/s52_runtime_executor.js",
|
||||
"contains": [
|
||||
"LightLabelState",
|
||||
"lastPosition",
|
||||
"lastDescription",
|
||||
"yoff:1"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
63
config/s52/cm95_s52_style_compiler_v1.json
Normal file
63
config/s52/cm95_s52_style_compiler_v1.json
Normal file
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"schema": "navsea.cm95.s52-style-compiler-config/v1",
|
||||
"name": "CM95 S-52 Fukuoka compiled style v1",
|
||||
"palette": "DAY_BRIGHT",
|
||||
"sprite": "http://192.168.200.184:8877/s52-sprite-lc-ap-v3",
|
||||
"background_color": "DEPDW",
|
||||
"fallback_color": "NODTA",
|
||||
"area_fill_opacity": 0.92,
|
||||
"area_pattern_opacity": 0.85,
|
||||
"line_width_fallback": 1.0,
|
||||
"complex_line_width_px": 5.0,
|
||||
"display_categories": ["Displaybase", "Standard", "Other"],
|
||||
"source": {
|
||||
"id": "cm95",
|
||||
"type": "vector",
|
||||
"tiles": ["http://192.168.200.184:8877/tiles-presentation-v10/{z}/{x}/{y}.pbf"],
|
||||
"bounds": [130.341796875, 33.57801474614399, 130.4296875, 33.65120829920498],
|
||||
"minzoom": 12,
|
||||
"maxzoom": 12
|
||||
},
|
||||
"resources": {
|
||||
"colors": "build/s52_resources_v2/base/s52_colors.json",
|
||||
"symbol_metadata": "build/s52_resources_v2/base/s52_symbol_metadata.json",
|
||||
"bitmap_pattern_metadata": "build/s52_resources_v2/base/s52_pattern_metadata.json",
|
||||
"vector_pattern_metadata": "build/s52_resources_v2/ap/s52-sprite-ap-pattern-metadata.json",
|
||||
"line_metadata": "build/s52_resources_v2/final/s52-sprite-full-line-metadata.json",
|
||||
"sprite_index": "build/s52_resources_v2/final/s52-sprite-full.json",
|
||||
"lookup_ir": "build/s52_presentation_ir_v2/lookup_rule_ir.json",
|
||||
"resource_contract": "build/s52_resources_v2/resource_contract.json"
|
||||
},
|
||||
"source_layers": [
|
||||
{"name": "anchor_caution_hazard_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "anchor_caution_hazard_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "anchorage_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "anchorage_outline", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "anchorage_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "baseline_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "baseline_line", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "bridge_structure", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "clearance_limit_line", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "clearance_limit_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "cm95_unmapped", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "depth_contour", "capabilities": ["line", "text"]},
|
||||
{"name": "facility_boundary_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "facility_boundary_outline", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "facility_boundary_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "fixed_fishing_gear_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "land_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "navigation_hazard_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "navigation_hazard_outline", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "navigation_hazard_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "navigation_marks", "capabilities": ["symbol", "text"]},
|
||||
{"name": "onshore_structure_area", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "onshore_structure_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "pilot_station_point", "capabilities": ["symbol", "text"]},
|
||||
{"name": "place_label_land", "capabilities": ["symbol", "text"]},
|
||||
{"name": "s52_areas", "capabilities": ["fill", "line", "symbol", "text"]},
|
||||
{"name": "s52_lines", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "s52_points", "capabilities": ["symbol", "text"]},
|
||||
{"name": "seabed_line", "capabilities": ["line", "symbol", "text"]},
|
||||
{"name": "seabed_text_point", "capabilities": ["symbol", "text"]}
|
||||
]
|
||||
}
|
||||
3334
config/sprite/semantic-sprite-20260808.json
Normal file
3334
config/sprite/semantic-sprite-20260808.json
Normal file
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
83
docs/PostgreSQL实现说明.md
Normal file
83
docs/PostgreSQL实现说明.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# CM95 Japan PostgreSQL/PostGIS 实现说明
|
||||
|
||||
## 目标
|
||||
|
||||
新工程独立完成:
|
||||
|
||||
CM95 Japan 原始数据 → PostgreSQL/PostGIS → NavSea PBF。
|
||||
|
||||
旧 MySQL 工程只作为行为参考,不是运行依赖,也不会被新工程读取、修改或迁移。
|
||||
|
||||
## 保持不变的业务层
|
||||
|
||||
以下文件从当前参考工程复制并保持逐字一致:
|
||||
|
||||
- CM93 binary parser
|
||||
- geometry normalization
|
||||
- scanner / dictionary loader
|
||||
- PBF mapping
|
||||
- tile policy
|
||||
- S-52 resolver
|
||||
|
||||
因此 WGS offset、Area/Coverage normalization、META/raw_geotype=16、raw preservation、确定性 FID 和 PBF mapping 继续沿用当前生产逻辑。
|
||||
|
||||
## PostgreSQL 数据层
|
||||
|
||||
数据库驱动使用 psycopg 3,空间能力使用 PostGIS。
|
||||
|
||||
主要对应关系:
|
||||
|
||||
- AUTO_INCREMENT → BIGSERIAL
|
||||
- JSON → JSONB
|
||||
- BLOB/MEDIUMBLOB → BYTEA
|
||||
- TINYINT(1) → BOOLEAN
|
||||
- Spatial index → PostGIS GiST
|
||||
- geometry WKB → ST_GeomFromWKB(..., 4326)
|
||||
- geometry readback → ST_AsBinary(...)
|
||||
- upsert → ON CONFLICT ... DO UPDATE
|
||||
- insert id → RETURNING id
|
||||
|
||||
所有标准化 Geometry 的 SRID 为 4326。
|
||||
|
||||
## PBF 空间查询
|
||||
|
||||
POINT / LINE / AREA 候选查询由 geometry 表的 GiST index 驱动,使用 bbox 运算符 && 配合 4326 tile envelope;同时继续保留 chart_feature 的数字 bbox 二次过滤。
|
||||
|
||||
SOUNDING 是 Point,因此同样使用 bbox 查询保持矩形 MBR 包含语义。PBF 的 Python 排序仍按 source_relative_path/source_record_index 执行,避免依赖数据库执行计划顺序。
|
||||
|
||||
## 当前源数据基线
|
||||
|
||||
2026-09-19 在 Linux 当前实际 /root/sourceserver/cm95-data/CM93JAPAN 扫描:
|
||||
|
||||
- Cells: 1492
|
||||
- A: 4
|
||||
- B: 15
|
||||
- C: 82
|
||||
- D: 354
|
||||
- E: 767
|
||||
- F: 247
|
||||
- G: 22
|
||||
- Z: 1
|
||||
|
||||
source-only 全量解析:
|
||||
|
||||
- Features: 1,282,431
|
||||
- Attributes: 1,506,625
|
||||
- Source parse failed: 0
|
||||
- Unknown objects: 0
|
||||
- Unknown attributes: 0
|
||||
|
||||
这套数据已不同于旧工程历史报告中的 784 Cell 基线。新 PostgreSQL 库应以当前 1492 Cell 源目录作为事实来源。
|
||||
|
||||
## 尚需真实 NAS 验证
|
||||
|
||||
代码已完成 PostgreSQL/PostGIS 方言切换,但在未配置 NAS 的 CM95_PG_* 连接参数前,不能声称以下步骤已经通过:
|
||||
|
||||
1. schema 在 NAS PostgreSQL/PostGIS 实际创建;
|
||||
2. 单 Cell / 小样本真实导入;
|
||||
3. 全量 1492 Cell 导入;
|
||||
4. PostgreSQL DB audit;
|
||||
5. 从 PostgreSQL 生成单 tile PBF;
|
||||
6. 全国 PBF production/finalize。
|
||||
|
||||
这些验证必须针对实际 NAS 数据库执行。
|
||||
27
pyproject.toml
Normal file
27
pyproject.toml
Normal file
@@ -0,0 +1,27 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "cm95-postgresql-pipeline"
|
||||
version = "0.1.0"
|
||||
description = "CM95 Japan source -> PostgreSQL/PostGIS -> NavSea MVT/PBF pipeline"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12,<3.13"
|
||||
dependencies = [
|
||||
"psycopg[binary]>=3.2,<4",
|
||||
"python-dotenv>=1.0,<2",
|
||||
"Shapely>=2.0,<3",
|
||||
"PyYAML>=6.0,<7",
|
||||
"mapbox-vector-tile>=2.2,<3",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
cm95pg = "cm93db.cli:main"
|
||||
cm95pbf = "cm95pbf.cli:main"
|
||||
|
||||
[tool.setuptools]
|
||||
package-dir = {"" = "src"}
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
5
requirements.txt
Normal file
5
requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
psycopg[binary]>=3.2,<4
|
||||
python-dotenv>=1.0,<2
|
||||
Shapely>=2.0,<3
|
||||
PyYAML>=6.0,<7
|
||||
mapbox-vector-tile>=2.2,<3
|
||||
109122
resources/s52/lookup_rule_ir.json
Normal file
109122
resources/s52/lookup_rule_ir.json
Normal file
File diff suppressed because it is too large
Load Diff
84
resources/s52/resource_contract.json
Normal file
84
resources/s52/resource_contract.json
Normal file
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"schema": "navsea.cm95.s52-resource-contract/v1",
|
||||
"status": "PASS",
|
||||
"reference": {
|
||||
"project": "OpenCPN",
|
||||
"commit": "f7211b4e1874c453e37a7f190aa191c55a25c32e",
|
||||
"chartsymbols_sha256": "c5678321ab3533605bb51f9376ad1909a50cedb4509afc34852d8d857a109de2",
|
||||
"csp_source_sha256": "d094d9345fa76acf601ca2c1a48da7d65f06f3086b8f810e6d1ab9f8a5014ea4"
|
||||
},
|
||||
"defined_resource_counts": {
|
||||
"SY": 1018,
|
||||
"AP": 30,
|
||||
"LC": 59,
|
||||
"total": 1107
|
||||
},
|
||||
"active_resource_counts": {
|
||||
"SY": 902,
|
||||
"AP": 27,
|
||||
"LC": 48,
|
||||
"total": 977
|
||||
},
|
||||
"built_resource_counts": {
|
||||
"SY": 1018,
|
||||
"AP": 30,
|
||||
"LC": 59,
|
||||
"total": 1107
|
||||
},
|
||||
"defined_missing_from_bundle": {
|
||||
"SY": [],
|
||||
"AP": [],
|
||||
"LC": []
|
||||
},
|
||||
"active_missing_from_bundle": {
|
||||
"SY": [],
|
||||
"AP": [],
|
||||
"LC": []
|
||||
},
|
||||
"upstream_undefined_references": {
|
||||
"SY": [
|
||||
"BCNCON81",
|
||||
"BOYLAT52",
|
||||
"BOYLAT53",
|
||||
"BOYLAT54",
|
||||
"BOYLAT55",
|
||||
"BOYLAT56",
|
||||
"BOYSPH79",
|
||||
"BOYSPR02",
|
||||
"BOYSPR03",
|
||||
"DANGER53",
|
||||
"DGPS01DRFSTA01",
|
||||
"ESSARE01",
|
||||
"FLTHAZ02",
|
||||
"LOWACC03",
|
||||
"NEWOBJ 01",
|
||||
"NEWOBJ01",
|
||||
"PSSARE01",
|
||||
"TOPSHP33",
|
||||
"TOWERS74|",
|
||||
"VEHTRF01",
|
||||
"boyspp50",
|
||||
"rdocal02",
|
||||
"rdocal03"
|
||||
],
|
||||
"AP": [],
|
||||
"LC": [
|
||||
"ARCSLN01",
|
||||
"NEWOBJ01"
|
||||
]
|
||||
},
|
||||
"undefined_reference_fallback": {
|
||||
"SY": "QUESMRK1",
|
||||
"LC": "QUESMRK1 symbol fallback",
|
||||
"AP": "QUESMRK1 pattern fallback",
|
||||
"source": "OpenCPN S52PLIB StringToRules"
|
||||
},
|
||||
"artifact_hashes": {
|
||||
"symbol_metadata_sha256": "5639de5d512b639c143d2ce88e209135e17640de094fdc63752ee6546806e77a",
|
||||
"bitmap_pattern_metadata_sha256": "738c4b8b0c387d8d6593d77e79fc8165c0d157a205f6dca197f6f5722155a875",
|
||||
"vector_pattern_metadata_sha256": "263bae788647b1728534e681769da2d24a4cc5ee086a056c8b1e36a7c4c3e78e",
|
||||
"line_metadata_sha256": "712e509cd70c89cdbed976797f23b3043590023cf9bece0cba4b900b100fa071",
|
||||
"sprite_index_sha256": "51519fb9a64f11dc100f894b2b54d14f56770ac9d511f15ae574ff5867af857d"
|
||||
},
|
||||
"failures": []
|
||||
}
|
||||
17
samples/README.md
Normal file
17
samples/README.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# 样本数据
|
||||
|
||||
本 ZIP 不重新分发用户的 CM93 数据。
|
||||
|
||||
建议将已经验证的小样本放在项目外,例如:
|
||||
|
||||
`D:\Ship\CM93V2\CM93JAPAN_TEST\04501020\`
|
||||
|
||||
然后运行:
|
||||
|
||||
`cm93db scan D:\Ship\CM93V2\CM93JAPAN_TEST\04501020`
|
||||
|
||||
和:
|
||||
|
||||
`cm93db inspect-dicts D:\Ship\CM93V2\CM93JAPAN_TEST`
|
||||
|
||||
正式 Codex 开发时,以真实样本作为格式回归基线。
|
||||
124
scripts/cm95_japan_postgresql_pipeline.sh
Executable file
124
scripts/cm95_japan_postgresql_pipeline.sh
Executable file
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
if [[ ! -f .env ]]; then
|
||||
echo "ERROR: .env 不存在。请从 .env.example 复制并填写 NAS PostgreSQL/PostGIS 连接参数。" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ! -x .venv/bin/python ]]; then
|
||||
echo "ERROR: .venv 不存在。先执行 python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/pip install -e ." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
source .venv/bin/activate
|
||||
set -a
|
||||
source .env
|
||||
set +a
|
||||
export PYTHONUNBUFFERED=1
|
||||
|
||||
SOURCE_ROOT="${CM95_SOURCE_DIR:-/root/sourceserver/cm95-data/CM93JAPAN}"
|
||||
ZOOMS="${CM95_PBF_ZOOMS:-0-12}"
|
||||
WORKERS="${CM95_PBF_WORKERS:-4}"
|
||||
MANIFEST="${CM95_FID_MANIFEST:-build/cm95_fid_manifest_v1.sqlite}"
|
||||
PLAN="${CM95_PRODUCTION_PLAN:-build/production/cm95_postgresql_japan_full_plan.json}"
|
||||
RELEASE_ROOT="${CM95_RELEASE_ROOT:-build/releases-postgresql-japan}"
|
||||
MAPPING="${CM95_PBF_MAPPING:-config/mappings/cm95_to_navsea_v1.yaml}"
|
||||
POLICY="${CM95_PBF_POLICY:-config/pbf/cm95_tile_policy_japan_national_v1.yaml}"
|
||||
AOI_NAME="${CM95_AOI_NAME:-japan-national-v1}"
|
||||
AOI_BBOX="${CM95_AOI_BBOX:-122.0,20.0,154.5,46.0}"
|
||||
TILE_ENUMERATION_MODE="${CM95_TILE_ENUMERATION_MODE:-aoi-complete}"
|
||||
S52_LOOKUPS="${CM95_S52_LOOKUPS:-resources/s52/lookup_rule_ir.json}"
|
||||
S52_RESOURCE_CONTRACT="${CM95_S52_RESOURCE_CONTRACT:-resources/s52/resource_contract.json}"
|
||||
|
||||
echo "== STEP 1: PostgreSQL/PostGIS environment =="
|
||||
cm95pg env-check
|
||||
|
||||
echo "== STEP 2: schema =="
|
||||
if [[ "${CM95_CREATE_DATABASE:-0}" == "1" ]]; then
|
||||
cm95pg db-init --create-database
|
||||
else
|
||||
cm95pg db-init
|
||||
fi
|
||||
cm95pg db-check
|
||||
|
||||
echo "== STEP 3: source scan =="
|
||||
cm95pg scan "$SOURCE_ROOT"
|
||||
|
||||
echo "== STEP 4: import CM95 Japan to PostgreSQL/PostGIS =="
|
||||
cm95pg import "$SOURCE_ROOT"
|
||||
|
||||
echo "== STEP 5: source/database audit =="
|
||||
cm95pg audit "$SOURCE_ROOT" \
|
||||
--precision-samples "${CM95_PRECISION_SAMPLES:-1024}" \
|
||||
--report logs/postgresql_full_audit.json
|
||||
|
||||
echo "== STEP 6: resolve dataset id =="
|
||||
DATASET_ID="$(SOURCE_ROOT="$SOURCE_ROOT" python - <<'PY'
|
||||
import os
|
||||
from pathlib import Path
|
||||
from cm93db.config import load_config
|
||||
from cm93db.database import connect
|
||||
root = str(Path(os.environ["SOURCE_ROOT"]).resolve())
|
||||
with connect(load_config()) as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT id FROM chart_dataset "
|
||||
"WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1",
|
||||
(root,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
raise SystemExit("dataset not found after import")
|
||||
print(int(row["id"]))
|
||||
PY
|
||||
)"
|
||||
echo "Dataset ID: $DATASET_ID"
|
||||
|
||||
echo "== STEP 7: deterministic FID manifest =="
|
||||
cm95pbf build-fid-manifest \
|
||||
--dataset-id "$DATASET_ID" \
|
||||
--manifest "$MANIFEST" \
|
||||
--rebuild-fid-manifest
|
||||
|
||||
echo "== STEP 8: production plan =="
|
||||
cm95pbf plan-production \
|
||||
--dataset-id "$DATASET_ID" \
|
||||
--zooms "$ZOOMS" \
|
||||
--profile delivery \
|
||||
--mapping "$MAPPING" \
|
||||
--policy "$POLICY" \
|
||||
--manifest "$MANIFEST" \
|
||||
--aoi-name "$AOI_NAME" \
|
||||
--aoi-bbox "$AOI_BBOX" \
|
||||
--tile-enumeration-mode "$TILE_ENUMERATION_MODE" \
|
||||
--s52-lookups "$S52_LOOKUPS" \
|
||||
--s52-resource-contract "$S52_RESOURCE_CONTRACT" \
|
||||
--s52-point-table Paper \
|
||||
--s52-boundary-table Plain \
|
||||
--s52-mariner-mode runtime \
|
||||
--tile-enumeration-mode coverage \
|
||||
--output "$PLAN"
|
||||
|
||||
echo "== STEP 9: PBF production =="
|
||||
cm95pbf build-production \
|
||||
--plan "$PLAN" \
|
||||
--release-root "$RELEASE_ROOT" \
|
||||
--workers "$WORKERS" \
|
||||
--max-attempts 3
|
||||
|
||||
echo "== STEP 10: production status =="
|
||||
cm95pbf status-production \
|
||||
--plan "$PLAN" \
|
||||
--release-root "$RELEASE_ROOT"
|
||||
|
||||
echo "== STEP 11: final release audit =="
|
||||
cm95pbf finalize-production \
|
||||
--plan "$PLAN" \
|
||||
--release-root "$RELEASE_ROOT" \
|
||||
--skip-style-contract
|
||||
|
||||
echo "CM95 Japan PostgreSQL/PostGIS -> PBF pipeline: PASS"
|
||||
50
scripts/cm95_postgresql_kyushu_compare.sh
Executable file
50
scripts/cm95_postgresql_kyushu_compare.sh
Executable file
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
[[ -f .env ]] || { echo "missing .env" >&2; exit 2; }
|
||||
source .venv/bin/activate
|
||||
set -a; source .env; set +a
|
||||
export PYTHONUNBUFFERED=1
|
||||
|
||||
MANIFEST="${CM95_FID_MANIFEST:-build/cm95_fid_manifest_v1.sqlite}"
|
||||
PLAN="${CM95_COMPARE_PLAN:-build/production/cm95_postgresql_kyushu_compare_plan.json}"
|
||||
RELEASE_ROOT="${CM95_COMPARE_RELEASE_ROOT:-build/releases-postgresql-compare}"
|
||||
MYSQL_RELEASE="${CM95_MYSQL_COMPARE_RELEASE:-/root/sourceserver/cm95/build/releases-s52-v2/cm95-d1-62a85bd40d3a-2b2e7d976a43}"
|
||||
WORKERS="${CM95_PBF_WORKERS:-4}"
|
||||
S52_LOOKUPS="${CM95_S52_LOOKUPS:-resources/s52/lookup_rule_ir.json}"
|
||||
S52_CONTRACT="${CM95_S52_RESOURCE_CONTRACT:-resources/s52/resource_contract.json}"
|
||||
|
||||
DATASET_ID="$(python - <<'PY'
|
||||
from cm93db.config import load_config
|
||||
from cm93db.database import connect
|
||||
with connect(load_config()) as c:
|
||||
with c.cursor() as cur:
|
||||
cur.execute("SELECT id FROM chart_dataset WHERE source_type=%s ORDER BY id LIMIT 1",("CM93",))
|
||||
row=cur.fetchone()
|
||||
if not row: raise SystemExit("dataset not found")
|
||||
print(int(row["id"]))
|
||||
PY
|
||||
)"
|
||||
|
||||
cm95pbf build-fid-manifest --dataset-id "$DATASET_ID" --manifest "$MANIFEST" --rebuild-fid-manifest
|
||||
cm95pbf plan-production --dataset-id "$DATASET_ID" --zooms 5-12 --profile delivery \
|
||||
--manifest "$MANIFEST" --s52-lookups "$S52_LOOKUPS" --s52-resource-contract "$S52_CONTRACT" \
|
||||
--s52-point-table Paper --s52-boundary-table Plain --s52-mariner-mode runtime \
|
||||
--aoi-name kyushu-main-nearby-v1 --aoi-bbox 128.0,29.5,132.5,34.8 \
|
||||
--tile-enumeration-mode aoi-complete --output "$PLAN"
|
||||
cm95pbf build-production --plan "$PLAN" --release-root "$RELEASE_ROOT" --workers "$WORKERS" --max-attempts 3
|
||||
cm95pbf status-production --plan "$PLAN" --release-root "$RELEASE_ROOT"
|
||||
cm95pbf finalize-production --plan "$PLAN" --release-root "$RELEASE_ROOT" --skip-style-contract
|
||||
|
||||
RELEASE_ID="$(python - "$PLAN" <<'PY'
|
||||
import json,sys
|
||||
print(json.load(open(sys.argv[1],encoding="utf-8"))["release_id"])
|
||||
PY
|
||||
)"
|
||||
PG_PBF="$RELEASE_ROOT/$RELEASE_ID/pbf/delivery"
|
||||
MYSQL_PBF="$MYSQL_RELEASE/pbf/delivery"
|
||||
python tools/compare_pbf_trees.py "$MYSQL_PBF" "$PG_PBF" --output logs/mysql_postgresql_kyushu_pbf_compare.json
|
||||
echo "POSTGRES_RELEASE_ID=$RELEASE_ID"
|
||||
echo "POSTGRES_PBF=$PG_PBF"
|
||||
echo "MYSQL_PBF=$MYSQL_PBF"
|
||||
46
scripts/cm95_postgresql_smoke.sh
Normal file
46
scripts/cm95_postgresql_smoke.sh
Normal file
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "$PROJECT_ROOT"
|
||||
|
||||
if [[ ! -f .env ]]; then
|
||||
echo "ERROR: .env 不存在;请填写 NAS PostgreSQL/PostGIS 参数。" >&2
|
||||
exit 2
|
||||
fi
|
||||
source .venv/bin/activate
|
||||
|
||||
SOURCE_ROOT="${CM95_SOURCE_DIR:-/root/sourceserver/cm95-data/CM93JAPAN}"
|
||||
LIMIT="${CM95_SMOKE_CELL_LIMIT:-20}"
|
||||
MANIFEST="build/smoke_fid_manifest.sqlite"
|
||||
|
||||
cm95pg env-check
|
||||
cm95pg db-init
|
||||
cm95pg db-check
|
||||
cm95pg scan "$SOURCE_ROOT"
|
||||
cm95pg import "$SOURCE_ROOT" --limit "$LIMIT"
|
||||
cm95pg audit "$SOURCE_ROOT" --limit "$LIMIT" --precision-samples 20 --report logs/postgresql_smoke_audit.json
|
||||
|
||||
DATASET_ID="$(SOURCE_ROOT="$SOURCE_ROOT" python - <<'PY'
|
||||
import os
|
||||
from pathlib import Path
|
||||
from cm93db.config import load_config
|
||||
from cm93db.database import connect
|
||||
root = str(Path(os.environ['SOURCE_ROOT']).resolve())
|
||||
with connect(load_config()) as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT id FROM chart_dataset WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1",
|
||||
(root,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
raise SystemExit('dataset not found')
|
||||
print(int(row['id']))
|
||||
PY
|
||||
)"
|
||||
|
||||
cm95pbf build-fid-manifest --dataset-id "$DATASET_ID" --manifest "$MANIFEST" --rebuild-fid-manifest
|
||||
cm95pbf build-tile --dataset-id "$DATASET_ID" --z 10 --x 914 --y 385 --profile delivery --manifest "$MANIFEST" --output-root build/smoke-pbf --report-root build/smoke-reports
|
||||
|
||||
echo "PostgreSQL/PostGIS smoke: PASS"
|
||||
226
sql/001_schema_postgresql.sql
Normal file
226
sql/001_schema_postgresql.sql
Normal file
@@ -0,0 +1,226 @@
|
||||
CREATE EXTENSION IF NOT EXISTS postgis;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_dataset (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_type VARCHAR(16) NOT NULL DEFAULT 'CM93',
|
||||
name VARCHAR(255) NOT NULL,
|
||||
source_root TEXT NOT NULL,
|
||||
source_fingerprint CHAR(64),
|
||||
parser_version VARCHAR(32) NOT NULL,
|
||||
normalized_srid INTEGER NOT NULL DEFAULT 4326,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_source_type ON chart_dataset(source_type);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_dictionary_object (
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
object_class_id SMALLINT NOT NULL,
|
||||
object_class VARCHAR(16) NOT NULL,
|
||||
geometry_code VARCHAR(8),
|
||||
display_code VARCHAR(8),
|
||||
description VARCHAR(255),
|
||||
raw_line TEXT NOT NULL,
|
||||
PRIMARY KEY (dataset_id, object_class_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_dict_object_class ON chart_dictionary_object(object_class);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_dictionary_attribute (
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
attribute_id SMALLINT NOT NULL,
|
||||
attribute_name VARCHAR(16) NOT NULL,
|
||||
source_type_name VARCHAR(32) NOT NULL,
|
||||
decoder_type CHAR(1) NOT NULL,
|
||||
min_value DOUBLE PRECISION,
|
||||
max_value DOUBLE PRECISION,
|
||||
raw_line TEXT NOT NULL,
|
||||
PRIMARY KEY (dataset_id, attribute_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_dict_attribute_name ON chart_dictionary_attribute(attribute_name);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_cell (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
source_relative_path VARCHAR(1024) NOT NULL,
|
||||
source_file_name VARCHAR(255) NOT NULL,
|
||||
source_size BIGINT NOT NULL,
|
||||
source_sha256 CHAR(64),
|
||||
scale_code CHAR(1) NOT NULL,
|
||||
native_scale INTEGER NOT NULL,
|
||||
cell_name VARCHAR(64) NOT NULL,
|
||||
prolog_word0 INTEGER,
|
||||
table1_length BIGINT,
|
||||
table2_length BIGINT,
|
||||
easting_min DOUBLE PRECISION,
|
||||
northing_min DOUBLE PRECISION,
|
||||
easting_max DOUBLE PRECISION,
|
||||
northing_max DOUBLE PRECISION,
|
||||
bbox_min_lon DOUBLE PRECISION,
|
||||
bbox_min_lat DOUBLE PRECISION,
|
||||
bbox_max_lon DOUBLE PRECISION,
|
||||
bbox_max_lat DOUBLE PRECISION,
|
||||
vector_record_count INTEGER,
|
||||
vector_point_count BIGINT,
|
||||
point2d_count INTEGER,
|
||||
point3d_count INTEGER,
|
||||
feature_count INTEGER,
|
||||
attribute_count BIGINT,
|
||||
import_status VARCHAR(16) NOT NULL DEFAULT 'DISCOVERED'
|
||||
CHECK (import_status IN ('DISCOVERED','IMPORTING','DONE','FAILED')),
|
||||
error_message TEXT,
|
||||
imported_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE (dataset_id, source_relative_path)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cell_scale ON chart_cell(dataset_id, scale_code);
|
||||
CREATE INDEX IF NOT EXISTS idx_cell_status ON chart_cell(dataset_id, import_status);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_coverage (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
|
||||
source_record_index INTEGER NOT NULL,
|
||||
object_class_id SMALLINT NOT NULL,
|
||||
compilation_scale INTEGER,
|
||||
horizontal_datum INTEGER,
|
||||
vertical_datum INTEGER,
|
||||
wgsox DOUBLE PRECISION,
|
||||
wgsoy DOUBLE PRECISION,
|
||||
attributes_json JSONB,
|
||||
geom geometry(Geometry,4326) NOT NULL,
|
||||
geometry_repaired BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
geometry_issue VARCHAR(255),
|
||||
UNIQUE (cell_id, source_record_index)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cov_cell ON chart_coverage(cell_id);
|
||||
CREATE INDEX IF NOT EXISTS spx_cov_geom ON chart_coverage USING GIST(geom);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_feature (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
|
||||
coverage_id BIGINT REFERENCES chart_coverage(id) ON DELETE SET NULL,
|
||||
source_record_index INTEGER NOT NULL,
|
||||
object_class_id SMALLINT,
|
||||
object_class VARCHAR(16) NOT NULL,
|
||||
geometry_type VARCHAR(16) NOT NULL
|
||||
CHECK (geometry_type IN ('POINT','LINE','AREA','SOUNDING','META','UNKNOWN')),
|
||||
scale_code CHAR(1) NOT NULL,
|
||||
native_scale INTEGER NOT NULL,
|
||||
raw_geotype SMALLINT,
|
||||
normalized_srid INTEGER NOT NULL DEFAULT 4326,
|
||||
bbox_min_lon DOUBLE PRECISION,
|
||||
bbox_min_lat DOUBLE PRECISION,
|
||||
bbox_max_lon DOUBLE PRECISION,
|
||||
bbox_max_lat DOUBLE PRECISION,
|
||||
attributes_json JSONB,
|
||||
raw_feature_blob BYTEA,
|
||||
raw_attribute_blob BYTEA,
|
||||
geometry_repaired BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
geometry_issue VARCHAR(255),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE (cell_id, source_record_index)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_class ON chart_feature(dataset_id, object_class);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_scale ON chart_feature(dataset_id, native_scale);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_cell ON chart_feature(cell_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_coverage ON chart_feature(coverage_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_feature_attribute (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
feature_id BIGINT NOT NULL REFERENCES chart_feature(id) ON DELETE CASCADE,
|
||||
ordinal_no SMALLINT NOT NULL,
|
||||
attribute_id SMALLINT NOT NULL,
|
||||
attribute_name VARCHAR(16) NOT NULL,
|
||||
value_type VARCHAR(16) NOT NULL
|
||||
CHECK (value_type IN ('BYTE','INT','LONG','REAL','WORD10','STRING','LIST','COMPLEX','UNKNOWN')),
|
||||
value_int BIGINT,
|
||||
value_real DOUBLE PRECISION,
|
||||
value_text TEXT,
|
||||
value_json JSONB,
|
||||
raw_value_blob BYTEA,
|
||||
UNIQUE (feature_id, ordinal_no)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_attr_name ON chart_feature_attribute(attribute_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_feature_attr_id ON chart_feature_attribute(attribute_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_point_geometry (
|
||||
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
|
||||
geom geometry(Point,4326) NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS spx_point_geom ON chart_point_geometry USING GIST(geom);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_line_geometry (
|
||||
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
|
||||
geom geometry(Geometry,4326) NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS spx_line_geom ON chart_line_geometry USING GIST(geom);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_area_geometry (
|
||||
feature_id BIGINT PRIMARY KEY REFERENCES chart_feature(id) ON DELETE CASCADE,
|
||||
geom geometry(Geometry,4326) NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS spx_area_geom ON chart_area_geometry USING GIST(geom);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_sounding_point (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
feature_id BIGINT NOT NULL REFERENCES chart_feature(id) ON DELETE CASCADE,
|
||||
point_index INTEGER NOT NULL,
|
||||
raw_x BIGINT NOT NULL,
|
||||
raw_y BIGINT NOT NULL,
|
||||
raw_z BIGINT NOT NULL,
|
||||
depth DOUBLE PRECISION NOT NULL,
|
||||
geom geometry(Point,4326) NOT NULL,
|
||||
UNIQUE (feature_id, point_index)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_sounding_depth ON chart_sounding_point(depth);
|
||||
CREATE INDEX IF NOT EXISTS spx_sounding_geom ON chart_sounding_point USING GIST(geom);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_raw_record (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
cell_id BIGINT NOT NULL REFERENCES chart_cell(id) ON DELETE CASCADE,
|
||||
record_kind VARCHAR(16) NOT NULL
|
||||
CHECK (record_kind IN ('HEADER','VECTOR','POINT2D','POINT3D','FEATURE','ATTRIBUTE','OTHER')),
|
||||
source_record_index INTEGER NOT NULL DEFAULT 0,
|
||||
raw_blob BYTEA NOT NULL,
|
||||
raw_metadata_json JSONB
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_raw_cell_kind
|
||||
ON chart_raw_record(cell_id, record_kind, source_record_index);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_import_job (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
dataset_id BIGINT NOT NULL REFERENCES chart_dataset(id) ON DELETE CASCADE,
|
||||
status VARCHAR(16) NOT NULL DEFAULT 'RUNNING'
|
||||
CHECK (status IN ('RUNNING','DONE','FAILED','CANCELLED')),
|
||||
started_at TIMESTAMPTZ NOT NULL,
|
||||
finished_at TIMESTAMPTZ,
|
||||
cells_discovered INTEGER NOT NULL DEFAULT 0,
|
||||
cells_done INTEGER NOT NULL DEFAULT 0,
|
||||
cells_failed INTEGER NOT NULL DEFAULT 0,
|
||||
features_source BIGINT NOT NULL DEFAULT 0,
|
||||
features_imported BIGINT NOT NULL DEFAULT 0,
|
||||
attributes_source BIGINT NOT NULL DEFAULT 0,
|
||||
attributes_imported BIGINT NOT NULL DEFAULT 0,
|
||||
unknown_object_count BIGINT NOT NULL DEFAULT 0,
|
||||
unknown_attribute_count BIGINT NOT NULL DEFAULT 0,
|
||||
geometry_invalid_count BIGINT NOT NULL DEFAULT 0,
|
||||
geometry_repaired_count BIGINT NOT NULL DEFAULT 0,
|
||||
summary_json JSONB
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_job_dataset ON chart_import_job(dataset_id, started_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chart_import_event (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
job_id BIGINT REFERENCES chart_import_job(id) ON DELETE CASCADE,
|
||||
cell_id BIGINT REFERENCES chart_cell(id) ON DELETE CASCADE,
|
||||
level VARCHAR(16) NOT NULL CHECK (level IN ('DEBUG','INFO','WARNING','ERROR')),
|
||||
event_code VARCHAR(64) NOT NULL,
|
||||
message TEXT NOT NULL,
|
||||
details_json JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_event_job ON chart_import_event(job_id, id);
|
||||
CREATE INDEX IF NOT EXISTS idx_event_cell ON chart_import_event(cell_id, id);
|
||||
CREATE INDEX IF NOT EXISTS idx_event_code ON chart_import_event(event_code);
|
||||
1
src/cm93db/__init__.py
Normal file
1
src/cm93db/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
__version__ = "0.1.0"
|
||||
299
src/cm93db/audit.py
Normal file
299
src/cm93db/audit.py
Normal file
@@ -0,0 +1,299 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
import math
|
||||
from typing import Any
|
||||
|
||||
from shapely import from_wkb
|
||||
|
||||
from .cm93_binary import parse_cell
|
||||
from .config import DbConfig
|
||||
from .database import (
|
||||
connect,
|
||||
assert_postgresql_compatible,
|
||||
server_version,
|
||||
as_wkb_sql,
|
||||
)
|
||||
from .dictionaries import load_object_dictionary, load_attribute_dictionary
|
||||
from .geometry import build_coverages, build_feature
|
||||
from .scanner import scan_cm93, CellFile
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceCellStat:
|
||||
rel: str
|
||||
sha256: str
|
||||
file_size: int
|
||||
scale: str
|
||||
features: int
|
||||
attributes: int
|
||||
points: int
|
||||
lines: int
|
||||
areas: int
|
||||
coverages: int
|
||||
sounding_features: int
|
||||
sounding_points: int
|
||||
unknown_objects: int
|
||||
unknown_attributes: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class PrecisionSample:
|
||||
rel: str
|
||||
record_index: int
|
||||
geometry_type: str
|
||||
expected_wkb: bytes
|
||||
point_index: int | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuditResult:
|
||||
source_cells: int = 0
|
||||
source_parse_failed: int = 0
|
||||
db_missing_cells: int = 0
|
||||
db_not_done_cells: int = 0
|
||||
count_mismatches: int = 0
|
||||
hash_mismatches: int = 0
|
||||
unknown_objects: int = 0
|
||||
unknown_attributes: int = 0
|
||||
precision_checked: int = 0
|
||||
precision_failed: int = 0
|
||||
max_precision_delta_deg: float = 0.0
|
||||
raw_preservation_mismatches: int = 0
|
||||
geometry_validity_issues: int = 0
|
||||
messages: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
return not any((self.source_parse_failed,self.db_missing_cells,self.db_not_done_cells,self.count_mismatches,self.hash_mismatches,self.raw_preservation_mismatches,self.unknown_objects,self.unknown_attributes,self.precision_failed))
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"passed": self.passed,
|
||||
"source_cells": self.source_cells,
|
||||
"source_parse_failed": self.source_parse_failed,
|
||||
"db_missing_cells": self.db_missing_cells,
|
||||
"db_not_done_cells": self.db_not_done_cells,
|
||||
"count_mismatches": self.count_mismatches,
|
||||
"hash_mismatches": self.hash_mismatches,
|
||||
"raw_preservation_mismatches": self.raw_preservation_mismatches,
|
||||
"geometry_validity_issues": self.geometry_validity_issues,
|
||||
"unknown_objects": self.unknown_objects,
|
||||
"unknown_attributes": self.unknown_attributes,
|
||||
"precision_checked": self.precision_checked,
|
||||
"precision_failed": self.precision_failed,
|
||||
"max_precision_delta_deg": self.max_precision_delta_deg,
|
||||
"messages": self.messages,
|
||||
}
|
||||
|
||||
|
||||
def _rel(cf: CellFile) -> str:
|
||||
return str(cf.relative_path).replace("/", "\\")
|
||||
|
||||
|
||||
def _select_cells(scan, limit: int | None, scales: set[str] | None, cell_name: str | None):
|
||||
cells=list(scan.cells)
|
||||
if scales:
|
||||
cells=[c for c in cells if c.scale_code in scales]
|
||||
if cell_name:
|
||||
t=cell_name.lower()
|
||||
cells=[c for c in cells if c.path.name.lower()==t or c.path.stem.lower()==t]
|
||||
if limit is not None:
|
||||
cells=cells[:limit]
|
||||
return cells
|
||||
|
||||
|
||||
def _source_stat(cell, cf: CellFile) -> SourceCellStat:
|
||||
p=l=a=cov=sf=sp=0
|
||||
for f in cell.features:
|
||||
if f.object_class == "_m_sor":
|
||||
cov += 1
|
||||
continue
|
||||
if f.geometry_primitive == 1: p += 1
|
||||
elif f.geometry_primitive == 2: l += 1
|
||||
elif f.geometry_primitive == 4: a += 1
|
||||
elif f.geometry_primitive == 8:
|
||||
sf += 1
|
||||
if f.point3d_index is not None:
|
||||
sp += len(cell.point3d[f.point3d_index].points)
|
||||
return SourceCellStat(
|
||||
rel=_rel(cf),sha256=cell.sha256,file_size=cell.file_size,scale=cf.scale_code,features=len(cell.features),attributes=cell.attribute_count,
|
||||
points=p,lines=l,areas=a,coverages=cov,sounding_features=sf,sounding_points=sp,
|
||||
unknown_objects=len(cell.unknown_object_ids),unknown_attributes=len(cell.unknown_attribute_ids),
|
||||
)
|
||||
|
||||
|
||||
def collect_source(root: Path, limit: int | None = None, scales: set[str] | None = None, cell_name: str | None = None,
|
||||
precision_samples: int = 0, validate_geometry: bool = False):
|
||||
scan=scan_cm93(root)
|
||||
if not scan.object_dictionary or not scan.attribute_dictionary:
|
||||
raise RuntimeError("CM93 字典未找到")
|
||||
objs={x.object_id:x for x in load_object_dictionary(scan.object_dictionary)}
|
||||
attrs={x.attribute_id:x for x in load_attribute_dictionary(scan.attribute_dictionary)}
|
||||
cells=_select_cells(scan,limit,scales,cell_name)
|
||||
stats: dict[str,SourceCellStat]={}
|
||||
failures: list[str]=[]
|
||||
samples: list[PrecisionSample]=[]
|
||||
geom_issues=0
|
||||
quota=max(1, math.ceil(precision_samples / max(1,len({c.scale_code for c in cells})))) if precision_samples else 0
|
||||
per_scale=defaultdict(int)
|
||||
|
||||
for n,cf in enumerate(cells,1):
|
||||
try:
|
||||
cell=parse_cell(cf.path,objs,attrs)
|
||||
st=_source_stat(cell,cf); stats[st.rel]=st
|
||||
want = precision_samples and per_scale[cf.scale_code] < quota and len(samples) < precision_samples
|
||||
if want or validate_geometry:
|
||||
covs=build_coverages(cell)
|
||||
for f in cell.features:
|
||||
b=build_feature(cell,f,covs,validate=validate_geometry)
|
||||
if validate_geometry and b.geometry_issue:
|
||||
geom_issues += 1
|
||||
if want and per_scale[cf.scale_code] < quota and len(samples) < precision_samples:
|
||||
if b.geom is not None and b.geometry_type in {"POINT","LINE","AREA"}:
|
||||
samples.append(PrecisionSample(st.rel,f.index,b.geometry_type,b.geom.wkb))
|
||||
per_scale[cf.scale_code]+=1
|
||||
elif b.geometry_type=="SOUNDING" and b.soundings:
|
||||
samples.append(PrecisionSample(st.rel,f.index,"SOUNDING",b.soundings[0][4].wkb,0))
|
||||
per_scale[cf.scale_code]+=1
|
||||
if n % 250 == 0 or n == len(cells):
|
||||
print(f"source audit [{n}/{len(cells)}]")
|
||||
except Exception as e:
|
||||
failures.append(f"{_rel(cf)}: {e}")
|
||||
return stats,failures,samples,geom_issues
|
||||
|
||||
|
||||
def _group_counts(conn, sql: str) -> dict[int,int]:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(sql)
|
||||
return {int(r["cell_id"]):int(r["n"]) for r in cur.fetchall()}
|
||||
|
||||
|
||||
def _dataset_id(conn, root: Path) -> int | None:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT id FROM chart_dataset WHERE source_type='CM93' AND source_root=%s ORDER BY id LIMIT 1", (str(root.resolve()),))
|
||||
r=cur.fetchone(); return int(r["id"]) if r else None
|
||||
|
||||
|
||||
def _precision_db_geom(conn, cell_id: int, sample: PrecisionSample):
|
||||
if sample.geometry_type == "SOUNDING":
|
||||
sql=(f"SELECT {as_wkb_sql('s.geom')} AS wkb FROM chart_feature f JOIN chart_sounding_point s ON s.feature_id=f.id "
|
||||
"WHERE f.cell_id=%s AND f.source_record_index=%s AND s.point_index=%s")
|
||||
params=(cell_id,sample.record_index,sample.point_index or 0)
|
||||
else:
|
||||
table={"POINT":"chart_point_geometry","LINE":"chart_line_geometry","AREA":"chart_area_geometry"}[sample.geometry_type]
|
||||
sql=(f"SELECT {as_wkb_sql('g.geom')} AS wkb FROM chart_feature f JOIN {table} g ON g.feature_id=f.id "
|
||||
"WHERE f.cell_id=%s AND f.source_record_index=%s")
|
||||
params=(cell_id,sample.record_index)
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(sql,params); r=cur.fetchone()
|
||||
return bytes(r["wkb"]) if r and r["wkb"] is not None else None
|
||||
|
||||
|
||||
def run_audit(cfg: DbConfig, root: Path, limit: int | None = None, scales: set[str] | None = None, cell_name: str | None = None,
|
||||
precision_samples: int = 128, tolerance_deg: float = 1e-12, validate_geometry: bool = False, source_only: bool = False, require_raw: bool = True) -> AuditResult:
|
||||
result=AuditResult()
|
||||
stats,failures,samples,geom_issues=collect_source(root,limit,scales,cell_name,precision_samples if not source_only else 0,validate_geometry)
|
||||
result.source_cells=len(stats)+len(failures)
|
||||
result.source_parse_failed=len(failures)
|
||||
result.messages.extend(f"SOURCE_PARSE_FAILED {m}" for m in failures[:100])
|
||||
result.unknown_objects=sum(s.unknown_objects for s in stats.values())
|
||||
result.unknown_attributes=sum(s.unknown_attributes for s in stats.values())
|
||||
print(f"Source cells parsed: {len(stats)}, failed: {len(failures)}")
|
||||
print(f"Source features: {sum(s.features for s in stats.values())}")
|
||||
print(f"Source attributes: {sum(s.attributes for s in stats.values())}")
|
||||
result.geometry_validity_issues=geom_issues
|
||||
if validate_geometry:
|
||||
print(f"Source geometry issues (Shapely validity/structure): {geom_issues}")
|
||||
if source_only:
|
||||
return result
|
||||
|
||||
conn=connect(cfg)
|
||||
try:
|
||||
assert_postgresql_compatible(server_version(conn))
|
||||
did=_dataset_id(conn,root)
|
||||
if did is None:
|
||||
result.db_missing_cells=len(stats)
|
||||
result.messages.append("数据库中找不到该 source_root 对应 dataset")
|
||||
return result
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT id,source_relative_path,import_status,source_sha256,feature_count,attribute_count FROM chart_cell WHERE dataset_id=%s", (did,))
|
||||
rows=cur.fetchall()
|
||||
db_by_rel={str(r["source_relative_path"]):r for r in rows}
|
||||
cellids=[int(r["id"]) for r in rows]
|
||||
fcnt=_group_counts(conn,f"SELECT cell_id,COUNT(*) n FROM chart_feature WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
|
||||
acnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_feature_attribute a JOIN chart_feature f ON f.id=a.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
|
||||
pcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_point_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
|
||||
lcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_line_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
|
||||
arcnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_area_geometry g JOIN chart_feature f ON f.id=g.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
|
||||
ccnt=_group_counts(conn,f"SELECT cell_id,COUNT(*) n FROM chart_coverage WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
|
||||
scnt=_group_counts(conn,f"SELECT f.cell_id,COUNT(*) n FROM chart_sounding_point s JOIN chart_feature f ON f.id=s.feature_id JOIN chart_cell c ON c.id=f.cell_id WHERE c.dataset_id={did} GROUP BY f.cell_id")
|
||||
rawbytes=_group_counts(conn,f"SELECT cell_id,SUM(OCTET_LENGTH(raw_blob)) n FROM chart_raw_record WHERE cell_id IN (SELECT id FROM chart_cell WHERE dataset_id={did}) GROUP BY cell_id")
|
||||
|
||||
for rel,s in stats.items():
|
||||
r=db_by_rel.get(rel)
|
||||
if not r:
|
||||
result.db_missing_cells+=1; result.messages.append(f"DB_MISSING {rel}"); continue
|
||||
cid=int(r["id"])
|
||||
if r["import_status"]!="DONE":
|
||||
result.db_not_done_cells+=1; result.messages.append(f"DB_NOT_DONE {rel} status={r['import_status']}")
|
||||
if r.get("source_sha256") and str(r["source_sha256"]) != s.sha256:
|
||||
result.hash_mismatches+=1; result.messages.append(f"HASH_MISMATCH {rel}")
|
||||
actual=(fcnt.get(cid,0),acnt.get(cid,0),pcnt.get(cid,0),lcnt.get(cid,0),arcnt.get(cid,0),ccnt.get(cid,0),scnt.get(cid,0))
|
||||
expected=(s.features,s.attributes,s.points,s.lines,s.areas,s.coverages,s.sounding_points)
|
||||
if actual != expected:
|
||||
result.count_mismatches+=1
|
||||
result.messages.append(f"COUNT_MISMATCH {rel} expected={expected} actual={actual}")
|
||||
if require_raw and rawbytes.get(cid,0) != s.file_size:
|
||||
result.raw_preservation_mismatches += 1
|
||||
result.messages.append(f"RAW_BYTES_MISMATCH {rel} expected={s.file_size} actual={rawbytes.get(cid,0)}")
|
||||
|
||||
# Precision audit: source-derived geometry versus geometry round-tripped through PostGIS.
|
||||
rel_to_id={k:int(v["id"]) for k,v in db_by_rel.items()}
|
||||
for sample in samples:
|
||||
cid=rel_to_id.get(sample.rel)
|
||||
if not cid:
|
||||
continue
|
||||
dbwkb=_precision_db_geom(conn,cid,sample)
|
||||
result.precision_checked+=1
|
||||
if dbwkb is None:
|
||||
result.precision_failed+=1
|
||||
result.messages.append(f"PRECISION_MISSING {sample.rel} record={sample.record_index}")
|
||||
continue
|
||||
expected=from_wkb(sample.expected_wkb); actual=from_wkb(dbwkb)
|
||||
try:
|
||||
ok=expected.equals_exact(actual,tolerance_deg)
|
||||
delta=expected.hausdorff_distance(actual) if not ok else 0.0
|
||||
except Exception:
|
||||
ok=False; delta=float("inf")
|
||||
if math.isfinite(delta):
|
||||
result.max_precision_delta_deg=max(result.max_precision_delta_deg,float(delta))
|
||||
if not ok:
|
||||
result.precision_failed+=1
|
||||
result.messages.append(f"PRECISION_FAIL {sample.rel} record={sample.record_index} type={sample.geometry_type} delta_deg={delta}")
|
||||
finally:
|
||||
conn.close()
|
||||
return result
|
||||
|
||||
|
||||
def print_result(r: AuditResult) -> None:
|
||||
print("\n=== CM93 AUDIT ===")
|
||||
print(f"source_cells : {r.source_cells}")
|
||||
print(f"source_parse_failed : {r.source_parse_failed}")
|
||||
print(f"db_missing_cells : {r.db_missing_cells}")
|
||||
print(f"db_not_done_cells : {r.db_not_done_cells}")
|
||||
print(f"count_mismatches : {r.count_mismatches}")
|
||||
print(f"hash_mismatches : {r.hash_mismatches}")
|
||||
print(f"raw_preservation_bad : {r.raw_preservation_mismatches}")
|
||||
print(f"geometry_validity : {r.geometry_validity_issues} (informational)")
|
||||
print(f"unknown_objects : {r.unknown_objects}")
|
||||
print(f"unknown_attributes : {r.unknown_attributes}")
|
||||
print(f"precision_checked : {r.precision_checked}")
|
||||
print(f"precision_failed : {r.precision_failed}")
|
||||
print(f"max_precision_delta° : {r.max_precision_delta_deg:.16g}")
|
||||
print(f"RESULT : {'PASS' if r.passed else 'FAIL'}")
|
||||
if r.messages:
|
||||
print("\nDetails (max 100 shown):")
|
||||
for m in r.messages[:100]: print(" -",m)
|
||||
468
src/cm93db/cell_parser.py
Normal file
468
src/cm93db/cell_parser.py
Normal file
@@ -0,0 +1,468 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
import hashlib
|
||||
import struct
|
||||
from typing import Any
|
||||
|
||||
from .decoder import decode_cm93
|
||||
from .dictionaries import AttributeDef, ObjectDef
|
||||
from .precision import CM93_COORD_MAX, CM93_SEMIMAJOR_M
|
||||
|
||||
|
||||
class CM93ParseError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CM93Header:
|
||||
lon_min: float
|
||||
lat_min: float
|
||||
lon_max: float
|
||||
lat_max: float
|
||||
easting_min: float
|
||||
northing_min: float
|
||||
easting_max: float
|
||||
northing_max: float
|
||||
vector_records: int
|
||||
vector_record_points: int
|
||||
m_46: int
|
||||
m_4a: int
|
||||
point3d_records: int
|
||||
point3d_points: int
|
||||
m_54: int
|
||||
point2d_records: int
|
||||
m_5a: int
|
||||
m_5c: int
|
||||
feature_records: int
|
||||
m_60: int
|
||||
m_64: int
|
||||
m_68: int
|
||||
m_6a: int
|
||||
m_6c: int
|
||||
related_object_pointers: int
|
||||
m_72: int
|
||||
m_76: int
|
||||
attribute_bytes: int
|
||||
m_7c: int
|
||||
|
||||
@property
|
||||
def transform_x_rate(self) -> float:
|
||||
delta_x = self.easting_max - self.easting_min
|
||||
if delta_x < 0.0:
|
||||
delta_x += CM93_SEMIMAJOR_M * 2.0 * 3.141592653589793
|
||||
return delta_x / CM93_COORD_MAX
|
||||
|
||||
@property
|
||||
def transform_y_rate(self) -> float:
|
||||
return (self.northing_max - self.northing_min) / CM93_COORD_MAX
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EdgeRecord:
|
||||
index: int
|
||||
points: tuple[tuple[int, int], ...]
|
||||
raw: bytes
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SoundingGroup:
|
||||
index: int
|
||||
points: tuple[tuple[int, int, int], ...]
|
||||
raw: bytes
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParsedAttribute:
|
||||
ordinal: int
|
||||
attribute_id: int
|
||||
name: str
|
||||
source_type: str
|
||||
decoder_type: str
|
||||
value_type: str
|
||||
value: Any
|
||||
value_int: int | None
|
||||
value_real: float | None
|
||||
value_text: str | None
|
||||
value_json: Any | None
|
||||
raw_value: bytes
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GeometryRef:
|
||||
edge_index: int
|
||||
segment_usage: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedFeature:
|
||||
record_index: int
|
||||
object_type: int
|
||||
object_class: str
|
||||
raw_geotype: int
|
||||
primitive: int
|
||||
geometry_refs: tuple[GeometryRef, ...] = ()
|
||||
point2d_index: int | None = None
|
||||
point3d_index: int | None = None
|
||||
related_indices: tuple[int, ...] = ()
|
||||
related_count_hint: int | None = None
|
||||
n_attributes: int = 0
|
||||
attributes: tuple[ParsedAttribute, ...] = ()
|
||||
raw_record: bytes = b""
|
||||
raw_attribute_block: bytes = b""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParsedCell:
|
||||
path: Path
|
||||
source_size: int
|
||||
sha256: str
|
||||
encoded_bytes: bytes
|
||||
decoded_bytes: bytes
|
||||
prolog_word0: int
|
||||
table1_length: int
|
||||
table2_length: int
|
||||
header: CM93Header
|
||||
edges: tuple[EdgeRecord, ...]
|
||||
point2d: tuple[tuple[int, int], ...]
|
||||
point3d: tuple[SoundingGroup, ...]
|
||||
features: tuple[ParsedFeature, ...]
|
||||
end_offset: int
|
||||
table_offsets: dict[str, int]
|
||||
attribute_count: int
|
||||
|
||||
|
||||
class _Reader:
|
||||
def __init__(self, data: bytes, offset: int = 0):
|
||||
self.data = data
|
||||
self.offset = offset
|
||||
|
||||
def _need(self, n: int) -> None:
|
||||
if self.offset + n > len(self.data):
|
||||
raise CM93ParseError(
|
||||
f"unexpected EOF at {self.offset}, need {n}, size={len(self.data)}"
|
||||
)
|
||||
|
||||
def u8(self) -> int:
|
||||
self._need(1)
|
||||
v = self.data[self.offset]
|
||||
self.offset += 1
|
||||
return v
|
||||
|
||||
def u16(self) -> int:
|
||||
self._need(2)
|
||||
v = struct.unpack_from("<H", self.data, self.offset)[0]
|
||||
self.offset += 2
|
||||
return v
|
||||
|
||||
def i32(self) -> int:
|
||||
self._need(4)
|
||||
v = struct.unpack_from("<i", self.data, self.offset)[0]
|
||||
self.offset += 4
|
||||
return v
|
||||
|
||||
def f32(self) -> float:
|
||||
self._need(4)
|
||||
v = struct.unpack_from("<f", self.data, self.offset)[0]
|
||||
self.offset += 4
|
||||
return float(v)
|
||||
|
||||
def f64(self) -> float:
|
||||
self._need(8)
|
||||
v = struct.unpack_from("<d", self.data, self.offset)[0]
|
||||
self.offset += 8
|
||||
return float(v)
|
||||
|
||||
def take(self, n: int) -> bytes:
|
||||
self._need(n)
|
||||
b = self.data[self.offset:self.offset+n]
|
||||
self.offset += n
|
||||
return b
|
||||
|
||||
def cstring(self) -> bytes:
|
||||
end = self.data.find(b"\x00", self.offset)
|
||||
if end < 0:
|
||||
raise CM93ParseError(f"unterminated string at {self.offset}")
|
||||
b = self.data[self.offset:end]
|
||||
self.offset = end + 1
|
||||
return b
|
||||
|
||||
|
||||
def _parse_header(r: _Reader) -> CM93Header:
|
||||
start = r.offset
|
||||
vals: list[Any] = [r.f64() for _ in range(8)]
|
||||
vector_records = r.u16()
|
||||
vector_record_points = r.i32()
|
||||
m_46 = r.i32()
|
||||
m_4a = r.i32()
|
||||
point3d_records = r.u16()
|
||||
point3d_points = r.i32()
|
||||
m_54 = r.i32()
|
||||
point2d_records = r.u16()
|
||||
m_5a = r.u16()
|
||||
m_5c = r.u16()
|
||||
feature_records = r.u16()
|
||||
m_60 = r.i32()
|
||||
m_64 = r.i32()
|
||||
m_68 = r.u16()
|
||||
m_6a = r.u16()
|
||||
m_6c = r.u16()
|
||||
related = r.i32()
|
||||
m_72 = r.i32()
|
||||
m_76 = r.u16()
|
||||
attr_bytes = r.i32()
|
||||
m_7c = r.i32()
|
||||
if r.offset - start != 128:
|
||||
raise CM93ParseError(f"header size != 128: {r.offset-start}")
|
||||
if min(vector_records, vector_record_points, point3d_records, point3d_points,
|
||||
point2d_records, feature_records, related, attr_bytes) < 0:
|
||||
raise CM93ParseError("negative header count")
|
||||
return CM93Header(
|
||||
*vals, vector_records, vector_record_points, m_46, m_4a,
|
||||
point3d_records, point3d_points, m_54, point2d_records,
|
||||
m_5a, m_5c, feature_records, m_60, m_64, m_68, m_6a, m_6c,
|
||||
related, m_72, m_76, attr_bytes, m_7c,
|
||||
)
|
||||
|
||||
|
||||
def _text(b: bytes) -> str:
|
||||
# CM93 data in the field is commonly ANSI/Windows text. Preserve bytes separately.
|
||||
return b.decode("cp1252", errors="replace")
|
||||
|
||||
|
||||
def _parse_attributes(
|
||||
block: bytes,
|
||||
nattr: int,
|
||||
attr_defs: dict[int, AttributeDef],
|
||||
) -> tuple[ParsedAttribute, ...]:
|
||||
r = _Reader(block)
|
||||
out: list[ParsedAttribute] = []
|
||||
for ordinal in range(nattr):
|
||||
item_start = r.offset
|
||||
aid = r.u8()
|
||||
d = attr_defs.get(aid)
|
||||
if d is None:
|
||||
raise CM93ParseError(f"unknown attribute id={aid} at attr {ordinal}")
|
||||
t = d.decoder_type
|
||||
value: Any
|
||||
vi: int | None = None
|
||||
vr: float | None = None
|
||||
vt: str | None = None
|
||||
vj: Any | None = None
|
||||
value_type = "UNKNOWN"
|
||||
if t == "B":
|
||||
vi = r.u8(); value = vi; value_type = "BYTE"
|
||||
elif t == "S":
|
||||
raw_s = r.cstring(); vt = _text(raw_s); value = vt; value_type = "STRING"
|
||||
elif t == "R":
|
||||
vr = r.f32(); value = vr; value_type = "REAL"
|
||||
elif t == "W":
|
||||
vi = r.u16(); vr = vi / 10.0; value = vr; value_type = "WORD10"
|
||||
elif t == "G":
|
||||
vi = r.i32(); value = vi; value_type = "LONG"
|
||||
elif t == "C":
|
||||
prefix = list(r.take(3)); raw_s = r.cstring(); vt = _text(raw_s)
|
||||
vj = {"prefix": prefix, "text": vt}; value = vj; value_type = "COMPLEX"
|
||||
elif t == "L":
|
||||
n = r.u8(); arr = list(r.take(n)); vj = arr; value = arr; value_type = "LIST"
|
||||
elif t == "I":
|
||||
# Kept for compatibility with old CM93 decoders; not present in supplied dictionary.
|
||||
vi = r.u16(); value = vi; value_type = "INT"
|
||||
else:
|
||||
raise CM93ParseError(
|
||||
f"unsupported attribute type={d.source_type_name}/{t} id={aid}"
|
||||
)
|
||||
raw_value = block[item_start+1:r.offset]
|
||||
out.append(ParsedAttribute(
|
||||
ordinal=ordinal,
|
||||
attribute_id=aid,
|
||||
name=d.name,
|
||||
source_type=d.source_type_name,
|
||||
decoder_type=t,
|
||||
value_type=value_type,
|
||||
value=value,
|
||||
value_int=vi,
|
||||
value_real=vr,
|
||||
value_text=vt,
|
||||
value_json=vj,
|
||||
raw_value=raw_value,
|
||||
))
|
||||
if r.offset != len(block):
|
||||
raise CM93ParseError(
|
||||
f"attribute block length mismatch: parsed={r.offset} actual={len(block)} nattr={nattr}"
|
||||
)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def parse_cell(
|
||||
path: Path,
|
||||
object_defs: dict[int, ObjectDef],
|
||||
attr_defs: dict[int, AttributeDef],
|
||||
*,
|
||||
keep_source_bytes: bool = True,
|
||||
) -> ParsedCell:
|
||||
encoded = path.read_bytes()
|
||||
decoded = decode_cm93(encoded)
|
||||
r = _Reader(decoded)
|
||||
word0 = r.u16()
|
||||
table1 = r.i32()
|
||||
table2 = r.i32()
|
||||
if word0 + table1 + table2 != len(decoded):
|
||||
raise CM93ParseError(
|
||||
f"cell length check failed: {word0}+{table1}+{table2}!={len(decoded)}"
|
||||
)
|
||||
if word0 != 138:
|
||||
raise CM93ParseError(f"unexpected prolog+header length: {word0}")
|
||||
header = _parse_header(r)
|
||||
table_offsets = {"prolog": 0, "header": 10, "vector": r.offset}
|
||||
|
||||
edges: list[EdgeRecord] = []
|
||||
total_vector_points = 0
|
||||
for i in range(header.vector_records):
|
||||
st = r.offset
|
||||
n = r.u16()
|
||||
pts = tuple((r.u16(), r.u16()) for _ in range(n))
|
||||
total_vector_points += n
|
||||
edges.append(EdgeRecord(i, pts, decoded[st:r.offset]))
|
||||
if total_vector_points != header.vector_record_points:
|
||||
raise CM93ParseError(
|
||||
f"vector point count mismatch: parsed={total_vector_points} header={header.vector_record_points}"
|
||||
)
|
||||
|
||||
table_offsets["point3d"] = r.offset
|
||||
groups: list[SoundingGroup] = []
|
||||
total_3d = 0
|
||||
for i in range(header.point3d_records):
|
||||
st = r.offset
|
||||
n = r.u16()
|
||||
pts = tuple((r.u16(), r.u16(), r.u16()) for _ in range(n))
|
||||
total_3d += n
|
||||
groups.append(SoundingGroup(i, pts, decoded[st:r.offset]))
|
||||
if total_3d != header.point3d_points:
|
||||
raise CM93ParseError(
|
||||
f"3D point count mismatch: parsed={total_3d} header={header.point3d_points}"
|
||||
)
|
||||
|
||||
table_offsets["point2d"] = r.offset
|
||||
point2d = tuple((r.u16(), r.u16()) for _ in range(header.point2d_records))
|
||||
|
||||
table_offsets["feature"] = r.offset
|
||||
features: list[ParsedFeature] = []
|
||||
attr_count = 0
|
||||
total_attr_bytes = 0
|
||||
for i in range(header.feature_records):
|
||||
st = r.offset
|
||||
object_type = r.u8()
|
||||
geotype = r.u8()
|
||||
record_len = r.u16()
|
||||
if record_len < 4:
|
||||
raise CM93ParseError(f"feature {i}: record length {record_len} < 4")
|
||||
record_end = st + record_len
|
||||
if record_end > len(decoded):
|
||||
raise CM93ParseError(f"feature {i}: record beyond EOF")
|
||||
primitive = geotype & 0x0F
|
||||
refs: list[GeometryRef] = []
|
||||
p2idx = None
|
||||
p3idx = None
|
||||
if primitive in (2, 4):
|
||||
n = r.u16()
|
||||
for _ in range(n):
|
||||
raw_idx = r.u16()
|
||||
edge_idx = raw_idx & 0x1FFF
|
||||
usage = raw_idx >> 13
|
||||
if edge_idx >= len(edges):
|
||||
raise CM93ParseError(f"feature {i}: edge index {edge_idx} out of range")
|
||||
refs.append(GeometryRef(edge_idx, usage))
|
||||
elif primitive == 1:
|
||||
p2idx = r.u16()
|
||||
if p2idx >= len(point2d):
|
||||
raise CM93ParseError(f"feature {i}: point2d index {p2idx} out of range")
|
||||
elif primitive == 8:
|
||||
p3idx = r.u16()
|
||||
if p3idx >= len(groups):
|
||||
raise CM93ParseError(f"feature {i}: point3d index {p3idx} out of range")
|
||||
elif primitive == 0:
|
||||
pass
|
||||
else:
|
||||
# Unknown low primitive cannot be skipped safely unless record boundary is honored.
|
||||
# We preserve the record and mark geometry unknown; remaining bytes are consumed below.
|
||||
pass
|
||||
|
||||
related: list[int] = []
|
||||
related_hint: int | None = None
|
||||
if geotype & 0x10:
|
||||
nr = r.u8()
|
||||
for _ in range(nr):
|
||||
idx = r.u16()
|
||||
if idx >= header.feature_records:
|
||||
raise CM93ParseError(f"feature {i}: related index {idx} out of range")
|
||||
related.append(idx)
|
||||
if geotype & 0x20:
|
||||
related_hint = r.u16() & 0xFF
|
||||
|
||||
nattr = 0
|
||||
attrs: tuple[ParsedAttribute, ...] = ()
|
||||
attr_block = b""
|
||||
if geotype & 0x80:
|
||||
nattr = r.u8()
|
||||
attr_len = record_end - r.offset
|
||||
if attr_len < 0:
|
||||
raise CM93ParseError(f"feature {i}: negative attribute length")
|
||||
attr_block = r.take(attr_len)
|
||||
attrs = _parse_attributes(attr_block, nattr, attr_defs)
|
||||
attr_count += nattr
|
||||
total_attr_bytes += len(attr_block)
|
||||
elif r.offset < record_end:
|
||||
# Known flags consumed fewer bytes than descriptor length. Preserve but do not guess.
|
||||
r.offset = record_end
|
||||
|
||||
if r.offset != record_end:
|
||||
raise CM93ParseError(
|
||||
f"feature {i}: record boundary mismatch parsed={r.offset-st} expected={record_len}"
|
||||
)
|
||||
obj = object_defs.get(object_type)
|
||||
object_class = obj.name if obj else f"UNKNOWN_{object_type}"
|
||||
features.append(ParsedFeature(
|
||||
record_index=i,
|
||||
object_type=object_type,
|
||||
object_class=object_class,
|
||||
raw_geotype=geotype,
|
||||
primitive=primitive,
|
||||
geometry_refs=tuple(refs),
|
||||
point2d_index=p2idx,
|
||||
point3d_index=p3idx,
|
||||
related_indices=tuple(related),
|
||||
related_count_hint=related_hint,
|
||||
n_attributes=nattr,
|
||||
attributes=attrs,
|
||||
raw_record=decoded[st:record_end],
|
||||
raw_attribute_block=attr_block,
|
||||
))
|
||||
|
||||
if total_attr_bytes != header.attribute_bytes:
|
||||
raise CM93ParseError(
|
||||
f"attribute byte count mismatch: parsed={total_attr_bytes} header={header.attribute_bytes}"
|
||||
)
|
||||
if r.offset != len(decoded):
|
||||
raise CM93ParseError(f"cell end mismatch: parsed={r.offset} file={len(decoded)}")
|
||||
|
||||
sha = hashlib.sha256(encoded).hexdigest()
|
||||
return ParsedCell(
|
||||
path=path,
|
||||
source_size=len(encoded),
|
||||
sha256=sha,
|
||||
encoded_bytes=encoded if keep_source_bytes else b"",
|
||||
decoded_bytes=decoded if keep_source_bytes else b"",
|
||||
prolog_word0=word0,
|
||||
table1_length=table1,
|
||||
table2_length=table2,
|
||||
header=header,
|
||||
edges=tuple(edges),
|
||||
point2d=point2d,
|
||||
point3d=tuple(groups),
|
||||
features=tuple(features),
|
||||
end_offset=r.offset,
|
||||
table_offsets=table_offsets,
|
||||
attribute_count=attr_count,
|
||||
)
|
||||
203
src/cm93db/cli.py
Normal file
203
src/cm93db/cli.py
Normal file
@@ -0,0 +1,203 @@
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import platform
|
||||
import sys
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from .config import load_config, load_source_dir
|
||||
from .database import (
|
||||
connect,
|
||||
server_version,
|
||||
assert_postgresql_compatible,
|
||||
postgis_version,
|
||||
create_database,
|
||||
apply_schema,
|
||||
)
|
||||
from .scanner import scan_cm93
|
||||
from .dictionaries import load_object_dictionary, load_attribute_dictionary
|
||||
from .schema_contract import REQUIRED_TABLES
|
||||
from .importer import ImportOptions, run_import
|
||||
from .audit import run_audit, print_result
|
||||
|
||||
|
||||
def project_root() -> Path:
|
||||
return Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _root(arg: str | None) -> Path:
|
||||
return Path(arg) if arg else load_source_dir()
|
||||
|
||||
|
||||
def _scales(values: list[str] | None) -> set[str] | None:
|
||||
if not values:
|
||||
return None
|
||||
out=set()
|
||||
for v in values:
|
||||
for ch in v.upper().replace(",",""):
|
||||
if ch.strip(): out.add(ch)
|
||||
bad=out-set("ZABCDEFG")
|
||||
if bad:
|
||||
raise SystemExit(f"无效比例尺: {sorted(bad)}")
|
||||
return out
|
||||
|
||||
|
||||
def cmd_env_check(_args):
|
||||
print(f"Python: {sys.version.split()[0]}")
|
||||
if sys.version_info[:2] != (3, 12):
|
||||
raise SystemExit("ERROR: 本项目固定 Python 3.12.x;推荐 3.12.10")
|
||||
print(f"Platform: {platform.platform()}")
|
||||
cfg = load_config()
|
||||
conn = connect(cfg)
|
||||
try:
|
||||
v = server_version(conn)
|
||||
assert_postgresql_compatible(v)
|
||||
print(f"PostgreSQL: {v}")
|
||||
print(f"PostGIS: {postgis_version(conn)}")
|
||||
print(f"Database: {cfg.database}")
|
||||
print("PostgreSQL/PostGIS compatibility: OK")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def cmd_db_init(args):
|
||||
cfg = load_config()
|
||||
if args.create_database:
|
||||
create_database(cfg)
|
||||
print(f"Database ensured: {cfg.database}")
|
||||
schema = project_root() / "sql" / "001_schema_postgresql.sql"
|
||||
apply_schema(cfg, schema)
|
||||
print(f"Schema initialized: {schema.name}")
|
||||
|
||||
|
||||
def cmd_db_check(_args):
|
||||
cfg = load_config()
|
||||
conn = connect(cfg)
|
||||
try:
|
||||
v = server_version(conn)
|
||||
assert_postgresql_compatible(v)
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"SELECT table_name FROM information_schema.tables "
|
||||
"WHERE table_schema='public' AND table_type='BASE TABLE'"
|
||||
)
|
||||
tables = {row["table_name"] for row in cur.fetchall()}
|
||||
missing = sorted(REQUIRED_TABLES - tables)
|
||||
present = REQUIRED_TABLES & tables
|
||||
print(f"PostgreSQL: {v}")
|
||||
print(f"PostGIS: {postgis_version(conn)}")
|
||||
print(f"CM95 tables: {len(present)}/{len(REQUIRED_TABLES)}")
|
||||
if missing:
|
||||
print("Missing:")
|
||||
for t in missing: print(f" - {t}")
|
||||
raise SystemExit(2)
|
||||
print("Schema contract: OK")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def cmd_scan(args):
|
||||
result = scan_cm93(_root(args.root))
|
||||
by_scale = {}
|
||||
for c in result.cells: by_scale[c.scale_code] = by_scale.get(c.scale_code, 0) + 1
|
||||
print(f"Root: {result.root}")
|
||||
print(f"Cells: {len(result.cells)}")
|
||||
for scale in sorted(by_scale): print(f" {scale}: {by_scale[scale]}")
|
||||
print(f"CM93OBJ.DIC: {result.object_dictionary or 'NOT FOUND'}")
|
||||
print(f"ATTR dictionary: {result.attribute_dictionary or 'NOT FOUND'}")
|
||||
|
||||
|
||||
def cmd_inspect_dicts(args):
|
||||
result = scan_cm93(_root(args.root))
|
||||
if not result.object_dictionary: raise SystemExit("CM93OBJ.DIC 未找到")
|
||||
if not result.attribute_dictionary: raise SystemExit("CM93ATTR.DIC/ATTRLUT.DIC 未找到")
|
||||
objects = load_object_dictionary(result.object_dictionary)
|
||||
attrs = load_attribute_dictionary(result.attribute_dictionary)
|
||||
print(f"Object dictionary: {len(objects)} records")
|
||||
print(f"Attribute dictionary: {len(attrs)} records")
|
||||
for key in ("DEPARE", "DEPCNT", "LIGHTS", "SOUNDG", "WRECKS", "_m_sor"):
|
||||
item = next((x for x in objects if x.name == key), None)
|
||||
print(f" {key}: {item.object_id if item else 'missing'}")
|
||||
for key in ("DRVAL1", "VALSOU", "_wgsox", "_wgsoy"):
|
||||
item = next((x for x in attrs if x.name == key), None)
|
||||
print(f" {key}: {item.attribute_id if item else 'missing'} / {item.source_type_name if item else '-'}")
|
||||
|
||||
|
||||
def cmd_import(args):
|
||||
cfg=load_config(); root=_root(args.root)
|
||||
opt=ImportOptions(root=root,limit=args.limit,scales=_scales(args.scale),cell_name=args.cell,
|
||||
reimport=args.reimport,retry_failed=not args.skip_failed,fail_fast=args.fail_fast,
|
||||
store_raw_tables=not args.no_raw_tables,shard_count=args.shard_count,
|
||||
shard_index=args.shard_index)
|
||||
s=run_import(cfg,opt)
|
||||
print("\n=== IMPORT SUMMARY ===")
|
||||
for k in ("cells_discovered","cells_done","cells_skipped","cells_failed","features_source","features_imported","attributes_source","attributes_imported","unknown_object_count","unknown_attribute_count","geometry_issue_count","sounding_points"):
|
||||
print(f"{k:24}: {getattr(s,k)}")
|
||||
if s.cells_failed or s.unknown_object_count or s.unknown_attribute_count:
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def cmd_audit(args):
|
||||
cfg=load_config() if not args.source_only else load_config()
|
||||
r=run_audit(cfg,_root(args.root),limit=args.limit,scales=_scales(args.scale),cell_name=args.cell,
|
||||
precision_samples=args.precision_samples,tolerance_deg=args.tolerance,
|
||||
validate_geometry=args.validate_geometry,source_only=args.source_only,require_raw=not args.allow_missing_raw)
|
||||
print_result(r)
|
||||
report = Path(args.report) if args.report else (project_root() / "logs" / f"cm93_audit_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json")
|
||||
report.parent.mkdir(parents=True,exist_ok=True)
|
||||
report.write_text(json.dumps(r.as_dict(),ensure_ascii=False,indent=2),encoding="utf-8")
|
||||
print(f"Audit report: {report}")
|
||||
if not r.passed: raise SystemExit(2)
|
||||
|
||||
|
||||
def _add_source_args(s, optional=True):
|
||||
s.add_argument(
|
||||
"root",
|
||||
nargs="?" if optional else None,
|
||||
help="CM95/CM93 Japan 根目录;省略时读取 .env 的 CM95_SOURCE_DIR",
|
||||
)
|
||||
|
||||
|
||||
def build_parser():
|
||||
p = argparse.ArgumentParser(prog="cm95pg")
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
s = sub.add_parser("env-check"); s.set_defaults(func=cmd_env_check)
|
||||
s = sub.add_parser("db-init"); s.add_argument("--create-database", action="store_true"); s.set_defaults(func=cmd_db_init)
|
||||
s = sub.add_parser("db-check"); s.set_defaults(func=cmd_db_check)
|
||||
s = sub.add_parser("scan"); _add_source_args(s); s.set_defaults(func=cmd_scan)
|
||||
s = sub.add_parser("inspect-dicts"); _add_source_args(s); s.set_defaults(func=cmd_inspect_dicts)
|
||||
|
||||
s = sub.add_parser("import", help="完整解析 CM95/CM93 Japan 并逐 cell 导入 PostgreSQL/PostGIS")
|
||||
_add_source_args(s)
|
||||
s.add_argument("--limit", type=int)
|
||||
s.add_argument("--scale", action="append", help="限定比例尺,例如 --scale G 或 --scale F,G")
|
||||
s.add_argument("--cell", help="只导入指定 cell 文件名/不带扩展名")
|
||||
s.add_argument("--reimport", action="store_true", help="重新导入已 DONE cell")
|
||||
s.add_argument("--skip-failed", action="store_true", help="不重试之前 FAILED 的 cell")
|
||||
s.add_argument("--fail-fast", action="store_true")
|
||||
s.add_argument("--no-raw-tables", action="store_true", help="不保存 decoded 原始 table;不建议正式母库使用")
|
||||
s.add_argument("--shard-count", type=int, help="可选并行分片总数;默认不分片")
|
||||
s.add_argument("--shard-index", type=int, help="当前分片编号,从 0 开始;需与 --shard-count 配合")
|
||||
s.set_defaults(func=cmd_import)
|
||||
|
||||
s = sub.add_parser("audit", help="重新解析源 CM93 并与数据库逐 cell 审计")
|
||||
_add_source_args(s)
|
||||
s.add_argument("--limit", type=int)
|
||||
s.add_argument("--scale", action="append")
|
||||
s.add_argument("--cell")
|
||||
s.add_argument("--precision-samples", type=int, default=128, help="数据库 WKB 双精度回读样本数")
|
||||
s.add_argument("--tolerance", type=float, default=1e-12, help="经纬度精度比较容差(degree)")
|
||||
s.add_argument("--validate-geometry", action="store_true", help="额外执行 Shapely validity 全量检查,较慢")
|
||||
s.add_argument("--source-only", action="store_true", help="只审计源文件解析,不查询数据库")
|
||||
s.add_argument("--allow-missing-raw", action="store_true", help="不把 raw table 未保存视为审计失败")
|
||||
s.add_argument("--report", help="JSON 审计报告路径;默认写入项目 logs/")
|
||||
s.set_defaults(func=cmd_audit)
|
||||
return p
|
||||
|
||||
|
||||
def main():
|
||||
args = build_parser().parse_args()
|
||||
args.func(args)
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
478
src/cm93db/cm93_binary.py
Normal file
478
src/cm93db/cm93_binary.py
Normal file
@@ -0,0 +1,478 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
import hashlib
|
||||
import math
|
||||
import struct
|
||||
from typing import Any
|
||||
|
||||
from .dictionaries import AttributeDef, ObjectDef
|
||||
from .precision import CM93_SEMIMAJOR_M
|
||||
|
||||
TABLE_0 = bytes([
|
||||
0xCD,0xEA,0xDC,0x48,0x3E,0x6D,0xCA,0x7B,0x52,0xE1,0xA4,0x8E,0xAB,0x05,0xA7,0x97,
|
||||
0xB9,0x60,0x39,0x85,0x7C,0x56,0x7A,0xBA,0x68,0x6E,0xF5,0x5D,0x02,0x4E,0x0F,0xA1,
|
||||
0x27,0x24,0x41,0x34,0x00,0x5A,0xFE,0xCB,0xD0,0xFA,0xF8,0x6C,0x74,0x96,0x9E,0x0E,
|
||||
0xC2,0x49,0xE3,0xE5,0xC0,0x3B,0x59,0x18,0xA9,0x86,0x8F,0x30,0xC3,0xA8,0x22,0x0A,
|
||||
0x14,0x1A,0xB2,0xC9,0xC7,0xED,0xAA,0x29,0x94,0x75,0x0D,0xAC,0x0C,0xF4,0xBB,0xC5,
|
||||
0x3F,0xFD,0xD9,0x9C,0x4F,0xD5,0x84,0x1E,0xB1,0x81,0x69,0xB4,0x09,0xB8,0x3C,0xAF,
|
||||
0xA3,0x08,0xBF,0xE0,0x9A,0xD7,0xF7,0x8C,0x67,0x66,0xAE,0xD4,0x4C,0xA5,0xEC,0xF9,
|
||||
0xB6,0x64,0x78,0x06,0x5B,0x9B,0xF2,0x99,0xCE,0xDB,0x53,0x55,0x65,0x8D,0x07,0x33,
|
||||
0x04,0x37,0x92,0x26,0x23,0xB5,0x58,0xDA,0x2F,0xB3,0x40,0x5E,0x7F,0x4B,0x62,0x80,
|
||||
0xE4,0x6F,0x73,0x1D,0xDF,0x17,0xCC,0x28,0x25,0x2D,0xEE,0x3A,0x98,0xE2,0x01,0xEB,
|
||||
0xDD,0xBC,0x90,0xB0,0xFC,0x95,0x76,0x93,0x46,0x57,0x2C,0x2B,0x50,0x11,0x0B,0xC1,
|
||||
0xF0,0xE7,0xD6,0x21,0x31,0xDE,0xFF,0xD8,0x12,0xA6,0x4D,0x8A,0x13,0x43,0x45,0x38,
|
||||
0xD2,0x87,0xA0,0xEF,0x82,0xF1,0x47,0x89,0x6A,0xC8,0x54,0x1B,0x16,0x7E,0x79,0xBD,
|
||||
0x6B,0x91,0xA2,0x71,0x36,0xB7,0x03,0x3D,0x72,0xC6,0x44,0x8B,0xCF,0x15,0x9F,0x32,
|
||||
0xC4,0x77,0x83,0x63,0x20,0x88,0xF6,0xAD,0xF3,0xE8,0x4A,0xE9,0x35,0x1C,0x5F,0x19,
|
||||
0x1F,0x7D,0x70,0xFB,0xD1,0x51,0x10,0xD3,0x2E,0x61,0x9D,0x5C,0x2A,0x42,0xBE,0xE6,
|
||||
])
|
||||
|
||||
_encode = bytes((v ^ 8) for v in TABLE_0)
|
||||
_decode_arr = bytearray(256)
|
||||
for i, v in enumerate(_encode):
|
||||
_decode_arr[v] = i
|
||||
DECODE_TABLE = bytes(_decode_arr)
|
||||
|
||||
|
||||
def decode_bytes(data: bytes) -> bytes:
|
||||
return data.translate(DECODE_TABLE)
|
||||
|
||||
|
||||
class CM93ParseError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Header:
|
||||
lon_min: float
|
||||
lat_min: float
|
||||
lon_max: float
|
||||
lat_max: float
|
||||
easting_min: float
|
||||
northing_min: float
|
||||
easting_max: float
|
||||
northing_max: float
|
||||
usn_vector_records: int
|
||||
n_vector_record_points: int
|
||||
m_46: int
|
||||
m_4a: int
|
||||
usn_point3d_records: int
|
||||
m_50: int
|
||||
m_54: int
|
||||
usn_point2d_records: int
|
||||
m_5a: int
|
||||
m_5c: int
|
||||
usn_feature_records: int
|
||||
m_60: int
|
||||
m_64: int
|
||||
m_68: int
|
||||
m_6a: int
|
||||
m_6c: int
|
||||
m_nrelated_object_pointers: int
|
||||
m_72: int
|
||||
m_76: int
|
||||
m_78: int
|
||||
m_7c: int
|
||||
|
||||
@property
|
||||
def x_rate(self) -> float:
|
||||
delta = self.easting_max - self.easting_min
|
||||
if delta < 0:
|
||||
delta += CM93_SEMIMAJOR_M * 2.0 * math.pi
|
||||
return delta / 65535.0
|
||||
|
||||
@property
|
||||
def y_rate(self) -> float:
|
||||
return (self.northing_max - self.northing_min) / 65535.0
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {k: getattr(self, k) for k in self.__dataclass_fields__}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Edge:
|
||||
index: int
|
||||
points: tuple[tuple[int, int], ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Point3DDescriptor:
|
||||
index: int
|
||||
points: tuple[tuple[int, int, int], ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParsedAttribute:
|
||||
ordinal: int
|
||||
attribute_id: int
|
||||
name: str
|
||||
decoder_type: str
|
||||
source_type_name: str
|
||||
value: Any
|
||||
raw_value: bytes
|
||||
raw_entry: bytes
|
||||
|
||||
|
||||
@dataclass
|
||||
class FeatureRecord:
|
||||
index: int
|
||||
object_type: int
|
||||
object_class: str
|
||||
geotype: int
|
||||
geometry_primitive: int
|
||||
geometry_refs: list[tuple[int, int]] = field(default_factory=list) # (edge index, usage)
|
||||
point2d_index: int | None = None
|
||||
point3d_index: int | None = None
|
||||
related_indices: list[int] = field(default_factory=list)
|
||||
relation20_raw: int | None = None
|
||||
attributes: list[ParsedAttribute] = field(default_factory=list)
|
||||
raw_record: bytes = b""
|
||||
raw_attribute_blob: bytes = b""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RawTables:
|
||||
prolog: bytes
|
||||
header: bytes
|
||||
vectors: bytes
|
||||
point3d: bytes
|
||||
point2d: bytes
|
||||
features: bytes
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedCell:
|
||||
path: Path
|
||||
file_size: int
|
||||
sha256: str
|
||||
word0: int
|
||||
table1_length: int
|
||||
table2_length: int
|
||||
header: Header
|
||||
edges: list[Edge]
|
||||
point3d: list[Point3DDescriptor]
|
||||
point2d: list[tuple[int, int]]
|
||||
features: list[FeatureRecord]
|
||||
raw_tables: RawTables
|
||||
unknown_object_ids: set[int]
|
||||
unknown_attribute_ids: set[int]
|
||||
attribute_count: int
|
||||
|
||||
|
||||
class Cursor:
|
||||
def __init__(self, data: bytes, pos: int = 0):
|
||||
self.data = data
|
||||
self.pos = pos
|
||||
|
||||
def remaining(self) -> int:
|
||||
return len(self.data) - self.pos
|
||||
|
||||
def take(self, n: int) -> bytes:
|
||||
if n < 0 or self.pos + n > len(self.data):
|
||||
raise CM93ParseError(f"越界读取: pos={self.pos}, n={n}, size={len(self.data)}")
|
||||
out = self.data[self.pos:self.pos+n]
|
||||
self.pos += n
|
||||
return out
|
||||
|
||||
def u8(self) -> int:
|
||||
return self.take(1)[0]
|
||||
|
||||
def u16(self) -> int:
|
||||
return struct.unpack("<H", self.take(2))[0]
|
||||
|
||||
def i32(self) -> int:
|
||||
return struct.unpack("<i", self.take(4))[0]
|
||||
|
||||
def f32(self) -> float:
|
||||
return struct.unpack("<f", self.take(4))[0]
|
||||
|
||||
def f64(self) -> float:
|
||||
return struct.unpack("<d", self.take(8))[0]
|
||||
|
||||
|
||||
HEADER_FMT = "<8dHiiiHiiHHHHiiHHH i H ii".replace(" ", "")
|
||||
# Explicit parsing is used instead of struct.unpack on the full header so that
|
||||
# field order remains auditable against the documented 128-byte CM93 sequence.
|
||||
|
||||
|
||||
def parse_header(decoded_header: bytes) -> Header:
|
||||
if len(decoded_header) != 128:
|
||||
raise CM93ParseError(f"CM93 header 长度应为 128,实际 {len(decoded_header)}")
|
||||
c = Cursor(decoded_header)
|
||||
vals = [c.f64() for _ in range(8)]
|
||||
h = Header(
|
||||
lon_min=vals[0], lat_min=vals[1], lon_max=vals[2], lat_max=vals[3],
|
||||
easting_min=vals[4], northing_min=vals[5], easting_max=vals[6], northing_max=vals[7],
|
||||
usn_vector_records=c.u16(),
|
||||
n_vector_record_points=c.i32(),
|
||||
m_46=c.i32(),
|
||||
m_4a=c.i32(),
|
||||
usn_point3d_records=c.u16(),
|
||||
m_50=c.i32(),
|
||||
m_54=c.i32(),
|
||||
usn_point2d_records=c.u16(),
|
||||
m_5a=c.u16(),
|
||||
m_5c=c.u16(),
|
||||
usn_feature_records=c.u16(),
|
||||
m_60=c.i32(),
|
||||
m_64=c.i32(),
|
||||
m_68=c.u16(),
|
||||
m_6a=c.u16(),
|
||||
m_6c=c.u16(),
|
||||
m_nrelated_object_pointers=c.i32(),
|
||||
m_72=c.i32(),
|
||||
m_76=c.u16(),
|
||||
m_78=c.i32(),
|
||||
m_7c=c.i32(),
|
||||
)
|
||||
if c.remaining() != 0:
|
||||
raise CM93ParseError(f"header 未完全消费: {c.remaining()} bytes")
|
||||
return h
|
||||
|
||||
|
||||
def _decode_text(raw: bytes) -> str:
|
||||
# Byte-preserving fallback is intentional. raw_value is stored separately.
|
||||
for enc in ("utf-8", "cp1252", "latin1"):
|
||||
try:
|
||||
return raw.decode(enc)
|
||||
except UnicodeDecodeError:
|
||||
pass
|
||||
return raw.decode("latin1")
|
||||
|
||||
|
||||
def _parse_attr_entry(c: Cursor, ordinal: int, attr_defs: dict[int, AttributeDef]) -> tuple[ParsedAttribute, bool]:
|
||||
start = c.pos
|
||||
aid = c.u8()
|
||||
d = attr_defs.get(aid)
|
||||
unknown = d is None
|
||||
if d is None:
|
||||
raise CM93ParseError(f"未知 attribute id={aid},无法确定值长度")
|
||||
vstart = c.pos
|
||||
t = d.decoder_type
|
||||
if t == "B":
|
||||
value = c.u8()
|
||||
elif t == "W":
|
||||
value = c.u16() / 10.0
|
||||
elif t == "G":
|
||||
value = c.i32()
|
||||
elif t == "R":
|
||||
value = c.f32()
|
||||
elif t == "S":
|
||||
z = c.data.find(b"\x00", c.pos)
|
||||
if z < 0:
|
||||
raise CM93ParseError(f"字符串属性 {d.name} 缺少 NUL 终止符")
|
||||
raw = c.data[c.pos:z]
|
||||
c.pos = z + 1
|
||||
value = _decode_text(raw)
|
||||
elif t == "C":
|
||||
prefix = list(c.take(3))
|
||||
z = c.data.find(b"\x00", c.pos)
|
||||
if z < 0:
|
||||
raise CM93ParseError(f"复杂属性 {d.name} 缺少 NUL 终止符")
|
||||
raw = c.data[c.pos:z]
|
||||
c.pos = z + 1
|
||||
value = {"prefix": prefix, "text": _decode_text(raw)}
|
||||
elif t == "L":
|
||||
n = c.u8()
|
||||
value = list(c.take(n))
|
||||
elif t == "I":
|
||||
value = c.u16()
|
||||
else:
|
||||
raise CM93ParseError(f"属性 {d.name}({aid}) 类型 {d.source_type_name}/{t} 无法解码")
|
||||
vend = c.pos
|
||||
return ParsedAttribute(
|
||||
ordinal=ordinal,
|
||||
attribute_id=aid,
|
||||
name=d.name,
|
||||
decoder_type=t,
|
||||
source_type_name=d.source_type_name,
|
||||
value=value,
|
||||
raw_value=c.data[vstart:vend],
|
||||
raw_entry=c.data[start:vend],
|
||||
), unknown
|
||||
|
||||
|
||||
def parse_cell(path: Path, object_defs: dict[int, ObjectDef], attr_defs: dict[int, AttributeDef]) -> ParsedCell:
|
||||
encoded = path.read_bytes()
|
||||
decoded = decode_bytes(encoded)
|
||||
if len(decoded) < 138:
|
||||
raise CM93ParseError(f"文件过短: {path}")
|
||||
c = Cursor(decoded)
|
||||
prolog_start = c.pos
|
||||
word0 = c.u16()
|
||||
table1 = c.i32()
|
||||
table2 = c.i32()
|
||||
prolog = decoded[prolog_start:c.pos]
|
||||
if word0 != 138:
|
||||
raise CM93ParseError(f"prolog/header 长度异常: word0={word0}, expected=138")
|
||||
if word0 + table1 + table2 != len(decoded):
|
||||
raise CM93ParseError(
|
||||
f"文件完整性失败: {word0}+{table1}+{table2}!={len(decoded)}"
|
||||
)
|
||||
|
||||
header_start = c.pos
|
||||
header = parse_header(c.take(128))
|
||||
raw_header = decoded[header_start:c.pos]
|
||||
|
||||
vector_start = c.pos
|
||||
edges: list[Edge] = []
|
||||
total_vector_points = 0
|
||||
for i in range(header.usn_vector_records):
|
||||
n = c.u16()
|
||||
pts = tuple((c.u16(), c.u16()) for _ in range(n))
|
||||
total_vector_points += n
|
||||
edges.append(Edge(i, pts))
|
||||
vector_end = c.pos
|
||||
if total_vector_points != header.n_vector_record_points:
|
||||
raise CM93ParseError(
|
||||
f"vector point count mismatch: header={header.n_vector_record_points}, parsed={total_vector_points}"
|
||||
)
|
||||
|
||||
p3_start = c.pos
|
||||
point3d: list[Point3DDescriptor] = []
|
||||
total_p3_points = 0
|
||||
for i in range(header.usn_point3d_records):
|
||||
n = c.u16()
|
||||
pts = tuple((c.u16(), c.u16(), c.u16()) for _ in range(n))
|
||||
total_p3_points += n
|
||||
point3d.append(Point3DDescriptor(i, pts))
|
||||
p3_end = c.pos
|
||||
if header.m_50 >= 0 and total_p3_points != header.m_50:
|
||||
raise CM93ParseError(f"3D point count mismatch: header={header.m_50}, parsed={total_p3_points}")
|
||||
|
||||
p2_start = c.pos
|
||||
point2d = [(c.u16(), c.u16()) for _ in range(header.usn_point2d_records)]
|
||||
p2_end = c.pos
|
||||
|
||||
feature_start = c.pos
|
||||
features: list[FeatureRecord] = []
|
||||
unknown_obj: set[int] = set()
|
||||
unknown_attr: set[int] = set()
|
||||
attr_total = 0
|
||||
|
||||
for fi in range(header.usn_feature_records):
|
||||
rec_start = c.pos
|
||||
object_type = c.u8()
|
||||
geotype = c.u8()
|
||||
desc_len = c.u16()
|
||||
if desc_len < 4:
|
||||
raise CM93ParseError(f"feature {fi}: obj_desc_bytes={desc_len}<4")
|
||||
rec_end = rec_start + desc_len
|
||||
if rec_end > len(decoded):
|
||||
raise CM93ParseError(f"feature {fi}: record 越界")
|
||||
od = object_defs.get(object_type)
|
||||
if od is None:
|
||||
unknown_obj.add(object_type)
|
||||
object_class = f"UNKNOWN_{object_type}"
|
||||
else:
|
||||
object_class = od.name
|
||||
prim = geotype & 0x0F
|
||||
f = FeatureRecord(fi, object_type, object_class, geotype, prim)
|
||||
|
||||
if prim in (2, 4):
|
||||
n = c.u16()
|
||||
for _ in range(n):
|
||||
packed = c.u16()
|
||||
edge_idx = packed & 0x1FFF
|
||||
usage = packed >> 13
|
||||
if edge_idx >= len(edges):
|
||||
raise CM93ParseError(f"feature {fi}: edge index {edge_idx} >= {len(edges)}")
|
||||
f.geometry_refs.append((edge_idx, usage))
|
||||
elif prim == 1:
|
||||
idx = c.u16()
|
||||
if idx >= len(point2d):
|
||||
raise CM93ParseError(f"feature {fi}: point2d index {idx} >= {len(point2d)}")
|
||||
f.point2d_index = idx
|
||||
elif prim == 8:
|
||||
idx = c.u16()
|
||||
if idx >= len(point3d):
|
||||
raise CM93ParseError(f"feature {fi}: point3d index {idx} >= {len(point3d)}")
|
||||
f.point3d_index = idx
|
||||
|
||||
if geotype & 0x10:
|
||||
nrel = c.u8()
|
||||
f.related_indices = [c.u16() for _ in range(nrel)]
|
||||
for rel in f.related_indices:
|
||||
if rel >= header.usn_feature_records:
|
||||
raise CM93ParseError(f"feature {fi}: related index {rel} 越界")
|
||||
if geotype & 0x20:
|
||||
f.relation20_raw = c.u16()
|
||||
# 0x40 is observed as a flag without a documented payload in OpenCPN.
|
||||
|
||||
if geotype & 0x80:
|
||||
nattr = c.u8()
|
||||
attr_blob_start = c.pos
|
||||
# By format definition obj_desc_bytes is the total record length.
|
||||
# Attribute payload is the remaining bytes in this record.
|
||||
attr_blob = decoded[attr_blob_start:rec_end]
|
||||
ac = Cursor(attr_blob)
|
||||
attrs: list[ParsedAttribute] = []
|
||||
for ordinal in range(nattr):
|
||||
try:
|
||||
a, was_unknown = _parse_attr_entry(ac, ordinal, attr_defs)
|
||||
except CM93ParseError as e:
|
||||
# Capture the leading id when possible for audit diagnostics.
|
||||
if ac.pos < len(attr_blob):
|
||||
unknown_attr.add(attr_blob[ac.pos])
|
||||
raise CM93ParseError(f"feature {fi} attribute {ordinal}: {e}") from e
|
||||
if was_unknown:
|
||||
unknown_attr.add(a.attribute_id)
|
||||
attrs.append(a)
|
||||
if ac.remaining() != 0:
|
||||
raise CM93ParseError(
|
||||
f"feature {fi}: attribute blob 剩余 {ac.remaining()} bytes; nattr={nattr}"
|
||||
)
|
||||
f.attributes = attrs
|
||||
f.raw_attribute_blob = attr_blob
|
||||
attr_total += len(attrs)
|
||||
c.pos = rec_end
|
||||
else:
|
||||
# Preserve any unparsed tail, but do not silently cross the record boundary.
|
||||
if c.pos > rec_end:
|
||||
raise CM93ParseError(f"feature {fi}: 已解析字段超过 record 长度")
|
||||
c.pos = rec_end
|
||||
|
||||
f.raw_record = decoded[rec_start:rec_end]
|
||||
features.append(f)
|
||||
|
||||
feature_end = c.pos
|
||||
if feature_end != len(decoded):
|
||||
raise CM93ParseError(f"文件尾不匹配: parsed={feature_end}, size={len(decoded)}, remaining={len(decoded)-feature_end}")
|
||||
parsed_attr_bytes = sum(len(f.raw_attribute_blob) for f in features)
|
||||
if parsed_attr_bytes != header.m_78:
|
||||
raise CM93ParseError(f"attribute byte count mismatch: header={header.m_78}, parsed={parsed_attr_bytes}")
|
||||
parsed_geom_refs = sum(len(f.geometry_refs) for f in features)
|
||||
if parsed_geom_refs != header.m_46 + header.m_4a:
|
||||
raise CM93ParseError(f"geometry ref count mismatch: header={header.m_46 + header.m_4a}, parsed={parsed_geom_refs}")
|
||||
parsed_related = sum(len(f.related_indices) for f in features)
|
||||
if parsed_related != header.m_nrelated_object_pointers:
|
||||
raise CM93ParseError(f"related pointer count mismatch: header={header.m_nrelated_object_pointers}, parsed={parsed_related}")
|
||||
|
||||
# Table-1/Table-2 lengths are retained and audited at file level. The internal
|
||||
# semantic subtable boundaries are determined from header record counts.
|
||||
return ParsedCell(
|
||||
path=path,
|
||||
file_size=len(encoded),
|
||||
sha256=hashlib.sha256(encoded).hexdigest(),
|
||||
word0=word0,
|
||||
table1_length=table1,
|
||||
table2_length=table2,
|
||||
header=header,
|
||||
edges=edges,
|
||||
point3d=point3d,
|
||||
point2d=point2d,
|
||||
features=features,
|
||||
raw_tables=RawTables(
|
||||
prolog=prolog,
|
||||
header=raw_header,
|
||||
vectors=decoded[vector_start:vector_end],
|
||||
point3d=decoded[p3_start:p3_end],
|
||||
point2d=decoded[p2_start:p2_end],
|
||||
features=decoded[feature_start:feature_end],
|
||||
),
|
||||
unknown_object_ids=unknown_obj,
|
||||
unknown_attribute_ids=unknown_attr,
|
||||
attribute_count=attr_total,
|
||||
)
|
||||
39
src/cm93db/config.py
Normal file
39
src/cm93db/config.py
Normal file
@@ -0,0 +1,39 @@
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DbConfig:
|
||||
host: str
|
||||
port: int
|
||||
database: str
|
||||
user: str
|
||||
password: str
|
||||
sslmode: str = "prefer"
|
||||
admin_database: str = "postgres"
|
||||
connect_timeout: int = 20
|
||||
|
||||
def load_config(require_database: bool = True) -> DbConfig:
|
||||
load_dotenv()
|
||||
database = os.getenv("CM95_PG_DATABASE", "cm95_chart").strip()
|
||||
if require_database and not database:
|
||||
raise RuntimeError("CM95_PG_DATABASE 未配置")
|
||||
return DbConfig(
|
||||
host=os.getenv("CM95_PG_HOST", "127.0.0.1").strip(),
|
||||
port=int(os.getenv("CM95_PG_PORT", "5432")),
|
||||
database=database,
|
||||
user=os.getenv("CM95_PG_USER", "cm95").strip(),
|
||||
password=os.getenv("CM95_PG_PASSWORD", ""),
|
||||
sslmode=os.getenv("CM95_PG_SSLMODE", "prefer").strip() or "prefer",
|
||||
admin_database=os.getenv("CM95_PG_ADMIN_DATABASE", "postgres").strip() or "postgres",
|
||||
connect_timeout=int(os.getenv("CM95_PG_CONNECT_TIMEOUT", "20")),
|
||||
)
|
||||
|
||||
def load_source_dir() -> Path:
|
||||
load_dotenv()
|
||||
raw = os.getenv("CM95_SOURCE_DIR", "").strip()
|
||||
if not raw:
|
||||
raise RuntimeError("未传入 CM95/CM93 Japan 路径,且 .env 中 CM95_SOURCE_DIR 未配置")
|
||||
return Path(raw)
|
||||
127
src/cm93db/database.py
Normal file
127
src/cm93db/database.py
Normal file
@@ -0,0 +1,127 @@
|
||||
from __future__ import annotations
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
from psycopg import connect as pg_connect, sql
|
||||
from psycopg.rows import dict_row
|
||||
|
||||
from .config import DbConfig
|
||||
|
||||
def connect(cfg: DbConfig, with_database: bool = True):
|
||||
return pg_connect(
|
||||
host=cfg.host,
|
||||
port=cfg.port,
|
||||
user=cfg.user,
|
||||
password=cfg.password,
|
||||
dbname=cfg.database if with_database else cfg.admin_database,
|
||||
sslmode=cfg.sslmode,
|
||||
connect_timeout=cfg.connect_timeout,
|
||||
autocommit=False,
|
||||
row_factory=dict_row,
|
||||
)
|
||||
|
||||
def server_version(conn) -> str:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SHOW server_version")
|
||||
row = cur.fetchone()
|
||||
return str(row["server_version"])
|
||||
|
||||
def postgresql_version_tuple(version: str) -> tuple[int, int]:
|
||||
m = re.match(r"^(\d+)(?:\.(\d+))?", version)
|
||||
if not m:
|
||||
raise RuntimeError(f"无法识别 PostgreSQL 版本: {version}")
|
||||
return int(m.group(1)), int(m.group(2) or 0)
|
||||
|
||||
def assert_postgresql_compatible(version: str) -> None:
|
||||
major, _minor = postgresql_version_tuple(version)
|
||||
if major < 14:
|
||||
raise RuntimeError(
|
||||
f"CM95 PostgreSQL 版要求 PostgreSQL 14+;当前服务器为 {version}"
|
||||
)
|
||||
|
||||
def postgis_version(conn) -> str:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT PostGIS_Version() AS v")
|
||||
row = cur.fetchone()
|
||||
return str(row["v"])
|
||||
|
||||
def assert_postgis_available(conn) -> str:
|
||||
try:
|
||||
return postgis_version(conn)
|
||||
except Exception as exc:
|
||||
conn.rollback()
|
||||
raise RuntimeError("目标 PostgreSQL 必须安装并启用 PostGIS extension") from exc
|
||||
|
||||
def geom_from_wkb_sql(value_sql: str = "%s", srid: int = 4326) -> str:
|
||||
return f"ST_GeomFromWKB({value_sql},{int(srid)})"
|
||||
|
||||
def geom_from_text_sql(value_sql: str = "%s", srid: int = 4326) -> str:
|
||||
return f"ST_GeomFromText({value_sql},{int(srid)})"
|
||||
|
||||
def as_wkb_sql(geom_sql: str) -> str:
|
||||
return f"ST_AsBinary({geom_sql})"
|
||||
|
||||
def create_database(cfg: DbConfig) -> None:
|
||||
conn = connect(cfg, with_database=False)
|
||||
try:
|
||||
conn.autocommit = True
|
||||
assert_postgresql_compatible(server_version(conn))
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT 1 FROM pg_database WHERE datname=%s", (cfg.database,))
|
||||
if cur.fetchone() is None:
|
||||
cur.execute(
|
||||
sql.SQL("CREATE DATABASE {}").format(sql.Identifier(cfg.database))
|
||||
)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def _split_sql(text: str) -> list[str]:
|
||||
statements: list[str] = []
|
||||
buf: list[str] = []
|
||||
in_single = False
|
||||
in_double = False
|
||||
i = 0
|
||||
while i < len(text):
|
||||
ch = text[i]
|
||||
nxt = text[i + 1] if i + 1 < len(text) else ""
|
||||
if not (in_single or in_double) and ch == "-" and nxt == "-":
|
||||
while i < len(text) and text[i] != "\n":
|
||||
i += 1
|
||||
continue
|
||||
if ch == "'" and not in_double:
|
||||
if in_single and nxt == "'":
|
||||
buf.extend([ch, nxt])
|
||||
i += 2
|
||||
continue
|
||||
in_single = not in_single
|
||||
elif ch == '"' and not in_single:
|
||||
in_double = not in_double
|
||||
if ch == ";" and not (in_single or in_double):
|
||||
stmt = "".join(buf).strip()
|
||||
if stmt:
|
||||
statements.append(stmt)
|
||||
buf = []
|
||||
else:
|
||||
buf.append(ch)
|
||||
i += 1
|
||||
|
||||
tail = "".join(buf).strip()
|
||||
if tail:
|
||||
statements.append(tail)
|
||||
return statements
|
||||
|
||||
def apply_schema(cfg: DbConfig, schema_path: Path) -> None:
|
||||
statements = _split_sql(schema_path.read_text(encoding="utf-8"))
|
||||
conn = connect(cfg)
|
||||
try:
|
||||
assert_postgresql_compatible(server_version(conn))
|
||||
with conn.cursor() as cur:
|
||||
for stmt in statements:
|
||||
cur.execute(stmt)
|
||||
conn.commit()
|
||||
assert_postgis_available(conn)
|
||||
except Exception:
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
38
src/cm93db/decoder.py
Normal file
38
src/cm93db/decoder.py
Normal file
@@ -0,0 +1,38 @@
|
||||
from __future__ import annotations
|
||||
|
||||
# CM93 byte substitution table. This is a format constant, not executable code.
|
||||
# Decode table is derived as: encode[i] = TABLE_0[i] ^ 8; decode[encode[i]] = i.
|
||||
TABLE_0 = bytes([
|
||||
0xCD,0xEA,0xDC,0x48,0x3E,0x6D,0xCA,0x7B,0x52,0xE1,0xA4,0x8E,0xAB,0x05,0xA7,0x97,
|
||||
0xB9,0x60,0x39,0x85,0x7C,0x56,0x7A,0xBA,0x68,0x6E,0xF5,0x5D,0x02,0x4E,0x0F,0xA1,
|
||||
0x27,0x24,0x41,0x34,0x00,0x5A,0xFE,0xCB,0xD0,0xFA,0xF8,0x6C,0x74,0x96,0x9E,0x0E,
|
||||
0xC2,0x49,0xE3,0xE5,0xC0,0x3B,0x59,0x18,0xA9,0x86,0x8F,0x30,0xC3,0xA8,0x22,0x0A,
|
||||
0x14,0x1A,0xB2,0xC9,0xC7,0xED,0xAA,0x29,0x94,0x75,0x0D,0xAC,0x0C,0xF4,0xBB,0xC5,
|
||||
0x3F,0xFD,0xD9,0x9C,0x4F,0xD5,0x84,0x1E,0xB1,0x81,0x69,0xB4,0x09,0xB8,0x3C,0xAF,
|
||||
0xA3,0x08,0xBF,0xE0,0x9A,0xD7,0xF7,0x8C,0x67,0x66,0xAE,0xD4,0x4C,0xA5,0xEC,0xF9,
|
||||
0xB6,0x64,0x78,0x06,0x5B,0x9B,0xF2,0x99,0xCE,0xDB,0x53,0x55,0x65,0x8D,0x07,0x33,
|
||||
0x04,0x37,0x92,0x26,0x23,0xB5,0x58,0xDA,0x2F,0xB3,0x40,0x5E,0x7F,0x4B,0x62,0x80,
|
||||
0xE4,0x6F,0x73,0x1D,0xDF,0x17,0xCC,0x28,0x25,0x2D,0xEE,0x3A,0x98,0xE2,0x01,0xEB,
|
||||
0xDD,0xBC,0x90,0xB0,0xFC,0x95,0x76,0x93,0x46,0x57,0x2C,0x2B,0x50,0x11,0x0B,0xC1,
|
||||
0xF0,0xE7,0xD6,0x21,0x31,0xDE,0xFF,0xD8,0x12,0xA6,0x4D,0x8A,0x13,0x43,0x45,0x38,
|
||||
0xD2,0x87,0xA0,0xEF,0x82,0xF1,0x47,0x89,0x6A,0xC8,0x54,0x1B,0x16,0x7E,0x79,0xBD,
|
||||
0x6B,0x91,0xA2,0x71,0x36,0xB7,0x03,0x3D,0x72,0xC6,0x44,0x8B,0xCF,0x15,0x9F,0x32,
|
||||
0xC4,0x77,0x83,0x63,0x20,0x88,0xF6,0xAD,0xF3,0xE8,0x4A,0xE9,0x35,0x1C,0x5F,0x19,
|
||||
0x1F,0x7D,0x70,0xFB,0xD1,0x51,0x10,0xD3,0x2E,0x61,0x9D,0x5C,0x2A,0x42,0xBE,0xE6,
|
||||
])
|
||||
|
||||
|
||||
def _make_decode_table() -> bytes:
|
||||
decode = bytearray(256)
|
||||
for plain, base in enumerate(TABLE_0):
|
||||
encoded = base ^ 0x08
|
||||
decode[encoded] = plain
|
||||
return bytes(decode)
|
||||
|
||||
|
||||
DECODE_TABLE = _make_decode_table()
|
||||
|
||||
|
||||
def decode_cm93(data: bytes) -> bytes:
|
||||
"""Decode a complete CM93 cell. Byte substitution is position-independent."""
|
||||
return data.translate(DECODE_TABLE)
|
||||
94
src/cm93db/dictionaries.py
Normal file
94
src/cm93db/dictionaries.py
Normal file
@@ -0,0 +1,94 @@
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
TYPE_MAP = {
|
||||
"aFLOAT": "R",
|
||||
"aBYTE": "B",
|
||||
"aSTRING": "S",
|
||||
"aCMPLX": "C",
|
||||
"aLIST": "L",
|
||||
"aWORD10": "W",
|
||||
"aLONG": "G",
|
||||
"aNotUsed": "?",
|
||||
}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ObjectDef:
|
||||
name: str
|
||||
object_id: int
|
||||
geometry_code: str
|
||||
display_code: str | None
|
||||
description: str | None
|
||||
raw_line: str
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AttributeDef:
|
||||
name: str
|
||||
attribute_id: int
|
||||
source_type_name: str
|
||||
decoder_type: str
|
||||
min_value: float | None
|
||||
max_value: float | None
|
||||
raw_line: str
|
||||
|
||||
def _read_lines(path: Path):
|
||||
for enc in ("utf-8-sig", "cp1252", "latin1"):
|
||||
try:
|
||||
return path.read_text(encoding=enc).splitlines()
|
||||
except UnicodeDecodeError:
|
||||
pass
|
||||
raise UnicodeDecodeError("unknown", b"", 0, 1, f"无法读取 {path}")
|
||||
|
||||
def load_object_dictionary(path: Path) -> list[ObjectDef]:
|
||||
out = []
|
||||
for line in _read_lines(path):
|
||||
if not line.strip():
|
||||
continue
|
||||
parts = line.rstrip("\r\n").split("|")
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
try:
|
||||
object_id = int(parts[1].strip())
|
||||
except ValueError:
|
||||
continue
|
||||
out.append(ObjectDef(
|
||||
name=parts[0].strip(),
|
||||
object_id=object_id,
|
||||
geometry_code=parts[2].strip(),
|
||||
display_code=parts[3].strip() if len(parts) > 3 else None,
|
||||
description=parts[4].strip() if len(parts) > 4 else None,
|
||||
raw_line=line,
|
||||
))
|
||||
return out
|
||||
|
||||
def load_attribute_dictionary(path: Path) -> list[AttributeDef]:
|
||||
out = []
|
||||
for line in _read_lines(path):
|
||||
if not line.strip():
|
||||
continue
|
||||
parts = [p.strip() for p in line.rstrip("\r\n").split("|")]
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
try:
|
||||
attribute_id = int(parts[1])
|
||||
except ValueError:
|
||||
continue
|
||||
source_type = parts[2]
|
||||
def f(i):
|
||||
if len(parts) <= i or parts[i] == "":
|
||||
return None
|
||||
try:
|
||||
return float(parts[i])
|
||||
except ValueError:
|
||||
return None
|
||||
out.append(AttributeDef(
|
||||
name=parts[0],
|
||||
attribute_id=attribute_id,
|
||||
source_type_name=source_type,
|
||||
decoder_type=TYPE_MAP.get(source_type, "?"),
|
||||
min_value=f(3),
|
||||
max_value=f(4),
|
||||
raw_line=line,
|
||||
))
|
||||
return out
|
||||
870
src/cm93db/geometry.py
Normal file
870
src/cm93db/geometry.py
Normal file
@@ -0,0 +1,870 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
from typing import Iterable, Any
|
||||
|
||||
try:
|
||||
from shapely import normalize as _shapely_normalize
|
||||
except ImportError:
|
||||
_shapely_normalize = None
|
||||
|
||||
from shapely.geometry import Point, LineString, MultiLineString, Polygon, MultiPolygon
|
||||
from shapely.geometry.base import BaseGeometry
|
||||
from shapely.ops import unary_union, polygonize_full
|
||||
from shapely.validation import explain_validity
|
||||
|
||||
from .cm93_binary import ParsedCell, FeatureRecord
|
||||
from .precision import CM93_SEMIMAJOR_M
|
||||
|
||||
|
||||
# 已通过 CM93 实际数据审计的 Area normalization 规则:
|
||||
# 1. valid Polygon/MultiPolygon 完全原样 PASS-THROUGH;
|
||||
# 2. invalid Polygon/MultiPolygon 仅使用原始 boundary 做 noding + polygonize_full;
|
||||
# 3. planar face 采用 odd-even parity 选择;
|
||||
# 4. 只对修复后的 candidate 做 GEOS canonical normalize;
|
||||
# 5. 不使用 make_valid() / buffer(0) 覆盖原始 CM93 语义;
|
||||
# 6. raw_feature_blob / chart_raw_record 仍由 importer 原样保留。
|
||||
_NORMALIZATION_BOUNDARY_TOLERANCE = 1e-10
|
||||
_POINT_ON_SEGMENT_EPS = 1e-12
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CoverageInfo:
|
||||
source_record_index: int
|
||||
geom_unshifted: BaseGeometry
|
||||
wgsox: float
|
||||
wgsoy: float
|
||||
compilation_scale: int | None
|
||||
horizontal_datum: int | None
|
||||
vertical_datum: int | None
|
||||
attributes: dict[str, Any]
|
||||
geometry_repaired: bool
|
||||
geometry_issue: str | None
|
||||
|
||||
|
||||
@dataclass
|
||||
class BuiltFeature:
|
||||
feature: FeatureRecord
|
||||
geometry_type: str
|
||||
geom: BaseGeometry | None
|
||||
coverage_record_index: int | None
|
||||
geometry_issue: str | None
|
||||
geometry_repaired: bool
|
||||
soundings: list[tuple[int, int, int, float, Point, int | None]]
|
||||
attributes_map: dict[str, Any]
|
||||
|
||||
|
||||
def attr_map(feature: FeatureRecord) -> dict[str, Any]:
|
||||
out: dict[str, Any] = {}
|
||||
for a in feature.attributes:
|
||||
if a.name in out:
|
||||
prev = out[a.name]
|
||||
if not isinstance(prev, list) or (prev and not isinstance(prev[0], (dict, list))):
|
||||
out[a.name] = [prev, a.value]
|
||||
else:
|
||||
prev.append(a.value)
|
||||
else:
|
||||
out[a.name] = a.value
|
||||
if feature.related_indices:
|
||||
out["_cm93_related_indices"] = feature.related_indices
|
||||
if feature.relation20_raw is not None:
|
||||
out["_cm93_relation20_raw"] = feature.relation20_raw
|
||||
return out
|
||||
|
||||
|
||||
def raw_xy_to_mercator(
|
||||
cell: ParsedCell,
|
||||
x: int,
|
||||
y: int,
|
||||
wgsox: float = 0.0,
|
||||
wgsoy: float = 0.0,
|
||||
) -> tuple[float, float]:
|
||||
h = cell.header
|
||||
easting = x * h.x_rate + h.easting_min - wgsox
|
||||
northing = y * h.y_rate + h.northing_min - wgsoy
|
||||
return easting, northing
|
||||
|
||||
|
||||
def mercator_to_lonlat(easting: float, northing: float) -> tuple[float, float]:
|
||||
lon = math.degrees(easting / CM93_SEMIMAJOR_M)
|
||||
lat = math.degrees(
|
||||
2.0 * math.atan(math.exp(northing / CM93_SEMIMAJOR_M))
|
||||
- math.pi / 2.0
|
||||
)
|
||||
# Normalize only after exact formula. This handles cells crossing the dateline.
|
||||
while lon > 180.0:
|
||||
lon -= 360.0
|
||||
while lon < -180.0:
|
||||
lon += 360.0
|
||||
return lon, lat
|
||||
|
||||
|
||||
def raw_xy_to_lonlat(
|
||||
cell: ParsedCell,
|
||||
x: int,
|
||||
y: int,
|
||||
wgsox: float = 0.0,
|
||||
wgsoy: float = 0.0,
|
||||
) -> tuple[float, float]:
|
||||
return mercator_to_lonlat(
|
||||
*raw_xy_to_mercator(cell, x, y, wgsox, wgsoy)
|
||||
)
|
||||
|
||||
|
||||
def _oriented_edge_points(
|
||||
cell: ParsedCell,
|
||||
edge_idx: int,
|
||||
usage: int,
|
||||
) -> list[tuple[int, int]]:
|
||||
pts = list(cell.edges[edge_idx].points)
|
||||
if usage & 4:
|
||||
pts.reverse()
|
||||
return pts
|
||||
|
||||
|
||||
def _append_joined(
|
||||
dst: list[tuple[int, int]],
|
||||
src: list[tuple[int, int]],
|
||||
) -> None:
|
||||
if not src:
|
||||
return
|
||||
if not dst:
|
||||
dst.extend(src)
|
||||
elif dst[-1] == src[0]:
|
||||
dst.extend(src[1:])
|
||||
else:
|
||||
# Preserve discontinuity; audit will report it through geometry validity/issue.
|
||||
dst.extend(src)
|
||||
|
||||
|
||||
def raw_line_parts(
|
||||
cell: ParsedCell,
|
||||
feature: FeatureRecord,
|
||||
) -> list[list[tuple[int, int]]]:
|
||||
parts: list[list[tuple[int, int]]] = []
|
||||
current: list[tuple[int, int]] = []
|
||||
for edge_idx, usage in feature.geometry_refs:
|
||||
seg = _oriented_edge_points(cell, edge_idx, usage)
|
||||
if not seg:
|
||||
continue
|
||||
if not current:
|
||||
current = list(seg)
|
||||
elif current[-1] == seg[0]:
|
||||
current.extend(seg[1:])
|
||||
else:
|
||||
if len(current) >= 2:
|
||||
parts.append(current)
|
||||
current = list(seg)
|
||||
if len(current) >= 2:
|
||||
parts.append(current)
|
||||
return parts
|
||||
|
||||
|
||||
def raw_line_points(
|
||||
cell: ParsedCell,
|
||||
feature: FeatureRecord,
|
||||
) -> list[tuple[int, int]]:
|
||||
# Compatibility helper for diagnostics. Disconnected parts are concatenated here;
|
||||
# production geometry uses raw_line_parts() and preserves disconnection.
|
||||
out: list[tuple[int, int]] = []
|
||||
for part in raw_line_parts(cell, feature):
|
||||
out.extend(part)
|
||||
return out
|
||||
|
||||
|
||||
def raw_area_rings(
|
||||
cell: ParsedCell,
|
||||
feature: FeatureRecord,
|
||||
) -> list[list[tuple[int, int]]]:
|
||||
rings: list[list[tuple[int, int]]] = []
|
||||
current: list[tuple[int, int]] = []
|
||||
start: tuple[int, int] | None = None
|
||||
for edge_idx, usage in feature.geometry_refs:
|
||||
seg = _oriented_edge_points(cell, edge_idx, usage)
|
||||
if not seg:
|
||||
continue
|
||||
if start is None:
|
||||
start = seg[0]
|
||||
_append_joined(current, seg)
|
||||
if current and start is not None and current[-1] == start:
|
||||
if len(current) >= 4:
|
||||
rings.append(current)
|
||||
current = []
|
||||
start = None
|
||||
if current:
|
||||
# Keep an open ring visible to audit. Shapely Polygon will close it, so the issue
|
||||
# is returned explicitly by the Area builder.
|
||||
rings.append(current)
|
||||
return rings
|
||||
|
||||
|
||||
def _transform_points(
|
||||
cell: ParsedCell,
|
||||
pts: Iterable[tuple[int, int]],
|
||||
wgsox: float,
|
||||
wgsoy: float,
|
||||
) -> list[tuple[float, float]]:
|
||||
return [
|
||||
raw_xy_to_lonlat(cell, x, y, wgsox, wgsoy)
|
||||
for x, y in pts
|
||||
]
|
||||
|
||||
|
||||
def _area_from_rings(
|
||||
coord_rings: list[list[tuple[float, float]]],
|
||||
) -> BaseGeometry | None:
|
||||
valid_rings = [r for r in coord_rings if len(r) >= 3]
|
||||
if not valid_rings:
|
||||
return None
|
||||
|
||||
# CM93 edge chains are emitted as closed contours. Most features are one exterior
|
||||
# followed by holes. If a later contour is not inside the current exterior, treat it
|
||||
# as another exterior rather than forcing an invalid hole.
|
||||
polygons: list[Polygon] = []
|
||||
current_exterior = valid_rings[0]
|
||||
holes: list[list[tuple[float, float]]] = []
|
||||
exterior_poly = Polygon(current_exterior)
|
||||
|
||||
for r in valid_rings[1:]:
|
||||
rp = Polygon(r)
|
||||
try:
|
||||
inside = exterior_poly.covers(rp.representative_point())
|
||||
except Exception:
|
||||
inside = False
|
||||
|
||||
if inside:
|
||||
holes.append(r)
|
||||
else:
|
||||
polygons.append(Polygon(current_exterior, holes))
|
||||
current_exterior = r
|
||||
holes = []
|
||||
exterior_poly = Polygon(current_exterior)
|
||||
|
||||
polygons.append(Polygon(current_exterior, holes))
|
||||
|
||||
if len(polygons) == 1:
|
||||
return polygons[0]
|
||||
return MultiPolygon(polygons)
|
||||
|
||||
|
||||
def _geom_issue(
|
||||
geom: BaseGeometry | None,
|
||||
extra: str | None = None,
|
||||
validate: bool = False,
|
||||
) -> str | None:
|
||||
parts = []
|
||||
if extra:
|
||||
parts.append(extra)
|
||||
if geom is None:
|
||||
parts.append("geometry missing")
|
||||
elif geom.is_empty:
|
||||
parts.append("geometry empty")
|
||||
elif validate and not geom.is_valid:
|
||||
parts.append(explain_validity(geom))
|
||||
return "; ".join(parts) if parts else None
|
||||
|
||||
|
||||
def _polygon_rings(
|
||||
geom: BaseGeometry,
|
||||
) -> list[list[tuple[float, float]]]:
|
||||
rings: list[list[tuple[float, float]]] = []
|
||||
|
||||
def add_polygon(poly: Polygon) -> None:
|
||||
rings.append([
|
||||
(float(c[0]), float(c[1]))
|
||||
for c in poly.exterior.coords
|
||||
])
|
||||
for ring in poly.interiors:
|
||||
rings.append([
|
||||
(float(c[0]), float(c[1]))
|
||||
for c in ring.coords
|
||||
])
|
||||
|
||||
if geom.geom_type == "Polygon":
|
||||
add_polygon(geom)
|
||||
elif geom.geom_type == "MultiPolygon":
|
||||
for poly in geom.geoms:
|
||||
add_polygon(poly)
|
||||
|
||||
return rings
|
||||
|
||||
|
||||
def _point_on_segment(
|
||||
px: float,
|
||||
py: float,
|
||||
ax: float,
|
||||
ay: float,
|
||||
bx: float,
|
||||
by: float,
|
||||
eps: float = _POINT_ON_SEGMENT_EPS,
|
||||
) -> bool:
|
||||
vx = bx - ax
|
||||
vy = by - ay
|
||||
wx = px - ax
|
||||
wy = py - ay
|
||||
|
||||
cross_value = abs(vx * wy - vy * wx)
|
||||
if cross_value > eps * max(1.0, abs(vx), abs(vy)):
|
||||
return False
|
||||
|
||||
dot_value = wx * vx + wy * vy
|
||||
if dot_value < -eps:
|
||||
return False
|
||||
|
||||
vv = vx * vx + vy * vy
|
||||
if dot_value - vv > eps:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _odd_even_inside(
|
||||
point: tuple[float, float],
|
||||
rings: list[list[tuple[float, float]]],
|
||||
) -> bool:
|
||||
x, y = point
|
||||
crossings = 0
|
||||
|
||||
for ring in rings:
|
||||
for i in range(len(ring) - 1):
|
||||
x1, y1 = ring[i]
|
||||
x2, y2 = ring[i + 1]
|
||||
|
||||
if _point_on_segment(x, y, x1, y1, x2, y2):
|
||||
# polygonize face representative_point should not normally lie
|
||||
# on the input boundary. Treat a numerical boundary hit as inside.
|
||||
return True
|
||||
|
||||
if (y1 > y) != (y2 > y):
|
||||
x_intersection = (
|
||||
x1
|
||||
+ (y - y1) * (x2 - x1) / (y2 - y1)
|
||||
)
|
||||
if x_intersection > x:
|
||||
crossings += 1
|
||||
|
||||
return (crossings % 2) == 1
|
||||
|
||||
|
||||
def _canonicalize_polygonal(
|
||||
geom: BaseGeometry,
|
||||
) -> BaseGeometry:
|
||||
if _shapely_normalize is not None:
|
||||
return _shapely_normalize(geom)
|
||||
|
||||
normalizer = getattr(geom, "normalize", None)
|
||||
if callable(normalizer):
|
||||
normalized = normalizer()
|
||||
if normalized is not None:
|
||||
return normalized
|
||||
|
||||
raise RuntimeError(
|
||||
"CM93 Area normalization requires Shapely/GEOS normalize support"
|
||||
)
|
||||
|
||||
|
||||
def _normalize_cm93_area(
|
||||
geom: BaseGeometry | None,
|
||||
structural_issue: str | None = None,
|
||||
) -> tuple[BaseGeometry | None, bool, str | None]:
|
||||
"""
|
||||
Normalize an OGC-invalid CM93 Area using the audited CM93 fill semantics.
|
||||
|
||||
Valid geometry is returned byte/structure untouched.
|
||||
Open/discontinuous source rings are not silently repaired.
|
||||
"""
|
||||
if geom is None:
|
||||
return None, False, _geom_issue(None, structural_issue)
|
||||
|
||||
if geom.is_empty:
|
||||
return geom, False, _geom_issue(geom, structural_issue)
|
||||
|
||||
if geom.geom_type not in ("Polygon", "MultiPolygon"):
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"area geometry has unexpected type {geom.geom_type}",
|
||||
)
|
||||
|
||||
# Structural source problems are evidence, not retraced-connector semantics.
|
||||
# Preserve them for audit instead of silently normalizing them away.
|
||||
if structural_issue:
|
||||
issue = _geom_issue(
|
||||
geom,
|
||||
structural_issue,
|
||||
validate=True,
|
||||
)
|
||||
return geom, False, issue
|
||||
|
||||
# Critical compatibility rule: already-valid geometry stays exactly as built.
|
||||
if geom.is_valid:
|
||||
return geom, False, None
|
||||
|
||||
original_reason = explain_validity(geom)
|
||||
|
||||
try:
|
||||
original_rings = _polygon_rings(geom)
|
||||
noded_boundary = unary_union(geom.boundary)
|
||||
polygonized, _cuts, _dangles, _invalid_rings = polygonize_full(
|
||||
noded_boundary
|
||||
)
|
||||
|
||||
if polygonized.geom_type == "Polygon":
|
||||
faces = [polygonized]
|
||||
elif polygonized.geom_type in ("MultiPolygon", "GeometryCollection"):
|
||||
faces = [
|
||||
g
|
||||
for g in polygonized.geoms
|
||||
if g.geom_type == "Polygon"
|
||||
]
|
||||
else:
|
||||
faces = []
|
||||
|
||||
selected: list[Polygon] = []
|
||||
|
||||
for face in faces:
|
||||
rp = face.representative_point()
|
||||
if _odd_even_inside(
|
||||
(float(rp.x), float(rp.y)),
|
||||
original_rings,
|
||||
):
|
||||
selected.append(face)
|
||||
|
||||
if not selected:
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; odd-even normalization produced no polygon face",
|
||||
)
|
||||
|
||||
candidate = unary_union(selected)
|
||||
|
||||
if candidate.geom_type not in ("Polygon", "MultiPolygon"):
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; odd-even normalization produced {candidate.geom_type}",
|
||||
)
|
||||
|
||||
if candidate.is_empty:
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; odd-even normalization produced empty geometry",
|
||||
)
|
||||
|
||||
if not candidate.is_valid:
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; normalized candidate invalid: "
|
||||
f"{explain_validity(candidate)}",
|
||||
)
|
||||
|
||||
# Safety guard learned from the dry-run audit: normalization may remove
|
||||
# retraced zero-area connector linework, but must never invent a new
|
||||
# boundary outside the original CM93 line network.
|
||||
candidate_extra_boundary = candidate.boundary.difference(
|
||||
noded_boundary
|
||||
)
|
||||
if (
|
||||
not candidate_extra_boundary.is_empty
|
||||
and float(candidate_extra_boundary.length)
|
||||
> _NORMALIZATION_BOUNDARY_TOLERANCE
|
||||
):
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; normalized candidate introduces "
|
||||
f"new boundary length={candidate_extra_boundary.length}",
|
||||
)
|
||||
|
||||
candidate = _canonicalize_polygonal(candidate)
|
||||
|
||||
if candidate.geom_type not in ("Polygon", "MultiPolygon"):
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; canonical candidate has type "
|
||||
f"{candidate.geom_type}",
|
||||
)
|
||||
|
||||
if candidate.is_empty or not candidate.is_valid:
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; canonical candidate is not valid",
|
||||
)
|
||||
|
||||
return candidate, True, None
|
||||
|
||||
except Exception as exc:
|
||||
return (
|
||||
geom,
|
||||
False,
|
||||
f"{original_reason}; odd-even normalization failed: "
|
||||
f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
|
||||
|
||||
def build_unshifted_geometry(
|
||||
cell: ParsedCell,
|
||||
feature: FeatureRecord,
|
||||
validate: bool = False,
|
||||
) -> tuple[BaseGeometry | None, str | None]:
|
||||
prim = feature.geometry_primitive
|
||||
|
||||
if prim == 1 and feature.point2d_index is not None:
|
||||
x, y = cell.point2d[feature.point2d_index]
|
||||
return Point(raw_xy_to_lonlat(cell, x, y)), None
|
||||
|
||||
if prim == 2:
|
||||
parts = raw_line_parts(cell, feature)
|
||||
if not parts:
|
||||
return None, "line has fewer than 2 points"
|
||||
coords = [
|
||||
_transform_points(cell, p, 0.0, 0.0)
|
||||
for p in parts
|
||||
]
|
||||
g = (
|
||||
LineString(coords[0])
|
||||
if len(coords) == 1
|
||||
else MultiLineString(coords)
|
||||
)
|
||||
return g, _geom_issue(g, validate=validate)
|
||||
|
||||
if prim == 4:
|
||||
raw_rings = raw_area_rings(cell, feature)
|
||||
open_count = sum(
|
||||
1
|
||||
for r in raw_rings
|
||||
if r and r[0] != r[-1]
|
||||
)
|
||||
rings = [
|
||||
_transform_points(cell, r, 0.0, 0.0)
|
||||
for r in raw_rings
|
||||
]
|
||||
g = _area_from_rings(rings)
|
||||
extra = f"{open_count} open ring(s)" if open_count else None
|
||||
return g, _geom_issue(g, extra, validate=validate)
|
||||
|
||||
return None, None
|
||||
|
||||
|
||||
def build_coverages(cell: ParsedCell) -> list[CoverageInfo]:
|
||||
out: list[CoverageInfo] = []
|
||||
|
||||
for f in cell.features:
|
||||
if f.object_class != "_m_sor":
|
||||
continue
|
||||
|
||||
geom, pre_issue = build_unshifted_geometry(
|
||||
cell,
|
||||
f,
|
||||
validate=False,
|
||||
)
|
||||
if geom is None:
|
||||
continue
|
||||
|
||||
raw_rings = raw_area_rings(cell, f) if f.geometry_primitive == 4 else []
|
||||
open_count = sum(
|
||||
1
|
||||
for r in raw_rings
|
||||
if r and r[0] != r[-1]
|
||||
)
|
||||
structural_issue = (
|
||||
f"{open_count} open ring(s)"
|
||||
if open_count
|
||||
else pre_issue
|
||||
)
|
||||
|
||||
if f.geometry_primitive == 4:
|
||||
geom, repaired, issue = _normalize_cm93_area(
|
||||
geom,
|
||||
structural_issue=structural_issue,
|
||||
)
|
||||
else:
|
||||
repaired = False
|
||||
issue = _geom_issue(
|
||||
geom,
|
||||
pre_issue,
|
||||
validate=False,
|
||||
)
|
||||
|
||||
if geom is None:
|
||||
continue
|
||||
|
||||
a = attr_map(f)
|
||||
|
||||
def number(name: str, default=0.0):
|
||||
v = a.get(name, default)
|
||||
if isinstance(v, list):
|
||||
v = v[0] if v else default
|
||||
try:
|
||||
return float(v)
|
||||
except (TypeError, ValueError):
|
||||
return float(default)
|
||||
|
||||
def integer(name: str):
|
||||
if name not in a:
|
||||
return None
|
||||
try:
|
||||
return int(a[name])
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
out.append(CoverageInfo(
|
||||
source_record_index=f.index,
|
||||
geom_unshifted=geom,
|
||||
wgsox=number("_wgsox", 0.0),
|
||||
wgsoy=number("_wgsoy", 0.0),
|
||||
compilation_scale=integer("CSCALE"),
|
||||
horizontal_datum=integer("HORDAT"),
|
||||
vertical_datum=integer("VERDAT"),
|
||||
attributes=a,
|
||||
geometry_repaired=repaired,
|
||||
geometry_issue=issue,
|
||||
))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def choose_coverage(
|
||||
coverages: list[CoverageInfo],
|
||||
geom_unshifted: BaseGeometry | None,
|
||||
) -> CoverageInfo | None:
|
||||
if not coverages:
|
||||
return None
|
||||
|
||||
if len(coverages) == 1:
|
||||
return coverages[0]
|
||||
|
||||
if geom_unshifted is None or geom_unshifted.is_empty:
|
||||
return coverages[0]
|
||||
|
||||
minx, miny, maxx, maxy = geom_unshifted.bounds
|
||||
probe = Point(
|
||||
(minx + maxx) / 2.0,
|
||||
(miny + maxy) / 2.0,
|
||||
)
|
||||
|
||||
for cov in coverages:
|
||||
try:
|
||||
if cov.geom_unshifted.covers(probe):
|
||||
return cov
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return coverages[0]
|
||||
|
||||
|
||||
def _depth(raw_z: int) -> float:
|
||||
return (
|
||||
float(raw_z - 12000)
|
||||
if raw_z >= 12000
|
||||
else raw_z / 10.0
|
||||
)
|
||||
|
||||
|
||||
def build_feature(
|
||||
cell: ParsedCell,
|
||||
feature: FeatureRecord,
|
||||
coverages: list[CoverageInfo],
|
||||
validate: bool = False,
|
||||
) -> BuiltFeature:
|
||||
amap = attr_map(feature)
|
||||
prim = feature.geometry_primitive
|
||||
|
||||
# _m_sor is stored spatially in chart_coverage. Reuse the already-normalized
|
||||
# coverage object here so chart_feature bbox/repair metadata matches it.
|
||||
if feature.object_class == "_m_sor":
|
||||
cov = next(
|
||||
(
|
||||
c
|
||||
for c in coverages
|
||||
if c.source_record_index == feature.index
|
||||
),
|
||||
None,
|
||||
)
|
||||
if cov is not None:
|
||||
return BuiltFeature(
|
||||
feature=feature,
|
||||
geometry_type="META",
|
||||
geom=cov.geom_unshifted,
|
||||
coverage_record_index=feature.index,
|
||||
geometry_issue=cov.geometry_issue,
|
||||
geometry_repaired=cov.geometry_repaired,
|
||||
soundings=[],
|
||||
attributes_map=amap,
|
||||
)
|
||||
|
||||
g, issue = build_unshifted_geometry(
|
||||
cell,
|
||||
feature,
|
||||
validate=validate,
|
||||
)
|
||||
return BuiltFeature(
|
||||
feature=feature,
|
||||
geometry_type="META",
|
||||
geom=g,
|
||||
coverage_record_index=feature.index,
|
||||
geometry_issue=issue,
|
||||
geometry_repaired=False,
|
||||
soundings=[],
|
||||
attributes_map=amap,
|
||||
)
|
||||
|
||||
# CM93 raw geotype 16 is a parent/no-geometry object. It is expected to
|
||||
# have no own Point/Line/Area and must not be counted as geometry invalid.
|
||||
if feature.geotype == 16:
|
||||
return BuiltFeature(
|
||||
feature=feature,
|
||||
geometry_type="META",
|
||||
geom=None,
|
||||
coverage_record_index=None,
|
||||
geometry_issue=None,
|
||||
geometry_repaired=False,
|
||||
soundings=[],
|
||||
attributes_map=amap,
|
||||
)
|
||||
|
||||
if prim == 8 and feature.point3d_index is not None:
|
||||
desc = cell.point3d[feature.point3d_index]
|
||||
soundings = []
|
||||
|
||||
for x, y, z in desc.points:
|
||||
unshifted = Point(raw_xy_to_lonlat(cell, x, y))
|
||||
cov = choose_coverage(coverages, unshifted)
|
||||
wx = cov.wgsox if cov else 0.0
|
||||
wy = cov.wgsoy if cov else 0.0
|
||||
p = Point(
|
||||
raw_xy_to_lonlat(
|
||||
cell,
|
||||
x,
|
||||
y,
|
||||
wx,
|
||||
wy,
|
||||
)
|
||||
)
|
||||
soundings.append((
|
||||
x,
|
||||
y,
|
||||
z,
|
||||
_depth(z),
|
||||
p,
|
||||
cov.source_record_index if cov else None,
|
||||
))
|
||||
|
||||
return BuiltFeature(
|
||||
feature=feature,
|
||||
geometry_type="SOUNDING",
|
||||
geom=None,
|
||||
coverage_record_index=None,
|
||||
geometry_issue=None,
|
||||
geometry_repaired=False,
|
||||
soundings=soundings,
|
||||
attributes_map=amap,
|
||||
)
|
||||
|
||||
# Unshifted geometry is used only to choose the WGS offset coverage.
|
||||
# Do not let optional validation of this intermediate copy create a
|
||||
# duplicate Area issue; final shifted Area is normalized below.
|
||||
unshifted, pre_issue = build_unshifted_geometry(
|
||||
cell,
|
||||
feature,
|
||||
validate=False,
|
||||
)
|
||||
cov = choose_coverage(coverages, unshifted)
|
||||
wx = cov.wgsox if cov else 0.0
|
||||
wy = cov.wgsoy if cov else 0.0
|
||||
|
||||
geometry_repaired = False
|
||||
|
||||
if prim == 1 and feature.point2d_index is not None:
|
||||
x, y = cell.point2d[feature.point2d_index]
|
||||
geom = Point(
|
||||
raw_xy_to_lonlat(
|
||||
cell,
|
||||
x,
|
||||
y,
|
||||
wx,
|
||||
wy,
|
||||
)
|
||||
)
|
||||
gtype = "POINT"
|
||||
issue = _geom_issue(
|
||||
geom,
|
||||
pre_issue,
|
||||
validate=validate,
|
||||
)
|
||||
|
||||
elif prim == 2:
|
||||
parts = raw_line_parts(cell, feature)
|
||||
coords = [
|
||||
_transform_points(cell, p, wx, wy)
|
||||
for p in parts
|
||||
]
|
||||
geom = (
|
||||
(
|
||||
LineString(coords[0])
|
||||
if len(coords) == 1
|
||||
else MultiLineString(coords)
|
||||
)
|
||||
if coords
|
||||
else None
|
||||
)
|
||||
gtype = "LINE"
|
||||
issue = _geom_issue(
|
||||
geom,
|
||||
pre_issue,
|
||||
validate=validate,
|
||||
)
|
||||
|
||||
elif prim == 4:
|
||||
raw_rings = raw_area_rings(cell, feature)
|
||||
open_count = sum(
|
||||
1
|
||||
for r in raw_rings
|
||||
if r and r[0] != r[-1]
|
||||
)
|
||||
rings = [
|
||||
_transform_points(cell, r, wx, wy)
|
||||
for r in raw_rings
|
||||
]
|
||||
raw_geom = _area_from_rings(rings)
|
||||
structural_issue = (
|
||||
f"{open_count} open ring(s)"
|
||||
if open_count
|
||||
else None
|
||||
)
|
||||
|
||||
geom, geometry_repaired, issue = _normalize_cm93_area(
|
||||
raw_geom,
|
||||
structural_issue=structural_issue,
|
||||
)
|
||||
gtype = "AREA"
|
||||
|
||||
else:
|
||||
geom = None
|
||||
gtype = "META" if prim == 0 else "UNKNOWN"
|
||||
issue = _geom_issue(
|
||||
geom,
|
||||
pre_issue,
|
||||
validate=validate,
|
||||
)
|
||||
|
||||
return BuiltFeature(
|
||||
feature=feature,
|
||||
geometry_type=gtype,
|
||||
geom=geom,
|
||||
coverage_record_index=(
|
||||
cov.source_record_index
|
||||
if cov
|
||||
else None
|
||||
),
|
||||
geometry_issue=issue,
|
||||
geometry_repaired=geometry_repaired,
|
||||
soundings=[],
|
||||
attributes_map=amap,
|
||||
)
|
||||
1125
src/cm93db/importer.py
Normal file
1125
src/cm93db/importer.py
Normal file
File diff suppressed because it is too large
Load Diff
39
src/cm93db/precision.py
Normal file
39
src/cm93db/precision.py
Normal file
@@ -0,0 +1,39 @@
|
||||
from __future__ import annotations
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
CM93_COORD_MAX = 65535.0
|
||||
CM93_SEMIMAJOR_M = 6378388.0
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CellTransform:
|
||||
easting_min: float
|
||||
northing_min: float
|
||||
easting_max: float
|
||||
northing_max: float
|
||||
|
||||
@property
|
||||
def x_rate(self) -> float:
|
||||
delta=self.easting_max-self.easting_min
|
||||
if delta < 0:
|
||||
delta += CM93_SEMIMAJOR_M * 2.0 * math.pi
|
||||
return delta / CM93_COORD_MAX
|
||||
|
||||
@property
|
||||
def y_rate(self) -> float:
|
||||
return (self.northing_max - self.northing_min) / CM93_COORD_MAX
|
||||
|
||||
def raw_xy_to_cm93_mercator(raw_x: int, raw_y: int, t: CellTransform) -> tuple[float, float]:
|
||||
if not 0 <= raw_x <= 65535 or not 0 <= raw_y <= 65535:
|
||||
raise ValueError("CM93 raw x/y 必须在 0..65535")
|
||||
return float(raw_x*t.x_rate+t.easting_min), float(raw_y*t.y_rate+t.northing_min)
|
||||
|
||||
def apply_mercator_offset(easting: float, northing: float, wgsox: float, wgsoy: float) -> tuple[float, float]:
|
||||
# CM93 coverage attributes are metre offsets. CM93 Transform applies them by
|
||||
# subtraction before the Mercator→lon/lat conversion.
|
||||
return easting-wgsox, northing-wgsoy
|
||||
|
||||
def cm93_mercator_to_lonlat(easting: float, northing: float) -> tuple[float, float]:
|
||||
lon=math.degrees(easting/CM93_SEMIMAJOR_M)
|
||||
lat=math.degrees(2.0*math.atan(math.exp(northing/CM93_SEMIMAJOR_M))-math.pi/2.0)
|
||||
return lon,lat
|
||||
59
src/cm93db/scanner.py
Normal file
59
src/cm93db/scanner.py
Normal file
@@ -0,0 +1,59 @@
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
SCALE_NATIVE = {
|
||||
"Z": 20_000_000,
|
||||
"A": 3_000_000,
|
||||
"B": 1_000_000,
|
||||
"C": 200_000,
|
||||
"D": 100_000,
|
||||
"E": 50_000,
|
||||
"F": 20_000,
|
||||
"G": 7_500,
|
||||
}
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CellFile:
|
||||
path: Path
|
||||
relative_path: Path
|
||||
scale_code: str
|
||||
native_scale: int
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ScanResult:
|
||||
root: Path
|
||||
cells: tuple[CellFile, ...]
|
||||
object_dictionary: Path | None
|
||||
attribute_dictionary: Path | None
|
||||
|
||||
def _find_case_insensitive(root: Path, names: set[str]) -> Path | None:
|
||||
lowers = {n.lower() for n in names}
|
||||
for p in root.rglob("*"):
|
||||
if p.is_file() and p.name.lower() in lowers:
|
||||
return p
|
||||
return None
|
||||
|
||||
def scan_cm93(root: Path) -> ScanResult:
|
||||
root = root.resolve()
|
||||
if not root.exists():
|
||||
raise FileNotFoundError(root)
|
||||
cells = []
|
||||
for p in root.rglob("*"):
|
||||
if not p.is_file():
|
||||
continue
|
||||
ext = p.suffix.upper().lstrip(".")
|
||||
if ext in SCALE_NATIVE:
|
||||
cells.append(
|
||||
CellFile(
|
||||
path=p,
|
||||
relative_path=p.relative_to(root),
|
||||
scale_code=ext,
|
||||
native_scale=SCALE_NATIVE[ext],
|
||||
)
|
||||
)
|
||||
cells.sort(key=lambda c: (c.scale_code, str(c.relative_path).lower()))
|
||||
obj = _find_case_insensitive(root, {"CM93OBJ.DIC"})
|
||||
attr = _find_case_insensitive(root, {"ATTRLUT.DIC", "CM93ATTR.DIC"})
|
||||
return ScanResult(root, tuple(cells), obj, attr)
|
||||
21
src/cm93db/schema_contract.py
Normal file
21
src/cm93db/schema_contract.py
Normal file
@@ -0,0 +1,21 @@
|
||||
SCHEMA_VERSION = 1
|
||||
TARGET_POSTGRESQL_MIN_MAJOR = 14
|
||||
POSTGIS_REQUIRED = True
|
||||
NORMALIZED_SRID = 4326
|
||||
|
||||
REQUIRED_TABLES = {
|
||||
"chart_dataset",
|
||||
"chart_dictionary_object",
|
||||
"chart_dictionary_attribute",
|
||||
"chart_cell",
|
||||
"chart_coverage",
|
||||
"chart_feature",
|
||||
"chart_feature_attribute",
|
||||
"chart_point_geometry",
|
||||
"chart_line_geometry",
|
||||
"chart_area_geometry",
|
||||
"chart_sounding_point",
|
||||
"chart_raw_record",
|
||||
"chart_import_job",
|
||||
"chart_import_event",
|
||||
}
|
||||
2
src/cm95pbf/__init__.py
Normal file
2
src/cm95pbf/__init__.py
Normal file
@@ -0,0 +1,2 @@
|
||||
__version__ = "0.4.14-s52-opencpn-lookup-flow"
|
||||
BUILDER_CONTRACT_VERSION = "cm95pbf-builder/0.4.14-s52-opencpn-lookup-flow"
|
||||
2178
src/cm95pbf/builder.py
Normal file
2178
src/cm95pbf/builder.py
Normal file
File diff suppressed because it is too large
Load Diff
436
src/cm95pbf/cli.py
Normal file
436
src/cm95pbf/cli.py
Normal file
@@ -0,0 +1,436 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from cm93db.config import load_config
|
||||
from cm93db.database import connect
|
||||
|
||||
from .builder import Cm95TileBuilder, TileJob
|
||||
from .identity import build_manifest
|
||||
from .policy import child_tiles
|
||||
from .production import (
|
||||
create_production_plan,
|
||||
migrate_production_release,
|
||||
parse_zooms,
|
||||
production_status,
|
||||
run_production_plan,
|
||||
)
|
||||
from .release import finalize_release
|
||||
|
||||
|
||||
DEFAULT_MAPPING = Path("config/mappings/cm95_to_navsea_v1.yaml")
|
||||
DEFAULT_POLICY = Path("config/pbf/cm95_tile_policy_v1.yaml")
|
||||
DEFAULT_MANIFEST = Path("build/cm95_fid_manifest_v1.sqlite")
|
||||
|
||||
|
||||
def _add_common_db(p):
|
||||
p.add_argument("--dataset-id", type=int, default=1)
|
||||
|
||||
|
||||
def _parse_aoi_bbox(value: str | None):
|
||||
if value is None or not str(value).strip():
|
||||
return None
|
||||
parts = [part.strip() for part in str(value).split(",")]
|
||||
if len(parts) != 4:
|
||||
raise argparse.ArgumentTypeError("AOI bbox 必须是 west,south,east,north")
|
||||
try:
|
||||
return tuple(float(part) for part in parts)
|
||||
except ValueError as exc:
|
||||
raise argparse.ArgumentTypeError("AOI bbox 必须是四个数字") from exc
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
p = argparse.ArgumentParser(
|
||||
prog="cm95pbf",
|
||||
description="CM95 PostgreSQL/PostGIS → NavSea-compatible MVT/PBF generator",
|
||||
)
|
||||
sub = p.add_subparsers(dest="command", required=True)
|
||||
|
||||
manifest = sub.add_parser(
|
||||
"build-fid-manifest",
|
||||
help="建立全 Dataset 确定性 int32 MVT Feature ID 清单",
|
||||
)
|
||||
_add_common_db(manifest)
|
||||
manifest.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
||||
manifest.add_argument("--rebuild-fid-manifest", action="store_true")
|
||||
|
||||
tile = sub.add_parser(
|
||||
"build-tile",
|
||||
help="生成单张 CM95 NavSea-compatible PBF",
|
||||
)
|
||||
_add_common_db(tile)
|
||||
tile.add_argument("--z", type=int, required=True)
|
||||
tile.add_argument("--x", type=int, required=True)
|
||||
tile.add_argument("--y", type=int, required=True)
|
||||
tile.add_argument("--profile", choices=["engineering", "delivery"], default="delivery")
|
||||
tile.add_argument("--mapping", default=str(DEFAULT_MAPPING))
|
||||
tile.add_argument("--policy", default=str(DEFAULT_POLICY))
|
||||
tile.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
||||
tile.add_argument("--output-root", default="build/pbf")
|
||||
tile.add_argument("--report-root", default="build/pbf-reports")
|
||||
tile.add_argument("--s52-lookups", default=None)
|
||||
tile.add_argument(
|
||||
"--s52-point-table",
|
||||
choices=["Simplified", "Paper"],
|
||||
default="Paper",
|
||||
)
|
||||
tile.add_argument(
|
||||
"--s52-boundary-table",
|
||||
choices=["Plain", "Symbolized"],
|
||||
default="Plain",
|
||||
)
|
||||
tile.add_argument(
|
||||
"--s52-mariner-mode",
|
||||
choices=["runtime", "reference"],
|
||||
default="runtime",
|
||||
help="runtime=保留船员设置相关事实;reference=按固定 S52MarinerSettings 物化,用于 OpenCPN 对照",
|
||||
)
|
||||
|
||||
sample = sub.add_parser(
|
||||
"build-sample-aoi",
|
||||
help="生成 z10 parent + z11 children + z12 grandchildren,共 21 张测试 tile",
|
||||
)
|
||||
_add_common_db(sample)
|
||||
sample.add_argument("--z10-x", type=int, default=914)
|
||||
sample.add_argument("--z10-y", type=int, default=385)
|
||||
sample.add_argument(
|
||||
"--profiles",
|
||||
default="engineering,delivery",
|
||||
help="逗号分隔:engineering,delivery",
|
||||
)
|
||||
sample.add_argument("--mapping", default=str(DEFAULT_MAPPING))
|
||||
sample.add_argument("--policy", default=str(DEFAULT_POLICY))
|
||||
sample.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
||||
sample.add_argument("--output-root", default="build/pbf")
|
||||
sample.add_argument("--report-root", default="build/pbf-reports")
|
||||
sample.add_argument("--s52-lookups", default=None)
|
||||
sample.add_argument(
|
||||
"--s52-point-table",
|
||||
choices=["Simplified", "Paper"],
|
||||
default="Paper",
|
||||
)
|
||||
sample.add_argument(
|
||||
"--s52-boundary-table",
|
||||
choices=["Plain", "Symbolized"],
|
||||
default="Plain",
|
||||
)
|
||||
sample.add_argument(
|
||||
"--s52-mariner-mode",
|
||||
choices=["runtime", "reference"],
|
||||
default="runtime",
|
||||
help="runtime=保留船员设置相关事实;reference=按固定 S52MarinerSettings 物化,用于 OpenCPN 对照",
|
||||
)
|
||||
|
||||
|
||||
prod_plan = sub.add_parser(
|
||||
"plan-production",
|
||||
help="按 Dataset coverage 枚举生产 tile,并冻结不可变生成计划",
|
||||
)
|
||||
_add_common_db(prod_plan)
|
||||
prod_plan.add_argument("--zooms", default="0-12", help="缩放级别,支持 0-12 或 0,1,...,12")
|
||||
prod_plan.add_argument("--profile", choices=["engineering", "delivery"], default="delivery")
|
||||
prod_plan.add_argument("--mapping", default=str(DEFAULT_MAPPING))
|
||||
prod_plan.add_argument("--policy", default=str(DEFAULT_POLICY))
|
||||
prod_plan.add_argument("--manifest", default=str(DEFAULT_MANIFEST))
|
||||
prod_plan.add_argument("--output", default="build/production/cm95_production_plan_v2.json")
|
||||
prod_plan.add_argument("--s52-lookups", default=None, help="S-52 lookup_rule_ir.json;提供后冻结 SHA256 并进入 production builder")
|
||||
prod_plan.add_argument(
|
||||
"--s52-resource-contract",
|
||||
default=None,
|
||||
help="S-52 resource_contract.json;S-52 正式生产必须为 PASS,并冻结 SHA256",
|
||||
)
|
||||
prod_plan.add_argument("--s52-point-table", choices=["Simplified", "Paper"], default="Paper")
|
||||
prod_plan.add_argument("--s52-boundary-table", choices=["Plain", "Symbolized"], default="Plain")
|
||||
prod_plan.add_argument("--s52-mariner-mode", choices=["runtime", "reference"], default="runtime")
|
||||
prod_plan.add_argument("--aoi-name", default=None, help="可审计 AOI 名称")
|
||||
prod_plan.add_argument("--aoi-bbox", type=_parse_aoi_bbox, default=None, metavar="W,S,E,N", help="WGS84 AOI bbox;aoi-complete 模式下枚举 bbox 内完整 XYZ 金字塔")
|
||||
prod_plan.add_argument(
|
||||
"--tile-enumeration-mode",
|
||||
choices=["auto", "coverage", "aoi-complete"],
|
||||
default="auto",
|
||||
help="auto=有 AOI 时使用 aoi-complete,否则 coverage;正式区域/全国生产应使用 aoi-complete",
|
||||
)
|
||||
|
||||
prod_build = sub.add_parser(
|
||||
"build-production",
|
||||
help="按冻结计划批量生成,支持断点续跑、并发和失败重试",
|
||||
)
|
||||
prod_build.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
|
||||
prod_build.add_argument("--release-root", default="build/releases")
|
||||
prod_build.add_argument("--workers", type=int, default=4)
|
||||
prod_build.add_argument("--max-attempts", type=int, default=3)
|
||||
|
||||
prod_migrate = sub.add_parser(
|
||||
"migrate-production-release",
|
||||
help="把旧 release 的 delivery-equivalent PASS tile 复用到新计划,只重建受影响/失败任务(支持 builder contract 升级)",
|
||||
)
|
||||
prod_migrate.add_argument("--old-release-dir", required=True)
|
||||
prod_migrate.add_argument("--new-plan", default="build/production/cm95_production_plan_v1_4_5.json")
|
||||
prod_migrate.add_argument("--release-root", default="build/releases")
|
||||
prod_migrate.add_argument("--copy-mode", choices=["hardlink", "copy"], default="hardlink")
|
||||
prod_migrate.add_argument(
|
||||
"--invalidate-tiles-file",
|
||||
default=None,
|
||||
help="每行一个 z/x/y;这些旧 PASS tile 强制留为 PENDING,适用于 builder contract 的定向重建",
|
||||
)
|
||||
|
||||
prod_status = sub.add_parser(
|
||||
"status-production",
|
||||
help="查看生产计划当前账本状态",
|
||||
)
|
||||
prod_status.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
|
||||
prod_status.add_argument("--release-root", default="build/releases")
|
||||
|
||||
prod_final = sub.add_parser(
|
||||
"finalize-production",
|
||||
help="对完成的生产产物做三方计数/FID/哈希盘点并生成 release manifest",
|
||||
)
|
||||
prod_final.add_argument("--plan", default="build/production/cm95_production_plan_v1.json")
|
||||
prod_final.add_argument("--release-root", default="build/releases")
|
||||
prod_final.add_argument("--style", default="config/style/style.navsea-delivery-full-semantic-unified-20260808-cm95-v1.json")
|
||||
prod_final.add_argument("--sprite-json", default="config/sprite/semantic-sprite-20260808.json")
|
||||
prod_final.add_argument("--gap-disposition", default="config/mappings/cm95_gap_disposition_v1.yaml")
|
||||
prod_final.add_argument("--presentation-conservation-audit", default=None, help="Source→PBF presentation conservation audit JSON;builder 0.4.11+ 正式 release 必填且必须 PASS")
|
||||
prod_final.add_argument("--runtime-capability-audit", default=None, help="目标 renderer runtime capability audit JSON;提供后必须 PASS")
|
||||
prod_final.add_argument("--skip-style-contract", action="store_true", help="跳过旧 NavSea style/sprite contract;S-52 runtime release 仍执行 runtime contract gate")
|
||||
|
||||
return p
|
||||
|
||||
|
||||
def ensure_manifest(dataset_id: int, manifest_path: Path) -> dict:
|
||||
cfg = load_config()
|
||||
with connect(cfg) as conn:
|
||||
return build_manifest(
|
||||
conn,
|
||||
dataset_id,
|
||||
manifest_path,
|
||||
rebuild=False,
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_parser().parse_args()
|
||||
cfg = load_config()
|
||||
|
||||
if args.command == "plan-production":
|
||||
manifest_path = Path(args.manifest)
|
||||
if not manifest_path.exists():
|
||||
print("FID manifest 不存在,先自动建立。")
|
||||
print(json.dumps(ensure_manifest(args.dataset_id, manifest_path), ensure_ascii=False, indent=2))
|
||||
result = create_production_plan(
|
||||
db=cfg,
|
||||
dataset_id=args.dataset_id,
|
||||
mapping_path=Path(args.mapping),
|
||||
policy_path=Path(args.policy),
|
||||
manifest_path=manifest_path,
|
||||
output_path=Path(args.output),
|
||||
zooms=parse_zooms(args.zooms),
|
||||
profile=args.profile,
|
||||
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
|
||||
s52_resource_contract_path=(
|
||||
Path(args.s52_resource_contract)
|
||||
if args.s52_resource_contract
|
||||
else None
|
||||
),
|
||||
s52_point_table=args.s52_point_table,
|
||||
s52_boundary_table=args.s52_boundary_table,
|
||||
s52_mariner_mode=args.s52_mariner_mode,
|
||||
aoi_name=args.aoi_name,
|
||||
aoi_bbox_wgs84=args.aoi_bbox,
|
||||
tile_enumeration_mode=args.tile_enumeration_mode,
|
||||
)
|
||||
print(json.dumps({
|
||||
"plan_id": result["plan_id"],
|
||||
"release_id": result["release_id"],
|
||||
"job_count": result["job_count"],
|
||||
"production_pipeline_version": result.get("production_pipeline_version"),
|
||||
"s52": result.get("s52"),
|
||||
"aoi": result.get("aoi"),
|
||||
"tile_enumeration_mode": result.get("tile_enumeration_mode"),
|
||||
"source_stats": result["source_stats"],
|
||||
"output": str(Path(args.output)),
|
||||
}, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
if args.command == "build-production":
|
||||
result = run_production_plan(
|
||||
db=cfg,
|
||||
plan_path=Path(args.plan),
|
||||
release_root=Path(args.release_root),
|
||||
workers=args.workers,
|
||||
max_attempts=args.max_attempts,
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
states = result.get("states") or {}
|
||||
if states.get("FAILED", 0) or states.get("PENDING", 0) or states.get("RUNNING", 0):
|
||||
raise SystemExit(2)
|
||||
return
|
||||
|
||||
if args.command == "migrate-production-release":
|
||||
result = migrate_production_release(
|
||||
old_release_dir=Path(args.old_release_dir),
|
||||
new_plan_path=Path(args.new_plan),
|
||||
release_root=Path(args.release_root),
|
||||
copy_mode=args.copy_mode,
|
||||
invalidate_tiles_file=(Path(args.invalidate_tiles_file) if args.invalidate_tiles_file else None),
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
if args.command == "status-production":
|
||||
print(json.dumps(production_status(Path(args.plan), Path(args.release_root)), ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
if args.command == "finalize-production":
|
||||
result = finalize_release(
|
||||
Path(args.plan),
|
||||
Path(args.release_root),
|
||||
None if args.skip_style_contract else (Path(args.style) if args.style else None),
|
||||
None if args.skip_style_contract else (Path(args.sprite_json) if args.sprite_json else None),
|
||||
Path(args.gap_disposition) if args.gap_disposition else None,
|
||||
Path(args.presentation_conservation_audit) if args.presentation_conservation_audit else None,
|
||||
Path(args.runtime_capability_audit) if args.runtime_capability_audit else None,
|
||||
)
|
||||
print(json.dumps({
|
||||
"status": result["status"],
|
||||
"release_id": result["release_id"],
|
||||
"planned_tile_jobs": result["planned_tile_jobs"],
|
||||
"pbf_file_count": result["pbf_file_count"],
|
||||
"planned_output_features": result["planned_output_features"],
|
||||
"decoded_pbf_features": result["decoded_pbf_features"],
|
||||
"semantic_completeness": result.get("semantic_completeness"),
|
||||
"semantic_backlog": result.get("semantic_backlog"),
|
||||
"non_renderable_omissions": result.get("non_renderable_omissions"),
|
||||
"non_renderable_omission_layers": result.get("non_renderable_omission_layers"),
|
||||
"tile_footprint_contract": result.get("tile_footprint_contract"),
|
||||
"style_sprite_contract": {
|
||||
"overall": (result.get("style_sprite_contract") or {}).get("overall"),
|
||||
"contract_unmatched_count": (result.get("style_sprite_contract") or {}).get("contract_unmatched_count"),
|
||||
"sprite_missing_count": (result.get("style_sprite_contract") or {}).get("sprite_missing_count"),
|
||||
},
|
||||
"s52_resource_contract": result.get("s52_resource_contract"),
|
||||
"s52_runtime_contract": result.get("s52_runtime_contract"),
|
||||
"s52_light_presentation_contract": result.get("s52_light_presentation_contract"),
|
||||
"s52_presentation_conservation_contract": result.get("s52_presentation_conservation_contract"),
|
||||
"s52_runtime_capability_contract": result.get("s52_runtime_capability_contract"),
|
||||
"quantization_fallbacks": result.get("quantization_fallbacks"),
|
||||
"quantization_omissions": result.get("quantization_omissions"),
|
||||
"nav_name_conflict_audit": {
|
||||
"overall": (result.get("nav_name_conflict_audit") or {}).get("overall"),
|
||||
"policy": (result.get("nav_name_conflict_audit") or {}).get("policy"),
|
||||
"report_occurrences": (result.get("nav_name_conflict_audit") or {}).get("report_occurrences"),
|
||||
"trace_occurrences": (result.get("nav_name_conflict_audit") or {}).get("trace_occurrences"),
|
||||
"safe_omission_occurrences": (result.get("nav_name_conflict_audit") or {}).get("safe_omission_occurrences"),
|
||||
"unique_conflict_outputs": (result.get("nav_name_conflict_audit") or {}).get("unique_conflict_outputs"),
|
||||
"affected_tiles": (result.get("nav_name_conflict_audit") or {}).get("affected_tiles"),
|
||||
},
|
||||
"failures": result["failures"],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
if result["status"] != "PASS":
|
||||
raise SystemExit(2)
|
||||
return
|
||||
|
||||
if args.command == "build-fid-manifest":
|
||||
with connect(cfg) as conn:
|
||||
result = build_manifest(
|
||||
conn,
|
||||
args.dataset_id,
|
||||
Path(args.manifest),
|
||||
rebuild=bool(args.rebuild_fid_manifest),
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
manifest_path = Path(args.manifest)
|
||||
if not manifest_path.exists():
|
||||
print("FID manifest 不存在,先自动建立。")
|
||||
print(
|
||||
json.dumps(
|
||||
ensure_manifest(args.dataset_id, manifest_path),
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
|
||||
if args.command == "build-tile":
|
||||
builder = Cm95TileBuilder(
|
||||
db=cfg,
|
||||
dataset_id=args.dataset_id,
|
||||
mapping_path=Path(args.mapping),
|
||||
policy_path=Path(args.policy),
|
||||
fid_manifest_path=manifest_path,
|
||||
output_root=Path(args.output_root) / args.profile,
|
||||
report_root=Path(args.report_root),
|
||||
profile=args.profile,
|
||||
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
|
||||
s52_point_table=args.s52_point_table,
|
||||
s52_boundary_table=args.s52_boundary_table,
|
||||
s52_mariner_mode=args.s52_mariner_mode,
|
||||
)
|
||||
result = builder.build_tile(
|
||||
TileJob(args.z, args.x, args.y)
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
if args.command == "build-sample-aoi":
|
||||
profiles = [
|
||||
token.strip()
|
||||
for token in str(args.profiles).split(",")
|
||||
if token.strip()
|
||||
]
|
||||
for profile in profiles:
|
||||
if profile not in {"engineering", "delivery"}:
|
||||
raise SystemExit(f"invalid profile: {profile}")
|
||||
|
||||
jobs = [(10, args.z10_x, args.z10_y)]
|
||||
jobs.extend(child_tiles(10, args.z10_x, args.z10_y, 11))
|
||||
jobs.extend(child_tiles(10, args.z10_x, args.z10_y, 12))
|
||||
|
||||
results = []
|
||||
for profile in profiles:
|
||||
builder = Cm95TileBuilder(
|
||||
db=cfg,
|
||||
dataset_id=args.dataset_id,
|
||||
mapping_path=Path(args.mapping),
|
||||
policy_path=Path(args.policy),
|
||||
fid_manifest_path=manifest_path,
|
||||
output_root=Path(args.output_root) / profile,
|
||||
report_root=Path(args.report_root),
|
||||
profile=profile,
|
||||
s52_lookups_path=(Path(args.s52_lookups) if args.s52_lookups else None),
|
||||
s52_point_table=args.s52_point_table,
|
||||
s52_boundary_table=args.s52_boundary_table,
|
||||
s52_mariner_mode=args.s52_mariner_mode,
|
||||
)
|
||||
for z, x, y in jobs:
|
||||
result = builder.build_tile(TileJob(z, x, y))
|
||||
results.append(result)
|
||||
print(
|
||||
f"[{profile}] {z}/{x}/{y} "
|
||||
f"features={result['output_feature_count']} "
|
||||
f"layers={result['output_layer_count']} "
|
||||
f"hard_gate_unmapped={len(result['hard_gate_unmapped'])}"
|
||||
)
|
||||
|
||||
summary = {
|
||||
"dataset_id": args.dataset_id,
|
||||
"root_tile": f"10/{args.z10_x}/{args.z10_y}",
|
||||
"profiles": profiles,
|
||||
"tile_count_per_profile": len(jobs),
|
||||
"results": results,
|
||||
}
|
||||
summary_path = Path(args.report_root) / "sample_aoi_summary_v1.json"
|
||||
summary_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
summary_path.write_text(
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(f"SUMMARY: {summary_path}")
|
||||
return
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
1124
src/cm95pbf/csp.py
Normal file
1124
src/cm95pbf/csp.py
Normal file
File diff suppressed because it is too large
Load Diff
329
src/cm95pbf/identity.py
Normal file
329
src/cm95pbf/identity.py
Normal file
@@ -0,0 +1,329 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
from typing import Iterable
|
||||
|
||||
ROLE_PRIMARY = 0
|
||||
ROLE_BOUNDARY = 1
|
||||
ROLE_POINT_ON_SURFACE = 2
|
||||
ROLE_SECONDARY = 3
|
||||
ROLE_FACTOR = 16
|
||||
MAX_SIGNED_INT32 = (1 << 31) - 1
|
||||
|
||||
|
||||
def feature_source_key(
|
||||
dataset_id: int,
|
||||
source_relative_path: str,
|
||||
source_record_index: int,
|
||||
geometry_type: str,
|
||||
) -> str:
|
||||
path = str(source_relative_path).replace("\\", "/")
|
||||
return (
|
||||
f"cm95:{int(dataset_id)}:{path}:"
|
||||
f"{int(source_record_index)}:{str(geometry_type).lower()}"
|
||||
)
|
||||
|
||||
|
||||
def sounding_source_key(
|
||||
dataset_id: int,
|
||||
source_relative_path: str,
|
||||
source_record_index: int,
|
||||
point_index: int,
|
||||
) -> str:
|
||||
return (
|
||||
feature_source_key(
|
||||
dataset_id,
|
||||
source_relative_path,
|
||||
source_record_index,
|
||||
"SOUNDING",
|
||||
)
|
||||
+ f":point:{int(point_index)}"
|
||||
)
|
||||
|
||||
|
||||
def dataset_fingerprint(conn, dataset_id: int) -> str:
|
||||
h = hashlib.sha256()
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
source_relative_path,
|
||||
COALESCE(source_sha256,'') AS source_sha256,
|
||||
source_size,
|
||||
scale_code,
|
||||
native_scale
|
||||
FROM chart_cell
|
||||
WHERE dataset_id=%s
|
||||
ORDER BY source_relative_path
|
||||
""",
|
||||
(dataset_id,),
|
||||
)
|
||||
for row in cur.fetchall():
|
||||
line = (
|
||||
f"{row['source_relative_path']}|{row['source_sha256']}|"
|
||||
f"{row['source_size']}|{row['scale_code']}|{row['native_scale']}\n"
|
||||
)
|
||||
h.update(line.encode("utf-8"))
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
class FidManifest:
|
||||
def __init__(self, path: Path):
|
||||
self.path = Path(path)
|
||||
self.db: sqlite3.Connection | None = None
|
||||
|
||||
def open(self) -> "FidManifest":
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.db = sqlite3.connect(self.path)
|
||||
self.db.row_factory = sqlite3.Row
|
||||
return self
|
||||
|
||||
def close(self) -> None:
|
||||
if self.db is not None:
|
||||
self.db.close()
|
||||
self.db = None
|
||||
|
||||
def __enter__(self) -> "FidManifest":
|
||||
return self.open()
|
||||
|
||||
def __exit__(self, exc_type, exc, tb) -> None:
|
||||
self.close()
|
||||
|
||||
def _require(self) -> sqlite3.Connection:
|
||||
if self.db is None:
|
||||
raise RuntimeError("manifest not open")
|
||||
return self.db
|
||||
|
||||
def init_schema(self) -> None:
|
||||
db = self._require()
|
||||
db.executescript(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS manifest_meta (
|
||||
k TEXT PRIMARY KEY,
|
||||
v TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS manifest_identity (
|
||||
source_key TEXT PRIMARY KEY,
|
||||
base_ord INTEGER NOT NULL UNIQUE,
|
||||
source_kind TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_manifest_identity_ord
|
||||
ON manifest_identity(base_ord);
|
||||
"""
|
||||
)
|
||||
db.commit()
|
||||
|
||||
def clear(self) -> None:
|
||||
db = self._require()
|
||||
self.init_schema()
|
||||
db.execute("DELETE FROM manifest_identity")
|
||||
db.execute("DELETE FROM manifest_meta")
|
||||
db.commit()
|
||||
|
||||
def set_meta(self, key: str, value: str) -> None:
|
||||
db = self._require()
|
||||
db.execute(
|
||||
"INSERT OR REPLACE INTO manifest_meta(k,v) VALUES(?,?)",
|
||||
(str(key), str(value)),
|
||||
)
|
||||
|
||||
def get_meta(self, key: str) -> str | None:
|
||||
db = self._require()
|
||||
row = db.execute(
|
||||
"SELECT v FROM manifest_meta WHERE k=?",
|
||||
(str(key),),
|
||||
).fetchone()
|
||||
return None if row is None else str(row["v"])
|
||||
|
||||
def insert_many(self, rows: Iterable[tuple[str, int, str]]) -> None:
|
||||
db = self._require()
|
||||
db.executemany(
|
||||
"""
|
||||
INSERT INTO manifest_identity(source_key,base_ord,source_kind)
|
||||
VALUES(?,?,?)
|
||||
""",
|
||||
rows,
|
||||
)
|
||||
|
||||
def commit(self) -> None:
|
||||
self._require().commit()
|
||||
|
||||
def base_ord(self, source_key: str) -> int:
|
||||
db = self._require()
|
||||
row = db.execute(
|
||||
"SELECT base_ord FROM manifest_identity WHERE source_key=?",
|
||||
(source_key,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise KeyError(f"source identity missing from manifest: {source_key}")
|
||||
return int(row["base_ord"])
|
||||
|
||||
def mvt_id(self, source_key: str, role_code: int = ROLE_PRIMARY) -> int:
|
||||
if role_code < 0 or role_code >= ROLE_FACTOR:
|
||||
raise ValueError(f"invalid role code: {role_code}")
|
||||
value = self.base_ord(source_key) * ROLE_FACTOR + int(role_code)
|
||||
if value > MAX_SIGNED_INT32:
|
||||
raise RuntimeError(
|
||||
"deterministic fid exceeded signed int32; "
|
||||
"increase identity design before publishing"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def build_manifest(
|
||||
mysql_conn,
|
||||
dataset_id: int,
|
||||
manifest_path: Path,
|
||||
rebuild: bool = False,
|
||||
) -> dict:
|
||||
fingerprint = dataset_fingerprint(mysql_conn, dataset_id)
|
||||
|
||||
with FidManifest(manifest_path) as manifest:
|
||||
manifest.init_schema()
|
||||
|
||||
old_dataset = manifest.get_meta("dataset_id")
|
||||
old_fingerprint = manifest.get_meta("dataset_fingerprint")
|
||||
|
||||
if old_dataset is not None and not rebuild:
|
||||
if old_dataset == str(dataset_id) and old_fingerprint == fingerprint:
|
||||
count = int(
|
||||
manifest._require()
|
||||
.execute("SELECT COUNT(*) FROM manifest_identity")
|
||||
.fetchone()[0]
|
||||
)
|
||||
return {
|
||||
"status": "REUSED",
|
||||
"dataset_id": dataset_id,
|
||||
"dataset_fingerprint": fingerprint,
|
||||
"identity_count": count,
|
||||
"manifest": str(manifest_path),
|
||||
}
|
||||
raise RuntimeError(
|
||||
"existing fid manifest belongs to a different dataset state; "
|
||||
"run with --rebuild-fid-manifest"
|
||||
)
|
||||
|
||||
manifest.clear()
|
||||
|
||||
base_ord = 0
|
||||
batch: list[tuple[str, int, str]] = []
|
||||
|
||||
with mysql_conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
c.source_relative_path,
|
||||
f.source_record_index,
|
||||
f.geometry_type
|
||||
FROM chart_feature f
|
||||
JOIN chart_cell c ON c.id=f.cell_id
|
||||
WHERE f.dataset_id=%s
|
||||
ORDER BY
|
||||
c.source_relative_path,
|
||||
f.source_record_index,
|
||||
f.geometry_type
|
||||
""",
|
||||
(dataset_id,),
|
||||
)
|
||||
while True:
|
||||
mysql_batch = cur.fetchmany(5000)
|
||||
if not mysql_batch:
|
||||
break
|
||||
for row in mysql_batch:
|
||||
base_ord += 1
|
||||
batch.append(
|
||||
(
|
||||
feature_source_key(
|
||||
dataset_id,
|
||||
row["source_relative_path"],
|
||||
row["source_record_index"],
|
||||
row["geometry_type"],
|
||||
),
|
||||
base_ord,
|
||||
"feature",
|
||||
)
|
||||
)
|
||||
manifest.insert_many(batch)
|
||||
batch.clear()
|
||||
|
||||
feature_identity_count = base_ord
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
c.source_relative_path,
|
||||
f.source_record_index,
|
||||
s.point_index
|
||||
FROM chart_sounding_point s
|
||||
JOIN chart_feature f ON f.id=s.feature_id
|
||||
JOIN chart_cell c ON c.id=f.cell_id
|
||||
WHERE f.dataset_id=%s
|
||||
ORDER BY
|
||||
c.source_relative_path,
|
||||
f.source_record_index,
|
||||
s.point_index
|
||||
""",
|
||||
(dataset_id,),
|
||||
)
|
||||
while True:
|
||||
mysql_batch = cur.fetchmany(5000)
|
||||
if not mysql_batch:
|
||||
break
|
||||
for row in mysql_batch:
|
||||
base_ord += 1
|
||||
batch.append(
|
||||
(
|
||||
sounding_source_key(
|
||||
dataset_id,
|
||||
row["source_relative_path"],
|
||||
row["source_record_index"],
|
||||
row["point_index"],
|
||||
),
|
||||
base_ord,
|
||||
"sounding_point",
|
||||
)
|
||||
)
|
||||
manifest.insert_many(batch)
|
||||
batch.clear()
|
||||
|
||||
manifest.set_meta("schema", "navsea.cm95.fid-manifest/v1")
|
||||
manifest.set_meta("dataset_id", str(dataset_id))
|
||||
manifest.set_meta("dataset_fingerprint", fingerprint)
|
||||
manifest.set_meta("role_factor", str(ROLE_FACTOR))
|
||||
manifest.set_meta("feature_identity_count", str(feature_identity_count))
|
||||
manifest.set_meta("identity_count", str(base_ord))
|
||||
manifest.commit()
|
||||
|
||||
return {
|
||||
"status": "BUILT",
|
||||
"dataset_id": dataset_id,
|
||||
"dataset_fingerprint": fingerprint,
|
||||
"feature_identity_count": feature_identity_count,
|
||||
"identity_count": base_ord,
|
||||
"manifest": str(manifest_path),
|
||||
}
|
||||
|
||||
|
||||
def validate_manifest(mysql_conn, dataset_id: int, manifest_path: Path) -> dict:
|
||||
fingerprint = dataset_fingerprint(mysql_conn, dataset_id)
|
||||
with FidManifest(manifest_path) as manifest:
|
||||
manifest.init_schema()
|
||||
dataset = manifest.get_meta("dataset_id")
|
||||
stored_fp = manifest.get_meta("dataset_fingerprint")
|
||||
if dataset != str(dataset_id) or stored_fp != fingerprint:
|
||||
raise RuntimeError(
|
||||
"fid manifest does not match current dataset; rebuild required"
|
||||
)
|
||||
count = int(
|
||||
manifest._require()
|
||||
.execute("SELECT COUNT(*) FROM manifest_identity")
|
||||
.fetchone()[0]
|
||||
)
|
||||
return {
|
||||
"dataset_id": dataset_id,
|
||||
"dataset_fingerprint": fingerprint,
|
||||
"identity_count": count,
|
||||
"manifest": str(manifest_path),
|
||||
}
|
||||
633
src/cm95pbf/mapping.py
Normal file
633
src/cm95pbf/mapping.py
Normal file
@@ -0,0 +1,633 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
DELIVERY_PROPERTY_ALLOWLIST = frozenset(
|
||||
{
|
||||
"canonical_object_type",
|
||||
"class_code",
|
||||
"chart_fill_pattern",
|
||||
"chart_fill_style",
|
||||
"chart_icon_image",
|
||||
"chart_label_position_code",
|
||||
"chart_label_subtext",
|
||||
"chart_label_text",
|
||||
"chart_line_color",
|
||||
"chart_line_width",
|
||||
"chart_symbol_code",
|
||||
"chart_text_color",
|
||||
"chart_text_style",
|
||||
"clearance_height_m",
|
||||
"contour_depth_m",
|
||||
"depth_max_m",
|
||||
"depth_min_m",
|
||||
"depth_value_m",
|
||||
"display_code",
|
||||
"least_depth_m",
|
||||
"light_color_code",
|
||||
"light_sector_mode",
|
||||
"arc_id",
|
||||
"arc_source_code",
|
||||
"name_ja",
|
||||
"place_name_en",
|
||||
"s52_csp",
|
||||
"s52_csp_resolved",
|
||||
"s52_csp_rule_opcodes",
|
||||
"s52_mariner_dependent",
|
||||
"s52_runtime_csp",
|
||||
"s52_is_safety_contour",
|
||||
"s52_display_category",
|
||||
"s52_area_color",
|
||||
"s52_area_pattern_code",
|
||||
"s52_arc_color",
|
||||
"s52_arc_end_deg",
|
||||
"s52_arc_outline_color",
|
||||
"s52_arc_outline_width",
|
||||
"s52_arc_radius_mm",
|
||||
"s52_arc_sector_radius_mm",
|
||||
"s52_arc_start_deg",
|
||||
"s52_arc_width",
|
||||
"s52_has_text",
|
||||
"s52_display_base_promoted",
|
||||
"s52_hazard_depth_m",
|
||||
"s52_hazard_procedure",
|
||||
"s52_hazard_quality_rule_chain",
|
||||
"s52_hazard_name_rule_chain",
|
||||
"s52_valsou_present",
|
||||
"s52_valsou_m",
|
||||
"s52_isolated_danger",
|
||||
"s52_surrounding_max_drval1_m",
|
||||
"s52_surrounding_min_drval2_m",
|
||||
"s52_quapos_code",
|
||||
"s52_quasou_codes",
|
||||
"s52_tecsou_codes",
|
||||
"s52_status_codes",
|
||||
"s52_watlev_code",
|
||||
"s52_expsou_code",
|
||||
"s52_catobs_code",
|
||||
"s52_catwrk_code",
|
||||
"s52_quapos_code_present",
|
||||
"s52_watlev_code_present",
|
||||
"s52_expsou_code_present",
|
||||
"s52_catobs_code_present",
|
||||
"s52_catwrk_code_present",
|
||||
"s52_line_color",
|
||||
"s52_line_complex_code",
|
||||
"s52_line_count",
|
||||
"s52_line_sequence",
|
||||
"s52_line_style",
|
||||
"s52_line_width",
|
||||
"s52_line1_style",
|
||||
"s52_line1_width",
|
||||
"s52_line1_color",
|
||||
"s52_line2_style",
|
||||
"s52_line2_width",
|
||||
"s52_line2_color",
|
||||
"s52_rule_opcodes",
|
||||
"s52_rule_chain",
|
||||
"s52_rule_chain_version",
|
||||
"s52_source_object_class",
|
||||
"s52_source_geometry",
|
||||
"s52_scamin_override",
|
||||
"s52_safe_dashed_rule_chain",
|
||||
"s52_safe_solid_rule_chain",
|
||||
"s52_resare_plain_rule_chain",
|
||||
"s52_resare_symbolized_rule_chain",
|
||||
"s52_lights_metres_rule_chain",
|
||||
"s52_lights_feet_rule_chain",
|
||||
"s52_light_arc_color",
|
||||
"s52_light_arc_radius_mm",
|
||||
"s52_light_mode",
|
||||
"s52_light_nominal_range_nm",
|
||||
"s52_light_obscured",
|
||||
"s52_light_label_text",
|
||||
"s52_light_presentations",
|
||||
"s52_light_rotation_deg",
|
||||
"s52_light_sector_radius_mm",
|
||||
"s52_light_symbol_code",
|
||||
"s52_sector_display_end_deg",
|
||||
"s52_sector_display_start_deg",
|
||||
"s52_sector_end_deg",
|
||||
"s52_sector_start_deg",
|
||||
"s52_sector_sweep_deg",
|
||||
"s52_display_priority",
|
||||
"s52_priority_override",
|
||||
"s52_lookup_rcid",
|
||||
"s52_object_class",
|
||||
"s52_symbol_code",
|
||||
"s52_symbol_count",
|
||||
"s52_symbol_rotation_deg",
|
||||
"s52_symbol_rotation_source",
|
||||
"s52_symbol_rule_sequence",
|
||||
"s52_symbol_sequence",
|
||||
"s52_sounding_depth_m",
|
||||
"s52_sounding_display_depth",
|
||||
"s52_sounding_depth_unit",
|
||||
"s52_sounding_is_shallow",
|
||||
"s52_sounding_low_quality",
|
||||
"s52_sounding_prefix",
|
||||
"s52_sounding_swept",
|
||||
"s52_sounding_symbol_sequence",
|
||||
"s52_soundg_point_csp",
|
||||
"s52_table_name",
|
||||
"s52_text_literal",
|
||||
"s52_text1_value",
|
||||
"s52_text1_hjust",
|
||||
"s52_text1_vjust",
|
||||
"s52_text1_space",
|
||||
"s52_text1_size",
|
||||
"s52_text1_body_size",
|
||||
"s52_text1_charspec",
|
||||
"s52_text1_xoff",
|
||||
"s52_text1_yoff",
|
||||
"s52_text1_color",
|
||||
"s52_text1_group",
|
||||
"s52_text2_value",
|
||||
"s52_text2_hjust",
|
||||
"s52_text2_vjust",
|
||||
"s52_text2_space",
|
||||
"s52_text2_size",
|
||||
"s52_text2_body_size",
|
||||
"s52_text2_charspec",
|
||||
"s52_text2_xoff",
|
||||
"s52_text2_yoff",
|
||||
"s52_text2_color",
|
||||
"s52_text2_group",
|
||||
"s52_text3_value",
|
||||
"s52_text3_hjust",
|
||||
"s52_text3_vjust",
|
||||
"s52_text3_space",
|
||||
"s52_text3_size",
|
||||
"s52_text3_body_size",
|
||||
"s52_text3_charspec",
|
||||
"s52_text3_xoff",
|
||||
"s52_text3_yoff",
|
||||
"s52_text3_color",
|
||||
"s52_text3_group",
|
||||
"s52_text_count",
|
||||
"shape_class_code",
|
||||
}
|
||||
)
|
||||
|
||||
DELIVERY_MAIN_STATUSES = frozenset(
|
||||
{
|
||||
"MAPPED",
|
||||
"MAPPED_CONDITIONAL",
|
||||
"COMPOSITE",
|
||||
"MAPPED_DERIVED_POINTS",
|
||||
}
|
||||
)
|
||||
|
||||
DELIVERY_DERIVED_STATUSES = frozenset(
|
||||
{
|
||||
"MAPPED",
|
||||
"MAPPED_CONDITIONAL",
|
||||
"DERIVED_LABEL_ONLY",
|
||||
}
|
||||
)
|
||||
|
||||
NAV_COMPONENT_CLASSES = frozenset(
|
||||
{
|
||||
"LIGHTS",
|
||||
"BCNLAT",
|
||||
"BCNSPP",
|
||||
"BOYSPP",
|
||||
"RTPBCN",
|
||||
"TOWERS",
|
||||
"_extgn",
|
||||
}
|
||||
)
|
||||
|
||||
LIGHT_COLOUR = {
|
||||
1: ("white", "W"),
|
||||
3: ("red", "R"),
|
||||
4: ("green", "G"),
|
||||
6: ("yellow", "Y"),
|
||||
}
|
||||
|
||||
LIGHT_CHARACTER = {
|
||||
1: "F",
|
||||
2: "Fl",
|
||||
3: "LFl",
|
||||
4: "Q",
|
||||
5: "VQ",
|
||||
6: "UQ",
|
||||
7: "Iso",
|
||||
8: "Oc",
|
||||
9: "IQ",
|
||||
10: "IVQ",
|
||||
11: "IUQ",
|
||||
12: "Mo",
|
||||
13: "F+Fl",
|
||||
14: "Fl+LFl",
|
||||
15: "Oc+Fl",
|
||||
16: "F+LFl",
|
||||
17: "Al.Oc",
|
||||
18: "Al.LFl",
|
||||
19: "Al.Fl",
|
||||
20: "Al.Gr",
|
||||
21: "Q+LFl",
|
||||
22: "VQ+LFl",
|
||||
23: "UQ+LFl",
|
||||
}
|
||||
|
||||
NATSUR_TEXT = {
|
||||
1: "M",
|
||||
2: "Cy",
|
||||
3: "Si",
|
||||
4: "S",
|
||||
5: "St",
|
||||
6: "G",
|
||||
7: "P",
|
||||
8: "Cb",
|
||||
9: "R",
|
||||
11: "Lv",
|
||||
14: "Co",
|
||||
17: "Sh",
|
||||
18: "Bo",
|
||||
}
|
||||
|
||||
|
||||
def scalarize(value: Any) -> Any:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, (int, float, str)):
|
||||
return value
|
||||
if isinstance(value, (list, dict, tuple)):
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
return str(value)
|
||||
|
||||
|
||||
def get_path(data: dict[str, Any], path: str) -> Any:
|
||||
cur: Any = data
|
||||
for token in str(path).split("."):
|
||||
if isinstance(cur, dict) and token in cur:
|
||||
cur = cur[token]
|
||||
else:
|
||||
return None
|
||||
return cur
|
||||
|
||||
|
||||
|
||||
def first_direct_name(attrs: dict[str, Any]) -> str | None:
|
||||
"""Return a direct source-owned navigation name, never a proximity-derived text."""
|
||||
for key in ("OBJNAM", "NOBJNM"):
|
||||
value = attrs.get(key)
|
||||
if isinstance(value, str):
|
||||
value = value.strip()
|
||||
if value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def unique_direct_names(rows: list[dict[str, Any]]) -> list[str]:
|
||||
out: list[str] = []
|
||||
for attrs in rows:
|
||||
value = first_direct_name(attrs)
|
||||
if value and value not in out:
|
||||
out.append(value)
|
||||
return out
|
||||
|
||||
|
||||
def format_number(value: Any) -> str:
|
||||
if value is None:
|
||||
return ""
|
||||
try:
|
||||
v = float(value)
|
||||
except Exception:
|
||||
return str(value)
|
||||
if abs(v - round(v)) < 1e-9:
|
||||
return str(int(round(v)))
|
||||
return f"{v:.2f}".rstrip("0").rstrip(".")
|
||||
|
||||
|
||||
def format_depth(value: Any) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
return format_number(value)
|
||||
|
||||
|
||||
def format_clearance(value: Any) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
return f"{format_number(value)}m"
|
||||
|
||||
|
||||
def _depth_token(value: Any) -> str:
|
||||
return format_number(value)
|
||||
|
||||
|
||||
# Current NavSea unified style only consumes these depth-area buckets.
|
||||
# Keep this list aligned with style.navsea-delivery-full-semantic-unified-20260808.
|
||||
NAVSEA_DEPTH_ZONE_INTERVALS: tuple[tuple[float, float, str], ...] = (
|
||||
(0, 2, "depth_zone_0-2m"),
|
||||
(0, 5, "depth_zone_0-5m"),
|
||||
(0, 10, "depth_zone_0-10m"),
|
||||
(0, 20, "depth_zone_0-20m"),
|
||||
(2, 5, "depth_zone_2-5m"),
|
||||
(2, 10, "depth_zone_2-10m"),
|
||||
(2, 20, "depth_zone_2-20m"),
|
||||
(5, 10, "depth_zone_5-10m"),
|
||||
(5, 20, "depth_zone_5-20m"),
|
||||
(10, 20, "depth_zone_10-20m"),
|
||||
(20, 100, "depth_zone_20-100m"),
|
||||
(20, 200, "depth_zone_20-200m"),
|
||||
(100, 200, "depth_zone_100-200m"),
|
||||
(200, 500, "depth_zone_200-500m"),
|
||||
(500, 1000, "depth_zone_500-1000m"),
|
||||
(1000, 2000, "depth_zone_1000-2000m"),
|
||||
(2000, 3000, "depth_zone_2000-3000m"),
|
||||
(3000, 4000, "depth_zone_3000-4000m"),
|
||||
(4000, 5000, "depth_zone_4000-5000m"),
|
||||
(5000, 6000, "depth_zone_5000-6000m"),
|
||||
(6000, 7000, "depth_zone_6000-7000m"),
|
||||
(7000, 8000, "depth_zone_7000-8000m"),
|
||||
(8000, 9000, "depth_zone_8000-9000m"),
|
||||
)
|
||||
|
||||
|
||||
def _smallest_supported_depth_bucket(lo: float, hi: float | None) -> str | None:
|
||||
if lo >= 9000 and (hi is None or hi >= lo):
|
||||
return "depth_zone_over_9000m"
|
||||
|
||||
candidates: list[tuple[float, float, float, str]] = []
|
||||
for bucket_lo, bucket_hi, token in NAVSEA_DEPTH_ZONE_INTERVALS:
|
||||
if hi is None:
|
||||
# A lower-bounded zone uses the tightest bucket containing the lower bound.
|
||||
if bucket_lo <= lo < bucket_hi:
|
||||
width = bucket_hi - bucket_lo
|
||||
edge_distance = abs(bucket_lo - lo)
|
||||
candidates.append((width, edge_distance, -bucket_lo, token))
|
||||
else:
|
||||
# Prefer the smallest supported interval that fully contains the source range.
|
||||
if bucket_lo <= lo and hi <= bucket_hi:
|
||||
width = bucket_hi - bucket_lo
|
||||
edge_distance = abs(bucket_lo - lo) + abs(bucket_hi - hi)
|
||||
candidates.append((width, edge_distance, -bucket_lo, token))
|
||||
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort()
|
||||
return candidates[0][3]
|
||||
|
||||
|
||||
def depth_zone_style(lo: Any, hi: Any) -> str:
|
||||
"""Normalize CM95 depth ranges to the existing NavSea style vocabulary.
|
||||
|
||||
Arbitrary CM95 intervals such as 7.5-10m, 13-20m or 20-50m must never
|
||||
leak into delivery PBF because the current style does not consume them.
|
||||
The chosen style bucket is the smallest existing NavSea interval that fully
|
||||
contains the source range. The original DRVAL1/DRVAL2 remain available in
|
||||
engineering trace/source attributes.
|
||||
"""
|
||||
if lo is None and hi is None:
|
||||
return "water_area"
|
||||
|
||||
try:
|
||||
lv = float(lo) if lo is not None else 0.0
|
||||
except Exception:
|
||||
return "water_area"
|
||||
|
||||
hv: float | None
|
||||
if hi is None:
|
||||
hv = None
|
||||
else:
|
||||
try:
|
||||
hv = float(hi)
|
||||
except Exception:
|
||||
hv = None
|
||||
|
||||
if hv is not None and hv < lv:
|
||||
lv, hv = hv, lv
|
||||
|
||||
token = _smallest_supported_depth_bucket(lv, hv)
|
||||
if token is not None:
|
||||
return token
|
||||
|
||||
# No truthful single bucket exists in the current style vocabulary.
|
||||
# Use neutral water rather than inventing a style value that will not render.
|
||||
return "water_area"
|
||||
|
||||
|
||||
def format_natsur(natsur: Any, natqua: Any) -> str | None:
|
||||
values = natsur if isinstance(natsur, list) else [natsur] if natsur is not None else []
|
||||
parts: list[str] = []
|
||||
for value in values:
|
||||
try:
|
||||
code = int(value)
|
||||
except Exception:
|
||||
code = None
|
||||
text = NATSUR_TEXT.get(code, str(value) if value is not None else "")
|
||||
if text and text not in parts:
|
||||
parts.append(text)
|
||||
if not parts:
|
||||
return None
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def s57_light_color(colour: Any) -> str | None:
|
||||
values = colour if isinstance(colour, list) else [colour] if colour is not None else []
|
||||
for value in values:
|
||||
try:
|
||||
code = int(value)
|
||||
except Exception:
|
||||
continue
|
||||
if code in LIGHT_COLOUR:
|
||||
return LIGHT_COLOUR[code][0]
|
||||
return None
|
||||
|
||||
|
||||
def light_colour_abbrs(colour: Any) -> list[str]:
|
||||
values = colour if isinstance(colour, list) else [colour] if colour is not None else []
|
||||
out: list[str] = []
|
||||
for value in values:
|
||||
try:
|
||||
code = int(value)
|
||||
except Exception:
|
||||
continue
|
||||
item = LIGHT_COLOUR.get(code)
|
||||
if item and item[1] not in out:
|
||||
out.append(item[1])
|
||||
return out
|
||||
|
||||
|
||||
def format_light_abbreviation(attrs: dict[str, Any]) -> str | None:
|
||||
parts: list[str] = []
|
||||
litchr = attrs.get("LITCHR")
|
||||
if litchr is not None:
|
||||
try:
|
||||
char = LIGHT_CHARACTER.get(int(litchr), f"L{int(litchr)}")
|
||||
except Exception:
|
||||
char = str(litchr)
|
||||
siggrp = attrs.get("SIGGRP")
|
||||
if siggrp and str(siggrp) not in {"(1)", "1"}:
|
||||
char += str(siggrp)
|
||||
parts.append(char)
|
||||
|
||||
colours = light_colour_abbrs(attrs.get("COLOUR"))
|
||||
if colours:
|
||||
parts.append(",".join(colours))
|
||||
|
||||
if attrs.get("SIGPER") is not None:
|
||||
parts.append(f"{format_number(attrs['SIGPER'])}s")
|
||||
if attrs.get("HEIGHT") is not None:
|
||||
parts.append(f"{format_number(attrs['HEIGHT'])}m")
|
||||
if attrs.get("VALNMR") is not None:
|
||||
parts.append(f"{format_number(attrs['VALNMR'])}M")
|
||||
|
||||
if not parts:
|
||||
colours = light_colour_abbrs(attrs.get("COLOUR"))
|
||||
if colours:
|
||||
return ",".join(colours)
|
||||
return None
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def nav_icon_from_bcnlat(attrs: dict[str, Any], lighted: bool = False) -> str:
|
||||
try:
|
||||
cat = int(attrs.get("CATLAM"))
|
||||
except Exception:
|
||||
cat = 0
|
||||
suffix = {
|
||||
1: "port",
|
||||
2: "starboard",
|
||||
3: "starboard",
|
||||
4: "port",
|
||||
}.get(cat, "special")
|
||||
if lighted:
|
||||
return f"beacon_light_mark_{suffix}"
|
||||
return f"beacon_small_{suffix}"
|
||||
|
||||
|
||||
def nav_icon_from_bcnspp(attrs: dict[str, Any], lighted: bool = False) -> str:
|
||||
return "beacon_light_mark_special" if lighted else "beacon_small_special"
|
||||
|
||||
|
||||
def nav_icon_from_boyspp(attrs: dict[str, Any], lighted: bool = False) -> str:
|
||||
try:
|
||||
shape = int(attrs.get("BOYSHP"))
|
||||
except Exception:
|
||||
shape = 0
|
||||
if lighted:
|
||||
if shape == 4:
|
||||
return "buoy_pillar_light_special"
|
||||
return "buoy_light_special"
|
||||
return {
|
||||
1: "buoy_cone",
|
||||
2: "buoy_cylinder",
|
||||
3: "buoy_sphere",
|
||||
4: "buoy_pillar",
|
||||
6: "buoy_barrel_light",
|
||||
}.get(shape, "buoy_lattice_special")
|
||||
|
||||
|
||||
def nav_icon_from_rtpbcn(attrs: dict[str, Any], lighted: bool = False) -> str:
|
||||
return "beacon_light" if lighted else "beacon_large"
|
||||
|
||||
|
||||
class MappingRegistry:
|
||||
def __init__(self, path: Path):
|
||||
self.path = Path(path)
|
||||
self.document = yaml.safe_load(self.path.read_text(encoding="utf-8"))
|
||||
self.rules: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
for rule in self.document.get("rules", []):
|
||||
key = (
|
||||
str(rule.get("cm95_object_class") or ""),
|
||||
str(rule.get("geometry_type") or ""),
|
||||
)
|
||||
self.rules[key] = rule
|
||||
|
||||
def resolve(self, object_class: str, geometry_type: str) -> dict[str, Any] | None:
|
||||
return self.rules.get((str(object_class), str(geometry_type)))
|
||||
|
||||
def eval_condition(self, condition: Any, attrs: dict[str, Any]) -> bool:
|
||||
if condition is None or condition == "":
|
||||
return True
|
||||
text = str(condition).strip()
|
||||
if text.endswith(" present"):
|
||||
field = text[:-8].strip()
|
||||
return get_path(attrs, field) is not None
|
||||
return False
|
||||
|
||||
def evaluate_fields(
|
||||
self,
|
||||
field_spec: dict[str, Any],
|
||||
attrs: dict[str, Any],
|
||||
context: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
context = context or {}
|
||||
merged = dict(attrs)
|
||||
merged.update(context)
|
||||
out: dict[str, Any] = {}
|
||||
for field, spec in (field_spec or {}).items():
|
||||
value = self._evaluate_spec(spec, merged)
|
||||
value = scalarize(value)
|
||||
if value is not None and value != "":
|
||||
out[str(field)] = value
|
||||
return out
|
||||
|
||||
def _evaluate_spec(self, spec: Any, attrs: dict[str, Any]) -> Any:
|
||||
if not isinstance(spec, dict):
|
||||
return spec
|
||||
if "const" in spec:
|
||||
return spec["const"]
|
||||
if "from" in spec:
|
||||
return get_path(attrs, str(spec["from"]))
|
||||
derive = spec.get("derive")
|
||||
if derive is None:
|
||||
return None
|
||||
text = str(derive)
|
||||
|
||||
if text == "wreck_surveyed if VALSOU present else wreck_submerged_dangerous":
|
||||
return "wreck_surveyed" if attrs.get("VALSOU") is not None else "wreck_submerged_dangerous"
|
||||
if text == "rock_awash if WATLEV in [2,4,5] else rock_submerged":
|
||||
try:
|
||||
watlev = int(attrs.get("WATLEV"))
|
||||
except Exception:
|
||||
watlev = None
|
||||
return "rock_awash" if watlev in {2, 4, 5} else "rock_submerged"
|
||||
if text == "format_depth(VALDCO)":
|
||||
return format_depth(attrs.get("VALDCO"))
|
||||
if text == "format_depth(chart_sounding_point.depth)":
|
||||
return format_depth(get_path(attrs, "chart_sounding_point.depth"))
|
||||
if text == "depth_zone_style(DRVAL1,DRVAL2)":
|
||||
return depth_zone_style(attrs.get("DRVAL1"), attrs.get("DRVAL2"))
|
||||
if text == "depth_zone_style(DRVAL1,null)":
|
||||
return depth_zone_style(attrs.get("DRVAL1"), None)
|
||||
if text == "format_natsur(NATSUR,NATQUA)":
|
||||
return format_natsur(attrs.get("NATSUR"), attrs.get("NATQUA"))
|
||||
if text == "_texta.text":
|
||||
return get_path(attrs, "_texta.text")
|
||||
if text == "s57_colour_to_light_color(COLOUR)":
|
||||
return s57_light_color(attrs.get("COLOUR"))
|
||||
if text == "format_light_abbreviation(LITCHR,SIGGRP,SIGPER,COLOUR,HEIGHT,VALNMR)":
|
||||
return format_light_abbreviation(attrs)
|
||||
if text == "first_present(OBJNAM,NOBJNM)":
|
||||
return first_direct_name(attrs)
|
||||
if text == "sector if SECTR1/SECTR2 present else omni":
|
||||
return "sector" if attrs.get("SECTR1") is not None or attrs.get("SECTR2") is not None else "omni"
|
||||
if text == "format_clearance(VERCLR)":
|
||||
return format_clearance(attrs.get("VERCLR"))
|
||||
if text == "nav_icon_from_bcnlat_attributes":
|
||||
return nav_icon_from_bcnlat(attrs, lighted=False)
|
||||
if text == "nav_icon_from_bcnspp_attributes":
|
||||
return nav_icon_from_bcnspp(attrs, lighted=False)
|
||||
if text == "nav_icon_from_boyspp_attributes":
|
||||
return nav_icon_from_boyspp(attrs, lighted=False)
|
||||
if text == "nav_icon_from_rtpbcn_attributes":
|
||||
return nav_icon_from_rtpbcn(attrs, lighted=False)
|
||||
return None
|
||||
|
||||
def minimize_delivery_properties(self, properties: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
key: scalarize(value)
|
||||
for key, value in properties.items()
|
||||
if key in DELIVERY_PROPERTY_ALLOWLIST and scalarize(value) is not None
|
||||
}
|
||||
98
src/cm95pbf/policy.py
Normal file
98
src/cm95pbf/policy.py
Normal file
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
|
||||
WEB_MERCATOR_HALF_WORLD = 20037508.342789244
|
||||
MAX_MERCATOR_LAT = 85.0511287798066
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TileBounds:
|
||||
z: int
|
||||
x: int
|
||||
y: int
|
||||
west_m: float
|
||||
south_m: float
|
||||
east_m: float
|
||||
north_m: float
|
||||
|
||||
@property
|
||||
def width_m(self) -> float:
|
||||
return self.east_m - self.west_m
|
||||
|
||||
@property
|
||||
def height_m(self) -> float:
|
||||
return self.north_m - self.south_m
|
||||
|
||||
@property
|
||||
def mercator_tuple(self) -> tuple[float, float, float, float]:
|
||||
return self.west_m, self.south_m, self.east_m, self.north_m
|
||||
|
||||
@property
|
||||
def wgs84_tuple(self) -> tuple[float, float, float, float]:
|
||||
west, south = mercator_to_lonlat(self.west_m, self.south_m)
|
||||
east, north = mercator_to_lonlat(self.east_m, self.north_m)
|
||||
return west, south, east, north
|
||||
|
||||
|
||||
def lonlat_to_mercator(lon: float, lat: float) -> tuple[float, float]:
|
||||
lat = max(-MAX_MERCATOR_LAT, min(MAX_MERCATOR_LAT, float(lat)))
|
||||
x = WEB_MERCATOR_HALF_WORLD * float(lon) / 180.0
|
||||
y = math.log(math.tan((90.0 + lat) * math.pi / 360.0)) / (math.pi / 180.0)
|
||||
y = WEB_MERCATOR_HALF_WORLD * y / 180.0
|
||||
return x, y
|
||||
|
||||
|
||||
def mercator_to_lonlat(x: float, y: float) -> tuple[float, float]:
|
||||
lon = float(x) / WEB_MERCATOR_HALF_WORLD * 180.0
|
||||
lat = float(y) / WEB_MERCATOR_HALF_WORLD * 180.0
|
||||
lat = 180.0 / math.pi * (
|
||||
2.0 * math.atan(math.exp(lat * math.pi / 180.0)) - math.pi / 2.0
|
||||
)
|
||||
return lon, lat
|
||||
|
||||
|
||||
def xyz_tile_bounds(z: int, x: int, y: int) -> TileBounds:
|
||||
n = 1 << int(z)
|
||||
if x < 0 or y < 0 or x >= n or y >= n:
|
||||
raise ValueError(f"invalid XYZ tile z={z} x={x} y={y}")
|
||||
tile_size = (WEB_MERCATOR_HALF_WORLD * 2.0) / n
|
||||
west = -WEB_MERCATOR_HALF_WORLD + x * tile_size
|
||||
east = west + tile_size
|
||||
north = WEB_MERCATOR_HALF_WORLD - y * tile_size
|
||||
south = north - tile_size
|
||||
return TileBounds(
|
||||
z=z,
|
||||
x=x,
|
||||
y=y,
|
||||
west_m=west,
|
||||
south_m=south,
|
||||
east_m=east,
|
||||
north_m=north,
|
||||
)
|
||||
|
||||
|
||||
def expanded_mercator_bounds(
|
||||
tile: TileBounds,
|
||||
buffer_tile_pixels: float,
|
||||
nominal_tile_pixels: float = 256.0,
|
||||
) -> tuple[float, float, float, float]:
|
||||
pad = tile.width_m * float(buffer_tile_pixels) / float(nominal_tile_pixels)
|
||||
return (
|
||||
tile.west_m - pad,
|
||||
tile.south_m - pad,
|
||||
tile.east_m + pad,
|
||||
tile.north_m + pad,
|
||||
)
|
||||
|
||||
|
||||
def child_tiles(z: int, x: int, y: int, child_z: int) -> list[tuple[int, int, int]]:
|
||||
if child_z < z:
|
||||
raise ValueError("child_z must be >= z")
|
||||
factor = 1 << (child_z - z)
|
||||
out: list[tuple[int, int, int]] = []
|
||||
for cx in range(x * factor, (x + 1) * factor):
|
||||
for cy in range(y * factor, (y + 1) * factor):
|
||||
out.append((child_z, cx, cy))
|
||||
return out
|
||||
410
src/cm95pbf/presentation.py
Normal file
410
src/cm95pbf/presentation.py
Normal file
@@ -0,0 +1,410 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
import re
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
SUPPORTED_OPCODES = frozenset(
|
||||
{"SY", "LS", "LC", "AC", "AP", "TX", "TE", "CS", "CA", "MP", "OP"}
|
||||
)
|
||||
|
||||
|
||||
OPENCPN_NATSUR_TEXT = {
|
||||
1: "mud", 2: "clay", 3: "silt", 4: "sand", 5: "stone",
|
||||
6: "gravel", 7: "pebbles", 8: "cobbles", 9: "rock", 11: "lava",
|
||||
14: "coral", 17: "shells", 18: "boulder", 51: "Wd", 56: "Bo",
|
||||
}
|
||||
|
||||
|
||||
def _strip_quotes(value: str) -> str:
|
||||
text = str(value or "").strip()
|
||||
if len(text) >= 2 and text[0] in {"'", '"'} and text[-1] == text[0]:
|
||||
return text[1:-1]
|
||||
return text
|
||||
|
||||
|
||||
def _resolve_text_param(value: str, attrs: dict[str, Any]) -> str:
|
||||
raw = str(value or "").strip()
|
||||
quoted = len(raw) >= 2 and raw[0] in {"'", '"'} and raw[-1] == raw[0]
|
||||
if quoted:
|
||||
return raw[1:-1]
|
||||
if "=" in raw:
|
||||
key, default = raw.split("=", 1)
|
||||
current = attrs.get(key)
|
||||
if current not in (None, ""):
|
||||
return str(current)
|
||||
return default.rstrip(")")
|
||||
return raw.rstrip(")")
|
||||
|
||||
|
||||
def _text_attr_value(name: str, attrs: dict[str, Any]) -> str | None:
|
||||
key = _strip_quotes(name)
|
||||
if key not in attrs or attrs.get(key) in (None, ""):
|
||||
return None
|
||||
value = attrs.get(key)
|
||||
if isinstance(value, dict):
|
||||
text_value = value.get("text")
|
||||
if text_value not in (None, ""):
|
||||
return str(text_value)
|
||||
return None
|
||||
if key == "NATSUR":
|
||||
values = value if isinstance(value, (list, tuple)) else str(value).split(",")
|
||||
out = []
|
||||
for item in values:
|
||||
try:
|
||||
out.append(OPENCPN_NATSUR_TEXT.get(int(item), "unk"))
|
||||
except (TypeError, ValueError):
|
||||
out.append("unk")
|
||||
return ",".join(out)
|
||||
if isinstance(value, (list, tuple)):
|
||||
return ",".join(str(x) for x in value)
|
||||
return str(value)
|
||||
|
||||
|
||||
def _format_te_value(fmt_raw: str, attr_raw: str, attrs: dict[str, Any]) -> str | None:
|
||||
value = _text_attr_value(attr_raw, attrs)
|
||||
if value is None:
|
||||
return None
|
||||
fmt = _strip_quotes(fmt_raw).replace("lf", "f")
|
||||
match = re.search(r"%(?:[-+0 #]*\d*(?:\.\d+)?)?([csfdi])", fmt)
|
||||
try:
|
||||
if match is None:
|
||||
return fmt
|
||||
conversion = match.group(1)
|
||||
if conversion == "f":
|
||||
return fmt % float(value)
|
||||
if conversion in {"d", "i"}:
|
||||
return fmt % int(float(value))
|
||||
if conversion == "c":
|
||||
return fmt % (int(float(value)) if str(value).lstrip("+-").isdigit() else str(value)[0])
|
||||
return fmt % value
|
||||
except (TypeError, ValueError, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def materialize_text_hints(instruction: str, attrs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Materialize OpenCPN-compatible TX/TE rules into at most three text slots."""
|
||||
out: dict[str, Any] = {}
|
||||
slot = 0
|
||||
for rule in parse_instruction_string(instruction):
|
||||
if rule.opcode not in {"TX", "TE"}:
|
||||
continue
|
||||
args = rule.args
|
||||
if rule.opcode == "TX":
|
||||
if len(args) < 9:
|
||||
continue
|
||||
first = str(args[0]).strip()
|
||||
if len(first) >= 2 and first[0] in {"'", '"'} and first[-1] == first[0]:
|
||||
text = _strip_quotes(first)
|
||||
else:
|
||||
text = _text_attr_value(first, attrs)
|
||||
layout = args[1:9]
|
||||
else:
|
||||
if len(args) < 10:
|
||||
continue
|
||||
text = _format_te_value(args[0], args[1], attrs)
|
||||
layout = args[2:10]
|
||||
if text in (None, ""):
|
||||
continue
|
||||
slot += 1
|
||||
if slot > 3:
|
||||
break
|
||||
prefix = f"s52_text{slot}_"
|
||||
out[prefix + "value"] = str(text)
|
||||
out[prefix + "hjust"] = _resolve_text_param(layout[0], attrs)
|
||||
out[prefix + "vjust"] = _resolve_text_param(layout[1], attrs)
|
||||
out[prefix + "space"] = _resolve_text_param(layout[2], attrs)
|
||||
charspec = _resolve_text_param(layout[3], attrs)
|
||||
out[prefix + "charspec"] = charspec
|
||||
if len(charspec) >= 4 and charspec[3:].isdigit():
|
||||
body_size = int(charspec[3:])
|
||||
out[prefix + "body_size"] = body_size
|
||||
# OpenCPN normalizes S-52 body size to the user ChartText base
|
||||
# instead of rendering 18/20 as literal pixel sizes. Use a
|
||||
# deterministic 10px web base; a future NavSea text-scale setting
|
||||
# can shift this derived size without losing the raw body size.
|
||||
normalized = int((min(20, max(0, body_size)) / 20.0) * 4.0)
|
||||
out[prefix + "size"] = max(10, 10 + normalized)
|
||||
out[prefix + "xoff"] = _resolve_text_param(layout[4], attrs)
|
||||
out[prefix + "yoff"] = _resolve_text_param(layout[5], attrs)
|
||||
out[prefix + "color"] = _resolve_text_param(layout[6], attrs)
|
||||
out[prefix + "group"] = _resolve_text_param(layout[7], attrs)
|
||||
if slot:
|
||||
out["s52_text_count"] = slot
|
||||
return out
|
||||
|
||||
|
||||
def materialize_symbol_hints(instruction: str, attrs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Resolve generic SY orientation parameters without changing symbol choice."""
|
||||
symbols: list[dict[str, Any]] = []
|
||||
for rule in parse_instruction_string(instruction):
|
||||
if rule.opcode != "SY" or not rule.args:
|
||||
continue
|
||||
entry: dict[str, Any] = {
|
||||
"sequence": rule.sequence,
|
||||
"code": rule.args[0].strip().strip("'\"")[:8],
|
||||
}
|
||||
if len(rule.args) >= 2:
|
||||
source = rule.args[1].strip().strip("'\"")
|
||||
raw_value = attrs.get(source, source)
|
||||
try:
|
||||
entry["rotation_deg"] = float(raw_value) % 360.0
|
||||
entry["rotation_source"] = source
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
symbols.append(entry)
|
||||
|
||||
if not symbols:
|
||||
return {}
|
||||
out: dict[str, Any] = {}
|
||||
if len(symbols) > 1 or any("rotation_deg" in item for item in symbols):
|
||||
out["s52_symbol_rule_sequence"] = symbols
|
||||
first = symbols[0]
|
||||
if "rotation_deg" in first:
|
||||
out["s52_symbol_rotation_deg"] = first["rotation_deg"]
|
||||
out["s52_symbol_rotation_source"] = first.get("rotation_source")
|
||||
return out
|
||||
|
||||
|
||||
def materialize_rule_ir(
|
||||
instruction: str,
|
||||
attrs: dict[str, Any],
|
||||
*,
|
||||
source: str,
|
||||
procedure: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return an ordered, executable representation of an S-52 rule string."""
|
||||
out: list[dict[str, Any]] = []
|
||||
for rule in parse_instruction_string(instruction):
|
||||
item = rule.to_dict()
|
||||
item.update({"source": source, "procedure": procedure})
|
||||
if rule.opcode in {"TX", "TE"}:
|
||||
text = materialize_text_hints(rule.raw, attrs)
|
||||
if text.get("s52_text1_value") not in (None, ""):
|
||||
item["text"] = {
|
||||
key.removeprefix("s52_text1_"): value
|
||||
for key, value in text.items()
|
||||
if key.startswith("s52_text1_")
|
||||
}
|
||||
if rule.opcode == "OP":
|
||||
item["side_effect"] = "upstream_parser_noop"
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
class PresentationParseError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PresentationRule:
|
||||
opcode: str
|
||||
args_raw: str
|
||||
args: tuple[str, ...]
|
||||
raw: str
|
||||
sequence: int
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
data = asdict(self)
|
||||
data["args"] = list(self.args)
|
||||
return data
|
||||
|
||||
|
||||
def _split_args(text: str) -> tuple[str, ...]:
|
||||
if not text.strip():
|
||||
return ()
|
||||
out: list[str] = []
|
||||
buf: list[str] = []
|
||||
quote: str | None = None
|
||||
depth = 0
|
||||
i = 0
|
||||
while i < len(text):
|
||||
ch = text[i]
|
||||
if quote is not None:
|
||||
buf.append(ch)
|
||||
if ch == quote:
|
||||
quote = None
|
||||
elif ch == "\\" and i + 1 < len(text):
|
||||
i += 1
|
||||
buf.append(text[i])
|
||||
elif ch in {"'", '"'}:
|
||||
quote = ch
|
||||
buf.append(ch)
|
||||
elif ch == "(":
|
||||
depth += 1
|
||||
buf.append(ch)
|
||||
elif ch == ")":
|
||||
depth = max(0, depth - 1)
|
||||
buf.append(ch)
|
||||
elif ch == "," and depth == 0:
|
||||
out.append("".join(buf).strip())
|
||||
buf.clear()
|
||||
else:
|
||||
buf.append(ch)
|
||||
i += 1
|
||||
out.append("".join(buf).strip())
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def parse_instruction_string(instruction: str) -> list[PresentationRule]:
|
||||
"""Parse S-52 INST using OpenCPN StringToRules/SCANFWRD semantics.
|
||||
|
||||
OpenCPN intentionally scans each command to the next semicolon/record end
|
||||
instead of balancing parentheses. Some production Presentation Library
|
||||
records rely on that tolerance, so this parser does the same.
|
||||
"""
|
||||
text = str(instruction or "").replace("\x1f", ";").strip()
|
||||
rules: list[PresentationRule] = []
|
||||
i = 0
|
||||
while i < len(text):
|
||||
while i < len(text) and (text[i].isspace() or text[i] == ";"):
|
||||
i += 1
|
||||
if i >= len(text):
|
||||
break
|
||||
if i + 2 >= len(text):
|
||||
raise PresentationParseError(f"truncated opcode at {i}: {text!r}")
|
||||
opcode = text[i : i + 2]
|
||||
if opcode not in SUPPORTED_OPCODES:
|
||||
raise PresentationParseError(
|
||||
f"unsupported opcode {opcode!r} at {i}: {text!r}"
|
||||
)
|
||||
start = i
|
||||
i += 2
|
||||
while i < len(text) and text[i].isspace():
|
||||
i += 1
|
||||
if i >= len(text) or text[i] != "(":
|
||||
raise PresentationParseError(
|
||||
f"missing '(' after {opcode} at {i}: {text!r}"
|
||||
)
|
||||
i += 1
|
||||
args_start = i
|
||||
while i < len(text) and text[i] != ";":
|
||||
i += 1
|
||||
payload = text[args_start:i].strip()
|
||||
if payload.endswith(")"):
|
||||
payload = payload[:-1].rstrip()
|
||||
raw = text[start:i].strip()
|
||||
rules.append(
|
||||
PresentationRule(
|
||||
opcode=opcode,
|
||||
args_raw=payload,
|
||||
args=_split_args(payload),
|
||||
raw=raw,
|
||||
sequence=len(rules),
|
||||
)
|
||||
)
|
||||
return rules
|
||||
|
||||
def compile_lookup_ir(row: dict[str, Any]) -> dict[str, Any]:
|
||||
rules = parse_instruction_string(str(row.get("instruction") or ""))
|
||||
return {
|
||||
"id": row.get("id"),
|
||||
"rcid": row.get("rcid"),
|
||||
"object_class": row.get("object_class"),
|
||||
"geometry_type": row.get("geometry_type"),
|
||||
"table_name": row.get("table_name"),
|
||||
"display_priority": row.get("display_priority"),
|
||||
"display_category": row.get("display_category"),
|
||||
"radar_priority": row.get("radar_priority"),
|
||||
"attribute_conditions_raw": list(row.get("attribute_conditions_raw") or []),
|
||||
"instruction": row.get("instruction") or "",
|
||||
"rules": [rule.to_dict() for rule in rules],
|
||||
}
|
||||
|
||||
|
||||
def referenced_resource(rule: PresentationRule) -> tuple[str, str] | None:
|
||||
if not rule.args:
|
||||
return None
|
||||
kind_by_opcode = {
|
||||
"SY": "symbol",
|
||||
"LC": "line_style",
|
||||
"AP": "pattern",
|
||||
"CS": "conditional",
|
||||
"AC": "color",
|
||||
}
|
||||
kind = kind_by_opcode.get(rule.opcode)
|
||||
if kind is None:
|
||||
return None
|
||||
value = rule.args[0].strip().strip("'\"")
|
||||
if rule.opcode in {"SY", "LC", "AP", "CS"}:
|
||||
value = value[:8]
|
||||
return (kind, value) if value else None
|
||||
|
||||
|
||||
def flatten_rules(rows: Iterable[dict[str, Any]]) -> list[PresentationRule]:
|
||||
out: list[PresentationRule] = []
|
||||
for row in rows:
|
||||
out.extend(parse_instruction_string(str(row.get("instruction") or "")))
|
||||
return out
|
||||
|
||||
|
||||
def static_rule_hints(instruction: str) -> dict[str, Any]:
|
||||
"""Materialize compact scalar hints; full rule IR remains lookup-side."""
|
||||
rules = parse_instruction_string(instruction)
|
||||
out: dict[str, Any] = {}
|
||||
opcodes = [rule.opcode for rule in rules]
|
||||
if opcodes:
|
||||
out["s52_rule_opcodes"] = ",".join(opcodes)
|
||||
|
||||
for rule in rules:
|
||||
args = rule.args
|
||||
if rule.opcode == "SY" and args and "s52_symbol_code" not in out:
|
||||
out["s52_symbol_code"] = args[0].strip().strip("'\"")[:8]
|
||||
elif rule.opcode == "AC" and args and "s52_area_color" not in out:
|
||||
out["s52_area_color"] = args[0].strip().strip("'\"")
|
||||
elif rule.opcode == "AP" and args and "s52_area_pattern_code" not in out:
|
||||
out["s52_area_pattern_code"] = args[0].strip().strip("'\"")[:8]
|
||||
elif rule.opcode == "LC" and args and "s52_line_complex_code" not in out:
|
||||
out["s52_line_complex_code"] = args[0].strip().strip("'\"")[:8]
|
||||
elif rule.opcode == "LS" and len(args) >= 3 and "s52_line_style" not in out:
|
||||
out["s52_line_style"] = args[0].strip().strip("'\"")
|
||||
out["s52_line_width"] = args[1].strip().strip("'\"")
|
||||
out["s52_line_color"] = args[2].strip().strip("'\"")
|
||||
elif rule.opcode == "CA" and len(args) >= 8:
|
||||
out["s52_arc_outline_color"] = args[0].strip().strip("\'\"")
|
||||
out["s52_arc_outline_width"] = args[1].strip().strip("\'\"")
|
||||
out["s52_arc_color"] = args[2].strip().strip("\'\"")
|
||||
out["s52_arc_width"] = args[3].strip().strip("\'\"")
|
||||
out["s52_arc_start_deg"] = args[4].strip().strip("\'\"")
|
||||
out["s52_arc_end_deg"] = args[5].strip().strip("\'\"")
|
||||
out["s52_arc_radius_mm"] = args[6].strip().strip("\'\"")
|
||||
out["s52_arc_sector_radius_mm"] = args[7].strip().strip("\'\"")
|
||||
elif rule.opcode == "OP" and args:
|
||||
out.setdefault("s52_priority_override", args[0].strip().strip("\'\""))
|
||||
elif rule.opcode in {"TX", "TE"}:
|
||||
out["s52_has_text"] = 1
|
||||
if rule.opcode == "TX" and args:
|
||||
literal = args[0].strip()
|
||||
if len(literal) >= 2 and literal[0] in {"\'", '"'} and literal[-1] == literal[0]:
|
||||
out.setdefault("s52_text_literal", literal[1:-1])
|
||||
elif rule.opcode == "CS" and args:
|
||||
name = args[0].strip().strip("'\"")[:8]
|
||||
existing = out.get("s52_csp")
|
||||
out["s52_csp"] = f"{existing},{name}" if existing else name
|
||||
line_rules = [
|
||||
rule for rule in rules if rule.opcode == "LS" and len(rule.args) >= 3
|
||||
]
|
||||
for index, rule in enumerate(line_rules, 1):
|
||||
prefix = f"s52_line{index}_"
|
||||
out[prefix + "style"] = rule.args[0].strip().strip("'\"")
|
||||
out[prefix + "width"] = rule.args[1].strip().strip("'\"")
|
||||
out[prefix + "color"] = rule.args[2].strip().strip("'\"")
|
||||
if len(line_rules) > 1:
|
||||
out["s52_line_sequence"] = "|".join(
|
||||
",".join(arg.strip().strip("'\"") for arg in rule.args[:3])
|
||||
for rule in line_rules
|
||||
)
|
||||
out["s52_line_count"] = len(line_rules)
|
||||
|
||||
symbol_codes = [
|
||||
rule.args[0].strip().strip("'\"")[:8]
|
||||
for rule in rules
|
||||
if rule.opcode == "SY" and rule.args
|
||||
]
|
||||
if symbol_codes:
|
||||
out.setdefault("s52_symbol_code", symbol_codes[0])
|
||||
out["s52_symbol_sequence"] = ",".join(symbol_codes)
|
||||
if len(symbol_codes) > 1:
|
||||
out["s52_symbol_count"] = len(symbol_codes)
|
||||
return out
|
||||
114
src/cm95pbf/presentation_audit.py
Normal file
114
src/cm95pbf/presentation_audit.py
Normal file
@@ -0,0 +1,114 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from decimal import Decimal, InvalidOperation
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
LIGHT_PRESENTATION_SIGNATURE_FIELDS = (
|
||||
"source_record_index",
|
||||
"s52_csp_resolved",
|
||||
"s52_display_priority",
|
||||
"s52_display_category",
|
||||
"s52_light_mode",
|
||||
"s52_light_symbol_code",
|
||||
"s52_arc_outline_color",
|
||||
"s52_arc_outline_width",
|
||||
"s52_arc_color",
|
||||
"s52_arc_width",
|
||||
"s52_arc_start_deg",
|
||||
"s52_arc_end_deg",
|
||||
"s52_arc_radius_mm",
|
||||
"s52_arc_sector_radius_mm",
|
||||
"s52_sector_start_deg",
|
||||
"s52_sector_end_deg",
|
||||
"s52_sector_sweep_deg",
|
||||
"s52_light_nominal_range_nm",
|
||||
"s52_light_rotation_deg",
|
||||
"s52_light_obscured",
|
||||
"s52_light_label_text",
|
||||
"s52_text_literal",
|
||||
)
|
||||
|
||||
_NUMERIC_FIELDS = {
|
||||
"source_record_index",
|
||||
"s52_arc_outline_width",
|
||||
"s52_arc_width",
|
||||
"s52_arc_start_deg",
|
||||
"s52_arc_end_deg",
|
||||
"s52_arc_radius_mm",
|
||||
"s52_arc_sector_radius_mm",
|
||||
"s52_sector_start_deg",
|
||||
"s52_sector_end_deg",
|
||||
"s52_sector_sweep_deg",
|
||||
"s52_light_nominal_range_nm",
|
||||
"s52_light_rotation_deg",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_number(value: Any) -> str | None:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
try:
|
||||
number = Decimal(str(value))
|
||||
except (InvalidOperation, ValueError):
|
||||
return str(value)
|
||||
if not number.is_finite():
|
||||
return str(value)
|
||||
number = number.normalize()
|
||||
if number == number.to_integral():
|
||||
return str(number.quantize(Decimal("1")))
|
||||
text = format(number, "f")
|
||||
return text.rstrip("0").rstrip(".") if "." in text else text
|
||||
|
||||
|
||||
def canonical_light_presentation(
|
||||
presentation: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
canonical: dict[str, Any] = {}
|
||||
for field in LIGHT_PRESENTATION_SIGNATURE_FIELDS:
|
||||
value = presentation.get(field)
|
||||
if field in _NUMERIC_FIELDS:
|
||||
value = _normalize_number(value)
|
||||
elif field == "s52_light_obscured":
|
||||
if value in (None, ""):
|
||||
value = None
|
||||
elif isinstance(value, str):
|
||||
value = value.strip().lower() in {"1", "true", "yes"}
|
||||
else:
|
||||
value = bool(value)
|
||||
elif value is not None:
|
||||
value = str(value)
|
||||
canonical[field] = value
|
||||
return canonical
|
||||
|
||||
|
||||
def light_presentation_signature(
|
||||
presentation: dict[str, Any],
|
||||
) -> str:
|
||||
canonical = canonical_light_presentation(presentation)
|
||||
payload = json.dumps(
|
||||
canonical,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def parse_light_presentations(value: Any) -> list[dict[str, Any]]:
|
||||
if value in (None, ""):
|
||||
return []
|
||||
if isinstance(value, str):
|
||||
value = json.loads(value)
|
||||
if not isinstance(value, list):
|
||||
raise ValueError("s52_light_presentations must decode to a list")
|
||||
rows: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(value):
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError(
|
||||
f"s52_light_presentations[{index}] must be an object"
|
||||
)
|
||||
rows.append(dict(item))
|
||||
return rows
|
||||
1072
src/cm95pbf/production.py
Normal file
1072
src/cm95pbf/production.py
Normal file
File diff suppressed because it is too large
Load Diff
847
src/cm95pbf/release.py
Normal file
847
src/cm95pbf/release.py
Normal file
@@ -0,0 +1,847 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import mapbox_vector_tile
|
||||
|
||||
from .production import ProductionLedger, load_plan, sha256_file
|
||||
from .style_contract import audit_pbf_style_and_sprite
|
||||
|
||||
|
||||
RELEASE_SCHEMA = "navsea.cm95.release-manifest/v1"
|
||||
|
||||
|
||||
def _builder_contract_at_least(value: Any, target: tuple[int, int, int]) -> bool:
|
||||
match = re.match(r"^cm95pbf-builder/(\d+)\.(\d+)\.(\d+)", str(value or ""))
|
||||
if not match:
|
||||
return False
|
||||
return tuple(int(part) for part in match.groups()) >= target
|
||||
|
||||
|
||||
def _utc_now():
|
||||
return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def _external_audit_contract(
|
||||
path: Path | None,
|
||||
*,
|
||||
required: bool,
|
||||
expected_schema: str,
|
||||
expected_plan_id: str | None = None,
|
||||
expected_pbf_dir: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {
|
||||
"required": required,
|
||||
"status": "NOT_AUDITED",
|
||||
"path": None,
|
||||
"sha256": None,
|
||||
"schema": None,
|
||||
"audit_status": None,
|
||||
"failures": [],
|
||||
}
|
||||
if path is None:
|
||||
if required:
|
||||
result["status"] = "FAIL"
|
||||
result["failures"].append("required_audit_missing")
|
||||
return result
|
||||
|
||||
path = Path(path)
|
||||
result["path"] = str(path)
|
||||
if not path.is_file():
|
||||
result["status"] = "FAIL"
|
||||
result["failures"].append("audit_file_missing")
|
||||
return result
|
||||
|
||||
result["sha256"] = sha256_file(path)
|
||||
try:
|
||||
obj = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
result["status"] = "FAIL"
|
||||
result["failures"].append(
|
||||
f"audit_parse_error:{type(exc).__name__}"
|
||||
)
|
||||
return result
|
||||
|
||||
result["schema"] = obj.get("schema")
|
||||
result["audit_status"] = obj.get("status")
|
||||
if obj.get("schema") != expected_schema:
|
||||
result["failures"].append("audit_schema_mismatch")
|
||||
if obj.get("status") != "PASS":
|
||||
result["failures"].append("audit_not_pass")
|
||||
if expected_plan_id is not None and obj.get("plan_id") != expected_plan_id:
|
||||
result["failures"].append("audit_plan_id_mismatch")
|
||||
if expected_pbf_dir is not None:
|
||||
raw_pbf_dir = obj.get("pbf_dir")
|
||||
if not raw_pbf_dir:
|
||||
result["failures"].append("audit_pbf_dir_missing")
|
||||
else:
|
||||
try:
|
||||
actual = Path(raw_pbf_dir).resolve()
|
||||
expected = Path(expected_pbf_dir).resolve()
|
||||
except Exception:
|
||||
result["failures"].append("audit_pbf_dir_invalid")
|
||||
else:
|
||||
if actual != expected:
|
||||
result["failures"].append("audit_pbf_dir_mismatch")
|
||||
|
||||
for key in (
|
||||
"expected_multi_light_groups",
|
||||
"actual_multi_light_groups",
|
||||
"expected_presentation_records",
|
||||
"actual_presentation_records",
|
||||
"signature_mismatch_count",
|
||||
"occurrence_drift_count",
|
||||
"occurrence_scope_mismatch_count",
|
||||
"required_runtime_kinds",
|
||||
"missing_runtime_kinds",
|
||||
"noncompliant_runtime_kinds",
|
||||
):
|
||||
if key in obj:
|
||||
result[key] = obj.get(key)
|
||||
|
||||
result["status"] = "PASS" if not result["failures"] else "FAIL"
|
||||
return result
|
||||
|
||||
|
||||
def _collect_nav_name_conflict_expectations(
|
||||
report_root: Path,
|
||||
conflict_reports: list[tuple[Path, int]],
|
||||
) -> dict[str, Any]:
|
||||
"""Collect ambiguous navigation-name outputs that must remain unlabeled.
|
||||
|
||||
This is intentionally fail-closed:
|
||||
- a report-side conflict must have exactly matching trace rows;
|
||||
- each trace row must explicitly say nav_name_source=conflict;
|
||||
- there must be at least two distinct direct-name candidates;
|
||||
- the corresponding PBF feature is validated later and must omit
|
||||
chart_label_text rather than guessing one candidate.
|
||||
"""
|
||||
expected_refs: dict[tuple[str, str, int], dict[str, Any]] = {}
|
||||
unique_outputs: dict[tuple[str, int], dict[str, Any]] = {}
|
||||
issues: list[dict[str, Any]] = []
|
||||
trace_occurrences = 0
|
||||
|
||||
for report_path, report_count in conflict_reports:
|
||||
rel_report = report_path.relative_to(report_root)
|
||||
trace_path = report_path.with_name(report_path.stem + ".trace.jsonl")
|
||||
pbf_rel = rel_report.with_suffix(".pbf").as_posix()
|
||||
|
||||
if not trace_path.exists():
|
||||
issues.append({
|
||||
"type": "missing_trace_file",
|
||||
"report": rel_report.as_posix(),
|
||||
"expected_conflicts": int(report_count),
|
||||
})
|
||||
continue
|
||||
|
||||
tile_rows: list[dict[str, Any]] = []
|
||||
with trace_path.open("r", encoding="utf-8") as handle:
|
||||
for lineno, raw in enumerate(handle, start=1):
|
||||
raw = raw.strip()
|
||||
if not raw:
|
||||
continue
|
||||
try:
|
||||
row = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
issues.append({
|
||||
"type": "invalid_trace_json",
|
||||
"trace": str(trace_path.relative_to(report_root)),
|
||||
"line": lineno,
|
||||
"error": str(exc),
|
||||
})
|
||||
continue
|
||||
if isinstance(row, dict) and row.get("nav_name_conflict") is True:
|
||||
tile_rows.append(row)
|
||||
|
||||
trace_occurrences += len(tile_rows)
|
||||
if len(tile_rows) != int(report_count):
|
||||
issues.append({
|
||||
"type": "report_trace_conflict_count_mismatch",
|
||||
"report": rel_report.as_posix(),
|
||||
"report_count": int(report_count),
|
||||
"trace_count": len(tile_rows),
|
||||
})
|
||||
|
||||
for row in tile_rows:
|
||||
output_id = row.get("output_id")
|
||||
layer = str(row.get("output_layer") or "")
|
||||
name_source = str(row.get("nav_name_source") or "")
|
||||
raw_candidates = row.get("nav_name_candidates") or []
|
||||
candidates: list[str] = []
|
||||
for value in raw_candidates:
|
||||
text = str(value).strip()
|
||||
if text and text not in candidates:
|
||||
candidates.append(text)
|
||||
|
||||
if output_id is None or not layer:
|
||||
issues.append({
|
||||
"type": "invalid_conflict_trace_identity",
|
||||
"report": rel_report.as_posix(),
|
||||
"row": row,
|
||||
})
|
||||
continue
|
||||
|
||||
try:
|
||||
fid = int(output_id)
|
||||
except (TypeError, ValueError):
|
||||
issues.append({
|
||||
"type": "invalid_conflict_output_id",
|
||||
"report": rel_report.as_posix(),
|
||||
"output_id": output_id,
|
||||
})
|
||||
continue
|
||||
|
||||
if name_source != "conflict":
|
||||
issues.append({
|
||||
"type": "conflict_trace_wrong_name_source",
|
||||
"report": rel_report.as_posix(),
|
||||
"output_id": fid,
|
||||
"nav_name_source": name_source,
|
||||
})
|
||||
|
||||
if len(candidates) < 2:
|
||||
issues.append({
|
||||
"type": "conflict_trace_insufficient_candidates",
|
||||
"report": rel_report.as_posix(),
|
||||
"output_id": fid,
|
||||
"candidates": candidates,
|
||||
})
|
||||
|
||||
ref_key = (pbf_rel, layer, fid)
|
||||
if ref_key in expected_refs:
|
||||
issues.append({
|
||||
"type": "duplicate_conflict_trace_reference",
|
||||
"pbf": pbf_rel,
|
||||
"layer": layer,
|
||||
"output_id": fid,
|
||||
})
|
||||
expected_refs[ref_key] = {
|
||||
"pbf": pbf_rel,
|
||||
"layer": layer,
|
||||
"output_id": fid,
|
||||
"source_id": row.get("source_id"),
|
||||
"relation20": row.get("relation20"),
|
||||
"native_scale": row.get("native_scale"),
|
||||
"candidates": candidates,
|
||||
}
|
||||
|
||||
unique_key = (layer, fid)
|
||||
unique = unique_outputs.setdefault(unique_key, {
|
||||
"output_id": fid,
|
||||
"output_layer": layer,
|
||||
"source_id": row.get("source_id"),
|
||||
"relation20": row.get("relation20"),
|
||||
"native_scale": row.get("native_scale"),
|
||||
"nav_name_candidates": list(candidates),
|
||||
"occurrences": 0,
|
||||
"tiles": [],
|
||||
})
|
||||
unique["occurrences"] += 1
|
||||
if pbf_rel not in unique["tiles"]:
|
||||
unique["tiles"].append(pbf_rel)
|
||||
for candidate in candidates:
|
||||
if candidate not in unique["nav_name_candidates"]:
|
||||
unique["nav_name_candidates"].append(candidate)
|
||||
|
||||
return {
|
||||
"report_occurrences": sum(int(count) for _path, count in conflict_reports),
|
||||
"trace_occurrences": trace_occurrences,
|
||||
"expected_refs": expected_refs,
|
||||
"unique_outputs": unique_outputs,
|
||||
"issues": issues,
|
||||
"seen_refs": set(),
|
||||
"labeled_conflicts": [],
|
||||
}
|
||||
|
||||
|
||||
def _observe_nav_name_conflict_feature(
|
||||
state: dict[str, Any],
|
||||
pbf_rel: str,
|
||||
layer: str,
|
||||
fid: Any,
|
||||
properties: dict[str, Any],
|
||||
) -> None:
|
||||
if fid is None:
|
||||
return
|
||||
try:
|
||||
key = (pbf_rel, str(layer), int(fid))
|
||||
except (TypeError, ValueError):
|
||||
return
|
||||
|
||||
expected = state["expected_refs"].get(key)
|
||||
if expected is None:
|
||||
return
|
||||
|
||||
state["seen_refs"].add(key)
|
||||
label = properties.get("chart_label_text")
|
||||
if label is not None and str(label).strip():
|
||||
state["labeled_conflicts"].append({
|
||||
**expected,
|
||||
"chart_label_text": str(label),
|
||||
})
|
||||
|
||||
|
||||
def _finalize_nav_name_conflict_audit(state: dict[str, Any]) -> dict[str, Any]:
|
||||
expected_keys = set(state["expected_refs"])
|
||||
seen_keys = set(state["seen_refs"])
|
||||
missing = sorted(expected_keys - seen_keys)
|
||||
|
||||
issues = list(state["issues"])
|
||||
if state["report_occurrences"] != state["trace_occurrences"]:
|
||||
issues.append({
|
||||
"type": "global_report_trace_conflict_count_mismatch",
|
||||
"report_occurrences": state["report_occurrences"],
|
||||
"trace_occurrences": state["trace_occurrences"],
|
||||
})
|
||||
|
||||
for pbf_rel, layer, fid in missing:
|
||||
issues.append({
|
||||
"type": "conflict_output_missing_from_pbf",
|
||||
"pbf": pbf_rel,
|
||||
"layer": layer,
|
||||
"output_id": fid,
|
||||
})
|
||||
|
||||
for item in state["labeled_conflicts"]:
|
||||
issues.append({
|
||||
"type": "ambiguous_conflict_was_labeled",
|
||||
**item,
|
||||
})
|
||||
|
||||
unique_details = []
|
||||
for row in state["unique_outputs"].values():
|
||||
item = dict(row)
|
||||
item["tiles"] = sorted(item.get("tiles") or [])
|
||||
unique_details.append(item)
|
||||
unique_details.sort(
|
||||
key=lambda row: (
|
||||
int(row.get("native_scale") or 0),
|
||||
str(row.get("source_id") or ""),
|
||||
int(row.get("output_id") or 0),
|
||||
)
|
||||
)
|
||||
|
||||
affected_tiles = sorted({
|
||||
key[0] for key in state["expected_refs"]
|
||||
})
|
||||
safe_omissions = len(seen_keys) - len(state["labeled_conflicts"])
|
||||
|
||||
return {
|
||||
"overall": "PASS" if not issues else "FAIL",
|
||||
"policy": "AMBIGUOUS_DIRECT_NAMES_OMIT_LABEL_FAIL_CLOSED",
|
||||
"report_occurrences": int(state["report_occurrences"]),
|
||||
"trace_occurrences": int(state["trace_occurrences"]),
|
||||
"safe_omission_occurrences": int(safe_omissions),
|
||||
"unique_conflict_outputs": len(unique_details),
|
||||
"affected_tiles": len(affected_tiles),
|
||||
"labeled_conflict_occurrences": len(state["labeled_conflicts"]),
|
||||
"missing_pbf_occurrences": len(missing),
|
||||
"issues": issues,
|
||||
"unique_conflicts": unique_details,
|
||||
}
|
||||
|
||||
|
||||
def finalize_release(
|
||||
plan_path: Path,
|
||||
release_root: Path,
|
||||
style_path: Path | None = None,
|
||||
sprite_json_path: Path | None = None,
|
||||
gap_disposition_path: Path | None = None,
|
||||
presentation_conservation_audit_path: Path | None = None,
|
||||
runtime_capability_audit_path: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
plan = load_plan(plan_path)
|
||||
release_dir = Path(release_root) / str(plan["release_id"])
|
||||
ledger = ProductionLedger(release_dir / "production_run_v1.sqlite")
|
||||
status = ledger.summary(str(plan["profile"]))
|
||||
if status["states"].get("FAILED", 0) or status["states"].get("PENDING", 0) or status["states"].get("RUNNING", 0):
|
||||
raise RuntimeError(f"release has unfinished jobs: {status['states']}")
|
||||
if status["states"].get("PASS", 0) != int(plan["job_count"]):
|
||||
raise RuntimeError(f"PASS job count mismatch: {status['states']} plan={plan['job_count']}")
|
||||
|
||||
pbf_root = release_dir / "pbf" / str(plan["profile"])
|
||||
report_root = release_dir / "reports" / str(plan["profile"])
|
||||
pbf_files = sorted(pbf_root.rglob("*.pbf")) if pbf_root.exists() else []
|
||||
report_files = sorted(
|
||||
p for p in report_root.rglob("*.json")
|
||||
if not p.name.endswith(".trace.json")
|
||||
) if report_root.exists() else []
|
||||
|
||||
hard_gate = []
|
||||
encoder_dropped = []
|
||||
name_conflicts = 0
|
||||
nav_name_conflict_reports: list[tuple[Path, int]] = []
|
||||
non_renderable_omissions = 0
|
||||
non_renderable_omission_layers = Counter()
|
||||
quantization_fallbacks = 0
|
||||
quantization_omissions = 0
|
||||
quantization_omission_layers = Counter()
|
||||
encoder_reconciled_candidates = 0
|
||||
encoder_reconciled_tiles = 0
|
||||
report_multi_light_groups = 0
|
||||
report_light_presentation_records = 0
|
||||
planned = 0
|
||||
output = 0
|
||||
report_layer_counts = Counter()
|
||||
for path in report_files:
|
||||
obj = json.loads(path.read_text(encoding="utf-8"))
|
||||
hard_gate += [{"report": str(path.relative_to(release_dir)), **x} for x in obj.get("hard_gate_unmapped") or []]
|
||||
encoder_dropped += [{"report": str(path.relative_to(release_dir)), **x} for x in obj.get("encoder_dropped_features") or []]
|
||||
nav_summary = obj.get("navigation_composites") or {}
|
||||
tile_name_conflicts = int(nav_summary.get("name_conflict_groups") or 0)
|
||||
report_multi_light_groups += int(nav_summary.get("multi_light_groups") or 0)
|
||||
report_light_presentation_records += int(nav_summary.get("light_presentation_records") or 0)
|
||||
name_conflicts += tile_name_conflicts
|
||||
if tile_name_conflicts:
|
||||
nav_name_conflict_reports.append((path, tile_name_conflicts))
|
||||
label_omissions = obj.get("non_renderable_omissions") or []
|
||||
non_renderable_omissions += len(label_omissions)
|
||||
for item in label_omissions:
|
||||
if isinstance(item, dict):
|
||||
non_renderable_omission_layers[str(item.get("layer") or "UNKNOWN")] += 1
|
||||
quantization_fallbacks += len(obj.get("quantization_fallbacks") or [])
|
||||
omissions = obj.get("quantization_omissions") or []
|
||||
quantization_omissions += len(omissions)
|
||||
reconcile_candidates = obj.get("encoder_drop_candidates") or []
|
||||
if reconcile_candidates:
|
||||
encoder_reconciled_tiles += 1
|
||||
for candidate in reconcile_candidates:
|
||||
if isinstance(candidate, dict):
|
||||
encoder_reconciled_candidates += len(candidate.get("dropped") or [])
|
||||
for item in omissions:
|
||||
if isinstance(item, dict):
|
||||
quantization_omission_layers[str(item.get("layer") or "UNKNOWN")] += 1
|
||||
planned += int(obj.get("planned_output_feature_count") or 0)
|
||||
output += int(obj.get("output_feature_count") or 0)
|
||||
report_layer_counts.update(obj.get("output_layer_counts") or {})
|
||||
|
||||
nav_name_conflict_state = _collect_nav_name_conflict_expectations(
|
||||
report_root,
|
||||
nav_name_conflict_reports,
|
||||
)
|
||||
|
||||
inventory = []
|
||||
decoded_features = 0
|
||||
decoded_layers = Counter()
|
||||
duplicate_feature_ids = []
|
||||
|
||||
s52_plan = dict(plan.get("s52") or {})
|
||||
s52_runtime_enabled = bool(s52_plan.get("enabled")) and s52_plan.get("mariner_mode") == "runtime"
|
||||
s52_runtime_failures: list[str] = []
|
||||
s52_runtime_csp_counts = Counter()
|
||||
s52_runtime_layer_counts = Counter()
|
||||
s52_runtime_dependent_count = 0
|
||||
s52_runtime_leaks = Counter()
|
||||
s52_runtime_fact_counts = Counter()
|
||||
s52_runtime_unknown_fact_exceptions = Counter()
|
||||
light_presentation_required = _builder_contract_at_least(
|
||||
plan.get("builder_contract_version"), (0, 4, 11)
|
||||
)
|
||||
pbf_multi_light_groups = 0
|
||||
pbf_light_presentation_records = 0
|
||||
light_presentation_failures: list[str] = []
|
||||
s52_dynamic_final_fields = {
|
||||
"s52_is_safety_contour",
|
||||
"s52_sounding_prefix",
|
||||
"s52_sounding_is_shallow",
|
||||
"s52_isolated_danger",
|
||||
"s52_display_base_promoted",
|
||||
}
|
||||
for path in pbf_files:
|
||||
raw = path.read_bytes()
|
||||
decoded = mapbox_vector_tile.decode(raw)
|
||||
seen = set()
|
||||
tile_features = 0
|
||||
for layer, payload in decoded.items():
|
||||
feats = payload.get("features", [])
|
||||
decoded_layers[layer] += len(feats)
|
||||
tile_features += len(feats)
|
||||
pbf_rel = str(path.relative_to(pbf_root)).replace("\\", "/")
|
||||
for feat in feats:
|
||||
fid = feat.get("id")
|
||||
props = feat.get("properties") or {}
|
||||
raw_light_presentations = props.get("s52_light_presentations")
|
||||
if raw_light_presentations not in (None, ""):
|
||||
try:
|
||||
light_presentations = (
|
||||
json.loads(raw_light_presentations)
|
||||
if isinstance(raw_light_presentations, str)
|
||||
else raw_light_presentations
|
||||
)
|
||||
except Exception as exc:
|
||||
light_presentation_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} invalid s52_light_presentations JSON: {type(exc).__name__}"
|
||||
)
|
||||
light_presentations = []
|
||||
if not isinstance(light_presentations, list) or len(light_presentations) < 2:
|
||||
light_presentation_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} multi-light presentation array must contain at least 2 items"
|
||||
)
|
||||
else:
|
||||
pbf_multi_light_groups += 1
|
||||
pbf_light_presentation_records += len(light_presentations)
|
||||
for index, presentation in enumerate(light_presentations):
|
||||
if not isinstance(presentation, dict):
|
||||
light_presentation_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} light presentation {index} is not an object"
|
||||
)
|
||||
continue
|
||||
if presentation.get("s52_csp_resolved") != "LIGHTS05":
|
||||
light_presentation_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} light presentation {index} is not resolved LIGHTS05"
|
||||
)
|
||||
if "source_record_index" not in presentation:
|
||||
light_presentation_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} light presentation {index} missing source_record_index"
|
||||
)
|
||||
if s52_runtime_enabled:
|
||||
if props.get("s52_mariner_dependent") == 1:
|
||||
s52_runtime_dependent_count += 1
|
||||
s52_runtime_layer_counts[layer] += 1
|
||||
csp = str(props.get("s52_runtime_csp") or "")
|
||||
if not csp:
|
||||
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} missing s52_runtime_csp")
|
||||
else:
|
||||
s52_runtime_csp_counts[csp] += 1
|
||||
for key in s52_dynamic_final_fields:
|
||||
if key in props:
|
||||
s52_runtime_leaks[key] += 1
|
||||
if csp in {"DEPARE01", "DEPARE02"}:
|
||||
if "depth_min_m" not in props:
|
||||
if props.get("canonical_object_type") == "dredged_area":
|
||||
s52_runtime_unknown_fact_exceptions[
|
||||
"DEPARE:dredged_area_missing_depth_min"
|
||||
] += 1
|
||||
else:
|
||||
s52_runtime_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} {csp} missing depth_min_m"
|
||||
)
|
||||
if "s52_area_color" in props:
|
||||
s52_runtime_leaks["DEPARE:s52_area_color"] += 1
|
||||
elif csp == "DEPCNT02" and "contour_depth_m" not in props:
|
||||
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} DEPCNT02 missing contour_depth_m")
|
||||
elif csp == "SOUNDG02" and "s52_sounding_depth_m" not in props:
|
||||
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} SOUNDG02 missing s52_sounding_depth_m")
|
||||
elif csp in {"OBSTRN04", "WRECKS02"} and "s52_hazard_depth_m" not in props:
|
||||
if csp == "OBSTRN04" and props.get("s52_expsou_code") == 1:
|
||||
s52_runtime_unknown_fact_exceptions[
|
||||
"OBSTRN04:EXPSOU1_unknown_hazard_depth"
|
||||
] += 1
|
||||
else:
|
||||
s52_runtime_failures.append(
|
||||
f"{pbf_rel}:{layer}:{fid} {csp} missing s52_hazard_depth_m"
|
||||
)
|
||||
elif "s52_runtime_csp" in props:
|
||||
s52_runtime_failures.append(f"{pbf_rel}:{layer}:{fid} runtime CSP without mariner-dependent marker")
|
||||
for key in (
|
||||
"depth_min_m", "depth_max_m", "contour_depth_m",
|
||||
"s52_sounding_depth_m", "s52_hazard_depth_m",
|
||||
"s52_surrounding_max_drval1_m", "s52_surrounding_min_drval2_m",
|
||||
):
|
||||
if key in props:
|
||||
s52_runtime_fact_counts[key] += 1
|
||||
_observe_nav_name_conflict_feature(
|
||||
nav_name_conflict_state,
|
||||
pbf_rel,
|
||||
layer,
|
||||
fid,
|
||||
props,
|
||||
)
|
||||
if fid is None:
|
||||
continue
|
||||
key = (layer, int(fid))
|
||||
if key in seen:
|
||||
duplicate_feature_ids.append({"file": str(path.relative_to(release_dir)), "layer": layer, "id": int(fid)})
|
||||
seen.add(key)
|
||||
decoded_features += tile_features
|
||||
inventory.append({
|
||||
"path": str(path.relative_to(release_dir)).replace("\\", "/"),
|
||||
"bytes": len(raw),
|
||||
"sha256": hashlib.sha256(raw).hexdigest(),
|
||||
"features": tile_features,
|
||||
"layers": len(decoded),
|
||||
})
|
||||
|
||||
style_sha = None
|
||||
sprite_sha = None
|
||||
style_contract = None
|
||||
if style_path is not None:
|
||||
style_path = Path(style_path)
|
||||
style_sha = sha256_file(style_path)
|
||||
if sprite_json_path is not None:
|
||||
sprite_json_path = Path(sprite_json_path)
|
||||
sprite_sha = sha256_file(sprite_json_path)
|
||||
style_contract = audit_pbf_style_and_sprite(
|
||||
pbf_root,
|
||||
style_path,
|
||||
sprite_json_path,
|
||||
)
|
||||
|
||||
gap_disposition_sha = None
|
||||
semantic_backlog = None
|
||||
semantic_completeness = "NOT_AUDITED"
|
||||
if gap_disposition_path is not None:
|
||||
import yaml
|
||||
gap_disposition_path = Path(gap_disposition_path)
|
||||
if gap_disposition_path.exists():
|
||||
gap_disposition_sha = sha256_file(gap_disposition_path)
|
||||
gap_obj = yaml.safe_load(gap_disposition_path.read_text(encoding="utf-8")) or {}
|
||||
semantic_backlog = dict(gap_obj.get("summary") or {})
|
||||
pending = int((semantic_backlog or {}).get("pending_semantic_review") or 0)
|
||||
semantic_completeness = "COMPLETE" if pending == 0 else "PENDING_REVIEW"
|
||||
|
||||
nav_name_conflict_audit = _finalize_nav_name_conflict_audit(
|
||||
nav_name_conflict_state
|
||||
)
|
||||
|
||||
if s52_runtime_leaks:
|
||||
s52_runtime_failures.append(f"dynamic final state leaked into runtime PBF: {dict(s52_runtime_leaks)}")
|
||||
if light_presentation_required:
|
||||
if report_multi_light_groups != pbf_multi_light_groups:
|
||||
light_presentation_failures.append(
|
||||
"report/PBF multi-light group count mismatch: "
|
||||
f"report={report_multi_light_groups} pbf={pbf_multi_light_groups}"
|
||||
)
|
||||
if report_light_presentation_records != pbf_light_presentation_records:
|
||||
light_presentation_failures.append(
|
||||
"report/PBF light presentation record count mismatch: "
|
||||
f"report={report_light_presentation_records} pbf={pbf_light_presentation_records}"
|
||||
)
|
||||
s52_light_presentation_contract = {
|
||||
"status": (
|
||||
"PASS" if not light_presentation_failures else "FAIL"
|
||||
) if light_presentation_required else "NOT_APPLICABLE",
|
||||
"required": light_presentation_required,
|
||||
"report_multi_light_groups": report_multi_light_groups,
|
||||
"pbf_multi_light_groups": pbf_multi_light_groups,
|
||||
"report_light_presentation_records": report_light_presentation_records,
|
||||
"pbf_light_presentation_records": pbf_light_presentation_records,
|
||||
"failure_count": len(light_presentation_failures),
|
||||
"failure_examples": light_presentation_failures[:100],
|
||||
}
|
||||
|
||||
presentation_conservation_required = _builder_contract_at_least(
|
||||
plan.get("builder_contract_version"), (0, 4, 11)
|
||||
)
|
||||
presentation_conservation_contract = _external_audit_contract(
|
||||
presentation_conservation_audit_path,
|
||||
required=presentation_conservation_required,
|
||||
expected_schema="navsea.cm95.s52-presentation-conservation-audit/v1",
|
||||
expected_plan_id=str(plan.get("plan_id") or ""),
|
||||
expected_pbf_dir=pbf_root,
|
||||
)
|
||||
runtime_capability_contract = _external_audit_contract(
|
||||
runtime_capability_audit_path,
|
||||
required=False,
|
||||
expected_schema="navsea.cm95.s52-runtime-capability-audit/v1",
|
||||
expected_plan_id=str(plan.get("plan_id") or ""),
|
||||
expected_pbf_dir=None,
|
||||
)
|
||||
|
||||
s52_runtime_contract = {
|
||||
"status": ("PASS" if not s52_runtime_failures else "FAIL") if s52_runtime_enabled else "NOT_APPLICABLE",
|
||||
"enabled": s52_runtime_enabled,
|
||||
"mariner_dependent_count": s52_runtime_dependent_count,
|
||||
"runtime_csp_counts": dict(sorted(s52_runtime_csp_counts.items())),
|
||||
"runtime_layer_counts": dict(sorted(s52_runtime_layer_counts.items())),
|
||||
"fact_counts": dict(sorted(s52_runtime_fact_counts.items())),
|
||||
"unknown_fact_exceptions": dict(sorted(s52_runtime_unknown_fact_exceptions.items())),
|
||||
"dynamic_final_leaks": dict(sorted(s52_runtime_leaks.items())),
|
||||
"failure_count": len(s52_runtime_failures),
|
||||
"failure_examples": s52_runtime_failures[:100],
|
||||
}
|
||||
|
||||
tile_enumeration_mode = str(plan.get("tile_enumeration_mode") or "coverage")
|
||||
pbf_rel_paths = {
|
||||
path.relative_to(pbf_root).as_posix()
|
||||
for path in pbf_files
|
||||
}
|
||||
explicit_empty_tiles: list[str] = []
|
||||
missing_nonempty_pbf_tiles: list[str] = []
|
||||
report_tile_paths: set[str] = set()
|
||||
for path in report_files:
|
||||
obj = json.loads(path.read_text(encoding="utf-8"))
|
||||
rel_pbf = path.relative_to(report_root).with_suffix(".pbf").as_posix()
|
||||
report_tile_paths.add(rel_pbf)
|
||||
if rel_pbf in pbf_rel_paths:
|
||||
continue
|
||||
planned_count = int(obj.get("planned_output_feature_count") or 0)
|
||||
output_count = int(obj.get("output_feature_count") or 0)
|
||||
hard_gate_count = len(obj.get("hard_gate_unmapped") or [])
|
||||
encoder_drop_count = len(obj.get("encoder_dropped_features") or [])
|
||||
if (
|
||||
planned_count == 0
|
||||
and output_count == 0
|
||||
and hard_gate_count == 0
|
||||
and encoder_drop_count == 0
|
||||
):
|
||||
explicit_empty_tiles.append(rel_pbf)
|
||||
else:
|
||||
missing_nonempty_pbf_tiles.append(rel_pbf)
|
||||
|
||||
unexpected_pbf_tiles = sorted(pbf_rel_paths - report_tile_paths)
|
||||
tile_footprint_contract = {
|
||||
"status": "NOT_APPLICABLE",
|
||||
"mode": tile_enumeration_mode,
|
||||
"planned_tile_jobs": int(plan["job_count"]),
|
||||
"pbf_file_count": len(pbf_files),
|
||||
"report_file_count": len(report_files),
|
||||
"explicit_empty_tile_count": len(explicit_empty_tiles),
|
||||
"materialized_or_empty_tile_count": len(pbf_files) + len(explicit_empty_tiles),
|
||||
"missing_nonempty_pbf_tiles": missing_nonempty_pbf_tiles,
|
||||
"unexpected_pbf_tiles": unexpected_pbf_tiles,
|
||||
}
|
||||
if tile_enumeration_mode == "aoi-complete":
|
||||
tile_footprint_contract["status"] = (
|
||||
"PASS"
|
||||
if (
|
||||
len(report_files) == int(plan["job_count"])
|
||||
and len(pbf_files) + len(explicit_empty_tiles) == int(plan["job_count"])
|
||||
and not missing_nonempty_pbf_tiles
|
||||
and not unexpected_pbf_tiles
|
||||
)
|
||||
else "FAIL"
|
||||
)
|
||||
|
||||
s52_resource_contract = {
|
||||
"status": "NOT_APPLICABLE",
|
||||
"path": plan.get("s52_resource_contract_path"),
|
||||
"sha256": None,
|
||||
"expected_sha256": (plan.get("s52") or {}).get("resource_contract_sha256"),
|
||||
"defined_resource_counts": None,
|
||||
"active_resource_counts": None,
|
||||
"failures": [],
|
||||
}
|
||||
expected_resource_sha = str(
|
||||
(plan.get("s52") or {}).get("resource_contract_sha256") or ""
|
||||
)
|
||||
if expected_resource_sha:
|
||||
resource_path_raw = plan.get("s52_resource_contract_path")
|
||||
resource_failures: list[str] = []
|
||||
resource_obj: dict[str, Any] = {}
|
||||
if not resource_path_raw:
|
||||
resource_failures.append("missing_resource_contract_path")
|
||||
else:
|
||||
resource_path = Path(resource_path_raw)
|
||||
if not resource_path.exists():
|
||||
resource_failures.append("resource_contract_file_missing")
|
||||
else:
|
||||
actual_sha = sha256_file(resource_path)
|
||||
s52_resource_contract["sha256"] = actual_sha
|
||||
if actual_sha != expected_resource_sha:
|
||||
resource_failures.append("resource_contract_sha256_mismatch")
|
||||
try:
|
||||
resource_obj = json.loads(
|
||||
resource_path.read_text(encoding="utf-8")
|
||||
)
|
||||
except Exception as exc:
|
||||
resource_failures.append(
|
||||
f"resource_contract_parse_error:{type(exc).__name__}"
|
||||
)
|
||||
else:
|
||||
if resource_obj.get("status") != "PASS":
|
||||
resource_failures.append("resource_contract_not_pass")
|
||||
s52_resource_contract["defined_resource_counts"] = (
|
||||
resource_obj.get("defined_resource_counts")
|
||||
)
|
||||
s52_resource_contract["active_resource_counts"] = (
|
||||
resource_obj.get("active_resource_counts")
|
||||
)
|
||||
s52_resource_contract["failures"] = resource_failures
|
||||
s52_resource_contract["status"] = (
|
||||
"PASS" if not resource_failures else "FAIL"
|
||||
)
|
||||
|
||||
failures = []
|
||||
if expected_resource_sha and s52_resource_contract["status"] != "PASS":
|
||||
failures.append("s52_resource_contract")
|
||||
if tile_enumeration_mode == "aoi-complete":
|
||||
if len(report_files) != int(plan["job_count"]):
|
||||
failures.append("aoi_complete_report_count_mismatch")
|
||||
if tile_footprint_contract["status"] != "PASS":
|
||||
failures.append("aoi_complete_tile_footprint_mismatch")
|
||||
if hard_gate:
|
||||
failures.append("hard_gate_unmapped")
|
||||
if encoder_dropped:
|
||||
failures.append("encoder_dropped_features")
|
||||
if name_conflicts and nav_name_conflict_audit.get("overall") != "PASS":
|
||||
failures.append("nav_name_conflict_audit")
|
||||
if planned != output:
|
||||
failures.append("planned_vs_report_output_mismatch")
|
||||
if output != decoded_features:
|
||||
failures.append("report_output_vs_decoded_pbf_mismatch")
|
||||
if duplicate_feature_ids:
|
||||
failures.append("duplicate_feature_ids_within_tile_layer")
|
||||
if style_contract is not None and style_contract.get("overall") != "PASS":
|
||||
failures.append("style_or_sprite_contract")
|
||||
if s52_runtime_enabled and s52_runtime_contract.get("status") != "PASS":
|
||||
failures.append("s52_runtime_contract")
|
||||
if light_presentation_required and s52_light_presentation_contract.get("status") != "PASS":
|
||||
failures.append("s52_light_presentation_contract")
|
||||
if presentation_conservation_required and presentation_conservation_contract.get("status") != "PASS":
|
||||
failures.append("s52_presentation_conservation_contract")
|
||||
if runtime_capability_audit_path is not None and runtime_capability_contract.get("status") != "PASS":
|
||||
failures.append("s52_runtime_capability_contract")
|
||||
|
||||
result = {
|
||||
"schema": RELEASE_SCHEMA,
|
||||
"created_at": _utc_now(),
|
||||
"status": "PASS" if not failures else "FAIL",
|
||||
"release_id": plan["release_id"],
|
||||
"plan_id": plan["plan_id"],
|
||||
"dataset_id": plan["dataset_id"],
|
||||
"dataset_fingerprint": plan["dataset_fingerprint"],
|
||||
"profile": plan["profile"],
|
||||
"zooms": plan["zooms"],
|
||||
"mapping_sha256": plan["mapping_sha256"],
|
||||
"policy_sha256": plan["policy_sha256"],
|
||||
"builder_contract_version": plan.get("builder_contract_version"),
|
||||
"production_pipeline_version": plan.get("production_pipeline_version"),
|
||||
"tile_enumeration_mode": tile_enumeration_mode,
|
||||
"tile_footprint_contract": tile_footprint_contract,
|
||||
"s52": plan.get("s52"),
|
||||
"aoi": plan.get("aoi"),
|
||||
"s52_resource_contract": s52_resource_contract,
|
||||
"s52_runtime_contract": s52_runtime_contract,
|
||||
"s52_light_presentation_contract": s52_light_presentation_contract,
|
||||
"s52_presentation_conservation_contract": presentation_conservation_contract,
|
||||
"s52_runtime_capability_contract": runtime_capability_contract,
|
||||
"style_sha256": style_sha,
|
||||
"sprite_json_sha256": sprite_sha,
|
||||
"gap_disposition_sha256": gap_disposition_sha,
|
||||
"semantic_completeness": semantic_completeness,
|
||||
"semantic_backlog": semantic_backlog,
|
||||
"style_sprite_contract": style_contract,
|
||||
"planned_tile_jobs": int(plan["job_count"]),
|
||||
"pbf_file_count": len(pbf_files),
|
||||
"report_file_count": len(report_files),
|
||||
"planned_output_features": planned,
|
||||
"report_output_features": output,
|
||||
"decoded_pbf_features": decoded_features,
|
||||
"non_renderable_omissions": non_renderable_omissions,
|
||||
"non_renderable_omission_layers": dict(sorted(non_renderable_omission_layers.items())),
|
||||
"quantization_fallbacks": quantization_fallbacks,
|
||||
"quantization_omissions": quantization_omissions,
|
||||
"quantization_omission_layers": dict(sorted(quantization_omission_layers.items())),
|
||||
"encoder_reconciled_tiles": encoder_reconciled_tiles,
|
||||
"encoder_reconciled_drop_candidates": encoder_reconciled_candidates,
|
||||
"hard_gate_unmapped": hard_gate,
|
||||
"encoder_dropped_features": encoder_dropped,
|
||||
"nav_name_conflicts": name_conflicts,
|
||||
"nav_name_conflict_omissions": nav_name_conflict_audit.get("safe_omission_occurrences", 0),
|
||||
"nav_name_conflict_unique_outputs": nav_name_conflict_audit.get("unique_conflict_outputs", 0),
|
||||
"nav_name_conflict_audit": nav_name_conflict_audit,
|
||||
"duplicate_feature_ids": duplicate_feature_ids,
|
||||
"output_layer_counts": dict(sorted(decoded_layers.items())),
|
||||
"failures": failures,
|
||||
"inventory": inventory,
|
||||
}
|
||||
out = release_dir / "release_manifest_v1.json"
|
||||
out.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return result
|
||||
652
src/cm95pbf/s52.py
Normal file
652
src/cm95pbf/s52.py
Normal file
@@ -0,0 +1,652 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import struct
|
||||
from typing import Any, Iterable
|
||||
|
||||
from .presentation import (
|
||||
materialize_rule_ir,
|
||||
materialize_symbol_hints,
|
||||
materialize_text_hints,
|
||||
parse_instruction_string,
|
||||
static_rule_hints,
|
||||
)
|
||||
from .csp import (
|
||||
S52MarinerSettings,
|
||||
is_mariner_dependent_csp,
|
||||
materialize_mariner_runtime_facts,
|
||||
resolve_csp_hints,
|
||||
)
|
||||
|
||||
|
||||
S52_DISPLAY_PRIORITY_BY_CODE = {
|
||||
"0": "No data",
|
||||
"1": "Group 1",
|
||||
"2": "Area 1",
|
||||
"3": "Area 2",
|
||||
"4": "Point Symbol",
|
||||
"5": "Line Symbol",
|
||||
"6": "Area Symbol",
|
||||
"7": "Routing",
|
||||
"8": "Hazards",
|
||||
"9": "Mariners",
|
||||
}
|
||||
|
||||
|
||||
def _apply_presentation_side_effects(out: dict[str, Any]) -> None:
|
||||
if int(out.get("s52_display_base_promoted") or 0) == 1:
|
||||
out["s52_display_category"] = "Displaybase"
|
||||
|
||||
|
||||
OPENCPN_CM93_CLASS_SUBSTITUTIONS: dict[str, str] = {
|
||||
"ITDARE": "DEPARE",
|
||||
"_m_sor": "M_COVR",
|
||||
"SPOGRD": "DMPGRD",
|
||||
"FSHHAV": "FSHFAC",
|
||||
"OFSPRD": "CTNARE",
|
||||
"BRTFAC": "BERTHS",
|
||||
"TNKCON": "SILTNK",
|
||||
"ROADPT": "ROADWY",
|
||||
"VEGARE": "VEGATN",
|
||||
"PRDINS": "PRDARE",
|
||||
"DYKARE": "DYKCON",
|
||||
"DYKCRW": "DYKCON",
|
||||
"SLTPAN": "LNDRGN",
|
||||
"TELPHC": "CONVYR",
|
||||
"DIFFUS": "OBSTRN",
|
||||
"SLIPWY": "SLCONS",
|
||||
"LNDSTS": "SLCONS",
|
||||
"RMPARE": "SLCONS",
|
||||
}
|
||||
|
||||
# OpenCPN's current Presentation Library contains these extension Lookup names
|
||||
# in lowercase, while this CM93 dictionary stores the corresponding classes
|
||||
# in uppercase. Keep the aliases explicit: lookup matching must remain
|
||||
# case-sensitive because other extension classes (for example TOPMAR/topmar)
|
||||
# have distinct semantics.
|
||||
CM93_PRESENTATION_CLASS_ALIASES: dict[str, str] = {
|
||||
"RIVBNK": "rivbnk",
|
||||
"CANBNK": "canbnk",
|
||||
}
|
||||
|
||||
CM93_LANDMARK_CLASS_TO_CATLMK: dict[str, int] = {
|
||||
"CHIMNY": 3,
|
||||
"DSHAER": 4,
|
||||
"FLGSTF": 5,
|
||||
"FLASTK": 6,
|
||||
"MSTCON": 7,
|
||||
"WNDMIL": 18,
|
||||
"WIMCON": 19,
|
||||
}
|
||||
|
||||
CM93_CLASS_SYNTHETIC_ATTRIBUTES: dict[str, dict[str, Any]] = {
|
||||
"SILBUI": {"CATSIL": 1},
|
||||
"SLTPAN": {"CATLND": 15},
|
||||
"TELPHC": {"CATCON": 1},
|
||||
"DIFFUS": {"CATOBS": 3},
|
||||
"SLIPWY": {"CATSLC": 13},
|
||||
"LNDSTS": {"CATSLC": 11},
|
||||
"RMPARE": {"CATSLC": 12},
|
||||
}
|
||||
|
||||
|
||||
def normalize_cm93_object_class_for_s52(object_class: str) -> str:
|
||||
"""Return the OpenCPN-compatible Presentation Library object class."""
|
||||
name = str(object_class or "")
|
||||
if name in CM93_LANDMARK_CLASS_TO_CATLMK:
|
||||
return "LNDMRK"
|
||||
if name == "SILBUI":
|
||||
return "SILTNK"
|
||||
name = OPENCPN_CM93_CLASS_SUBSTITUTIONS.get(name, name)
|
||||
return CM93_PRESENTATION_CLASS_ALIASES.get(name, name)
|
||||
|
||||
|
||||
def normalize_cm93_object_for_s52(
|
||||
object_class: str, geometry_type: str, attrs: dict[str, Any]
|
||||
) -> tuple[str, dict[str, Any]]:
|
||||
"""Normalize CM93 class/attributes into the fixed OpenCPN S-52 view."""
|
||||
source_class = str(object_class or "")
|
||||
lookup_class = normalize_cm93_object_class_for_s52(source_class)
|
||||
out = normalize_cm93_attributes_for_s52(attrs)
|
||||
|
||||
if source_class == "$AREAS" and str(out.get("$SCODE") or "") == "II25":
|
||||
out["$SCODE"] = "BACKGROUND"
|
||||
|
||||
if str(geometry_type).upper() == "POINT":
|
||||
catlmk = CM93_LANDMARK_CLASS_TO_CATLMK.get(source_class)
|
||||
if catlmk is not None:
|
||||
out.setdefault("CATLMK", catlmk)
|
||||
|
||||
aton = (
|
||||
lookup_class.startswith("LIT")
|
||||
or lookup_class.startswith("LIGHTS")
|
||||
or lookup_class.startswith("BCN")
|
||||
or lookup_class.startswith("_slgto")
|
||||
or lookup_class.startswith("_boygn")
|
||||
or lookup_class.startswith("_bcngn")
|
||||
or lookup_class.startswith("_extgn")
|
||||
or lookup_class.startswith("TOWERS")
|
||||
or lookup_class.startswith("BOY")
|
||||
)
|
||||
if aton and "OBJNAM" not in out and "INFORM" in out:
|
||||
out["OBJNAM"] = out.pop("INFORM")
|
||||
|
||||
for key, value in CM93_CLASS_SYNTHETIC_ATTRIBUTES.get(source_class, {}).items():
|
||||
out.setdefault(key, value)
|
||||
return lookup_class, out
|
||||
|
||||
|
||||
CM93_COLMAR_TO_S57_COLOUR: dict[int, str] = {
|
||||
1: "4", # green
|
||||
2: "2", # black
|
||||
3: "3", # red
|
||||
4: "6", # yellow
|
||||
5: "1", # white
|
||||
6: "11", # orange
|
||||
7: "2,6", # black/yellow
|
||||
8: "2,6,2", # black/yellow/black
|
||||
9: "6,2", # yellow/black
|
||||
10: "6,2,6", # yellow/black/yellow
|
||||
11: "3,1", # red/white
|
||||
12: "4,3,4", # green/red/green
|
||||
13: "3,4,3", # red/green/red
|
||||
14: "2,3,2", # black/red/black
|
||||
15: "6,3,6", # yellow/red/yellow
|
||||
16: "4,3", # green/red
|
||||
17: "3,4", # red/green
|
||||
18: "4,1", # green/white
|
||||
}
|
||||
|
||||
|
||||
TOPMAR01_FLOATING_SYMBOLS: dict[int, str] = {
|
||||
1: "TOPMAR02", 2: "TOPMAR04", 3: "TOPMAR10", 4: "TOPMAR12",
|
||||
5: "TOPMAR13", 6: "TOPMAR14", 7: "TOPMAR65", 8: "TOPMAR17",
|
||||
9: "TOPMAR16", 10: "TOPMAR08", 11: "TOPMAR07", 12: "TOPMAR14",
|
||||
13: "TOPMAR05", 14: "TOPMAR06", 17: "TMARDEF2", 18: "TOPMAR10",
|
||||
19: "TOPMAR13", 20: "TOPMAR14", 21: "TOPMAR13", 22: "TOPMAR14",
|
||||
23: "TOPMAR14", 24: "TOPMAR02", 25: "TOPMAR04", 26: "TOPMAR10",
|
||||
27: "TOPMAR17", 28: "TOPMAR18", 29: "TOPMAR02", 30: "TOPMAR17",
|
||||
31: "TOPMAR14", 32: "TOPMAR10", 33: "TMARDEF2",
|
||||
}
|
||||
|
||||
TOPMAR01_RIGID_SYMBOLS: dict[int, str] = {
|
||||
1: "TOPMAR22", 2: "TOPMAR24", 3: "TOPMAR30", 4: "TOPMAR32",
|
||||
5: "TOPMAR33", 6: "TOPMAR34", 7: "TOPMAR85", 8: "TOPMAR86",
|
||||
9: "TOPMAR36", 10: "TOPMAR28", 11: "TOPMAR27", 12: "TOPMAR14",
|
||||
13: "TOPMAR25", 14: "TOPMAR26", 15: "TOPMAR88", 16: "TOPMAR87",
|
||||
17: "TMARDEF1", 18: "TOPMAR30", 19: "TOPMAR33", 20: "TOPMAR34",
|
||||
21: "TOPMAR33", 22: "TOPMAR34", 23: "TOPMAR34", 24: "TOPMAR22",
|
||||
25: "TOPMAR24", 26: "TOPMAR30", 27: "TOPMAR86", 28: "TOPMAR89",
|
||||
29: "TOPMAR22", 30: "TOPMAR86", 31: "TOPMAR14", 32: "TOPMAR30",
|
||||
33: "TMARDEF1",
|
||||
}
|
||||
|
||||
|
||||
def _s57_list_string(value: Any) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, (list, tuple)):
|
||||
return ",".join(str(int(x)) for x in value)
|
||||
if isinstance(value, bool):
|
||||
return str(int(value))
|
||||
if isinstance(value, int):
|
||||
return str(value)
|
||||
if isinstance(value, float) and value.is_integer():
|
||||
return str(int(value))
|
||||
return str(value)
|
||||
|
||||
|
||||
def normalize_cm93_attributes_for_s52(attrs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Return the fixed OpenCPN CM93-to-S57 typed attribute view."""
|
||||
out = dict(attrs)
|
||||
|
||||
for key, value in list(out.items()):
|
||||
if len(str(key)) == 6 and isinstance(value, (list, tuple)):
|
||||
normalized = _s57_list_string(value)
|
||||
if normalized is not None:
|
||||
out[key] = normalized
|
||||
|
||||
if "COLOUR" in out:
|
||||
value = _s57_list_string(out.get("COLOUR"))
|
||||
if value is not None:
|
||||
out["COLOUR"] = value
|
||||
elif "COLMAR" in out:
|
||||
try:
|
||||
code = int(out["COLMAR"])
|
||||
except (TypeError, ValueError):
|
||||
code = -1
|
||||
translated = CM93_COLMAR_TO_S57_COLOUR.get(code)
|
||||
if translated is not None:
|
||||
out["COLOUR"] = translated
|
||||
|
||||
for key in ("QUASOU", "CATLIT"):
|
||||
value = out.get(key)
|
||||
if isinstance(value, int):
|
||||
out[key] = str(int(value))
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def split_lookup_condition(condition: str) -> tuple[str, str]:
|
||||
text = str(condition)
|
||||
if len(text) < 6:
|
||||
raise ValueError(f"invalid S-52 lookup condition: {condition!r}")
|
||||
return text[:6].upper(), text[6:]
|
||||
|
||||
|
||||
def _c_atoi(value: str) -> int:
|
||||
"""Minimal C atoi semantics used by fixed OpenCPN FindBestLUP."""
|
||||
match = re.match(r"^[\s]*([+-]?\d+)", str(value))
|
||||
return int(match.group(1)) if match else 0
|
||||
|
||||
|
||||
def _c_atof_float32(value: str) -> float:
|
||||
"""OpenCPN assigns atof() to float before exact REAL comparison."""
|
||||
try:
|
||||
parsed = float(str(value).strip() or "0")
|
||||
except ValueError:
|
||||
parsed = 0.0
|
||||
return struct.unpack("f", struct.pack("f", parsed))[0]
|
||||
|
||||
|
||||
def lookup_condition_matches(condition: str, attrs: dict[str, Any]) -> bool:
|
||||
"""Match one condition as fixed OpenCPN FindBestLUP does."""
|
||||
acronym, expected = split_lookup_condition(condition)
|
||||
actual_present = acronym in attrs and attrs.get(acronym) is not None
|
||||
|
||||
if expected.startswith(" "):
|
||||
return actual_present
|
||||
if expected.startswith("?"):
|
||||
# Preserve the fixed-source quirk: this path never increments countATT.
|
||||
return False
|
||||
if not actual_present:
|
||||
return False
|
||||
|
||||
actual = attrs.get(acronym)
|
||||
if isinstance(actual, bool):
|
||||
actual = int(actual)
|
||||
if isinstance(actual, int):
|
||||
return actual == _c_atoi(expected)
|
||||
if isinstance(actual, float):
|
||||
return actual == _c_atof_float32(expected)
|
||||
return str(actual) == expected
|
||||
|
||||
|
||||
def _lookup_sequence(row: dict[str, Any]) -> int:
|
||||
try:
|
||||
return int(row.get("id") or 0)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def sort_lookup_candidates(rows: Iterable[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Match fixed OpenCPN CompareLUPObjects."""
|
||||
return sorted(
|
||||
rows,
|
||||
key=lambda row: (
|
||||
-len(row.get("attribute_conditions_raw") or []),
|
||||
_lookup_sequence(row),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _has_lookup_visible_attributes(attrs: dict[str, Any]) -> bool:
|
||||
return any(len(str(key)) == 6 and value is not None for key, value in attrs.items())
|
||||
|
||||
|
||||
def find_best_lookup(
|
||||
rows: Iterable[dict[str, Any]],
|
||||
attrs: dict[str, Any],
|
||||
*,
|
||||
table_name: str | None = None,
|
||||
geometry_type: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Reproduce non-strict fixed OpenCPN FindBestLUP selection."""
|
||||
normalized = normalize_cm93_attributes_for_s52(attrs)
|
||||
candidates = []
|
||||
for row in rows:
|
||||
if table_name and str(row.get("table_name")) != table_name:
|
||||
continue
|
||||
if geometry_type and str(row.get("geometry_type")) != geometry_type:
|
||||
continue
|
||||
candidates.append(row)
|
||||
|
||||
ordered = sort_lookup_candidates(candidates)
|
||||
if not ordered:
|
||||
return None
|
||||
|
||||
first = ordered[0]
|
||||
if not _has_lookup_visible_attributes(normalized):
|
||||
return first
|
||||
|
||||
for row in ordered:
|
||||
conditions = row.get("attribute_conditions_raw") or []
|
||||
if not conditions:
|
||||
continue
|
||||
count_att = sum(
|
||||
1 for condition in conditions
|
||||
if lookup_condition_matches(condition, normalized)
|
||||
)
|
||||
if count_att == len(conditions):
|
||||
return row
|
||||
|
||||
for row in ordered:
|
||||
if not (row.get("attribute_conditions_raw") or []):
|
||||
return row
|
||||
return first
|
||||
|
||||
|
||||
def primary_symbol_code(row: dict[str, Any] | None) -> str | None:
|
||||
if not row:
|
||||
return None
|
||||
symbols = row.get("sy") or []
|
||||
if not symbols:
|
||||
return None
|
||||
return str(symbols[0])
|
||||
|
||||
|
||||
def conditional_procedures(row: dict[str, Any] | None) -> tuple[str, ...]:
|
||||
if not row:
|
||||
return ()
|
||||
return tuple(str(x) for x in (row.get("cs") or []))
|
||||
|
||||
|
||||
def _s52_int_list(value: Any) -> list[int]:
|
||||
if value is None or value == "":
|
||||
return []
|
||||
if isinstance(value, (list, tuple)):
|
||||
raw = value
|
||||
elif isinstance(value, str):
|
||||
text = value.strip().strip("[]")
|
||||
raw = [part.strip() for part in text.split(",") if part.strip()]
|
||||
else:
|
||||
raw = [value]
|
||||
out: list[int] = []
|
||||
for item in raw:
|
||||
try:
|
||||
out.append(int(item))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
def _s52_float(value: Any, default: float | None = None) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _lights05_color(colours: list[int]) -> str:
|
||||
if len(colours) == 1:
|
||||
if 3 in colours:
|
||||
return "red"
|
||||
if 4 in colours:
|
||||
return "green"
|
||||
if any(code in colours for code in (1, 6, 11)):
|
||||
return "yellow_white"
|
||||
return "magenta"
|
||||
if len(colours) == 2:
|
||||
if 1 in colours and 3 in colours:
|
||||
return "red"
|
||||
if 1 in colours and 4 in colours:
|
||||
return "green"
|
||||
return "magenta"
|
||||
|
||||
|
||||
def _lights05_flare_symbol(colours: list[int]) -> str:
|
||||
color = _lights05_color(colours)
|
||||
if color == "red":
|
||||
return "LIGHTS11"
|
||||
if color == "green":
|
||||
return "LIGHTS12"
|
||||
if color == "yellow_white":
|
||||
return "LIGHTS13"
|
||||
return "LITDEF11"
|
||||
|
||||
|
||||
def resolve_lights05(attrs: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Resolve OpenCPN PLIB LIGHTS05/LIGHTS06 into runtime-friendly fields."""
|
||||
catlit = _s52_int_list(attrs.get("CATLIT"))
|
||||
if 8 in catlit or 11 in catlit:
|
||||
return {"s52_csp_resolved": "LIGHTS05", "s52_light_mode": "special", "s52_light_symbol_code": "LIGHTS82"}
|
||||
if 9 in catlit:
|
||||
return {"s52_csp_resolved": "LIGHTS05", "s52_light_mode": "special", "s52_light_symbol_code": "LIGHTS81"}
|
||||
|
||||
colours = _s52_int_list(attrs.get("COLOUR")) or [12]
|
||||
valnmr = _s52_float(attrs.get("VALNMR"), 9.0) or 9.0
|
||||
sectr1 = _s52_float(attrs.get("SECTR1"))
|
||||
sectr2 = _s52_float(attrs.get("SECTR2"))
|
||||
arc_color = _lights05_color(colours)
|
||||
litvis = _s52_int_list(attrs.get("LITVIS"))
|
||||
obscured = any(code in litvis for code in (3, 7, 8))
|
||||
if obscured:
|
||||
arc_color = "obscured"
|
||||
|
||||
base: dict[str, Any] = {
|
||||
"s52_csp_resolved": "LIGHTS05",
|
||||
"s52_light_arc_color": arc_color,
|
||||
"s52_light_nominal_range_nm": float(valnmr),
|
||||
}
|
||||
if sectr1 is None or sectr2 is None:
|
||||
if valnmr < 10.0:
|
||||
base.update({"s52_light_mode": "flare", "s52_light_symbol_code": _lights05_flare_symbol(colours), "s52_light_rotation_deg": 135.0})
|
||||
return base
|
||||
radius = 3.0 if valnmr < 7.0 else 10.0 if valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0
|
||||
base.update({"s52_light_mode": "all_round", "s52_light_arc_radius_mm": radius})
|
||||
return base
|
||||
|
||||
sweep = (sectr2 - sectr1) % 360.0
|
||||
if sweep < 1.0:
|
||||
base.update({"s52_light_mode": "all_round", "s52_light_arc_radius_mm": 10.0 if valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0})
|
||||
return base
|
||||
|
||||
arc_radius = 10.0 if 0 < valnmr < 15.0 else 15.0 if valnmr < 30.0 else 20.0
|
||||
base.update({
|
||||
"s52_light_mode": "sector",
|
||||
"s52_sector_start_deg": float(sectr1),
|
||||
"s52_sector_end_deg": float(sectr2),
|
||||
"s52_sector_sweep_deg": float(sweep),
|
||||
"s52_sector_display_start_deg": float((sectr1 + 180.0) % 360.0),
|
||||
"s52_sector_display_end_deg": float((sectr2 + 180.0) % 360.0),
|
||||
"s52_light_arc_radius_mm": float(arc_radius),
|
||||
"s52_light_sector_radius_mm": 25.0,
|
||||
"s52_light_obscured": bool(obscured),
|
||||
})
|
||||
return base
|
||||
|
||||
|
||||
class S52LookupRegistry:
|
||||
"""Read extracted S-52 lookups once and expose PBF-ready properties."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
path,
|
||||
point_table: str = "Paper",
|
||||
boundary_table: str = "Plain",
|
||||
mariner_mode: str = "runtime",
|
||||
):
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
self.path = Path(path)
|
||||
self.point_table = str(point_table)
|
||||
self.boundary_table = str(boundary_table)
|
||||
self.mariner_mode = str(mariner_mode)
|
||||
if self.mariner_mode not in {"runtime", "reference"}:
|
||||
raise ValueError("mariner_mode must be runtime or reference")
|
||||
self.settings = S52MarinerSettings(symbolized_boundaries=self.boundary_table == "Symbolized")
|
||||
rows = json.loads(self.path.read_text(encoding="utf-8"))
|
||||
self.by_class: dict[str, list[dict[str, Any]]] = {}
|
||||
for row in rows:
|
||||
row = dict(row)
|
||||
row["_static_hints"] = static_rule_hints(
|
||||
str(row.get("instruction") or "")
|
||||
)
|
||||
object_class = str(row.get("object_class") or "")
|
||||
self.by_class.setdefault(object_class, []).append(row)
|
||||
|
||||
def resolve(
|
||||
self,
|
||||
object_class: str,
|
||||
geometry_type: str,
|
||||
attrs: dict[str, Any],
|
||||
) -> dict[str, Any] | None:
|
||||
geometry = str(geometry_type).upper()
|
||||
table_geometry = {
|
||||
"POINT": (self.point_table, "Point"),
|
||||
"SOUNDING": (self.point_table, "Point"),
|
||||
"LINE": ("Lines", "Line"),
|
||||
"AREA": (self.boundary_table, "Area"),
|
||||
}.get(geometry)
|
||||
if table_geometry is None:
|
||||
return None
|
||||
table_name, lookup_geometry = table_geometry
|
||||
lookup_class, lookup_attrs = normalize_cm93_object_for_s52(
|
||||
object_class, geometry_type, attrs
|
||||
)
|
||||
return find_best_lookup(
|
||||
self.by_class.get(lookup_class, []),
|
||||
lookup_attrs,
|
||||
table_name=table_name,
|
||||
geometry_type=lookup_geometry,
|
||||
)
|
||||
|
||||
def _plain_area_instruction(self, object_class: str) -> str | None:
|
||||
row = find_best_lookup(
|
||||
self.by_class.get(object_class, []),
|
||||
{},
|
||||
table_name="Plain",
|
||||
geometry_type="Area",
|
||||
)
|
||||
if row is None:
|
||||
return None
|
||||
instruction = str(row.get("instruction") or "").strip()
|
||||
return instruction or None
|
||||
|
||||
def properties(
|
||||
self,
|
||||
object_class: str,
|
||||
geometry_type: str,
|
||||
attrs: dict[str, Any],
|
||||
*,
|
||||
next_safe_contour_m: float | None = None,
|
||||
sounding_depth_m: float | None = None,
|
||||
associated_depth_ranges: list[dict[str, Any]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
lookup_class, normalized_attrs = normalize_cm93_object_for_s52(
|
||||
object_class, geometry_type, attrs
|
||||
)
|
||||
row = self.resolve(object_class, geometry_type, attrs)
|
||||
if row is None:
|
||||
return {}
|
||||
|
||||
out: dict[str, Any] = {
|
||||
"s52_lookup_rcid": str(row.get("rcid") or ""),
|
||||
"s52_display_category": str(row.get("display_category") or ""),
|
||||
"s52_display_priority": str(row.get("display_priority") or ""),
|
||||
"s52_table_name": str(row.get("table_name") or ""),
|
||||
}
|
||||
instruction = str(row.get("instruction") or "")
|
||||
out.update(dict(row.get("_static_hints") or {}))
|
||||
out.update(materialize_text_hints(instruction, normalized_attrs))
|
||||
out.update(materialize_symbol_hints(instruction, normalized_attrs))
|
||||
if lookup_class != str(object_class):
|
||||
out["s52_object_class"] = lookup_class
|
||||
procedures = [
|
||||
token.strip()
|
||||
for token in str(out.get("s52_csp") or "").split(",")
|
||||
if token.strip()
|
||||
]
|
||||
safe_dashed_instruction = self._plain_area_instruction("SAFECD")
|
||||
safe_solid_instruction = self._plain_area_instruction("SAFECN")
|
||||
unresolved: list[str] = []
|
||||
resolved_by_procedure: dict[str, dict[str, Any]] = {}
|
||||
for procedure in procedures:
|
||||
if self.mariner_mode == "runtime" and is_mariner_dependent_csp(procedure):
|
||||
runtime_facts = materialize_mariner_runtime_facts(
|
||||
procedure,
|
||||
lookup_class,
|
||||
str(geometry_type),
|
||||
normalized_attrs,
|
||||
sounding_depth_m=sounding_depth_m,
|
||||
associated_depth_ranges=associated_depth_ranges,
|
||||
)
|
||||
if procedure == "DEPCNT02":
|
||||
if safe_dashed_instruction:
|
||||
runtime_facts["s52_safe_dashed_rule_chain"] = materialize_rule_ir(
|
||||
safe_dashed_instruction,
|
||||
normalized_attrs,
|
||||
source="lookup",
|
||||
procedure="SAFECD",
|
||||
)
|
||||
if safe_solid_instruction:
|
||||
runtime_facts["s52_safe_solid_rule_chain"] = materialize_rule_ir(
|
||||
safe_solid_instruction,
|
||||
normalized_attrs,
|
||||
source="lookup",
|
||||
procedure="SAFECN",
|
||||
)
|
||||
out.update(runtime_facts)
|
||||
unresolved.append(procedure)
|
||||
continue
|
||||
resolved = resolve_csp_hints(
|
||||
procedure,
|
||||
lookup_class,
|
||||
str(geometry_type),
|
||||
normalized_attrs,
|
||||
self.settings,
|
||||
next_safe_contour_m=next_safe_contour_m,
|
||||
sounding_depth_m=sounding_depth_m,
|
||||
associated_depth_ranges=associated_depth_ranges,
|
||||
safe_dashed_instruction=safe_dashed_instruction,
|
||||
safe_solid_instruction=safe_solid_instruction,
|
||||
)
|
||||
if resolved:
|
||||
out.update(resolved)
|
||||
resolved_by_procedure[procedure] = resolved
|
||||
else:
|
||||
unresolved.append(procedure)
|
||||
ordered_chain: list[dict[str, Any]] = []
|
||||
for rule in parse_instruction_string(instruction):
|
||||
if rule.opcode != "CS" or not rule.args:
|
||||
ordered_chain.extend(
|
||||
materialize_rule_ir(rule.raw, normalized_attrs, source="lookup")
|
||||
)
|
||||
continue
|
||||
procedure = rule.args[0].strip().strip("'\"")[:8]
|
||||
if procedure in unresolved:
|
||||
dynamic = materialize_rule_ir(
|
||||
rule.raw, normalized_attrs, source="lookup"
|
||||
)[0]
|
||||
dynamic["dynamic"] = True
|
||||
dynamic["procedure"] = procedure
|
||||
ordered_chain.append(dynamic)
|
||||
else:
|
||||
ordered_chain.extend(
|
||||
list(resolved_by_procedure.get(procedure, {}).get("s52_rule_chain") or [])
|
||||
)
|
||||
out["s52_rule_chain_version"] = 1
|
||||
out["s52_rule_chain"] = ordered_chain
|
||||
out["s52_source_object_class"] = lookup_class
|
||||
out["s52_source_geometry"] = str(geometry_type).upper()
|
||||
if unresolved:
|
||||
out["s52_csp"] = ",".join(unresolved)
|
||||
else:
|
||||
out.pop("s52_csp", None)
|
||||
_apply_presentation_side_effects(out)
|
||||
return {key: value for key, value in out.items() if value != ""}
|
||||
|
||||
|
||||
def resolve_topmar01_symbol(attrs: dict[str, Any], *, floating: bool) -> str:
|
||||
"""Resolve S-52 TOPMAR01 exactly as OpenCPN PLIB for known TOPSHP values."""
|
||||
value = attrs.get("TOPSHP")
|
||||
if value is None or value == "":
|
||||
return "QUESMRK1"
|
||||
try:
|
||||
topshp = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return "QUESMRK1"
|
||||
if floating:
|
||||
return TOPMAR01_FLOATING_SYMBOLS.get(topshp, "TMARDEF2")
|
||||
return TOPMAR01_RIGID_SYMBOLS.get(topshp, "TMARDEF1")
|
||||
174
src/cm95pbf/style_contract.py
Normal file
174
src/cm95pbf/style_contract.py
Normal file
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import Counter, defaultdict
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import mapbox_vector_tile
|
||||
|
||||
|
||||
SPRITE_PROPERTY_FIELDS = (
|
||||
"chart_icon_image",
|
||||
"chart_fill_pattern",
|
||||
"chart_line_pattern",
|
||||
)
|
||||
|
||||
|
||||
def ev(expr: Any, props: dict[str, Any]):
|
||||
if not isinstance(expr, list):
|
||||
return expr
|
||||
if not expr:
|
||||
return None
|
||||
op = expr[0]
|
||||
if op == "get":
|
||||
return props.get(expr[1])
|
||||
if op == "has":
|
||||
return expr[1] in props
|
||||
if op == "!":
|
||||
return not bool(ev(expr[1], props))
|
||||
if op == "all":
|
||||
return all(bool(ev(x, props)) for x in expr[1:])
|
||||
if op == "any":
|
||||
return any(bool(ev(x, props)) for x in expr[1:])
|
||||
if op == "==":
|
||||
return ev(expr[1], props) == ev(expr[2], props)
|
||||
if op == "!=":
|
||||
return ev(expr[1], props) != ev(expr[2], props)
|
||||
if op == "slice":
|
||||
value = ev(expr[1], props)
|
||||
if value is None:
|
||||
return None
|
||||
start = int(ev(expr[2], props))
|
||||
end = int(ev(expr[3], props)) if len(expr) > 3 else None
|
||||
return str(value)[start:end]
|
||||
if op == "to-number":
|
||||
value = ev(expr[1], props)
|
||||
try:
|
||||
num = float(value)
|
||||
return int(num) if num.is_integer() else num
|
||||
except Exception:
|
||||
return 0
|
||||
if op == "literal":
|
||||
return expr[1]
|
||||
if op == "match":
|
||||
value = ev(expr[1], props)
|
||||
pairs = expr[2:-1]
|
||||
default = expr[-1]
|
||||
for i in range(0, len(pairs), 2):
|
||||
label = pairs[i]
|
||||
output = pairs[i + 1]
|
||||
if isinstance(label, list) and label and label[0] == "literal":
|
||||
labels = label[1]
|
||||
if value in labels:
|
||||
return ev(output, props)
|
||||
elif isinstance(label, list):
|
||||
if value in label:
|
||||
return ev(output, props)
|
||||
elif value == label:
|
||||
return ev(output, props)
|
||||
return ev(default, props)
|
||||
raise RuntimeError(f"unsupported style filter op: {op}")
|
||||
|
||||
|
||||
def audit_pbf_style_and_sprite(
|
||||
pbf_root: Path,
|
||||
style_path: Path,
|
||||
sprite_json_path: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
pbf_root = Path(pbf_root)
|
||||
style = json.loads(Path(style_path).read_text(encoding="utf-8"))
|
||||
sprite_keys = None
|
||||
if sprite_json_path is not None:
|
||||
sprite_obj = json.loads(Path(sprite_json_path).read_text(encoding="utf-8"))
|
||||
sprite_keys = set(sprite_obj)
|
||||
|
||||
style_by_source_layer = defaultdict(list)
|
||||
for layer in style.get("layers", []):
|
||||
if layer.get("source") == "navsea_delivery" and layer.get("source-layer"):
|
||||
style_by_source_layer[layer["source-layer"]].append(layer)
|
||||
|
||||
files = sorted(pbf_root.rglob("*.pbf"))
|
||||
contract_unmatched = Counter()
|
||||
zoom_hidden = Counter()
|
||||
visible = Counter()
|
||||
sprite_missing = Counter()
|
||||
sprite_values = Counter()
|
||||
total_features = 0
|
||||
|
||||
for path in files:
|
||||
rel = path.relative_to(pbf_root)
|
||||
if len(rel.parts) < 3:
|
||||
raise RuntimeError(f"PBF path must end with z/x/y.pbf: {path}")
|
||||
z = int(rel.parts[-3])
|
||||
decoded = mapbox_vector_tile.decode(path.read_bytes())
|
||||
for source_layer, layer in decoded.items():
|
||||
style_layers = style_by_source_layer.get(source_layer, [])
|
||||
for feature in layer.get("features", []):
|
||||
total_features += 1
|
||||
props = feature.get("properties") or {}
|
||||
|
||||
filter_matches = []
|
||||
for sl in style_layers:
|
||||
flt = sl.get("filter")
|
||||
if flt is None or bool(ev(flt, props)):
|
||||
filter_matches.append(sl)
|
||||
|
||||
key = (z, source_layer)
|
||||
if not filter_matches:
|
||||
contract_unmatched[key] += 1
|
||||
else:
|
||||
zoom_matches = []
|
||||
for sl in filter_matches:
|
||||
minz = sl.get("minzoom")
|
||||
maxz = sl.get("maxzoom")
|
||||
if minz is not None and z < minz:
|
||||
continue
|
||||
if maxz is not None and z >= maxz:
|
||||
continue
|
||||
zoom_matches.append(sl)
|
||||
if zoom_matches:
|
||||
visible[key] += 1
|
||||
else:
|
||||
zoom_hidden[key] += 1
|
||||
|
||||
if sprite_keys is not None:
|
||||
for field in SPRITE_PROPERTY_FIELDS:
|
||||
value = props.get(field)
|
||||
if isinstance(value, str) and value:
|
||||
sprite_values[(field, value)] += 1
|
||||
if value not in sprite_keys:
|
||||
sprite_missing[(field, value)] += 1
|
||||
|
||||
return {
|
||||
"schema": "navsea.cm95.style-sprite-contract-audit/v1",
|
||||
"pbf_file_count": len(files),
|
||||
"feature_count": total_features,
|
||||
"style_source_layer_count": len(style_by_source_layer),
|
||||
"contract_unmatched_count": sum(contract_unmatched.values()),
|
||||
"zoom_hidden_count": sum(zoom_hidden.values()),
|
||||
"visible_count": sum(visible.values()),
|
||||
"sprite_key_count": None if sprite_keys is None else len(sprite_keys),
|
||||
"sprite_missing_count": sum(sprite_missing.values()),
|
||||
"contract_unmatched": [
|
||||
{"z": z, "source_layer": sl, "count": count}
|
||||
for (z, sl), count in sorted(contract_unmatched.items())
|
||||
],
|
||||
"zoom_hidden": [
|
||||
{"z": z, "source_layer": sl, "count": count}
|
||||
for (z, sl), count in sorted(zoom_hidden.items())
|
||||
],
|
||||
"sprite_missing": [
|
||||
{"field": field, "sprite_key": key, "count": count}
|
||||
for (field, key), count in sorted(sprite_missing.items())
|
||||
],
|
||||
"sprite_values": [
|
||||
{"field": field, "sprite_key": key, "count": count}
|
||||
for (field, key), count in sorted(sprite_values.items())
|
||||
],
|
||||
"overall": (
|
||||
"PASS"
|
||||
if sum(contract_unmatched.values()) == 0 and sum(sprite_missing.values()) == 0
|
||||
else "FAIL"
|
||||
),
|
||||
}
|
||||
47
tools/compare_pbf_trees.py
Executable file
47
tools/compare_pbf_trees.py
Executable file
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
import argparse, hashlib, json
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
def digest(p: Path):
|
||||
h=hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for b in iter(lambda:f.read(1024*1024),b""):
|
||||
h.update(b)
|
||||
return p.stat().st_size,h.hexdigest()
|
||||
|
||||
def inventory(root: Path):
|
||||
return {str(p.relative_to(root)):p for p in root.rglob("*.pbf") if p.is_file()}
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser()
|
||||
ap.add_argument("mysql_root")
|
||||
ap.add_argument("postgres_root")
|
||||
ap.add_argument("--output",default="logs/mysql_postgresql_pbf_compare.json")
|
||||
ap.add_argument("--workers",type=int,default=8)
|
||||
args=ap.parse_args()
|
||||
mr,pr=Path(args.mysql_root),Path(args.postgres_root)
|
||||
mi,pi=inventory(mr),inventory(pr)
|
||||
common=sorted(set(mi)&set(pi))
|
||||
def one(rel):
|
||||
ms,mh=digest(mi[rel]); ps,ph=digest(pi[rel])
|
||||
return rel,ms,mh,ps,ph
|
||||
identical=0; diffs=[]
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as ex:
|
||||
for rel,ms,mh,ps,ph in ex.map(one,common):
|
||||
if mh==ph and ms==ps: identical+=1
|
||||
else: diffs.append({"path":rel,"mysql_bytes":ms,"postgres_bytes":ps,"mysql_sha256":mh,"postgres_sha256":ph})
|
||||
out={
|
||||
"mysql_root":str(mr.resolve()),"postgres_root":str(pr.resolve()),
|
||||
"mysql_files":len(mi),"postgres_files":len(pi),"common_files":len(common),
|
||||
"identical_files":identical,"different_files":len(diffs),
|
||||
"mysql_only":sorted(set(mi)-set(pi)),"postgres_only":sorted(set(pi)-set(mi)),
|
||||
"different_examples":diffs[:100],
|
||||
"result":"PASS" if len(mi)==len(pi)==identical and not diffs else "DIFF",
|
||||
}
|
||||
op=Path(args.output);op.parent.mkdir(parents=True,exist_ok=True)
|
||||
op.write_text(json.dumps(out,ensure_ascii=False,indent=2),encoding="utf-8")
|
||||
print(json.dumps({k:v for k,v in out.items() if k not in ("mysql_only","postgres_only","different_examples")},ensure_ascii=False,indent=2))
|
||||
print(f"report={op}")
|
||||
if __name__=="__main__": main()
|
||||
1
web/assets/maplibre-gl.css
Normal file
1
web/assets/maplibre-gl.css
Normal file
File diff suppressed because one or more lines are too long
59
web/assets/maplibre-gl.js
Normal file
59
web/assets/maplibre-gl.js
Normal file
File diff suppressed because one or more lines are too long
61
web/mysql_postgresql_compare.html
Normal file
61
web/mysql_postgresql_compare.html
Normal file
@@ -0,0 +1,61 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>CM95 MySQL vs PostgreSQL PBF 对比</title>
|
||||
<link rel="stylesheet" href="/assets/maplibre-gl.css">
|
||||
<style>
|
||||
*{box-sizing:border-box}html,body{height:100%;margin:0;background:#111;color:#eee;font-family:system-ui,-apple-system,"Segoe UI",sans-serif}
|
||||
body{overflow:hidden}#bar{height:52px;display:flex;align-items:center;gap:12px;padding:0 12px;background:#17191d;border-bottom:1px solid #3b3f46;font-size:12px}
|
||||
#bar strong{font-size:14px;white-space:nowrap}#bar button{border:1px solid #555;background:#272b31;color:#fff;border-radius:5px;padding:5px 9px;cursor:pointer}
|
||||
#bar label{display:flex;gap:5px;align-items:center;white-space:nowrap}.muted{color:#aeb7c3}#readout{margin-left:auto;font-variant-numeric:tabular-nums;white-space:nowrap}
|
||||
#maps{height:calc(100% - 52px);display:grid;grid-template-columns:1fr 1fr;gap:2px;background:#555}
|
||||
.pane{position:relative;min-width:0;overflow:hidden;background:#d8edf5}.map{position:absolute;inset:0}
|
||||
.title{position:absolute;z-index:5;left:10px;top:10px;padding:7px 9px;border-radius:6px;background:rgba(18,20,24,.88);font-size:12px;line-height:1.4}
|
||||
.title b{display:block;font-size:14px}.state{font-size:11px;color:#cbd2dc}.ok{color:#91e39b}.err{color:#ff8c8c}
|
||||
.inspect{position:absolute;z-index:5;left:10px;right:10px;bottom:10px;max-height:28%;overflow:auto;padding:7px 9px;border-radius:6px;background:rgba(18,20,24,.9);font:11px/1.4 ui-monospace,SFMono-Regular,Consolas,monospace;white-space:pre-wrap}
|
||||
#stats{position:absolute;z-index:10;top:62px;left:50%;transform:translateX(-50%);max-width:72%;padding:7px 10px;border-radius:6px;background:rgba(18,20,24,.9);font:11px/1.35 ui-monospace,SFMono-Regular,Consolas,monospace;pointer-events:none}
|
||||
@media(max-width:820px){#maps{grid-template-columns:1fr;grid-template-rows:1fr 1fr}#bar .muted{display:none}#stats{max-width:94%}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="bar"><strong>CM95 PBF:MySQL vs PostgreSQL</strong><span class="muted">同源数据 / 同一 S-52 style / 同步视图</span>
|
||||
<label><input id="sync" type="checkbox" checked>同步</label><button id="reset">九州</button><button id="info">统计</button><span id="readout">--</span></div>
|
||||
<div id="stats" hidden></div>
|
||||
<div id="maps">
|
||||
<section class="pane"><div id="mysql-map" class="map"></div><div class="title"><b>左:MySQL</b><span id="mysql-state" class="state">加载中…</span></div><div id="mysql-inspect" class="inspect">点击任一侧,同点检查 Feature。</div></section>
|
||||
<section class="pane"><div id="pg-map" class="map"></div><div class="title"><b>右:PostgreSQL / PostGIS</b><span id="pg-state" class="state">加载中…</span></div><div id="pg-inspect" class="inspect">点击任一侧,同点检查 Feature。</div></section>
|
||||
</div>
|
||||
<script src="/assets/maplibre-gl.js"></script>
|
||||
<script>
|
||||
const BOUNDS=[[128.0,29.5],[132.5,34.8]], START={center:[130.25,32.15],zoom:6.6,bearing:0,pitch:0};
|
||||
const sync=document.getElementById("sync"), readout=document.getElementById("readout"), stats=document.getElementById("stats");
|
||||
let styleTemplate=null,syncing=false,m1=null,m2=null;
|
||||
function state(id,text,ok=true){const e=document.getElementById(id);e.textContent=text;e.className="state "+(ok?"ok":"err")}
|
||||
function blank(){return {version:8,sources:{},layers:[{id:"bg",type:"background",paint:{"background-color":"#d8edf5"}}]}}
|
||||
function styleFor(kind){const s=JSON.parse(JSON.stringify(styleTemplate));const src=s.sources&&s.sources.cm95;if(!src)throw new Error("style 缺少 sources.cm95");
|
||||
src.type="vector";src.tiles=[location.origin+"/"+kind+"/{z}/{x}/{y}.pbf"];src.minzoom=5;src.maxzoom=12;src.bounds=[128,29.5,132.5,34.8];delete src.url;s.sprite=location.origin+"/sprite";return s}
|
||||
const mysqlMap=new maplibregl.Map({container:"mysql-map",style:blank(),...START,attributionControl:false});
|
||||
const pgMap=new maplibregl.Map({container:"pg-map",style:blank(),...START,attributionControl:false});
|
||||
mysqlMap.addControl(new maplibregl.NavigationControl(),"top-right");pgMap.addControl(new maplibregl.NavigationControl(),"top-right");
|
||||
function camera(m){const c=m.getCenter();return {center:[c.lng,c.lat],zoom:m.getZoom(),bearing:m.getBearing(),pitch:m.getPitch()}}
|
||||
function link(a,b){if(syncing||!sync.checked)return;syncing=true;try{b.jumpTo(camera(a))}finally{syncing=false}}
|
||||
mysqlMap.on("move",()=>{link(mysqlMap,pgMap);showReadout(mysqlMap)});pgMap.on("move",()=>{link(pgMap,mysqlMap);showReadout(pgMap)});
|
||||
function showReadout(m){const c=m.getCenter();readout.textContent="z="+m.getZoom().toFixed(2)+" "+c.lng.toFixed(5)+", "+c.lat.toFixed(5)}
|
||||
document.getElementById("reset").onclick=()=>{mysqlMap.fitBounds(BOUNDS,{padding:36,duration:0});link(mysqlMap,pgMap)};
|
||||
document.getElementById("info").onclick=()=>{stats.hidden=!stats.hidden};
|
||||
async function load(){try{const [sr,ir]=await Promise.all([fetch("/style.json",{cache:"no-store"}),fetch("/api/info",{cache:"no-store"})]);styleTemplate=await sr.json();const info=await ir.json();
|
||||
stats.textContent="MySQL: "+info.mysql.release+" | "+info.mysql.files+" tiles\nPostgreSQL: "+info.postgresql.release+" | "+info.postgresql.files+" tiles\nCommon tile hash: "+info.compare.identical+"/"+info.compare.common+" identical";
|
||||
mysqlMap.setStyle(styleFor("mysql"));pgMap.setStyle(styleFor("postgres"));}catch(e){state("mysql-state",String(e),false);state("pg-state",String(e),false)}}
|
||||
mysqlMap.on("style.load",()=>state("mysql-state","S-52 / MySQL PBF",true));pgMap.on("style.load",()=>state("pg-state","S-52 / PostgreSQL PBF",true));
|
||||
function key(f){const p=f.properties||{};return [f.sourceLayer||"",f.id??"",p.canonical_object_type||"",p.source_object_class||""].join("|")}
|
||||
function summary(f){const p=f.properties||{};return [f.sourceLayer||"-","id="+(f.id??"-"),p.canonical_object_type||"",p.source_object_class||"",p.chart_label_text||"",p.contour_depth_m!==undefined?"depth="+p.contour_depth_m:""].filter(Boolean).join(" ")}
|
||||
function inspect(map,ll,id){const el=document.getElementById(id);if(!map.loaded()){el.textContent="地图加载中";return}const p=map.project(ll),box=[[p.x-5,p.y-5],[p.x+5,p.y+5]],u=new Map();
|
||||
for(const f of map.queryRenderedFeatures(box)){if(!u.has(key(f)))u.set(key(f),f)}const a=[...u.values()].slice(0,24);el.textContent=ll.lng.toFixed(6)+", "+ll.lat.toFixed(6)+"\n命中 "+u.size+" 项\n"+a.map((f,i)=>String(i+1).padStart(2,"0")+". "+summary(f)).join("\n")}
|
||||
function compare(ll){if(m1)m1.remove();if(m2)m2.remove();m1=new maplibregl.Marker().setLngLat(ll).addTo(mysqlMap);m2=new maplibregl.Marker().setLngLat(ll).addTo(pgMap);inspect(mysqlMap,ll,"mysql-inspect");inspect(pgMap,ll,"pg-inspect")}
|
||||
mysqlMap.on("click",e=>compare(e.lngLat));pgMap.on("click",e=>compare(e.lngLat));
|
||||
mysqlMap.on("error",e=>{const m=e?.error?.message||"";if(!m.includes("404"))state("mysql-state",m,false)});pgMap.on("error",e=>{const m=e?.error?.message||"";if(!m.includes("404"))state("pg-state",m,false)});
|
||||
window.addEventListener("resize",()=>{mysqlMap.resize();pgMap.resize()});showReadout(mysqlMap);load();
|
||||
</script>
|
||||
</body></html>
|
||||
105
web/mysql_postgresql_compare_server.py
Executable file
105
web/mysql_postgresql_compare_server.py
Executable file
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
def parse_args():
|
||||
p=argparse.ArgumentParser(description="CM95 MySQL vs PostgreSQL PBF comparison server")
|
||||
p.add_argument("--host",default="0.0.0.0")
|
||||
p.add_argument("--port",type=int,default=18889)
|
||||
p.add_argument("--mysql-pbf-root",required=True)
|
||||
p.add_argument("--postgres-pbf-root",required=True)
|
||||
p.add_argument("--mysql-release",default="mysql-current")
|
||||
p.add_argument("--postgres-release",default="postgres-current")
|
||||
return p.parse_args()
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
h=hashlib.sha256()
|
||||
with path.open("rb") as f:
|
||||
for b in iter(lambda:f.read(1024*1024),b""):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
def pbf_inventory(root: Path):
|
||||
return {str(p.relative_to(root)):p for p in root.rglob("*.pbf") if p.is_file()}
|
||||
|
||||
def main():
|
||||
args=parse_args()
|
||||
web=Path(__file__).resolve().parent
|
||||
mysql_root=Path(args.mysql_pbf_root).resolve()
|
||||
pg_root=Path(args.postgres_pbf_root).resolve()
|
||||
html=web/"mysql_postgresql_compare.html"
|
||||
style=web/"resources/style.s52.json"
|
||||
assets=web/"assets"
|
||||
resources=web/"resources"
|
||||
for p in [mysql_root,pg_root,html,style,assets/"maplibre-gl.js",assets/"maplibre-gl.css",resources/"sprite.json",resources/"sprite.png"]:
|
||||
if not p.exists():
|
||||
raise SystemExit(f"missing: {p}")
|
||||
mi=pbf_inventory(mysql_root)
|
||||
pi=pbf_inventory(pg_root)
|
||||
common=sorted(set(mi)&set(pi))
|
||||
identical=0
|
||||
different=[]
|
||||
for rel in common:
|
||||
if mi[rel].stat().st_size==pi[rel].stat().st_size and sha256(mi[rel])==sha256(pi[rel]):
|
||||
identical+=1
|
||||
elif len(different)<50:
|
||||
different.append(rel)
|
||||
info={
|
||||
"mysql":{"release":args.mysql_release,"files":len(mi),"root":str(mysql_root)},
|
||||
"postgresql":{"release":args.postgres_release,"files":len(pi),"root":str(pg_root)},
|
||||
"compare":{"common":len(common),"identical":identical,"different":len(common)-identical,"different_examples":different},
|
||||
}
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def send(self,data:bytes,ctype:str,status=200):
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type",ctype)
|
||||
self.send_header("Content-Length",str(len(data)))
|
||||
self.send_header("Cache-Control","no-cache")
|
||||
self.send_header("Access-Control-Allow-Origin","*")
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
def serve_file(self,path:Path,ctype:str):
|
||||
if not path.is_file():
|
||||
self.send_error(404)
|
||||
return
|
||||
self.send(path.read_bytes(),ctype)
|
||||
def do_GET(self):
|
||||
path=urlparse(self.path).path
|
||||
if path in ("/","/index.html","/compare"):
|
||||
return self.serve_file(html,"text/html; charset=utf-8")
|
||||
if path=="/api/info":
|
||||
return self.send(json.dumps(info,ensure_ascii=False,indent=2).encode(),"application/json; charset=utf-8")
|
||||
if path=="/style.json":
|
||||
return self.serve_file(style,"application/json; charset=utf-8")
|
||||
if path in ("/assets/maplibre-gl.js","/assets/maplibre-gl.css"):
|
||||
fn=path.rsplit("/",1)[-1]
|
||||
ct="application/javascript; charset=utf-8" if fn.endswith(".js") else "text/css; charset=utf-8"
|
||||
return self.serve_file(assets/fn,ct)
|
||||
if path in ("/sprite.json","/sprite.png","/sprite@2x.json","/sprite@2x.png"):
|
||||
fn=path[1:]
|
||||
ct="application/json; charset=utf-8" if fn.endswith(".json") else "image/png"
|
||||
return self.serve_file(resources/fn,ct)
|
||||
for prefix,root in (("/mysql/",mysql_root),("/postgres/",pg_root)):
|
||||
if path.startswith(prefix) and path.endswith(".pbf"):
|
||||
rel=path[len(prefix):]
|
||||
parts=rel.split("/")
|
||||
if len(parts)!=3 or not all(x.isdigit() for x in [parts[0],parts[1],parts[2][:-4]]):
|
||||
self.send_error(404); return
|
||||
return self.serve_file(root/rel,"application/vnd.mapbox-vector-tile")
|
||||
self.send_error(404)
|
||||
def log_message(self,fmt,*args):
|
||||
pass
|
||||
|
||||
print(json.dumps(info,ensure_ascii=False,indent=2))
|
||||
print(f"Compare: http://{args.host}:{args.port}/")
|
||||
ThreadingHTTPServer((args.host,args.port),Handler).serve_forever()
|
||||
|
||||
if __name__=="__main__":
|
||||
main()
|
||||
7338
web/resources/sprite.json
Normal file
7338
web/resources/sprite.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
web/resources/sprite.png
Normal file
BIN
web/resources/sprite.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 394 KiB |
7338
web/resources/sprite@2x.json
Normal file
7338
web/resources/sprite@2x.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
web/resources/sprite@2x.png
Normal file
BIN
web/resources/sprite@2x.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 562 KiB |
42381
web/resources/style.s52.json
Normal file
42381
web/resources/style.s52.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user