tei/pbf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import of NavSea pbf project

Browse Source
This commit is contained in:
OpenAI Codex
2026-03-17 19:48:15 +08:00
commit 5f3f7409b3
84 changed files with 43889 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
.gitignore vendored Normal file
View File

@@ -0,0 +1,37 @@
# Python environments
.venv/
venv/
env/
# Python cache
__pycache__/
*.py[cod]
*.pyo
*.pyd
# Editor / OS
.DS_Store
Thumbs.db
.idea/
.vscode/
# Local temp and logs
.tmp/
logs/
*.log
# Test / coverage caches
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
htmlcov/
# Build artifacts
build/
dist/
*.egg-info/
# Local runtime outputs
report/*.tmp
report/*.cache

197
NavSea_Delivery_vs_Source_Audit_Report_Karatsu_10nm.md Normal file
View File

@@ -0,0 +1,197 @@
# NavSea 使用版与原始版审计报告
范围：唐津中心 10 海里 AOI
原始版目录：`/home/wwwroot/newpec/exported_auto/tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles`
使用版目录：`/home/wwwroot/pbf-delivery-karatsu-10nm`
使用版样式：[navsea-delivery.json](/home/wwwroot/newpec/navsea-delivery.json)
## 1. 结论
在唐津中心 10 海里范围内，使用版与原始版相比：
- 瓦片文件集合一致
- feature 总数一致
- `source-layer` 集合一致
- 图层 `extent` 一致
- 没有发生 layer 重排
这意味着：
- 使用版没有删对象
- 使用版没有少瓦片
- 使用版没有改几何精度
- 使用版的主要变化集中在属性增强，而不是空间内容变化
## 2. 审计范围
- AOI 瓦片数：`59`
- zoom 分布：`0, 5, 6, 7, 8, 9, 10, 11, 12`
- 使用版样式名称：`NavSea Delivery Karatsu 10nm`
- 使用版瓦片源：`http://192.168.200.184/pbf-delivery-karatsu-10nm/{z}/{x}/{y}.pbf`
## 3. 瓦片集合审计
- 使用版瓦片数：`59`
- 同名原始瓦片存在数：`59`
- 使用版缺少原始对应文件：`0`
结论：
- 使用版与原始版在 `z/x/y.pbf` 文件名层面一一对应
## 4. Feature 数量审计
- 原始版 feature 总数：`142424`
- 使用版 feature 总数：`142424`
- feature 数量不一致的瓦片数：`0`
结论：
- 唐津 10 海里范围内，使用版没有减少或增加 feature
## 5. Layer 结构审计
- 原始版 distinct `source-layer` 数：`37`
- 使用版 distinct `source-layer` 数：`37`
- layer 重排 feature 数：`0`
- `extent` 不一致层数：`0`
结论：
- 本 AOI 内，使用版仍沿用原始 layer 结构
- `render_layer` 虽然被写入了属性，但这 10 海里范围内没有出现 `source_layer -> render_layer` 的实际迁移
- 几何编码精度和原始版一致
## 6. 属性差异审计
### 6.1 新增字段
使用版在原始属性基础上新增了这些字段：
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `fid_algo_id`
- `fid_key_id`
- `render_layer`
- `semantic_key`
### 6.2 删除字段
审计结果中，没有发现“原始版有、使用版没有”的属性键：
- 删除字段数：`0`
说明：
- 使用版保留了原始字段体系
- 变化不是“删旧字段”，而是“在旧字段上增加新字段”
### 6.3 `fid` 的变化
`fid` 键仍然存在，但值已经从旧版原始值改成了 NavSea 加密后的 16 进制值。
示例：
- 原始版 `/12/3525/1642.pbf`：`fid=10004352`
- 使用版 `/12/3525/1642.pbf`：`fid=3493BD24`
同时使用版会补充：
- `fid_algo_id=feistel32_aes_cyclewalk_v1`
- `fid_key_id=navsea-fid-key-v1`
说明：
- 使用版不再直接暴露旧版原始 `fid`
- 与工程版不同，使用版不会保留 `fid_legacy_raw`、`fid_navsea_int` 这类追溯字段
## 7. 样例对比
### 7.1 `/0/0/0.pbf`
原始版样例：
- layer：`P基本線`
- `fid=5002231`
使用版样例：
- layer：`P基本線`
- `fid=3720E480`
- `fid_algo_id=feistel32_aes_cyclewalk_v1`
- `fid_key_id=navsea-fid-key-v1`
- `canonical_object_type=2-5m`
- `canonical_family=surface`
- `detection_key=surface:2-5m`
- `render_layer=P基本線`
### 7.2 `/12/3525/1642.pbf`
原始版样例：
- layer：`L739`
- `fid=10004352`
使用版样例：
- layer：`L739`
- `fid=3493BD24`
- `fid_algo_id=feistel32_aes_cyclewalk_v1`
- `fid_key_id=navsea-fid-key-v1`
- `canonical_object_type=平水境界`
- `canonical_family=line`
- `detection_key=line:平水境界`
- `render_layer=L739`
## 8. 对渲染的影响
从这次审计结果看，使用版对渲染最重要的结论有两条：
- 几何、feature 数量、layer 结构没有变化，所以“画不出来”的风险不来自空间内容减少
- 真正变化的是属性层，尤其是 `fid` 被替换为加密值，以及增加了语义字段
因此，对旧样式兼容性的影响主要来自：
- 样式是否依赖旧 `fid` 直接做逻辑
- 样式是否开始使用新的 `canonical_*` / `render_layer` / `detection_key`
而不是来自：
- 瓦片缺失
- layer 缺失
- feature 丢失
- extent 改变
## 9. 与工程版的区别
使用版与工程版相比，主要少了这些追溯字段：
- `fid_legacy_raw`
- `fid_navsea_int`
- `source_layer_jp`
- `source_layer_std`
- `normalization_bundle_id`
- `source_layer_rule_id`
- `trace_status`
- `feature_id`
结论：
- 使用版适合渲染、检索、业务使用
- 工程版适合审计、回查、问题定位
## 10. 最终判断
唐津 10 海里范围内，使用版相对于原始版的变化可以概括为：
- 空间内容未变
- 图层结构未变
- 原始字段未删
- `fid` 被替换为 NavSea 加密值
- 叠加了新的语义字段
- 去掉了工程追溯字段
一句话总结：
> 使用版不是“瘦身后的原始版”，而是“保留原始内容、替换对象 ID、叠加语义信息、去掉工程追溯字段”的交付版。

107
NavSea_Geometry_Enablement.md Normal file
View File

@@ -0,0 +1,107 @@
# NavSea Geometry Enablement
## 目的
为 `navsea_detection_objects` 补上真实 geometry，并在数据库中启用真正的空间索引能力。
## 当前现状
当前 `pbf_analysis` 库中没有真实坐标 geometry 列，只有：
- `feature_id`
- `z/x/y`
- `vt_layer`
- `geom_type`
因此当前系统只能做：
- 语义分类
- tile 级查询优化
- 基于 `z/x/y` 的近似空间诊断
不能做：
- 真正的半径查询
- 真正的航线走廊查询
- 真正的船首前方空间查询
- 真正的 `SPATIAL INDEX` 查询计划
## 启用方式
使用脚本：
```bash
.venv/bin/python navsea_geometry_enable.py
```
默认要求数据库中存在 staging 表：
```text
feature_geometry_stage
```
可以通过环境变量改名：
```bash
NAVSEA_GEOMETRY_SOURCE_TABLE=your_table .venv/bin/python navsea_geometry_enable.py
```
## 支持的 staging 结构
脚本支持以下任一输入形式：
### 1. 直接 geometry 列
```sql
feature_id BIGINT
geometry GEOMETRY
```
### 2. WKT
```sql
feature_id BIGINT
geometry_wkt LONGTEXT
```
### 3. WKB
```sql
feature_id BIGINT
geometry_wkb BLOB
```
### 4. 点坐标
```sql
feature_id BIGINT
lon DOUBLE
lat DOUBLE
```
仅适用于 Point 类对象。
## 脚本输出
如果 staging 表满足要求，脚本会创建：
- `navsea_feature_geometry`
- `navsea_detection_objects_spatial`
其中：
- `navsea_feature_geometry` 保存真实 geometry，并建立 `SPATIAL KEY idx_geometry`
- `navsea_detection_objects_spatial` 是 `navsea_detection_objects` 的空间版只读视图
## 推荐的后续查询
当 geometry 可用后，可以在 `navsea_detection_objects_spatial` 上执行：
- 半径查询
- 航线缓冲带查询
- 前方扇形查询
- 空间相交 / 邻近查询
## 当前阻塞
当前唯一阻塞不是脚本，而是缺少 geometry staging 数据源。

347
NavSea_Legacy_Field_Exit_Roadmap.md Normal file
View File

@@ -0,0 +1,347 @@
# NavSea 摆脱旧字段体系路线图
目标：让 NavSea 的使用版样式与交付版 `pbf` 不再依赖旧海图字段体系，最终做到：
- 样式主逻辑不再直接读取 `分類番号`
- 样式主逻辑不再直接读取 `形状分類番号`
- 样式主逻辑不再直接读取 `表示用番号`
- 样式主逻辑不再直接读取 `灯色`、`灯略記`、`明弧/分孤`
- 检索和渲染都优先依赖 `canonical_*` 与 `chart_*`
## 1. 当前状态
当前使用版已经迈出第一步，但还没有真正摆脱旧字段体系。
### 1.1 已完成
- `fid` 已替换为 NavSea 自有 16 进制 ID
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `semantic_key`
- `render_layer`
### 1.2 仍未完成
当前使用版样式仍直接读取这些旧字段：
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `表示位置`
- `名称`
- `名称補助`
- `日本語地名`
- `英文字地名`
- `水深値(m)`
- `高さ/深度(m)`
说明：
- 其中 `名称`、`名称補助`、`日本語地名`、`英文字地名` 不应视为要被完全淘汰的“坏字段”
- 它们属于内容字段，应保留或映射为 `chart_label_*`
- 真正要退出的是“旧渲染控制字段”
## 2. 旧字段退出的对象范围
优先退出的旧字段：
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `明弧/分孤`
- `表示位置`
- `水深値(m)`
- `高さ(m)`
- `高さ/深度(m)`
- `角度`
不建议简单删除，而应先完成“等价替代字段”。
## 3. 对应的新字段体系
### 3.1 对象语义层
继续保留并强化：
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `semantic_key`
这层回答：
- 这个对象是什么
### 3.2 渲染语义层
必须补齐：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_text_style`
- `chart_priority`
- `chart_visibility_min`
- `chart_visibility_max`
- `chart_collision_group`
这层回答：
- 这个对象应该怎么画
### 3.3 专项语义层
必须补齐：
- `chart_label_text`
- `chart_label_subtext`
- `chart_label_anchor`
- `chart_label_dx`
- `chart_label_dy`
- `light_color_code`
- `light_character_code`
- `light_sector_mode`
- `hazard_class`
- `hazard_severity`
- `area_usage_class`
- `depth_value_m`
- `clearance_height_m`
- `least_depth_m`
- `bearing_deg`
这层回答：
- 具体符号、文字和业务检索所需的细节是什么
## 4. 替代关系
### 4.1 灯标体系
旧字段：
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `明弧/分孤`
新字段：
- `chart_symbol_family=navigation_light` 或 `navigation_mark`
- `chart_symbol_code`
- `light_color_code`
- `light_character_code`
- `light_sector_mode`
- `chart_label_text`
- `chart_label_subtext`
退出条件：
- 样式中灯标相关层不再直接 `get` 旧灯标字段
### 4.2 危险物/鱼礁体系
旧字段：
- `分類番号`
- `表示用番号`
新字段：
- `canonical_object_type`
- `hazard_class`
- `hazard_severity`
- `chart_symbol_code`
- `chart_fill_style`
退出条件：
- 危险物和鱼礁图形不再依赖旧编码号判断
### 4.3 等深线/水深体系
旧字段：
- `表示用番号`
- `水深値(m)`
- `高さ/深度(m)`
新字段：
- `chart_line_style`
- `chart_text_style`
- `depth_value_m`
- `least_depth_m`
退出条件：
- 等深线粗细、类别和数字显示不再依赖旧字段
### 4.4 边界/区域体系
旧字段：
- `分類番号`
- `表示用番号`
新字段：
- `canonical_object_type`
- `chart_line_style`
- `chart_fill_style`
- `area_usage_class`
退出条件：
- 锚地、航路、鱼业区、危险界等区域表现不再用旧编码做主判断
### 4.5 地名体系
旧字段：
- `名称`
- `名称補助`
- `日本語地名`
- `英文字地名`
- `表示位置`
新字段：
- `chart_label_text`
- `chart_label_subtext`
- `chart_label_anchor`
- `chart_label_dx`
- `chart_label_dy`
- `chart_text_style`
说明：
- 地名文本本身不是要淘汰的“旧体系”
- 要退出的是“用原始字段名直接驱动样式”的方式
## 5. 实施阶段
## 阶段 A：补齐数据字段
目标：
- builder 输出完整 `chart_*`
- builder 输出完整 `light_*`
- builder 输出完整 `hazard_*`
- builder 输出完整数值归一化字段
验收：
- 使用版 `pbf` 中，核心对象都能拿到可直接渲染的新字段
## 阶段 B：建立纯新字段样式
目标：
- 新样式不再直接读取 `分類番号`
- 新样式不再直接读取 `形状分類番号`
- 新样式不再直接读取 `表示用番号`
例外：
- 文本内容字段暂时允许保留
验收：
- `style json` 中 `["get", "..."]` 不再出现旧渲染控制字段
## 阶段 C：做等价性审计
目标：
- 对比旧样式与新样式的主要对象覆盖
- 对比使用版旧字段驱动与新字段驱动的渲染结果
验收：
- 核心对象组通过审计：
- 灯标
- 危险物
- 鱼礁
- 等深线
- 水深数字
- 锚地/航路/鱼业区
- 地名
## 阶段 D：旧字段降级为兼容字段
目标：
- 使用版样式停止直接依赖旧渲染控制字段
- 旧字段只保留在数据里，供兼容、回查或历史对照
验收：
- 使用版样式完全跑在新字段体系上
- 工程版仍保留旧字段和 trace 以便审计
## 阶段 E：旧字段退场
目标：
- 仅在确认无样式、无检索、无检测依赖后，才从交付版中移除旧渲染控制字段
注意：
- 这不是近期动作
- 必须放在最后
## 6. 退出门禁
一个旧字段只有在同时满足以下条件时，才允许从使用版交付中移除：
- 已存在稳定的新字段替代
- 使用版样式不再直接读取它
- 检索规则不再直接读取它
- 碰撞或检测链路不再直接读取它
- 工程版仍能回查原值
- 审计报告确认关键对象渲染未明显退化
## 7. 下一步最值得做的事
如果目标是“真正摆脱旧字段体系”，下一步最值得直接落地的是：
1. 在 builder 里补齐第一批 `chart_*` 字段
重点对象：
- 灯标
- 危险物
- 鱼礁
- 等深线
- 水深数字
- 锚地/航路/鱼业区
2. 出一份“纯新字段样式”
要求：
- 不再读取 `分類番号`
- 不再读取 `形状分類番号`
- 不再读取 `表示用番号`
3. 对唐津 10 海里做一次新旧样式对照审计
目标：
- 看哪些对象已经可以完全脱离旧字段
- 看哪些对象还需要旧字段兜底
## 8. 最终判断
完全摆脱旧字段体系是可行的，但前提不是“删除旧字段”，而是：
- 先补齐新字段
- 再迁移样式
- 再做等价性审计
- 最后才让旧字段退出主流程
一句话总结：
> 下一步不是“把旧字段删掉”，而是“把旧字段一项一项替换掉，直到样式和业务都不再依赖它们”。

182
NavSea_Object_Taxonomy_Manual_Review.md Normal file
View File

@@ -0,0 +1,182 @@
# NavSea Object Taxonomy Manual Review
## 1 Review Objective
This package is for manual confirmation of the current NavSea object taxonomy derived from the `pbf_analysis` database snapshot in this workspace.
The review focuses on confirming:
- canonical object type naming
- family grouping
- detection grouping
- geometry unification or splitting
- ambiguous object handling
The current pipeline already separates source-preserved render layers, semantic object layers, and detection-oriented object taxonomy. This review package highlights the places where human confirmation is still needed before treating the taxonomy as stable.
## 2 Review Priority Groups
### Priority A - Immediate navigation safety relevance
Hazards, wrecks, rocks, reefs, dangerous obstacles, navigation marks, route controls, anchorage controls, restricted areas, and fishery hazards that can affect collision, grounding, or entanglement decisions.
### Priority B - Navigation context / operational relevance
Boundaries, facilities, subsea cables and pipelines, piers, floating structures, and other contextual objects that matter to operations even when they are not direct collision targets.
### Priority C - Background / rendering objects
Labels, bathymetry support layers, clip layers, land masks, and other source-preserved layers that may be rendering support rather than semantic end objects.
## 3 Manual Review Candidate Table
| review_priority | layer_name | class_name | shape_name | vt_layer | observed_geom_types | total_feature_count | suggested_canonical_object_type | suggested_canonical_family | suggested_detection_class | suggested_review_reason | needs_manual_review |
| --- | --- | --- | --- | --- | --- | ---: | --- | --- | --- | --- | --- |
| A | 航行危険障害物 | 暗岩 | | p航行危険障害物 | Point | 10,366 | 暗岩 | hazard | obstacle | Direct grounding hazard; confirm whether rock / reef subtyping should remain separate from generic obstacle handling. | YES |
| A | 航行危険障害物 | 洗岩 | | p航行危険障害物 | Point | 3,323 | 洗岩 | hazard | obstacle | High-salience grounding hazard; confirm stable canonical naming under hazard family. | YES |
| A | 航行危険障害物 | 干出岩 | | p航行危険障害物 | Point | 1,178 | 干出岩 | hazard | obstacle | Intertidal hazard object with clear navigation relevance; confirm whether it stays separate from other rock hazards. | YES |
| A | 投錨注意障害物 | 険悪物 | | p投錨注意障害物 / P投錨注意障害物 | Point, Polygon, MultiPolygon | 26,836 | 険悪物 | hazard | obstacle | Same object appears as point and area; confirm whether symbolic and areal forms should be unified or split. | YES |
| A | 投錨注意障害物 | 魚礁 | | p投錨注意障害物 / P投錨注意障害物 | Point, Polygon, MultiPolygon | 16,608 | 魚礁 | hazard | obstacle | Artificial reef can imply fishery semantics or navigation hazard; family assignment needs human confirmation. | YES |
| A | 航路標識点 | | 灯 (Lt) | p航路標識群 | Point | 14,936 | 灯 (Lt) | navigation_aid | navigation_mark | Core navigation mark subtype; confirm whether light subtypes should be normalized under a broader canonical lighthouse / light object. | YES |
| A | 航路標識点 | | 防波堤灯台 | p航路標識群 | Point | 4,679 | 防波堤灯台 | navigation_aid | navigation_mark | Shape-driven subtype may be a naming variant of a broader light / lighthouse family. | YES |
| A | 錨泊地等 | | | p錨泊地等 / P錨泊地等 | Point, Polygon | 234 | 錨泊 (指定)地 | anchorage | anchorage | Same anchorage concept appears as symbol and area; confirm whether annotation and area extent should remain one object type. | YES |
| A | 施設・境界線等 | 港則法による境界 | | L741 | LineString, MultiLineString | 1,696 | 港則法による境界 | boundary | boundary_control | Regulatory meaning affects route compliance and alerting; boundary family and detection role should be confirmed manually. | YES |
| A | 海岸線等深線等 | 浅所危険界 | | P基本線 | Polygon, MultiPolygon | 31,537 | 浅所危険界 | hazard | obstacle | Shoal hazard boundary may be treated as hazard extent, boundary object, or bathymetry support; requires explicit confirmation. | YES |
| A | 漁具定置箇所 | 区画漁業 | | P漁具定置箇所 | Polygon, MultiPolygon | 19,025 | 区画漁業 | fishery | fishery_area | Fishery extent has entanglement and operational relevance; confirm fishery-area grouping and naming. | YES |
| A | 漁具定置箇所 | 定置漁業 | | P漁具定置箇所 | Polygon, MultiPolygon | 3,570 | 定置漁業 | fishery | fishery_area | Fixed fishery installations should be checked against other fishery-area types for canonical consolidation. | YES |
| B | 海岸線等深線等 | 浮施設・桟橋 | | L基本線 / P基本線 | LineString, MultiLineString, Polygon, MultiPolygon | 30,090 | 浮施設・桟橋 | infrastructure | structure | Strong multi-geometry ambiguity across line and area representations; confirm unify vs split between centerline and footprint semantics. | YES |
| B | 海岸線等深線等 | 未測海域 | | L危険界 / P基本線 | LineString, Polygon, MultiPolygon | 4,554 | 未測海域 | water_area | water_area | Appears both as boundary and area extent; confirm whether unsurveyed-region boundary should be separated from area fill. | YES |
| B | 施設・境界線等 | 海底線 (電信電話) | | L海底線 | LineString, MultiLineString | 12,817 | 海底線 (電信電話) | utility | utility_line | Utility line can also imply entanglement / anchor risk; confirm utility vs hazard emphasis. | YES |
| B | 施設・境界線等 | 海底輸送管 (水) | | L海底線 | LineString, MultiLineString | 655 | 海底輸送管 (水) | utility | utility_line | Pipeline semantics are operationally important and come from a container layer; confirm naming and grouping. | YES |
| B | 施設・境界線等 | 養殖場 | | P施設・境界線等 | Polygon | 716 | 養殖場 | fishery | fishery_area | Fishery object currently originates from a facilities / boundary container layer; confirm semantic extraction is correct. | YES |
| B | 施設・境界線等 | 漁網 | | P施設・境界線等 | Polygon | 178 | 漁網 | fishery | fishery_area | Net-related entanglement risk should be confirmed as fishery semantics rather than generic facility content. | YES |
| B | 航路 | 航路 (法律による航路) | | L航路 / P航路 | LineString, Polygon, MultiPolygon | 404 | 航路 (法律による航路) | route | route_reference | Legal route appears as line and area; confirm whether centerline guidance and regulated area should be split. | YES |
| C | | | | P基本線ククリ | LineString, MultiLineString | 1,080,780 | P基本線ククリ | overlay_support | render_support | High-volume clip / outline layer likely acts as rendering support rather than semantic object content. | YES |
| C | | | | P危険界ククリ | LineString, MultiLineString | 51,962 | P危険界ククリ | overlay_support | render_support | Source-preserved hazard-outline layer; confirm it should stay container / render-only. | YES |
| C | | | | P投錨注意障害物ククリ | LineString, MultiLineString | 8,743 | P投錨注意障害物ククリ | overlay_support | render_support | Outline layer around anchoring hazards may be symbolic support rather than independent semantic object. | YES |
| C | | | | P航路ククリ | LineString, MultiLineString | 864 | P航路ククリ | overlay_support | render_support | Route clip / outline layer should likely remain render support; confirm no semantic promotion is needed. | YES |
| C | | | | L海底地形 | LineString, MultiLineString | 2,075,172 | L海底地形 | overlay_support | render_support | Very large source-preserved layer may be bathymetry render support rather than end-user semantic taxonomy. | YES |
| C | | | | p地名 | Point | 36,247 | p地名 | overlay_support | render_support | Label layer behaves as background annotation support, not operational semantic object content. | NO |
| C | | | | p地名陸 | Point | 35,388 | p地名陸 | overlay_support | render_support | Label layer behaves as background annotation support, not operational semantic object content. | NO |
| C | | | | P穴 | Polygon, MultiPolygon | 367,217 | P穴 | overlay_support | render_support | Land-mask support layer appears stable as rendering support and does not currently require semantic taxonomy splitting. | NO |
## 4 Ambiguity Categories
### Multi-Geometry Ambiguity
Current high-value multi-geometry objects include:
- 浮施設・桟橋: 30,090 features across `LineString`, `MultiLineString`, `Polygon`, `MultiPolygon`
- 険悪物: 26,836 features across `Point`, `Polygon`, `MultiPolygon`
- 魚礁: 16,608 features across `Point`, `Polygon`, `MultiPolygon`
- 未測海域: 4,554 features across `LineString`, `Polygon`, `MultiPolygon`
- 航路 (法律による航路): 404 features across `LineString`, `Polygon`, `MultiPolygon`
- 錨泊 (指定)地: 234 features across `Point`, `Polygon`
Primary review question: should these remain one canonical object with mixed geometry support, or should point-symbol, line-boundary, and area-extent variants be split into separate semantic objects?
### Navigation Meaning Ambiguity
The following objects directly affect collision, grounding, anchor, or entanglement reasoning and should be reviewed first:
- 暗岩
- 洗岩
- 干出岩
- 険悪物
- 魚礁
- 浅所危険界
- 漁網
- 養殖場
- 海底線 / 海底輸送管
Primary review question: does each object belong in `hazard`, `fishery`, `utility`, or another family when detection behavior is considered?
### Source Layer Container Ambiguity
The current layer rules identify several likely container layers:
- `P基本線`: `container_layer`, 34 semantic object types, 701,625 features
- `P施設・境界線等`: `container_layer`, 16 semantic object types, 6,425 features
- `L海底線`: `container_layer`, 5 semantic object types, 14,391 features
- `L基本線`: `container_layer`, 5 semantic object types, 53,615 features
Primary review question: should these layers remain source-preserved render containers while downstream products use extracted semantic objects instead?
### Regulatory / Boundary Ambiguity
Objects with strong legal or control semantics include:
- 港則法による境界
- 航路 (法律による航路)
- 錨泊 (指定)地
- 浅所危険界
- 未測海域
Primary review question: should regulatory boundary geometry and filled-area geometry be modeled as the same object type or as separate rule / extent objects?
## 5 First-Pass Recommended Review Scope
Focus first on classes and layers most similar to the task brief:
- 航行危険障害物
- 投錨注意障害物
- 航路標識点
- 航路境界等 / 航路 (法律による航路)
- 錨泊地等
- 施設・境界線等
- 海岸線等深線等 / 海岸線等深線等小縮尺版
- 漁具定置箇所
Recommended first-pass order:
1. Confirm hazard objects and shoal / reef / wreck naming
2. Confirm navigation-mark subtype normalization
3. Confirm anchorage, route, and regulatory boundary grouping
4. Confirm fishery and utility objects extracted from container layers
5. Confirm which clip / support layers remain render-only
## 6 Proposed Review Decisions
| Candidate Object | Suggested canonical_object_type | Suggested canonical_family | Suggested detection_class | Reason | Manual confirmation required |
| --- | --- | --- | --- | --- | --- |
| 険悪物 | 険悪物 | hazard | obstacle | Large mixed point / area hazard set; important to decide whether symbol and extent remain unified. | YES |
| 魚礁 | 魚礁 | hazard | obstacle | Could be treated as hazard, fishery, or hybrid operational object depending on downstream use. | YES |
| 暗岩 / 洗岩 / 干出岩 | Keep as separate named hazard objects | hazard | obstacle | Distinct rock-hazard names are meaningful for chart interpretation and should not be collapsed without explicit approval. | YES |
| 浅所危険界 | 浅所危険界 | hazard | obstacle | Current suggestion treats it as hazard extent, but it may need a boundary-style subtype. | YES |
| 灯 (Lt) and light subtypes | Normalize under light / navigation mark family while preserving subtype label | navigation_aid | navigation_mark | Many shape-driven names likely belong to a common aid-to-navigation family with subtype metadata. | YES |
| 錨泊 (指定)地 | 錨泊 (指定)地 | anchorage | anchorage | Point annotation and polygon extent probably represent the same concept, but a split between label and area remains plausible. | YES |
| 港則法による境界 | 港則法による境界 | boundary | boundary_control | Legal boundary semantics justify keeping a distinct boundary object if user confirms. | YES |
| 航路 (法律による航路) | 航路 (法律による航路) | route | route_reference | Line and polygon forms may need separate centerline and regulated-area objects. | YES |
| 浮施設・桟橋 | 浮施設・桟橋 | infrastructure | structure | Strong line / area dual representation suggests a possible split into linear and areal facility objects. | YES |
| 未測海域 | 未測海域 | water_area | water_area | Boundary plus filled-area modeling should be reviewed before finalizing canonical geometry rules. | YES |
| 海底線 / 海底輸送管 | Keep separate utility object types by infrastructure kind | utility | utility_line | Operationally relevant and extracted from a container layer; keep distinct unless user prefers a broader subsea-utility object. | YES |
| 区画漁業 / 定置漁業 / 養殖場 / 漁網 | Keep as fishery-area objects with subtype distinction | fishery | fishery_area | These support entanglement-aware review and likely should stay distinct within one fishery family. | YES |
| P基本線ククリ and similar `ククリ` layers | Keep as source-preserved render support | overlay_support | render_support | Current evidence suggests these are clip / outline / render-control layers rather than semantic end objects. | YES |
| p地名 / p地名陸 / P穴 | Keep as render-support / background objects | overlay_support | render_support | These appear stable and low-risk as non-semantic support layers. | NO |
## 7 Layer Review Appendix
| source_layer | semantic_granularity | current_rule_guess | review_reason | needs_manual_review |
| --- | --- | --- | --- | --- |
| P基本線 | container_layer | Container surface layer feeding bathymetry, hazard, water-area, and facility objects | Very high semantic fan-out: 34 object types across 701,625 features. | YES |
| P施設・境界線等 | container_layer | Container surface layer for facilities, boundaries, fishery, and place objects | Mixed semantic content suggests it should remain a container layer rather than a single semantic layer. | YES |
| L海底線 | container_layer | Container linear layer for cable and pipeline objects | Contains multiple subsea utility types with shared rendering source. | YES |
| L基本線 | container_layer | Container linear layer for piers, berths, removed objects, and linear marine structures | Same source layer mixes several operationally different linear object types. | YES |
| p航路標識群 | semantic_layer | Stable semantic point layer with shape-driven navigation-mark subtypes | Semantic layer is good, but subtype normalization still needs manual confirmation. | YES |
| p投錨注意障害物 | semantic_layer | Stable semantic hazard layer with mixed object meanings | Contains hazards, reefs, current phenomena, and seabed-related points that may need family confirmation. | YES |
| p航行危険障害物 | semantic_layer | Stable semantic hazard layer | High-priority safety layer; naming and family assignments should be checked even if the layer itself stays semantic. | YES |
| P錨泊地等 | semantic_layer | Semantic area layer for anchorage and restriction objects | Needs confirmation that point and area anchorage variants are modeled consistently. | YES |
| L等深線 | semantic_layer | Stable contour layer | Geometry behavior looks expected for contour lines; low ambiguity. | NO |
| p地名 | style_or_source_layer | Source-preserved label layer | Background label support; not currently a semantic review focus. | NO |
| p地名陸 | style_or_source_layer | Source-preserved land label layer | Background label support; not currently a semantic review focus. | NO |
| P穴 | style_or_source_layer | Source-preserved area support layer | Stable render-support role; no strong evidence for semantic promotion. | NO |
## 8 Review Outcome Guidance
Please confirm the following decisions after manual review:
1. Confirm canonical object naming for hazards, aids to navigation, route controls, anchorage objects, fishery objects, and utility objects.
2. Confirm family grouping for ambiguous objects such as `魚礁`, `浅所危険界`, `未測海域`, and subsea infrastructure.
3. Confirm detection grouping, especially where the current suggestion uses `obstacle`, `navigation_mark`, `anchorage`, `boundary_control`, `utility_line`, or `fishery_area`.
4. Confirm geometry unification rules for mixed point / line / area objects such as `浮施設・桟橋`, `険悪物`, `魚礁`, `未測海域`, `航路 (法律による航路)`, and `錨泊 (指定)地`.
5. Confirm which source layers remain container or render-support layers, especially `P基本線`, `P施設・境界線等`, `L海底線`, `L基本線`, and the `ククリ` layers.

10
NavSea_Original_vs_Engineering_Render_Audit_Karatsu_10nm.json Normal file
View File

@@ -0,0 +1,10 @@
{
"original_feature_instances": 136461,
"engineering_feature_instances": 136461,
"result_count": 136461,
"status_counts": {
"exact_match": 136461
},
"source_layer_issue_counts": [],
"mismatch_examples": []
}

29
NavSea_Original_vs_Engineering_Render_Audit_Karatsu_10nm.md Normal file
View File

@@ -0,0 +1,29 @@
# NavSea 原始版 vs 工程版渲染审计报告
## 范围
- 原始样式: `/mnt/sda1/www/newpec/style.patched.local.json`
- 工程样式: `/mnt/sda1/www/newpec/navsea-engineering.json`
- 原始瓦片根目录: `/home/wwwroot/newpec/exported_auto/tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles`
- 工程瓦片根目录: `/home/wwwroot/pbf-engineering-karatsu-10nm`
- 原始 feature 实例数: `136461`
- 工程 feature 实例数: `136461`
- 审计结果数: `136461`
## 匹配口径
- 主键优先使用 legacy `fid`。
- 没有 `fid` 的对象，回退到 `geometry + 稳定旧属性`。
- 审计粒度保留 tile 实例，因为渲染具有 zoom 敏感性。
## 结果统计
- `exact_match`: `136461`
## 主要问题层
- 没有发现差异。
## 差异样例
- 没有差异样例。

10
NavSea_Original_vs_Engineering_Render_Audit_Kyushu.json Normal file
View File

@@ -0,0 +1,10 @@
{
"original_feature_instances": 2301189,
"engineering_feature_instances": 2301189,
"result_count": 2301189,
"status_counts": {
"exact_match": 2301189
},
"source_layer_issue_counts": [],
"mismatch_examples": []
}

29
NavSea_Original_vs_Engineering_Render_Audit_Kyushu.md Normal file
View File

@@ -0,0 +1,29 @@
# NavSea 原始版 vs 工程版渲染审计报告
## 范围
- 原始样式: `/mnt/sda1/www/newpec/style.patched.local.json`
- 工程样式: `/mnt/sda1/www/newpec/navsea-engineering-kyushu.json`
- 原始瓦片根目录: `/home/wwwroot/newpec/exported_auto/tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles`
- 工程瓦片根目录: `/home/wwwroot/pbf-engineering-kyushu`
- 原始 feature 实例数: `2301189`
- 工程 feature 实例数: `2301189`
- 审计结果数: `2301189`
## 匹配口径
- 主键优先使用 legacy `fid`。
- 没有 `fid` 的对象，回退到 `geometry + 稳定旧属性`。
- 审计粒度保留 tile 实例，因为渲染具有 zoom 敏感性。
## 结果统计
- `exact_match`: `2301189`
## 主要问题层
- 没有发现差异。
## 差异样例
- 没有差异样例。

405
NavSea_Render_Semantic_Redesign_Report.md Normal file
View File

@@ -0,0 +1,405 @@
# NavSea 渲染语义重构报告
## 1 目标
本报告提出一套更清晰的 NavSea PBF 交付渲染描述体系。
这次重构的目标，不是立刻移除旧的海图渲染体系，而是把下面三件事情拆开：
- 对象本身的语义
- 海图渲染意图
- 最终样式实现
这样 NavSea 才能同时支持：
- 兼容旧海图风格的渲染
- 更清晰、可维护的样式逻辑
- 碰撞检测、检索与语义分析
- 逐步摆脱对原始旧海图编码的强依赖
## 2 当前问题
当前渲染栈把多种职责混在了一起。
### 2.1 当前渲染依赖的输入
旧版渲染主要依赖这些字段：
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `明弧/分孤`
- `表示位置`
- `名称`
- 展开的 `at` 属性
这些字段当然有用，但它们的问题也很明显：它们直接承载的是“海图怎么画”的细节编码，而不是清晰、稳定的渲染语义，因此维护成本高、理解成本高、迁移成本也高。
### 2.2 当前已经叠加的新语义
新 `pbf` 已经增加了这些语义字段：
- `canonical_object_type`
- `canonical_family`
- `semantic_key`
- `detection_key`
- `render_layer`
这些字段已经能很好地服务于检测和分类，但还不足以完整替代海图渲染所需的细粒度表达。
### 2.3 当前的核心缺口
现在的数据层同时存在两套东西：
- 面向海图绘制细节的旧编码
- 面向系统理解的语义分类
但缺少一层专门回答“应该怎么画”的渲染语义层。也就是目前还没有一套稳定字段，明确回答这些问题：
- 该使用哪一类符号族
- 该按点、线、面还是文字渲染
- 该显示什么文本
- 渲染优先级是什么
- 应该选用哪一种线型、面型或符号变体
## 3 建议的三层模型
NavSea 的渲染体系建议拆成三层。
### 3.1 A 层：对象语义层
这一层回答“这是什么对象”。
代表字段：
- `canonical_object_type`
- `canonical_family`
- `detection_key`
示例：
- `暗岩`
- `魚礁`
- `港湾灯台`
- `港則法による境界`
### 3.2 B 层：渲染语义层
这一层回答“这个对象在海图上应该怎么画”。
这是本报告建议新增的核心层。
代表字段：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_label_text`
- `chart_label_subtext`
- `chart_label_anchor`
- `chart_priority`
### 3.3 C 层：样式层
MapLibre 样式层尽量只消费“渲染语义层”以及少量保留的原始字段。
这样可以让样式规则明显缩短，也更容易维护和演进。
## 4 建议新增的渲染字段
建议在新的交付 `pbf` 中补充如下字段。
### 4.1 核心渲染字段
| 字段 | 类型 | 用途 |
| --- | --- | --- |
| `chart_render_type` | string | 主渲染类型：`symbol`、`line`、`fill`、`label`、`none` |
| `chart_symbol_family` | string | 符号族，例如 `navigation_light`、`buoy`、`beacon`、`hazard`、`facility`、`depth_mark` |
| `chart_symbol_code` | string | 稳定的符号编码，例如 `lighthouse`、`rock_awash`、`anchorage_mark` |
| `chart_line_style` | string | 稳定的线型预设，例如 `boundary_dashed`、`contour_minor`、`subsea_cable` |
| `chart_fill_style` | string | 稳定的填充预设，例如 `depth_zone_0_5`、`fishery_area`、`hazard_area` |
| `chart_priority` | integer | 绘制优先级，用于排序和碰撞决策 |
| `chart_visibility_min` | integer | 最小显示缩放级别 |
| `chart_visibility_max` | integer | 最大显示缩放级别 |
### 4.2 标注字段
| 字段 | 类型 | 用途 |
| --- | --- | --- |
| `chart_label_text` | string | 主标注文本 |
| `chart_label_subtext` | string | 次级标注，例如辅助说明或灯略记 |
| `chart_label_anchor` | string | 标准化锚点，例如 `top`、`bottom`、`left`、`right`、`center` |
| `chart_label_dx` | number | 文本 X 方向偏移 |
| `chart_label_dy` | number | 文本 Y 方向偏移 |
| `chart_text_style` | string | 文字样式预设，例如 `place_name`、`light_name`、`depth_text` |
### 4.3 灯标专项字段
| 字段 | 类型 | 用途 |
| --- | --- | --- |
| `light_color_code` | string | 归一化后的灯色，例如 `white`、`red`、`green`、`yellow`、`mixed` |
| `light_character_code` | string | 归一化后的灯质，例如 `Fl`、`Oc`、`Iso`、`F`、`V-AIS` |
| `light_arc_code` | string | 可选的灯弧编码，用于需要时表达分区差异 |
| `light_sector_mode` | string | `sector`、`omni`、`none` |
### 4.4 危险物 / 区域专项字段
| 字段 | 类型 | 用途 |
| --- | --- | --- |
| `hazard_class` | string | `rock`、`wreck`、`obstruction`、`reef`、`shoal`、`current` 等 |
| `hazard_severity` | string | `critical`、`major`、`minor`、`context` |
| `area_usage_class` | string | `anchorage`、`route`、`fishery`、`restricted`、`facility`、`land`、`water` |
## 5 建议的映射策略
新的渲染语义层，应该同时从以下两类输入推导得到：
- 新的语义分类字段
- 为保留海图细节而保留的部分旧字段
### 5.1 映射原则
对象身份和大类语义，优先使用新的 taxonomy 字段。
只有在确实需要保留海图细节表现时，才继续引用旧字段。
### 5.2 映射示例
#### 灯标 / 航标
输入字段：
- `canonical_object_type`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `名称`
输出字段：
- `chart_render_type = symbol`
- `chart_symbol_family = navigation_light`
- `chart_symbol_code = lighthouse | light_beacon | buoy | beacon | vais`
- `light_color_code = ...`
- `light_character_code = ...`
- `chart_label_text = 名称`
- `chart_label_subtext = 灯略記`
#### 危险点状对象
输入字段：
- `canonical_object_type`
- `分類番号`
输出字段：
- `chart_render_type = symbol`
- `chart_symbol_family = hazard`
- `chart_symbol_code = rock_awash | isolated_danger | wreck | reef | obstruction`
- `hazard_class = rock | reef | wreck | obstruction`
- `hazard_severity = critical | major | minor`
#### 危险面 / 鱼礁区 / 礁盘区
输入字段：
- `canonical_object_type`
- `分類番号`
- geometry type
输出字段：
- `chart_render_type = fill`
- `chart_fill_style = hazard_area | reef_area | obstruction_area`
- `hazard_class = reef | obstruction | hazard_zone`
#### 水深分带
输入字段：
- `canonical_object_type`
输出字段：
- `chart_render_type = fill`
- `chart_fill_style = depth_zone_0_2 | depth_zone_0_5 | depth_zone_5_10 ...`
- `area_usage_class = water`
#### 边界类对象
输入字段：
- `canonical_object_type`
输出字段：
- `chart_render_type = line`
- `chart_line_style = regulatory_boundary | route_boundary | hazard_boundary`
## 6 建议继续保留的原始字段
即使完成重构，也不建议立刻把所有旧字段从交付 `pbf` 里删除。
### 6.1 为渲染保留
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `明弧/分孤`
- `表示位置`
- `名称`
- `名称補助`
- `日本語地名`
- `英文字地名`
### 6.2 为关联 / 检索保留
- `fid`
- `canonical_object_type`
- `canonical_family`
- `detection_key`
### 6.3 后续可评估移除
这些字段建议在新的渲染语义层稳定之后，再评估是否移除：
- `semantic_key`
- 大块原始 `at` 负载，前提是所需信息都已经物化为显式字段
- 不再被样式、检索或检测引用的旧字段
## 7 样式架构建议
### 7.1 当前样式模式
当前样式主要是直接对旧编码做分支判断。
例如：
- 如果 `分類番号 = 403`，就使用某一个图标
- 如果 `表示用番号 = 31135504`，就切换到另一个图标
这种方式精确，但维护难度很高。
### 7.2 建议中的新样式模式
未来样式应优先基于“渲染语义字段”分支。
例如：
- 如果 `chart_symbol_family = navigation_light`，就进入灯标符号体系
- 如果 `chart_fill_style = depth_zone_0_5`，就使用浅水区填充
- 如果 `chart_line_style = regulatory_boundary`，就使用监管边界线型
只有少量特殊情形，才继续回退到旧字段。
### 7.3 预期结果
这样做会带来以下收益：
- 样式文件更短
- 规则归属更清晰
- 后续迁移更容易
- 多产品之间更容易保持一致
## 8 迁移计划
### 阶段 1：混合交付
保留：
- 旧渲染关键字段
- 新的语义 taxonomy 字段
新增：
- 新的渲染语义字段
使用方式：
- 旧样式继续承担兼容渲染
- 混合样式用于逐步迁移
### 阶段 2：迁移到语义样式
构建一套新的样式，优先读取：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_fill_style`
- `chart_line_style`
- `chart_label_text`
旧字段只在少量细节场景下兜底。
### 阶段 3：稳定渲染字段体系
当渲染语义层经过验证后：
- 逐步减少样式对原始旧编码的直接依赖
- 只保留少量用于回退和兼容的旧字段
## 9 示例输出 Feature 模型
灯台示例：
```json
{
"fid": 12345678,
"canonical_object_type": "港湾灯台",
"canonical_family": "navigation_aid",
"detection_key": "symbol:港湾灯台",
"chart_render_type": "symbol",
"chart_symbol_family": "navigation_light",
"chart_symbol_code": "lighthouse",
"chart_priority": 900,
"chart_visibility_min": 7,
"chart_label_text": "鷹島灯台",
"chart_label_subtext": "Fl W 5s",
"light_color_code": "white",
"light_character_code": "Fl"
}
```
鱼礁危险区示例：
```json
{
"fid": 23456789,
"canonical_object_type": "魚礁",
"canonical_family": "hazard",
"detection_key": "mixed:魚礁",
"chart_render_type": "fill",
"chart_symbol_family": "hazard",
"chart_fill_style": "reef_area",
"hazard_class": "reef",
"hazard_severity": "major",
"chart_priority": 850
}
```
## 10 建议的下一步
建议按下面的顺序推进实现：
1. 在出瓦片流程中定义新的渲染语义字段
2. 将这些字段物化到交付 `pbf`
3. 在过渡期继续保留旧的渲染关键字段
4. 构建一套优先读取新渲染语义字段的混合样式
5. 优先验证灯标体系、危险物体系、鱼业/鱼礁对象和边界对象，再推进更大范围清理
## 11 结论
渲染描述体系完全可以重构，而且值得重构。
正确的方向，不是立刻抛弃旧海图字段，而是在“原始海图属性”和“最终样式”之间，增加一层专门的“渲染语义层”。
这层语义应当提供稳定、明确、可维护的海图渲染含义，同时在必要时保留与旧海图表达方式的兼容能力。

677
NavSea_Semantic_Pipeline.md Normal file
View File

@@ -0,0 +1,677 @@
# NavSea 语义重建流水线说明
## 1. 流水线概览
NavSea 当前的数据库流水线可以分成两段：
1. 上游导入段
将原始 PBF 拆入数据库，形成 `features` 与 `properties` 两张核心表。
2. 语义重建与校验段
在不修改现有渲染逻辑、不修改 `style.json`、不修改原始 PBF 结构的前提下，建立语义叠加层与校验层。
当前真实的流水线可概括为：
```text
原始 PBF
→ PBF 导入（外部前置步骤，结果进入 features / properties）
→ 属性展开（at_attributes）
→ 语义投影（feature_semantic）
→ 对象类型候选生成（object_type_candidates）
→ 审计与统计（object_catalog / object_type_stats / style_layers / tile_density）
→ 规范对象规则（canonical_object_rules / canonical_layer_rules）
→ 语义叠加层构建（pbf_relayer_candidates）
→ 检测目录与渲染兼容层（pbf_detection_catalog / pbf_render_compatibility）
→ 语义校验（classification / geometry / render / detection / spatial）
→ 最终 NavSea 语义数据集（数据库中的已校验语义叠加结果）
```
需要明确一点：当前仓库中的脚本并不直接生成“新的 NavSea 矢量瓦片文件”。当前系统产出的是一套可重建、可审计、可校验的数据库语义数据集；下游若要生成新的 NavSea 矢量瓦片，应以这些表为输入继续执行后续生产步骤。
## 2. 分步流水线
### Step 00 - 原始 PBF 导入
目的
将原始 PBF 解析并导入数据库。
输入
原始 PBF 数据
输出
`features`
`properties`
说明
这一步是当前语义流水线的前置条件。当前仓库中没有提供这一步的导入脚本，但后续所有步骤都假定这两张表已存在。
### Step 01 - 提取 `at` 属性
目的
把 `properties` 表中 `k='at'` 的 JSON 属性展开成结构化键值表。
输入表
`properties`
输出表
`at_attributes`
脚本
`navsea_audit.py`
### Step 02 - 构建语义投影表
目的
把原始要素与已展开的属性汇总到单要素视角，形成后续语义分析使用的基础表。
输入表
`features`
`at_attributes`
输出表
`feature_semantic`
字段要点
`feature_id`、`z/x/y`、`vt_layer`、`geom_type`、`class_name`、`shape_name`、`layer_name`
脚本
`navsea_audit.py`
### Step 03 - 构建对象目录
目的
从 `feature_semantic` 聚合出对象目录，用于观察每类对象在不同来源层和几何类型上的分布。
输入表
`feature_semantic`
输出表
`object_catalog`
脚本
`navsea_audit.py`
### Step 04 - 生成对象类型候选
目的
为每个要素生成对象类型候选，优先级为：
```text
class_name > shape_name > vt_layer
```
输入表
`feature_semantic`
输出表
`object_type_candidates`
脚本
`navsea_audit.py`
### Step 05 - 构建对象类型统计
目的
统计每个对象类型与几何类型的分布，为规则生成和几何白名单提供依据。
输入表
`object_type_candidates`
输出表
`object_type_stats`
脚本
`navsea_audit.py`
### Step 06 - 构建样式层分析
目的
从 `style.json` 中抽取现有渲染层信息，并与对象候选做交叉比对，确认渲染轨仍可复用。
输入
`src/pbf/style.json`
`object_type_candidates`
输出表
`style_layers`
`style_mapping`
脚本
`navsea_audit.py`
### Step 07 - 构建密度与基础异常分析
目的
计算瓦片密度与审计阶段的几何异常，为后续语义校验提供分析输入。
输入表
`features`
`feature_semantic`
输出表
`tile_density`
`tile_layer_density`
`tile_density_top100`
`anomaly_navigation_geom`
`anomaly_reef_geom`
`geometry_consistency_audit`
说明
`geometry_consistency_audit` 是审计阶段保留下来的旧版几何分布快照，不是当前正式校验表。
脚本
`navsea_audit.py`
### Step 08 - 构建来源层规则
目的
基于现有 `source_layer`、样式绑定关系和语义密度，判断每个来源层属于：
```text
semantic_layer
container_layer
style_or_source_layer
```
输入表
`feature_semantic`
`style_layers`
输出表
`canonical_layer_rules`
脚本
`navsea_relayer.py`
### Step 09 - 构建规范对象规则
目的
为每个 `canonical_object_type` 固化规范对象规则，包括：
- `canonical_family`
- `classification_basis`
- `source_layer_scope`
- `allowed_geom_types`
- `preferred_geom_type`
输入表
`object_catalog`
`object_type_stats`
输出表
`canonical_object_rules`
补充输出表
`object_geometry_allowlist`
脚本
`navsea_relayer.py`
### Step 10 - 构建语义叠加层
目的
在保留现有渲染轨的前提下，为每个要素附加：
- 规范对象类型
- 规范对象族
- 检测键
- 语义键
- 渲染兼容层
输入表
`feature_semantic`
`object_type_candidates`
`canonical_layer_rules`
`canonical_object_rules`
输出表
`pbf_relayer_candidates`
脚本
`navsea_relayer.py`
### Step 11 - 构建语义统计与检测目录
目的
从语义叠加层生成两个下游核心视图：
1. 渲染兼容统计
用于确认 `render_layer` 与原有 `source_layer` 的数量关系
2. 检测目录
用于安全检测、对象检测和规则匹配
输入表
`pbf_relayer_candidates`
输出表
`pbf_source_object_stats`
`pbf_render_compatibility`
`pbf_detection_catalog`
脚本
`navsea_relayer.py`
### Step 12 - 运行分类校验
目的
确认每个 `canonical_object_type` 的映射是否稳定，是否存在未知对象、歧义映射或规则冲突。
输入表
`pbf_source_object_stats`
`canonical_object_rules`
输出表
`classification_validation`
脚本
`navsea_semantic_validation.py`
### Step 13 - 运行几何一致性校验
目的
确认每个规范对象的几何类型是否符合白名单与规则定义。
输入表
`pbf_detection_catalog`
`object_geometry_allowlist`
输出表
`geometry_consistency`
脚本
`navsea_semantic_validation.py`
### Step 14 - 运行渲染等价校验
目的
确认语义叠加层没有破坏现有 style 渲染轨。
输入表
`style_layers`
`features`
`pbf_render_compatibility`
输出表
`style_render_equivalence`
脚本
`navsea_semantic_validation.py`
### Step 15 - 运行检测目录完整性校验
目的
确认 `detection_key` 对象目录稳定且几何一致。
输入表
`pbf_detection_catalog`
`geometry_consistency`
输出表
`detection_catalog_integrity`
脚本
`navsea_semantic_validation.py`
### Step 16 - 运行空间密度诊断
目的
基于 `features` 直接计算的瓦片密度和分 zoom 阈值，识别空间热点瓦片。
输入表
`tile_density`
`tile_layer_density`
`features`
`pbf_relayer_candidates`
输出表
`spatial_anomalies`
规则要点
- `tile_density` 只允许来自 `features`
- 每个 zoom 单独计算阈值
- 跳过 `z=0`
- 支持 `DENSITY_STATISTICS_ERROR`
- 每个异常 tile 单独落一行
脚本
`navsea_semantic_validation.py`
### Step 17 - 生成总体验证结论
目的
把分类、几何、渲染、检测目录和空间诊断汇总成一条总状态。
输入表
`classification_validation`
`geometry_consistency`
`style_render_equivalence`
`detection_catalog_integrity`
`spatial_anomalies`
输出表
`semantic_validation_summary`
脚本
`navsea_semantic_validation.py`
## 3. 表依赖图
### 3.1 主干数据血缘
```text
原始 PBF
↓
features + properties
↓
at_attributes
↓
feature_semantic
├─→ object_catalog
├─→ object_type_candidates
│ ↓
│ object_type_stats
│
└─→ style_layers ← style.json
object_catalog + object_type_stats
↓
canonical_object_rules
↓
object_geometry_allowlist
feature_semantic + style_layers
↓
canonical_layer_rules
feature_semantic + object_type_candidates + canonical_layer_rules + canonical_object_rules
↓
pbf_relayer_candidates
↓
pbf_source_object_stats
├─→ pbf_render_compatibility
└─→ pbf_detection_catalog
```
### 3.2 并行校验血缘
```text
pbf_source_object_stats + canonical_object_rules
↓
classification_validation
pbf_detection_catalog + object_geometry_allowlist
↓
geometry_consistency
style_layers + features + pbf_render_compatibility
↓
style_render_equivalence
pbf_detection_catalog + geometry_consistency
↓
detection_catalog_integrity
features
↓
tile_density + tile_layer_density
↓
spatial_anomalies
classification_validation
geometry_consistency
style_render_equivalence
detection_catalog_integrity
spatial_anomalies
↓
semantic_validation_summary
```
## 4. 表分类
下表按照当前数据库中的实际表进行分类。
| 表名 | 分类 | 说明 |
| --- | --- | --- |
| `features` | CORE TABLE | 原始要素主表，保存 `id`、`z/x/y`、`vt_layer`、`geom_type` |
| `properties` | CORE TABLE | 原始属性键值表 |
| `at_attributes` | SEMANTIC TABLE | `properties.k='at'` 的展开结果 |
| `feature_semantic` | SEMANTIC TABLE | 单要素语义投影表 |
| `object_type_candidates` | SEMANTIC TABLE | 对象类型候选表 |
| `object_catalog` | ANALYSIS TABLE | 对象目录与分布分析 |
| `object_type_stats` | ANALYSIS TABLE | 对象类型与几何统计 |
| `style_layers` | ANALYSIS TABLE | 从 `style.json` 提取的渲染层信息 |
| `style_mapping` | ANALYSIS TABLE | 样式层与对象候选的交叉映射 |
| `canonical_layer_rules` | SEMANTIC TABLE | 来源层规则表 |
| `canonical_object_rules` | SEMANTIC TABLE | 规范对象规则表 |
| `object_geometry_allowlist` | SEMANTIC TABLE | 对象几何白名单 |
| `pbf_relayer_candidates` | SEMANTIC TABLE | 语义叠加主表 |
| `pbf_source_object_stats` | ANALYSIS TABLE | 语义叠加聚合统计表 |
| `pbf_render_compatibility` | ANALYSIS TABLE | 渲染兼容统计表 |
| `pbf_detection_catalog` | SEMANTIC TABLE | 安全检测目录表 |
| `classification_validation` | VALIDATION TABLE | 分类稳定性校验表 |
| `geometry_consistency` | VALIDATION TABLE | 当前正式几何一致性校验表 |
| `geometry_consistency_audit` | ANALYSIS TABLE | 审计阶段保留的旧版几何快照 |
| `style_render_equivalence` | VALIDATION TABLE | 渲染等价校验表 |
| `detection_catalog_integrity` | VALIDATION TABLE | 检测目录完整性校验表 |
| `spatial_anomalies` | VALIDATION TABLE | 空间密度异常表 |
| `semantic_validation_summary` | VALIDATION TABLE | 整体验证摘要 |
| `tile_density` | ANALYSIS TABLE | 每瓦片 feature 数量 |
| `tile_layer_density` | ANALYSIS TABLE | 每瓦片、每来源层 feature 数量 |
| `tile_density_top100` | ANALYSIS TABLE | 最高密度瓦片 Top100 |
| `anomaly_navigation_geom` | ANALYSIS TABLE | 航标类几何异常审计 |
| `anomaly_reef_geom` | ANALYSIS TABLE | 鱼礁类几何异常审计 |
| `feature_flat` | TEMP TABLE | 当前数据库中的遗留/临时表，不在现行语义脚本主路径上 |
## 5. 官方执行顺序
当前建议的正式执行顺序如下：
1. 导入原始 PBF，生成 `features` 和 `properties`
2. 运行 `navsea_audit.py`，构建基础语义表、分析表和密度表
3. 运行 `navsea_relayer.py`，生成规范规则与语义叠加层
4. 运行 `navsea_semantic_validation.py`，生成分类、几何、渲染、检测目录和空间校验结果
5. 读取 `semantic_validation_summary` 确认总体状态
6. 以 `pbf_relayer_candidates` 为主表，结合 `features`、`properties`、`at_attributes` 产出下游 NavSea 数据集或矢量瓦片生产输入
## 6. 执行命令
### 6.1 构建基础语义与分析表
```bash
.venv/bin/python navsea_audit.py
```
作用
从 `features` / `properties` 出发，构建：
- `at_attributes`
- `feature_semantic`
- `object_catalog`
- `geometry_consistency_audit`
- `object_type_candidates`
- `object_type_stats`
- `style_layers`
- `style_mapping`
- `tile_density`
- `tile_layer_density`
- `tile_density_top100`
- 审计异常表
### 6.2 构建语义叠加层
```bash
.venv/bin/python navsea_relayer.py
```
作用
生成：
- `canonical_layer_rules`
- `canonical_object_rules`
- `object_geometry_allowlist`
- `pbf_relayer_candidates`
- `pbf_source_object_stats`
- `pbf_render_compatibility`
- `pbf_detection_catalog`
### 6.3 运行语义校验
```bash
.venv/bin/python navsea_semantic_validation.py
```
作用
生成：
- `classification_validation`
- `geometry_consistency`
- `style_render_equivalence`
- `detection_catalog_integrity`
- `spatial_anomalies`
- `semantic_validation_summary`
### 6.4 仅重建密度表
如果只需要更新空间密度与热点诊断，可以单独执行：
```bash
.venv/bin/python - <<'PY'
from navsea_audit import NavSeaAudit, DbConfig
NavSeaAudit(DbConfig()).build_tile_density()
PY
```
随后再执行：
```bash
.venv/bin/python navsea_semantic_validation.py
```
## 7. 最终输出数据集
### 7.1 当前系统中的“最终语义数据集”
当前系统中，最接近“最终 NavSea 语义要素表”的主表是：
`pbf_relayer_candidates`
它保存了：
- `feature_id`
- `z/x/y`
- `source_layer`
- `geom_type`
- `class_name`
- `shape_name`
- `layer_name`
- `object_type_source`
- `canonical_object_type`
- `semantic_granularity`
- `canonical_family`
- `classification_basis`
- `source_layer_scope`
- `semantic_key`
- `detection_key`
- `render_layer`
### 7.2 原始属性与语义属性的对应关系
`pbf_relayer_candidates` 本身并不保存原始全部属性值，因此最终语义数据集应按下述方式理解：
- 原始要素主键与瓦片定位来自 `features`
- 原始属性全集来自 `properties`
- 展开的 `at` 属性来自 `at_attributes`
- 语义投影属性来自 `feature_semantic`
- 规范对象分类来自 `pbf_relayer_candidates`
也就是说，当前 NavSea 最终语义数据集实际上是一个“主表 + 关联表”结构：
```text
主表：
pbf_relayer_candidates
关联恢复：
features → 原始要素标识、瓦片坐标、几何类型
properties → 原始属性键值
at_attributes → 展开的 at 属性
feature_semantic → class_name / shape_name / layer_name
```
### 7.3 关于 geometry 的说明
当前数据库 schema 中，`features` 表仅保存：
- `id`
- `z/x/y`
- `vt_layer`
- `geom_type`
它不保存完整坐标几何体字段。
因此，当前数据库中的“最终语义数据集”是面向语义重建、校验和下游生成的语义索引层，而不是一张带完整坐标几何的空间实体表。
### 7.4 下游生成 NavSea 数据的推荐入口
如果后续要生成 NavSea 下游数据或重新生成矢量瓦片，推荐以：
- `pbf_relayer_candidates` 作为语义主表
- `features` 作为原始要素锚点
- `properties` / `at_attributes` 作为原始属性源
共同构成生产输入。
## 8. 从零重建步骤
### 最小可复现步骤
1. 将原始 PBF 导入数据库，得到 `features` 与 `properties`
2. 执行 `navsea_audit.py`
3. 执行 `navsea_relayer.py`
4. 执行 `navsea_semantic_validation.py`
5. 检查 `semantic_validation_summary`
6. 若需要下游数据生产，则从 `pbf_relayer_candidates` 联合 `features`、`properties`、`at_attributes` 提取最终数据集
### 推荐的重建命令顺序
```bash
.venv/bin/python navsea_audit.py
.venv/bin/python navsea_relayer.py
.venv/bin/python navsea_semantic_validation.py
```
### 重建完成后的关键检查点
至少应检查以下表是否已刷新：
- `feature_semantic`
- `object_type_candidates`
- `canonical_object_rules`
- `canonical_layer_rules`
- `pbf_relayer_candidates`
- `pbf_detection_catalog`
- `classification_validation`
- `geometry_consistency`
- `style_render_equivalence`
- `detection_catalog_integrity`
- `spatial_anomalies`
- `semantic_validation_summary`
### 当前系统的审计终点
当前系统的“可审计终点”不是一个新的矢量瓦片文件，而是：
- 一套可复现的语义规则表
- 一套可复现的语义叠加主表
- 一套可重复执行的验证表
即：
```text
canonical_layer_rules
canonical_object_rules
pbf_relayer_candidates
pbf_detection_catalog
semantic_validation_summary
```
这套终点定义了 NavSea 当前语义流水线的正式数据库输出。

54
NavSea_Source_Layer_Consumption_Table.md Normal file
View File

@@ -0,0 +1,54 @@
# NavSea Source Layer Consumption Table
用途：汇总当前交付体系中每个 `source-layer` 的全量 feature 数，以及当前 [navsea.json](/home/wwwroot/newpec/navsea.json) 里哪些 style layer 正在消费它。
统计口径：
- feature 数来自 `pbf_analysis.pbf_relayer_candidates`
- 样式消费关系来自当前 [navsea.json](/home/wwwroot/newpec/navsea.json)
- 仅保留当前交付 `pbf` 与当前样式实际使用的 `37` 个 `source-layer`
| source-layer | feature_count | style_layer_count | style_layer_ids |
| --- | ---: | ---: | --- |
| `L739` | 429 | 1 | `water-boundary` |
| `L741` | 1696 | 1 | `regulatory-boundary` |
| `L基本線` | 53615 | 1 | `coast-structures-line` |
| `L概略等深線` | 5551 | 1 | `depth-contours-major` |
| `L海底地形` | 2075172 | 2 | `bathymetry-support`, `bathymetry-depth-labels` |
| `L海底線` | 14391 | 1 | `subsea-cables` |
| `L等深線` | 445051 | 2 | `depth-contours`, `depth-contour-labels` |
| `L陸上構造物陸` | 1258262 | 1 | `land-structures-line` |
| `L高さ制限` | 1028 | 1 | `height-limit-line` |
| `P754ククリ` | 771 | 1 | `special-pattern-line-754` |
| `P危険界ククリ` | 51962 | 1 | `danger-outline` |
| `P基本線` | 701625 | 6 | `sea-area-fill`, `tidal-flat`, `river-water`, `unsurveyed-area`, `shoal-danger-area`, `coast-structures-area` |
| `P基本線ククリ` | 1080780 | 1 | `clip-outline` |
| `P投錨注意障害物` | 58626 | 1 | `hazard-polygons` |
| `P投錨注意障害物ククリ` | 8743 | 1 | `anchor-danger-outline` |
| `P施設・境界線等` | 0 | 1 | `facility-zones` |
| `P施設・境界線等ククリ` | 566 | 1 | `facility-zone-outline` |
| `P橋りょう等構造物` | 25775 | 1 | `bridge-area` |
| `P漁具定置箇所` | 22595 | 1 | `fishery-areas` |
| `P潜堤` | 3881 | 1 | `submerged-structures` |
| `P穴` | 367217 | 1 | `land-hole` |
| `P航行危険障害物` | 0 | 1 | `navigation-hazard-polygons` |
| `P航行危険障害物ククリ` | 198 | 1 | `navigation-hazard-outline` |
| `P錨泊地等` | 0 | 1 | `anchorage-areas` |
| `P錨泊地等ククリ` | 297 | 1 | `anchorage-outline` |
| `P陸上構造物陸` | 35622 | 1 | `land-structures-area` |
| `P陸域` | 198655 | 1 | `land-area` |
| `p地名` | 36247 | 1 | `place-labels-sea` |
| `p地名陸` | 35388 | 1 | `place-labels-land` |
| `p底質` | 99863 | 2 | `bottom-material-points`, `bottom-material-labels` |
| `p投錨注意障害物` | 0 | 1 | `anchor-hazard-points` |
| `p施設・境界線等` | 6425 | 1 | `facility-points` |
| `p航行危険障害物` | 19427 | 1 | `hazard-points` |
| `p航路標識群` | 38520 | 4 | `nav-light-flare`, `nav-marks`, `nav-mark-labels`, `nav-light-abbr-labels` |
| `p錨泊地等` | 674 | 1 | `anchorage-symbols` |
| `p陸上構造物` | 32708 | 1 | `landmark-points` |
| `p高さ制限` | 858 | 1 | `height-limit-points` |
说明：
- 上表里的 `feature_count = 0`，表示当前 `pbf_relayer_candidates` 全量统计里没有对应记录，但在唐津 AOI 测试瓦片中这些 layer 仍然实际存在并已被样式消费。
- 当前 [navsea.json](/home/wwwroot/newpec/navsea.json) 对这 `37` 个 `source-layer` 都已经有覆盖，不存在“数据里有 layer 但样式完全不消费”的情况。

273
NavSea_Style_Render_Change_Audit_Karatsu_10nm.md Normal file
View File

@@ -0,0 +1,273 @@
# NavSea 原始样式与使用版样式渲染变更审计
范围：唐津中心 10 海里 AOI
原始样式：[style.json](/root/weather/src/pbf/style.json)
使用版样式：[navsea-delivery.json](/home/wwwroot/newpec/navsea-delivery.json)
使用版瓦片：`/home/wwwroot/pbf-delivery-karatsu-10nm`
## 1. 结论
结合原始 `style.json` 和使用版样式来看，渲染的“对象内容”没有减少，但渲染“表达方式”已经明显改变。
最核心的判断是：
- AOI 内真实存在的 `37` 个 `source-layer`，原始样式和使用版样式都覆盖到了
- 但样式层组织从 `87` 层收成了 `50` 层
- 很多 `source-layer` 的渲染被重新归并、重命名、重分层
- 样式直接依赖的旧字段减少了，新版开始混合使用旧字段和新语义字段
一句话总结：
> 这不是“少画了东西”，而是“同一批东西换了一套渲染组织方式”。
## 2. 总体统计
### 2.1 样式规模
- 原始样式层数：`87`
- 使用版样式层数：`50`
### 2.2 AOI 内实际图层覆盖
- AOI 内实际存在的 `source-layer`：`37`
- 原始样式覆盖到的 AOI `source-layer`：`37`
- 使用版样式覆盖到的 AOI `source-layer`：`37`
结论：
- 在唐津 10 海里范围内，使用版并没有漏掉某个实际存在的 `source-layer`
- 变化主要不是“有没有画”，而是“怎么画”
## 3. 字段依赖变化
### 3.1 原始样式直接读取的字段数
原始样式通过 `["get", ...]` 直接依赖的字段共 `21` 个，主要包括：
- `分類番号`
- `名称`
- `名称補助`
- `形状分類番号`
- `日本語地名`
- `明弧/分孤`
- `水深値(m)`
- `灯略記`
- `灯色`
- `英文字地名`
- `表示位置`
- `表示用番号`
- `角度`
- `高さ(m)`
- `高さ/深度(m)`
- `cog`
- `delay`
- `hdg`
- `length`
- `rot`
- `sog`
### 3.2 使用版样式直接读取的字段数
使用版样式直接依赖的字段共 `13` 个：
- `canonical_object_type`
- `分類番号`
- `名称`
- `名称補助`
- `形状分類番号`
- `日本語地名`
- `水深値(m)`
- `灯略記`
- `灯色`
- `英文字地名`
- `表示位置`
- `表示用番号`
- `高さ/深度(m)`
### 3.3 差异判断
使用版不再直接读取这些旧字段：
- `明弧/分孤`
- `角度`
- `高さ(m)`
- `cog`
- `delay`
- `hdg`
- `length`
- `rot`
- `sog`
同时新增了新语义字段依赖：
- `canonical_object_type`
结论：
- 使用版已经开始摆脱部分旧海图编码和动态字段依赖
- 但当前仍保留了对 `分類番号`、`形状分類番号`、`表示用番号`、`灯色` 等旧字段的依赖
## 4. AOI 内渲染组织变化
### 4.1 没变的部分
- 两边都覆盖了 AOI 内的全部 `37` 个 `source-layer`
- 本 AOI 内不存在“旧样式能画、使用版完全不画”的数据层
### 4.2 变化最大的部分
以下 `source-layer` 在使用版里被显著重组了：
- `L概略等深線`
- 原始样式：`8` 个样式层
- 使用版：`1` 个样式层 `depth-contours-major`
- `L等深線`
- 原始样式：`5` 个样式层
- 使用版：`2` 个样式层 `depth-contours` + `depth-contour-labels`
- `P基本線`
- 原始样式：`4` 个样式层
- 使用版：`6` 个样式层
- 说明：使用版把同一面层按语义拆成了多种面表现，比如 `sea-area-fill`、`tidal-flat`、`river-water`、`unsurveyed-area`、`shoal-danger-area`
- `p航路標識群`
- 原始样式：`5` 个样式层
- 使用版：`4` 个样式层
- `p底質`
- 原始样式：`1` 个样式层
- 使用版：`2` 个样式层
- 说明：使用版把底质本体和文字拆开了
### 4.3 被简化的部分
这些层在使用版中从多层压缩为单层：
- `P橋りょう等構造物`：`2 -> 1`
- `P漁具定置箇所`：`2 -> 1`
- `P潜堤`：`2 -> 1`
- `p地名`：`2 -> 1`
- `p地名陸`：`2 -> 1`
- `p高さ制限`：`2 -> 1`
结论：
- 使用版不是简单复制旧样式
- 使用版对不少对象做了“压层”和“归并”
- 同时也对少数核心面层做了“按语义细分”
## 5. 典型图层对比
### 5.1 `L739`
- 原始样式：`#b#d2#d5#d10#BK#at#L739`
- 使用版：`water-boundary`
变化：
- 从旧式缩尺/颜色编码命名，变成语义命名
### 5.2 `L741`
- 原始样式：`#b#d2#d5#d10#BS#at#L741`
- 使用版：`regulatory-boundary`
变化：
- 图层语义比旧版更清楚
### 5.3 `L海底地形`
- 原始样式：`#b#海底地形`、`#b#C000`
- 使用版：`bathymetry-support`、`bathymetry-depth-labels`
变化：
- 使用版明确拆成“支撑线”和“水深数字”
### 5.4 `L等深線`
- 原始样式：`#b#L等深線`、`#d2#L等深線`、`#d5#L等深線`、`#d10#L等深線`、`#b#d2#d5#d10#C140`
- 使用版：`depth-contours`、`depth-contour-labels`
变化：
- 原始样式按缩尺和强调层堆叠
- 使用版变成“线 + 标注”的直接语义拆分
### 5.5 `P基本線`
- 原始样式：`#b#at#P基本線`、`#d2#at#P基本線`、`#d5#at#P基本線`、`#d10#at#P基本線`
- 使用版：
- `sea-area-fill`
- `tidal-flat`
- `river-water`
- `unsurveyed-area`
- `shoal-danger-area`
- `coast-structures-area`
变化：
- 使用版在这一层上变化最大
- 同一个原始面层被拆成多个语义面层
- 视觉表达会比旧样式更“语义化”
## 6. 原始样式中存在、使用版中不再出现的 `source-layer`
原始样式定义了 `57` 个 `source-layer`，使用版只有 `37` 个。
但这里要注意：
- 这 `20` 个差异层并不是“使用版漏画了 AOI 数据”
- 它们是原始样式里定义过、但在当前 AOI 与当前使用版中未出现的层
主要包括：
- `L700`
- `L701`
- `L702`
- `L725`
- `L738`
- `L740`
- `L748`
- `L749`
- `L危険界`
- `L航路`
- `P721ククリ`
- `P730ククリ`
- `P投錨注意障害物透明`
- `P施設・境界線等透明`
- `P航路`
- `P航路ククリ`
- `P誘導線ククリ`
- `dummy`
- `pパイロットステーション`
- `p航路境界等`
结论：
- 这是“原始样式定义范围更大”
- 不是“使用版把 AOI 内实际对象删掉了”
## 7. 最终判断
结合原始样式和使用版样式来看，渲染变化可以归纳成三类：
### 7.1 没变的
- AOI 内实际要画的数据层都还在
- 使用版没有丢掉任何 AOI 内真实存在的 `source-layer`
### 7.2 变了但不属于缺失
- 图层命名体系从旧的技术型命名变成了新的语义命名
- 很多对象从“按缩尺堆层”变成“按语义拆层”
- 样式总层数明显减少
- 旧字段依赖减少，新语义字段开始进入样式
### 7.3 需要继续注意的
- 使用版虽然是交付版，但还没有完全摆脱旧字段体系
- 某些旧样式细节如果依赖 `明弧/分孤`、`角度`、`高さ(m)` 等字段，在使用版里会弱化或消失
- 现在的变化更偏“渲染体系重构”，不是“数据减少”
一句话总结：
> 从原始 `style.json` 切到使用版样式后，唐津 10 海里范围内“画的对象”没有少，但“画法”已经发生了统计上明确可见的重构。

634
PBF_ANALYSIS_DATABASE_TABLES.md Normal file
View File

@@ -0,0 +1,634 @@
# pbf_analysis 数据库表功能说明
## 1. 目的
这份文档说明当前 `pbf_analysis` 数据库中各张 NavSea 相关表的用途、所处阶段、与当前工程的关系。
目标是让后续接手的人能快速回答这些问题：
- 哪些表是原始解析结果
- 哪些表是语义分类与归层结果
- 哪些表是规则注册表
- 哪些表是检测/碰撞/航行分析专题视图
- 哪些表是样式分析与渲染审计结果
- 哪些表是历史分析或中间产物
## 2. 总体分层
当前 `pbf_analysis` 里的表可以按用途分成 6 类：
1. 原始与基础解析层
2. 语义分类与归层层
3. 几何与空间支撑层
4. 规则注册与映射层
5. 样式分析与渲染审计层
6. 业务专题与异常分析层
说明：
- `BASE TABLE` 表示实体表
- `VIEW` 表示视图
- `table_rows` 为 MySQL 估计值，仅用于了解量级
## 3. 关键原则
当前数据库不是只存“最终结果”，而是同时承担四类职责：
- 保存原始解析和展平结果
- 保存分类和标准化结果
- 保存可追溯规则注册表
- 保存审计与专题查询结果
后续开发应优先遵循：
- 映射规则尽量收口到规则表
- 审计结果进专门审计表
- 原始值与标准值并存，避免不可逆覆盖
## 4. 表分组说明
### 4.1 原始与基础解析层
这些表主要用于把原始 `pbf` 解码成可查询的结构化数据。
#### `features`
- 类型：`BASE TABLE`
- 估计行数：`6653014`
- 关键字段：
- `id, z, x, y, vt_layer, geom_type`
- 用途：
- 保存原始瓦片的基础 feature 索引
- 作为很多后续表的原始主干
#### `properties`
- 类型：`BASE TABLE`
- 估计行数：`22213975`
- 关键字段：
- `feature_id, k, v`
- 用途：
- 存原始属性键值对
- 便于后续抽取 `fid`、`分類番号`、`名称` 等字段
#### `at_attributes`
- 类型：`BASE TABLE`
- 估计行数：`6674189`
- 关键字段：
- `feature_id, k, v`
- 用途：
- 拆分原始 `at` 属性集合
- 为分类、渲染兼容、规则归纳提供细粒度字段来源
#### `feature_flat`
- 类型：`VIEW`
- 关键字段：
- `feature_id, vt_layer, class_id, fid, at`
- 用途：
- 对原始 feature 做便于分析的展平视图
- 常用于快速查看对象分类号和 `fid`
#### `feature_semantic`
- 类型：`BASE TABLE`
- 估计行数：`6367524`
- 关键字段：
- `feature_id, z, x, y, vt_layer, geom_type, class_name, shape_name, layer_name`
- 用途：
- 原始对象语义的早期抽取层
- 为后续 taxonomy 和 render 分析提供基础口径
#### `feature_geojson_lookup`
- 类型：`BASE TABLE`
- 估计行数：`6636967`
- 关键字段：
- `z, x, y, feature_id, fid, vt_layer, geom_type`
- 用途：
- 做 feature 与瓦片实例的查询映射
- 便于回查到具体 tile
### 4.2 几何与空间支撑层
这些表负责把原始 feature 补成可空间分析的对象。
#### `feature_geometry_stage`
- 类型：`BASE TABLE`
- 估计行数：`13597637`
- 关键字段：
- `feature_id, geometry_wkt`
- 用途：
- 几何导入与中间转换阶段表
- 为后续空间启用、几何核查服务
#### `navsea_feature_geometry`
- 类型：`BASE TABLE`
- 估计行数：`6685117`
- 关键字段：
- `feature_id, geometry, geometry_source`
- 用途：
- NavSea 当前正式使用的 feature 几何表
- 支撑碰撞、空间专题和检测视图
#### `geometry_consistency`
- 类型：`BASE TABLE`
- 估计行数：`327`
- 关键字段：
- `canonical_object_type, geometry_type, feature_count, geometry_consistency_status`
- 用途：
- 统计某类对象的几何类型是否符合预期
#### `geometry_consistency_audit`
- 类型：`BASE TABLE`
- 估计行数：`331`
- 关键字段：
- `canonical_object_type, geometry_type, feature_count, geometry_consistency_status`
- 用途：
- 与 `geometry_consistency` 类似，用于审计记录与复核
#### `object_geometry_allowlist`
- 类型：`BASE TABLE`
- 估计行数：`310`
- 关键字段：
- `canonical_object_type, geometry_type, rule_source, is_allowed, notes`
- 用途：
- 定义哪些对象类型允许哪些几何类型
- 是 geometry 审计的规则依据之一
### 4.3 语义分类与归层层
这些表描述“对象是什么”和“应该归到哪一层”。
#### `object_catalog`
- 类型：`BASE TABLE`
- 估计行数：`319`
- 关键字段：
- `layer_name, class_name, shape_name, vt_layer, geom_type, feature_count`
- 用途：
- 原始对象目录
- 手工 taxonomy 审查和统计的基础表
#### `object_type_candidates`
- 类型：`BASE TABLE`
- 估计行数：`6473314`
- 关键字段：
- `feature_id, object_type_source, object_type, vt_layer, geom_type`
- 用途：
- 某个 feature 可能属于哪些对象类型的候选集合
- 是 taxonomy 判定前的候选层
#### `object_type_stats`
- 类型：`BASE TABLE`
- 估计行数：`255`
- 关键字段：
- `object_type, geom_type, feature_count`
- 用途：
- 对候选对象类型做统计
#### `canonical_object_rules`
- 类型：`BASE TABLE`
- 估计行数：`221`
- 关键字段：
- `canonical_object_type, canonical_family, classification_basis, source_layer_scope, ...`
- 用途：
- 旧版人工整理的对象级归类规则汇总表
- 偏分析结果，不是最新规则注册表
#### `canonical_layer_rules`
- 类型：`BASE TABLE`
- 估计行数：`49`
- 关键字段：
- `source_layer, semantic_granularity, canonical_family, render_strategy, ...`
- 用途：
- 旧版人工整理的层级归并规则表
- 用于理解老系统如何按层做语义合并
#### `feature_semantic`
- 已在基础层列出
- 同时也是对象语义候选的重要来源
#### `pbf_relayer_candidates`
- 类型：`BASE TABLE`
- 估计行数：`6547009`
- 关键字段：
- `feature_id, z, x, y, source_layer, geom_type, ... canonical_object_type, canonical_family, semantic_key, detection_key, render_layer`
- 用途：
- 当前构建 `pbf` 时最重要的结果表之一
- 保存 feature 经过分类和归层后的候选结果
- `navsea_tile_builder.py` 会结合该表与原始 `pbf` 一起输出新瓦片
#### `pbf_source_object_stats`
- 类型：`BASE TABLE`
- 估计行数：`215`
- 关键字段：
- `source_layer, render_layer, canonical_family, canonical_object_type, geom_type, detection_key, semantic_key, feature_count`
- 用途：
- 统计不同源层在重构后如何分布
#### `pbf_render_compatibility`
- 类型：`BASE TABLE`
- 估计行数：`58`
- 关键字段：
- `source_layer, render_layer, canonical_family, semantic_granularity, style_bound, feature_count, canonical_object_types`
- 用途：
- 评估重构后数据与旧样式兼容性的分析表
#### `pbf_detection_catalog`
- 类型：`BASE TABLE`
- 估计行数：`359`
- 关键字段：
- `detection_key, canonical_family, canonical_object_type, geom_type, feature_count, source_layers`
- 用途：
- 检测视角的对象目录
- 支撑 detection_key 设计与校验
### 4.4 规则注册与映射层
这组表是现在最重要的“规则中心”，用于收口旧版到新版的映射逻辑。
#### `navsea_rule_bundle`
- 类型：`BASE TABLE`
- 估计行数：`1`
- 关键字段：
- `bundle_id, bundle_version, status, effective_date, ...`
- 用途：
- 规则包总表
- 记录当前使用的是哪一套 taxonomy/render/source-layer/field-value 规则
#### `navsea_source_layer_rules`
- 类型：`BASE TABLE`
- 估计行数：`49`
- 关键字段：
- `source_layer_jp, source_layer_std, semantic_granularity, canonical_family, render_strategy, ...`
- 用途：
- 日文 `source-layer` 到标准层名的映射规则
- 是 source-layer 可逆标准化的正式注册表
#### `navsea_field_name_rules`
- 类型：`BASE TABLE`
- 估计行数：`17`
- 关键字段：
- `field_name_jp, field_name_std, field_group_name, keep_in_delivery, keep_in_engineering, ...`
- 用途：
- 旧字段名到标准字段名的映射
- 同时定义字段在哪种输出中保留
#### `navsea_field_value_rules`
- 类型：`BASE TABLE`
- 估计行数：`20`
- 关键字段：
- `rule_id, field_name_jp, legacy_value, standardized_field, standardized_value, ...`
- 用途：
- 旧字段值到新标准字段值的映射
- 例如灯色、表示位置等值映射
#### `navsea_taxonomy_rules`
- 类型：`BASE TABLE`
- 估计行数：`8`
- 关键字段：
- `rule_id, priority, enabled, match_scope, match_expr_json, output_json, canonical_family, canonical_object_type, detection_key_template`
- 用途：
- 正式 taxonomy 规则表
- 定义 feature 在什么条件下被归成哪个 `canonical_object_type`
#### `navsea_render_rules`
- 类型：`BASE TABLE`
- 估计行数：`9`
- 关键字段：
- `rule_id, priority, match_expr_json, output_json, chart_render_type, chart_symbol_code, chart_fill_style, ...`
- 用途：
- 正式渲染语义规则表
- 定义 feature 应输出哪些 `chart_* / hazard_* / area_*` 字段
#### `navsea_mapping_registry_vw`
- 类型：`VIEW`
- 用途：
- 统一查看当前各类映射规则
- 适合人工巡检和管理
#### `navsea_legacy_to_new_mapping_vw`
- 类型：`VIEW`
- 用途：
- 从旧值查新值
- 适合排查“旧字段为什么被映射成这个新字段”
#### `navsea_new_to_legacy_mapping_vw`
- 类型：`VIEW`
- 用途：
- 从新值反查旧值
- 适合可逆追述和规则审计
### 4.5 NavSea 对象字典与能力表
#### `navsea_object_taxonomy`
- 类型：`BASE TABLE`
- 估计行数：`127`
- 关键字段：
- `canonical_object_type, object_family, detection_class, description`
- 用途：
- NavSea 对象字典
- 定义各 canonical object 的基础说明
#### `navsea_object_capabilities`
- 类型：`BASE TABLE`
- 估计行数：`259`
- 关键字段：
- `canonical_object_type, capability`
- 用途：
- 定义对象能力，例如是否用于碰撞、导航参考、缠绕风险等
- 给下游专题视图提供能力标签
### 4.6 样式分析与渲染审计层
#### `style_layers`
- 类型：`BASE TABLE`
- 估计行数：`113`
- 关键字段：
- `layer_id, source_layer, icon, line_color, fill_color, layer_type`
- 用途：
- 解析旧样式后的 layer 清单
#### `style_mapping`
- 类型：`BASE TABLE`
- 估计行数：`42`
- 关键字段：
- `layer_id, source_layer, layer_type, icon, line_color, fill_color, object_type, geom_type, feature_count`
- 用途：
- 分析旧样式的 source-layer 消费关系
#### `style_render_equivalence`
- 类型：`BASE TABLE`
- 估计行数：`53`
- 关键字段：
- `style_layer, feature_count_before, feature_count_after, difference, render_status`
- 用途：
- 用于分析重构前后样式层渲染覆盖是否等价
#### `navsea_render_audit_run`
- 类型：`BASE TABLE`
- 估计行数：`1`
- 关键字段：
- `audit_name, original_style_path, engineering_style_path, original_tile_root, engineering_tile_root, result_count, status_counts_json, ...`
- 用途：
- 存放“最新一次”渲染审计的摘要
- 当前约定只保留最新一轮结果
#### `navsea_render_audit_result`
- 类型：`BASE TABLE`
- 估计行数：`2357346`
- 关键字段：
- `audit_name, object_instance_key, object_instance_id, object_id, fid_legacy, tile_z, tile_x, tile_y, source_layer, status, original_components_json, engineering_components_json`
- 用途：
- 渲染审计结果明细表
- 可按 `fid_legacy` 追查某个对象在原始样式和工程样式下的渲染差异
### 4.7 业务专题与异常分析层
这组表/视图偏业务消费和专题分析。
#### `navsea_detection_objects`
- 类型：`VIEW`
- 用途：
- 检测对象总视图
- 面向 detection 任务使用
#### `navsea_detection_objects_spatial`
- 类型：`VIEW`
- 用途：
- 检测对象的空间分析视图
#### `navsea_collision_objects`
- 类型：`VIEW`
- 用途：
- 碰撞相关对象视图
#### `navsea_grounding_objects`
- 类型：`VIEW`
- 用途：
- 搁浅风险相关对象视图
#### `navsea_entangle_objects`
- 类型：`VIEW`
- 用途：
- 缠绕风险相关对象视图
#### `navsea_boundary_reference_objects`
- 类型：`VIEW`
- 用途：
- 边界参考对象视图
#### `navsea_route_reference_objects`
- 类型：`VIEW`
- 用途：
- 航路参考对象视图
#### `navsea_navigation_mark_objects`
- 类型：`VIEW`
- 用途：
- 航标专题对象视图
#### `anomaly_navigation_geom`
- 类型：`BASE TABLE`
- 估计行数：`0`
- 用途：
- 导航相关几何异常对象表
#### `anomaly_reef_geom`
- 类型：`BASE TABLE`
- 估计行数：`13`
- 用途：
- 鱼礁/礁石类几何异常对象表
#### `spatial_anomalies`
- 类型：`BASE TABLE`
- 估计行数：`693`
- 关键字段：
- `anomaly_type, affected_feature_count, region_hint, severity`
- 用途：
- 空间异常摘要
#### `classification_validation`
- 类型：`BASE TABLE`
- 估计行数：`177`
- 用途：
- 分类结果验证摘要
#### `detection_catalog_integrity`
- 类型：`BASE TABLE`
- 估计行数：`343`
- 用途：
- 检测目录完整性校验
#### `semantic_validation_summary`
- 类型：`BASE TABLE`
- 估计行数：`1`
- 用途：
- 语义验证总摘要
### 4.8 密度与覆盖统计层
#### `tile_density`
- 类型：`BASE TABLE`
- 估计行数：`65467`
- 关键字段：
- `z, x, y, feature_count`
- 用途：
- 每个 tile 的 feature 密度统计
#### `tile_density_top100`
- 类型：`BASE TABLE`
- 估计行数：`100`
- 用途：
- feature 最密的 top100 tile
#### `tile_layer_density`
- 类型：`BASE TABLE`
- 估计行数：`207315`
- 关键字段：
- `z, x, y, vt_layer, feature_count`
- 用途：
- 按 tile 和 layer 分解的密度统计
## 5. 当前工程最重要的表
如果只看当前构建与审计链路，最重要的是这些：
### 构建时必看
- `pbf_relayer_candidates`
- `navsea_rule_bundle`
- `navsea_source_layer_rules`
- `navsea_field_name_rules`
- `navsea_field_value_rules`
- `navsea_taxonomy_rules`
- `navsea_render_rules`
- `navsea_feature_geometry`
### 审计时必看
- `navsea_render_audit_run`
- `navsea_render_audit_result`
- `style_layers`
- `style_mapping`
- `style_render_equivalence`
### 数据理解时必看
- `object_catalog`
- `pbf_source_object_stats`
- `pbf_detection_catalog`
- `navsea_object_taxonomy`
- `navsea_object_capabilities`
## 6. 当前推荐使用方式
### 想查某个对象为什么被分类成某个新对象
优先查：
- `pbf_relayer_candidates`
- `navsea_taxonomy_rules`
- `navsea_legacy_to_new_mapping_vw`
### 想查某个对象为什么被渲染成某种 `chart_*`
优先查：
- `navsea_render_rules`
- `navsea_mapping_registry_vw`
- 工程版 `pbf` 中对应 feature 的输出字段
### 想查某个对象原始版和工程版渲染是否一致
优先查：
- `navsea_render_audit_result`
- `navsea_render_audit_run`
### 想做碰撞、搁浅、航路专题分析
优先查：
- `navsea_collision_objects`
- `navsea_grounding_objects`
- `navsea_route_reference_objects`
- `navsea_navigation_mark_objects`
## 7. 维护建议
当前数据库里同时有“旧分析表”和“新规则注册表”，后续维护建议如下：
- 正式规则源以 `tasks/pbf/mappings/*.yaml` 为主
- 数据库规则表作为运行时注册中心
- 不要再把同一套逻辑分散写进多个临时分析表
- 新增规则时优先更新：
- `navsea_rule_bundle`
- `navsea_source_layer_rules`
- `navsea_field_name_rules`
- `navsea_field_value_rules`
- `navsea_taxonomy_rules`
- `navsea_render_rules`
## 8. 与 Git 管理的关系
建议把下面这些内容一起纳入项目 Git：
- 本文档
- `tasks/pbf/` 下的规范文档
- `tasks/pbf/mappings/` 下的规则源文件
- `navsea_mapping_registry_sync.py`
- `navsea_tile_builder.py`
- `navsea_render_audit.py`
这样数据库中的规则、代码中的构建逻辑、审计流程就能形成完整闭环。

139
PROJECT_PATH_AND_GIT_PLAN.md Normal file
View File

@@ -0,0 +1,139 @@
# 项目路径与 Git 接管说明
## 正式项目路径
当前项目的正式工作目录是：
- `/root/sourceserver/pbf`
以后所有相关命令、脚本、VSCode 工作区、日志路径，都应以这个目录为准。
旧路径：
- `/root/weather`
当前已经不是主项目目录，不能再作为构建和审计脚本的启动路径使用。
## 当前状态
已确认以下关键文件都在正式目录中：
- `navsea_tile_builder.py`
- `navsea_render_audit.py`
- `navsea_mapping_registry.py`
- `navsea_mapping_registry_sync.py`
- `tasks/pbf/`
- `src/pbf/`
已确认当前目录还不是 Git 仓库：
- `git rev-parse --is-inside-work-tree` 返回 `NOT_GIT`
## 后续统一约定
以后统一使用：
- 工作目录：`/root/sourceserver/pbf`
- Python 环境：`/root/sourceserver/pbf/.venv`
- 构建日志目录：`/root/sourceserver/pbf/logs`
## 全量工程版构建命令
```bash
mkdir -p /root/sourceserver/pbf/logs
cd /root/sourceserver/pbf
nohup env \
PYTHONUNBUFFERED=1 \
PYTHONPATH=/root/sourceserver/pbf/.venv/lib/python3.12/site-packages \
python3 /root/sourceserver/pbf/navsea_tile_builder.py \
--all-tiles \
--zmin 0 \
--zmax 14 \
--output /home/wwwroot/pbf-engineering-full \
--workers 1 \
--engineering \
--fid-key 'thisMyWorld@2026' \
--fid-key-id navsea-fid-key-v1 \
--bundle-id navsea-core \
> /root/sourceserver/pbf/logs/navsea_full_engineering_build.log 2>&1 &
```
## Git 服务器接管建议
目标服务器：
- `192.168.200.184`
建议流程：
1. 在服务器上先创建空仓库
2. 在本地项目目录初始化 Git
3. 添加 `.gitignore`
4. 首次提交
5. 添加远端并推送
## 推荐命令
### 1. 初始化本地仓库
```bash
cd /root/sourceserver/pbf
git init
```
### 2. 创建主分支
```bash
git branch -M main
```
### 3. 添加远端
如果 Git 服务器上仓库地址是：
- `ssh://git@192.168.200.184/<group>/pbf.git`
则命令为：
```bash
git remote add origin ssh://git@192.168.200.184/<group>/pbf.git
```
如果是 HTTP 地址，则改成对应 URL。
### 4. 首次提交
```bash
git add .
git commit -m "Initial import of NavSea pbf project"
```
### 5. 首次推送
```bash
git push -u origin main
```
## 提交前建议排除的内容
建议不要直接提交这些目录或文件：
- `.venv/`
- `__pycache__/`
- `logs/`
- 大体量生成产物
- 临时输出目录
- 本地缓存
如果准备正式接管 Git，建议先补一份 `.gitignore`，再做首次提交。
## 推荐下一步
最稳的顺序是：
1. 以后统一从 `/root/sourceserver/pbf` 工作
2. 先补 `.gitignore`
3. 初始化 Git
4. 在 `192.168.200.184` 上创建空仓库
5. 推送首版代码

53
RUNNING.md Normal file
View File

@@ -0,0 +1,53 @@
# Running the weather pipeline
## Python
Use `python3`. This workspace does not provide a `python` shim.
After installing dependencies in the project virtual environment, prefer `.venv/bin/python`.
## Required packages
Install:
```bash
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```
The pipeline requires these runtime dependencies:
- `requests`
- `numpy`
- `xarray`
- `cfgrib`
- `eccodes`
- `mercantile`
- `mapbox-vector-tile`
## Commands
Run the full pipeline:
```bash
.venv/bin/python src/weather_pipeline.py
```
Run individual steps:
```bash
.venv/bin/python src/gfs_downloader.py
.venv/bin/python src/grid_builder_v2.py
.venv/bin/python src/vector_tile_generator.py
```
Wave fields in the weather tiles come from the downloader and grid builder. If you want `wh`/`wdir`/`wp` to contain real data instead of compatibility zeros from older grid JSON, rerun the downloader before rebuilding grids and tiles.
The weather tile generator now builds `z=2,4,6,8,10,12` by default so wind barb density stays usable after zooming in. If you need a custom set, override with `WEATHER_TILE_ZOOMS`, for example:
```bash
WEATHER_TILE_ZOOMS=2,4,6,8,10,12,14 .venv/bin/python src/vector_tile_generator.py
```
## Known environment blockers in this session
- Network access may prevent `src/gfs_downloader.py` from reaching NOAA.

115
UPGRADE_REPORT.md Normal file
View File

@@ -0,0 +1,115 @@
# GFS Downloader 升级报告
## 任务目标
优化 gfs_downloader.py，改进数据下载效率，特别是对于日本附近区域的气象数据。
## 升级内容
### 1. API 变更
**旧方法**: 直接下载完整 GFS 文件
```
BASE_URL = https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod
文件大小: ~500MB/file
```
**新方法**: 使用 NOAA GRIB Subset API，按需下载指定区域和变量
```
BASE_URL = https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl
文件大小: ~4.2MB/file
```
### 2. 数据范围限制
**下载区域**:
- 左经度: 120°E
- 右经度: 150°E
- 上纬度: 50°N
- 下纬度: 20°N
- 覆盖: 日本海、东海、太平洋日本海域
**下载变量** (5个):
- UGRD: 东西方向风速
- VGRD: 南北方向风速
- APCP: 累积降水量
- PRMSL: 平均海平面气压
- TMP: 温度
注: 移除了 HTSGW、DIRPW、PERPW（浪高、浪向、浪周期）
原因: 这些变量在 GFS 0.25°数据集中不可用，应从波浪数据集获取
**下载高度层** (3个):
- 10m above ground
- Surface
- Mean sea level
### 3. 性能改进
| 指标 | 旧方法 | 新方法 | 改进 |
|------|-------|-------|------|
| 单文件大小 | 485.6 MB | 3.3-4.4 MB | **99.3%** ↓ |
| 72小时预报文件数 | 6个 | 25个 | 100% 完整率 |
| 72小时总大小 | 2.7 GB | 105 MB | **96%** ↓ |
| 平均单位数据量 | 450 MB/预报 | 4.2 MB/预报 | **99%** ↓ |
| 下载时间 (估计) | ~1小时+ | ~10分钟 | 85% ↓ |
### 4. 测试结果
✅ **数据完整性**: 25个预报时次全部下载成功
✅ **文件有效性**: GRIB 格式正确，数据可正常解析
✅ **覆盖范围**: 日本附近区域完整覆盖 (1440×721 格点)
✅ **变量可用性**: UGRD、VGRD、PRMSL、TMP、APCP 全部可读
### 5. 下载统计
```
下载文件数: 25个
总数据量: 105 MB
预报周期: 2026-03-12 00Z
分辨率: 0.25° × 0.25°
时间间距: 3小时 (f000-f072)
文件大小分布:
3.4 MB: 1个
4.0 MB: 1个
4.1 MB: 2个
4.2 MB: 9个
4.3 MB: 9个
4.4 MB: 3个
```
## 关键改进
1. **存储优化**: 数据量从 2.7GB 减至 105MB，节省 96% 存储空间
2. **网络优化**: 下载时间减少 85%，更快获得气象数据
3. **区域专化**: 仅下载日本附近数据，避免全球数据浪费
4. **自动化**: 脚本自动重试、跳过已完成文件，支持续下载
## 下一步建议
1. **波浪数据集成**: 单独配置浪高、浪向、浪周期数据源
- 来源: NOAA WW3 或 GFS Wave 数据
- 下载器: filter_gfs_wave_0p25.pl
2. **多区域支持**: 配置不同地区的下载参数
- 东南亚区域、太平洋区域等
3. **增量更新**: 每 3 小时更新最新的 GFS 预报周期
4. **数据验证**: 添加 MD5 校验、数据完整性检查
## 验证命令
```bash
# 查看下载的文件
ls -lh data/grib/*.grib2
# 验证 GRIB 文件可读性
source /root/weather_test_env/bin/activate
python3 << 'EOF'
import xarray as xr
ds = xr.open_dataset("data/grib/20260312_00_f000.grib2",
engine="cfgrib",
filter_by_keys={'typeOfLevel': 'surface'})
print(ds.data_vars)

629
navsea_audit.py Normal file
View File

@@ -0,0 +1,629 @@
import csv
import json
import os
import tempfile
import time
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
import pymysql
ROOT = Path(__file__).resolve().parent
STYLE_JSON_PATH = ROOT / "src" / "pbf" / "style.json"
REPORT_PATH = ROOT / "navsea_classification_report.md"
@dataclass(frozen=True)
class DbConfig:
host: str = os.getenv("NAVSEA_DB_HOST", "localhost")
port: int = int(os.getenv("NAVSEA_DB_PORT", "3306"))
user: str = os.getenv("NAVSEA_DB_USER", "root")
password: str = os.getenv("NAVSEA_DB_PASSWORD", "2chi9ks2")
database: str = os.getenv("NAVSEA_DB_NAME", "pbf_analysis")
unix_socket: str | None = os.getenv("NAVSEA_DB_SOCKET", "/tmp/mysql.sock")
class NavSeaAudit:
def __init__(self, db_config: DbConfig) -> None:
self.db_config = db_config
self.metrics: dict[str, int | float | str] = {}
@contextmanager
def connect(self, *, streaming: bool = False, local_infile: bool = False):
kwargs = {
"host": self.db_config.host,
"port": self.db_config.port,
"user": self.db_config.user,
"password": self.db_config.password,
"database": self.db_config.database,
"charset": "utf8mb4",
"autocommit": False,
"local_infile": local_infile,
"cursorclass": pymysql.cursors.SSCursor if streaming else pymysql.cursors.Cursor,
}
if self.db_config.unix_socket and self.db_config.host in {"localhost", "127.0.0.1"}:
kwargs["unix_socket"] = self.db_config.unix_socket
connection = pymysql.connect(**kwargs)
try:
yield connection
connection.commit()
except Exception:
connection.rollback()
raise
finally:
connection.close()
def run(self) -> None:
start = time.time()
self.validate_inputs()
self.parse_at_attributes()
self.build_feature_semantic()
self.build_object_catalog()
self.build_geometry_consistency()
self.build_object_type_candidates()
self.build_object_type_stats()
self.build_style_layers()
self.build_style_mapping()
self.build_tile_density()
self.build_anomaly_tables()
self.generate_report()
self.metrics["pipeline_runtime_seconds"] = round(time.time() - start, 2)
print(f"Pipeline completed in {self.metrics['pipeline_runtime_seconds']} seconds")
def validate_inputs(self) -> None:
if not STYLE_JSON_PATH.exists():
raise FileNotFoundError(f"style.json not found: {STYLE_JSON_PATH}")
with self.connect() as conn:
with conn.cursor() as cur:
for table in ("features", "properties"):
cur.execute("SHOW TABLES LIKE %s", (table,))
if cur.fetchone() is None:
raise RuntimeError(f"required table missing: {table}")
cur.execute("SELECT COUNT(*) FROM features")
self.metrics["source_feature_count"] = cur.fetchone()[0]
cur.execute("SELECT COUNT(*) FROM properties")
self.metrics["source_property_count"] = cur.fetchone()[0]
cur.execute("SELECT COUNT(*) FROM properties WHERE k='at'")
self.metrics["source_at_count"] = cur.fetchone()[0]
print(
"Validated inputs:",
self.metrics["source_feature_count"],
"features,",
self.metrics["source_property_count"],
"properties,",
self.metrics["source_at_count"],
"at rows",
)
def parse_at_attributes(self) -> None:
print("Parsing at attributes...")
parsed_rows = 0
source_rows = 0
parse_errors = 0
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
newline="",
suffix=".tsv",
delete=False,
dir="/tmp",
) as temp_file:
temp_path = Path(temp_file.name)
writer = csv.writer(
temp_file,
delimiter="\t",
quotechar='"',
escapechar="\\",
lineterminator="\n",
)
with self.connect(streaming=True) as conn:
with conn.cursor() as cur:
cur.execute("SELECT feature_id, v FROM properties WHERE k='at'")
for feature_id, raw_value in cur:
source_rows += 1
try:
pairs = json.loads(raw_value)
except json.JSONDecodeError:
parse_errors += 1
continue
if not isinstance(pairs, list):
parse_errors += 1
continue
for pair in pairs:
if not isinstance(pair, list) or len(pair) != 2:
parse_errors += 1
continue
key, value = pair
writer.writerow(
(
int(feature_id),
"" if key is None else str(key),
"" if value is None else str(value),
)
)
parsed_rows += 1
with self.connect(local_infile=True) as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS at_attributes")
cur.execute(
"""
CREATE TABLE at_attributes (
feature_id BIGINT NOT NULL,
k VARCHAR(100) NOT NULL,
v TEXT NULL,
KEY idx_feature_id (feature_id),
KEY idx_k (k),
KEY idx_feature_k (feature_id, k)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
"""
LOAD DATA LOCAL INFILE %s
INTO TABLE at_attributes
CHARACTER SET utf8mb4
FIELDS TERMINATED BY '\t'
ENCLOSED BY '"'
ESCAPED BY '\\\\'
LINES TERMINATED BY '\n'
(feature_id, k, v)
""",
(str(temp_path),),
)
temp_path.unlink(missing_ok=True)
self.metrics["at_source_rows"] = source_rows
self.metrics["at_attribute_rows"] = parsed_rows
self.metrics["at_parse_errors"] = parse_errors
print(
"Parsed at attributes:",
source_rows,
"source rows ->",
parsed_rows,
"attribute rows,",
parse_errors,
"parse errors",
)
def build_feature_semantic(self) -> None:
print("Building feature_semantic...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS feature_semantic")
cur.execute(
"""
CREATE TABLE feature_semantic AS
SELECT
f.id AS feature_id,
f.z,
f.x,
f.y,
f.vt_layer,
f.geom_type,
MAX(CASE WHEN a.k='分類' THEN a.v END) AS class_name,
MAX(CASE WHEN a.k='形状分類' THEN a.v END) AS shape_name,
MAX(CASE WHEN a.k='レイヤ' THEN a.v END) AS layer_name
FROM features f
LEFT JOIN at_attributes a
ON f.id = a.feature_id
GROUP BY
f.id,
f.z,
f.x,
f.y,
f.vt_layer,
f.geom_type
"""
)
cur.execute("ALTER TABLE feature_semantic ADD PRIMARY KEY (feature_id)")
cur.execute("ALTER TABLE feature_semantic ADD KEY idx_vt_layer (vt_layer)")
cur.execute("ALTER TABLE feature_semantic ADD KEY idx_class_name (class_name(100))")
cur.execute("ALTER TABLE feature_semantic ADD KEY idx_geom_type (geom_type)")
cur.execute("SELECT COUNT(*) FROM feature_semantic")
self.metrics["feature_semantic_rows"] = cur.fetchone()[0]
def build_object_catalog(self) -> None:
print("Building object_catalog...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS object_catalog")
cur.execute(
"""
CREATE TABLE object_catalog AS
SELECT
layer_name,
class_name,
shape_name,
vt_layer,
geom_type,
COUNT(*) AS feature_count
FROM feature_semantic
GROUP BY
layer_name,
class_name,
shape_name,
vt_layer,
geom_type
"""
)
cur.execute("ALTER TABLE object_catalog ADD KEY idx_class_name (class_name(100))")
cur.execute("ALTER TABLE object_catalog ADD KEY idx_vt_layer (vt_layer)")
cur.execute("SELECT COUNT(*) FROM object_catalog")
self.metrics["object_catalog_rows"] = cur.fetchone()[0]
def build_geometry_consistency(self) -> None:
print("Building geometry_consistency...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS geometry_consistency")
cur.execute(
"""
CREATE TABLE geometry_consistency AS
SELECT
class_name,
geom_type,
COUNT(*) AS feature_count
FROM feature_semantic
GROUP BY class_name, geom_type
"""
)
cur.execute("ALTER TABLE geometry_consistency ADD KEY idx_class_name (class_name(100))")
cur.execute("SELECT COUNT(*) FROM geometry_consistency")
self.metrics["geometry_consistency_rows"] = cur.fetchone()[0]
def build_object_type_candidates(self) -> None:
print("Building object_type_candidates...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS object_type_candidates")
cur.execute(
"""
CREATE TABLE object_type_candidates AS
SELECT
feature_id,
CASE
WHEN class_name IS NOT NULL AND class_name <> '' THEN 'class_name'
WHEN shape_name IS NOT NULL AND shape_name <> '' THEN 'shape_name'
ELSE 'vt_layer'
END AS object_type_source,
COALESCE(NULLIF(class_name, ''), NULLIF(shape_name, ''), vt_layer) AS object_type,
vt_layer,
geom_type
FROM feature_semantic
"""
)
cur.execute("ALTER TABLE object_type_candidates ADD PRIMARY KEY (feature_id)")
cur.execute("ALTER TABLE object_type_candidates ADD KEY idx_object_type (object_type(100))")
cur.execute("ALTER TABLE object_type_candidates ADD KEY idx_vt_layer (vt_layer)")
cur.execute("SELECT COUNT(*) FROM object_type_candidates")
self.metrics["object_type_candidate_rows"] = cur.fetchone()[0]
def build_object_type_stats(self) -> None:
print("Building object_type_stats...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS object_type_stats")
cur.execute(
"""
CREATE TABLE object_type_stats AS
SELECT
object_type,
geom_type,
COUNT(*) AS feature_count
FROM object_type_candidates
GROUP BY object_type, geom_type
"""
)
cur.execute("ALTER TABLE object_type_stats ADD KEY idx_object_type (object_type(100))")
cur.execute("SELECT COUNT(*) FROM object_type_stats")
self.metrics["object_type_stats_rows"] = cur.fetchone()[0]
cur.execute("SELECT COUNT(DISTINCT object_type) FROM object_type_candidates")
self.metrics["distinct_object_types"] = cur.fetchone()[0]
def build_style_layers(self) -> None:
print("Building style_layers from style.json...")
style_data = json.loads(STYLE_JSON_PATH.read_text(encoding="utf-8"))
rows = []
for layer in style_data.get("layers", []):
paint = layer.get("paint", {})
layout = layer.get("layout", {})
rows.append(
(
layer.get("id"),
layer.get("source-layer"),
self.stringify_style_value(layout.get("icon-image")),
self.stringify_style_value(paint.get("line-color")),
self.stringify_style_value(paint.get("fill-color")),
layer.get("type"),
)
)
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS style_layers")
cur.execute(
"""
CREATE TABLE style_layers (
layer_id VARCHAR(200) NOT NULL,
source_layer VARCHAR(200) NULL,
icon VARCHAR(2000) NULL,
line_color VARCHAR(2000) NULL,
fill_color VARCHAR(2000) NULL,
layer_type VARCHAR(50) NULL,
KEY idx_source_layer (source_layer),
KEY idx_layer_id (layer_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.executemany(
"""
INSERT INTO style_layers
(layer_id, source_layer, icon, line_color, fill_color, layer_type)
VALUES (%s, %s, %s, %s, %s, %s)
""",
rows,
)
self.metrics["style_layer_rows"] = len(rows)
def build_style_mapping(self) -> None:
print("Building style_mapping...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS style_mapping")
cur.execute(
"""
CREATE TABLE style_mapping AS
SELECT
s.layer_id,
s.source_layer,
s.layer_type,
s.icon,
s.line_color,
s.fill_color,
o.object_type,
o.geom_type,
COUNT(*) AS feature_count
FROM style_layers s
JOIN object_type_candidates o
ON s.source_layer = o.object_type
GROUP BY
s.layer_id,
s.source_layer,
s.layer_type,
s.icon,
s.line_color,
s.fill_color,
o.object_type,
o.geom_type
"""
)
cur.execute("ALTER TABLE style_mapping ADD KEY idx_object_type (object_type(100))")
cur.execute("SELECT COUNT(*) FROM style_mapping")
self.metrics["style_mapping_rows"] = cur.fetchone()[0]
def build_tile_density(self) -> None:
print("Building tile_density, tile_layer_density and tile_density_top100...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS tile_density")
cur.execute(
"""
CREATE TABLE tile_density AS
SELECT
f.z,
f.x,
f.y,
COUNT(DISTINCT f.id) AS feature_count
FROM features AS f
GROUP BY
f.z,
f.x,
f.y
"""
)
cur.execute("ALTER TABLE tile_density ADD KEY idx_tile (z, x, y)")
cur.execute("ALTER TABLE tile_density ADD KEY idx_feature_count (feature_count)")
cur.execute("DROP TABLE IF EXISTS tile_layer_density")
cur.execute(
"""
CREATE TABLE tile_layer_density AS
SELECT
f.z,
f.x,
f.y,
f.vt_layer,
COUNT(DISTINCT f.id) AS feature_count
FROM features AS f
GROUP BY
f.z,
f.x,
f.y,
f.vt_layer
"""
)
cur.execute("ALTER TABLE tile_layer_density ADD KEY idx_tile (z, x, y)")
cur.execute("ALTER TABLE tile_layer_density ADD KEY idx_layer (vt_layer)")
cur.execute("ALTER TABLE tile_layer_density ADD KEY idx_feature_count (feature_count)")
cur.execute("DROP TABLE IF EXISTS tile_density_top100")
cur.execute(
"""
CREATE TABLE tile_density_top100 AS
SELECT *
FROM tile_density
ORDER BY feature_count DESC
LIMIT 100
"""
)
cur.execute("SELECT COUNT(*) FROM tile_density")
self.metrics["tile_density_rows"] = cur.fetchone()[0]
cur.execute("SELECT COUNT(*) FROM tile_layer_density")
self.metrics["tile_layer_density_rows"] = cur.fetchone()[0]
def build_anomaly_tables(self) -> None:
print("Building anomaly tables...")
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS anomaly_navigation_geom")
cur.execute(
"""
CREATE TABLE anomaly_navigation_geom AS
SELECT *
FROM feature_semantic
WHERE class_name='灯台'
AND geom_type <> 'Point'
"""
)
cur.execute("DROP TABLE IF EXISTS anomaly_reef_geom")
cur.execute(
"""
CREATE TABLE anomaly_reef_geom AS
SELECT *
FROM feature_semantic
WHERE class_name='魚礁'
AND geom_type NOT IN ('Point', 'Polygon')
"""
)
cur.execute("SELECT COUNT(*) FROM anomaly_navigation_geom")
self.metrics["anomaly_navigation_geom_rows"] = cur.fetchone()[0]
cur.execute("SELECT COUNT(*) FROM anomaly_reef_geom")
self.metrics["anomaly_reef_geom_rows"] = cur.fetchone()[0]
def generate_report(self) -> None:
print("Generating navsea_classification_report.md...")
total_features = self.scalar("SELECT COUNT(*) FROM features")
total_object_types = self.scalar("SELECT COUNT(DISTINCT object_type) FROM object_type_candidates")
top_object_types = self.fetchall(
"""
SELECT object_type, geom_type, feature_count
FROM object_type_stats
ORDER BY feature_count DESC
LIMIT 50
"""
)
geometry_consistency = self.fetchall(
"""
SELECT class_name, geom_type, feature_count
FROM geometry_consistency
ORDER BY feature_count DESC
LIMIT 100
"""
)
style_mapping = self.fetchall(
"""
SELECT layer_id, source_layer, object_type, geom_type, feature_count
FROM style_mapping
ORDER BY feature_count DESC
LIMIT 100
"""
)
tile_density = self.fetchall(
"""
SELECT z, x, y, feature_count
FROM tile_density_top100
ORDER BY feature_count DESC, z, x, y
"""
)
source_breakdown = self.fetchall(
"""
SELECT object_type_source, COUNT(*) AS feature_count
FROM object_type_candidates
GROUP BY object_type_source
ORDER BY feature_count DESC
"""
)
lines = [
"# NavSea Classification Report",
"",
"## Dataset Summary",
"",
f"- Total features: {total_features}",
f"- Total properties: {self.metrics['source_property_count']}",
f"- Total `at` rows: {self.metrics['source_at_count']}",
f"- Parsed `at` attribute rows: {self.metrics['at_attribute_rows']}",
f"- `at` parse errors: {self.metrics['at_parse_errors']}",
f"- Total object types: {total_object_types}",
f"- Style layers parsed: {self.metrics['style_layer_rows']}",
f"- Style mappings found: {self.metrics['style_mapping_rows']}",
"",
"## Object Type Source Breakdown",
"",
self.render_table(
["object_type_source", "feature_count"],
source_breakdown,
),
"",
"## Top Object Types",
"",
self.render_table(["object_type", "geom_type", "feature_count"], top_object_types),
"",
"## Geometry Consistency",
"",
self.render_table(["class_name", "geom_type", "feature_count"], geometry_consistency),
"",
"## Style Mapping",
"",
self.render_table(
["layer_id", "source_layer", "object_type", "geom_type", "feature_count"],
style_mapping,
),
"",
"## Tile Density Top 100",
"",
self.render_table(["z", "x", "y", "feature_count"], tile_density),
"",
"## Spatial Sanity Checks",
"",
f"- anomaly_navigation_geom rows: {self.metrics['anomaly_navigation_geom_rows']}",
f"- anomaly_reef_geom rows: {self.metrics['anomaly_reef_geom_rows']}",
"",
]
REPORT_PATH.write_text("\n".join(lines), encoding="utf-8")
def scalar(self, sql: str):
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute(sql)
return cur.fetchone()[0]
def fetchall(self, sql: str):
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute(sql)
return cur.fetchall()
@staticmethod
def stringify_style_value(value):
if value is None:
return None
if isinstance(value, str):
return value
return json.dumps(value, ensure_ascii=False)
@staticmethod
def render_table(headers, rows) -> str:
lines = [
"| " + " | ".join(headers) + " |",
"| " + " | ".join(["---"] * len(headers)) + " |",
]
for row in rows:
lines.append("| " + " | ".join("" if value is None else str(value) for value in row) + " |")
return "\n".join(lines)
def main() -> None:
NavSeaAudit(DbConfig()).run()
if __name__ == "__main__":
main()

337
navsea_classification_report.md Normal file
View File

@@ -0,0 +1,337 @@
# NavSea Classification Report
## Dataset Summary
- Total features: 6685117
- Total properties: 23659423
- Total `at` rows: 2823055
- Parsed `at` attribute rows: 6614767
- `at` parse errors: 0
- Total object types: 177
- Style layers parsed: 87
- Style mappings found: 42
## Object Type Source Breakdown
| object_type_source | feature_count |
| --- | --- |
| vt_layer | 3862062 |
| class_name | 2784535 |
| shape_name | 38520 |
## Top Object Types
| object_type | geom_type | feature_count |
| --- | --- | --- |
| L海底地形 | LineString | 1866386 |
| 道路 | LineString | 1256228 |
| P基本線ククリ | LineString | 1074484 |
| 等深線 | LineString | 434442 |
| P穴 | Polygon | 361754 |
| L海底地形 | MultiLineString | 208786 |
| P陸域 | Polygon | 193114 |
| 干潮帯 | Polygon | 114265 |
| 底質 | Point | 99863 |
| 5-10m | Polygon | 66271 |
| 防波堤 | Polygon | 62117 |
| 2-5m | Polygon | 59238 |
| P危険界ククリ | LineString | 51743 |
| 0-2m | Polygon | 47385 |
| p地名 | Point | 36247 |
| p地名陸 | Point | 35388 |
| 10-20m | Polygon | 34190 |
| 浅所危険界 | Polygon | 29950 |
| ビル | Polygon | 26813 |
| 0-5m | Polygon | 26366 |
| 橋 (水門) | Polygon | 25708 |
| 一本線で表わす桟橋 | LineString | 25497 |
| 険悪物 | Point | 22314 |
| 5-10m | MultiPolygon | 21217 |
| 区画漁業 | Polygon | 18759 |
| 浮施設・桟橋 | LineString | 16873 |
| 1000-2000m | Polygon | 15663 |
| 灯 (Lt) | Point | 14936 |
| 0-5m | MultiPolygon | 14841 |
| 干潮帯 | MultiPolygon | 14617 |
| 20-100m | Polygon | 14605 |
| 2000-3000m | Polygon | 14540 |
| 100-200m | Polygon | 13018 |
| 浮施設・桟橋 | Polygon | 12763 |
| 海底線 (電信電話) | LineString | 12712 |
| 魚礁 | Point | 12477 |
| 500-1000m | Polygon | 12363 |
| 3000-4000m | Polygon | 11683 |
| 5000-6000m | Polygon | 11147 |
| 200-500m | Polygon | 10932 |
| 塔、やぐら、風車 | Point | 10818 |
| 暗岩 | Point | 10366 |
| 2-5m | MultiPolygon | 10183 |
| 4000-5000m | Polygon | 10099 |
| 10-20m | MultiPolygon | 9881 |
| 山頂 | Point | 9686 |
| 一本線で表わす海上バース | LineString | 9366 |
| 河川域 | Polygon | 9081 |
| タンク | Polygon | 8765 |
| P投錨注意障害物ククリ | LineString | 8465 |
## Geometry Consistency
| class_name | geom_type | feature_count |
| --- | --- | --- |
| | LineString | 3003658 |
| 道路 | LineString | 1256228 |
| | Polygon | 554868 |
| 等深線 | LineString | 434442 |
| | MultiLineString | 215850 |
| | Point | 115202 |
| 干潮帯 | Polygon | 114265 |
| 底質 | Point | 99863 |
| 5-10m | Polygon | 66271 |
| 防波堤 | Polygon | 62117 |
| 2-5m | Polygon | 59238 |
| 0-2m | Polygon | 47385 |
| 10-20m | Polygon | 34190 |
| 浅所危険界 | Polygon | 29950 |
| ビル | Polygon | 26813 |
| 0-5m | Polygon | 26366 |
| 橋 (水門) | Polygon | 25708 |
| 一本線で表わす桟橋 | LineString | 25497 |
| 険悪物 | Point | 22314 |
| 5-10m | MultiPolygon | 21217 |
| 区画漁業 | Polygon | 18759 |
| 浮施設・桟橋 | LineString | 16873 |
| 1000-2000m | Polygon | 15663 |
| 0-5m | MultiPolygon | 14841 |
| 干潮帯 | MultiPolygon | 14617 |
| 20-100m | Polygon | 14605 |
| 2000-3000m | Polygon | 14540 |
| 100-200m | Polygon | 13018 |
| 浮施設・桟橋 | Polygon | 12763 |
| 海底線 (電信電話) | LineString | 12712 |
| 魚礁 | Point | 12477 |
| 500-1000m | Polygon | 12363 |
| 3000-4000m | Polygon | 11683 |
| 5000-6000m | Polygon | 11147 |
| | MultiPolygon | 11004 |
| 200-500m | Polygon | 10932 |
| 塔、やぐら、風車 | Point | 10818 |
| 暗岩 | Point | 10366 |
| 2-5m | MultiPolygon | 10183 |
| 4000-5000m | Polygon | 10099 |
| 10-20m | MultiPolygon | 9881 |
| 山頂 | Point | 9686 |
| 一本線で表わす海上バース | LineString | 9366 |
| 河川域 | Polygon | 9081 |
| タンク | Polygon | 8765 |
| 煙突 | Point | 8123 |
| 等深線 | MultiLineString | 6736 |
| 0-2m | MultiPolygon | 6312 |
| 概略等深線 | LineString | 5497 |
| 0-10m | Polygon | 5435 |
| 6000-7000m | Polygon | 5016 |
| 険悪物 | Polygon | 4519 |
| 未測海域 | Polygon | 4190 |
| 海草 | Point | 4175 |
| 魚礁 | Polygon | 4118 |
| 潜提 | Polygon | 3780 |
| 定置漁業 | Polygon | 3566 |
| 洗岩 | Point | 3323 |
| 漁港 | Point | 3169 |
| 湖沼等深線 | LineString | 3124 |
| 防波堤 | MultiPolygon | 3032 |
| 漁業協同組合 | Point | 2914 |
| 河川域 | MultiPolygon | 2714 |
| 撤去跡 | Polygon | 2407 |
| サンドウェーブ | Point | 2356 |
| 7000-8000m | Polygon | 2306 |
| 急潮・波紋・激潮 | Point | 2206 |
| ポンツーン、パイル、杭 | Point | 2119 |
| 0-10m | MultiPolygon | 2082 |
| 道路 | MultiLineString | 2034 |
| 20-100m | MultiPolygon | 1867 |
| 港則法による境界 | LineString | 1665 |
| 浅所危険界 | MultiPolygon | 1587 |
| 500-1000m | MultiPolygon | 1461 |
| 2-10m | Polygon | 1367 |
| 1000-2000m | MultiPolygon | 1364 |
| 干出岩 | Point | 1178 |
| 2000-3000m | MultiPolygon | 1141 |
| 100-200m | MultiPolygon | 1135 |
| 0-20m | Polygon | 1111 |
| 200-500m | MultiPolygon | 1100 |
| 8000-9000m | Polygon | 1100 |
| 一本線で表わす潜提 | LineString | 1096 |
| 架空線、送電線 | LineString | 1028 |
| 陸上水域 | Polygon | 971 |
| 湖沼域 | Polygon | 870 |
| 5-20m | Polygon | 838 |
| 孤立危険物 | Point | 824 |
| 全沈没船 (危険なし) | Point | 811 |
| 3000-4000m | MultiPolygon | 801 |
| 一本線で表わす撤去跡 | LineString | 780 |
| ドルフィン | Point | 756 |
| 湖沼等深線 | MultiLineString | 749 |
| 5000-6000m | MultiPolygon | 737 |
| 海底線 (電力) | LineString | 735 |
| 4000-5000m | MultiPolygon | 730 |
| 養殖場 | Polygon | 716 |
| 海底設置物、放水口、取水口 | Point | 715 |
| 海底輸送管 (水) | LineString | 649 |
| 一般港湾 (港則法区域、重要港湾) | Point | 610 |
## Style Mapping
| layer_id | source_layer | object_type | geom_type | feature_count |
| --- | --- | --- | --- | --- |
| #b#C000 | L海底地形 | L海底地形 | LineString | 1866386 |
| #b#海底地形 | L海底地形 | L海底地形 | LineString | 1866386 |
| #b#d2#d5#d10#P基本線ククリ | P基本線ククリ | P基本線ククリ | LineString | 1074484 |
| #b#d2#d5#d10#P穴 | P穴 | P穴 | Polygon | 361754 |
| #b#C000 | L海底地形 | L海底地形 | MultiLineString | 208786 |
| #b#海底地形 | L海底地形 | L海底地形 | MultiLineString | 208786 |
| #b#d2#d5#d10#land#P100 | P陸域 | P陸域 | Polygon | 193114 |
| #b#d2#d5#d10#P危険界ククリ | P危険界ククリ | P危険界ククリ | LineString | 51743 |
| #b#d2#d5#d10#C800#EN | p地名 | p地名 | Point | 36247 |
| #b#d2#d5#d10#C800#JA | p地名 | p地名 | Point | 36247 |
| #b#d2#d5#d10#land#C800#EN | p地名陸 | p地名陸 | Point | 35388 |
| #b#d2#d5#d10#land#C800#JA | p地名陸 | p地名陸 | Point | 35388 |
| #b#d2#d5#d10#L4#P投錨注意障害物ククリ | P投錨注意障害物ククリ | P投錨注意障害物ククリ | LineString | 8465 |
| #b#d2#d5#d10#P基本線ククリ | P基本線ククリ | P基本線ククリ | MultiLineString | 6296 |
| #b#d2#d5#d10#land#P100 | P陸域 | P陸域 | MultiPolygon | 5541 |
| #b#d2#d5#d10#P穴 | P穴 | P穴 | MultiPolygon | 5463 |
| #b#d2#d5#d10#at#P4#P投錨注意障害物 | P投錨注意障害物 | p投錨注意障害物 | Point | 3926 |
| #b#d2#d5#d10#at#S4#S投錨注意障害物 | p投錨注意障害物 | p投錨注意障害物 | Point | 3926 |
| #b#d2#d5#d10#C600#名称 | p高さ制限 | p高さ制限 | Point | 858 |
| #b#d2#d5#d10#C600#高さ | p高さ制限 | p高さ制限 | Point | 858 |
| #b#d2#d5#d10#BK#P航路ククリ | P航路ククリ | P航路ククリ | LineString | 755 |
| #b#d2#d5#d10#BS#P754ククリ | P754ククリ | P754ククリ | LineString | 730 |
| #b#d2#d5#d10#BS#P施設・境界線等ククリ | P施設・境界線等ククリ | P施設・境界線等ククリ | LineString | 521 |
| #b#d2#d5#d10#L4#P投錨注意障害物ククリ | P投錨注意障害物ククリ | P投錨注意障害物ククリ | MultiLineString | 278 |
| #b#d2#d5#d10#BB#P錨泊地等ククリ | P錨泊地等ククリ | P錨泊地等ククリ | LineString | 240 |
| #b#d2#d5#d10#at#BB#P錨泊地等 | P錨泊地等 | p錨泊地等 | Point | 221 |
| #b#d2#d5#d10#at#BB#S720 | p錨泊地等 | p錨泊地等 | Point | 221 |
| #b#d2#d5#d10#P危険界ククリ | P危険界ククリ | P危険界ククリ | MultiLineString | 219 |
| #b#d2#d5#d10#L4#P航行危険障害物ククリ | P航行危険障害物ククリ | P航行危険障害物ククリ | LineString | 191 |
| #b#d2#d5#d10#BK#P航路ククリ | P航路ククリ | P航路ククリ | MultiLineString | 109 |
| #b#d2#d5#d10#BS#P誘導線ククリ | P誘導線ククリ | P誘導線ククリ | LineString | 101 |
| #b#d2#d5#d10#BB#P錨泊地等ククリ | P錨泊地等ククリ | P錨泊地等ククリ | MultiLineString | 57 |
| #b#d2#d5#d10#BS#P施設・境界線等ククリ | P施設・境界線等ククリ | P施設・境界線等ククリ | MultiLineString | 45 |
| #b#d2#d5#d10#BS#P754ククリ | P754ククリ | P754ククリ | MultiLineString | 41 |
| #b#d2#d5#d10#at#P4#P航行危険障害物 | P航行危険障害物 | p航行危険障害物 | Point | 24 |
| #b#d2#d5#d10#at#S4#S航行危険障害物 | p航行危険障害物 | p航行危険障害物 | Point | 24 |
| #b#d2#d5#d10#BB#P721ククリ | P721ククリ | P721ククリ | LineString | 24 |
| #b#d2#d5#d10#at#BK#S760 | p航路境界等 | p航路境界等 | Point | 18 |
| #b#d2#d5#d10#BS#P730ククリ | P730ククリ | P730ククリ | LineString | 18 |
| #b#d2#d5#d10#BS#P730ククリ | P730ククリ | P730ククリ | MultiLineString | 9 |
| #b#d2#d5#d10#L4#P航行危険障害物ククリ | P航行危険障害物ククリ | P航行危険障害物ククリ | MultiLineString | 7 |
| #b#d2#d5#d10#BB#P721ククリ | P721ククリ | P721ククリ | MultiLineString | 3 |
## Tile Density Top 100
| z | x | y | feature_count |
| --- | --- | --- | --- |
| 7 | 110 | 51 | 32300 |
| 6 | 55 | 25 | 25153 |
| 7 | 111 | 51 | 21202 |
| 5 | 27 | 12 | 20867 |
| 7 | 111 | 50 | 16595 |
| 8 | 222 | 102 | 16494 |
| 11 | 1750 | 869 | 14203 |
| 8 | 220 | 103 | 13548 |
| 11 | 1794 | 814 | 13256 |
| 5 | 28 | 12 | 13157 |
| 11 | 1794 | 813 | 13068 |
| 11 | 1793 | 813 | 12948 |
| 11 | 1759 | 815 | 12137 |
| 11 | 1777 | 815 | 11981 |
| 11 | 1760 | 813 | 11881 |
| 11 | 1736 | 878 | 11198 |
| 11 | 1802 | 810 | 10984 |
| 7 | 109 | 54 | 10901 |
| 7 | 112 | 51 | 10530 |
| 11 | 1815 | 790 | 10515 |
| 11 | 1785 | 815 | 10512 |
| 7 | 112 | 50 | 10308 |
| 8 | 220 | 102 | 10217 |
| 6 | 56 | 25 | 10198 |
| 8 | 218 | 108 | 9803 |
| 11 | 1781 | 815 | 9684 |
| 7 | 109 | 51 | 9655 |
| 11 | 1792 | 813 | 9357 |
| 11 | 1845 | 752 | 9303 |
| 9 | 444 | 204 | 8858 |
| 11 | 1761 | 823 | 8657 |
| 11 | 1780 | 817 | 8365 |
| 11 | 1829 | 783 | 8304 |
| 11 | 1818 | 809 | 8031 |
| 11 | 1802 | 816 | 8004 |
| 11 | 1751 | 868 | 7873 |
| 7 | 113 | 50 | 7820 |
| 8 | 223 | 101 | 7614 |
| 11 | 1786 | 815 | 7600 |
| 7 | 114 | 48 | 7588 |
| 7 | 110 | 52 | 7577 |
| 11 | 1831 | 780 | 7476 |
| 7 | 109 | 53 | 7472 |
| 11 | 1768 | 818 | 7452 |
| 8 | 228 | 97 | 7437 |
| 11 | 1761 | 824 | 7433 |
| 11 | 1729 | 881 | 7388 |
| 9 | 440 | 205 | 7366 |
| 11 | 1796 | 821 | 7328 |
| 11 | 1824 | 761 | 7293 |
| 11 | 1763 | 830 | 7277 |
| 5 | 27 | 13 | 7181 |
| 11 | 1781 | 816 | 7147 |
| 8 | 224 | 102 | 7108 |
| 8 | 219 | 103 | 7023 |
| 8 | 221 | 102 | 7018 |
| 11 | 1826 | 787 | 6960 |
| 11 | 1783 | 821 | 6912 |
| 12 | 3659 | 1567 | 6889 |
| 11 | 1777 | 823 | 6873 |
| 11 | 1801 | 811 | 6862 |
| 11 | 1777 | 816 | 6848 |
| 11 | 1791 | 813 | 6773 |
| 11 | 1818 | 808 | 6717 |
| 9 | 437 | 217 | 6661 |
| 12 | 3520 | 1626 | 6639 |
| 11 | 1830 | 782 | 6622 |
| 11 | 1790 | 812 | 6591 |
| 11 | 1784 | 814 | 6566 |
| 11 | 1757 | 826 | 6563 |
| 11 | 1778 | 816 | 6550 |
| 11 | 1803 | 798 | 6488 |
| 11 | 1781 | 807 | 6424 |
| 12 | 3587 | 1626 | 6414 |
| 11 | 1782 | 822 | 6394 |
| 11 | 1786 | 814 | 6371 |
| 11 | 1819 | 806 | 6289 |
| 11 | 1829 | 755 | 6237 |
| 11 | 1780 | 816 | 6231 |
| 11 | 1800 | 816 | 6212 |
| 11 | 1798 | 803 | 6184 |
| 12 | 3586 | 1626 | 6156 |
| 9 | 439 | 206 | 6111 |
| 8 | 219 | 107 | 6084 |
| 11 | 1778 | 818 | 6074 |
| 11 | 1757 | 825 | 6063 |
| 11 | 1804 | 799 | 6057 |
| 11 | 1750 | 868 | 5961 |
| 12 | 3500 | 1738 | 5955 |
| 7 | 114 | 49 | 5942 |
| 11 | 1797 | 820 | 5918 |
| 12 | 3589 | 1628 | 5917 |
| 12 | 3555 | 1631 | 5852 |
| 9 | 457 | 195 | 5849 |
| 11 | 1824 | 769 | 5796 |
| 12 | 3665 | 1727 | 5740 |
| 11 | 1793 | 815 | 5727 |
| 12 | 3649 | 1523 | 5715 |
| 12 | 3588 | 1626 | 5697 |
| 11 | 1832 | 863 | 5684 |
## Spatial Sanity Checks
- anomaly_navigation_geom rows: 0
- anomaly_reef_geom rows: 13

434
navsea_detection_pipeline.py Normal file
View File

@@ -0,0 +1,434 @@
from __future__ import annotations
import os
from dataclasses import dataclass
import pymysql
@dataclass(frozen=True)
class DbConfig:
host: str = os.getenv("NAVSEA_DB_HOST", "localhost")
port: int = int(os.getenv("NAVSEA_DB_PORT", "3306"))
user: str = os.getenv("NAVSEA_DB_USER", "root")
password: str = os.getenv("NAVSEA_DB_PASSWORD", "2chi9ks2")
database: str = os.getenv("NAVSEA_DB_NAME", "pbf_analysis")
unix_socket: str | None = os.getenv("NAVSEA_DB_SOCKET", "/tmp/mysql.sock")
FAMILY_DEFAULT_CAPABILITIES = {
"hazard": {"collision"},
"navigation_aid": {"navigation_mark"},
"bathymetry": {"depth_reference"},
"seabed": {"bottom_reference"},
"fishery": {"entangle"},
"route": {"route_reference"},
"boundary": {"boundary_reference"},
"anchorage": {"anchorage_reference"},
"restricted_area": {"boundary_reference"},
"infrastructure": {"structure_reference"},
"utility": {"utility_hazard"},
"landmark": {"landmark_reference"},
"place": {"place_reference"},
"water_area": {"water_reference"},
"monitoring": {"traffic_monitoring"},
"overlay_support": {"render_support"},
"unknown": {"review_required"},
}
class DetectionPipeline:
def __init__(self, config: DbConfig) -> None:
self.config = config
def connect(self):
kwargs = {
"host": self.config.host,
"port": self.config.port,
"user": self.config.user,
"password": self.config.password,
"database": self.config.database,
"charset": "utf8mb4",
"autocommit": False,
}
if self.config.unix_socket and self.config.host in {"localhost", "127.0.0.1"}:
kwargs["unix_socket"] = self.config.unix_socket
return pymysql.connect(**kwargs)
def run(self) -> None:
with self.connect() as conn:
with conn.cursor() as cur:
object_types = self.fetch_object_types(cur)
taxonomy_rows = [self.classify_object_type(name) for name in object_types]
capability_rows = self.build_capabilities(taxonomy_rows)
self.create_taxonomy_table(cur, taxonomy_rows)
self.create_capabilities_table(cur, capability_rows)
self.create_detection_view(cur)
self.create_capability_views(cur)
self.ensure_query_indexes(cur)
conn.commit()
print(
"Built navsea_object_taxonomy, navsea_object_capabilities, "
"navsea_detection_objects, capability views and supporting query indexes"
)
@staticmethod
def fetch_object_types(cur) -> list[str]:
cur.execute(
"""
SELECT canonical_object_type
FROM canonical_object_rules
ORDER BY canonical_object_type
"""
)
return [row[0] for row in cur.fetchall()]
@staticmethod
def is_depth_band(name: str) -> bool:
compact = name.replace(" ", "")
return compact.endswith("m") or "m以深" in compact
@staticmethod
def classify_object_type(name: str) -> tuple[str, str, str, str]:
if name.startswith(("P", "L", "p")) and (
name.endswith("ククリ")
or name in {
"P穴",
"P陸域",
"P危険界ククリ",
"P投錨注意障害物ククリ",
"P施設・境界線等ククリ",
"P航路ククリ",
"P錨泊地等ククリ",
"P754ククリ",
"P721ククリ",
"P730ククリ",
"L海底地形",
"p地名",
"p地名陸",
"p高さ制限",
}
):
return (
name,
"overlay_support",
"render_support",
"保留原始 source_layer 的叠加支持对象，不作为独立语义实体替换原始渲染层。",
)
if DetectionPipeline.is_depth_band(name):
return (name, "bathymetry", "depth_zone", "水深分带对象，用于浅滩、深度范围和搁浅风险分析。")
if any(token in name for token in ("等深線",)):
return (name, "bathymetry", "depth_contour", "等深线对象，用于水深变化与航线安全分析。")
if any(token in name for token in ("底質", "海底地形", "サンドウェーブ", "海底火山")):
return (name, "seabed", "seabed_feature", "海底形态或底质对象，用于海底环境识别。")
if any(
token in name
for token in (
"険悪物",
"障害物",
"危険物",
"危険全沈没船",
"沈船",
"全沈没船",
"沈木",
"暗岩",
"洗岩",
"干出岩",
"水上岩",
"孤立危険物",
"魚礁",
"浅所危険界",
"撤去跡",
"掃海済み",
"サンゴ礁",
)
):
return (name, "hazard", "obstacle", "对航行存在碰撞或搁浅风险的危险物对象。")
if any(
token in name
for token in (
"灯",
"灯台",
"灯標",
"灯浮標",
"浮標",
"立標",
"導灯",
"指向灯",
"V-AIS",
"管制信号所",
)
):
return (name, "navigation_aid", "navigation_mark", "航标、灯标或导助航对象。")
if any(token in name for token in ("航路", "進路矢印", "分離通航方式", "誘導線", "指導線")):
return (name, "route", "route_reference", "航路、导向线或通航组织对象。")
if any(
token in name
for token in (
"境界",
"危険界",
"境界線",
"制限区域",
"専用用途海域",
"航泊禁止区域",
"錨泊禁止区域",
"航空機進入区域",
)
):
return (name, "boundary", "boundary_control", "边界、限制区或规则控制对象。")
if "錨泊" in name or "錨地" in name:
return (name, "anchorage", "anchorage", "锚地或锚泊控制对象。")
if any(token in name for token in ("漁", "養殖場", "漁網", "海草")):
return (name, "fishery", "fishery_area", "渔业、养殖或缠绕风险相关对象。")
if any(
token in name
for token in (
"防波堤",
"潜提",
"桟橋",
"海上バース",
"ドルフィン",
"ポンツーン",
"橋",
"道路",
"ビル",
"タンク",
"煙突",
"塔",
"やぐら",
"ケーソン",
"土砂捨て場",
"石油開発台",
)
):
return (name, "infrastructure", "structure", "港口、岸线或陆上海工构造物对象。")
if any(token in name for token in ("海底線", "輸送管", "送電線", "架空線", "放水口", "取水口")):
return (name, "utility", "utility_line", "海底线缆、管道或公用设施对象。")
if any(token in name for token in ("河川域", "湖沼域", "陸上水域", "干潮帯", "未測海域", "平水境界")):
return (name, "water_area", "water_area", "水域、潮滩或未测区域对象。")
if any(
token in name
for token in (
"その他 (記念碑等)",
"陸上顕著物",
"地名",
"山頂",
"港湾",
"漁港",
"マリーナ",
"フィッシャリーナ",
"海の駅",
"漁業協同組合",
"海事関係署",
"税関",
"パイロットステーション",
)
):
return (name, "place", "place_label", "地名、显著地物或港区服务设施类参考对象。")
if any(token in name for token in ("廃棄物捨て場", "貯木場")):
return (name, "infrastructure", "structure", "作业、堆置或弃置区域类设施对象。")
if any(token in name for token in ("無線局", "レーダー局")):
return (name, "monitoring", "monitoring_station", "通信、雷达或监测设施对象。")
if any(token in name for token in ("急潮", "波紋", "激潮", "渦流", "水源方向")):
return (name, "hazard", "current_hazard", "流态、水流或潮流风险相关对象。")
return (name, "unknown", "review_required", "尚未细化归类的对象，需要人工复核。")
@staticmethod
def build_capabilities(
taxonomy_rows: list[tuple[str, str, str, str]]
) -> list[tuple[str, str]]:
capability_rows: set[tuple[str, str]] = set()
for canonical_object_type, object_family, detection_class, _ in taxonomy_rows:
capabilities = set(FAMILY_DEFAULT_CAPABILITIES.get(object_family, {"review_required"}))
if detection_class in {"obstacle"}:
capabilities.add("collision")
if detection_class in {"depth_zone", "depth_contour"}:
capabilities.add("grounding")
if "魚礁" in canonical_object_type or "浅所" in canonical_object_type or "干潮帯" in canonical_object_type:
capabilities.add("grounding")
if any(token in canonical_object_type for token in ("漁", "養殖場", "漁網", "海草")):
capabilities.add("entangle")
if any(token in canonical_object_type for token in ("防波堤",)):
capabilities.add("wave_barrier")
if any(token in canonical_object_type for token in ("架空線", "送電線", "高さ制限")):
capabilities.add("overhead_clearance")
if any(token in canonical_object_type for token in ("航路", "誘導線", "進路矢印", "分離通航方式")):
capabilities.add("route_reference")
if any(token in canonical_object_type for token in ("錨泊", "錨地")):
capabilities.add("anchorage_reference")
if any(token in canonical_object_type for token in ("境界", "区域", "危険界")):
capabilities.add("boundary_reference")
if any(token in canonical_object_type for token in ("海底線", "輸送管")):
capabilities.add("snag_risk")
if any(token in canonical_object_type for token in ("灯", "標", "浮標", "導灯", "V-AIS", "管制信号所")):
capabilities.add("navigation_mark")
if object_family in {"place", "infrastructure", "monitoring"}:
capabilities.add("landmark_reference")
for capability in sorted(capabilities):
capability_rows.add((canonical_object_type, capability))
return sorted(capability_rows)
@staticmethod
def create_taxonomy_table(cur, rows: list[tuple[str, str, str, str]]) -> None:
cur.execute("DROP TABLE IF EXISTS navsea_object_taxonomy")
cur.execute(
"""
CREATE TABLE navsea_object_taxonomy (
canonical_object_type VARCHAR(191) NOT NULL,
object_family VARCHAR(64) NOT NULL,
detection_class VARCHAR(64) NOT NULL,
description TEXT NULL,
PRIMARY KEY (canonical_object_type),
KEY idx_family_class (object_family, detection_class)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.executemany(
"""
INSERT INTO navsea_object_taxonomy (
canonical_object_type,
object_family,
detection_class,
description
)
VALUES (%s, %s, %s, %s)
""",
rows,
)
@staticmethod
def create_capabilities_table(cur, rows: list[tuple[str, str]]) -> None:
cur.execute("DROP TABLE IF EXISTS navsea_object_capabilities")
cur.execute(
"""
CREATE TABLE navsea_object_capabilities (
canonical_object_type VARCHAR(191) NOT NULL,
capability VARCHAR(64) NOT NULL,
PRIMARY KEY (canonical_object_type, capability),
KEY idx_capability (capability, canonical_object_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.executemany(
"""
INSERT INTO navsea_object_capabilities (
canonical_object_type,
capability
)
VALUES (%s, %s)
""",
rows,
)
@staticmethod
def create_detection_view(cur) -> None:
cur.execute("DROP VIEW IF EXISTS navsea_detection_objects")
cur.execute(
"""
CREATE VIEW navsea_detection_objects AS
SELECT
r.feature_id,
CAST(NULL AS CHAR(1)) AS geometry,
r.z,
r.x,
r.y,
r.geom_type,
r.source_layer,
r.class_name,
r.shape_name,
r.layer_name,
t.canonical_object_type,
t.object_family,
t.detection_class,
c.capability,
r.detection_key
FROM pbf_relayer_candidates r
JOIN navsea_object_taxonomy t
ON r.canonical_object_type = t.canonical_object_type
LEFT JOIN navsea_object_capabilities c
ON r.canonical_object_type = c.canonical_object_type
"""
)
@staticmethod
def create_capability_views(cur) -> None:
views = {
"navsea_collision_objects": "collision",
"navsea_grounding_objects": "grounding",
"navsea_entangle_objects": "entangle",
"navsea_navigation_mark_objects": "navigation_mark",
"navsea_route_reference_objects": "route_reference",
"navsea_boundary_reference_objects": "boundary_reference",
}
for view_name, capability in views.items():
cur.execute(f"DROP VIEW IF EXISTS {view_name}")
cur.execute(
f"""
CREATE VIEW {view_name} AS
SELECT *
FROM navsea_detection_objects
WHERE capability = %s
""",
(capability,),
)
@staticmethod
def add_index_if_missing(cur, table: str, index_name: str, ddl: str) -> None:
cur.execute(f"SHOW INDEX FROM {table} WHERE Key_name=%s", (index_name,))
if cur.fetchone() is None:
cur.execute(ddl)
def ensure_query_indexes(self, cur) -> None:
self.add_index_if_missing(
cur,
"pbf_relayer_candidates",
"idx_navsea_tile",
"ALTER TABLE pbf_relayer_candidates ADD KEY idx_navsea_tile (z, x, y)",
)
self.add_index_if_missing(
cur,
"pbf_relayer_candidates",
"idx_navsea_object",
"ALTER TABLE pbf_relayer_candidates ADD KEY idx_navsea_object (canonical_object_type(100), geom_type)",
)
self.add_index_if_missing(
cur,
"pbf_relayer_candidates",
"idx_navsea_detection_key",
"ALTER TABLE pbf_relayer_candidates ADD KEY idx_navsea_detection_key (detection_key(64))",
)
self.add_index_if_missing(
cur,
"pbf_relayer_candidates",
"idx_navsea_source_layer",
"ALTER TABLE pbf_relayer_candidates ADD KEY idx_navsea_source_layer (source_layer(100), geom_type)",
)
def main() -> None:
DetectionPipeline(DbConfig()).run()
if __name__ == "__main__":
main()

292
navsea_geojson_stage_loader.py Normal file
View File

@@ -0,0 +1,292 @@
from __future__ import annotations
import json
import os
import re
import tempfile
from collections import defaultdict
from dataclasses import dataclass
from pathlib import Path
import pymysql
TILE_PATTERN = re.compile(r"^(?P<z>\d+)_(?P<x>\d+)_(?P<y>\d+)\.json$")
@dataclass(frozen=True)
class DbConfig:
host: str = os.getenv("NAVSEA_DB_HOST", "localhost")
port: int = int(os.getenv("NAVSEA_DB_PORT", "3306"))
user: str = os.getenv("NAVSEA_DB_USER", "root")
password: str = os.getenv("NAVSEA_DB_PASSWORD", "2chi9ks2")
database: str = os.getenv("NAVSEA_DB_NAME", "pbf_analysis")
unix_socket: str | None = os.getenv("NAVSEA_DB_SOCKET", "/tmp/mysql.sock")
class GeoJsonStageLoader:
def __init__(
self,
config: DbConfig,
geojson_dir: Path,
stage_table: str = "feature_geometry_stage",
) -> None:
self.config = config
self.geojson_dir = geojson_dir
self.stage_table = stage_table
def connect(self, *, streaming: bool = False, local_infile: bool = False):
kwargs = {
"host": self.config.host,
"port": self.config.port,
"user": self.config.user,
"password": self.config.password,
"database": self.config.database,
"charset": "utf8mb4",
"autocommit": False,
"local_infile": local_infile,
"cursorclass": pymysql.cursors.SSCursor if streaming else pymysql.cursors.Cursor,
}
if self.config.unix_socket and self.config.host in {"localhost", "127.0.0.1"}:
kwargs["unix_socket"] = self.config.unix_socket
return pymysql.connect(**kwargs)
def load(self) -> None:
tile_files = self.list_tile_files()
if not tile_files:
raise RuntimeError(f"no geojson files found in {self.geojson_dir}")
lookup = {tile: path for tile, path in tile_files}
ordered_tiles = [tile for tile, _ in tile_files]
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
newline="",
suffix=".tsv",
delete=False,
dir="/tmp",
) as temp_file:
temp_path = Path(temp_file.name)
self.build_lookup_table()
staged_rows = self.stream_and_stage(lookup, ordered_tiles, temp_file)
with self.connect(local_infile=True) as conn:
with conn.cursor() as cur:
self.create_stage_table(cur)
cur.execute(
f"""
LOAD DATA LOCAL INFILE %s
INTO TABLE {self.stage_table}
CHARACTER SET utf8mb4
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
(feature_id, geometry_wkt)
""",
(str(temp_path),),
)
cur.execute(f"SELECT COUNT(*) FROM {self.stage_table}")
loaded_rows = cur.fetchone()[0]
conn.commit()
temp_path.unlink(missing_ok=True)
print(f"Loaded {loaded_rows} rows into {self.stage_table} from {len(tile_files)} geojson tiles")
if loaded_rows != staged_rows:
raise RuntimeError(f"stage row mismatch: staged={staged_rows}, loaded={loaded_rows}")
def list_tile_files(self) -> list[tuple[tuple[int, int, int], Path]]:
rows = []
for path in self.geojson_dir.glob("*.json"):
match = TILE_PATTERN.match(path.name)
if not match:
continue
rows.append(
(
(int(match.group("z")), int(match.group("x")), int(match.group("y"))),
path,
)
)
rows.sort()
return rows
def stream_and_stage(self, lookup: dict[tuple[int, int, int], Path], ordered_tiles: list[tuple[int, int, int]], temp_file) -> int:
staged_rows = 0
tile_index = 0
current_tile = None
current_rows: list[tuple[int, str, str, str]] = []
with self.connect(streaming=True) as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT
z,
x,
y,
feature_id,
fid,
vt_layer,
geom_type
FROM feature_geojson_lookup FORCE INDEX (idx_tile_order)
ORDER BY z, x, y, feature_id
"""
)
for z, x, y, feature_id, fid, vt_layer, geom_type in cur:
tile = (int(z), int(x), int(y))
if current_tile is None:
current_tile = tile
if tile != current_tile:
staged_rows += self.flush_tile(
current_tile,
current_rows,
lookup,
temp_file,
)
tile_index += 1
if tile_index % 5000 == 0:
print(f"processed {tile_index}/{len(ordered_tiles)} tiles, staged {staged_rows} geometries")
current_tile = tile
current_rows = []
current_rows.append((int(feature_id), str(fid), str(vt_layer), str(geom_type)))
if current_tile is not None:
staged_rows += self.flush_tile(current_tile, current_rows, lookup, temp_file)
tile_index += 1
if tile_index != len(ordered_tiles):
raise RuntimeError(f"tile count mismatch: db_stream={tile_index}, geojson={len(ordered_tiles)}")
return staged_rows
def build_lookup_table(self) -> None:
with self.connect() as conn:
with conn.cursor() as cur:
cur.execute("DROP TABLE IF EXISTS feature_geojson_lookup")
cur.execute(
"""
CREATE TABLE feature_geojson_lookup AS
SELECT
f.z,
f.x,
f.y,
f.id AS feature_id,
pfid.v AS fid,
pvt.v AS vt_layer,
f.geom_type
FROM features f FORCE INDEX (idx_tile_order)
LEFT JOIN properties pfid
ON pfid.feature_id = f.id AND pfid.k = 'fid'
LEFT JOIN properties pvt
ON pvt.feature_id = f.id AND pvt.k = 'vt_layer'
"""
)
cur.execute("ALTER TABLE feature_geojson_lookup ADD PRIMARY KEY (feature_id)")
cur.execute("ALTER TABLE feature_geojson_lookup ADD KEY idx_tile_order (z, x, y, feature_id)")
conn.commit()
def flush_tile(
self,
tile: tuple[int, int, int],
db_rows: list[tuple[int, str, str, str]],
lookup: dict[tuple[int, int, int], Path],
temp_file,
) -> int:
path = lookup.get(tile)
if path is None:
raise RuntimeError(f"missing geojson for tile {tile}")
data = json.loads(path.read_text(encoding="utf-8"))
features = data.get("features", [])
geo_groups: dict[tuple[str, str, str], list[str]] = defaultdict(list)
for feature in features:
props = feature.get("properties", {})
fid = props.get("fid")
vt_layer = props.get("vt_layer")
geometry = feature.get("geometry")
if fid is None or vt_layer is None or geometry is None:
continue
key = (str(fid), str(vt_layer), str(geometry.get("type")))
geo_groups[key].append(self.geometry_to_wkt(geometry))
db_groups: dict[tuple[str, str, str], list[int]] = defaultdict(list)
for feature_id, fid, vt_layer, geom_type in db_rows:
key = (fid, vt_layer, geom_type)
db_groups[key].append(feature_id)
if set(geo_groups) != set(db_groups):
missing_in_db = list(sorted(set(geo_groups) - set(db_groups)))[:10]
missing_in_geo = list(sorted(set(db_groups) - set(geo_groups)))[:10]
raise RuntimeError(
f"tile key mismatch for {path.name}: "
f"missing_in_db={missing_in_db}, missing_in_geo={missing_in_geo}"
)
inserted = 0
for key in sorted(geo_groups):
geo_wkts = geo_groups[key]
db_feature_ids = db_groups[key]
if len(geo_wkts) != len(db_feature_ids):
raise RuntimeError(
f"tile multiplicity mismatch for {path.name}, key={key}: "
f"geojson={len(geo_wkts)}, db={len(db_feature_ids)}"
)
for feature_id, geometry_wkt in zip(db_feature_ids, geo_wkts):
temp_file.write(f"{feature_id}\t{geometry_wkt}\n")
inserted += 1
return inserted
@staticmethod
def create_stage_table(cur) -> None:
cur.execute("DROP TABLE IF EXISTS feature_geometry_stage")
cur.execute(
"""
CREATE TABLE feature_geometry_stage (
feature_id BIGINT NOT NULL,
geometry_wkt LONGTEXT NOT NULL,
PRIMARY KEY (feature_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
@staticmethod
def geometry_to_wkt(geometry: dict) -> str:
geom_type = geometry["type"]
coordinates = geometry["coordinates"]
def fmt_pair(pair):
return f"{pair[0]} {pair[1]}"
def fmt_ring(ring):
return "(" + ", ".join(fmt_pair(pair) for pair in ring) + ")"
if geom_type == "Point":
return f"POINT ({fmt_pair(coordinates)})"
if geom_type == "MultiPoint":
return "MULTIPOINT (" + ", ".join(f"({fmt_pair(pair)})" for pair in coordinates) + ")"
if geom_type == "LineString":
return "LINESTRING (" + ", ".join(fmt_pair(pair) for pair in coordinates) + ")"
if geom_type == "MultiLineString":
return "MULTILINESTRING (" + ", ".join("(" + ", ".join(fmt_pair(pair) for pair in line) + ")" for line in coordinates) + ")"
if geom_type == "Polygon":
return "POLYGON (" + ", ".join(fmt_ring(ring) for ring in coordinates) + ")"
if geom_type == "MultiPolygon":
polygons = []
for polygon in coordinates:
polygons.append("(" + ", ".join(fmt_ring(ring) for ring in polygon) + ")")
return "MULTIPOLYGON (" + ", ".join(polygons) + ")"
raise RuntimeError(f"unsupported geometry type: {geom_type}")
def main() -> None:
loader = GeoJsonStageLoader(
DbConfig(),
Path("/home/wwwroot/newpec/exported_auto/geojson"),
)
loader.load()
if __name__ == "__main__":
main()

174
navsea_geometry_enable.py Normal file
View File

@@ -0,0 +1,174 @@
from __future__ import annotations
import os
from dataclasses import dataclass
import pymysql
@dataclass(frozen=True)
class DbConfig:
host: str = os.getenv("NAVSEA_DB_HOST", "localhost")
port: int = int(os.getenv("NAVSEA_DB_PORT", "3306"))
user: str = os.getenv("NAVSEA_DB_USER", "root")
password: str = os.getenv("NAVSEA_DB_PASSWORD", "2chi9ks2")
database: str = os.getenv("NAVSEA_DB_NAME", "pbf_analysis")
unix_socket: str | None = os.getenv("NAVSEA_DB_SOCKET", "/tmp/mysql.sock")
source_table: str = os.getenv("NAVSEA_GEOMETRY_SOURCE_TABLE", "feature_geometry_stage")
class GeometryEnablement:
def __init__(self, config: DbConfig) -> None:
self.config = config
def connect(self):
kwargs = {
"host": self.config.host,
"port": self.config.port,
"user": self.config.user,
"password": self.config.password,
"database": self.config.database,
"charset": "utf8mb4",
"autocommit": False,
}
if self.config.unix_socket and self.config.host in {"localhost", "127.0.0.1"}:
kwargs["unix_socket"] = self.config.unix_socket
return pymysql.connect(**kwargs)
def run(self) -> None:
with self.connect() as conn:
with conn.cursor() as cur:
source_mode = self.detect_source_mode(cur)
self.create_geometry_table(cur, source_mode)
self.create_spatial_view(cur)
conn.commit()
print("Built navsea_feature_geometry and navsea_detection_objects_spatial")
def detect_source_mode(self, cur) -> str:
cur.execute("SHOW TABLES LIKE %s", (self.config.source_table,))
if cur.fetchone() is None:
raise RuntimeError(
"geometry staging table missing: "
f"{self.config.source_table}. "
"Expected one of: "
"(feature_id, geometry_wkt), "
"(feature_id, geometry_wkb), "
"(feature_id, lon, lat), "
"or (feature_id, geometry GEOMETRY)."
)
cur.execute(
"""
SELECT COLUMN_NAME, DATA_TYPE
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA=%s AND TABLE_NAME=%s
""",
(self.config.database, self.config.source_table),
)
columns = {name: data_type for name, data_type in cur.fetchall()}
if "feature_id" not in columns:
raise RuntimeError(f"{self.config.source_table} missing required column: feature_id")
if "geometry" in columns and columns["geometry"] == "geometry":
return "geometry"
if "geometry_wkt" in columns:
return "geometry_wkt"
if "geometry_wkb" in columns:
return "geometry_wkb"
if {"lon", "lat"}.issubset(columns):
return "lonlat"
raise RuntimeError(
f"{self.config.source_table} does not expose a supported geometry source. "
"Supported layouts: geometry GEOMETRY, geometry_wkt, geometry_wkb, or lon/lat."
)
def create_geometry_table(self, cur, source_mode: str) -> None:
cur.execute("DROP TABLE IF EXISTS navsea_feature_geometry")
cur.execute(
"""
CREATE TABLE navsea_feature_geometry (
feature_id BIGINT NOT NULL,
geometry GEOMETRY NOT NULL,
geometry_source VARCHAR(32) NOT NULL,
PRIMARY KEY (feature_id),
SPATIAL KEY idx_geometry (geometry)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4
"""
)
source_table = self.config.source_table
if source_mode == "geometry":
cur.execute(
f"""
INSERT INTO navsea_feature_geometry (feature_id, geometry, geometry_source)
SELECT feature_id, geometry, 'geometry'
FROM {source_table}
"""
)
elif source_mode == "geometry_wkt":
cur.execute(
f"""
INSERT INTO navsea_feature_geometry (feature_id, geometry, geometry_source)
SELECT feature_id, GeomFromText(geometry_wkt), 'geometry_wkt'
FROM {source_table}
WHERE geometry_wkt IS NOT NULL AND geometry_wkt <> ''
"""
)
elif source_mode == "geometry_wkb":
cur.execute(
f"""
INSERT INTO navsea_feature_geometry (feature_id, geometry, geometry_source)
SELECT feature_id, GeomFromWKB(geometry_wkb), 'geometry_wkb'
FROM {source_table}
WHERE geometry_wkb IS NOT NULL
"""
)
elif source_mode == "lonlat":
cur.execute(
f"""
INSERT INTO navsea_feature_geometry (feature_id, geometry, geometry_source)
SELECT feature_id, Point(lon, lat), 'lonlat'
FROM {source_table}
WHERE lon IS NOT NULL AND lat IS NOT NULL
"""
)
else:
raise RuntimeError(f"unsupported source mode: {source_mode}")
@staticmethod
def create_spatial_view(cur) -> None:
cur.execute("DROP VIEW IF EXISTS navsea_detection_objects_spatial")
cur.execute(
"""
CREATE VIEW navsea_detection_objects_spatial AS
SELECT
d.feature_id,
g.geometry,
d.z,
d.x,
d.y,
d.geom_type,
d.source_layer,
d.class_name,
d.shape_name,
d.layer_name,
d.canonical_object_type,
d.object_family,
d.detection_class,
d.capability,
d.detection_key
FROM navsea_detection_objects d
JOIN navsea_feature_geometry g
ON d.feature_id = g.feature_id
"""
)
def main() -> None:
GeometryEnablement(DbConfig()).run()
if __name__ == "__main__":
main()

237
navsea_mapping_registry.py Normal file
View File

@@ -0,0 +1,237 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any
import pymysql
@dataclass(frozen=True)
class FieldValueRule:
field_name_jp: str
legacy_value: str
standardized_field: str
standardized_value: str
geom_scope: tuple[str, ...]
source_layer_scope: tuple[str, ...]
canonical_object_scope: tuple[str, ...]
def matches(self, context: dict[str, str]) -> bool:
if self.geom_scope and context.get("geom_type", "") not in self.geom_scope:
return False
if self.source_layer_scope and context.get("source_layer", "") not in self.source_layer_scope:
return False
if self.canonical_object_scope and context.get("canonical_object_type", "") not in self.canonical_object_scope:
return False
return True
@dataclass(frozen=True)
class RenderRule:
rule_id: str
priority: int
match_expr: dict[str, Any]
output: dict[str, Any]
def matches(self, context: dict[str, Any]) -> bool:
return evaluate_match_expr(self.match_expr, context)
def normalize_csv_scope(value: str | None) -> tuple[str, ...]:
if not value:
return ()
return tuple(part.strip() for part in str(value).split(",") if part.strip())
def evaluate_match_expr(match_expr: dict[str, Any], context: dict[str, Any]) -> bool:
clauses = match_expr.get("all") or []
for clause in clauses:
field = clause.get("field")
op = clause.get("op")
expected = clause.get("value")
actual = context.get(str(field))
actual_text = "" if actual is None else str(actual)
if op == "eq":
if actual_text != str(expected):
return False
elif op == "neq":
if actual_text == str(expected):
return False
elif op == "in":
if actual_text not in {str(item) for item in (expected or [])}:
return False
elif op == "contains":
if str(expected) not in actual_text:
return False
elif op == "contains_any":
values = [str(item) for item in (expected or [])]
if not any(item in actual_text for item in values):
return False
elif op == "is_null":
if actual not in (None, ""):
return False
elif op == "not_null":
if actual in (None, ""):
return False
else:
return False
return True
class NavSeaMappingRegistry:
def __init__(
self,
bundle_id: str,
bundle_version: str,
source_layer_rules: dict[str, dict[str, str]],
field_value_rules: dict[tuple[str, str], list[FieldValueRule]],
render_rules: list[RenderRule],
) -> None:
self.bundle_id = bundle_id
self.bundle_version = bundle_version
self.source_layer_rules = source_layer_rules
self.field_value_rules = field_value_rules
self.render_rules = render_rules
@classmethod
def load(
cls,
conn: pymysql.Connection,
bundle_id: str | None = None,
) -> "NavSeaMappingRegistry":
with conn.cursor() as cur:
if bundle_id:
cur.execute(
"""
SELECT bundle_id, bundle_version
FROM navsea_rule_bundle
WHERE bundle_id = %s
ORDER BY created_at DESC
LIMIT 1
""",
(bundle_id,),
)
else:
cur.execute(
"""
SELECT bundle_id, bundle_version
FROM navsea_rule_bundle
ORDER BY created_at DESC
LIMIT 1
"""
)
bundle = cur.fetchone()
if not bundle:
raise RuntimeError("navsea mapping registry is empty")
resolved_bundle_id = str(bundle["bundle_id"])
resolved_bundle_version = str(bundle["bundle_version"])
cur.execute(
"""
SELECT source_layer_jp, source_layer_std
FROM navsea_source_layer_rules
WHERE bundle_id = %s AND bundle_version = %s
""",
(resolved_bundle_id, resolved_bundle_version),
)
source_layer_rules = {
str(row["source_layer_jp"]): {
"source_layer_std": str(row["source_layer_std"]),
"source_layer_rule_id": f"SLR:{row['source_layer_jp']}",
}
for row in cur.fetchall()
}
cur.execute(
"""
SELECT
field_name_jp,
legacy_value,
standardized_field,
standardized_value,
geom_scope,
source_layer_scope,
canonical_object_scope
FROM navsea_field_value_rules
WHERE bundle_id = %s AND bundle_version = %s
ORDER BY rule_priority ASC, rule_id ASC, rule_revision ASC
""",
(resolved_bundle_id, resolved_bundle_version),
)
field_value_rules: dict[tuple[str, str], list[FieldValueRule]] = {}
for row in cur.fetchall():
key = (str(row["field_name_jp"]), str(row["legacy_value"]))
field_value_rules.setdefault(key, []).append(
FieldValueRule(
field_name_jp=str(row["field_name_jp"]),
legacy_value=str(row["legacy_value"]),
standardized_field=str(row["standardized_field"]),
standardized_value=str(row["standardized_value"]),
geom_scope=normalize_csv_scope(row.get("geom_scope")),
source_layer_scope=normalize_csv_scope(row.get("source_layer_scope")),
canonical_object_scope=normalize_csv_scope(row.get("canonical_object_scope")),
)
)
cur.execute(
"""
SELECT rule_id, priority, match_expr_json, output_json
FROM navsea_render_rules
WHERE bundle_id = %s AND bundle_version = %s AND enabled = 1
ORDER BY priority ASC, rule_id ASC, rule_revision ASC
""",
(resolved_bundle_id, resolved_bundle_version),
)
render_rules = []
for row in cur.fetchall():
render_rules.append(
RenderRule(
rule_id=str(row["rule_id"]),
priority=int(row["priority"]),
match_expr=json.loads(row["match_expr_json"]),
output=json.loads(row["output_json"]),
)
)
return cls(
bundle_id=resolved_bundle_id,
bundle_version=resolved_bundle_version,
source_layer_rules=source_layer_rules,
field_value_rules=field_value_rules,
render_rules=render_rules,
)
def resolve_source_layer(self, source_layer_jp: str) -> tuple[str, str]:
rule = self.source_layer_rules.get(source_layer_jp)
if not rule:
return source_layer_jp, f"SLR:{source_layer_jp}:identity"
return rule["source_layer_std"], rule["source_layer_rule_id"]
def standardize_field_value(
self,
field_name_jp: str,
legacy_value: object,
standardized_field: str,
*,
context: dict[str, str] | None = None,
) -> str | None:
if legacy_value in (None, ""):
return None
key = (field_name_jp, str(legacy_value))
rules = self.field_value_rules.get(key, [])
match_context = context or {}
for rule in rules:
if rule.standardized_field != standardized_field:
continue
if rule.matches(match_context):
return rule.standardized_value
return None
def resolve_render_rule(self, context: dict[str, Any]) -> tuple[dict[str, Any], str | None]:
for rule in self.render_rules:
if rule.matches(context):
return dict(rule.output), rule.rule_id
return {}, None

524
navsea_mapping_registry_sync.py Normal file
View File

@@ -0,0 +1,524 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import pymysql
import yaml
ROOT = Path(__file__).resolve().parent
MAPPINGS_DIR = ROOT / "tasks" / "pbf" / "mappings"
@dataclass(frozen=True)
class DbConfig:
host: str = "localhost"
port: int = 3306
user: str = "root"
password: str = "2chi9ks2"
database: str = "pbf_analysis"
unix_socket: str | None = "/tmp/mysql.sock"
DDL_STATEMENTS = [
"DROP VIEW IF EXISTS navsea_new_to_legacy_mapping_vw",
"DROP VIEW IF EXISTS navsea_legacy_to_new_mapping_vw",
"DROP VIEW IF EXISTS navsea_mapping_registry_vw",
"DROP TABLE IF EXISTS navsea_render_rules",
"DROP TABLE IF EXISTS navsea_taxonomy_rules",
"DROP TABLE IF EXISTS navsea_field_value_rules",
"DROP TABLE IF EXISTS navsea_field_name_rules",
"DROP TABLE IF EXISTS navsea_source_layer_rules",
"DROP TABLE IF EXISTS navsea_rule_bundle",
"""
CREATE TABLE IF NOT EXISTS navsea_rule_bundle (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
status VARCHAR(20) NOT NULL,
effective_date DATE DEFAULT NULL,
taxonomy_ruleset VARCHAR(64) DEFAULT NULL,
render_ruleset VARCHAR(64) DEFAULT NULL,
source_layer_ruleset VARCHAR(64) DEFAULT NULL,
field_name_ruleset VARCHAR(64) DEFAULT NULL,
field_value_ruleset VARCHAR(64) DEFAULT NULL,
source_of_truth VARCHAR(32) NOT NULL DEFAULT 'yaml+sql',
notes TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS navsea_source_layer_rules (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
source_layer_jp VARCHAR(100) NOT NULL,
source_layer_std VARCHAR(100) NOT NULL,
semantic_granularity VARCHAR(50) DEFAULT NULL,
canonical_family VARCHAR(100) DEFAULT NULL,
render_strategy VARCHAR(50) DEFAULT NULL,
preserve_source_layer TINYINT(1) NOT NULL DEFAULT 1,
style_bound TINYINT(1) NOT NULL DEFAULT 1,
style_types_json LONGTEXT,
notes TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version, source_layer_jp),
KEY idx_source_layer_std (source_layer_std),
KEY idx_source_layer_family (canonical_family)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS navsea_field_name_rules (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
field_name_jp VARCHAR(100) NOT NULL,
field_name_std VARCHAR(100) NOT NULL,
field_group_name VARCHAR(50) NOT NULL,
keep_in_delivery TINYINT(1) NOT NULL DEFAULT 1,
keep_in_engineering TINYINT(1) NOT NULL DEFAULT 1,
normalization_class VARCHAR(64) NOT NULL,
notes TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version, field_name_jp),
UNIQUE KEY uniq_field_name_std (bundle_id, bundle_version, field_name_std)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS navsea_field_value_rules (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
rule_id VARCHAR(64) NOT NULL,
rule_revision INT NOT NULL,
field_name_jp VARCHAR(100) NOT NULL,
legacy_value VARCHAR(191) NOT NULL,
standardized_field VARCHAR(100) NOT NULL,
standardized_value VARCHAR(191) NOT NULL,
geom_scope VARCHAR(100) DEFAULT NULL,
source_layer_scope VARCHAR(200) DEFAULT NULL,
canonical_object_scope VARCHAR(191) DEFAULT NULL,
rule_priority INT NOT NULL DEFAULT 100,
notes TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version, rule_id, rule_revision),
KEY idx_field_value_lookup (field_name_jp, legacy_value),
KEY idx_standardized_lookup (standardized_field, standardized_value)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS navsea_taxonomy_rules (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
rule_id VARCHAR(64) NOT NULL,
rule_revision INT NOT NULL,
priority INT NOT NULL,
enabled TINYINT(1) NOT NULL DEFAULT 1,
match_scope VARCHAR(32) NOT NULL,
match_expr_json LONGTEXT NOT NULL,
output_json LONGTEXT NOT NULL,
canonical_family VARCHAR(100) NOT NULL,
canonical_object_type VARCHAR(191) NOT NULL,
detection_key_template VARCHAR(191) DEFAULT NULL,
rule_reason TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version, rule_id, rule_revision),
KEY idx_taxonomy_object (canonical_object_type),
KEY idx_taxonomy_family (canonical_family)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE TABLE IF NOT EXISTS navsea_render_rules (
bundle_id VARCHAR(64) NOT NULL,
bundle_version VARCHAR(32) NOT NULL,
rule_id VARCHAR(64) NOT NULL,
rule_revision INT NOT NULL,
priority INT NOT NULL,
enabled TINYINT(1) NOT NULL DEFAULT 1,
match_expr_json LONGTEXT NOT NULL,
output_json LONGTEXT NOT NULL,
chart_render_type VARCHAR(32) DEFAULT NULL,
chart_symbol_family VARCHAR(64) DEFAULT NULL,
chart_symbol_code VARCHAR(64) DEFAULT NULL,
chart_line_style VARCHAR(64) DEFAULT NULL,
chart_fill_style VARCHAR(64) DEFAULT NULL,
chart_text_style VARCHAR(64) DEFAULT NULL,
chart_priority INT DEFAULT NULL,
chart_visibility_min INT DEFAULT NULL,
chart_visibility_max INT DEFAULT NULL,
area_usage_class VARCHAR(64) DEFAULT NULL,
hazard_class VARCHAR(64) DEFAULT NULL,
hazard_severity VARCHAR(64) DEFAULT NULL,
rule_reason TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NULL DEFAULT NULL,
PRIMARY KEY (bundle_id, bundle_version, rule_id, rule_revision),
KEY idx_render_symbol (chart_symbol_code),
KEY idx_render_fill (chart_fill_style),
KEY idx_render_text (chart_text_style)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
""",
"""
CREATE OR REPLACE VIEW navsea_mapping_registry_vw AS
SELECT
'source_layer' AS mapping_type,
bundle_id,
bundle_version,
source_layer_jp AS legacy_key,
source_layer_jp AS legacy_value,
'source_layer_std' AS new_key,
source_layer_std AS new_value,
NULL AS rule_id,
notes
FROM navsea_source_layer_rules
UNION ALL
SELECT
'field_name' AS mapping_type,
bundle_id,
bundle_version,
field_name_jp AS legacy_key,
field_name_jp AS legacy_value,
'field_name_std' AS new_key,
field_name_std AS new_value,
NULL AS rule_id,
notes
FROM navsea_field_name_rules
UNION ALL
SELECT
'field_value' AS mapping_type,
bundle_id,
bundle_version,
field_name_jp AS legacy_key,
legacy_value AS legacy_value,
standardized_field AS new_key,
standardized_value AS new_value,
rule_id AS rule_id,
notes
FROM navsea_field_value_rules
UNION ALL
SELECT
'taxonomy' AS mapping_type,
bundle_id,
bundle_version,
'match_expr' AS legacy_key,
CAST(match_expr_json AS CHAR(4096)) AS legacy_value,
'canonical_object_type' AS new_key,
canonical_object_type AS new_value,
rule_id AS rule_id,
rule_reason AS notes
FROM navsea_taxonomy_rules
UNION ALL
SELECT
'render' AS mapping_type,
bundle_id,
bundle_version,
'match_expr' AS legacy_key,
CAST(match_expr_json AS CHAR(4096)) AS legacy_value,
'render_output' AS new_key,
CAST(output_json AS CHAR(4096)) AS new_value,
rule_id AS rule_id,
rule_reason AS notes
FROM navsea_render_rules
""",
"""
CREATE OR REPLACE VIEW navsea_legacy_to_new_mapping_vw AS
SELECT
mapping_type,
bundle_id,
bundle_version,
legacy_key,
legacy_value,
new_key,
new_value,
rule_id,
notes
FROM navsea_mapping_registry_vw
""",
"""
CREATE OR REPLACE VIEW navsea_new_to_legacy_mapping_vw AS
SELECT
mapping_type,
bundle_id,
bundle_version,
new_key,
new_value,
legacy_key,
legacy_value,
rule_id,
notes
FROM navsea_mapping_registry_vw
""",
]
def load_yaml(path: Path) -> dict[str, Any]:
with path.open("r", encoding="utf-8") as f:
return yaml.safe_load(f) or {}
def json_or_none(value: Any) -> str | None:
if value in (None, "", [], {}):
return None
return json.dumps(value, ensure_ascii=False, sort_keys=True)
def connect(db: DbConfig) -> pymysql.Connection:
kwargs: dict[str, Any] = {
"host": db.host,
"port": db.port,
"user": db.user,
"password": db.password,
"database": db.database,
"charset": "utf8mb4",
"autocommit": False,
"cursorclass": pymysql.cursors.DictCursor,
}
if db.unix_socket:
kwargs["unix_socket"] = db.unix_socket
return pymysql.connect(**kwargs)
def purge_bundle(cur: pymysql.cursors.Cursor, bundle_id: str, bundle_version: str) -> None:
tables = (
"navsea_source_layer_rules",
"navsea_field_name_rules",
"navsea_field_value_rules",
"navsea_taxonomy_rules",
"navsea_render_rules",
"navsea_rule_bundle",
)
for table in tables:
cur.execute(
f"DELETE FROM {table} WHERE bundle_id=%s AND bundle_version=%s",
(bundle_id, bundle_version),
)
def load_canonical_layer_rules(cur: pymysql.cursors.Cursor) -> dict[str, dict[str, Any]]:
cur.execute(
"""
SELECT
source_layer,
semantic_granularity,
canonical_family,
render_strategy,
preserve_source_layer,
style_bound,
style_types,
notes
FROM canonical_layer_rules
"""
)
return {row["source_layer"]: row for row in cur.fetchall()}
def sync() -> dict[str, int]:
bundle = load_yaml(MAPPINGS_DIR / "navsea_rule_bundle_v1.yaml")
source_layer_rules = load_yaml(MAPPINGS_DIR / "navsea_source_layer_rules_v1.yaml")
field_name_rules = load_yaml(MAPPINGS_DIR / "navsea_field_name_rules_v1.yaml")
field_value_rules = load_yaml(MAPPINGS_DIR / "navsea_field_value_rules_v1.yaml")
taxonomy_rules = load_yaml(MAPPINGS_DIR / "navsea_taxonomy_rules_v1.yaml")
render_rules = load_yaml(MAPPINGS_DIR / "navsea_render_rules_v1.yaml")
bundle_id = bundle["bundle_id"]
bundle_version = bundle["bundle_version"]
counts = {
"source_layer_rules": 0,
"field_name_rules": 0,
"field_value_rules": 0,
"taxonomy_rules": 0,
"render_rules": 0,
}
with connect(DbConfig()) as conn:
with conn.cursor() as cur:
for ddl in DDL_STATEMENTS:
cur.execute(ddl)
purge_bundle(cur, bundle_id, bundle_version)
cur.execute(
"""
INSERT INTO navsea_rule_bundle (
bundle_id, bundle_version, status, effective_date,
taxonomy_ruleset, render_ruleset, source_layer_ruleset,
field_name_ruleset, field_value_ruleset, notes
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
bundle["status"],
bundle.get("effective_date"),
bundle.get("taxonomy_ruleset"),
bundle.get("render_ruleset"),
bundle.get("source_layer_ruleset"),
bundle.get("field_name_ruleset"),
bundle.get("field_value_ruleset"),
bundle.get("notes"),
),
)
canonical_layers = load_canonical_layer_rules(cur)
for rule in source_layer_rules.get("rules", []):
layer_meta = canonical_layers.get(rule["source_layer_jp"], {})
style_types = layer_meta.get("style_types")
style_types_json = None
if style_types:
style_types_json = json.dumps(
[part.strip() for part in str(style_types).split(",") if part.strip()],
ensure_ascii=False,
)
cur.execute(
"""
INSERT INTO navsea_source_layer_rules (
bundle_id, bundle_version, source_layer_jp, source_layer_std,
semantic_granularity, canonical_family, render_strategy,
preserve_source_layer, style_bound, style_types_json, notes
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
rule["source_layer_jp"],
rule["source_layer_std"],
layer_meta.get("semantic_granularity"),
layer_meta.get("canonical_family"),
layer_meta.get("render_strategy"),
int(layer_meta.get("preserve_source_layer", 1)),
int(layer_meta.get("style_bound", 1)),
style_types_json,
rule.get("notes") or layer_meta.get("notes"),
),
)
counts["source_layer_rules"] += 1
for rule in field_name_rules.get("rules", []):
cur.execute(
"""
INSERT INTO navsea_field_name_rules (
bundle_id, bundle_version, field_name_jp, field_name_std, field_group_name,
keep_in_delivery, keep_in_engineering, normalization_class, notes
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
rule["field_name_jp"],
rule["field_name_std"],
rule["field_group"],
int(bool(rule["keep_in_delivery"])),
int(bool(rule["keep_in_engineering"])),
rule["normalization_class"],
rule.get("notes"),
),
)
counts["field_name_rules"] += 1
for rule in field_value_rules.get("rules", []):
cur.execute(
"""
INSERT INTO navsea_field_value_rules (
bundle_id, bundle_version, rule_id, rule_revision, field_name_jp, legacy_value,
standardized_field, standardized_value, geom_scope, source_layer_scope,
canonical_object_scope, rule_priority, notes
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
rule["rule_id"],
int(rule["rule_revision"]),
rule["field_name_jp"],
str(rule["legacy_value"]),
rule["standardized_field"],
rule["standardized_value"],
rule.get("geom_scope"),
rule.get("source_layer_scope"),
rule.get("canonical_object_scope"),
int(rule.get("rule_priority", 100)),
rule.get("notes"),
),
)
counts["field_value_rules"] += 1
for rule in taxonomy_rules.get("rules", []):
output = rule["output"]
cur.execute(
"""
INSERT INTO navsea_taxonomy_rules (
bundle_id, bundle_version, rule_id, rule_revision, priority, enabled,
match_scope, match_expr_json, output_json, canonical_family,
canonical_object_type, detection_key_template, rule_reason
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
rule["rule_id"],
int(rule["rule_revision"]),
int(rule["priority"]),
int(bool(rule["enabled"])),
rule["match_scope"],
json_or_none(rule["match_expr"]),
json_or_none(output),
output["canonical_family"],
output["canonical_object_type"],
output.get("detection_key_template"),
rule.get("reason"),
),
)
counts["taxonomy_rules"] += 1
for rule in render_rules.get("rules", []):
output = rule["output"]
cur.execute(
"""
INSERT INTO navsea_render_rules (
bundle_id, bundle_version, rule_id, rule_revision, priority, enabled,
match_expr_json, output_json, chart_render_type, chart_symbol_family,
chart_symbol_code, chart_line_style, chart_fill_style, chart_text_style,
chart_priority, chart_visibility_min, chart_visibility_max,
area_usage_class, hazard_class, hazard_severity, rule_reason
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
bundle_id,
bundle_version,
rule["rule_id"],
int(rule["rule_revision"]),
int(rule["priority"]),
int(bool(rule["enabled"])),
json_or_none(rule["match_expr"]),
json_or_none(output),
output.get("chart_render_type"),
output.get("chart_symbol_family"),
output.get("chart_symbol_code"),
output.get("chart_line_style"),
output.get("chart_fill_style"),
output.get("chart_text_style"),
output.get("chart_priority"),
output.get("chart_visibility_min"),
output.get("chart_visibility_max"),
output.get("area_usage_class"),
output.get("hazard_class"),
output.get("hazard_severity"),
rule.get("reason"),
),
)
counts["render_rules"] += 1
conn.commit()
return counts
if __name__ == "__main__":
result = sync()
print(json.dumps(result, ensure_ascii=False, indent=2))

118
navsea_mvt_to_geojson.py Normal file
View File

@@ -0,0 +1,118 @@
from __future__ import annotations
import json
import math
from pathlib import Path
import mapbox_vector_tile
INPUT_ROOT = Path("/home/wwwroot/newpec/exported_auto/tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles")
OUTPUT_ROOT = Path("/home/wwwroot/newpec/exported_auto/geojson_linux")
def tile_coord_to_lonlat(z: int, x: int, y: int, tx: float, ty: float, extent: int) -> list[float]:
lon = ((x + (tx / extent)) / (2**z)) * 360.0 - 180.0
lat = math.degrees(
math.atan(
math.sinh(
math.pi * (1.0 - (2.0 * (y + (ty / extent)) / (2**z)))
)
)
)
return [lon, lat]
def convert_geometry(geometry: dict, z: int, x: int, y: int, extent: int) -> dict:
geom_type = geometry["type"]
coords = geometry["coordinates"]
if geom_type == "Point":
return {"type": "Point", "coordinates": tile_coord_to_lonlat(z, x, y, coords[0], coords[1], extent)}
if geom_type == "MultiPoint":
return {
"type": "MultiPoint",
"coordinates": [tile_coord_to_lonlat(z, x, y, px, py, extent) for px, py in coords],
}
if geom_type == "LineString":
return {
"type": "LineString",
"coordinates": [tile_coord_to_lonlat(z, x, y, px, py, extent) for px, py in coords],
}
if geom_type == "MultiLineString":
return {
"type": "MultiLineString",
"coordinates": [
[tile_coord_to_lonlat(z, x, y, px, py, extent) for px, py in line]
for line in coords
],
}
if geom_type == "Polygon":
return {
"type": "Polygon",
"coordinates": [
[tile_coord_to_lonlat(z, x, y, px, py, extent) for px, py in ring]
for ring in coords
],
}
if geom_type == "MultiPolygon":
return {
"type": "MultiPolygon",
"coordinates": [
[
[tile_coord_to_lonlat(z, x, y, px, py, extent) for px, py in ring]
for ring in polygon
]
for polygon in coords
],
}
raise RuntimeError(f"unsupported geometry type: {geom_type}")
def convert_tile(path: Path) -> tuple[dict, int]:
z = int(path.parent.parent.name)
x = int(path.parent.name)
y = int(path.stem)
with path.open("rb") as fh:
decoded = mapbox_vector_tile.decode(fh.read())
features = []
for vt_layer, payload in decoded.items():
extent = int(payload.get("extent") or 4096)
for feature in payload.get("features", []):
properties = dict(feature.get("properties") or {})
properties["vt_layer"] = vt_layer
out_feature = {
"type": "Feature",
"geometry": convert_geometry(feature["geometry"], z, x, y, extent),
"properties": properties,
}
if feature.get("id") is not None:
out_feature["id"] = feature["id"]
features.append(out_feature)
return {"type": "FeatureCollection", "features": features}, len(features)
def main() -> None:
OUTPUT_ROOT.mkdir(parents=True, exist_ok=True)
tile_paths = sorted(INPUT_ROOT.glob("*/*/*.pbf"))
total_features = 0
for index, path in enumerate(tile_paths, start=1):
collection, count = convert_tile(path)
total_features += count
output_path = OUTPUT_ROOT / f"{path.parent.parent.name}_{path.parent.name}_{path.stem}.json"
output_path.write_text(
json.dumps(collection, ensure_ascii=False, separators=(",", ":")),
encoding="utf-8",
)
if index % 5000 == 0:
print(f"processed {index}/{len(tile_paths)} tiles, wrote {total_features} features")
print(f"Converted {len(tile_paths)} tiles into {OUTPUT_ROOT} with {total_features} features")
if __name__ == "__main__":
main()

660
navsea_relayer.py Normal file
View File

@@ -0,0 +1,660 @@
import json
from collections import Counter, defaultdict
from pathlib import Path
import pymysql
ROOT = Path(__file__).resolve().parent
STYLE_JSON_PATH = ROOT / "src" / "pbf" / "style.json"
MANUAL_OBJECT_RULES = {
"P754ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P危険界ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P基本線ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P投錨注意障害物ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P施設・境界線等ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P航路ククリ": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved clip/outline layer.",
},
"P穴": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Source-preserved area layer.",
},
"P陸域": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Source-preserved area layer.",
},
"L海底地形": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Source-preserved bathymetry line layer.",
},
"p地名": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Source-preserved place label layer.",
},
"p地名陸": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Source-preserved place label layer.",
},
"p高さ制限": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Source-preserved point annotation layer.",
},
"p錨泊地等": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Source-preserved point annotation layer.",
},
"サンドウェーブ": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Same object appears in multiple source layers but remains a point hazard.",
},
"海底設置物、放水口、取水口": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Stable point object despite multiple source layers.",
},
"全沈没船 (危険なし)": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Stable point object despite multiple source layers.",
},
"測定済みの沈船": {
"family": "symbol",
"allowed_geom_types": ["Point"],
"notes": "Stable point object despite multiple source layers.",
},
"道路": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Road centerlines legitimately appear in multi-part line form.",
},
"等深線": {
"family": "line",
"allowed_geom_types": ["LineString", "MultiLineString"],
"notes": "Depth contours legitimately appear in multi-part line form.",
},
"防波堤": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Breakwaters are modeled as areas, including multipart polygons.",
},
"障害物": {
"family": "mixed",
"allowed_geom_types": ["Point", "Polygon", "MultiPolygon"],
"notes": "Obstacle objects can be rendered as symbols or area extents.",
},
"未測海域": {
"family": "mixed",
"allowed_geom_types": ["LineString", "Polygon", "MultiPolygon"],
"notes": "Unsurveyed regions appear as boundaries and areas.",
},
"浮施設・桟橋": {
"family": "mixed",
"allowed_geom_types": ["LineString", "MultiLineString", "Polygon", "MultiPolygon"],
"notes": "Floating facilities and piers appear as line and area geometry.",
},
"航路 (法律による航路)": {
"family": "mixed",
"allowed_geom_types": ["LineString", "Polygon", "MultiPolygon"],
"notes": "Legally defined routes appear as lines and areas.",
},
"険悪物": {
"family": "mixed",
"allowed_geom_types": ["Point", "Polygon", "MultiPolygon"],
"notes": "Hazards may be represented as point symbols or area extents.",
},
"魚礁": {
"family": "mixed",
"allowed_geom_types": ["Point", "Polygon", "MultiPolygon"],
"notes": "Artificial reefs may be represented as point symbols or area extents.",
},
"養殖場": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Aquaculture zones are area features.",
},
"漁網": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Fishing nets are area extents in this dataset.",
},
"錨泊 (指定)地": {
"family": "mixed",
"allowed_geom_types": ["Point", "Polygon", "MultiPolygon"],
"notes": "Anchorage can appear as symbol or area depending on source layer.",
},
"ケーソン仮置き場": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Caisson storage areas are polygons.",
},
"土砂捨て場": {
"family": "surface",
"allowed_geom_types": ["Polygon", "MultiPolygon"],
"notes": "Spoil grounds are polygons.",
},
}
DEFAULT_GEOMETRIES = {
"symbol": ["Point"],
"line": ["LineString", "MultiLineString"],
"surface": ["Polygon", "MultiPolygon"],
"mixed": [],
}
def connect():
return pymysql.connect(
host="localhost",
user="root",
password="2chi9ks2",
database="pbf_analysis",
charset="utf8mb4",
unix_socket="/tmp/mysql.sock",
autocommit=False,
)
def load_style_source_layers():
style = json.loads(STYLE_JSON_PATH.read_text(encoding="utf-8"))
layers = {}
for layer in style.get("layers", []):
source_layer = layer.get("source-layer")
if not source_layer:
continue
layers.setdefault(source_layer, {"style_rows": 0, "layer_types": set()})
layers[source_layer]["style_rows"] += 1
if layer.get("type"):
layers[source_layer]["layer_types"].add(layer["type"])
return layers
def geom_bucket(geom_type: str | None) -> str:
if geom_type in {"Point", "MultiPoint"}:
return "symbol"
if geom_type in {"LineString", "MultiLineString"}:
return "line"
if geom_type in {"Polygon", "MultiPolygon"}:
return "surface"
return "mixed"
def infer_family(geom_types: set[str]) -> str:
buckets = {geom_bucket(geom_type) for geom_type in geom_types if geom_type}
if not buckets:
return "mixed"
if len(buckets) == 1:
return next(iter(buckets))
return "mixed"
def choose_allowed_geometries(object_type: str, family: str, observed_geom_types: list[str]) -> tuple[list[str], str]:
manual = MANUAL_OBJECT_RULES.get(object_type)
if manual:
return manual["allowed_geom_types"], "manual_override"
defaults = DEFAULT_GEOMETRIES.get(family, [])
if defaults:
return defaults, "family_default"
return observed_geom_types, "observed_fallback"
def build_canonical_layer_rules(cur, style_layers):
cur.execute("DROP TABLE IF EXISTS canonical_layer_rules")
cur.execute(
"""
CREATE TABLE canonical_layer_rules (
source_layer VARCHAR(100) NOT NULL,
semantic_granularity VARCHAR(50) NOT NULL,
canonical_family VARCHAR(100) NOT NULL,
render_strategy VARCHAR(50) NOT NULL,
preserve_source_layer TINYINT(1) NOT NULL DEFAULT 1,
style_bound TINYINT(1) NOT NULL DEFAULT 1,
style_rows INT NOT NULL DEFAULT 0,
style_types VARCHAR(200) NULL,
notes TEXT NULL,
PRIMARY KEY (source_layer)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
"""
SELECT
vt_layer,
SUM(feature_count) AS feature_count,
COUNT(DISTINCT COALESCE(NULLIF(class_name, ''), NULLIF(shape_name, ''), vt_layer)) AS semantic_type_count,
SUM(CASE WHEN class_name IS NULL OR class_name='' THEN feature_count ELSE 0 END) AS missing_class_count,
SUM(CASE WHEN shape_name IS NULL OR shape_name='' THEN feature_count ELSE 0 END) AS missing_shape_count
FROM object_catalog
GROUP BY vt_layer
"""
)
inserts = []
for vt_layer, feature_count, semantic_type_count, missing_class_count, missing_shape_count in cur.fetchall():
style_info = style_layers.get(vt_layer, {"style_rows": 0, "layer_types": set()})
if vt_layer.startswith("p"):
canonical_family = "symbol"
elif vt_layer.startswith("L"):
canonical_family = "line"
elif vt_layer.startswith("P"):
canonical_family = "surface"
else:
canonical_family = "mixed"
if semantic_type_count >= 5 and (vt_layer.startswith("P") or vt_layer.startswith("L")):
semantic_granularity = "container_layer"
notes = (
f"Layer contains {semantic_type_count} semantic types across {feature_count} features; "
"retain source layer for rendering and derive semantic overlays for analysis."
)
elif missing_class_count == feature_count and missing_shape_count == feature_count:
semantic_granularity = "style_or_source_layer"
notes = "Layer has no semantic class/shape annotations; source layer remains the stable identity."
else:
semantic_granularity = "semantic_layer"
notes = "Layer is close to a stable semantic object and can be preserved as-is."
inserts.append(
(
vt_layer,
semantic_granularity,
canonical_family,
"keep_source_layer",
1,
1 if style_info["style_rows"] > 0 else 0,
style_info["style_rows"],
",".join(sorted(style_info["layer_types"])) or None,
notes,
)
)
cur.executemany(
"""
INSERT INTO canonical_layer_rules (
source_layer,
semantic_granularity,
canonical_family,
render_strategy,
preserve_source_layer,
style_bound,
style_rows,
style_types,
notes
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
inserts,
)
def build_canonical_object_rules(cur):
cur.execute("DROP TABLE IF EXISTS canonical_object_rules")
cur.execute(
"""
CREATE TABLE canonical_object_rules (
canonical_object_type VARCHAR(191) NOT NULL,
canonical_family VARCHAR(100) NOT NULL,
classification_basis VARCHAR(50) NOT NULL,
source_layer_scope VARCHAR(50) NOT NULL,
source_layer_count INT NOT NULL,
geom_type_count INT NOT NULL,
object_type_sources VARCHAR(100) NOT NULL,
source_layers TEXT NULL,
observed_geom_types VARCHAR(200) NULL,
allowed_geom_types VARCHAR(200) NULL,
preferred_geom_type VARCHAR(20) NULL,
geometry_rule_source VARCHAR(50) NOT NULL,
notes TEXT NULL,
PRIMARY KEY (canonical_object_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
"""
SELECT
COALESCE(NULLIF(class_name, ''), NULLIF(shape_name, ''), vt_layer) AS object_type,
GROUP_CONCAT(
DISTINCT CASE
WHEN class_name IS NOT NULL AND class_name <> '' THEN 'class_name'
WHEN shape_name IS NOT NULL AND shape_name <> '' THEN 'shape_name'
ELSE 'vt_layer'
END
ORDER BY 1 SEPARATOR ','
) AS object_type_sources,
SUM(feature_count) AS feature_count,
COUNT(DISTINCT vt_layer) AS source_layer_count,
GROUP_CONCAT(DISTINCT vt_layer ORDER BY vt_layer SEPARATOR ',') AS source_layers,
COUNT(DISTINCT geom_type) AS geom_type_count,
GROUP_CONCAT(DISTINCT geom_type ORDER BY geom_type SEPARATOR ',') AS observed_geom_types
FROM object_catalog
GROUP BY COALESCE(NULLIF(class_name, ''), NULLIF(shape_name, ''), vt_layer)
"""
)
object_rows = cur.fetchall()
cur.execute(
"""
SELECT
object_type,
geom_type,
feature_count
FROM object_type_stats
"""
)
geom_counts: dict[str, Counter[str]] = defaultdict(Counter)
for object_type, geom_type, feature_count in cur.fetchall():
geom_counts[object_type][geom_type] = feature_count
inserts = []
for (
object_type,
object_type_sources,
feature_count,
source_layer_count,
source_layers,
geom_type_count,
observed_geom_types,
) in object_rows:
observed_geom_list = [part for part in (observed_geom_types or "").split(",") if part]
observed_geom_set = set(observed_geom_list)
manual = MANUAL_OBJECT_RULES.get(object_type)
canonical_family = manual["family"] if manual else infer_family(observed_geom_set)
classification_basis = (
"source_layer_preserved"
if object_type_sources == "vt_layer"
else "semantic_object"
)
source_layer_scope = "multi_source" if source_layer_count > 1 else "single_source"
allowed_geom_types, geometry_rule_source = choose_allowed_geometries(
object_type,
canonical_family,
observed_geom_list,
)
preferred_geom_type = None
if geom_counts.get(object_type):
preferred_geom_type = geom_counts[object_type].most_common(1)[0][0]
notes = (
manual["notes"]
if manual
else (
"Source-preserved object type derived from vt_layer only."
if classification_basis == "source_layer_preserved"
else "Semantic object type derived from class_name/shape_name."
)
)
inserts.append(
(
object_type,
canonical_family,
classification_basis,
source_layer_scope,
source_layer_count,
geom_type_count,
object_type_sources,
source_layers,
",".join(observed_geom_list) or None,
",".join(allowed_geom_types) or None,
preferred_geom_type,
geometry_rule_source,
notes,
)
)
cur.executemany(
"""
INSERT INTO canonical_object_rules (
canonical_object_type,
canonical_family,
classification_basis,
source_layer_scope,
source_layer_count,
geom_type_count,
object_type_sources,
source_layers,
observed_geom_types,
allowed_geom_types,
preferred_geom_type,
geometry_rule_source,
notes
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
inserts,
)
def build_object_geometry_allowlist(cur):
cur.execute("DROP TABLE IF EXISTS object_geometry_allowlist")
cur.execute(
"""
CREATE TABLE object_geometry_allowlist (
canonical_object_type VARCHAR(191) NOT NULL,
geometry_type VARCHAR(20) NOT NULL,
rule_source VARCHAR(50) NOT NULL,
is_allowed TINYINT(1) NOT NULL DEFAULT 1,
notes TEXT NULL,
PRIMARY KEY (canonical_object_type, geometry_type)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
"""
SELECT
canonical_object_type,
geometry_rule_source,
allowed_geom_types,
notes
FROM canonical_object_rules
"""
)
inserts = []
for canonical_object_type, geometry_rule_source, allowed_geom_types, notes in cur.fetchall():
for geometry_type in [part for part in (allowed_geom_types or "").split(",") if part]:
inserts.append(
(
canonical_object_type,
geometry_type,
geometry_rule_source,
1,
notes,
)
)
if inserts:
cur.executemany(
"""
INSERT INTO object_geometry_allowlist (
canonical_object_type,
geometry_type,
rule_source,
is_allowed,
notes
)
VALUES (%s, %s, %s, %s, %s)
""",
inserts,
)
def build_relayer_outputs(cur):
cur.execute("DROP TABLE IF EXISTS pbf_relayer_candidates")
cur.execute(
"""
CREATE TABLE pbf_relayer_candidates AS
SELECT
o.feature_id,
s.z,
s.x,
s.y,
s.vt_layer AS source_layer,
s.geom_type,
s.class_name,
s.shape_name,
s.layer_name,
o.object_type_source,
o.object_type AS canonical_object_type,
r.semantic_granularity,
obj.canonical_family,
obj.classification_basis,
obj.source_layer_scope,
r.render_strategy,
r.preserve_source_layer,
r.style_bound,
CASE
WHEN obj.classification_basis = 'source_layer_preserved' THEN CONCAT('source:', s.vt_layer)
ELSE CONCAT(obj.canonical_family, ':', o.object_type)
END AS semantic_key,
CASE
WHEN obj.classification_basis = 'source_layer_preserved' THEN CONCAT(obj.canonical_family, ':', s.vt_layer)
ELSE CONCAT(obj.canonical_family, ':', o.object_type)
END AS detection_key,
CASE
WHEN r.preserve_source_layer = 1 THEN s.vt_layer
ELSE o.object_type
END AS render_layer
FROM feature_semantic s
JOIN object_type_candidates o
ON s.feature_id = o.feature_id
JOIN canonical_layer_rules r
ON s.vt_layer = r.source_layer
JOIN canonical_object_rules obj
ON o.object_type = obj.canonical_object_type
"""
)
cur.execute("DROP TABLE IF EXISTS pbf_source_object_stats")
cur.execute(
"""
CREATE TABLE pbf_source_object_stats AS
SELECT
source_layer,
render_layer,
canonical_family,
classification_basis,
source_layer_scope,
semantic_granularity,
style_bound,
canonical_object_type,
geom_type,
detection_key,
semantic_key,
COUNT(*) AS feature_count
FROM pbf_relayer_candidates
GROUP BY
source_layer,
render_layer,
canonical_family,
classification_basis,
source_layer_scope,
semantic_granularity,
style_bound,
canonical_object_type,
geom_type,
detection_key,
semantic_key
"""
)
cur.execute("ALTER TABLE pbf_source_object_stats ADD KEY idx_source_layer (source_layer(100))")
cur.execute("ALTER TABLE pbf_source_object_stats ADD KEY idx_render_layer (render_layer(100))")
cur.execute("ALTER TABLE pbf_source_object_stats ADD KEY idx_detection_key (detection_key(50))")
cur.execute("DROP TABLE IF EXISTS pbf_render_compatibility")
cur.execute(
"""
CREATE TABLE pbf_render_compatibility AS
SELECT
s.source_layer,
s.render_layer,
s.canonical_family,
s.semantic_granularity,
s.style_bound,
SUM(s.feature_count) AS feature_count,
COUNT(*) AS canonical_object_types
FROM pbf_source_object_stats s
GROUP BY
s.source_layer,
s.render_layer,
s.canonical_family,
s.semantic_granularity,
s.style_bound
"""
)
cur.execute("DROP TABLE IF EXISTS pbf_detection_catalog")
cur.execute(
"""
CREATE TABLE pbf_detection_catalog AS
SELECT
detection_key,
canonical_family,
canonical_object_type,
geom_type,
SUM(feature_count) AS feature_count,
COUNT(DISTINCT source_layer) AS source_layers
FROM pbf_source_object_stats
GROUP BY
detection_key,
canonical_family,
canonical_object_type,
geom_type
"""
)
def refresh_relayer_tables():
style_layers = load_style_source_layers()
with connect() as conn:
with conn.cursor() as cur:
build_canonical_layer_rules(cur, style_layers)
build_canonical_object_rules(cur)
build_object_geometry_allowlist(cur)
build_relayer_outputs(cur)
conn.commit()
def main():
refresh_relayer_tables()
print(
"Built canonical_layer_rules, canonical_object_rules, object_geometry_allowlist, "
"pbf_relayer_candidates, pbf_render_compatibility, pbf_detection_catalog"
)
if __name__ == "__main__":
main()

997
navsea_render_audit.py Normal file
View File

@@ -0,0 +1,997 @@
#!/usr/bin/env python3
"""
NavSea render audit for original vs engineering tiles/styles.
This script compares one original style/tile set against one engineering style/tile set.
Audit goal:
- use legacy fid as the primary object identity
- fall back to geometry + stable legacy properties when fid is missing
- evaluate each style layer against decoded tile features
- extract per-object render observations (icon/text/line/fill)
- compare original and engineering render observations at tile-instance scope
- emit Markdown and JSON audit reports for review
- persist the latest audit result into MySQL for fid-level trace-back
Important scope note:
- render comparison is performed per tile feature instance
- identity is "fid first", but zoom/tile instance is preserved because rendering is zoom-sensitive
"""
from __future__ import annotations
import argparse
import hashlib
import json
from collections import Counter, defaultdict
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import mapbox_vector_tile
import pymysql
DEFAULT_ORIGINAL_STYLE_PATH = Path("/mnt/sda1/www/newpec/style.patched.local.json")
DEFAULT_ENGINEERING_STYLE_PATH = Path("/mnt/sda1/www/newpec/navsea-engineering.json")
DEFAULT_ORIGINAL_TILE_ROOT = Path(
"/home/wwwroot/newpec/exported_auto/"
"tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles"
)
DEFAULT_ENGINEERING_TILE_ROOT = Path("/home/wwwroot/pbf-engineering-karatsu-10nm")
DEFAULT_REPORT_MD_PATH = Path("/root/weather/NavSea_Original_vs_Engineering_Render_Audit_Karatsu_10nm.md")
DEFAULT_REPORT_JSON_PATH = Path("/root/weather/NavSea_Original_vs_Engineering_Render_Audit_Karatsu_10nm.json")
DEFAULT_AUDIT_NAME = "navsea_original_vs_engineering_karatsu_10nm"
DB_BATCH_SIZE = 1000
MAX_EXAMPLES = 30
ORIGINAL_STYLE_PATH = DEFAULT_ORIGINAL_STYLE_PATH
ENGINEERING_STYLE_PATH = DEFAULT_ENGINEERING_STYLE_PATH
ORIGINAL_TILE_ROOT = DEFAULT_ORIGINAL_TILE_ROOT
ENGINEERING_TILE_ROOT = DEFAULT_ENGINEERING_TILE_ROOT
REPORT_MD_PATH = DEFAULT_REPORT_MD_PATH
REPORT_JSON_PATH = DEFAULT_REPORT_JSON_PATH
AUDIT_NAME = DEFAULT_AUDIT_NAME
DERIVED_PROPERTY_PREFIXES = (
"canonical_",
"semantic_",
"detection_",
"render_",
"chart_",
"light_",
"hazard_",
"area_",
"source_layer_",
)
DERIVED_PROPERTY_KEYS = {
"fid_algo_id",
"fid_key_id",
"fid_navsea_int",
"fid_legacy_raw",
"normalization_bundle_id",
"source_layer_rule_id",
"trace_status",
"feature_id",
"depth_value_m",
"clearance_height_m",
"least_depth_m",
"bearing_deg",
}
STABLE_FALLBACK_KEYS = (
"分類番号",
"形状分類番号",
"表示用番号",
"灯色",
"灯略記",
"明弧/分孤",
"表示位置",
"名称",
"名称補助",
"日本語地名",
"英文字地名",
"水深値(m)",
"高さ(m)",
"高さ/深度(m)",
"角度",
)
EXPRESSION_OPS = {
"get",
"match",
"coalesce",
"concat",
"number",
"literal",
"rgba",
"case",
"has",
"any",
"all",
"==",
"!=",
">=",
"<=",
">",
"<",
"/",
"*",
"interpolate",
"step",
"zoom",
}
@dataclass(frozen=True)
class DbConfig:
host: str = "localhost"
port: int = 3306
user: str = "root"
password: str = "2chi9ks2"
database: str = "pbf_analysis"
unix_socket: str | None = "/tmp/mysql.sock"
@dataclass(frozen=True)
class AuditFeature:
dataset: str
tile_z: int
tile_x: int
tile_y: int
layer_name: str
geom_type: str
object_id: str
fid_legacy: str | None
properties: dict[str, Any]
@dataclass(frozen=True)
class RenderObservation:
component_type: str
style_layer_id: str
signature: str
payload: dict[str, Any]
def canonical_json(value: Any) -> str:
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
def hash_text(text: str) -> str:
return hashlib.sha1(text.encode("utf-8")).hexdigest()[:16]
def text_or_none(value: Any) -> str | None:
if value is None:
return None
text = str(value).strip()
return text or None
def numeric_or_text(value: Any) -> float | str | None:
if value is None:
return None
if isinstance(value, (int, float)):
return float(value)
text = text_or_none(value)
if text is None:
return None
try:
return float(text)
except ValueError:
return text
def normalize_color(value: Any) -> str | None:
if value is None:
return None
if isinstance(value, str):
return value
if isinstance(value, list):
return canonical_json(value)
return str(value)
def compare_numbers(lhs: Any, rhs: Any, op: str) -> bool:
lhs_num = numeric_or_text(lhs)
rhs_num = numeric_or_text(rhs)
if isinstance(lhs_num, float) and isinstance(rhs_num, float):
if op == ">=":
return lhs_num >= rhs_num
if op == "<=":
return lhs_num <= rhs_num
if op == ">":
return lhs_num > rhs_num
if op == "<":
return lhs_num < rhs_num
lhs_text = "" if lhs is None else str(lhs)
rhs_text = "" if rhs is None else str(rhs)
if op == ">=":
return lhs_text >= rhs_text
if op == "<=":
return lhs_text <= rhs_text
if op == ">":
return lhs_text > rhs_text
return lhs_text < rhs_text
def evaluate_expression(expr: Any, feature: AuditFeature, zoom: int) -> Any:
if not isinstance(expr, list):
return expr
if not expr:
return expr
head = expr[0]
if not isinstance(head, str) or head not in EXPRESSION_OPS:
return [evaluate_expression(item, feature, zoom) for item in expr]
if head == "get":
key = expr[1]
return feature.properties.get(key)
if head == "literal":
return expr[1]
if head == "coalesce":
for item in expr[1:]:
value = evaluate_expression(item, feature, zoom)
if value not in (None, ""):
return value
return None
if head == "concat":
return "".join("" if (value := evaluate_expression(item, feature, zoom)) is None else str(value) for item in expr[1:])
if head == "number":
value = evaluate_expression(expr[1], feature, zoom)
parsed = numeric_or_text(value)
if isinstance(parsed, float):
return parsed
fallback = evaluate_expression(expr[2], feature, zoom) if len(expr) > 2 else None
return fallback
if head == "rgba":
values = [evaluate_expression(item, feature, zoom) for item in expr[1:5]]
return f"rgba({values[0]},{values[1]},{values[2]},{values[3]})"
if head == "has":
key = expr[1]
return key in feature.properties and feature.properties.get(key) not in (None, "")
if head == "zoom":
return zoom
if head in {"==", "!="}:
lhs = evaluate_expression(expr[1], feature, zoom)
rhs = evaluate_expression(expr[2], feature, zoom)
result = lhs == rhs
return result if head == "==" else not result
if head in {">=", "<=", ">", "<"}:
lhs = evaluate_expression(expr[1], feature, zoom)
rhs = evaluate_expression(expr[2], feature, zoom)
return compare_numbers(lhs, rhs, head)
if head == "any":
return any(bool(evaluate_expression(item, feature, zoom)) for item in expr[1:])
if head == "all":
return all(bool(evaluate_expression(item, feature, zoom)) for item in expr[1:])
if head == "case":
clauses = expr[1:]
for idx in range(0, len(clauses) - 1, 2):
if bool(evaluate_expression(clauses[idx], feature, zoom)):
return evaluate_expression(clauses[idx + 1], feature, zoom)
return evaluate_expression(clauses[-1], feature, zoom) if clauses else None
if head == "match":
value = evaluate_expression(expr[1], feature, zoom)
arms = expr[2:]
fallback = arms[-1] if arms else None
for idx in range(0, len(arms) - 1, 2):
label = arms[idx]
result = arms[idx + 1]
if isinstance(label, list):
if value in [evaluate_expression(item, feature, zoom) for item in label]:
return evaluate_expression(result, feature, zoom)
else:
if value == evaluate_expression(label, feature, zoom):
return evaluate_expression(result, feature, zoom)
return evaluate_expression(fallback, feature, zoom)
if head == "/":
lhs = evaluate_expression(expr[1], feature, zoom)
rhs = evaluate_expression(expr[2], feature, zoom)
lhs_num = numeric_or_text(lhs)
rhs_num = numeric_or_text(rhs)
if isinstance(lhs_num, float) and isinstance(rhs_num, float) and rhs_num != 0:
return lhs_num / rhs_num
return None
if head == "*":
lhs = evaluate_expression(expr[1], feature, zoom)
rhs = evaluate_expression(expr[2], feature, zoom)
lhs_num = numeric_or_text(lhs)
rhs_num = numeric_or_text(rhs)
if isinstance(lhs_num, float) and isinstance(rhs_num, float):
return lhs_num * rhs_num
return None
if head == "step":
input_value = evaluate_expression(expr[1], feature, zoom)
input_num = numeric_or_text(input_value)
if not isinstance(input_num, float):
return evaluate_expression(expr[2], feature, zoom)
result = evaluate_expression(expr[2], feature, zoom)
stops = expr[3:]
for idx in range(0, len(stops), 2):
stop = evaluate_expression(stops[idx], feature, zoom)
stop_num = numeric_or_text(stop)
if not isinstance(stop_num, float):
continue
if idx + 1 >= len(stops):
break
if input_num >= stop_num:
result = evaluate_expression(stops[idx + 1], feature, zoom)
else:
break
return result
if head == "interpolate":
input_value = evaluate_expression(expr[2], feature, zoom)
input_num = numeric_or_text(input_value)
if not isinstance(input_num, float):
return None
stops = expr[3:]
prev_stop = None
prev_value = None
for idx in range(0, len(stops), 2):
stop_num = numeric_or_text(evaluate_expression(stops[idx], feature, zoom))
stop_value = evaluate_expression(stops[idx + 1], feature, zoom) if idx + 1 < len(stops) else None
if not isinstance(stop_num, float):
continue
if input_num == stop_num:
return stop_value
if input_num < stop_num:
return prev_value if prev_value is not None else stop_value
prev_stop = stop_num
prev_value = stop_value
return prev_value
return None
def layer_visible(layer: dict[str, Any], zoom: int) -> bool:
if zoom < int(layer.get("minzoom", 0)):
return False
maxzoom = layer.get("maxzoom")
if maxzoom is not None and zoom >= int(maxzoom):
return False
layout = layer.get("layout") or {}
if layout.get("visibility") == "none":
return False
return True
def layer_matches_feature(layer: dict[str, Any], feature: AuditFeature) -> bool:
if layer.get("source-layer") != feature.layer_name:
return False
if not layer_visible(layer, feature.tile_z):
return False
filter_expr = layer.get("filter")
if filter_expr is None:
return True
return bool(evaluate_expression(filter_expr, feature, feature.tile_z))
def evaluate_style_value(layer: dict[str, Any], section: str, key: str, feature: AuditFeature) -> Any:
payload = layer.get(section) or {}
if key not in payload:
return None
return evaluate_expression(payload[key], feature, feature.tile_z)
def build_signature(component_type: str, payload: dict[str, Any]) -> str:
normalized = {
k: payload[k]
for k in sorted(payload)
if k != "layer_id" and payload[k] not in (None, "", [], {})
}
return f"{component_type}:{canonical_json(normalized)}"
def collect_layer_observations(layer: dict[str, Any], feature: AuditFeature) -> list[RenderObservation]:
observations: list[RenderObservation] = []
layer_id = str(layer["id"])
layer_type = str(layer.get("type", ""))
if layer_type == "symbol":
icon_image = evaluate_style_value(layer, "layout", "icon-image", feature)
text_value = evaluate_style_value(layer, "layout", "text-field", feature)
text_color = evaluate_style_value(layer, "paint", "text-color", feature)
text_anchor = evaluate_style_value(layer, "layout", "text-anchor", feature)
text_size = evaluate_style_value(layer, "layout", "text-size", feature)
icon_size = evaluate_style_value(layer, "layout", "icon-size", feature)
if icon_image not in (None, ""):
payload = {
"layer_id": layer_id,
"icon_image": icon_image,
"icon_size": icon_size,
}
observations.append(
RenderObservation(
component_type="icon",
style_layer_id=layer_id,
signature=build_signature("icon", payload),
payload=payload,
)
)
if text_value not in (None, ""):
payload = {
"layer_id": layer_id,
"text_value": str(text_value),
"text_color": normalize_color(text_color),
"text_anchor": text_anchor,
"text_size": text_size,
}
observations.append(
RenderObservation(
component_type="text",
style_layer_id=layer_id,
signature=build_signature("text", payload),
payload=payload,
)
)
return observations
if layer_type == "line":
payload = {
"layer_id": layer_id,
"line_color": normalize_color(evaluate_style_value(layer, "paint", "line-color", feature)),
"line_width": evaluate_style_value(layer, "paint", "line-width", feature),
"line_dasharray": evaluate_style_value(layer, "paint", "line-dasharray", feature),
"line_pattern": evaluate_style_value(layer, "paint", "line-pattern", feature),
}
observations.append(
RenderObservation(
component_type="line",
style_layer_id=layer_id,
signature=build_signature("line", payload),
payload=payload,
)
)
return observations
if layer_type == "fill":
payload = {
"layer_id": layer_id,
"fill_color": normalize_color(evaluate_style_value(layer, "paint", "fill-color", feature)),
"fill_pattern": evaluate_style_value(layer, "paint", "fill-pattern", feature),
"fill_outline_color": normalize_color(evaluate_style_value(layer, "paint", "fill-outline-color", feature)),
"fill_opacity": evaluate_style_value(layer, "paint", "fill-opacity", feature),
}
observations.append(
RenderObservation(
component_type="fill",
style_layer_id=layer_id,
signature=build_signature("fill", payload),
payload=payload,
)
)
return observations
return observations
def collect_render_observations(style: dict[str, Any], feature: AuditFeature) -> list[RenderObservation]:
observations: list[RenderObservation] = []
for layer in style.get("layers", []):
if "source-layer" not in layer:
continue
if not layer_matches_feature(layer, feature):
continue
observations.extend(collect_layer_observations(layer, feature))
return observations
def load_style(path: Path) -> dict[str, Any]:
return json.loads(path.read_text(encoding="utf-8"))
def iter_tile_paths(root: Path) -> list[Path]:
return sorted(root.glob("*/*/*.pbf"))
def stable_fallback_properties(properties: dict[str, Any]) -> dict[str, Any]:
return {
key: properties[key]
for key in STABLE_FALLBACK_KEYS
if key in properties and properties[key] not in (None, "")
}
def geometry_hash(geometry: dict[str, Any]) -> str:
return hash_text(canonical_json(geometry))
def extract_legacy_fid(properties: dict[str, Any]) -> str | None:
legacy = properties.get("fid_legacy_raw")
if legacy not in (None, ""):
return str(legacy)
fid = properties.get("fid")
if fid not in (None, ""):
return str(fid)
return None
def object_identity(layer_name: str, geometry: dict[str, Any], properties: dict[str, Any]) -> tuple[str, str | None]:
geom_type = str(geometry.get("type", ""))
legacy_source_layer = text_or_none(properties.get("source_layer_jp")) or layer_name
legacy_fid = extract_legacy_fid(properties)
if legacy_fid:
return f"fid:{legacy_fid}|src:{legacy_source_layer}|geom:{geom_type}", legacy_fid
fallback_payload = {
"source_layer": legacy_source_layer,
"geom_type": geom_type,
"geometry_hash": geometry_hash(geometry),
"stable_props": stable_fallback_properties(properties),
}
return f"fallback:{hash_text(canonical_json(fallback_payload))}", None
def feature_instance_id(feature: AuditFeature) -> str:
return (
f"{feature.object_id}|z:{feature.tile_z}|x:{feature.tile_x}|"
f"y:{feature.tile_y}|layer:{feature.layer_name}"
)
def decode_tile_instances(dataset: str, tile_root: Path, style: dict[str, Any], tile_path: Path) -> dict[str, dict[str, Any]]:
instances: dict[str, dict[str, Any]] = {}
rel = tile_path.relative_to(tile_root)
z = int(rel.parts[0])
x = int(rel.parts[1])
y = int(tile_path.stem)
decoded = mapbox_vector_tile.decode(tile_path.read_bytes())
for layer_name, payload in decoded.items():
for feature in payload.get("features", []):
geometry = feature.get("geometry") or {}
properties = dict(feature.get("properties") or {})
object_id, legacy_fid = object_identity(layer_name, geometry, properties)
audit_feature = AuditFeature(
dataset=dataset,
tile_z=z,
tile_x=x,
tile_y=y,
layer_name=layer_name,
geom_type=str(geometry.get("type", "")),
object_id=object_id,
fid_legacy=legacy_fid,
properties=properties,
)
instance_id = feature_instance_id(audit_feature)
instances[instance_id] = {
"feature": audit_feature,
"observations": collect_render_observations(style, audit_feature),
}
return instances
def summarize_observations(observations: list[RenderObservation]) -> dict[str, list[str]]:
grouped: dict[str, set[str]] = defaultdict(set)
for observation in observations:
grouped[observation.component_type].add(observation.signature)
return {key: sorted(values) for key, values in grouped.items()}
def normalize_component_map(component_map: dict[str, list[str]]) -> dict[str, tuple[str, ...]]:
return {key: tuple(values) for key, values in sorted(component_map.items())}
def compare_instance(
original: dict[str, Any] | None,
engineering: dict[str, Any] | None,
) -> dict[str, Any]:
if original is None:
eng_feature = engineering["feature"]
return {
"status": "extra_in_engineering",
"object_instance_id": feature_instance_id(eng_feature),
"object_id": eng_feature.object_id,
"fid_legacy": eng_feature.fid_legacy,
"tile": f"{eng_feature.tile_z}/{eng_feature.tile_x}/{eng_feature.tile_y}",
"tile_z": eng_feature.tile_z,
"tile_x": eng_feature.tile_x,
"tile_y": eng_feature.tile_y,
"source_layer": eng_feature.layer_name,
"canonical_object_type": text_or_none(eng_feature.properties.get("canonical_object_type")),
"original_components": {},
"engineering_components": normalize_component_map(
summarize_observations(engineering["observations"])
),
}
if engineering is None:
orig_feature = original["feature"]
return {
"status": "missing_in_engineering",
"object_instance_id": feature_instance_id(orig_feature),
"object_id": orig_feature.object_id,
"fid_legacy": orig_feature.fid_legacy,
"tile": f"{orig_feature.tile_z}/{orig_feature.tile_x}/{orig_feature.tile_y}",
"tile_z": orig_feature.tile_z,
"tile_x": orig_feature.tile_x,
"tile_y": orig_feature.tile_y,
"source_layer": orig_feature.layer_name,
"canonical_object_type": None,
"original_components": normalize_component_map(
summarize_observations(original["observations"])
),
"engineering_components": {},
}
orig_feature = original["feature"]
eng_feature = engineering["feature"]
original_components = normalize_component_map(summarize_observations(original["observations"]))
engineering_components = normalize_component_map(summarize_observations(engineering["observations"]))
if original_components == engineering_components:
status = "exact_match"
elif not original_components and engineering_components:
status = "extra_in_engineering"
elif original_components and not engineering_components:
status = "missing_in_engineering"
else:
status = "mismatch"
return {
"status": status,
"object_instance_id": feature_instance_id(orig_feature),
"object_id": orig_feature.object_id,
"fid_legacy": orig_feature.fid_legacy or eng_feature.fid_legacy,
"tile": f"{orig_feature.tile_z}/{orig_feature.tile_x}/{orig_feature.tile_y}",
"tile_z": orig_feature.tile_z,
"tile_x": orig_feature.tile_x,
"tile_y": orig_feature.tile_y,
"source_layer": text_or_none(eng_feature.properties.get("source_layer_jp")) or orig_feature.layer_name,
"canonical_object_type": text_or_none(eng_feature.properties.get("canonical_object_type")),
"original_components": original_components,
"engineering_components": engineering_components,
}
def format_component_map(component_map: dict[str, tuple[str, ...]] | dict[str, list[str]]) -> str:
if not component_map:
return "none"
parts = []
for key in sorted(component_map):
values = component_map[key]
parts.append(f"{key}={list(values)}")
return "; ".join(parts)
def write_reports(
*,
original_count: int,
engineering_count: int,
result_count: int,
status_counter: Counter[str],
source_layer_counter: Counter[tuple[str, str]],
mismatch_examples: list[dict[str, Any]],
) -> None:
payload = {
"original_feature_instances": original_count,
"engineering_feature_instances": engineering_count,
"result_count": result_count,
"status_counts": dict(status_counter),
"source_layer_issue_counts": [
{"status": status, "source_layer": source_layer, "count": count}
for (status, source_layer), count in source_layer_counter.most_common(30)
],
"mismatch_examples": mismatch_examples,
}
REPORT_JSON_PATH.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
lines = [
"# NavSea 原始版 vs 工程版渲染审计报告",
"",
"## 范围",
"",
f"- 原始样式: `{ORIGINAL_STYLE_PATH}`",
f"- 工程样式: `{ENGINEERING_STYLE_PATH}`",
f"- 原始瓦片根目录: `{ORIGINAL_TILE_ROOT}`",
f"- 工程瓦片根目录: `{ENGINEERING_TILE_ROOT}`",
f"- 原始 feature 实例数: `{original_count}`",
f"- 工程 feature 实例数: `{engineering_count}`",
f"- 审计结果数: `{result_count}`",
"",
"## 匹配口径",
"",
"- 主键优先使用 legacy `fid`。",
"- 没有 `fid` 的对象，回退到 `geometry + 稳定旧属性`。",
"- 审计粒度保留 tile 实例，因为渲染具有 zoom 敏感性。",
"",
"## 结果统计",
"",
]
for status, count in status_counter.most_common():
lines.append(f"- `{status}`: `{count}`")
lines.extend([
"",
"## 主要问题层",
"",
])
if not source_layer_counter:
lines.append("- 没有发现差异。")
else:
for (status, source_layer), count in source_layer_counter.most_common(20):
lines.append(f"- `{status}` | `{source_layer}` | `{count}`")
lines.extend([
"",
"## 差异样例",
"",
])
if not mismatch_examples:
lines.append("- 没有差异样例。")
else:
for item in mismatch_examples:
lines.append(
f"- `{item['status']}` | tile=`{item['tile']}` | source_layer=`{item['source_layer']}` | "
f"fid=`{item['fid_legacy']}` | object=`{item['canonical_object_type'] or 'n/a'}`"
)
lines.append(f" original: {format_component_map(item['original_components'])}")
lines.append(f" engineering: {format_component_map(item['engineering_components'])}")
REPORT_MD_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")
def db_connect() -> pymysql.Connection:
cfg = DbConfig()
kwargs: dict[str, Any] = {
"host": cfg.host,
"port": cfg.port,
"user": cfg.user,
"password": cfg.password,
"database": cfg.database,
"charset": "utf8mb4",
"autocommit": False,
}
if cfg.unix_socket:
kwargs["unix_socket"] = cfg.unix_socket
return pymysql.connect(**kwargs)
def ensure_audit_tables(cur: pymysql.cursors.Cursor) -> None:
cur.execute(
"""
CREATE TABLE IF NOT EXISTS navsea_render_audit_run (
audit_name VARCHAR(128) NOT NULL,
original_style_path VARCHAR(512) NOT NULL,
engineering_style_path VARCHAR(512) NOT NULL,
original_tile_root VARCHAR(512) NOT NULL,
engineering_tile_root VARCHAR(512) NOT NULL,
original_feature_instances INT NOT NULL,
engineering_feature_instances INT NOT NULL,
result_count INT NOT NULL,
status_counts_json LONGTEXT NOT NULL,
source_layer_issue_counts_json LONGTEXT NOT NULL,
report_md_path VARCHAR(512) NOT NULL,
report_json_path VARCHAR(512) NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (audit_name)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
"""
CREATE TABLE IF NOT EXISTS navsea_render_audit_result (
audit_name VARCHAR(64) NOT NULL,
object_instance_key VARCHAR(64) NOT NULL,
object_instance_id TEXT NOT NULL,
object_id VARCHAR(255) NOT NULL,
fid_legacy VARCHAR(64) DEFAULT NULL,
tile_z INT NOT NULL,
tile_x INT NOT NULL,
tile_y INT NOT NULL,
source_layer VARCHAR(100) DEFAULT NULL,
canonical_object_type VARCHAR(191) DEFAULT NULL,
status VARCHAR(32) NOT NULL,
original_components_json LONGTEXT NOT NULL,
engineering_components_json LONGTEXT NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (audit_name, object_instance_key),
KEY idx_render_audit_fid (fid_legacy),
KEY idx_render_audit_status (status),
KEY idx_render_audit_layer (source_layer)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
def insert_result_batch(cur: pymysql.cursors.Cursor, rows: list[tuple[Any, ...]]) -> None:
if not rows:
return
cur.executemany(
"""
INSERT INTO navsea_render_audit_result (
audit_name,
object_instance_key,
object_instance_id,
object_id,
fid_legacy,
tile_z,
tile_x,
tile_y,
source_layer,
canonical_object_type,
status,
original_components_json,
engineering_components_json
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
rows,
)
def persist_audit_run(
cur: pymysql.cursors.Cursor,
*,
original_count: int,
engineering_count: int,
result_count: int,
status_counter: Counter[str],
source_layer_counter: Counter[tuple[str, str]],
) -> None:
source_layer_issue_counts = [
{"status": status, "source_layer": source_layer, "count": count}
for (status, source_layer), count in source_layer_counter.most_common(100)
]
cur.execute(
"""
INSERT INTO navsea_render_audit_run (
audit_name,
original_style_path,
engineering_style_path,
original_tile_root,
engineering_tile_root,
original_feature_instances,
engineering_feature_instances,
result_count,
status_counts_json,
source_layer_issue_counts_json,
report_md_path,
report_json_path
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(
AUDIT_NAME,
str(ORIGINAL_STYLE_PATH),
str(ENGINEERING_STYLE_PATH),
str(ORIGINAL_TILE_ROOT),
str(ENGINEERING_TILE_ROOT),
original_count,
engineering_count,
result_count,
json.dumps(dict(status_counter), ensure_ascii=False),
json.dumps(source_layer_issue_counts, ensure_ascii=False),
str(REPORT_MD_PATH),
str(REPORT_JSON_PATH),
),
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Audit original vs engineering render outputs.")
parser.add_argument("--audit-name", default=DEFAULT_AUDIT_NAME)
parser.add_argument("--original-style", type=Path, default=DEFAULT_ORIGINAL_STYLE_PATH)
parser.add_argument("--engineering-style", type=Path, default=DEFAULT_ENGINEERING_STYLE_PATH)
parser.add_argument("--original-tile-root", type=Path, default=DEFAULT_ORIGINAL_TILE_ROOT)
parser.add_argument("--engineering-tile-root", type=Path, default=DEFAULT_ENGINEERING_TILE_ROOT)
parser.add_argument("--report-md", type=Path, default=DEFAULT_REPORT_MD_PATH)
parser.add_argument("--report-json", type=Path, default=DEFAULT_REPORT_JSON_PATH)
return parser.parse_args()
def main() -> None:
global AUDIT_NAME
global ORIGINAL_STYLE_PATH
global ENGINEERING_STYLE_PATH
global ORIGINAL_TILE_ROOT
global ENGINEERING_TILE_ROOT
global REPORT_MD_PATH
global REPORT_JSON_PATH
args = parse_args()
AUDIT_NAME = args.audit_name
ORIGINAL_STYLE_PATH = args.original_style
ENGINEERING_STYLE_PATH = args.engineering_style
ORIGINAL_TILE_ROOT = args.original_tile_root
ENGINEERING_TILE_ROOT = args.engineering_tile_root
REPORT_MD_PATH = args.report_md
REPORT_JSON_PATH = args.report_json
original_style = load_style(ORIGINAL_STYLE_PATH)
engineering_style = load_style(ENGINEERING_STYLE_PATH)
engineering_tiles = iter_tile_paths(ENGINEERING_TILE_ROOT)
original_count = 0
engineering_count = 0
result_count = 0
status_counter: Counter[str] = Counter()
source_layer_counter: Counter[tuple[str, str]] = Counter()
mismatch_examples: list[dict[str, Any]] = []
with db_connect() as conn:
with conn.cursor() as cur:
ensure_audit_tables(cur)
cur.execute("DELETE FROM navsea_render_audit_result")
cur.execute("DELETE FROM navsea_render_audit_run")
conn.commit()
rows: list[tuple[Any, ...]] = []
for engineering_tile in engineering_tiles:
rel = engineering_tile.relative_to(ENGINEERING_TILE_ROOT)
original_tile = ORIGINAL_TILE_ROOT / rel
if not original_tile.exists():
continue
original_instances = decode_tile_instances("original", ORIGINAL_TILE_ROOT, original_style, original_tile)
engineering_instances = decode_tile_instances(
"engineering", ENGINEERING_TILE_ROOT, engineering_style, engineering_tile
)
original_count += len(original_instances)
engineering_count += len(engineering_instances)
all_instance_ids = sorted(set(original_instances) | set(engineering_instances))
for instance_id in all_instance_ids:
item = compare_instance(original_instances.get(instance_id), engineering_instances.get(instance_id))
result_count += 1
status_counter[item["status"]] += 1
if item["status"] != "exact_match":
source_layer_counter[(item["status"], item["source_layer"] or "unknown")] += 1
if len(mismatch_examples) < MAX_EXAMPLES:
mismatch_examples.append(item)
rows.append(
(
AUDIT_NAME,
hash_text(item["object_instance_id"]),
item["object_instance_id"],
item["object_id"],
item["fid_legacy"],
item["tile_z"],
item["tile_x"],
item["tile_y"],
item["source_layer"],
item["canonical_object_type"],
item["status"],
json.dumps(item["original_components"], ensure_ascii=False),
json.dumps(item["engineering_components"], ensure_ascii=False),
)
)
if len(rows) >= DB_BATCH_SIZE:
insert_result_batch(cur, rows)
conn.commit()
rows.clear()
insert_result_batch(cur, rows)
persist_audit_run(
cur,
original_count=original_count,
engineering_count=engineering_count,
result_count=result_count,
status_counter=status_counter,
source_layer_counter=source_layer_counter,
)
conn.commit()
write_reports(
original_count=original_count,
engineering_count=engineering_count,
result_count=result_count,
status_counter=status_counter,
source_layer_counter=source_layer_counter,
mismatch_examples=mismatch_examples,
)
print(
json.dumps(
{
"original_feature_instances": original_count,
"engineering_feature_instances": engineering_count,
"results": result_count,
"report_md": str(REPORT_MD_PATH),
"report_json": str(REPORT_JSON_PATH),
"audit_name": AUDIT_NAME,
"db_run_table": "navsea_render_audit_run",
"db_result_table": "navsea_render_audit_result",
},
ensure_ascii=False,
indent=2,
)
)
if __name__ == "__main__":
main()

440
navsea_semantic_validation.py Normal file
View File

@@ -0,0 +1,440 @@
from dataclasses import dataclass
import os
import pymysql
@dataclass(frozen=True)
class DbConfig:
host: str = os.getenv("NAVSEA_DB_HOST", "localhost")
port: int = int(os.getenv("NAVSEA_DB_PORT", "3306"))
user: str = os.getenv("NAVSEA_DB_USER", "root")
password: str = os.getenv("NAVSEA_DB_PASSWORD", "2chi9ks2")
database: str = os.getenv("NAVSEA_DB_NAME", "pbf_analysis")
unix_socket: str | None = os.getenv("NAVSEA_DB_SOCKET", "/tmp/mysql.sock")
class SemanticOverlayValidator:
REQUIRED_TABLES = {
"canonical_layer_rules": {"source_layer", "semantic_granularity", "canonical_family"},
"canonical_object_rules": {
"canonical_object_type",
"canonical_family",
"classification_basis",
"source_layer_scope",
"allowed_geom_types",
},
"object_geometry_allowlist": {
"canonical_object_type",
"geometry_type",
"is_allowed",
},
"pbf_relayer_candidates": {
"feature_id",
"source_layer",
"geom_type",
"canonical_object_type",
"canonical_family",
"classification_basis",
"source_layer_scope",
"semantic_granularity",
"detection_key",
"render_layer",
"object_type_source",
"style_bound",
"z",
"x",
"y",
},
"pbf_render_compatibility": {
"source_layer",
"render_layer",
"feature_count",
},
"pbf_source_object_stats": {
"source_layer",
"render_layer",
"canonical_family",
"semantic_granularity",
"canonical_object_type",
"geom_type",
"feature_count",
},
"pbf_detection_catalog": {
"detection_key",
"canonical_object_type",
"canonical_family",
"geom_type",
"feature_count",
},
"style_layers": {"source_layer"},
"features": {"id", "z", "x", "y", "vt_layer", "geom_type"},
"tile_density": {"z", "x", "y", "feature_count"},
"tile_layer_density": {"z", "x", "y", "vt_layer", "feature_count"},
}
def __init__(self, db_config: DbConfig) -> None:
self.db_config = db_config
def connect(self):
kwargs = {
"host": self.db_config.host,
"port": self.db_config.port,
"user": self.db_config.user,
"password": self.db_config.password,
"database": self.db_config.database,
"charset": "utf8mb4",
"autocommit": False,
}
if self.db_config.unix_socket and self.db_config.host in {"localhost", "127.0.0.1"}:
kwargs["unix_socket"] = self.db_config.unix_socket
return pymysql.connect(**kwargs)
def validate_schema(self) -> None:
with self.connect() as conn:
with conn.cursor() as cur:
for table, required_columns in self.REQUIRED_TABLES.items():
cur.execute(
"""
SELECT COLUMN_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA=%s AND TABLE_NAME=%s
""",
(self.db_config.database, table),
)
columns = {row[0] for row in cur.fetchall()}
if not columns:
raise RuntimeError(f"required table missing: {table}")
missing = required_columns - columns
if missing:
raise RuntimeError(f"table {table} missing columns: {sorted(missing)}")
def run(self) -> None:
self.validate_schema()
with self.connect() as conn:
with conn.cursor() as cur:
self.drop_validation_tables(cur)
self.build_classification_validation(cur)
self.build_geometry_consistency(cur)
self.build_style_render_equivalence(cur)
self.build_detection_catalog_integrity(cur)
self.build_spatial_anomalies(cur)
self.build_summary(cur)
conn.commit()
print(
"Built classification_validation, geometry_consistency, "
"style_render_equivalence, detection_catalog_integrity, "
"spatial_anomalies, semantic_validation_summary"
)
@staticmethod
def drop_validation_tables(cur) -> None:
for table in (
"classification_validation",
"geometry_consistency",
"style_render_equivalence",
"detection_catalog_integrity",
"spatial_anomalies",
"semantic_validation_summary",
):
cur.execute(f"DROP TABLE IF EXISTS {table}")
@staticmethod
def build_classification_validation(cur) -> None:
cur.execute(
"""
CREATE TABLE classification_validation AS
SELECT
base.canonical_object_type,
base.feature_count,
base.distinct_source_layers,
base.distinct_geom_types,
CASE
WHEN base.canonical_object_type IS NULL OR base.canonical_object_type = '' THEN 'UNKNOWN_OBJECT'
WHEN rules.canonical_object_type IS NULL THEN 'UNKNOWN_OBJECT'
WHEN rules.classification_basis = 'source_layer_preserved' THEN 'OK'
WHEN base.distinct_families > 1 THEN 'RULE_CONFLICT'
WHEN rules.source_layer_scope = 'single_source' AND base.distinct_source_layers > 1 THEN 'AMBIGUOUS_MAPPING'
ELSE 'OK'
END AS classification_status
FROM (
SELECT
canonical_object_type,
SUM(feature_count) AS feature_count,
COUNT(DISTINCT source_layer) AS distinct_source_layers,
COUNT(DISTINCT geom_type) AS distinct_geom_types,
COUNT(DISTINCT canonical_family) AS distinct_families
FROM pbf_source_object_stats
GROUP BY canonical_object_type
) AS base
LEFT JOIN canonical_object_rules rules
ON base.canonical_object_type = rules.canonical_object_type
"""
)
cur.execute("ALTER TABLE classification_validation ADD KEY idx_status (classification_status)")
cur.execute("ALTER TABLE classification_validation ADD KEY idx_object (canonical_object_type(100))")
@staticmethod
def build_geometry_consistency(cur) -> None:
cur.execute(
"""
CREATE TABLE geometry_consistency AS
SELECT
g.canonical_object_type,
g.geom_type AS geometry_type,
g.feature_count,
CASE
WHEN allowlist.is_allowed = 1 THEN 'OK'
WHEN g.feature_share < 0.05 THEN 'GEOMETRY_DRIFT'
ELSE 'UNEXPECTED_GEOMETRY'
END AS geometry_consistency_status
FROM (
SELECT
canonical_object_type,
canonical_family,
geom_type,
SUM(feature_count) AS feature_count,
SUM(feature_count) / totals.total_features AS feature_share
FROM pbf_detection_catalog
JOIN (
SELECT
canonical_object_type,
SUM(feature_count) AS total_features
FROM pbf_detection_catalog
GROUP BY canonical_object_type
) AS totals
USING (canonical_object_type)
GROUP BY
canonical_object_type,
canonical_family,
geom_type,
totals.total_features
) AS g
LEFT JOIN object_geometry_allowlist allowlist
ON g.canonical_object_type = allowlist.canonical_object_type
AND g.geom_type = allowlist.geometry_type
"""
)
cur.execute("ALTER TABLE geometry_consistency ADD KEY idx_object (canonical_object_type(100))")
cur.execute("ALTER TABLE geometry_consistency ADD KEY idx_status (geometry_consistency_status)")
@staticmethod
def build_style_render_equivalence(cur) -> None:
cur.execute(
"""
CREATE TABLE style_render_equivalence AS
SELECT
style_scope.style_layer,
COALESCE(before_counts.feature_count_before, 0) AS feature_count_before,
COALESCE(after_counts.feature_count_after, 0) AS feature_count_after,
COALESCE(after_counts.feature_count_after, 0) - COALESCE(before_counts.feature_count_before, 0) AS difference,
CASE
WHEN COALESCE(before_counts.feature_count_before, 0) = COALESCE(after_counts.feature_count_after, 0) THEN 'OK'
WHEN COALESCE(before_counts.feature_count_before, 0) > 0 AND COALESCE(after_counts.feature_count_after, 0) = 0 THEN 'RENDER_BREAK_RISK'
ELSE 'COUNT_MISMATCH'
END AS render_status
FROM (
SELECT DISTINCT source_layer AS style_layer
FROM style_layers
WHERE source_layer IS NOT NULL
) AS style_scope
LEFT JOIN (
SELECT
vt_layer AS style_layer,
COUNT(*) AS feature_count_before
FROM features
GROUP BY vt_layer
) AS before_counts
ON style_scope.style_layer = before_counts.style_layer
LEFT JOIN (
SELECT
LEFT(render_layer, 100) AS style_layer,
SUM(feature_count) AS feature_count_after
FROM pbf_render_compatibility
GROUP BY LEFT(render_layer, 100)
) AS after_counts
ON style_scope.style_layer = after_counts.style_layer
"""
)
cur.execute("ALTER TABLE style_render_equivalence ADD KEY idx_status (render_status)")
cur.execute("ALTER TABLE style_render_equivalence ADD KEY idx_style_layer (style_layer)")
@staticmethod
def build_detection_catalog_integrity(cur) -> None:
cur.execute(
"""
CREATE TABLE detection_catalog_integrity AS
SELECT
d.detection_key,
SUM(d.feature_count) AS feature_count,
d.geom_type AS geometry_type,
CASE
WHEN d.detection_key IS NULL OR d.detection_key = '' THEN 'UNKNOWN_DETECTION_OBJECT'
WHEN COALESCE(object_meta.distinct_objects, 0) > 1 THEN 'UNKNOWN_DETECTION_OBJECT'
WHEN gc.geometry_consistency_status IS NOT NULL AND gc.geometry_consistency_status <> 'OK' THEN 'GEOMETRY_INCONSISTENT'
ELSE 'OK'
END AS catalog_status
FROM pbf_detection_catalog d
LEFT JOIN (
SELECT
detection_key,
COUNT(DISTINCT canonical_object_type) AS distinct_objects
FROM pbf_detection_catalog
GROUP BY detection_key
) AS object_meta
ON d.detection_key = object_meta.detection_key
LEFT JOIN geometry_consistency gc
ON d.canonical_object_type = gc.canonical_object_type
AND d.geom_type = gc.geometry_type
GROUP BY
d.detection_key,
d.geom_type,
object_meta.distinct_objects,
gc.geometry_consistency_status
"""
)
cur.execute("ALTER TABLE detection_catalog_integrity ADD KEY idx_status (catalog_status)")
cur.execute("ALTER TABLE detection_catalog_integrity ADD KEY idx_detection_key (detection_key(50))")
@staticmethod
def build_spatial_anomalies(cur) -> None:
cur.execute(
"""
CREATE TABLE spatial_anomalies AS
SELECT *
FROM (
SELECT
'OUT_OF_BOUNDS_TILE_INDEX' AS anomaly_type,
COUNT(*) AS affected_feature_count,
CONCAT('z=', MIN(z), '..', MAX(z), '; sample=', MIN(CONCAT(z, '/', x, '/', y))) AS region_hint,
'HIGH' AS severity
FROM pbf_relayer_candidates
WHERE z < 0
OR x < 0
OR y < 0
OR x >= POW(2, z)
OR y >= POW(2, z)
UNION ALL
SELECT
'DENSITY_STATISTICS_ERROR' AS anomaly_type,
tile_rows.feature_count AS affected_feature_count,
CONCAT('z=', tile_rows.z, '; tile=', tile_rows.z, '/', tile_rows.x, '/', tile_rows.y) AS region_hint,
'HIGH' AS severity
FROM tile_density AS tile_rows
CROSS JOIN (
SELECT COUNT(*) AS total_features
FROM features
) AS dataset_stats
WHERE tile_rows.feature_count > dataset_stats.total_features
UNION ALL
SELECT
CASE
WHEN layer_hint.vt_layer IN ('L海底地形', 'L等深線', 'L概略等深線', 'L海底線', 'p底質')
THEN 'TILE_OVERDENSE_BATHYMETRY'
WHEN layer_hint.vt_layer IN ('L陸上構造物陸', 'P陸上構造物陸', 'P橋りょう等構造物', 'p陸上構造物')
THEN 'TILE_OVERDENSE_LAND_STRUCTURE'
WHEN layer_hint.vt_layer LIKE '%ククリ'
OR layer_hint.vt_layer IN ('P基本線', 'L基本線', 'P危険界ククリ', 'P投錨注意障害物ククリ')
THEN 'TILE_OVERDENSE_OUTLINE'
ELSE 'EXTREME_TILE_DENSITY'
END AS anomaly_type,
tile_rows.feature_count AS affected_feature_count,
CONCAT(
'z=',
tile_rows.z,
'; tile=',
tile_rows.z,
'/',
tile_rows.x,
'/',
tile_rows.y,
'; top_layer=',
COALESCE(layer_hint.vt_layer, '<unknown>'),
'; layer_features=',
COALESCE(layer_hint.feature_count, 0)
) AS region_hint,
'MEDIUM' AS severity
FROM tile_density AS tile_rows
JOIN (
SELECT
z,
AVG(feature_count) AS avg_feature_count,
STD(feature_count) AS std_feature_count
FROM tile_density
GROUP BY z
) AS zoom_stats
ON tile_rows.z = zoom_stats.z
LEFT JOIN (
SELECT
winners.z,
winners.x,
winners.y,
MIN(winners.vt_layer) AS vt_layer,
winners.feature_count
FROM tile_layer_density AS winners
JOIN (
SELECT
z,
x,
y,
MAX(feature_count) AS feature_count
FROM tile_layer_density
GROUP BY z, x, y
) AS maxima
ON winners.z = maxima.z
AND winners.x = maxima.x
AND winners.y = maxima.y
AND winners.feature_count = maxima.feature_count
GROUP BY
winners.z,
winners.x,
winners.y,
winners.feature_count
) AS layer_hint
ON tile_rows.z = layer_hint.z
AND tile_rows.x = layer_hint.x
AND tile_rows.y = layer_hint.y
WHERE tile_rows.z >= 1
AND tile_rows.feature_count > (
zoom_stats.avg_feature_count + (5 * COALESCE(zoom_stats.std_feature_count, 0))
)
) AS anomaly_union
WHERE affected_feature_count IS NOT NULL AND affected_feature_count > 0
"""
)
cur.execute("ALTER TABLE spatial_anomalies ADD KEY idx_severity (severity)")
@staticmethod
def build_summary(cur) -> None:
cur.execute(
"""
CREATE TABLE semantic_validation_summary AS
SELECT
(SELECT COUNT(*) FROM pbf_relayer_candidates) AS total_features_checked,
(SELECT COUNT(*) FROM classification_validation WHERE classification_status <> 'OK') AS classification_errors,
(SELECT COUNT(*) FROM geometry_consistency WHERE geometry_consistency_status <> 'OK') AS geometry_errors,
(SELECT COUNT(*) FROM style_render_equivalence WHERE render_status <> 'OK') AS render_equivalence_errors,
(SELECT COUNT(*) FROM detection_catalog_integrity WHERE catalog_status <> 'OK') AS detection_integrity_errors,
(SELECT COUNT(*) FROM spatial_anomalies) AS spatial_anomalies,
CASE
WHEN (SELECT COUNT(*) FROM style_render_equivalence WHERE render_status = 'RENDER_BREAK_RISK') > 0 THEN 'FAIL'
WHEN (SELECT COUNT(*) FROM classification_validation WHERE classification_status IN ('RULE_CONFLICT', 'UNKNOWN_OBJECT')) > 0 THEN 'FAIL'
WHEN (SELECT COUNT(*) FROM geometry_consistency WHERE geometry_consistency_status = 'UNEXPECTED_GEOMETRY') > 0 THEN 'FAIL'
WHEN (SELECT COUNT(*) FROM detection_catalog_integrity WHERE catalog_status <> 'OK') > 0 THEN 'WARNING'
WHEN (SELECT COUNT(*) FROM spatial_anomalies) > 0 THEN 'WARNING'
ELSE 'PASS'
END AS validation_status
"""
)
def main() -> None:
SemanticOverlayValidator(DbConfig()).run()
if __name__ == "__main__":
main()

1349
navsea_tile_builder.py Normal file
View File

File diff suppressed because it is too large Load Diff

140
report/classification_distribution.txt Normal file
View File

@@ -0,0 +1,140 @@
630 1258262
100 887065
130 358545
141 239851
120 185447
142 156922
590 99863
140 93628
148 87488
150 83499
145 74972
143 53198
200 44071
830 32781
420 31318
121 30090
610 26855
600 26206
123 25497
428 24683
122 24679
910 19025
160 18833
260 17027
220 16472
270 15681
230 14153
850 13939
250 13824
710 12817
280 12484
240 12032
820 11975
291 11884
290 10829
650 10818
404 10366
681 9686
125 9367
620 8765
175 8493
640 8123
292 5499
860 5030
151 4554
170 4319
425 4175
800 3873
900 3570
403 3323
520 3169
840 3128
661 2914
810 2607
293 2540
152 2446
421 2356
422 2206
432 2119
124 1761
146 1720
741 1696
870 1665
754 1487
732 1455
144 1388
742 1251
294 1248
402 1178
161 1107
176 1097
162 990
149 988
409 824
415 811
171 780
433 756
711 744
434 715
712 655
427 637
413 626
505 610
295 549
530 539
880 500
510 454
660 436
739 429
720 396
719 380
755 377
426 308
662 289
759 262
750 260
756 257
698 231
550 203
682 181
147 150
753 146
748 145
749 140
724 132
412 125
713 108
431 107
221 99
725 95
424 92
500 88
746 86
747 85
745 84
429 82
743 81
408 73
760 69
714 67
761 64
730 54
740 51
721 46
757 45
540 44
723 44
410 43
680 32
405 31
210 28
414 20
758 17
700 14
406 10
702 10
890 10
401 8
411 1
430 1

80
report/geometry_summary.txt Normal file
View File

@@ -0,0 +1,80 @@
L700 LineString 14
L702 LineString 10
L725 LineString 95
L739 LineString 424
L739 MultiLineString 5
L740 LineString 51
L741 LineString 1665
L741 MultiLineString 31
L748 LineString 145
L749 LineString 140
L危険界 LineString 47
L基本線 LineString 53612
L基本線 MultiLineString 3
L概略等深線 LineString 5497
L概略等深線 MultiLineString 54
L海底地形 LineString 1866386
L海底地形 MultiLineString 208786
L海底線 LineString 14271
L海底線 MultiLineString 120
L等深線 LineString 437566
L等深線 MultiLineString 7485
L航路 LineString 116
L陸上構造物陸 LineString 1256228
L陸上構造物陸 MultiLineString 2034
L高さ制限 LineString 1028
P721ククリ LineString 24
P721ククリ MultiLineString 3
P730ククリ LineString 18
P730ククリ MultiLineString 9
P754ククリ LineString 730
P754ククリ MultiLineString 41
pパイロットステーション Point 88
P危険界ククリ LineString 51743
P危険界ククリ MultiLineString 219
p地名 Point 36247
p地名陸 Point 35388
P基本線 MultiPolygon 99559
P基本線 Polygon 602066
P基本線ククリ LineString 1074484
P基本線ククリ MultiLineString 6296
p底質 Point 99863
P投錨注意障害物 MultiPolygon 16
p投錨注意障害物 Point 49907
P投錨注意障害物 Polygon 8703
P投錨注意障害物ククリ LineString 8465
P投錨注意障害物ククリ MultiLineString 278
p施設・境界線等 Point 5019
P施設・境界線等 Polygon 1406
P施設・境界線等ククリ LineString 521
P施設・境界線等ククリ MultiLineString 45
P施設・境界線等透明 Polygon 27
P橋りょう等構造物 MultiPolygon 67
P橋りょう等構造物 Polygon 25708
P漁具定置箇所 MultiPolygon 270
P漁具定置箇所 Polygon 22325
P潜堤 MultiPolygon 101
P潜堤 Polygon 3780
P穴 MultiPolygon 5463
P穴 Polygon 361754
p航行危険障害物 Point 19268
P航行危険障害物 Polygon 159
P航行危険障害物ククリ LineString 191
P航行危険障害物ククリ MultiLineString 7
P航路 MultiPolygon 3
P航路 Polygon 349
P航路ククリ LineString 755
P航路ククリ MultiLineString 109
p航路境界等 Point 395
p航路標識群 Point 38520
P誘導線ククリ LineString 101
p錨泊地等 Point 366
P錨泊地等 Polygon 308
P錨泊地等ククリ LineString 240
P錨泊地等ククリ MultiLineString 57
p陸上構造物 Point 32708
P陸上構造物陸 MultiPolygon 42
P陸上構造物陸 Polygon 35580
P陸域 MultiPolygon 5541
P陸域 Polygon 193114
p高さ制限 Point 858

204
report/layer_properties.txt Normal file
View File

@@ -0,0 +1,204 @@
L700 at 14
L700 fid 14
L700 vt_layer 14
L700 分類番号 14
L702 at 10
L702 fid 10
L702 vt_layer 10
L702 分類番号 10
L725 at 95
L725 fid 95
L725 vt_layer 95
L725 分類番号 95
L739 at 429
L739 fid 429
L739 vt_layer 429
L739 分類番号 429
L740 at 51
L740 fid 51
L740 vt_layer 51
L740 分類番号 51
L741 at 1696
L741 fid 1696
L741 vt_layer 1696
L741 分類番号 1696
L748 at 145
L748 fid 145
L748 vt_layer 145
L748 分類番号 145
L749 at 140
L749 fid 140
L749 vt_layer 140
L749 分類番号 140
L危険界 at 47
L危険界 fid 47
L危険界 vt_layer 47
L危険界 分類番号 47
L基本線 at 53615
L基本線 fid 53615
L基本線 vt_layer 53615
L基本線 分類番号 53615
L概略等深線 at 5551
L概略等深線 fid 5551
L概略等深線 vt_layer 5551
L概略等深線 分類番号 5551
L概略等深線 高さ/深度(m) 5551
L海底地形 fid 2075172
L海底地形 vt_layer 2075172
L海底地形 水深値(m) 2075172
L海底線 at 14391
L海底線 fid 14391
L海底線 vt_layer 14391
L海底線 分類番号 14391
L等深線 at 445051
L等深線 fid 445051
L等深線 vt_layer 445051
L等深線 分類番号 445051
L等深線 高さ/深度(m) 445051
L航路 at 116
L航路 fid 116
L航路 vt_layer 116
L航路 分類番号 116
L陸上構造物陸 at 1258262
L陸上構造物陸 fid 1258262
L陸上構造物陸 vt_layer 1258262
L陸上構造物陸 分類番号 1258262
L高さ制限 at 1028
L高さ制限 fid 1028
L高さ制限 vt_layer 1028
L高さ制限 分類番号 1028
P721ククリ fid 27
P721ククリ vt_layer 27
P721ククリ 分類番号 27
P730ククリ fid 27
P730ククリ vt_layer 27
P730ククリ 分類番号 27
P754ククリ fid 771
P754ククリ vt_layer 771
P754ククリ 分類番号 771
pパイロットステーション at 88
pパイロットステーション fid 88
pパイロットステーション vt_layer 88
pパイロットステーション 分類番号 88
P危険界ククリ fid 51962
P危険界ククリ vt_layer 51962
P危険界ククリ 分類番号 51962
p地名 fid 36247
p地名 vt_layer 36247
p地名 分類番号 36247
p地名 日本語地名 36247
p地名 縮尺選択コード 36247
p地名 英文字地名 18969
p地名 表示重要度 36247
p地名陸 fid 35388
p地名陸 vt_layer 35388
p地名陸 分類番号 35388
p地名陸 日本語地名 35388
p地名陸 縮尺選択コード 35388
p地名陸 英文字地名 29131
p地名陸 表示重要度 35388
P基本線 at 701625
P基本線 fid 701625
P基本線 vt_layer 701625
P基本線 分類番号 701625
P基本線ククリ fid 1080780
P基本線ククリ vt_layer 1080780
P基本線ククリ 分類番号 1080780
p底質 at 99863
p底質 fid 99863
p底質 vt_layer 99863
p底質 分類番号 99863
p底質 名称 99863
p底質 表示位置 99863
p投錨注意障害物 at 54700
p投錨注意障害物 fid 58626
p投錨注意障害物 vt_layer 58626
p投錨注意障害物 分類番号 58626
P投錨注意障害物ククリ fid 8743
P投錨注意障害物ククリ vt_layer 8743
P投錨注意障害物ククリ 分類番号 8743
p施設・境界線等 at 6425
p施設・境界線等 fid 6425
p施設・境界線等 Sガイドページ 1752
p施設・境界線等 vt_layer 6425
p施設・境界線等 分類番号 6425
p施設・境界線等 名称 5019
P施設・境界線等ククリ fid 566
P施設・境界線等ククリ vt_layer 566
P施設・境界線等ククリ 分類番号 566
P施設・境界線等透明 at 27
P施設・境界線等透明 fid 27
P施設・境界線等透明 vt_layer 27
P施設・境界線等透明 分類番号 27
P橋りょう等構造物 at 25775
P橋りょう等構造物 fid 25775
P橋りょう等構造物 vt_layer 25775
P橋りょう等構造物 分類番号 25775
P漁具定置箇所 at 22595
P漁具定置箇所 fid 22595
P漁具定置箇所 vt_layer 22595
P漁具定置箇所 分類番号 22595
P潜堤 at 3881
P潜堤 fid 3881
P潜堤 vt_layer 3881
P潜堤 分類番号 3881
P穴 fid 367217
P穴 vt_layer 367217
p航行危険障害物 at 19403
p航行危険障害物 fid 19427
p航行危険障害物 vt_layer 19427
p航行危険障害物 分類番号 19427
P航行危険障害物ククリ fid 198
P航行危険障害物ククリ vt_layer 198
P航行危険障害物ククリ 分類番号 198
P航路 at 352
P航路 fid 352
P航路 vt_layer 352
P航路 分類番号 352
P航路ククリ fid 864
P航路ククリ vt_layer 864
P航路ククリ 分類番号 864
p航路境界等 at 377
p航路境界等 fid 395
p航路境界等 vt_layer 395
p航路境界等 分類番号 395
p航路境界等 角度 395
p航路標識群 at 38520
p航路標識群 fid 38520
p航路標識群 vt_layer 38520
p航路標識群 ローマ字名称 16765
p航路標識群 名称 17611
p航路標識群 名称補助 3559
p航路標識群 形状分類番号 38520
p航路標識群 明弧/分孤 1072
p航路標識群 灯略記 33446
p航路標識群 灯色 38520
p航路標識群 目的分類番号 38520
p航路標識群 表示位置 38520
p航路標識群 表示用番号 38520
P誘導線ククリ fid 101
P誘導線ククリ vt_layer 101
P誘導線ククリ 分類番号 101
P錨泊地等 at 453
P錨泊地等 fid 674
P錨泊地等 vt_layer 674
P錨泊地等 分類番号 674
P錨泊地等ククリ fid 297
P錨泊地等ククリ vt_layer 297
P錨泊地等ククリ 分類番号 297
p陸上構造物 at 32708
p陸上構造物 fid 32708
p陸上構造物 vt_layer 32708
p陸上構造物 分類番号 32708
P陸上構造物陸 at 35622
P陸上構造物陸 fid 35622
P陸上構造物陸 vt_layer 35622
P陸上構造物陸 分類番号 35622
P陸域 fid 198655
P陸域 vt_layer 198655
P陸域 分類番号 198655
p高さ制限 fid 858
p高さ制限 vt_layer 858
p高さ制限 分類番号 858
p高さ制限 名称 344
p高さ制限 高さ(m) 858

49
report/layer_summary.txt Normal file
View File

@@ -0,0 +1,49 @@
L海底地形 2075172
L陸上構造物陸 1258262
P基本線ククリ 1080780
P基本線 701625
L等深線 445051
P穴 367217
P陸域 198655
p底質 99863
p投錨注意障害物 58626
L基本線 53615
P危険界ククリ 51962
p航路標識群 38520
p地名 36247
P陸上構造物陸 35622
p地名陸 35388
p陸上構造物 32708
P橋りょう等構造物 25775
P漁具定置箇所 22595
p航行危険障害物 19427
L海底線 14391
P投錨注意障害物ククリ 8743
p施設・境界線等 6425
L概略等深線 5551
P潜堤 3881
L741 1696
L高さ制限 1028
P航路ククリ 864
p高さ制限 858
P754ククリ 771
P錨泊地等 674
P施設・境界線等ククリ 566
L739 429
p航路境界等 395
P航路 352
P錨泊地等ククリ 297
P航行危険障害物ククリ 198
L748 145
L749 140
L航路 116
P誘導線ククリ 101
L725 95
pパイロットステーション 88
L740 51
L危険界 47
P730ククリ 27
P721ククリ 27
P施設・境界線等透明 27
L700 14
L702 10

23
report/property_keys.txt Normal file
View File

@@ -0,0 +1,23 @@
vt_layer 6685117
fid 6685117
分類番号 4204208
at 2823055
水深値(m) 2075172
高さ/深度(m) 450602
表示位置 138383
名称 122837
縮尺選択コード 71635
日本語地名 71635
表示重要度 71635
英文字地名 48100
表示用番号 38520
灯色 38520
目的分類番号 38520
形状分類番号 38520
灯略記 33446
ローマ字名称 16765
名称補助 3559
Sガイドページ 1752
明弧/分孤 1072
高さ(m) 858
角度 395

100
report/tile_density.txt Normal file
View File

@@ -0,0 +1,100 @@
7 110 51 32300
6 55 25 25153
7 111 51 21202
5 27 12 20867
7 111 50 16595
8 222 102 16494
11 1750 869 14203
8 220 103 13548
11 1794 814 13256
5 28 12 13157
11 1794 813 13068
11 1793 813 12948
11 1759 815 12137
11 1777 815 11981
11 1760 813 11881
11 1736 878 11198
11 1802 810 10984
7 109 54 10901
7 112 51 10530
11 1815 790 10515
11 1785 815 10512
7 112 50 10308
8 220 102 10217
6 56 25 10198
8 218 108 9803
11 1781 815 9684
7 109 51 9655
11 1792 813 9357
11 1845 752 9303
9 444 204 8858
11 1761 823 8657
11 1780 817 8365
11 1829 783 8304
11 1818 809 8031
11 1802 816 8004
11 1751 868 7873
7 113 50 7820
8 223 101 7614
11 1786 815 7600
7 114 48 7588
7 110 52 7577
11 1831 780 7476
7 109 53 7472
11 1768 818 7452
8 228 97 7437
11 1761 824 7433
11 1729 881 7388
9 440 205 7366
11 1796 821 7328
11 1824 761 7293
11 1763 830 7277
5 27 13 7181
11 1781 816 7147
8 224 102 7108
8 219 103 7023
8 221 102 7018
11 1826 787 6960
11 1783 821 6912
12 3659 1567 6889
11 1777 823 6873
11 1801 811 6862
11 1777 816 6848
11 1791 813 6773
11 1818 808 6717
9 437 217 6661
12 3520 1626 6639
11 1830 782 6622
11 1790 812 6591
11 1784 814 6566
11 1757 826 6563
11 1778 816 6550
11 1803 798 6488
11 1781 807 6424
12 3587 1626 6414
11 1782 822 6394
11 1786 814 6371
11 1819 806 6289
11 1829 755 6237
11 1780 816 6231
11 1800 816 6212
11 1798 803 6184
12 3586 1626 6156
9 439 206 6111
8 219 107 6084
11 1778 818 6074
11 1757 825 6063
11 1804 799 6057
11 1750 868 5961
12 3500 1738 5955
7 114 49 5942
11 1797 820 5918
12 3589 1628 5917
12 3555 1631 5852
9 457 195 5849
11 1824 769 5796
12 3665 1727 5740
11 1793 815 5727
12 3649 1523 5715
12 3588 1626 5697
11 1832 863 5684

8
requirements.txt Normal file
View File

@@ -0,0 +1,8 @@
requests>=2.31
numpy>=1.26
xarray>=2024.6
cfgrib>=0.9.14
eccodes>=1.7
mercantile>=1.2
mapbox-vector-tile>=2.1
PyMySQL>=1.1

211
src/gfs_downloader.py Normal file
View File

@@ -0,0 +1,211 @@
from pathlib import Path
from datetime import datetime, timedelta, timezone
import time
try:
import requests
except ModuleNotFoundError: # pragma: no cover - dependency guard for runtime environments
requests = None
BASE_URL = "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl"
WAVE_BASE_URL = "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfswave.pl"
PROJECT_ROOT = Path(__file__).resolve().parent.parent
OUTPUT_DIR = PROJECT_ROOT / "data" / "grib"
REQUEST_TIMEOUT = (10, 120)
RETRY_LIMIT = 3
MIN_FILE_SIZE_BYTES = 1024
FORECAST_HOURS = [
0, 3, 6, 9, 12, 15, 18, 21, 24,
27, 30, 33, 36, 39, 42, 45,
48, 51, 54, 57, 60, 63, 66,
69, 72,
]
REGION = {
"leftlon": 120,
"rightlon": 150,
"toplat": 50,
"bottomlat": 20,
}
ATMOS_VARIABLES = [
"UGRD",
"VGRD",
"APCP",
"PRMSL",
"TMP",
]
WAVE_VARIABLES = [
"HTSGW",
"DIRPW",
"PERPW",
]
ATMOS_LEVELS = [
"lev_10_m_above_ground",
"lev_surface",
"lev_mean_sea_level",
]
WAVE_LEVELS = [
"lev_surface",
]
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
def get_cycle(reference_time=None):
now = reference_time or datetime.now(timezone.utc)
# NOMADS availability usually lags the wall clock. Bias one cycle back.
candidate = now - timedelta(hours=5)
hour = (candidate.hour // 6) * 6
cycle_time = candidate.replace(hour=hour, minute=0, second=0, microsecond=0)
return cycle_time.strftime("%Y%m%d"), f"{cycle_time.hour:02d}"
def build_atmos_request(date, cycle, forecast_hour):
filename = f"gfs.t{cycle}z.pgrb2.0p25.f{forecast_hour}"
params = {
"file": filename,
"leftlon": REGION["leftlon"],
"rightlon": REGION["rightlon"],
"toplat": REGION["toplat"],
"bottomlat": REGION["bottomlat"],
"dir": f"/gfs.{date}/{cycle}/atmos",
}
for variable in ATMOS_VARIABLES:
params[f"var_{variable}"] = "on"
for level in ATMOS_LEVELS:
params[level] = "on"
return params
def build_wave_request(date, cycle, forecast_hour):
filename = f"gfswave.t{cycle}z.global.0p25.f{forecast_hour}.grib2"
params = {
"file": filename,
"leftlon": REGION["leftlon"],
"rightlon": REGION["rightlon"],
"toplat": REGION["toplat"],
"bottomlat": REGION["bottomlat"],
"dir": f"/gfs.{date}/{cycle}/wave/gridded",
}
for variable in WAVE_VARIABLES:
params[f"var_{variable}"] = "on"
for level in WAVE_LEVELS:
params[level] = "on"
return params
def validate_response(response):
content_type = response.headers.get("Content-Type", "").lower()
if "html" in content_type or "text/plain" in content_type:
preview = response.text[:200].strip().replace("\n", " ")
raise ValueError(f"unexpected response content type {content_type}: {preview}")
def is_fatal_network_error(error):
error_text = str(error)
fatal_markers = (
"NameResolutionError",
"Failed to resolve",
"Temporary failure in name resolution",
)
return any(marker in error_text for marker in fatal_markers)
def download_file(session, base_url, params, output_path):
temp_path = output_path.with_suffix(".grib2.part")
if output_path.exists() and output_path.stat().st_size >= MIN_FILE_SIZE_BYTES:
print("skip", output_path)
return
for attempt in range(1, RETRY_LIMIT + 1):
try:
print(f"downloading {output_path} (attempt {attempt}/{RETRY_LIMIT})")
with session.get(
base_url,
params=params,
stream=True,
timeout=REQUEST_TIMEOUT,
) as response:
response.raise_for_status()
validate_response(response)
with temp_path.open("wb") as file_handle:
for chunk in response.iter_content(1024 * 1024):
if chunk:
file_handle.write(chunk)
if temp_path.stat().st_size < MIN_FILE_SIZE_BYTES:
raise ValueError(f"downloaded file too small: {temp_path.stat().st_size} bytes")
temp_path.replace(output_path)
return
except (requests.RequestException, ValueError) as exc:
if temp_path.exists():
temp_path.unlink()
print(" download failed:", exc)
if is_fatal_network_error(exc):
raise RuntimeError("fatal network error while reaching NOAA") from exc
if attempt == RETRY_LIMIT:
raise
time.sleep(attempt * 2)
def download_forecast(session, date, cycle, forecast_hour):
atmos_path = OUTPUT_DIR / f"{date}_{cycle}_f{forecast_hour}.grib2"
wave_path = OUTPUT_DIR / f"{date}_{cycle}_f{forecast_hour}_wave.grib2"
download_file(
session,
BASE_URL,
build_atmos_request(date, cycle, forecast_hour),
atmos_path,
)
download_file(
session,
WAVE_BASE_URL,
build_wave_request(date, cycle, forecast_hour),
wave_path,
)
def main():
if requests is None:
raise RuntimeError("requests is required to download GFS data")
date, cycle = get_cycle()
print("cycle:", date, cycle)
session = requests.Session()
session.headers["User-Agent"] = "weather-pipeline/1.0"
failures = []
for forecast_hour in FORECAST_HOURS:
forecast_hour_str = f"{forecast_hour:03d}"
try:
download_forecast(session, date, cycle, forecast_hour_str)
except RuntimeError as exc:
failures.append((forecast_hour_str, str(exc)))
break
except Exception as exc:
failures.append((forecast_hour_str, str(exc)))
if failures:
for forecast_hour, error in failures:
print(f"failed forecast {forecast_hour}: {error}")
raise SystemExit(1)
if __name__ == "__main__":
main()

189
src/grid_builder_v2.py Normal file
View File

@@ -0,0 +1,189 @@
from pathlib import Path
import json
import warnings
import numpy as np
import xarray as xr
try:
import cfgrib
except ModuleNotFoundError: # pragma: no cover - dependency guard for runtime environments
cfgrib = None
PROJECT_ROOT = Path(__file__).resolve().parent.parent
INPUT_DIR = PROJECT_ROOT / "data" / "grib"
OUTPUT_DIR = PROJECT_ROOT / "data" / "grid"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
REGION = {
"lon_min": 120,
"lon_max": 150,
"lat_min": 20,
"lat_max": 50,
}
VARIABLE_CANDIDATES = {
"u10": ("u10", "u"),
"v10": ("v10", "v"),
"tp": ("tp", "prate", "unknown"),
"msl": ("prmsl", "msl", "pres"),
"temp": ("t2m", "t"),
"wave_h": ("htsgw", "swh", "wvhgt"),
"wave_dir": ("dirpw", "mwd", "wvdir"),
"wave_period": ("perpw", "mwp", "wvper"),
}
def compute_wind(u_component, v_component):
speed = np.sqrt(u_component ** 2 + v_component ** 2)
# Meteorological direction: where the wind comes from, in degrees clockwise from north.
direction = (270 - np.degrees(np.arctan2(v_component, u_component))) % 360
return speed, direction
def load_datasets(path):
if cfgrib is None:
raise RuntimeError("cfgrib is required to build grids from GRIB2 files")
with xr.set_options(use_new_combine_kwarg_defaults=True):
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore",
message="In a future version of xarray the default value for compat will change",
category=FutureWarning,
)
return cfgrib.xarray_store.open_datasets(str(path))
def find_variable(datasets, candidates):
for candidate in candidates:
for dataset in datasets:
if candidate in dataset:
return dataset[candidate]
return None
def get_lat_lon(datasets):
for dataset in datasets:
if "latitude" in dataset and "longitude" in dataset:
return dataset["latitude"].values, dataset["longitude"].values
raise ValueError("no latitude/longitude coordinates found in GRIB datasets")
def to_2d_values(data_array, lat_size, lon_size):
if data_array is None:
return np.zeros((lat_size, lon_size), dtype=float)
values = np.asarray(data_array.squeeze().values)
if values.ndim != 2:
raise ValueError(f"expected 2D field, got shape {values.shape} for {data_array.name}")
if values.shape != (lat_size, lon_size):
raise ValueError(
f"field {data_array.name} shape {values.shape} does not match coordinates {(lat_size, lon_size)}"
)
return values
def normalize_longitudes(lon_values, fields):
lon_values = np.asarray(lon_values, dtype=float)
normalized_lon = ((lon_values + 180) % 360) - 180
sort_idx = np.argsort(normalized_lon)
normalized_lon = normalized_lon[sort_idx]
normalized_fields = [field[:, sort_idx] for field in fields]
return normalized_lon, normalized_fields
def select_region(lat_values, lon_values, fields):
lat_mask = (lat_values >= REGION["lat_min"]) & (lat_values <= REGION["lat_max"])
lon_mask = (lon_values >= REGION["lon_min"]) & (lon_values <= REGION["lon_max"])
lat_idx = np.where(lat_mask)[0]
lon_idx = np.where(lon_mask)[0]
if lat_idx.size == 0 or lon_idx.size == 0:
raise ValueError("selected region is empty after coordinate filtering")
return (
lat_values[lat_idx],
lon_values[lon_idx],
[field[np.ix_(lat_idx, lon_idx)] for field in fields],
)
def process_file(path):
datasets = load_datasets(path)
try:
wave_path = path.with_name(f"{path.stem}_wave.grib2")
wave_datasets = load_datasets(wave_path) if wave_path.exists() else []
all_datasets = [*datasets, *wave_datasets]
lat_values, lon_values = get_lat_lon(datasets)
lat_size = len(lat_values)
lon_size = len(lon_values)
u10 = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["u10"]), lat_size, lon_size)
v10 = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["v10"]), lat_size, lon_size)
rain = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["tp"]), lat_size, lon_size)
pressure = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["msl"]), lat_size, lon_size)
temp = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["temp"]), lat_size, lon_size)
wave_h = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["wave_h"]), lat_size, lon_size)
wave_dir = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["wave_dir"]), lat_size, lon_size)
wave_period = to_2d_values(find_variable(all_datasets, VARIABLE_CANDIDATES["wave_period"]), lat_size, lon_size)
wind_speed, wind_dir = compute_wind(u10, v10)
lon_values, normalized_fields = normalize_longitudes(
lon_values,
[wind_speed, wind_dir, rain, temp, pressure, wave_h, wave_dir, wave_period],
)
wind_speed, wind_dir, rain, temp, pressure, wave_h, wave_dir, wave_period = normalized_fields
lat_region, lon_region, region_fields = select_region(
lat_values,
lon_values,
[wind_speed, wind_dir, rain, temp, pressure, wave_h, wave_dir, wave_period],
)
wind_speed, wind_dir, rain, temp, pressure, wave_h, wave_dir, wave_period = region_fields
return {
"lat": lat_region.tolist(),
"lon": lon_region.tolist(),
"wind_speed": wind_speed.tolist(),
"wind_dir": wind_dir.tolist(),
"rain": rain.tolist(),
"temp": temp.tolist(),
"pressure": pressure.tolist(),
"wave_h": wave_h.tolist(),
"wave_dir": wave_dir.tolist(),
"wave_period": wave_period.tolist(),
}
finally:
for dataset in datasets:
dataset.close()
if 'wave_datasets' in locals():
for dataset in wave_datasets:
dataset.close()
def main():
for path in sorted(INPUT_DIR.glob("*.grib2")):
if path.stem.endswith("_wave"):
continue
print("processing", path.name)
try:
grid = process_file(path)
except Exception as exc:
print(" error processing", path.name, exc)
continue
output_path = OUTPUT_DIR / f"grid_{path.stem}.json"
payload = {
"time": path.stem,
"grid": grid,
}
with output_path.open("w", encoding="utf-8") as file_handle:
json.dump(payload, file_handle)
print("saved", output_path)
if __name__ == "__main__":
main()

104
src/pbf/audit_pbf_mysql.py Normal file
View File

@@ -0,0 +1,104 @@
import pymysql
import os
OUTPUT_DIR = "report"
os.makedirs(OUTPUT_DIR, exist_ok=True)
conn = pymysql.connect(
host="localhost",
user="root",
password="2chi9ks2",
database="pbf_analysis",
charset="utf8mb4"
)
cur = conn.cursor()
def write_file(name, rows):
path = os.path.join(OUTPUT_DIR, name)
with open(path,"w",encoding="utf8") as f:
for r in rows:
f.write("\t".join([str(x) for x in r])+"\n")
print("write:",path)
print("Layer summary...")
cur.execute("""
SELECT vt_layer, COUNT(*)
FROM features
GROUP BY vt_layer
ORDER BY COUNT(*) DESC
""")
write_file("layer_summary.txt",cur.fetchall())
print("Geometry summary...")
cur.execute("""
SELECT vt_layer, geom_type, COUNT(*)
FROM features
GROUP BY vt_layer, geom_type
ORDER BY vt_layer
""")
write_file("geometry_summary.txt",cur.fetchall())
print("Property keys...")
cur.execute("""
SELECT k, COUNT(*)
FROM properties
GROUP BY k
ORDER BY COUNT(*) DESC
""")
write_file("property_keys.txt",cur.fetchall())
print("Layer properties...")
cur.execute("""
SELECT f.vt_layer, p.k, COUNT(*)
FROM properties p
JOIN features f ON p.feature_id=f.id
GROUP BY f.vt_layer,p.k
ORDER BY f.vt_layer
""")
write_file("layer_properties.txt",cur.fetchall())
print("Classification distribution...")
cur.execute("""
SELECT v, COUNT(*)
FROM properties
WHERE k='分類番号'
GROUP BY v
ORDER BY COUNT(*) DESC
""")
write_file("classification_distribution.txt",cur.fetchall())
print("Tile density...")
cur.execute("""
SELECT z,x,y,COUNT(*)
FROM features
GROUP BY z,x,y
ORDER BY COUNT(*) DESC
LIMIT 100
""")
write_file("tile_density.txt",cur.fetchall())
print("done.")

5652
src/pbf/style.json Normal file
View File

File diff suppressed because it is too large Load Diff

5652
src/pbf/style.karatsu-10nm-compatible.json Normal file
View File

File diff suppressed because it is too large Load Diff

1892
src/pbf/style.karatsu-10nm-v2.json Normal file
View File

File diff suppressed because it is too large Load Diff

5652
src/pbf/style.navsea-compatible.json Normal file
View File

File diff suppressed because it is too large Load Diff

1391
src/pbf/style.navsea-delivery-karatsu-10nm.json Normal file
View File

File diff suppressed because it is too large Load Diff

1391
src/pbf/style.navsea-engineering-karatsu-10nm.json Normal file
View File

File diff suppressed because it is too large Load Diff

692
src/pbf/style.navsea-redesign.json Normal file
View File

@@ -0,0 +1,692 @@
{
"version": 8,
"name": "navsea-redesign",
"glyphs": "http://192.168.200.184/newpec/fonts/{fontstack}/{range}.pbf",
"sources": {
"navsea": {
"type": "vector",
"minzoom": 0,
"maxzoom": 12,
"tiles": [
"http://192.168.200.184/pbf/{z}/{x}/{y}.pbf"
],
"attribution": "NavSea semantic vector tiles"
}
},
"layers": [
{
"id": "background",
"type": "background",
"paint": {
"background-color": "#dfe8d8"
}
},
{
"id": "land-area",
"type": "fill",
"source": "navsea",
"source-layer": "P陸域",
"paint": {
"fill-color": "#d9cfab"
}
},
{
"id": "land-hole",
"type": "fill",
"source": "navsea",
"source-layer": "P穴",
"paint": {
"fill-color": "#bfc8b1"
}
},
{
"id": "water-depth-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"match",
[
"get",
"canonical_family"
],
[
"surface"
],
true,
false
],
"paint": {
"fill-color": [
"match",
[
"get",
"canonical_object_type"
],
"0-2m",
"#d8f3f6",
"0-5m",
"#d2eff4",
"2-5m",
"#c8eaf0",
"5-10m",
"#bce3ec",
"10-20m",
"#b0dbe8",
"20-100m",
"#a0d2e0",
"100-200m",
"#93c7d6",
"200-500m",
"#7eb7ca",
"500-1000m",
"#68a6bc",
"1000-2000m",
"#558eab",
"#dff3f7"
],
"fill-opacity": 0.9
}
},
{
"id": "shoal-hazard-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"==",
[
"get",
"canonical_object_type"
],
"浅所危険界"
],
"paint": {
"fill-color": "#f2a65a",
"fill-opacity": 0.45
}
},
{
"id": "tidal-flat-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"==",
[
"get",
"canonical_object_type"
],
"干潮帯"
],
"paint": {
"fill-color": "#d7d0b4",
"fill-opacity": 0.75
}
},
{
"id": "river-lake-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"match",
[
"get",
"canonical_object_type"
],
[
"河川域",
"湖沼域",
"陸上水域"
],
true,
false
],
"paint": {
"fill-color": "#b8d8e8",
"fill-opacity": 0.85
}
},
{
"id": "unsurveyed-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"==",
[
"get",
"canonical_object_type"
],
"未測海域"
],
"paint": {
"fill-color": "#f7f0b8",
"fill-opacity": 0.5
}
},
{
"id": "coast-structure-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"match",
[
"get",
"canonical_object_type"
],
[
"防波堤",
"浮施設・桟橋",
"撤去跡"
],
true,
false
],
"paint": {
"fill-color": "#8d8578",
"fill-opacity": 0.85
}
},
{
"id": "bridge-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P橋りょう等構造物",
"paint": {
"fill-color": "#736b5f",
"fill-opacity": 0.85
}
},
{
"id": "fishery-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P漁具定置箇所",
"paint": {
"fill-color": "#89b26f",
"fill-opacity": 0.45,
"fill-outline-color": "#56764a"
}
},
{
"id": "land-structure-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P陸上構造物陸",
"paint": {
"fill-color": "#8e6f3e",
"fill-opacity": 0.8
}
},
{
"id": "coast-outline",
"type": "line",
"source": "navsea",
"source-layer": "P基本線ククリ",
"paint": {
"line-color": "#736b5f",
"line-width": 1.2
}
},
{
"id": "danger-outline",
"type": "line",
"source": "navsea",
"source-layer": "P危険界ククリ",
"paint": {
"line-color": "#c95c2a",
"line-width": 1.3,
"line-dasharray": [
2,
1
]
}
},
{
"id": "anchor-danger-outline",
"type": "line",
"source": "navsea",
"source-layer": "P投錨注意障害物ククリ",
"paint": {
"line-color": "#c03d3d",
"line-width": 1.1,
"line-dasharray": [
1,
1
]
}
},
{
"id": "bathymetry-lines",
"type": "line",
"source": "navsea",
"source-layer": "L海底地形",
"paint": {
"line-color": "#7fa6b7",
"line-width": 0.5,
"line-opacity": 0.45
}
},
{
"id": "depth-contours",
"type": "line",
"source": "navsea",
"source-layer": "L等深線",
"paint": {
"line-color": "#487b94",
"line-width": [
"interpolate",
[
"linear"
],
[
"zoom"
],
5,
0.3,
8,
0.6,
12,
1.1
],
"line-opacity": 0.85
}
},
{
"id": "subsea-lines",
"type": "line",
"source": "navsea",
"source-layer": "L海底線",
"paint": {
"line-color": "#8d4f9c",
"line-width": 1.2,
"line-dasharray": [
3,
1
]
}
},
{
"id": "coast-structure-lines",
"type": "line",
"source": "navsea",
"source-layer": "L基本線",
"paint": {
"line-color": "#60584d",
"line-width": 1.4
}
},
{
"id": "land-structure-lines",
"type": "line",
"source": "navsea",
"source-layer": "L陸上構造物陸",
"paint": {
"line-color": "#805f2c",
"line-width": 1
}
},
{
"id": "water-boundary-line",
"type": "line",
"source": "navsea",
"source-layer": "L739",
"paint": {
"line-color": "#4b89a5",
"line-width": 1.1
}
},
{
"id": "regulatory-boundary-line",
"type": "line",
"source": "navsea",
"source-layer": "L741",
"paint": {
"line-color": "#cf5b47",
"line-width": 1.2,
"line-dasharray": [
2,
2
]
}
},
{
"id": "height-limit-line",
"type": "line",
"source": "navsea",
"source-layer": "L高さ制限",
"paint": {
"line-color": "#b1466b",
"line-width": 1.1
}
},
{
"id": "hazard-area-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P投錨注意障害物",
"paint": {
"fill-color": "#d85f4c",
"fill-opacity": 0.42,
"fill-outline-color": "#9b3324"
}
},
{
"id": "hazard-points",
"type": "circle",
"source": "navsea",
"source-layer": "p航行危険障害物",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#d5452f",
"circle-stroke-color": "#fff4df",
"circle-stroke-width": 1.1
}
},
{
"id": "anchor-hazard-points",
"type": "circle",
"source": "navsea",
"source-layer": "p投錨注意障害物",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#cf7d2b",
"circle-stroke-color": "#fff4df",
"circle-stroke-width": 1
}
},
{
"id": "nav-marks",
"type": "circle",
"source": "navsea",
"source-layer": "p航路標識群",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#f6ce4f",
"circle-stroke-color": "#313131",
"circle-stroke-width": 1
}
},
{
"id": "facility-points",
"type": "circle",
"source": "navsea",
"source-layer": "p施設・境界線等",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
8,
2,
12,
5
],
"circle-color": "#4d6b86",
"circle-stroke-color": "#ffffff",
"circle-stroke-width": 0.8
}
},
{
"id": "landmark-points",
"type": "circle",
"source": "navsea",
"source-layer": "p陸上構造物",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
8,
2,
12,
4.5
],
"circle-color": "#6b5840",
"circle-stroke-color": "#f7f1e2",
"circle-stroke-width": 0.8
}
},
{
"id": "bottom-material-points",
"type": "circle",
"source": "navsea",
"source-layer": "p底質",
"paint": {
"circle-radius": 1.5,
"circle-color": "#3f7183",
"circle-opacity": 0.55
}
},
{
"id": "height-limit-points",
"type": "circle",
"source": "navsea",
"source-layer": "p高さ制限",
"paint": {
"circle-radius": 3,
"circle-color": "#a83566",
"circle-stroke-color": "#ffffff",
"circle-stroke-width": 0.8
}
},
{
"id": "sea-place-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p地名",
"layout": {
"text-field": [
"coalesce",
[
"get",
"日本語地名"
],
[
"get",
"英文字地名"
],
[
"get",
"name"
]
],
"text-font": [
"NotoSerifCJK-Regular"
],
"text-size": [
"interpolate",
[
"linear"
],
[
"zoom"
],
6,
9,
12,
14
],
"text-anchor": "center",
"text-max-width": 10
},
"paint": {
"text-color": "#284857",
"text-halo-color": "#f6fbff",
"text-halo-width": 1
}
},
{
"id": "land-place-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p地名陸",
"layout": {
"text-field": [
"coalesce",
[
"get",
"日本語地名"
],
[
"get",
"英文字地名"
],
[
"get",
"name"
]
],
"text-font": [
"NotoSerifCJK-Regular"
],
"text-size": [
"interpolate",
[
"linear"
],
[
"zoom"
],
6,
9,
12,
14
],
"text-anchor": "center",
"text-max-width": 10
},
"paint": {
"text-color": "#3a3327",
"text-halo-color": "#fff9ef",
"text-halo-width": 1
}
},
{
"id": "nav-object-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p航路標識群",
"minzoom": 10,
"layout": {
"text-field": [
"coalesce",
[
"get",
"名称"
],
[
"get",
"name"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSansCJK-Regular"
],
"text-size": 11,
"text-offset": [
0,
1.1
],
"text-anchor": "top",
"text-max-width": 12
},
"paint": {
"text-color": "#2f2f2f",
"text-halo-color": "#fff8e0",
"text-halo-width": 1.1
}
},
{
"id": "hazard-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p航行危険障害物",
"minzoom": 10,
"layout": {
"text-field": [
"coalesce",
[
"get",
"名称"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSansCJK-Regular"
],
"text-size": 11,
"text-offset": [
0,
1.1
],
"text-anchor": "top"
},
"paint": {
"text-color": "#7f221a",
"text-halo-color": "#fff3eb",
"text-halo-width": 1
}
}
]
}

679
src/pbf/style.navsea-semantic-only.json Normal file
View File

@@ -0,0 +1,679 @@
{
"version": 8,
"name": "navsea-semantic-only",
"glyphs": "http://192.168.200.184/newpec/fonts/{fontstack}/{range}.pbf",
"sources": {
"navsea": {
"type": "vector",
"minzoom": 0,
"maxzoom": 12,
"tiles": [
"http://192.168.200.184/pbf/{z}/{x}/{y}.pbf"
],
"attribution": "NavSea semantic vector tiles"
}
},
"layers": [
{
"id": "background",
"type": "background",
"paint": {
"background-color": "#edf3ef"
}
},
{
"id": "land-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P陸域",
"paint": {
"fill-color": "#d8cda8"
}
},
{
"id": "hole-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P穴",
"paint": {
"fill-color": "#c7cfbb"
}
},
{
"id": "surface-water",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"match",
[
"get",
"canonical_object_type"
],
[
"0-2m",
"0-5m",
"2-5m",
"5-10m",
"10-20m",
"20-100m",
"100-200m",
"200-500m",
"500-1000m",
"1000-2000m",
"2000-3000m",
"3000-4000m",
"4000-5000m",
"5000-6000m",
"6000-7000m",
"7000-8000m",
"8000-9000m",
"9000m以深",
"干潮帯",
"河川域",
"湖沼域",
"陸上水域"
],
true,
false
],
"paint": {
"fill-color": [
"match",
[
"get",
"canonical_object_type"
],
"干潮帯",
"#d7cfb4",
"河川域",
"#afd8e7",
"湖沼域",
"#afd8e7",
"陸上水域",
"#afd8e7",
"0-2m",
"#d8f1f4",
"0-5m",
"#d0edf1",
"2-5m",
"#c4e7ed",
"5-10m",
"#b6dfe8",
"10-20m",
"#a8d6e1",
"20-100m",
"#96cbd8",
"100-200m",
"#84bfd0",
"200-500m",
"#71afc2",
"500-1000m",
"#629fb4",
"1000-2000m",
"#548ea4",
"2000-3000m",
"#497c93",
"3000-4000m",
"#3f6c84",
"4000-5000m",
"#365e75",
"5000-6000m",
"#2f5268",
"6000-7000m",
"#28475d",
"7000-8000m",
"#213d52",
"8000-9000m",
"#1d3448",
"#182c3f"
],
"fill-opacity": 0.88
}
},
{
"id": "hazard-area",
"type": "fill",
"source": "navsea",
"source-layer": "P基本線",
"filter": [
"match",
[
"get",
"canonical_object_type"
],
[
"浅所危険界",
"未測海域",
"浮施設・桟橋",
"防波堤",
"撤去跡"
],
true,
false
],
"paint": {
"fill-color": [
"match",
[
"get",
"canonical_object_type"
],
"浅所危険界",
"#ee9f56",
"未測海域",
"#eadf9b",
"浮施設・桟橋",
"#88857d",
"防波堤",
"#7d796f",
"#9d7c67"
],
"fill-opacity": 0.45
}
},
{
"id": "fishery-area",
"type": "fill",
"source": "navsea",
"source-layer": "P漁具定置箇所",
"paint": {
"fill-color": "#7db065",
"fill-opacity": 0.38,
"fill-outline-color": "#507541"
}
},
{
"id": "hazard-polygon",
"type": "fill",
"source": "navsea",
"source-layer": "P投錨注意障害物",
"paint": {
"fill-color": "#d15d49",
"fill-opacity": 0.4,
"fill-outline-color": "#8b3226"
}
},
{
"id": "bridge-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P橋りょう等構造物",
"paint": {
"fill-color": "#71685b",
"fill-opacity": 0.8
}
},
{
"id": "land-structure-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P陸上構造物陸",
"paint": {
"fill-color": "#896d3d",
"fill-opacity": 0.82
}
},
{
"id": "submerged-fill",
"type": "fill",
"source": "navsea",
"source-layer": "P潜堤",
"paint": {
"fill-color": "#757d95",
"fill-opacity": 0.45
}
},
{
"id": "depth-lines",
"type": "line",
"source": "navsea",
"source-layer": "L等深線",
"paint": {
"line-color": "#447a94",
"line-width": [
"interpolate",
[
"linear"
],
[
"zoom"
],
5,
0.3,
8,
0.6,
12,
1.1
],
"line-opacity": 0.9
}
},
{
"id": "major-depth-lines",
"type": "line",
"source": "navsea",
"source-layer": "L概略等深線",
"paint": {
"line-color": "#275f77",
"line-width": [
"interpolate",
[
"linear"
],
[
"zoom"
],
5,
0.5,
8,
0.9,
12,
1.5
]
}
},
{
"id": "bathymetry-support",
"type": "line",
"source": "navsea",
"source-layer": "L海底地形",
"paint": {
"line-color": "#8aadb9",
"line-width": 0.45,
"line-opacity": 0.4
}
},
{
"id": "cable-line",
"type": "line",
"source": "navsea",
"source-layer": "L海底線",
"paint": {
"line-color": "#7d4a94",
"line-width": 1.2,
"line-dasharray": [
3,
1
]
}
},
{
"id": "coast-structure-line",
"type": "line",
"source": "navsea",
"source-layer": "L基本線",
"paint": {
"line-color": "#635d53",
"line-width": 1.4
}
},
{
"id": "land-structure-line",
"type": "line",
"source": "navsea",
"source-layer": "L陸上構造物陸",
"paint": {
"line-color": "#7b5d2e",
"line-width": 1
}
},
{
"id": "boundary-line",
"type": "line",
"source": "navsea",
"source-layer": "L739",
"paint": {
"line-color": "#4f8ba5",
"line-width": 1.1
}
},
{
"id": "regulatory-line",
"type": "line",
"source": "navsea",
"source-layer": "L741",
"paint": {
"line-color": "#c95b46",
"line-width": 1.2,
"line-dasharray": [
2,
2
]
}
},
{
"id": "height-limit-line",
"type": "line",
"source": "navsea",
"source-layer": "L高さ制限",
"paint": {
"line-color": "#a93a62",
"line-width": 1.1
}
},
{
"id": "coast-outline",
"type": "line",
"source": "navsea",
"source-layer": "P基本線ククリ",
"paint": {
"line-color": "#7d7566",
"line-width": 1
}
},
{
"id": "danger-outline",
"type": "line",
"source": "navsea",
"source-layer": "P危険界ククリ",
"paint": {
"line-color": "#c35f31",
"line-width": 1.2,
"line-dasharray": [
2,
1
]
}
},
{
"id": "anchor-outline",
"type": "line",
"source": "navsea",
"source-layer": "P投錨注意障害物ククリ",
"paint": {
"line-color": "#b64335",
"line-width": 1.1,
"line-dasharray": [
1,
1
]
}
},
{
"id": "hazard-points",
"type": "circle",
"source": "navsea",
"source-layer": "p航行危険障害物",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#d1442f",
"circle-stroke-color": "#fff4ea",
"circle-stroke-width": 1
}
},
{
"id": "anchor-hazard-points",
"type": "circle",
"source": "navsea",
"source-layer": "p投錨注意障害物",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#d18a38",
"circle-stroke-color": "#fff6e7",
"circle-stroke-width": 1
}
},
{
"id": "nav-points",
"type": "circle",
"source": "navsea",
"source-layer": "p航路標識群",
"paint": {
"circle-radius": [
"interpolate",
[
"linear"
],
[
"zoom"
],
7,
2,
10,
4,
12,
6
],
"circle-color": "#f1cb4d",
"circle-stroke-color": "#313131",
"circle-stroke-width": 1
}
},
{
"id": "facility-points",
"type": "circle",
"source": "navsea",
"source-layer": "p施設・境界線等",
"paint": {
"circle-radius": 3.5,
"circle-color": "#4f6b86",
"circle-stroke-color": "#ffffff",
"circle-stroke-width": 0.8
}
},
{
"id": "bottom-points",
"type": "circle",
"source": "navsea",
"source-layer": "p底質",
"paint": {
"circle-radius": 1.8,
"circle-color": "#3d7182",
"circle-opacity": 0.65
}
},
{
"id": "landmark-points",
"type": "circle",
"source": "navsea",
"source-layer": "p陸上構造物",
"paint": {
"circle-radius": 3,
"circle-color": "#6a5640",
"circle-stroke-color": "#fff7eb",
"circle-stroke-width": 0.8
}
},
{
"id": "height-points",
"type": "circle",
"source": "navsea",
"source-layer": "p高さ制限",
"paint": {
"circle-radius": 3,
"circle-color": "#aa365f",
"circle-stroke-color": "#ffffff",
"circle-stroke-width": 0.8
}
},
{
"id": "sea-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p地名",
"layout": {
"text-field": [
"coalesce",
[
"get",
"日本語地名"
],
[
"get",
"英文字地名"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSerifCJK-Regular"
],
"text-size": [
"interpolate",
[
"linear"
],
[
"zoom"
],
6,
9,
12,
15
],
"text-anchor": "center",
"text-max-width": 10
},
"paint": {
"text-color": "#264858",
"text-halo-color": "#f8fcff",
"text-halo-width": 1
}
},
{
"id": "land-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p地名陸",
"layout": {
"text-field": [
"coalesce",
[
"get",
"日本語地名"
],
[
"get",
"英文字地名"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSerifCJK-Regular"
],
"text-size": [
"interpolate",
[
"linear"
],
[
"zoom"
],
6,
9,
12,
15
],
"text-anchor": "center",
"text-max-width": 10
},
"paint": {
"text-color": "#3a3228",
"text-halo-color": "#fff8ef",
"text-halo-width": 1
}
},
{
"id": "nav-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p航路標識群",
"minzoom": 10,
"layout": {
"text-field": [
"coalesce",
[
"get",
"名称"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSansCJK-Regular"
],
"text-size": 11,
"text-offset": [
0,
1.1
],
"text-anchor": "top"
},
"paint": {
"text-color": "#303030",
"text-halo-color": "#fff8df",
"text-halo-width": 1
}
},
{
"id": "hazard-labels",
"type": "symbol",
"source": "navsea",
"source-layer": "p航行危険障害物",
"minzoom": 10,
"layout": {
"text-field": [
"coalesce",
[
"get",
"名称"
],
[
"get",
"canonical_object_type"
]
],
"text-font": [
"NotoSansCJK-Regular"
],
"text-size": 11,
"text-offset": [
0,
1.1
],
"text-anchor": "top"
},
"paint": {
"text-color": "#7c2219",
"text-halo-color": "#fff4ec",
"text-halo-width": 1
}
}
]
}

1391
src/pbf/style.navsea-v2.json Normal file
View File

File diff suppressed because it is too large Load Diff

2232
src/pbf/style.pruned.json Normal file
View File

File diff suppressed because it is too large Load Diff

136
src/vector_tile_generator.py Normal file
View File

@@ -0,0 +1,136 @@
import os
from collections import defaultdict
from pathlib import Path
import json
try:
import mapbox_vector_tile
except ModuleNotFoundError: # pragma: no cover - dependency guard for runtime environments
mapbox_vector_tile = None
try:
import mercantile
except ModuleNotFoundError: # pragma: no cover - dependency guard for runtime environments
mercantile = None
PROJECT_ROOT = Path(__file__).resolve().parent.parent
GRID_DIR = PROJECT_ROOT / "data" / "grid"
OUTPUT_DIR = Path("/home/wwwroot/weather")
DEFAULT_ZOOMS = [2, 4, 6, 8, 10, 12]
def get_zoom_levels():
raw_value = os.environ.get("WEATHER_TILE_ZOOMS", "")
if not raw_value.strip():
return DEFAULT_ZOOMS
zooms = []
for chunk in raw_value.split(","):
chunk = chunk.strip()
if not chunk:
continue
zoom = int(chunk)
if zoom < 0:
raise ValueError(f"Invalid zoom level: {zoom}")
zooms.append(zoom)
if not zooms:
raise ValueError("WEATHER_TILE_ZOOMS did not contain any usable zoom levels")
return sorted(set(zooms))
def load_grid(path):
with path.open(encoding="utf-8") as file_handle:
data = json.load(file_handle)
return data["time"], data["grid"]
def get_grid_field(grid, field_name, latitudes, longitudes):
values = grid.get(field_name)
if values is not None:
return values
return [[0.0 for _ in longitudes] for _ in latitudes]
def grid_to_features(grid):
latitudes = grid["lat"]
longitudes = grid["lon"]
wind_speed = grid["wind_speed"]
wind_dir = grid["wind_dir"]
rain = grid["rain"]
temp = grid["temp"]
pressure = grid["pressure"]
wave_h = get_grid_field(grid, "wave_h", latitudes, longitudes)
wave_dir = get_grid_field(grid, "wave_dir", latitudes, longitudes)
wave_period = get_grid_field(grid, "wave_period", latitudes, longitudes)
features = []
for lat_index, latitude in enumerate(latitudes):
for lon_index, longitude in enumerate(longitudes):
features.append(
{
"geometry": {"type": "Point", "coordinates": [longitude, latitude]},
"properties": {
"ws": wind_speed[lat_index][lon_index],
"wd": wind_dir[lat_index][lon_index],
"r": rain[lat_index][lon_index],
"t": temp[lat_index][lon_index],
"p": pressure[lat_index][lon_index],
"wh": wave_h[lat_index][lon_index],
"wdir": wave_dir[lat_index][lon_index],
"wp": wave_period[lat_index][lon_index],
},
}
)
return features
def bucket_features_by_tile(features, zoom):
buckets = defaultdict(list)
for feature in features:
longitude, latitude = feature["geometry"]["coordinates"]
tile = mercantile.tile(longitude, latitude, zoom)
buckets[(tile.x, tile.y)].append(feature)
return buckets
def write_tile(tile_time, zoom, tile_x, tile_y, features):
bounds = mercantile.bounds(mercantile.Tile(x=tile_x, y=tile_y, z=zoom))
tile_dir = OUTPUT_DIR / tile_time / str(zoom) / str(tile_x)
tile_dir.mkdir(parents=True, exist_ok=True)
tile_path = tile_dir / f"{tile_y}.pbf"
layer = {"name": "weather", "features": features}
tile_data = mapbox_vector_tile.encode(
layer,
default_options={
"quantize_bounds": (bounds.west, bounds.south, bounds.east, bounds.north),
},
)
tile_path.write_bytes(tile_data)
print("tile", tile_time, zoom, tile_x, tile_y)
def generate_tiles(tile_time, features):
for zoom in get_zoom_levels():
buckets = bucket_features_by_tile(features, zoom)
for (tile_x, tile_y), tile_features in buckets.items():
write_tile(tile_time, zoom, tile_x, tile_y, tile_features)
def main():
if mapbox_vector_tile is None or mercantile is None:
raise RuntimeError("mapbox-vector-tile and mercantile are required to generate vector tiles")
for path in sorted(GRID_DIR.glob("*.json")):
print("processing", path.name)
tile_time, grid = load_grid(path)
features = grid_to_features(grid)
generate_tiles(tile_time, features)
if __name__ == "__main__":
main()

86
src/weather_pipeline.py Normal file
View File

@@ -0,0 +1,86 @@
from pathlib import Path
import importlib.util
import subprocess
import sys
import time
SCRIPT_DIR = Path(__file__).resolve().parent
DOWNLOADER = SCRIPT_DIR / "gfs_downloader.py"
GRID_BUILDER = SCRIPT_DIR / "grid_builder_v2.py"
TILE_GENERATOR = SCRIPT_DIR / "vector_tile_generator.py"
STEP_DEPENDENCIES = {
"GFS Downloader": ("requests",),
"Grid Builder v2": ("numpy", "cfgrib"),
"Vector Tile Generator": ("mercantile", "mapbox_vector_tile"),
}
def check_dependencies():
missing = []
for step_name, modules in STEP_DEPENDENCIES.items():
missing_modules = [module for module in modules if importlib.util.find_spec(module) is None]
if missing_modules:
missing.append(f"{step_name}: {', '.join(missing_modules)}")
return missing
def run_step(name, script_path):
print("\n==========================")
print("Running:", name)
print("==========================\n")
start = time.time()
command = [sys.executable, "-u", str(script_path)]
process = subprocess.Popen(
command,
cwd=str(SCRIPT_DIR.parent),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
bufsize=1,
)
assert process.stdout is not None
for line in process.stdout:
print(line, end="")
return_code = process.wait()
if return_code != 0:
raise RuntimeError(f"{name} failed with exit code {return_code}")
end = time.time()
print("\nFinished:", name)
print("Time:", round(end - start, 2), "seconds")
def main():
print("Weather Pipeline Starting...")
print("Date:", time.strftime("%Y-%m-%d %H:%M:%S"))
missing_dependencies = check_dependencies()
if missing_dependencies:
print("\n" + "!" * 50)
print("Pipeline Failed: missing runtime dependencies")
for item in missing_dependencies:
print("-", item)
print("!" * 50)
raise SystemExit(1)
try:
run_step("GFS Downloader", DOWNLOADER)
run_step("Grid Builder v2", GRID_BUILDER)
run_step("Vector Tile Generator", TILE_GENERATOR)
print("\n" + "=" * 50)
print("Weather Pipeline Completed Successfully!")
print("=" * 50)
except RuntimeError as exc:
print("\n" + "!" * 50)
print("Pipeline Failed:", str(exc))
print("!" * 50)
raise SystemExit(1)
if __name__ == "__main__":
main()

54
tasks/NavSea_Semantic_Overlay_Validation_System.md Normal file
View File

@@ -0,0 +1,54 @@
# NavSea Semantic Overlay Validation System
## Purpose
Validate the NavSea semantic overlay without touching rendering logic, style configuration, or vector tile structure.
## Inputs
The task reads the existing semantic overlay and audit tables in database `pbf_analysis`, including:
- `canonical_layer_rules`
- `canonical_object_rules`
- `object_geometry_allowlist`
- `pbf_relayer_candidates`
- `pbf_render_compatibility`
- `pbf_detection_catalog`
- `style_layers`
- `features`
- `tile_density`
## Execution
Rebuild the semantic overlay metadata and then run validation:
```bash
.venv/bin/python navsea_relayer.py
.venv/bin/python navsea_semantic_validation.py
```
## Validation Tables
The validation task creates or refreshes:
- `classification_validation`
- `geometry_consistency`
- `style_render_equivalence`
- `detection_catalog_integrity`
- `spatial_anomalies`
- `semantic_validation_summary`
## Current Design Notes
- Rendering compatibility is verified at database level only.
- `canonical_object_rules` is the stable object-level rule source.
- `object_geometry_allowlist` is the geometry whitelist used to separate valid multi-geometry objects from true anomalies.
- `spatial_anomalies` is intentionally lightweight and currently focuses on tile-density and tile-index anomalies.
## Success Criteria
- No render equivalence break
- No classification errors
- No geometry consistency errors
- No detection catalog integrity errors
- Spatial anomalies limited to acceptable warnings

229
tasks/WeatherServer_GridBuilder Normal file
View File

@@ -0,0 +1,229 @@
# NavSea Weather Server
Task: WeatherServer_GridBuilder
Architecture: NavSea V11
Codex: codex6
Status: TODO
---
# 1 任务目标
实现 Weather Grid Builder 模块。
功能：
将 GRIB 数据解析为统一的 Weather Grid 数据结构。
输入：
data/grib/*.grib2
输出：
data/grid/*.json
Weather Grid 将作为：
Vector Tile Generator
Routing Engine
Weather Analysis
的基础数据源。
---
# 2 输入数据
GRIB 文件来自 Downloader。
示例：
data/grib/
20260312_00_f000.grib2
20260312_00_f003.grib2
20260312_00_f006.grib2
变量：
UGRD
VGRD
APCP
PRMSL
TMP
---
# 3 Weather Grid 数据结构
每个 grid 点结构：
{
"lat": float,
"lon": float,
"wind_speed": float,
"wind_dir": float,
"rain": float,
"temp": float,
"pressure": float
}
说明：
lat 纬度
lon 经度
wind_speed m/s
wind_dir 度
rain mm
temp 摄氏度
pressure hPa
---
# 4 风速计算
使用 U/V 分量计算：
wind_speed = sqrt(u² + v²)
---
# 5 风向计算
公式：
wind_dir = (atan2(u, v) * 180 / π + 360) % 360
结果：
0–360°
---
# 6 Grid Builder 输出
文件：
data/grid/
示例：
grid_20260312_00_f000.json
结构：
{
"time": "20260312_00_f000",
"points": [...]
}
---
# 7 创建文件
weather_server/grid/grid_builder.py
---
# 8 实现代码
```python
import os
import json
import numpy as np
import xarray as xr
INPUT_DIR = "data/grib"
OUTPUT_DIR = "data/grid"
os.makedirs(OUTPUT_DIR, exist_ok=True)
def compute_wind(u, v):
speed = np.sqrt(u**2 + v**2)
direction = (np.degrees(np.arctan2(u, v)) + 360) % 360
return speed, direction
def process_file(path):
ds = xr.open_dataset(path, engine="cfgrib")
u = ds["u10"].values
v = ds["v10"].values
rain = ds["tp"].values
pressure = ds["msl"].values
temp = ds["t2m"].values
lat = ds.latitude.values
lon = ds.longitude.values
wind_speed, wind_dir = compute_wind(u, v)
points = []
for i in range(len(lat)):
for j in range(len(lon)):
point = {
"lat": float(lat[i]),
"lon": float(lon[j]),
"wind_speed": float(wind_speed[i][j]),
"wind_dir": float(wind_dir[i][j]),
"rain": float(rain[i][j]),
"temp": float(temp[i][j]),
"pressure": float(pressure[i][j])
}
points.append(point)
return points
def main():
for file in os.listdir(INPUT_DIR):
if not file.endswith(".grib2"):
continue
path = os.path.join(INPUT_DIR, file)
print("processing", file)
points = process_file(path)
output = os.path.join(
OUTPUT_DIR,
f"grid_{file.replace('.grib2','')}.json"
)
data = {
"time": file.replace(".grib2",""),
"points": points
}
with open(output, "w") as f:
json.dump(data, f)
print("saved", output)
if __name__ == "__main__":
main()

215
tasks/WeatherServer_GridBuilder_v2.md Normal file
View File

@@ -0,0 +1,215 @@
# NavSea Weather Server
Task: WeatherServer_GridBuilder_v2
Architecture: NavSea V11
Codex: codex6
Status: TODO
---
# 1 任务目标
升级 Grid Builder：
grid_builder.py → GridBuilder v2
目标：
1 裁剪日本区域 grid
2 使用数组结构保存 grid
3 大幅减少 JSON 文件大小
---
# 2 当前问题
旧版 GridBuilder：
生成全球 grid：
1440 × 721
≈ 1,038,240 points
JSON 文件：
≈ 215MB
这是不可接受的。
Downloader 已经只下载：
120E – 150E
20N – 50N
GridBuilder 必须只输出这个区域。
---
# 3 区域范围
REGION：
lon_min = 120
lon_max = 150
lat_min = 20
lat_max = 50
理论 grid：
(150-120)/0.25 = 120
(50-20)/0.25 = 120
≈ 14400 grid points
---
# 4 新 Grid 数据结构
旧结构：
points list
{
"points":[
{lat,lon,...}
]
}
新结构：
grid arrays
{
"time": "...",
"lat": [...],
"lon": [...],
"wind_speed": [...],
"wind_dir": [...],
"rain": [...],
"temp": [...],
"pressure": [...]
}
优点：
1 文件更小
2 读取更快
3 tile generator 更容易
---
# 5 创建文件
weather_server/grid/grid_builder_v2.py
---
# 6 实现代码
```python
import os
import json
import numpy as np
import xarray as xr
INPUT_DIR = "data/grib"
OUTPUT_DIR = "data/grid"
os.makedirs(OUTPUT_DIR, exist_ok=True)
REGION = {
"lon_min":120,
"lon_max":150,
"lat_min":20,
"lat_max":50
}
def compute_wind(u, v):
speed = np.sqrt(u**2 + v**2)
direction = (np.degrees(np.arctan2(u, v)) + 360) % 360
return speed, direction
def process_file(path):
ds = xr.open_dataset(path, engine="cfgrib")
u = ds["u10"].values
v = ds["v10"].values
rain = ds["tp"].values
pressure = ds["msl"].values
temp = ds["t2m"].values
lat = ds.latitude.values
lon = ds.longitude.values
wind_speed, wind_dir = compute_wind(u, v)
lat_idx = np.where(
(lat >= REGION["lat_min"]) &
(lat <= REGION["lat_max"])
)[0]
lon_idx = np.where(
(lon >= REGION["lon_min"]) &
(lon <= REGION["lon_max"])
)[0]
lat_region = lat[lat_idx]
lon_region = lon[lon_idx]
wind_speed = wind_speed[np.ix_(lat_idx, lon_idx)]
wind_dir = wind_dir[np.ix_(lat_idx, lon_idx)]
rain = rain[np.ix_(lat_idx, lon_idx)]
temp = temp[np.ix_(lat_idx, lon_idx)]
pressure = pressure[np.ix_(lat_idx, lon_idx)]
return {
"lat": lat_region.tolist(),
"lon": lon_region.tolist(),
"wind_speed": wind_speed.tolist(),
"wind_dir": wind_dir.tolist(),
"rain": rain.tolist(),
"temp": temp.tolist(),
"pressure": pressure.tolist()
}
def main():
for file in os.listdir(INPUT_DIR):
if not file.endswith(".grib2"):
continue
path = os.path.join(INPUT_DIR, file)
print("processing", file)
grid = process_file(path)
output = os.path.join(
OUTPUT_DIR,
f"grid_{file.replace('.grib2','')}.json"
)
data = {
"time": file.replace(".grib2",""),
"grid": grid
}
with open(output, "w") as f:
json.dump(data, f)
print("saved", output)
if __name__ == "__main__":
main()

248
tasks/WeatherServer_VectorTileGenerator.md Normal file
View File

@@ -0,0 +1,248 @@
# NavSea Weather Server
Task: WeatherServer_VectorTileGenerator
Architecture: NavSea V11
Codex: codex6
Status: TODO
---
# 1 任务目标
实现 Vector Tile Generator。
功能：
将 Weather Grid JSON 转换为 MapLibre Vector Tile (PBF)。
输入：
data/grid/*.json
输出：
output/weather/{time}/{z}/{x}/{y}.pbf
用于 NavSea 客户端加载天气图层。
---
# 2 使用库
需要安装：
pip install mercantile
pip install mapbox-vector-tile
---
# 3 输入数据
GridBuilder v2 生成：
data/grid/
grid_20260312_00_f000.json
grid_20260312_00_f003.json
结构：
{
"time": "...",
"grid": {
"lat": [...],
"lon": [...],
"wind_speed": [[...]],
"wind_dir": [[...]],
"rain": [[...]],
"temp": [[...]],
"pressure": [[...]]
}
}
grid 尺寸：
121 × 121
---
# 4 Tile Zoom 设计
Weather tile 只需要 4 个 zoom：
2
4
6
8
---
# 5 Tile 输出结构
output/weather/
time/
z/
x/
y.pbf
示例：
output/weather/20260312_00_f000/4/10/7.pbf
---
# 6 Feature 结构
每个 grid 点 → 一个 feature
geometry：
POINT(lon lat)
properties：
{
"ws": wind_speed
"wd": wind_dir
"r": rain
"t": temp
"p": pressure
}
字段缩写减少 tile 大小。
---
# 7 创建文件
weather_server/tiles/vector_tile_generator.py
---
# 8 实现代码
```python
import os
import json
import mercantile
import mapbox_vector_tile
GRID_DIR = "data/grid"
OUTPUT_DIR = "output/weather"
ZOOMS = [2,4,6,8]
def load_grid(path):
with open(path) as f:
data = json.load(f)
return data["time"], data["grid"]
def grid_to_features(grid):
lat = grid["lat"]
lon = grid["lon"]
ws = grid["wind_speed"]
wd = grid["wind_dir"]
rain = grid["rain"]
temp = grid["temp"]
pres = grid["pressure"]
features = []
for i in range(len(lat)):
for j in range(len(lon)):
feature = {
"geometry":{
"type":"Point",
"coordinates":[lon[j], lat[i]]
},
"properties":{
"ws":ws[i][j],
"wd":wd[i][j],
"r":rain[i][j],
"t":temp[i][j],
"p":pres[i][j]
}
}
features.append(feature)
return features
def generate_tiles(time, features):
for z in ZOOMS:
tiles = mercantile.tiles(120,20,150,50,z)
for tile in tiles:
bounds = mercantile.bounds(tile)
tile_features = []
for f in features:
lon, lat = f["geometry"]["coordinates"]
if (
bounds.west <= lon <= bounds.east
and bounds.south <= lat <= bounds.north
):
tile_features.append(f)
if not tile_features:
continue
layer = {
"weather": tile_features
}
tile_data = mapbox_vector_tile.encode(layer)
path = os.path.join(
OUTPUT_DIR,
time,
str(z),
str(tile.x)
)
os.makedirs(path, exist_ok=True)
filename = os.path.join(
path,
f"{tile.y}.pbf"
)
with open(filename,"wb") as f:
f.write(tile_data)
print("tile", time, z, tile.x, tile.y)
def main():
for file in os.listdir(GRID_DIR):
if not file.endswith(".json"):
continue
path = os.path.join(GRID_DIR, file)
print("processing", file)
time, grid = load_grid(path)
features = grid_to_features(grid)
generate_tiles(time, features)
if __name__ == "__main__":
main()

528
tasks/classfild.md Normal file
View File

@@ -0,0 +1,528 @@
# NavSea Data Classification & Validation Pipeline
Version: v1.0
Goal:
从当前数据库中的 PBF 数据自动：
1. 解析 at 字段
2. 构建语义属性表
3. 构建对象目录
4. 自动推导对象类型
5. 验证分类正确性
6. 统计 tile 密度
7. 输出 NavSea 分类报告
所有中间结果写入 MySQL 表。
数据库在 192.168.200.184. root/2chi9ks2
可以建立数据库，也可以使用python代码，必要的时候可以下载所需的包。
---
# TASK 1
Parse AT Attributes
Goal:
解析 properties 表中 k='at' 的 JSON 字符串。
Create Table:
```sql
CREATE TABLE IF NOT EXISTS at_attributes (
feature_id BIGINT,
k VARCHAR(100),
v TEXT
);
```
Logic:
1. 查询
```sql
SELECT feature_id, v
FROM properties
WHERE k='at';
```
2. v 是 JSON array
Example:
```
[["レイヤ","航路標識点"],["形状分類","シーバース灯"],["灯色","W (白)"]]
```
3. 解析后写入：
```
feature_id | k | v
--------------------
2682088 | レイヤ | 航路標識点
2682088 | 形状分類 | シーバース灯
2682088 | 灯色 | W (白)
```
Implementation:
Python
Libraries:
```
pymysql
json
```
Output:
```
table: at_attributes
```
---
# TASK 2
Build Feature Semantic Table
Goal:
将 feature + vt_layer + at 属性合并。
Create Table:
```sql
CREATE TABLE feature_semantic AS
SELECT
f.id AS feature_id,
f.vt_layer,
f.geom_type,
MAX(CASE WHEN a.k='分類' THEN a.v END) AS class_name,
MAX(CASE WHEN a.k='形状分類' THEN a.v END) AS shape_name,
MAX(CASE WHEN a.k='レイヤ' THEN a.v END) AS layer_name
FROM features f
LEFT JOIN at_attributes a
ON f.id = a.feature_id
GROUP BY f.id;
```
Output table:
```
feature_semantic
```
Columns:
```
feature_id
vt_layer
geom_type
class_name
shape_name
layer_name
```
---
# TASK 3
Generate Object Catalog
Goal:
统计所有语义对象。
Create Table:
```sql
CREATE TABLE object_catalog AS
SELECT
layer_name,
class_name,
shape_name,
vt_layer,
geom_type,
COUNT(*) AS feature_count
FROM feature_semantic
GROUP BY
layer_name,
class_name,
shape_name,
vt_layer,
geom_type;
```
Output:
```
object_catalog
```
Purpose:
得到完整对象目录。
Example:
```
魚礁 | p施設 | Point | 24683
灯台 | p航路標識群 | Point | 18000
等深線 | L等深線 | Line | 440000
```
---
# TASK 4
Geometry Consistency Check
Goal:
检查对象是否使用一致 geometry。
Create Table:
```sql
CREATE TABLE geometry_consistency AS
SELECT
class_name,
geom_type,
COUNT(*) AS feature_count
FROM feature_semantic
GROUP BY class_name, geom_type;
```
Output:
```
geometry_consistency
```
Purpose:
发现异常对象。
Example anomaly:
```
灯台 | Polygon
```
---
# TASK 5
Candidate Object Type Detection
Goal:
自动推导 object_type。
规则优先级：
1 class_name
2 shape_name
3 vt_layer
Create Table:
```sql
CREATE TABLE object_type_candidates AS
SELECT
feature_id,
COALESCE(class_name, shape_name, vt_layer) AS object_type,
geom_type
FROM feature_semantic;
```
Output:
```
object_type_candidates
```
---
# TASK 6
Object Type Statistics
Goal:
统计对象数量。
Create Table:
```sql
CREATE TABLE object_type_stats AS
SELECT
object_type,
geom_type,
COUNT(*) AS feature_count
FROM object_type_candidates
GROUP BY object_type, geom_type;
```
Output:
```
object_type_stats
```
Purpose:
识别主要对象。
---
# TASK 7
Style Cross Reference
Goal:
分析 style.json。
Extract:
```
layer_id
source-layer
icon-image
line-color
fill-color
```
Create Table:
```sql
CREATE TABLE style_layers (
layer_id VARCHAR(200),
source_layer VARCHAR(200),
icon VARCHAR(200),
line_color VARCHAR(200),
fill_color VARCHAR(200)
);
```
Join:
```sql
CREATE TABLE style_mapping AS
SELECT
s.layer_id,
s.icon,
o.object_type,
o.geom_type
FROM style_layers s
JOIN object_type_candidates o
ON s.source_layer = o.object_type;
```
Output:
```
style_mapping
```
Purpose:
确认对象 → 图标关系。
---
# TASK 8
Tile Density Analysis
Goal:
统计 tile feature 密度。
Create Table:
```sql
CREATE TABLE tile_density AS
SELECT
z,
x,
y,
COUNT(*) AS feature_count
FROM features
GROUP BY z,x,y;
```
Output:
```
tile_density
```
---
# TASK 9
Tile Density Top 100
Create Table:
```sql
CREATE TABLE tile_density_top100 AS
SELECT *
FROM tile_density
ORDER BY feature_count DESC
LIMIT 100;
```
Purpose:
识别高密度 tile。
---
# TASK 10
Spatial Sanity Checks
Goal:
发现明显错误。
Examples:
Navigation lights not point:
```sql
CREATE TABLE anomaly_navigation_geom AS
SELECT *
FROM feature_semantic
WHERE class_name='灯台'
AND geom_type!='Point';
```
Reef not point/polygon:
```sql
CREATE TABLE anomaly_reef_geom AS
SELECT *
FROM feature_semantic
WHERE class_name='魚礁'
AND geom_type NOT IN ('Point','Polygon');
```
---
# TASK 11
NavSea Classification Report
Generate markdown:
```
navsea_classification_report.md
```
Content:
## Dataset Summary
Total features:
```sql
SELECT COUNT(*) FROM features;
```
Total object types:
```sql
SELECT COUNT(DISTINCT object_type)
FROM object_type_candidates;
```
---
## Top Object Types
```sql
SELECT *
FROM object_type_stats
ORDER BY feature_count DESC
LIMIT 50;
```
---
## Geometry Consistency
```
geometry_consistency
```
---
## Style Mapping
```
style_mapping
```
---
## Tile Density
```
tile_density_top100
```
---
# Final Deliverables
Database Tables:
```
at_attributes
feature_semantic
object_catalog
geometry_consistency
object_type_candidates
object_type_stats
style_layers
style_mapping
tile_density
tile_density_top100
anomaly_navigation_geom
anomaly_reef_geom
```
Final Document:
```
navsea_classification_report.md
```
---
# Execution
Single command:
```
python navsea_audit.py
```
Pipeline:
```
parse_at
→ semantic_table
→ object_catalog
→ classification
→ validation
→ report
```
---
# Success Criteria
The system must allow answering:
1 What objects exist
2 How many features each object has
3 What geometry they use
4 How they are styled
5 Whether classification is consistent
6 Whether tile density is reasonable

151
tasks/firsttest.md Normal file
View File

@@ -0,0 +1,151 @@
# NavSea Weather Server Step-1
GRIB 解析环境测试
目标：
验证服务器可以正确解析 GRIB 文件。
---
# 1 安装系统依赖
在 Linux 服务器执行：
apt update
apt install -y python3 python3-pip
apt install -y libeccodes0 libeccodes-dev
apt install -y python3-eccodes
---
# 2 安装 Python 库
pip3 install cfgrib
pip3 install xarray
pip3 install numpy
---
# 3 下载测试 GRIB 文件
创建测试目录：
mkdir weather_test
cd weather_test
下载 NOAA GFS 示例：
wget https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260312/00/atmos/gfs.t00z.pgrb2.0p25.f003 -O test.grib2
（如果日期目录不存在，可以换最新日期）
---
# 4 创建测试脚本
创建文件：
test_grib.py
内容：
import xarray as xr
file = "test.grib2"
print("Opening GRIB file...")
ds = xr.open_dataset(file, engine="cfgrib")
print("Dataset loaded")
print(ds)
print("\nVariables:")
print(ds.data_vars)
---
# 5 运行测试
python3 test_grib.py
如果成功，你会看到类似输出：
Dataset loaded
Dimensions:
latitude: 721
longitude: 1440
Variables:
u10
v10
t2m
msl
tp
---
# 6 测试读取风数据
修改脚本：
import xarray as xr
import numpy as np
file = "test.grib2"
ds = xr.open_dataset(file, engine="cfgrib")
u = ds['u10']
v = ds['v10']
speed = np.sqrt(u**2 + v**2)
print("Wind sample:")
print(speed.values[0][0])
---
# 7 成功标志
如果成功运行：
说明：
GRIB解析正常
Python环境正常
可以进入 Weather Server 开发
---
# 8 常见错误
错误：
eccodes library not found
解决：
apt install libeccodes-dev
---
# 9 完成后下一步
下一步开发：
GRIB Downloader
模块：
weather_server/downloader/gfs_downloader.py
负责：
自动下载 GFS GRIB 数据。

413
tasks/pbf/AllPbf.md Normal file
View File

@@ -0,0 +1,413 @@
# NavSea Server Codex Tasks v2
Architecture: NavSea V11 / Codex6
Scope: **Server Side Only**
---
# Global Rules（全局强制规则）
以下规则适用于所有 NavSea Server Tasks。
## Rule 1 — 必须保留原始 PBF 属性
语义系统 **不得删除或覆盖任何原始 PBF feature 属性**。
必须保留包括但不限于：
* 日文对象名称
* 海图原始属性
* 灯塔信息
* 深度信息
* 渔区信息
* 海图符号属性
* 任何原始 metadata
语义字段只能 **新增**，不能替换。
示例：
```
class = 灯
shape = 灯標
name = 大島灯台
```
新增语义字段：
```
canonical_object_type = navigation_aid_light
object_family = navigation_aid
detection_class = navigation_mark
```
---
## Rule 2 — 不得修改 style.json 依赖属性
任何被 style.json 使用的属性字段 **必须保持不变**。
典型字段包括：
```
source_layer
class
shape
name
symbol
depth
type
category
```
语义系统 **不得重命名或删除这些字段**。
---
## Rule 3 — 语义字段只能新增
NavSea Semantic Overlay 只能增加字段，例如：
```
canonical_object_type
object_family
detection_class
capability
detection_key
```
不得改变：
```
geometry
source_layer
original_attributes
```
---
# Task 01 — Semantic Overlay Validation System v2
## 任务目标
构建验证系统，确保语义重建不会破坏原始 PBF 渲染结构。
验证内容包括：
* feature 数量一致
* geometry 类型一致
* style 渲染一致
* 原始属性完整保留
## 输入表
Codex 自动检测：
```
canonical_layer_rules
pbf_relayer_candidates
pbf_render_compatibility
pbf_detection_catalog
```
## 输出验证表
```
classification_validation
geometry_consistency
style_render_equivalence
attribute_preservation_check
semantic_validation_summary
```
## 新增验证
必须新增：
```
attribute_preservation_check
```
用于确认：
```
原始 PBF 属性未丢失
```
---
# Task 02 — Object Taxonomy Builder v2
## 任务目标
建立稳定的海图对象分类体系。
## 输出表
```
navsea_object_taxonomy
```
## 字段
```
canonical_object_type
object_family
detection_class
description
```
## 示例
```
rock → hazard → obstacle
wreck → hazard → obstacle
reef → hazard → obstacle
light → navigation_aid → navigation_mark
```
## 约束
Taxonomy 不能改变：
```
source_layer
original_attributes
```
---
# Task 03 — Object Capability System v2
## 任务目标
定义海图对象在航行中的能力（capability）。
## 输出表
```
navsea_object_capabilities
```
## 字段
```
canonical_object_type
capability
```
## 示例
```
rock → collision
wreck → collision
reef → grounding
fishery → entangle
light → navigation_mark
breakwater → wave_barrier
```
## 目的
使检测系统可以查询：
```
所有 collision 对象
所有 grounding 对象
```
---
# Task 04 — Detection Object View v2
## 任务目标
创建统一检测对象视图。
## 视图
```
navsea_detection_objects
```
## 字段
```
feature_id
geometry
canonical_object_type
object_family
detection_class
capability
detection_key
```
## 数据来源
```
pbf_relayer_candidates
navsea_object_taxonomy
navsea_object_capabilities
```
## 约束
该视图：
```
只读
不修改原始数据
```
---
# Task 05 — Spatial Index Builder v2
## 任务目标
优化空间查询性能。
当前数据规模：
```
~6.6 million features
```
## Codex 任务
自动分析 geometry 字段并创建空间索引。
优化查询：
```
半径查询
航线走廊查询
船舶前方查询
```
## 目标对象
```
navsea_detection_objects
semantic feature tables
```
## 约束
不得改变表结构。
---
# Task 08 — Semantic Layer Coverage Report v2
## 任务目标
统计语义系统覆盖率。
检测：
```
未分类对象
layer 覆盖率
canonical_object 覆盖率
```
## 输出表
```
semantic_layer_coverage
canonical_object_coverage
source_layer_coverage
unclassified_features
coverage_summary
```
## 用途
发现：
```
规则遗漏
新对象
数据异常
```
---
# Task 09 — Vector Tile Semantic Drift Detection v2
## 任务目标
检测 PBF 更新导致的语义漂移。
## 检测内容
```
新对象类型
source_layer 变化
geometry 类型变化
属性结构变化
语义规则冲突
```
## 输出表
```
semantic_drift_summary
new_object_types
source_layer_drift
geometry_drift
attribute_drift
rule_conflict_report
```
## 用途
确保新 PBF 数据不会破坏语义系统。
---
# 推荐执行顺序
```
1 Semantic Overlay Validation
2 Semantic Coverage Report
3 Object Taxonomy
4 Object Capability
5 Detection Object View
6 Spatial Index Builder
7 Drift Detection
```
---
# 最终服务器结构
完成全部任务后服务器语义系统结构：
```
PBF
↓
Semantic Reconstruction
↓
Taxonomy
↓
Capability
↓
Detection Objects
↓
Spatial Index
↓
Validation / Drift Monitoring
```
该结构确保：
```
PBF 稳定
原始属性保留
style.json 可安全重建
```
---

107
tasks/pbf/NavSea_Mapping_DB_Registry_v1.md Normal file
View File

@@ -0,0 +1,107 @@
# NavSea 映射关系数据库固化方案 v1
版本：v1
用途：把“旧版原始字段 / 层名 / 取值”和“新版标准化字段 / 语义 / 渲染语义”的关系正式固化到数据库里，作为后续规则审计、问题排查和构建回放的基础。
## 1 当前落地位置
本版已在 `pbf_analysis` 中建立以下正式对象：
- `navsea_rule_bundle`
- `navsea_source_layer_rules`
- `navsea_field_name_rules`
- `navsea_field_value_rules`
- `navsea_taxonomy_rules`
- `navsea_render_rules`
- `navsea_mapping_registry_vw`
- `navsea_legacy_to_new_mapping_vw`
- `navsea_new_to_legacy_mapping_vw`
## 2 规则源文件
数据库中的规则来自以下版本化文件：
- [navsea_rule_bundle_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_rule_bundle_v1.yaml)
- [navsea_source_layer_rules_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_source_layer_rules_v1.yaml)
- [navsea_field_name_rules_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_field_name_rules_v1.yaml)
- [navsea_field_value_rules_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_field_value_rules_v1.yaml)
- [navsea_taxonomy_rules_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_taxonomy_rules_v1.yaml)
- [navsea_render_rules_v1.yaml](/root/weather/tasks/pbf/mappings/navsea_render_rules_v1.yaml)
## 3 同步脚本
同步脚本：
- [navsea_mapping_registry_sync.py](/root/weather/navsea_mapping_registry_sync.py)
- [navsea_mapping_registry.py](/root/weather/navsea_mapping_registry.py)
作用：
- 建表
- 从 YAML 导入 bundle / layer / field / value / taxonomy / render 规则
- 生成统一审计视图
- 在运行时由 builder 直接读取数据库规则，不再在多个文件中重复维护同一份映射常量
## 4 双向查询口径
旧版查新版：
```sql
SELECT *
FROM navsea_legacy_to_new_mapping_vw
WHERE legacy_key = '分類番号' AND legacy_value = '428';
```
新版查旧版：
```sql
SELECT *
FROM navsea_new_to_legacy_mapping_vw
WHERE new_key = 'chart_fill_style' AND new_value = 'fish_reef_area';
```
## 5 当前 v1 覆盖内容
- `source-layer` 日文层名到标准层名
- 旧字段名到标准字段名
- 关键旧字段取值到标准值
- taxonomy 规则
- render 规则
## 6 当前边界
本版已经把映射关系正式固化到了数据库里，并且 builder 已开始直接消费数据库规则，当前已接入：
- `source-layer` 标准化
- 旧字段值到新字段值的标准化
- 首批 render 规则
但还没有把 builder 的全部 `infer_*` 分支完全改造成“只从数据库执行规则”。
当前状态是：
- 规则资产已经固化在数据库
- 代码侧只保留兜底逻辑，数据库规则优先
- 后续可以逐步把执行层切到数据库规则驱动
## 6.1 未决对象处理规则
本体系中的“兜底”不等于“静默归入一个大筐”。
当前执行约束是：
- 如果没有命中特定 render 规则，不允许把 `REN-FALLBACK` 当成正常命中结果
- 这类对象会被标记为 `trace_status = needs_review`
- builder 会在每次构建后自动输出一份 `mapping_audit` 报告
- 报告中会列出未决对象的来源层、对象类型、几何类型、样本 `fid` 和原因
这条约束的目的是：
- 不让未覆盖对象悄悄混进“正常渲染”
- 不让规则缺口被一个泛化 fallback 掩盖
- 让后续补规则时有明确的对象清单
## 7 建议下一步
- 增加 `navsea_feature_rule_trace` 的数据库写入
- 让 builder 在 engineering 口径记录实际命中的 `taxonomy_rule_id` / `render_rule_id`
- 逐步把 `infer_*` 硬编码分支迁移到规则表执行

153
tasks/pbf/NavSea_Object_Taxonomy_Manual_Review_Task.md Normal file
View File

@@ -0,0 +1,153 @@
# NavSea Codex Task
# Object Taxonomy Manual Review Package Generator
Architecture: NavSea V11 / Codex6 Scope: Server Side Documentation /
Review Preparation
------------------------------------------------------------------------
# Task Name
NavSea_Object_Taxonomy_Manual_Review_Package
------------------------------------------------------------------------
# Task Goal
Generate a manual review package for NavSea object taxonomy definition.
This task prepares a human-reviewable object classification package so
that the user can confirm:
1. canonical object naming
2. object family assignment
3. detection class assignment
4. whether mixed geometry objects should be unified or split
5. whether some source layers are container layers rather than semantic
layers
------------------------------------------------------------------------
# Inputs
object_catalog object_type_stats feature_semantic object_type_candidates
canonical_layer_rules canonical_object_rules pbf_source_object_stats
pbf_detection_catalog
Optional reference:
style_layers style_mapping
------------------------------------------------------------------------
# Output
NavSea_Object_Taxonomy_Manual_Review.md
------------------------------------------------------------------------
# Required Structure of the Review Document
## 1 Review Objective
Explain that this package is for manual confirmation of:
- canonical object type naming
- family grouping
- detection grouping
- geometry unification or splitting
- ambiguous object handling
------------------------------------------------------------------------
## 2 Review Priority Groups
Priority A --- Immediate navigation safety relevance
Examples: hazards, wrecks, rocks, reefs, dangerous obstacles, navigation
marks, channels, anchorage, restricted areas, fishery hazards
Priority B --- Navigation context / operational relevance
Examples: boundaries, facilities, subsea cables / pipelines, port
structures, land structures relevant to navigation
Priority C --- Background / rendering objects
Examples: labels, land background, rendering support layers
------------------------------------------------------------------------
## 3 Manual Review Candidate Table
Required columns:
review_priority layer_name class_name shape_name vt_layer
observed_geom_types total_feature_count suggested_canonical_object_type
suggested_canonical_family suggested_detection_class
suggested_review_reason needs_manual_review
needs_manual_review values:
YES NO
------------------------------------------------------------------------
## 4 Ambiguity Categories
### Multi-Geometry Ambiguity
Same object appearing as multiple geometry types.
### Navigation Meaning Ambiguity
Objects implying collision / grounding / entanglement risk.
### Source Layer Container Ambiguity
Layers that may act as container or rendering layers.
### Regulatory / Boundary Ambiguity
Objects representing legal or navigational boundaries.
------------------------------------------------------------------------
## 5 First-Pass Recommended Review Scope
Focus first on classes similar to:
航行危険障害物 投錨注意障害物 航路標識点 航路境界等 錨泊地等
施設・境界線等 海岸線等深線等 漁具定置箇所
------------------------------------------------------------------------
## 6 Proposed Review Decisions
Each candidate object should include:
Candidate Object Suggested canonical_object_type Suggested
canonical_family Suggested detection_class Reason Manual confirmation
required
------------------------------------------------------------------------
## 7 Layer Review Appendix
Columns:
source_layer semantic_granularity current_rule_guess review_reason
needs_manual_review
------------------------------------------------------------------------
## 8 Review Outcome Guidance
The user will confirm:
1 Confirm canonical object naming 2 Confirm family grouping 3 Confirm
detection grouping 4 Confirm geometry unification 5 Confirm container
layers
------------------------------------------------------------------------

36
tasks/pbf/NavSea_PBF_schema_v1.md Normal file
View File

@@ -0,0 +1,36 @@
# NavSea PBF Schema v1
用途：定义 NavSea Vector Tile 的数据结构。
## 顶层字段
- identity
- chart_semantics
- labels
- light
- hazard
- area
- source_legacy
## identity
- fid
- canonical_object_type
- canonical_family
- detection_key
- render_layer
## chart_semantics
- chart_render_type
- chart_symbol_family
- chart_symbol_code
- chart_priority
- chart_visibility_min
- chart_visibility_max
## labels
- chart_label_text
- chart_label_subtext
- chart_label_anchor

226
tasks/pbf/NavSea_PBF_schema_v2.md Normal file
View File

@@ -0,0 +1,226 @@
# NavSea PBF Schema v2
版本：v2 Draft
用途：定义 NavSea 交付用 Vector Tile 的字段结构，兼顾渲染、碰撞、检索和旧海图兼容。
## 1 设计目标
v2 schema 要同时满足四件事：
- 保持旧海图渲染内容不明显缩水
- 支撑新的语义渲染体系
- 支撑碰撞检测和检索
- 控制字段职责，避免样式继续直接依赖大量原始编码
## 2 字段分组
v2 建议把交付字段分成 8 组：
- identity
- taxonomy
- render_semantics
- labels
- light
- hazard_area
- normalized_numeric
- source_legacy
## 3 identity
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `fid` | string | 是 | NavSea 自有对象标识，使用可逆算法生成的 16 进制字符串 |
| `fid_legacy_raw` | integer/string | 否 | 旧版原始 `fid`，建议只保留在 engineering 口径或 SQL trace |
| `fid_algo_id` | string | 否 | `fid` 生成算法标识，例如 `feistel32_aes_cyclewalk_v1` |
| `fid_key_id` | string | 否 | `fid` 生成时使用的密钥标识 |
| `render_layer` | string | 是 | 交付瓦片中的目标图层名 |
| `source_layer` | string | 否 | 原始图层名，建议保留到工程版或过渡期交付版 |
说明：
- 对“只做图片渲染”的产品可以弱化 `fid`
- 对当前 NavSea 的交付目标，不建议删除 `fid`
- 不建议继续直接暴露旧版原始 `fid`
- 建议基于 `tasks/pbf/fiddecode.md` 的可逆算法生成 NavSea 自有 `fid`
- 工程实现建议使用 `signed int32` 全域作为 legacy `fid` 的可逆映射范围
## 4 taxonomy
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `canonical_family` | string | 是 | 大类语义 |
| `canonical_object_type` | string | 是 | 细粒度对象类型 |
| `detection_key` | string | 是 | 用于检测和规则系统的稳定分类键 |
| `semantic_key` | string | 否 | 内部语义归并键，过渡期可保留 |
## 5 render_semantics
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `chart_render_type` | string | 是 | `symbol` / `line` / `fill` / `label` / `none` |
| `chart_symbol_family` | string | 否 | 符号族，例如 `navigation_light`、`hazard`、`depth_mark` |
| `chart_symbol_code` | string | 否 | 稳定符号码，例如 `lighthouse`、`rock_awash` |
| `chart_line_style` | string | 否 | 线型预设，例如 `boundary_dashed`、`contour_major` |
| `chart_fill_style` | string | 否 | 面填充预设，例如 `depth_zone_0_5`、`reef_area` |
| `chart_text_style` | string | 否 | 文字样式预设，例如 `place_name_sea`、`light_label` |
| `chart_priority` | integer | 是 | 绘制优先级，用于层级和碰撞 |
| `chart_visibility_min` | integer | 否 | 最小显示 zoom |
| `chart_visibility_max` | integer | 否 | 最大显示 zoom |
| `chart_collision_group` | string | 否 | 文本或符号碰撞组，例如 `place_label`、`light_label` |
说明：
- `chart_render_type` 和 `chart_priority` 建议作为最基础必填渲染字段
- 样式层优先读取 `chart_*` 字段，不再优先读取旧编码
## 6 labels
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `chart_label_text` | string | 否 | 主标注文本 |
| `chart_label_subtext` | string | 否 | 次级标注，例如灯略记 |
| `chart_label_anchor` | string | 否 | `top` / `bottom` / `left` / `right` / `center` |
| `chart_label_dx` | number | 否 | X 偏移 |
| `chart_label_dy` | number | 否 | Y 偏移 |
说明：
- 旧样式中的 `名称`、`名称補助`、`日本語地名`、`英文字地名` 仍可保留
- 新样式应优先使用 `chart_label_*`
## 7 light
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `light_color_code` | string | 否 | `white` / `red` / `green` / `yellow` / `mixed` |
| `light_character_code` | string | 否 | `Fl` / `Oc` / `Iso` / `F` / `V-AIS` |
| `light_sector_mode` | string | 否 | `sector` / `omni` / `none` |
| `light_arc_code` | string | 否 | 灯弧附加表达，是否交付由产品决定 |
说明：
- 灯弧本身可以不渲染
- 但灯质、灯色、略记对检索和标签仍有价值
## 8 hazard_area
| 字段 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `hazard_class` | string | 否 | `rock` / `reef` / `wreck` / `obstruction` / `shoal` |
| `hazard_severity` | string | 否 | `critical` / `major` / `minor` / `context` |
| `area_usage_class` | string | 否 | `anchorage` / `route` / `fishery` / `restricted` / `water` |
## 9 normalized_numeric
建议把部分旧数值字段归一化，方便检索和规则判断。
| 字段 | 类型 | 必填 | 来源建议 |
| --- | --- | --- | --- |
| `depth_value_m` | number | 否 | 由 `水深値(m)` 归一化 |
| `clearance_height_m` | number | 否 | 由 `高さ(m)` 归一化 |
| `least_depth_m` | number | 否 | 由 `高さ/深度(m)` 中深度语义归一化 |
| `bearing_deg` | number | 否 | 由 `角度` 归一化 |
说明：
- 这些字段主要服务于检索、检测和统一显示
- 原始字段在过渡期可并存
## 10 source_legacy
以下字段是根据当前旧 `style.json` 的实际依赖，建议在 v2 过渡期继续保留的旧字段：
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `灯色`
- `灯略記`
- `明弧/分孤`
- `表示位置`
- `名称`
- `名称補助`
- `日本語地名`
- `英文字地名`
- `水深値(m)`
- `高さ(m)`
- `高さ/深度(m)`
- `角度`
说明：
- 当前旧样式实际还使用了船舶动态字段 `cog`、`sog`、`hdg`、`length`、`rot`、`delay`
- 这些字段来自其他源，不属于 NavSea 静态海图对象 schema 的核心范围
## 11 v2 最低交付要求
每个 NavSea feature 至少应满足：
- `fid`
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `render_layer`
- `chart_render_type`
- `chart_priority`
## 12 工程版与交付版
建议分两种输出口径：
### 12.1 delivery schema
用途：前端渲染、检索、碰撞、正式交付。
保留：
- identity
- taxonomy
- render_semantics
- labels
- light
- hazard_area
- normalized_numeric
- 关键 `source_legacy`
### 12.2 engineering schema
用途：调试、抽查、问题复盘、映射核验。
在 delivery schema 基础上可额外保留：
- `fid_legacy_raw`
- `fid_algo_id`
- `fid_key_id`
- 更完整的原始 `at`
- 更多 `source_layer` / 原始属性
- 临时校验字段
## 13 v2 与当前 builder 的差距
当前 `navsea_tile_builder.py` 已经输出：
- 原始全部属性
- `canonical_object_type`
- `canonical_family`
- `semantic_key`
- `detection_key`
- `render_layer`
当前尚未系统输出的 v2 关键字段包括：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_text_style`
- `chart_priority`
- `chart_visibility_min`
- `chart_visibility_max`
- `chart_label_*`
- `light_*`
- `hazard_*`
- `area_usage_class`
- `normalized_numeric`
这部分是 builder 下一阶段的主要补齐目标。

331
tasks/pbf/NavSea_PBF_v1_渲染语义契约草案.md Normal file
View File

@@ -0,0 +1,331 @@
# NavSea PBF v1 渲染语义契约草案
版本：Draft v1
用途：用于 NavSea PBF 重构讨论与 Codex 实现参考
---
# 1 重构目标
NavSea 当前 PBF 渲染依赖大量旧海图编码字段，例如：
- 分類番号
- 形状分類番号
- 表示用番号
- 灯色
- 灯略記
- 明弧 / 分孤
- 表示位置
这些字段本质上是 **旧海图绘制编码**，而不是稳定的语义描述。
当前系统同时存在两种表达：
1. 旧海图渲染编码
2. 新的语义分类字段
但缺少一层专门表达 **“海图上应该如何表达该对象”** 的语义层。
因此本重构目标是建立：
NavSea PBF 的稳定 **渲染语义契约层**。
---
# 2 三层模型
NavSea 渲染体系应拆分为三层：
## A 层：对象语义层
回答：
**这个对象是什么**
示例字段：
- canonical_object_type
- canonical_family
- detection_key
示例值：
- 港湾灯台
- 暗岩
- 魚礁
- 港則法境界
---
## B 层：渲染语义层
回答：
**在海图上应该如何表达**
示例字段：
- chart_render_type
- chart_symbol_family
- chart_symbol_code
- chart_priority
- chart_visibility_min
- chart_visibility_max
---
## C 层：样式实现层
由 MapLibre Style 决定：
- 颜色
- 线型
- 图标
- 填充
- 文字样式
样式层应尽量只依赖：
- chart_* 渲染语义字段
- 少量 legacy 字段
---
# 3 建议的核心渲染字段
## 3.1 主渲染字段
| 字段 | 类型 | 说明 |
|-----|------|------|
| chart_render_type | string | symbol / line / fill / label / none |
| chart_symbol_family | string | 符号族 |
| chart_symbol_code | string | 符号代码 |
| chart_priority | integer | 渲染优先级 |
| chart_visibility_min | integer | 最小显示 zoom |
| chart_visibility_max | integer | 最大显示 zoom |
---
## 3.2 标签字段
| 字段 | 类型 |
|-----|------|
| chart_label_text | string |
| chart_label_subtext | string |
| chart_label_anchor | string |
| chart_label_dx | number |
| chart_label_dy | number |
---
## 3.3 灯标字段
| 字段 | 类型 |
|-----|------|
| light_color_code | string |
| light_character_code | string |
| light_sector_mode | string |
| light_arc_code | string |
示例：
light_color_code = white
light_character_code = Fl
light_sector_mode = sector
---
## 3.4 危险物字段
| 字段 | 类型 |
|-----|------|
| hazard_class | string |
| hazard_severity | string |
hazard_class 示例：
- rock
- reef
- wreck
- obstruction
- shoal
hazard_severity 示例：
- critical
- major
- minor
- context
---
## 3.5 区域字段
| 字段 | 类型 |
|-----|------|
| area_usage_class | string |
示例：
- anchorage
- route
- fishery
- restricted
- facility
- land
- water
---
# 4 不建议直接写入 PBF 的字段
以下字段如果直接写入 PBF，可能会锁死样式：
- chart_line_style
- chart_fill_style
- chart_text_style
原因：
这些字段已经接近 **样式预设名称**，而不是语义。
例如：
depth_zone_0_5
boundary_dashed
place_name
未来若更换视觉风格，会产生耦合。
因此建议 PBF 只表达：
- line semantic
- fill semantic
- label role
具体视觉样式交由 Style 决定。
---
# 5 建议保留的 legacy 字段
为了兼容旧样式与历史数据，建议继续保留：
分類番号
形状分類番号
表示用番号
灯色
灯略記
明弧
分孤
表示位置
名称
名称補助
日本語地名
英文字地名
用途：
- 样式兜底
- 检索
- 历史兼容
---
# 6 Feature 示例
## 灯台
{
"fid": 12345678,
"canonical_object_type": "港湾灯台",
"canonical_family": "navigation_aid",
"detection_key": "symbol:港湾灯台",
"chart_render_type": "symbol",
"chart_symbol_family": "navigation_light",
"chart_symbol_code": "lighthouse",
"chart_priority": 900,
"chart_visibility_min": 7,
"chart_label_text": "鷹島灯台",
"chart_label_subtext": "Fl W 5s",
"light_color_code": "white",
"light_character_code": "Fl"
}## 鱼礁危险区
{
"fid": 23456789,
"canonical_object_type": "魚礁",
"canonical_family": "hazard",
"detection_key": "mixed:魚礁",
"chart_render_type": "fill",
"chart_symbol_family": "hazard",
"chart_symbol_code": "reef_area",
"hazard_class": "reef",
"hazard_severity": "major",
"chart_priority": 850
}
---
# 7 建议的实现顺序
阶段 1：
出瓦片流程新增 chart_* 字段。
阶段 2：
PBF 同时包含：
- 新 taxonomy 字段
- 新 chart semantics
- legacy 字段
阶段 3：
样式优先读取：
chart_render_type
chart_symbol_family
chart_symbol_code
缺失时回退 legacy 字段。
阶段 4：
逐步减少样式对 legacy 编码依赖。
---
# 8 第一阶段优先实现对象
建议优先完成四类对象：
1 灯标 / 航标
2 危险点对象
3 危险面对象（鱼礁 / 礁盘 / 障碍区）
4 边界对象
原因：
这些对象最依赖旧编码，也是最复杂的渲染类型。
---
# 9 结论
NavSea PBF 不应继续依赖旧海图编码作为核心渲染逻辑。
正确方向是：
在 **原始属性** 与 **样式实现** 之间建立稳定的 **渲染语义层**。
该层应：
- 表达稳定语义
- 不锁定具体视觉样式
- 支持新旧样式共存
- 为检测、检索、渲染提供统一数据面

334
tasks/pbf/NavSea_Reversible_Normalization_Audit_Plan_v1.md Normal file
View File

@@ -0,0 +1,334 @@
# NavSea 可逆标准化审计方案 v1
版本：v1 Draft
用途：定义 NavSea 可逆标准化规则的审计方案，用于验证标准化过程是否完整、可逆、稳定、可重复、对渲染无破坏。
## 1 审计目标
本审计方案要验证五件事：
- 原始信息有没有丢
- 标准化规则有没有被正确应用
- 每个结果能不能回查
- 同一规则重复执行是否稳定
- 标准化后渲染和业务能力有没有被破坏
## 2 审计对象
审计覆盖以下对象：
- 规则文件
- SQL 规则表
- feature trace 表
- delivery `pbf`
- engineering `pbf`
- style 消费关系
## 3 审计维度
## 3.1 完整性审计
检查是否所有需要标准化的对象都进入了规则体系。
检查项：
- 每个实际出现的 `source_layer_jp` 都有标准化规则
- 每个实际出现的关键原始字段都有保留策略
- 每个 delivery feature 都有对应 trace
## 3.2 可逆性审计
检查是否能从标准值回查原始值。
检查项：
- `source_layer_std` 是否能映射回 `source_layer_jp`
- 标准化字段是否有对应规则 ID
- feature 是否能查到 bundle 版本
## 3.3 稳定性审计
检查相同输入在相同规则版本下是否得到相同输出。
检查项：
- 同一输入重复运行结果一致
- 同一 `rule_id + revision` 输出不漂移
## 3.4 渲染兼容审计
检查标准化后是否破坏当前样式表现。
检查项：
- style 引用的 layer 是否仍然存在
- 关键对象是否仍能显示
- 旧兼容表现是否没有异常缩水
## 3.5 业务可用性审计
检查标准化后是否仍满足检索、碰撞、点击、排障需求。
检查项：
- `fid` 是否保留
- taxonomy 字段是否保留
- 关键内容型日文字段是否保留
- trace 是否可回查
## 4 审计表设计
建议新增以下审计表：
- `navsea_normalization_audit_run`
- `navsea_normalization_coverage_audit`
- `navsea_normalization_reversibility_audit`
- `navsea_normalization_stability_audit`
- `navsea_render_compatibility_audit`
## 5 每次审计运行应记录
建议在 `navsea_normalization_audit_run` 中记录：
- `audit_run_id`
- `bundle_id`
- `bundle_version`
- `source_snapshot`
- `target_scope`
- `started_at`
- `finished_at`
- `status`
- `notes`
## 6 审计检查项
## 6.1 规则覆盖检查
目标：确认没有“实际数据里出现，但没有规则”的对象。
建议 SQL：
- 所有实际出现的 `source_layer_jp`
- 左连接 `navsea_source_layer_rules`
- 找出未命中项
通过标准：
- 未命中数量必须为 `0`
## 6.2 原值保留检查
目标：确认原始关键字段没有在工程体系中丢失。
检查字段：
- `fid`
- `fid_legacy_raw`
- `source_layer_jp`
- `分類番号`
- `形状分類番号`
- `表示用番号`
- `名称`
- `日本語地名`
- `at_raw`
通过标准：
- engineering 口径中必须可回查
## 6.2.1 `fid` 可逆检查
目标：确认 NavSea 自有 `fid` 不是旧 `fid` 原样外露，且能可逆回查。
检查项：
- `delivery` / `engineering` 主 `fid` 是否为 16 进制字符串
- `fid_legacy_raw` 是否保留于 engineering / trace
- `fid_algo_id` 是否存在
- `fid_key_id` 是否存在
- 是否可通过 `decrypt_number` 从 `fid_navsea_int` 还原 `fid_legacy_raw`
通过标准：
- 抽样回解正确率必须为 `100%`
- 不允许出现“新 `fid` 与旧 `fid` 只是格式改写”的情况
## 6.3 规则追溯检查
目标：确认每个标准化结果都有 trace。
检查项：
- `source_layer_std` 是否有 `source_layer_rule_id`
- `canonical_object_type` 是否有 `taxonomy_rule_id`
- `chart_*` 是否有 `render_rule_id`
通过标准：
- 关键标准化字段 trace 覆盖率必须达到 `100%`
## 6.4 可逆回查检查
目标：确认能从标准值回查原值。
抽样要求：
- 随机抽样 `fid`
- 随机抽样 `source_layer_std`
- 随机抽样 `canonical_object_type`
- 随机抽样 `chart_symbol_code`
检查：
- 是否能回查 `fid_legacy_raw`
- 是否能回查原始 `source_layer_jp`
- 是否能回查 `source_fields_used_json`
- 是否能回查规则文件版本
## 6.5 重放一致性检查
目标：确认相同输入反复执行不会漂移。
方法：
1. 使用同一 bundle 版本运行两次
2. 比较以下字段：
- `source_layer_std`
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `chart_render_type`
- `chart_symbol_code`
通过标准：
- 差异率必须为 `0`
## 6.6 delivery / engineering 差异检查
目标：确认 delivery 简化没有破坏回查能力。
检查项：
- delivery 是否去掉了一部分原值字段
- engineering 是否仍保留完整 trace
- 是否能通过 `fid` 从 delivery 回查到 engineering / SQL trace
通过标准：
- 回查链条必须完整
## 6.7 渲染覆盖检查
目标：确认标准化后 style 仍消费当前存在的 `source-layer`。
检查项：
- 数据侧实际 `source-layer` 集合
- 样式侧引用的 `source-layer` 集合
- 差集
通过标准：
- 若目标是“全覆盖样式”，差集必须为 `0`
- 若目标是“有意忽略样式”，必须给出 ignore 清单和理由
## 7 审计输出
每次审计至少输出四类结果：
### 7.1 覆盖报告
内容：
- 实际 `source-layer`
- 已映射数量
- 未映射数量
- 未映射列表
### 7.2 可逆性报告
内容：
- 可回查率
- 无 trace feature 数量
- 缺失规则 ID 数量
### 7.3 稳定性报告
内容：
- 两次重放差异数
- 差异字段分布
- 受影响 feature 列表
### 7.4 渲染兼容报告
内容：
- style 消费的 `source-layer`
- 数据存在但样式未消费的层
- 关键对象抽样截图或抽样核验结果
## 8 审计频率
建议分三类：
### 8.1 每次规则变更后
必须执行：
- 规则覆盖检查
- 规则追溯检查
- 可逆回查检查
### 8.2 每次大批量构建后
必须执行：
- delivery / engineering 差异检查
- 渲染覆盖检查
- 抽样渲染兼容检查
### 8.3 每次 bundle 升级前
必须执行：
- 重放一致性检查
- 差异审计
## 9 不通过条件
出现以下任一情况，审计应判定不通过：
- 实际出现的 `source-layer` 无规则
- 关键标准化字段无 trace
- 无法从标准值回查原值
- 同版本重复执行结果漂移
- delivery 无法通过 `fid` 回查 engineering / SQL trace
- 样式消费覆盖与目标不符且无明确豁免
## 10 v1 最低可执行方案
如果要先快速落地，建议第一阶段至少做到：
1. 建 `source-layer` 标准化规则表
2. 建 `fid` 标准化规则文件和 `fid` 可逆回查检查
3. 在 trace 表中补 `source_layer_rule_id`
4. 建立 `source_layer_jp -> source_layer_std` 覆盖审计
5. 建立 `fid -> trace` 回查检查
6. 每次样式更新都跑 `source-layer` 覆盖差集
## 11 结论
NavSea 的可逆标准化不能只靠规则文档存在，必须配套审计。
审计的核心不是“看起来转换成功”，而是验证：
- 原值未丢
- 规则有据
- 结果可回查
- 重放可一致
- 渲染不被破坏
只有规则和审计同时成立，标准化这件事才算真正严谨。

356
tasks/pbf/NavSea_Reversible_Normalization_Rules_v1.md Normal file
View File

@@ -0,0 +1,356 @@
# NavSea 可逆标准化规则 v1
版本：v1 Draft
用途：定义 NavSea 在“日文原始数据 -> 标准化字段/层名/渲染语义”过程中必须遵循的可逆标准化规则，确保转换严谨、可追溯、可回退、可重复执行。
## 1 目标
NavSea 的标准化不是“清洗掉旧数据”，而是：
- 保留原始信息
- 新增标准化结果
- 记录完整映射过程
- 允许从标准化结果反查原始值
一句话原则：
> 标准化只能增加表达层，不能破坏原始信息层。
## 2 适用范围
本规则适用于以下对象：
- 日文 `source-layer`
- 原始日文字段名
- 原始日文字段值
- taxonomy 标准化字段
- render 语义字段
- 输出 `pbf`
- SQL 规则表与 trace 表
## 3 核心原则
### 3.1 原始值不可丢
所有参与标准化的关键输入都必须有原始保留位。
包括但不限于：
- 原始 `source-layer`
- 原始字段名
- 原始字段值
- 原始 `at`
- 原始 `fid`
### 3.2 标准化结果必须新增，不可覆盖原值
禁止直接把原值物理改写成标准值后不留痕迹。
正确做法：
- 原值保留
- 标准值新增
- 二者由规则表与 trace 表关联
### 3.3 每一步标准化都必须有规则来源
任何标准化结果都必须能回答：
- 是哪条规则做的
- 规则版本是什么
- 使用了哪些原始字段
### 3.4 每一步标准化都必须可逆
可逆的含义不是“自动完全恢复所有原始结构”，而是：
- 能从标准值回查原始值
- 能从 feature 回查到规则和版本
- 能重演同一批输入得到同一批输出
### 3.5 delivery 可以简化，engineering 不可失真
delivery `pbf` 可以只暴露必要字段。
但 engineering `pbf` / SQL 审计表必须保留足够信息，支持回查。
## 4 不可做的事
以下操作在 v1 规则下视为禁止：
- 直接删除原始 `source-layer`，且不保留 `source_layer_jp`
- 直接删除原始字段值，且不保留原值副本
- 在 SQL 里手工改结果，不回写规则文件
- 在代码里加临时 if 分支，不形成规则 ID 和版本
- 只保留标准化结果，不保留 trace
## 5 字段分层
NavSea 标准化后，字段应分成三层。
### 5.1 原始层
职责：完整保留输入信息。
建议字段：
- `source_layer_jp`
- `original_properties_json`
- `at_raw`
- `fid`
- 原始 `分類番号`
- 原始 `形状分類番号`
- 原始 `表示用番号`
- 原始 `名称`
- 原始 `日本語地名`
### 5.2 标准化层
职责：输出结构化、可消费的标准字段。
建议字段：
- `source_layer_std`
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_text_style`
### 5.3 trace 层
职责：连接“原始层”和“标准化层”。
建议字段：
- `normalization_bundle_id`
- `source_layer_rule_id`
- `taxonomy_rule_id`
- `render_rule_id`
- `rule_revision`
- `source_fields_used_json`
- `trace_status`
## 6 可逆标准化对象
## 6.1 `source-layer` 标准化
建议采用双字段：
- `source_layer_jp`
- `source_layer_std`
示例：
- `source_layer_jp = P錨泊地等`
- `source_layer_std = anchorage_area`
要求：
- 原始日文层名不可直接丢弃
- 标准层名必须来自规则表，不得手工散落在代码里
## 6.1.1 `fid` 标准化
NavSea 不应直接对外继续使用旧版原始 `fid` 作为最终对象 ID。
规则要求：
- `delivery` / `engineering` 对外主 `fid` 使用 NavSea 自有 `fid`
- NavSea 自有 `fid` 由旧版 `fid` 经 `tasks/pbf/fiddecode.md` 中的可逆算法生成
- 生成结果使用 16 进制字符串表达
建议定义：
- `fid_legacy_raw`：旧版原始整数 `fid`
- `fid_navsea_int`：经过可逆加密后的非负整数
- `fid`：最终对外使用的 16 进制字符串，例如固定宽度大写 HEX
- `fid_algo_id`：例如 `feistel32_aes_cyclewalk_v1`
- `fid_key_id`：用于标识采用了哪把密钥
建议生成规则：
1. 读取旧版 `fid`
2. 使用 `fiddecode.md` 中的 `encrypt_number` 算法生成 `fid_navsea_int`
3. 将 `fid_navsea_int` 编码为固定宽度 16 进制字符串
4. 在 engineering / trace 中保留 `fid_legacy_raw`
补充口径：
- `fid` 可逆算法沿用 `fiddecode.md` 中的 Feistel + AES cycle-walking 方案
- 为保证对真实历史数据全覆盖，工程实现应采用 `signed int32` 全域
- 不应把示例文档中的局部样本范围误当成生产限制
要求：
- `fid` 必须可逆回查到 `fid_legacy_raw`
- 不允许只保留新 `fid`，却丢失旧 `fid`
- 不允许直接把旧 `fid` 原样转成 HEX 当作“新 `fid`”
## 6.2 原始字段名标准化
对原始字段名建议也采用双字段表达。
例如：
- `field_name_jp = 形状分類番号`
- `field_name_std = shape_class_code`
注意：
- 这里只是定义映射关系
- 不代表 delivery `pbf` 必须立刻把所有原字段名换掉
## 6.3 原始字段值标准化
例如：
- `灯色 = 1` -> `light_color_code = white`
- `形状分類番号 = 335` -> `nav_mark_variant = vais`
要求：
- 原值保留
- 标准值新增
- 必须记录规则 ID
## 7 保留策略
## 7.1 必须保留的原始日文内容
以下属于“内容型日文”，不能因为标准化就删除：
- `名称`
- `名称補助`
- `日本語地名`
- `灯略記`
- 其他对展示、检索、业务解释有价值的原始说明文本
## 7.2 可标准化但需保留原值的结构型日文
以下属于“结构型日文”，可以标准化，但必须保留原值：
- 日文 `source-layer`
- 原始字段名
- `at` 中的结构型键值
- 旧样式直接消费的技术性编码字段
## 7.3 delivery 可隐藏，engineering 必保留
对结构型日文建议：
- delivery `pbf` 可以只暴露标准字段
- engineering `pbf` / SQL 审计表必须保留原值和映射关系
## 8 规则文件要求
为保证可逆与可重复，以下规则必须文件化：
- `source-layer` 标准化规则
- taxonomy 规则
- render 规则
- 字段名标准化规则
- 字段值标准化规则
建议目录：
```text
tasks/pbf/mappings/
navsea_fid_rules_v1.yaml
navsea_source_layer_rules_v1.yaml
navsea_taxonomy_rules_v1.yaml
navsea_render_rules_v1.yaml
navsea_field_name_rules_v1.yaml
navsea_field_value_rules_v1.yaml
navsea_rule_bundle_v1.yaml
```
## 9 SQL 结构要求
为保证可逆与可查询，建议至少有以下表：
- `navsea_rule_bundle`
- `navsea_source_layer_rules`
- `navsea_taxonomy_rules`
- `navsea_render_rules`
- `navsea_field_name_rules`
- `navsea_field_value_rules`
- `navsea_feature_rule_trace`
## 10 feature trace 最低要求
每个被标准化的 feature 至少应记录：
- `feature_id`
- `fid`
- `fid_legacy_raw`
- `fid_navsea_int`
- `fid_algo_id`
- `fid_key_id`
- `source_layer_jp`
- `source_layer_std`
- `normalization_bundle_id`
- `source_layer_rule_id`
- `taxonomy_rule_id`
- `render_rule_id`
- `source_fields_used_json`
- `trace_status`
## 11 delivery / engineering 输出要求
## 11.1 delivery pbf
目的：给前端渲染和检索使用。
要求：
- 可适度简化字段
- 但不能影响业务使用
- 若隐藏原值，必须仍可通过 SQL trace 回查
## 11.2 engineering pbf
目的：审计、复盘、问题定位。
要求：
- 保留原值
- 保留标准值
- 保留规则 ID
- 保留 bundle 版本
## 12 可逆性验收条件
只有同时满足以下条件，才可称为“可逆标准化”：
1. 能从标准值回查原始值
2. 能从 feature 回查规则 ID 和版本
3. 能重放同一批输入并得到同一批输出
4. 原始内容型日文未被误删
5. delivery 的简化不影响 engineering 的回查能力
## 13 v1 落地建议
第一阶段建议先做到：
1. 所有日文 `source-layer` 增加 `source_layer_std`
2. `fid` 改为 NavSea 自有 16 进制 `fid`
3. 新增 `fid` 规则文件和 `fid` trace 字段
4. 新增 `source-layer` 规则文件
5. 在 trace 表中记录 `source_layer_rule_id`
6. 保留原始内容型日文字段
7. 严禁在没有 trace 的情况下直接删除原始结构字段
## 14 结论
NavSea 的标准化必须是：
- 可逆的
- 可追溯的
- 可重复执行的
- 区分原始层、标准化层和 trace 层的
换句话说，标准化不是“替换”，而是“带证据的映射”。

30
tasks/pbf/NavSea_object_render_mapping_table_v1.md Normal file
View File

@@ -0,0 +1,30 @@
# NavSea Object → Render Mapping Table v1
版本：v1 Draft
用途：定义对象语义到渲染语义的映射。
canonical_object_type → chart_render_type / chart_symbol_family / chart_symbol_code
## 灯标
| canonical_object_type | render_type | symbol_family | symbol_code |
|---|---|---|---|
| 港湾灯台 | symbol | navigation_light | lighthouse |
| 灯柱 | symbol | navigation_light | light_beacon |
| 灯浮標 | symbol | navigation_light | light_buoy |
## 危险物
| canonical_object_type | render_type | symbol_family | symbol_code |
|---|---|---|---|
| 暗岩 | symbol | hazard | rock_awash |
| 岩礁 | symbol | hazard | reef |
| 沈船 | symbol | hazard | wreck |
| 障害物 | symbol | hazard | obstruction |
## 区域
| canonical_object_type | render_type | symbol_family | symbol_code |
|---|---|---|---|
| 魚礁 | fill | hazard | reef_area |
| 錨地 | fill | usage_area | anchorage_area |

165
tasks/pbf/NavSea_object_render_mapping_table_v2.md Normal file
View File

@@ -0,0 +1,165 @@
# NavSea Object -> Render Mapping Table v2
版本：v2 Draft
用途：定义对象语义到渲染语义的映射规则，作为 `taxonomy -> render_semantics` 的中间规范。
## 1 设计原则
- 先按 `canonical_object_type` 映射
- 再按几何类型和旧字段细化具体样式
- 样式层优先读取 `chart_*` 字段
- 旧字段只保留为细节兜底，不再作为主判定入口
## 2 输出字段
映射表的目标输出至少包括：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_text_style`
- `chart_priority`
- `chart_label_text`
- `chart_label_subtext`
- `light_color_code`
- `light_character_code`
- `hazard_class`
- `hazard_severity`
- `area_usage_class`
## 3 首批强制覆盖对象
下表是建议优先落地的 v2 核心映射对象。
| canonical_family | canonical_object_type | geometry | chart_render_type | symbol_family | symbol_or_style | priority | legacy_detail_dependency |
| --- | --- | --- | --- | --- | --- | --- | --- |
| navigation_aid | 港湾灯台 | Point | symbol | navigation_light | `chart_symbol_code=lighthouse` | 900 | `灯色` `灯略記` `表示用番号` |
| navigation_aid | 灯柱 | Point | symbol | navigation_light | `chart_symbol_code=light_beacon` | 890 | `灯色` `灯略記` `形状分類番号` |
| navigation_aid | 灯浮標 | Point | symbol | navigation_light | `chart_symbol_code=light_buoy` | 880 | `灯色` `灯略記` `形状分類番号` |
| navigation_aid | AIS航标 | Point | symbol | navigation_light | `chart_symbol_code=vais` | 870 | `表示用番号` |
| navigation_aid | 浮標 | Point | symbol | navigation_mark | `chart_symbol_code=buoy` | 860 | `形状分類番号` `表示用番号` |
| navigation_aid | 立標 | Point | symbol | navigation_mark | `chart_symbol_code=beacon` | 860 | `形状分類番号` `表示用番号` |
| hazard | 暗岩 | Point | symbol | hazard | `chart_symbol_code=rock_awash` | 910 | `分類番号` |
| hazard | 岩礁 | Point | symbol | hazard | `chart_symbol_code=reef` | 900 | `分類番号` |
| hazard | 礁盘 | Polygon | fill | hazard | `chart_fill_style=reef_area` | 860 | `分類番号` |
| hazard | 魚礁 | Polygon | fill | hazard | `chart_fill_style=reef_area` | 850 | `分類番号` `表示用番号` |
| hazard | 沈船 | Point | symbol | hazard | `chart_symbol_code=wreck` | 905 | `分類番号` |
| hazard | 障害物 | Point | symbol | hazard | `chart_symbol_code=obstruction` | 895 | `分類番号` |
| hazard | 潜堤 | Line/Polygon | line/fill | hazard | `chart_line_style=submerged_reef` / `chart_fill_style=hazard_area` | 845 | `分類番号` |
| depth | 水深点 | Point | label | depth_mark | `chart_text_style=depth_text` | 700 | `水深値(m)` |
| depth | 水深区 | Polygon | fill | depth_zone | `chart_fill_style=depth_zone_*` | 650 | `分類番号` |
| depth | 等深线 | LineString | line | contour | `chart_line_style=contour_major_or_minor` | 640 | `表示用番号` |
| depth | 概略等深线 | LineString | line | contour | `chart_line_style=contour_overview` | 620 | `表示用番号` |
| boundary | 危険界 | LineString | line | boundary | `chart_line_style=hazard_boundary` | 800 | `分類番号` |
| boundary | 基本线 | LineString | line | boundary | `chart_line_style=baseline` | 780 | `分類番号` |
| boundary | 航路境界 | LineString | line | boundary | `chart_line_style=route_boundary` | 770 | `分類番号` |
| usage_area | 航路 | Polygon/LineString | fill/line | usage_area | `chart_fill_style=route_area` / `chart_line_style=route_axis` | 760 | `分類番号` |
| usage_area | 錨地 | Polygon | fill | usage_area | `chart_fill_style=anchorage_area` | 750 | `分類番号` |
| usage_area | 錨泊地 | Polygon | fill | usage_area | `chart_fill_style=anchorage_area` | 750 | `分類番号` |
| usage_area | 漁業区 | Polygon | fill | usage_area | `chart_fill_style=fishery_area` | 740 | `分類番号` |
| usage_area | パイロットステーション | Point | symbol | usage_area | `chart_symbol_code=pilot_station` | 735 | `表示用番号` |
| facility | 防波堤 | LineString/Polygon | line/fill | facility | `chart_line_style=breakwater` | 730 | `分類番号` |
| facility | 橋梁 | LineString | line | facility | `chart_line_style=bridge` | 725 | `高さ(m)` |
| facility | 海底电缆 | LineString | line | facility | `chart_line_style=subsea_cable` | 720 | `分類番号` |
| facility | 高さ制限 | Point/LineString | label/line | facility | `chart_text_style=clearance_label` | 715 | `高さ(m)` `高さ/深度(m)` |
| seabed | 底質 | Point | label | seabed | `chart_text_style=seabed_text` | 500 | `名称` |
| place | 海上地名 | Point | label | place | `chart_text_style=place_name_sea` | 450 | `日本語地名` `英文字地名` |
| place | 陆上地名 | Point | label | place | `chart_text_style=place_name_land` | 440 | `日本語地名` `英文字地名` |
## 4 细化规则
### 4.1 灯标对象
主映射依据：
- `canonical_object_type`
细化字段：
- `灯色 -> light_color_code`
- `灯略記 -> light_character_code` 与 `chart_label_subtext`
- `名称 -> chart_label_text`
- `表示用番号` / `形状分類番号` 决定具体 icon 变体
说明：
- 灯弧不必作为核心渲染对象单独保留
- 但 `明弧/分孤` 可以继续保留，用于特殊产品或工程校验
### 4.2 危险物
主映射依据：
- `canonical_object_type`
细化输出：
- `hazard_class`
- `hazard_severity`
建议：
- `暗岩`、`沈船` 默认 `hazard_severity >= major`
- `魚礁` 按区域危险对象处理，但允许叠加 `area_usage_class=fishery`
### 4.3 水深和等深线
建议：
- 水深点统一转成 `chart_render_type=label`
- 水深区统一转成 `chart_render_type=fill`
- 等深线和概略等深线统一转成 `chart_render_type=line`
细化：
- `表示用番号` 或原 line layer 类型决定 `contour_major` / `contour_minor` / `contour_overview`
### 4.4 边界和区域
建议：
- “区域本体”与“边界线”分别映射
- 航路、锚地、鱼业区优先落到 `usage_area`
- 基本线、危险界、航路境界优先落到 `boundary`
## 5 label 映射规则
| 对象类型 | chart_label_text | chart_label_subtext | chart_text_style |
| --- | --- | --- | --- |
| 港湾灯台 / 灯柱 / 灯浮標 | `名称` | `灯略記` | `light_label` |
| 水深点 | 格式化 `depth_value_m` | 空 | `depth_text` |
| 海上地名 | `日本語地名` 优先，其次 `英文字地名` | 空 | `place_name_sea` |
| 陆上地名 | `日本語地名` 优先，其次 `英文字地名` | 空 | `place_name_land` |
| 高さ制限 | 规范化高度文本 | 空 | `clearance_label` |
## 6 v2 映射必须输出的附加字段
除了图形类型本身，映射表还应尽量输出：
- `chart_priority`
- `chart_visibility_min`
- `chart_visibility_max`
- `chart_collision_group`
原因：
- 这些字段能把样式层里的很多散乱 zoom/filter 规则收回到数据侧
## 7 v2 当前缺口
相较 v1，本表仍需继续扩充以下对象：
- 导灯
- 灯船
- 信号所
- 制限区域
- 未测深区
- 干出区
- 海底管线
- 岸壁 / 栈桥 / 码头
- 海底线
- 水文现象对象
这些建议作为 v2.1 的扩展项。

56
tasks/pbf/NavSea_taxonomy_v1.md Normal file
View File

@@ -0,0 +1,56 @@
# NavSea Taxonomy v1（对象语义分类体系）
版本：v1 Draft
用途：定义 NavSea 的对象语义分类（Object Taxonomy）。
taxonomy 只回答一个问题：
> 这个对象是什么
而不涉及如何绘制。
## 结构
canonical_family → canonical_object_type
## canonical_family
- navigation_aid 航标
- buoy 浮标
- hazard 危险物
- depth 水深
- boundary 边界
- facility 港口设施
- usage_area 使用区域
- landmark 地标
- place 地名
- seabed 海床
- hydro_feature 水文现象
## 示例
navigation_aid
- 灯台
- 港湾灯台
- 灯柱
- 灯浮標
- AIS航标
hazard
- 暗岩
- 岩礁
- 礁盘
- 沈船
- 障害物
- 危険物
- 浅滩
depth
- 水深点
- 水深区
- 等深线
taxonomy 与渲染无关。

306
tasks/pbf/NavSea_taxonomy_v2.md Normal file
View File

@@ -0,0 +1,306 @@
# NavSea Taxonomy v2
版本：v2 Draft
用途：定义 NavSea 的对象语义分类体系（Object Taxonomy），为渲染、检索、碰撞检测和统计分析提供稳定的对象身份。
taxonomy 只回答一个问题：
> 这个对象是什么
taxonomy 不直接回答怎么画，也不直接替代旧海图字段。
## 1 设计原则
### 1.1 taxonomy 与 render 分层
- `canonical_family` / `canonical_object_type` 负责语义归类
- `chart_*` 字段负责渲染表达
- 旧字段如 `分類番号`、`形状分類番号`、`表示用番号` 仍可在过渡期保留，用于兼容旧样式细节
### 1.2 family 必须互斥
`canonical_family` 必须尽量稳定、互斥、可统计。
例如：
- `buoy` 不再单独作为 family
- 灯浮标、灯柱、灯台、AIS 航标统一归到 `navigation_aid`
- `place` 只表示地名，不包含设施或标志物
### 1.3 object_type 可以细分
`canonical_object_type` 允许保留更细粒度的对象名，用于：
- 检索
- 规则判断
- 语义统计
- 到渲染语义的映射
## 2 taxonomy 层级
结构：
`canonical_family -> canonical_object_type`
## 3 canonical_family 定义
### 3.1 navigation_aid
定义：用于导航指示、警示、定位或电子助航的对象。
典型 object_type：
- 灯台
- 港湾灯台
- 灯柱
- 灯浮標
- 導灯
- 灯船
- 浮標
- 立標
- AIS航标
- 信号所
说明：
- 原 v1 中的 `buoy` 并入本 family
- “浮标是不是灯浮标”属于 object_type 层的问题，不在 family 层拆开
### 3.2 hazard
定义：对航行、锚泊或作业形成风险的对象、区域或状态。
典型 object_type：
- 暗岩
- 岩礁
- 礁盘
- 魚礁
- 沈船
- 障害物
- 危険物
- 浅滩
- 潜堤
- 洞穴/穴
说明：
- 鱼礁虽然可能带有利用属性，但从航行风险角度优先归到 `hazard`
- 若后续需要区分“危险物”和“人工利用构造”，可通过 `hazard_class` 或 `area_usage_class` 细分，而不是拆 family
### 3.3 depth
定义：描述水深、水深区、等深线和相关深度表达的对象。
典型 object_type：
- 水深点
- 水深区
- 等深线
- 概略等深线
- 干出区
- 未测深区
### 3.4 boundary
定义：描述法律、管理、航路或危险范围边界的对象。
典型 object_type：
- 基本线
- 危険界
- 航路境界
- 港則法による境界
- 制限区域境界
- 管辖边界
### 3.5 facility
定义：人工建造的固定设施、结构物和工程对象。
典型 object_type：
- 防波堤
- 岸壁
- 栈桥
- 桥梁
- 海底电缆
- 海底管线
- 陆上构造物
- 水上设施
说明：
- “设施”与“边界”分离
- “设施”与“地标”分离
### 3.6 usage_area
定义：表达水域或区域用途、限制或功能属性的对象。
典型 object_type：
- 錨地
- 錨泊地
- 漁業区
- 航路
- パイロットステーション
- 禁止区域
- 制限区域
说明：
- 如果对象主要表达“用途”而不是“实体构造”，优先归到本 family
### 3.7 landmark
定义：用于识别、定位或视觉参照的显著对象，但不属于典型助航设施。
典型 object_type：
- 山顶
- 烟囱
- 塔
- 目标物
- 显著建筑
### 3.8 place
定义：地名和位置名称。
典型 object_type：
- 海上地名
- 陆上地名
- 岬角地名
- 港名
说明：
- 只表示名称，不表示实体类别
- 与 `facility` 分离，避免“有名字的设施”误归到 place
### 3.9 seabed
定义：描述海床性质、底质和海底表面特征的对象。
典型 object_type：
- 底質
- 海底地形
- 海底线
### 3.10 hydro_feature
定义：描述流、潮、涌浪、水流方向等水文现象的对象。
典型 object_type：
- 潮流
- 流向
- 渦流
- 浪向
## 4 taxonomy 命名规则
### 4.1 family 命名
- 使用稳定英文 snake_case
- 面向系统处理
- 不与具体样式实现耦合
### 4.2 object_type 命名
- 允许使用与业务更贴近的中文或日文对象名
- 但同义词必须收敛到单一标准写法
建议：
- 输出层面统一使用当前已采用的 `canonical_object_type`
- 数据治理时维护“原始名 -> 标准对象名”的映射表
## 5 v2 首批强制覆盖对象
以下对象建议作为 v2 taxonomy 的首批强制覆盖范围，因为它们同时影响旧样式兼容和后续语义渲染：
- 港湾灯台
- 灯柱
- 灯浮標
- AIS航标
- 暗岩
- 岩礁
- 礁盘
- 魚礁
- 沈船
- 障害物
- 潜堤
- 水深点
- 水深区
- 等深线
- 概略等深线
- 危険界
- 基本线
- 航路
- 航路境界
- 錨地
- 錨泊地
- 漁業区
- 防波堤
- 橋梁
- 海底电缆
- 底質
- 海上地名
- 陆上地名
- 高さ制限
## 6 taxonomy 与旧字段的关系
taxonomy 不替代旧字段，但要尽量减少下游直接理解旧字段的负担。
建议职责划分如下：
- `canonical_family`：供统计、筛选、检索、碰撞分类使用
- `canonical_object_type`：供精细语义判断和渲染映射使用
- 旧字段：仅在兼容旧样式细节时保留
## 7 v2 边界决策
### 7.1 不再单独使用 buoy family
原因：
- 会与 `navigation_aid` 重叠
- 对检索和渲染映射帮助有限
### 7.2 鱼礁优先归到 hazard
原因：
- 当前产品目标明确包含碰撞检测
- 航行风险语义优先级高于用途语义
补充表达：
- 若需要表达作业用途，可通过渲染语义或 `area_usage_class=fishery` 补充
### 7.3 航路归到 usage_area，航路边界归到 boundary
原因：
- 区分“区域本身”和“边界线”
- 有利于样式层分别控制 fill 和 line
## 8 v2 输出要求
每个交付 feature 至少应满足：
- 有 `canonical_family`
- 有 `canonical_object_type`
- 有 `detection_key`
如果对象未完成 taxonomy 映射，不能直接输出为空，应至少回退到：
- `canonical_family = unknown`
- `canonical_object_type = 未分类对象`
并进入后续人工补录清单。

21
tasks/pbf/NavSea_tile_generation_pipeline_v1.md Normal file
View File

@@ -0,0 +1,21 @@
# NavSea Tile Generation Pipeline v1
描述 NavSea 从原始数据到 Vector Tile 的流程。
raw hydro data
↓
taxonomy classification
↓
canonical_object_type
↓
render mapping table
↓
chart semantic fields
↓
vector tile PBF
↓
MapLibre style rendering
该流程实现：
数据 → 语义 → 渲染 → 屏幕。

249
tasks/pbf/NavSea_tile_generation_pipeline_v2.md Normal file
View File

@@ -0,0 +1,249 @@
# NavSea Tile Generation Pipeline v2
版本：v2 Draft
用途：定义 NavSea 从原始海图数据到交付 Vector Tile 的生产流程和质量要求。
## 1 目标
v2 pipeline 不是简单“改字段名”，而是把交付链路明确拆成：
`原始数据 -> 语义分类 -> 渲染语义 -> 字段物化 -> PBF 交付 -> 样式渲染`
同时满足：
- 旧内容兼容
- 新样式迁移
- 检索和碰撞可用
- 工程可验证
## 2 输入
### 2.1 原始输入
- 原始 Vector Tile：`/home/wwwroot/newpec/exported_auto/tile.mapple-on.jp__newpec-mvt-20260106__z___x___y_.pbf/tiles`
- 原始要素属性：包括 `fid`、`分類番号`、`形状分類番号`、`表示用番号`、`名称`、`灯色` 等
### 2.2 数据库输入
当前数据库中已具备或部分具备：
- taxonomy 归类结果
- render_layer 归并结果
- detection 分类结果
现有 builder 查询表：
- `pbf_relayer_candidates`
- `properties`
## 3 输出类型
建议把输出分成两类：
### 3.1 delivery tiles
用途：
- 正式渲染
- 检索
- 碰撞检测
要求：
- 保留关键旧字段
- 输出稳定 taxonomy 字段
- 输出新的 render_semantics 字段
- 不包含过量调试字段
### 3.2 engineering tiles
用途：
- 抽样校验
- 问题定位
- 新旧比对
要求：
- 可保留更完整的原始字段
- 可保留更多校验字段
## 4 v2 标准流程
### 步骤 1：读取原始瓦片
按 `z/x/y` 读取原始 `.pbf`，解码所有 source layer。
要求：
- 保留每个 source layer 的原始 extent
- 不改变 feature geometry
### 步骤 2：按 fid/source_layer/geom_type 关联数据库记录
建议匹配键：
- `fid`
- `source_layer`
- `geom_type`
目的：
- 将原始 feature 与数据库里的 taxonomy / detection / render_layer 结果稳定关联
### 步骤 3：taxonomy 物化
输出：
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `semantic_key`（可选）
要求：
- 未命中 taxonomy 的对象要进入 fallback 分类
- 不允许静默输出空语义
### 步骤 4：render mapping 物化
依据：
- `canonical_object_type`
- 几何类型
- 必要旧字段
输出：
- `chart_render_type`
- `chart_symbol_family`
- `chart_symbol_code`
- `chart_line_style`
- `chart_fill_style`
- `chart_text_style`
- `chart_priority`
- `chart_visibility_min`
- `chart_visibility_max`
### 步骤 5：label / light / numeric 归一化
输出：
- `chart_label_text`
- `chart_label_subtext`
- `light_color_code`
- `light_character_code`
- `depth_value_m`
- `clearance_height_m`
- `least_depth_m`
- `bearing_deg`
要求：
- 同类对象统一字段命名
- 尽量避免样式和检索继续直接解析原始文本字段
### 步骤 6：决定输出 layer
建议优先级：
1. 使用数据库已确定的 `render_layer`
2. 若无 `render_layer`，按渲染语义计算默认层
3. 再不行才回退原 `source_layer`
说明：
- v2 的长期方向是减少直接依赖旧 source layer 命名
- 但过渡期允许回退，保证旧 style 不崩
### 步骤 7：编码输出 PBF
要求：
- 保持原 feature geometry 不变
- 保持原 layer extent 不变
- 输出写入目标目录 `/home/wwwroot/pbf/{z}/{x}/{y}.pbf`
## 5 质量门禁
v2 pipeline 必须增加明确的 QA 检查。
### 5.1 文件层检查
- 新旧目标区域 `z/x/y.pbf` 文件名集合一致
- 瓦片数一致
- 输出层级目录完整
### 5.2 几何层检查
- 抽样瓦片 extent 一致
- 抽样 feature 数量不异常缩水
- 几何类型不发生意外变化
### 5.3 属性层检查
- `fid` 保留
- taxonomy 字段不为空
- 核心 render 字段存在
- 关键旧字段未被错误清洗
### 5.4 渲染层检查
至少验证以下对象在新样式或兼容样式中可正常显示：
- 灯标体系
- 危险物体系
- 鱼礁/鱼业区
- 水深区和等深线
- 基本线 / 危险界 / 航路境界
- 海上地名 / 陆上地名
## 6 与当前实现的差距
当前 `navsea_tile_builder.py` 已经完成：
- 原始瓦片读取
- 数据库关联
- taxonomy 字段叠加
- `render_layer` 重分层
- 原始 extent 保持
当前还缺：
- 统一的 render mapping 物化
- label/light/numeric 归一化
- delivery 与 engineering 两种输出口径
- 更系统的 QA 报告
## 7 v2 推荐实施顺序
### 阶段 1：不破坏现有交付
- 保持当前 builder 可用
- 在 builder 中新增 `chart_*` 字段，不删旧字段
### 阶段 2：构建混合样式
- 新样式优先读取 `chart_*`
- 旧字段仅用于灯标细节、危险物图案和少数特例
### 阶段 3：建立 QA 报表
- 每次构建输出 coverage / feature count / extent 抽样报告
- 对 taxonomy 未命中项输出清单
### 阶段 4：精简交付字段
- 在新样式和检索稳定后，逐步减少无用旧字段
- 但保留 `fid`、taxonomy 和关键兼容字段
## 8 v2 验收标准
一版合格的 v2 delivery tiles 至少应满足：
- 与原始目标区域瓦片文件集合一致
- 抽样瓦片 extent 正确
- 主要对象类型 taxonomy 完整
- `chart_*` 核心字段可用于新样式
- 旧兼容样式仍可显示关键对象
- 检索和碰撞可通过 `fid + taxonomy + normalized_numeric` 使用

534
tasks/pbf/NavSea_traceable_mapping_architecture_v1.md Normal file
View File

@@ -0,0 +1,534 @@
# NavSea 可追溯映射体系设计 v1
版本：v1 Draft
用途：定义 NavSea “旧版原始数据 -> 新版语义字段 / 渲染字段”的可追溯映射体系，确保后续可以稳定定位分类错误、渲染错误和规则变更影响。
## 1 目标
这套设计解决的不是“如何分类”本身，而是“分类和渲染映射的过程如何可追溯”。
必须满足：
- 能知道某个 feature 命中了哪条 taxonomy 规则
- 能知道某个 feature 命中了哪条 render 规则
- 能知道规则来自哪个版本
- 能知道规则判断时参考了哪些原始字段
- 能从最终 `pbf` 或 `fid` 回查到完整映射过程
一句话概括：
> 最终结果不是终点，规则命中过程也必须被记录成正式资产。
## 2 为什么必须做可追溯
如果只保留结果字段，例如：
- `canonical_family`
- `canonical_object_type`
- `detection_key`
- `chart_render_type`
那后续虽然能看到“结果是什么”，但看不到：
- 为什么会分成这个结果
- 是哪条规则起了作用
- 如果结果错了，到底该改 taxonomy 还是改 render mapping
- 某次规则调整影响了哪些 feature
而 NavSea 后续一定会反复遇到这些问题：
- 某个对象分错类
- 某类对象在新样式里显示错误
- 某次规则升级导致历史区域表现变化
- 检索命中和渲染表现不一致
这时如果没有 trace，就只能人工猜。
## 3 设计原则
### 3.1 文件是规则源头
规则的 source of truth 必须是版本可控文件，而不是代码里的隐式判断，也不是数据库里唯一的一份规则。
建议：
- taxonomy 规则放 YAML/CSV 文件
- render mapping 规则放 YAML/CSV 文件
- 文档文件负责解释，不作为程序执行源
### 3.2 SQL 是运行时载体
数据库负责：
- 导入规则
- 物化计算结果
- 记录 feature 命中过程
- 支撑查询和审计
但数据库不是唯一规则源头。
### 3.3 结果和过程都要存
至少要同时存两类数据：
- 规则定义
- feature 命中结果
只存最终结果表不够。
### 3.4 规则必须有稳定 ID
每条规则必须有稳定 `rule_id`，不能只靠“第几条规则”或描述文本识别。
### 3.5 规则必须有版本
每次可影响结果的变更都要有版本号或 bundle 版本。
建议使用：
- `bundle_version`
- `rule_revision`
## 4 总体架构
建议分成 4 层。
### 4.1 规则源文件层
建议目录：
- `tasks/pbf/mappings/navsea_taxonomy_rules_v1.yaml`
- `tasks/pbf/mappings/navsea_render_rules_v1.yaml`
- `tasks/pbf/mappings/navsea_rule_bundle_v1.yaml`
职责：
- 人可读
- 可进入 git
- 可 code review
- 可回滚
### 4.2 规则表层
建议导入 MySQL 生成：
- `navsea_rule_bundle`
- `navsea_taxonomy_rules`
- `navsea_render_rules`
职责：
- 运行时查询
- 规则生效控制
- 按版本执行构建
### 4.3 结果表层
建议生成：
- `navsea_feature_taxonomy_result`
- `navsea_feature_render_result`
- `navsea_feature_rule_trace`
- `navsea_mapping_run`
职责：
- 记录某次构建的命中结果
- 记录 feature 与 rule 的关系
- 记录本次运行使用的规则版本
### 4.4 交付层
交付层包括：
- delivery pbf
- engineering pbf
其中：
- delivery pbf 只保留必要结果字段
- engineering pbf 或内部查询表保留 trace 回查能力
## 5 推荐文件结构
建议新增目录：
```text
tasks/pbf/mappings/
navsea_rule_bundle_v1.yaml
navsea_taxonomy_rules_v1.yaml
navsea_render_rules_v1.yaml
```
建议新增导入脚本：
```text
scripts/pbf/
import_navsea_rules.py
run_navsea_mapping.py
export_navsea_trace_report.py
```
## 6 规则文件设计
## 6.1 bundle 文件
bundle 文件用于声明：
- 当前规则集版本
- taxonomy 规则文件版本
- render 规则文件版本
- 适用范围
- 是否为当前默认版本
示例字段：
- `bundle_id`
- `bundle_version`
- `status`
- `effective_date`
- `taxonomy_ruleset`
- `render_ruleset`
- `notes`
## 6.2 taxonomy 规则文件
每条 taxonomy 规则至少包含：
- `rule_id`
- `rule_revision`
- `priority`
- `enabled`
- `match_scope`
- `match_expr`
- `output.canonical_family`
- `output.canonical_object_type`
- `output.detection_key_template`
- `reason`
其中：
- `rule_id` 是稳定主键，例如 `TAX-NAV-001`
- `rule_revision` 是规则修订号，例如 `3`
- `priority` 用于冲突时排序
- `match_expr` 描述匹配条件
- `reason` 解释为什么有这条规则
## 6.3 render 规则文件
每条 render 规则至少包含：
- `rule_id`
- `rule_revision`
- `priority`
- `enabled`
- `match_expr`
- `output.chart_render_type`
- `output.chart_symbol_family`
- `output.chart_symbol_code`
- `output.chart_line_style`
- `output.chart_fill_style`
- `output.chart_text_style`
- `output.chart_priority`
- `reason`
render 规则的输入通常包括：
- `canonical_object_type`
- `canonical_family`
- geometry type
- 必要旧字段
## 7 match_expr 建议
不建议把匹配逻辑写成自由文本。
建议使用结构化表达，例如：
```yaml
match_expr:
all:
- field: source_layer
op: in
value: [p航路標識群, P航行危険障害物]
- field: class_name
op: contains_any
value: [港湾灯台, 灯柱, 灯浮標]
- field: geom_type
op: eq
value: Point
```
支持的运算建议限制在少数几类：
- `eq`
- `neq`
- `in`
- `contains`
- `contains_any`
- `is_null`
- `not_null`
- `regex`
这样规则可审计、可导入、可执行。
## 8 SQL 表设计
## 8.1 `navsea_rule_bundle`
建议字段：
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `bundle_id` | varchar(64) PK | 规则集 ID |
| `bundle_version` | varchar(32) | 规则集版本 |
| `status` | varchar(20) | `draft` / `active` / `retired` |
| `taxonomy_ruleset` | varchar(64) | taxonomy 文件版本 |
| `render_ruleset` | varchar(64) | render 文件版本 |
| `effective_date` | datetime | 生效时间 |
| `created_at` | datetime | 创建时间 |
| `notes` | text | 备注 |
## 8.2 `navsea_taxonomy_rules`
建议字段：
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `rule_id` | varchar(64) PK | 稳定规则 ID |
| `bundle_id` | varchar(64) | 所属 bundle |
| `rule_revision` | int | 修订号 |
| `priority` | int | 优先级，值越小越先匹配 |
| `enabled` | tinyint | 是否启用 |
| `match_scope` | varchar(32) | `feature` / `layer` / `object` |
| `match_expr_json` | json | 结构化匹配表达式 |
| `canonical_family` | varchar(100) | 输出 family |
| `canonical_object_type` | varchar(191) | 输出 object type |
| `detection_key_template` | varchar(191) | 输出 detection 模板 |
| `rule_reason` | text | 规则说明 |
| `created_at` | datetime | 创建时间 |
| `updated_at` | datetime | 更新时间 |
## 8.3 `navsea_render_rules`
建议字段：
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `rule_id` | varchar(64) PK | 稳定规则 ID |
| `bundle_id` | varchar(64) | 所属 bundle |
| `rule_revision` | int | 修订号 |
| `priority` | int | 优先级 |
| `enabled` | tinyint | 是否启用 |
| `match_expr_json` | json | 匹配表达式 |
| `chart_render_type` | varchar(32) | 输出 |
| `chart_symbol_family` | varchar(64) | 输出 |
| `chart_symbol_code` | varchar(64) | 输出 |
| `chart_line_style` | varchar(64) | 输出 |
| `chart_fill_style` | varchar(64) | 输出 |
| `chart_text_style` | varchar(64) | 输出 |
| `chart_priority` | int | 输出 |
| `chart_visibility_min` | int | 输出 |
| `chart_visibility_max` | int | 输出 |
| `rule_reason` | text | 规则说明 |
| `created_at` | datetime | 创建时间 |
| `updated_at` | datetime | 更新时间 |
## 8.4 `navsea_mapping_run`
记录一次完整构建执行。
建议字段：
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `run_id` | bigint PK | 执行 ID |
| `bundle_id` | varchar(64) | 本次使用的规则集 |
| `run_type` | varchar(32) | `full` / `aoi` / `qa` |
| `tile_scope` | text | 范围说明 |
| `started_at` | datetime | 开始时间 |
| `finished_at` | datetime | 结束时间 |
| `status` | varchar(20) | `running` / `success` / `failed` |
| `source_snapshot` | varchar(128) | 源数据快照标识 |
| `notes` | text | 备注 |
## 8.5 `navsea_feature_rule_trace`
这是最关键的 trace 表。
建议字段：
| 字段 | 类型 | 说明 |
| --- | --- | --- |
| `run_id` | bigint | 所属执行 |
| `feature_id` | bigint | 内部 feature 主键 |
| `fid` | varchar(191) | 原始对象标识 |
| `z` | int | tile z |
| `x` | int | tile x |
| `y` | int | tile y |
| `source_layer` | varchar(100) | 原始 layer |
| `geom_type` | varchar(32) | 几何类型 |
| `taxonomy_rule_id` | varchar(64) | 命中的 taxonomy 规则 |
| `taxonomy_rule_revision` | int | taxonomy 修订号 |
| `render_rule_id` | varchar(64) | 命中的 render 规则 |
| `render_rule_revision` | int | render 修订号 |
| `bundle_id` | varchar(64) | 规则集版本 |
| `classification_basis` | varchar(64) | 使用的主要依据 |
| `source_fields_used_json` | json | 实际参与判定的字段和值摘要 |
| `canonical_family` | varchar(100) | 产出结果 |
| `canonical_object_type` | varchar(191) | 产出结果 |
| `detection_key` | varchar(191) | 产出结果 |
| `chart_render_type` | varchar(32) | 产出结果 |
| `chart_symbol_family` | varchar(64) | 产出结果 |
| `chart_symbol_code` | varchar(64) | 产出结果 |
| `chart_line_style` | varchar(64) | 产出结果 |
| `chart_fill_style` | varchar(64) | 产出结果 |
| `chart_text_style` | varchar(64) | 产出结果 |
| `trace_status` | varchar(20) | `matched` / `fallback` / `manual_review` |
| `created_at` | datetime | 记录时间 |
说明：
- `source_fields_used_json` 不要求保存整份原始属性
- 但必须记录本次命中实际使用的关键字段和值摘要
例如：
```json
{
"fid": "123456",
"分類番号": "403",
"表示用番号": "31135504",
"名称": "鷹島灯台",
"geom_type": "Point"
}
```
## 9 交付与追溯分离策略
为了兼顾交付体积和可追溯性，建议分两条线：
### 9.1 delivery pbf
保留：
- 业务必需字段
- taxonomy 结果字段
- render 结果字段
- 必要旧字段
不强制保留：
- 完整 trace 细节
### 9.2 engineering trace
通过以下方式保留完整追溯能力：
- SQL trace 表
- engineering pbf
- 按 `fid` / `feature_id` 回查脚本
建议 engineering pbf 可额外保留：
- `taxonomy_rule_id`
- `render_rule_id`
- `bundle_id`
- `trace_status`
## 10 builder 接入方式
当前 [navsea_tile_builder.py](/root/weather/navsea_tile_builder.py) 是从 `pbf_relayer_candidates` 读取结果并写入 `pbf`。
建议改造成两段式：
### 阶段 A：规则求值阶段
输入：
- 原始 feature
- 规则表
输出：
- `navsea_feature_taxonomy_result`
- `navsea_feature_render_result`
- `navsea_feature_rule_trace`
### 阶段 B：PBF 输出阶段
输入：
- 已物化结果表
输出：
- delivery pbf
- engineering pbf
这样做的好处是：
- “规则判断”与“PBF 编码”解耦
- 同一份判定结果可重复出图
- QA 和回归检查更容易做
## 11 问题排查路径
未来排查一个对象时，建议统一走这条链路：
1. 先从前端或瓦片里拿到 `fid`
2. 用 `fid` 或 `feature_id` 查 `navsea_feature_rule_trace`
3. 看命中的 `taxonomy_rule_id` 和 `render_rule_id`
4. 回到规则表查看该规则定义
5. 必要时回到规则源文件查看变更历史
6. 确认问题属于：
- 原始数据异常
- taxonomy 规则错误
- render 规则错误
- 样式解释错误
## 12 变更管理
建议所有规则变更都遵循：
- 改规则文件
- 提交 git
- 生成新 `bundle_version`
- 导入 SQL
- 执行局部或全量重算
- 输出差异报告
不建议：
- 直接手改结果表
- 直接在 SQL 里无版本地改规则
- 直接在代码里插入一条临时 if 判断后不回写规则文件
## 13 最小可落地版本
如果要尽快开始，建议第一阶段先做到：
1. 新增规则源文件目录
2. 给 taxonomy 和 render 规则都定义 `rule_id`
3. 新增 `navsea_rule_bundle`
4. 新增 `navsea_feature_rule_trace`
5. builder 在出 engineering 数据时至少写出：
- `bundle_id`
- `taxonomy_rule_id`
- `render_rule_id`
- `trace_status`
这样即使规则体系还不完整，也已经具备“能追”的能力。
## 14 结论
NavSea 的“旧版 -> 新版”对应关系，不应只是一套结果字段，也不应只存在于代码里。
它应该被正式建设成一套可追溯资产，包括：
- 可版本化的规则文件
- 可运行的 SQL 规则表
- 可查询的 feature trace 表
- 与 delivery / engineering pbf 配套的回查能力
后续查错、验收、回归、争议复盘，都会依赖这套体系。

344
tasks/pbf/OutPbf.md Normal file
View File

@@ -0,0 +1,344 @@
# NavSea Codex Task
# Karatsu AOI Vector Tile Build
Architecture: NavSea V11 / Codex6
Scope: Server Side Tile Build
---
# Task Name
NavSea_Karatsu_AOI_PBF_Builder
---
# Task Goal
Generate a NavSea Vector Tile dataset for the area surrounding **Karatsu City (唐津市)**.
The tile build must cover approximately:
```
10 nautical mile radius
centered on Karatsu city
```
The tiles must be generated for **all zoom levels supported by NavSea**.
The generated tiles will be used to validate:
- semantic overlay
- render compatibility
- navigation object detection
- tile density behavior
- style compatibility
---
# Output Location
Tiles must be written to:
```
/home/wwwroot/pbf
```
Directory structure must follow standard vector tile layout:
```
/home/wwwroot/pbf/{z}/{x}/{y}.pbf
```
Example:
```
/home/wwwroot/pbf/10/864/440.pbf
```
Existing files in the directory may be overwritten.
---
# AOI Definition
Center location:
```
Karatsu City
Japan
```
Recommended reference coordinate:
```
Lat: 33.4425
Lon: 129.9697
```
Coverage radius:
```
10 nautical miles
≈ 18.52 km
```
Codex may implement this AOI using one of the following:
```
1 center point + radius
2 bounding box derived from radius
3 tile range approximation
```
The resulting tiles must fully cover the requested radius.
---
# Zoom Levels
Tiles must be generated for:
```
z = 0 → 14
```
(Or the maximum zoom available in the dataset.)
---
# Data Sources
Tile features must be built from the validated NavSea semantic dataset.
Primary semantic source:
```
pbf_relayer_candidates
```
Additional required tables:
```
features
properties
at_attributes
feature_semantic
```
These tables must be joined to reconstruct the full feature.
---
# Feature Reconstruction Rules
Each output tile feature must include:
## Geometry
From:
```
features
```
Fields:
```
z
x
y
geom_type
```
The geometry encoding must match the original vector tile geometry.
---
## Original Attributes
All original attributes from:
```
properties
```
must be preserved.
Examples include:
```
class
shape
name
symbol
depth
category
```
These attributes must **not be removed or renamed**.
---
## Semantic Attributes
Additional semantic fields must be included from:
```
pbf_relayer_candidates
```
Including:
```
canonical_object_type
canonical_family
semantic_key
detection_key
render_layer
```
These fields must be **added**, not replacing original fields.
---
# Feature Selection
Features must be selected based on tile coordinates intersecting the AOI.
Selection process:
```
1 determine tile range covering AOI
2 select features matching tile coordinates
3 group features by tile
4 encode each tile
```
---
# Tile Encoding
Each tile must be encoded as a **Mapbox Vector Tile (MVT)**.
Encoding rules:
```
Layer name = render_layer
```
If `render_layer` is null, fallback to:
```
source_layer
```
Geometry must be encoded according to:
```
geom_type
```
---
# Performance Requirements
The tile builder must:
```
avoid full-table scans where possible
use tile coordinate filtering
process tiles incrementally
```
The builder must not attempt to load the entire dataset into memory.
---
# Build Script
Codex must implement a script:
```
navsea_tile_builder.py
```
Capabilities required:
```
AOI tile calculation
database feature extraction
tile grouping
vector tile encoding
parallel tile generation
filesystem output
```
---
# Suggested Execution
The build script should be executable as:
```bash
python navsea_tile_builder.py \
--center-lat 33.4425 \
--center-lon 129.9697 \
--radius-nm 10 \
--zmin 0 \
--zmax 14 \
--output /home/wwwroot/pbf
```
---
# Expected Result
After the build completes, the directory should contain:
```
/home/wwwroot/pbf/
├─ 0/
├─ 1/
├─ 2/
├─ ...
└─ 14/
```
Each containing:
```
{x}/{y}.pbf
```
The tiles must render correctly in MapLibre / OpenCPN / tile viewers.
---
# Verification
After build, the following checks should pass:
```
tiles exist for requested AOI
tile hierarchy is correct
vector tiles decode successfully
original attributes preserved
semantic fields present
```
Example feature should resemble:
```json
{
"class": "灯",
"shape": "灯標",
"name": "唐津灯台",
"canonical_object_type": "navigation_light",
"canonical_family": "navigation_aid",
"detection_key": "nav_light"
}
```
---
# End Task

104
tasks/pbf/fiddecode.md Normal file
View File

@@ -0,0 +1,104 @@
from Crypto.Cipher import AES
import hashlib
import struct
# Example implementation updated for the full signed int32 legacy fid domain.
# This keeps the algorithm reversible for all observed NavSea legacy fid values.
MIN_VAL = -(1 << 31)
MAX_VAL = (1 << 31) - 1
N = MAX_VAL - MIN_VAL + 1 # 4294967296
ROUNDS = 10
DOMAIN_BITS = 32
DOMAIN_SIZE = 1 << DOMAIN_BITS
def _prf(key: bytes, round_no: int, value: int) -> int:
"""
AES-based PRF, returns 16-bit integer.
"""
cipher = AES.new(key, AES.MODE_ECB)
data = struct.pack(">IQ", round_no, value)
block = hashlib.sha256(data).digest()[:16]
out = cipher.encrypt(block)
return int.from_bytes(out[-2:], "big")
def _feistel_permute(x: int, key: bytes, encrypt: bool) -> int:
"""
32-bit Feistel permutation.
"""
if not (0 <= x < DOMAIN_SIZE):
raise ValueError("x out of 32-bit domain")
l = (x >> 16) & 0xFFFF
r = x & 0xFFFF
rounds = range(ROUNDS) if encrypt else reversed(range(ROUNDS))
for rnd in rounds:
if encrypt:
f = _prf(key, rnd, r)
l, r = r, l ^ f
else:
f = _prf(key, rnd, l)
l, r = r ^ f, l
return ((l << 16) | r) & 0xFFFFFFFF
def encrypt_number(x: int, key: bytes) -> int:
"""
Encrypt x in [-2147483648, 2147483647]
Return non-negative integer in [0, N-1]
"""
if len(key) not in (16, 24, 32):
raise ValueError("AES key must be 16, 24, or 32 bytes")
if not (MIN_VAL <= x <= MAX_VAL):
raise ValueError("input out of range")
# Map original range to [0, N-1]
r = x - MIN_VAL
# Cycle-walking into [0, N-1]
while True:
r = _feistel_permute(r, key, True)
if r < N:
return r
def decrypt_number(y: int, key: bytes) -> int:
"""
Decrypt non-negative integer y in [0, N-1]
Return original integer in [-2147483648, 2147483647]
"""
if len(key) not in (16, 24, 32):
raise ValueError("AES key must be 16, 24, or 32 bytes")
if not (0 <= y < N):
raise ValueError("encrypted value out of range")
r = y
# Reverse cycle-walking
while True:
r = _feistel_permute(r, key, False)
if r < N:
return r + MIN_VAL
if __name__ == "__main__":
key = b"example-secret!!" # 16 bytes
samples = [
-2147483648,
-1,
0,
1,
12345678,
2147483647,
]
for x in samples:
enc = encrypt_number(x, key)
dec = decrypt_number(enc, key)
print(f"x={x}, enc={enc}, dec={dec}, ok={x == dec}")

123
tasks/pbf/mappings/navsea_field_name_rules_v1.yaml Normal file
View File

@@ -0,0 +1,123 @@
ruleset: navsea_field_name_rules_v1
bundle_id: navsea-core
version: v1-draft
rules:
- field_name_jp: fid
field_name_std: fid_legacy_raw
field_group: identity
keep_in_delivery: false
keep_in_engineering: true
normalization_class: reversible_identity
notes: Preserve the original legacy fid only in engineering and trace outputs.
- field_name_jp: at
field_name_std: at_raw
field_group: source_legacy
keep_in_delivery: false
keep_in_engineering: true
normalization_class: raw_blob
notes: Raw attribute pack remains engineering-only by default.
- field_name_jp: 分類番号
field_name_std: class_code
field_group: source_legacy
keep_in_delivery: true
keep_in_engineering: true
normalization_class: legacy_render_code
notes: Legacy class code used during transition and audits.
- field_name_jp: 形状分類番号
field_name_std: shape_class_code
field_group: source_legacy
keep_in_delivery: true
keep_in_engineering: true
normalization_class: legacy_render_code
notes: Legacy shape class code used during transition and audits.
- field_name_jp: 表示用番号
field_name_std: display_code
field_group: source_legacy
keep_in_delivery: true
keep_in_engineering: true
normalization_class: legacy_render_code
notes: Legacy display code retained until all rendering migrates to chart_* fields.
- field_name_jp: 灯色
field_name_std: light_color_raw_code
field_group: light
keep_in_delivery: true
keep_in_engineering: true
normalization_class: field_value_normalization
notes: Legacy raw color code maps into light_color_code.
- field_name_jp: 灯略記
field_name_std: light_character_remark
field_group: light
keep_in_delivery: true
keep_in_engineering: true
normalization_class: content_text
notes: User-visible light remark should remain reversible.
- field_name_jp: 明弧/分孤
field_name_std: light_sector_remark
field_group: light
keep_in_delivery: true
keep_in_engineering: true
normalization_class: field_value_normalization
notes: Legacy light sector remark maps into light_sector_mode and trace fields.
- field_name_jp: 表示位置
field_name_std: label_position_code_legacy
field_group: labels
keep_in_delivery: true
keep_in_engineering: true
normalization_class: field_value_normalization
notes: Legacy label position code maps into chart_label_position_code.
- field_name_jp: 名称
field_name_std: name_ja
field_group: labels
keep_in_delivery: true
keep_in_engineering: true
normalization_class: content_text
notes: Primary Japanese name content must remain available.
- field_name_jp: 名称補助
field_name_std: name_subtext_ja
field_group: labels
keep_in_delivery: true
keep_in_engineering: true
normalization_class: content_text
notes: Secondary Japanese name content must remain available.
- field_name_jp: 日本語地名
field_name_std: place_name_ja
field_group: labels
keep_in_delivery: true
keep_in_engineering: true
normalization_class: content_text
notes: Japanese place name content must remain available.
- field_name_jp: 英文字地名
field_name_std: place_name_en
field_group: labels
keep_in_delivery: true
keep_in_engineering: true
normalization_class: content_text
notes: English place name content remains available for fallback labels.
- field_name_jp: 水深値(m)
field_name_std: depth_value_m_legacy
field_group: normalized_numeric
keep_in_delivery: true
keep_in_engineering: true
normalization_class: numeric_normalization
notes: Legacy depth string is normalized into depth_value_m.
- field_name_jp: 高さ(m)
field_name_std: clearance_height_m_legacy
field_group: normalized_numeric
keep_in_delivery: true
keep_in_engineering: true
normalization_class: numeric_normalization
notes: Legacy height string is normalized into clearance_height_m.
- field_name_jp: 高さ/深度(m)
field_name_std: height_or_depth_m_legacy
field_group: normalized_numeric
keep_in_delivery: true
keep_in_engineering: true
normalization_class: numeric_normalization
notes: Mixed height/depth legacy string maps into least_depth_m or clearance semantics.
- field_name_jp: 角度
field_name_std: bearing_deg_legacy
field_group: normalized_numeric
keep_in_delivery: true
keep_in_engineering: true
normalization_class: numeric_normalization
notes: Legacy angle string is normalized into bearing_deg.

148
tasks/pbf/mappings/navsea_field_value_rules_v1.yaml Normal file
View File

@@ -0,0 +1,148 @@
ruleset: navsea_field_value_rules_v1
bundle_id: navsea-core
version: v1-draft
rules:
- rule_id: FVAL-LIGHT-001
rule_revision: 1
field_name_jp: 灯色
legacy_value: "1"
standardized_field: light_color_code
standardized_value: green
notes: Legacy light color code 1 maps to green in current data.
- rule_id: FVAL-LIGHT-002
rule_revision: 1
field_name_jp: 灯色
legacy_value: "2"
standardized_field: light_color_code
standardized_value: red
notes: Legacy light color code 2 maps to red.
- rule_id: FVAL-LIGHT-003
rule_revision: 1
field_name_jp: 灯色
legacy_value: "3"
standardized_field: light_color_code
standardized_value: yellow
notes: Legacy light color code 3 maps to yellow.
- rule_id: FVAL-LIGHT-004
rule_revision: 1
field_name_jp: 灯色
legacy_value: "4"
standardized_field: light_color_code
standardized_value: white
notes: Legacy light color code 4 maps to white.
- rule_id: FVAL-LIGHT-005
rule_revision: 1
field_name_jp: 灯色
legacy_value: "5"
standardized_field: light_color_code
standardized_value: blue
notes: Legacy light color code 5 maps to blue.
- rule_id: FVAL-LIGHT-006
rule_revision: 1
field_name_jp: 灯色
legacy_value: "6"
standardized_field: light_color_code
standardized_value: violet
notes: Legacy light color code 6 maps to violet.
- rule_id: FVAL-LIGHT-007
rule_revision: 1
field_name_jp: 灯色
legacy_value: "7"
standardized_field: light_color_code
standardized_value: orange
notes: Legacy light color code 7 maps to orange.
- rule_id: FVAL-LIGHT-008
rule_revision: 1
field_name_jp: 灯色
legacy_value: "8"
standardized_field: light_color_code
standardized_value: amber
notes: Legacy light color code 8 maps to amber.
- rule_id: FVAL-LIGHT-009
rule_revision: 1
field_name_jp: 灯色
legacy_value: "9"
standardized_field: light_color_code
standardized_value: mixed
notes: Legacy light color code 9 maps to mixed.
- rule_id: FVAL-LABELPOS-001
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "0"
standardized_field: chart_label_position_code
standardized_value: tl
notes: Legacy label position 0 maps to top-left.
- rule_id: FVAL-LABELPOS-002
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "1"
standardized_field: chart_label_position_code
standardized_value: t
notes: Legacy label position 1 maps to top.
- rule_id: FVAL-LABELPOS-003
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "2"
standardized_field: chart_label_position_code
standardized_value: tr
notes: Legacy label position 2 maps to top-right.
- rule_id: FVAL-LABELPOS-004
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "4"
standardized_field: chart_label_position_code
standardized_value: l
notes: Legacy label position 4 maps to left.
- rule_id: FVAL-LABELPOS-005
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "5"
standardized_field: chart_label_position_code
standardized_value: c
notes: Legacy label position 5 maps to center.
- rule_id: FVAL-LABELPOS-006
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "6"
standardized_field: chart_label_position_code
standardized_value: r
notes: Legacy label position 6 maps to right.
- rule_id: FVAL-LABELPOS-007
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "8"
standardized_field: chart_label_position_code
standardized_value: bl
notes: Legacy label position 8 maps to bottom-left.
- rule_id: FVAL-LABELPOS-008
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "9"
standardized_field: chart_label_position_code
standardized_value: b
notes: Legacy label position 9 maps to bottom.
- rule_id: FVAL-LABELPOS-009
rule_revision: 1
field_name_jp: 表示位置
legacy_value: "10"
standardized_field: chart_label_position_code
standardized_value: br
notes: Legacy label position 10 maps to bottom-right.
- rule_id: FVAL-CLASS-001
rule_revision: 1
field_name_jp: 分類番号
legacy_value: "428"
standardized_field: chart_symbol_code
standardized_value: fish_reef
geom_scope: Point
canonical_object_scope: 魚礁
notes: Legacy class 428 point object renders as fish reef symbol.
- rule_id: FVAL-CLASS-002
rule_revision: 1
field_name_jp: 分類番号
legacy_value: "428"
standardized_field: chart_fill_style
standardized_value: fish_reef_area
geom_scope: Polygon,MultiPolygon
canonical_object_scope: 魚礁
notes: Legacy class 428 area object renders as fish reef area pattern.

171
tasks/pbf/mappings/navsea_render_rules_v1.yaml Normal file
View File

@@ -0,0 +1,171 @@
ruleset: navsea_render_rules_v1
bundle_id: navsea-core
version: v1-draft
rules:
- rule_id: REN-NAV-001
rule_revision: 1
priority: 100
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 港湾灯台
- field: geom_type
op: eq
value: Point
output:
chart_render_type: symbol
chart_symbol_family: navigation_light
chart_symbol_code: lighthouse
chart_priority: 900
chart_visibility_min: 7
chart_visibility_max: 24
reason: 港湾灯台应以高优先级灯标符号渲染。
- rule_id: REN-HAZ-001
rule_revision: 1
priority: 120
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 魚礁
- field: geom_type
op: eq
value: Point
output:
chart_render_type: symbol
chart_symbol_family: hazard
chart_symbol_code: fish_reef
chart_priority: 860
area_usage_class: fishery
hazard_class: fish_reef
hazard_severity: major
reason: 点状鱼礁以鱼礁专用危险物符号表达。
- rule_id: REN-HAZ-002
rule_revision: 1
priority: 121
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 魚礁
- field: geom_type
op: in
value: [Polygon, MultiPolygon]
output:
chart_render_type: fill
chart_symbol_family: hazard
chart_fill_style: fish_reef_area
chart_priority: 850
area_usage_class: fishery
hazard_class: fish_reef
hazard_severity: major
reason: 面状鱼礁区以鱼礁专用危险区填充表达，同时保留 fishery 用途语义。
- rule_id: REN-DEPTH-001
rule_revision: 1
priority: 200
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 等深线
output:
chart_render_type: line
chart_symbol_family: contour
chart_line_style: contour_minor
chart_priority: 640
chart_visibility_min: 8
chart_visibility_max: 24
reason: 等深线统一归入 contour line 体系。
- rule_id: REN-SEABED-001
rule_revision: 1
priority: 210
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 底質
output:
chart_render_type: label
chart_symbol_family: seabed
chart_text_style: seabed_text
chart_priority: 500
chart_visibility_min: 11
chart_visibility_max: 24
reason: 底质对象以海底属性文字渲染。
- rule_id: REN-PLACE-001
rule_revision: 1
priority: 220
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 海上地名
output:
chart_render_type: label
chart_symbol_family: place
chart_text_style: place_name_sea
chart_priority: 450
chart_visibility_min: 7
chart_visibility_max: 24
reason: 海上地名使用海上地名字体体系渲染。
- rule_id: REN-PLACE-002
rule_revision: 1
priority: 221
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 陆上地名
output:
chart_render_type: label
chart_symbol_family: place
chart_text_style: place_name_land
chart_priority: 440
chart_visibility_min: 7
chart_visibility_max: 24
reason: 陆上地名使用陆地地名字体体系渲染。
- rule_id: REN-USAGE-001
rule_revision: 1
priority: 230
enabled: true
match_expr:
all:
- field: canonical_object_type
op: eq
value: 錨泊地
- field: geom_type
op: in
value: [Polygon, MultiPolygon]
output:
chart_render_type: fill
chart_symbol_family: usage_area
chart_fill_style: anchorage_area
chart_priority: 750
area_usage_class: anchorage
reason: 锚泊地区域以 anchorage_area 语义填充表达。
- rule_id: REN-FALLBACK-001
rule_revision: 1
priority: 9999
enabled: true
match_expr:
all: []
output:
chart_render_type: none
chart_priority: 0
reason: 未命中渲染规则的对象默认不进入正式语义渲染，待补录。

12
tasks/pbf/mappings/navsea_rule_bundle_v1.yaml Normal file
View File

@@ -0,0 +1,12 @@
bundle_id: navsea-core
bundle_version: v1-draft
status: draft
effective_date: 2026-03-14
taxonomy_ruleset: navsea_taxonomy_rules_v1
render_ruleset: navsea_render_rules_v1
source_layer_ruleset: navsea_source_layer_rules_v1
field_name_ruleset: navsea_field_name_rules_v1
field_value_ruleset: navsea_field_value_rules_v1
notes: >
Draft rule bundle for traceable NavSea taxonomy and render mapping.
File is intended as source-of-truth metadata and should be imported into SQL.

151
tasks/pbf/mappings/navsea_source_layer_rules_v1.yaml Normal file
View File

@@ -0,0 +1,151 @@
ruleset: navsea_source_layer_rules_v1
bundle_id: navsea-core
version: v1-draft
rules:
- source_layer_jp: L700
source_layer_std: depth_zone_700
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L702
source_layer_std: depth_zone_702
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L725
source_layer_std: depth_zone_725
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L739
source_layer_std: depth_zone_739
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L740
source_layer_std: depth_zone_740
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L741
source_layer_std: depth_zone_741
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L748
source_layer_std: depth_zone_748
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L749
source_layer_std: depth_zone_749
notes: Legacy coded depth zone layer kept as stable standardized code.
- source_layer_jp: L危険界
source_layer_std: hazard_boundary_line
notes: Standardized line identity for hazard boundary linework.
- source_layer_jp: L基本線
source_layer_std: baseline_line
notes: Standardized line identity for baseline linework.
- source_layer_jp: L概略等深線
source_layer_std: depth_contour_overview
notes: Overview contour line layer.
- source_layer_jp: L海底地形
source_layer_std: bathymetry_line
notes: Bathymetry support line layer.
- source_layer_jp: L海底線
source_layer_std: seabed_line
notes: Seabed line feature container.
- source_layer_jp: L等深線
source_layer_std: depth_contour
notes: Standard contour line layer.
- source_layer_jp: L航路
source_layer_std: route_axis_line
notes: Route axis or route line layer.
- source_layer_jp: L陸上構造物陸
source_layer_std: onshore_structure_line
notes: Onshore structure line layer.
- source_layer_jp: L高さ制限
source_layer_std: clearance_limit_line
notes: Clearance restriction line layer.
- source_layer_jp: P721ククリ
source_layer_std: clip_outline_721
notes: Stable outline layer preserved by numeric legacy code.
- source_layer_jp: P730ククリ
source_layer_std: clip_outline_730
notes: Stable outline layer preserved by numeric legacy code.
- source_layer_jp: P754ククリ
source_layer_std: clip_outline_754
notes: Stable outline layer preserved by numeric legacy code.
- source_layer_jp: pパイロットステーション
source_layer_std: pilot_station_point
notes: Pilot station point layer.
- source_layer_jp: P危険界ククリ
source_layer_std: hazard_boundary_outline
notes: Hazard boundary outline layer.
- source_layer_jp: p地名
source_layer_std: place_label_sea
notes: Marine place label point layer.
- source_layer_jp: p地名陸
source_layer_std: place_label_land
notes: Land place label point layer.
- source_layer_jp: P基本線
source_layer_std: baseline_area
notes: Baseline area or container layer.
- source_layer_jp: P基本線ククリ
source_layer_std: baseline_outline
notes: Baseline outline layer.
- source_layer_jp: p底質
source_layer_std: seabed_text_point
notes: Seabed label point layer.
- source_layer_jp: p投錨注意障害物
source_layer_std: anchor_caution_hazard_point
notes: Point hazards relevant to anchoring caution.
- source_layer_jp: P投錨注意障害物ククリ
source_layer_std: anchor_caution_hazard_outline
notes: Outline of anchoring caution hazard area.
- source_layer_jp: P施設・境界線等
source_layer_std: facility_boundary_area
notes: Facility and boundary area container.
- source_layer_jp: P施設・境界線等ククリ
source_layer_std: facility_boundary_outline
notes: Facility and boundary outline layer.
- source_layer_jp: P施設・境界線等透明
source_layer_std: facility_boundary_area_transparent
notes: Transparent facility and boundary area layer.
- source_layer_jp: P橋りょう等構造物
source_layer_std: bridge_structure
notes: Bridge and related structure layer.
- source_layer_jp: P漁具定置箇所
source_layer_std: fixed_fishing_gear_area
notes: Fixed fishing gear area layer.
- source_layer_jp: P潜堤
source_layer_std: submerged_reef_area
notes: Submerged reef area layer.
- source_layer_jp: P穴
source_layer_std: hole_area
notes: Hole or depression area layer.
- source_layer_jp: p航行危険障害物
source_layer_std: navigation_hazard_point
notes: Navigation hazard point layer.
- source_layer_jp: P航行危険障害物ククリ
source_layer_std: navigation_hazard_outline
notes: Navigation hazard outline layer.
- source_layer_jp: P航路
source_layer_std: route_area
notes: Route area layer.
- source_layer_jp: P航路ククリ
source_layer_std: route_outline
notes: Route outline layer.
- source_layer_jp: p航路境界等
source_layer_std: route_boundary_point
notes: Route boundary point annotation layer.
- source_layer_jp: p航路標識群
source_layer_std: navigation_marks
notes: Navigation marks point layer.
- source_layer_jp: P誘導線ククリ
source_layer_std: leading_line_outline
notes: Leading line outline layer.
- source_layer_jp: p錨泊地等
source_layer_std: anchorage_point
notes: Anchorage point symbol layer.
- source_layer_jp: P錨泊地等ククリ
source_layer_std: anchorage_outline
notes: Anchorage outline layer.
- source_layer_jp: p陸上構造物
source_layer_std: onshore_structure_point
notes: Onshore structure point layer.
- source_layer_jp: P陸上構造物陸
source_layer_std: onshore_structure_area
notes: Onshore structure area layer.
- source_layer_jp: P陸域
source_layer_std: land_area
notes: Land area polygon layer.
- source_layer_jp: p高さ制限
source_layer_std: clearance_limit_point
notes: Clearance restriction point layer.

134
tasks/pbf/mappings/navsea_taxonomy_rules_v1.yaml Normal file
View File

@@ -0,0 +1,134 @@
ruleset: navsea_taxonomy_rules_v1
bundle_id: navsea-core
version: v1-draft
rules:
- rule_id: TAX-NAV-001
rule_revision: 1
priority: 100
enabled: true
match_scope: feature
match_expr:
all:
- field: class_name
op: contains_any
value: [港湾灯台, 灯台]
- field: geom_type
op: eq
value: Point
output:
canonical_family: navigation_aid
canonical_object_type: 港湾灯台
detection_key_template: symbol:港湾灯台
reason: 港湾灯台属于典型点状助航对象。
- rule_id: TAX-DEPTH-001
rule_revision: 1
priority: 140
enabled: true
match_scope: layer
match_expr:
all:
- field: source_layer
op: eq
value: L等深線
output:
canonical_family: depth
canonical_object_type: 等深线
detection_key_template: line:等深线
reason: 等深线来源层具有稳定的深度线语义。
- rule_id: TAX-SEABED-001
rule_revision: 1
priority: 150
enabled: true
match_scope: layer
match_expr:
all:
- field: source_layer
op: eq
value: p底質
output:
canonical_family: seabed
canonical_object_type: 底質
detection_key_template: label:底質
reason: 底质标签层具有稳定的海底属性语义。
- rule_id: TAX-PLACE-001
rule_revision: 1
priority: 160
enabled: true
match_scope: layer
match_expr:
all:
- field: source_layer
op: eq
value: p地名
output:
canonical_family: place
canonical_object_type: 海上地名
detection_key_template: label:海上地名
reason: 海上地名点层具有稳定的地名语义。
- rule_id: TAX-PLACE-002
rule_revision: 1
priority: 161
enabled: true
match_scope: layer
match_expr:
all:
- field: source_layer
op: eq
value: p地名陸
output:
canonical_family: place
canonical_object_type: 陆上地名
detection_key_template: label:陆上地名
reason: 陆上地名点层具有稳定的地名语义。
- rule_id: TAX-USAGE-001
rule_revision: 1
priority: 180
enabled: true
match_scope: feature
match_expr:
all:
- field: class_name
op: contains_any
value: [錨泊, 錨地]
output:
canonical_family: usage_area
canonical_object_type: 錨泊地
detection_key_template: mixed:錨泊地
reason: 锚泊类对象统一归入 usage_area 体系。
- rule_id: TAX-HAZ-001
rule_revision: 1
priority: 120
enabled: true
match_scope: feature
match_expr:
all:
- field: class_name
op: contains
value: 魚礁
- field: geom_type
op: in
value: [Point, Polygon, MultiPolygon]
output:
canonical_family: hazard
canonical_object_type: 魚礁
detection_key_template: mixed:魚礁
reason: 魚礁在当前业务目标下优先视为危险对象，并允许后续叠加用途语义。
- rule_id: TAX-FALLBACK-001
rule_revision: 1
priority: 9999
enabled: true
match_scope: feature
match_expr:
all: []
output:
canonical_family: unknown
canonical_object_type: 未分类对象
detection_key_template: unknown:{source_layer}
reason: 所有未命中 taxonomy 规则的对象统一回退到待补录分类。

114
tasks/pipline.md Normal file
View File

@@ -0,0 +1,114 @@
# NavSea Weather Server
Task: WeatherServer_Pipeline
Architecture: NavSea V11
Codex: codex6
Status: TODO
---
# 1 任务目标
创建 Weather Pipeline。
Pipeline 负责自动执行整个天气数据生产流程：
Downloader
→ Grid Builder
→ Vector Tile Generator
输入：
NOAA GFS
输出：
/weather/{time}/{z}/{x}/{y}.pbf
用于 NavSea 客户端加载天气图层。
---
# 2 Pipeline 结构
weather_server/
downloader/
grid/
tiles/
pipeline/
创建文件：
weather_server/pipeline/weather_pipeline.py
---
# 3 Pipeline 执行流程
step1 下载 GRIB
step2 生成 Weather Grid
step3 生成 Vector Tiles
---
# 4 创建 Pipeline 文件
weather_server/pipeline/weather_pipeline.py
---
# 5 实现代码
```python
import os
import subprocess
import time
DOWNLOADER = "weather_server/downloader/gfs_downloader.py"
GRID_BUILDER = "weather_server/grid/grid_builder_v2.py"
TILE_GENERATOR = "weather_server/tiles/vector_tile_generator.py"
def run_step(name, script):
print("\n==========================")
print("Running:", name)
print("==========================\n")
start = time.time()
result = subprocess.run(
["python", script],
capture_output=True,
text=True
)
print(result.stdout)
if result.returncode != 0:
print("ERROR:", result.stderr)
raise RuntimeError(name + " failed")
end = time.time()
print("\nFinished:", name)
print("Time:", round(end-start,2),"seconds")
def main():
print("\nNavSea Weather Pipeline\n")
run_step("Downloader", DOWNLOADER)
run_step("Grid Builder", GRID_BUILDER)
run_step("Vector Tile Generator", TILE_GENERATOR)
print("\nWeather pipeline completed.\n")
if __name__ == "__main__":
main()

83
tasks/test2.md Normal file
View File

@@ -0,0 +1,83 @@
# NavSea Weather Server
Step-2 测试：GFS Downloader
创建文件：
gfs_downloader.py
------------------------------------------------
import os
import requests
from datetime import datetime
BASE_URL = "https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod"
FORECAST_HOURS = [
0,3,6,9,12,15,18,21,24,
27,30,33,36,39,42,45,
48,51,54,57,60,63,66,
69,72
]
OUTPUT_DIR = "data/grib"
os.makedirs(OUTPUT_DIR, exist_ok=True)
def get_cycle():
now = datetime.utcnow()
hour = (now.hour // 6) * 6
cycle = f"{hour:02d}"
date = now.strftime("%Y%m%d")
return date, cycle
def download_file(url, path):
if os.path.exists(path):
print("skip", path)
return
print("downloading", url)
r = requests.get(url, stream=True)
if r.status_code != 200:
print("failed", url)
return
with open(path, "wb") as f:
for chunk in r.iter_content(1024*1024):
f.write(chunk)
def main():
date, cycle = get_cycle()
print("cycle:", date, cycle)
for fh in FORECAST_HOURS:
fh_str = f"{fh:03d}"
filename = f"gfs.t{cycle}z.pgrb2.0p25.f{fh_str}"
url = f"{BASE_URL}/gfs.{date}/{cycle}/atmos/{filename}"
output = os.path.join(
OUTPUT_DIR,
f"{date}_{cycle}_f{fh_str}.grib2"
)
download_file(url, output)
if __name__ == "__main__":
main()

264
tasks/updatedownload.md Normal file
View File

@@ -0,0 +1,264 @@
# NavSea Weather Server
Task: WeatherServer_GFS_Subset_Downloader
Architecture: V11
Codex: codex6
Status: TODO
---
# 任务目标
升级现有 gfs_downloader.py
当前问题：
直接下载完整 GFS 文件：
gfs.t00z.pgrb2.0p25.f000
文件大小：
≈ 500MB
下载 72 小时预测需要：
≈ 12GB
这是不可接受的。
解决方案：
使用 NOAA 提供的 GRIB Subset API：
filter_gfs_0p25.pl
只下载：
1 指定区域
2 指定变量
3 指定高度层
目标：
将单个 GRIB 文件缩小到：
2MB – 10MB
---
# 下载区域
NavSea 天气服务器只需要日本附近区域：
leftlon = 120
rightlon = 150
toplat = 50
bottomlat = 20
覆盖：
日本海
东海
太平洋日本海域
---
# 下载变量
Weather System V1 需要以下变量：
UGRD
VGRD
APCP
PRMSL
TMP
HTSGW
DIRPW
PERPW
说明：
UGRD VGRD → 风
APCP → 降水
PRMSL → 气压
TMP → 温度
HTSGW → 浪高
DIRPW → 浪方向
PERPW → 浪周期
变量参数来自 NOAA GFS 参数列表。 :contentReference[oaicite:0]{index=0}
---
# 下载高度层
需要以下层：
10 m above ground
surface
mean sea level
参数：
lev_10_m_above_ground
lev_surface
lev_mean_sea_level
---
# URL 构造规则
基础地址：
https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl
示例：
https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?
file=gfs.t00z.pgrb2.0p25.f003
&lev_10_m_above_ground=on
&lev_surface=on
&lev_mean_sea_level=on
&var_UGRD=on
&var_VGRD=on
&var_APCP=on
&var_PRMSL=on
&var_TMP=on
&var_HTSGW=on
&var_DIRPW=on
&var_PERPW=on
&leftlon=120
&rightlon=150
&toplat=50
&bottomlat=20
&dir=%2Fgfs.20260312%2F00%2Fatmos
---
# 修改文件
weather_server/downloader/gfs_downloader.py
---
# 新代码
```python
import os
import requests
from datetime import datetime
BASE_URL = "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl"
OUTPUT_DIR = "data/grib"
FORECAST_HOURS = [
0,3,6,9,12,15,18,21,24,
27,30,33,36,39,42,45,
48,51,54,57,60,63,66,
69,72
]
REGION = {
"leftlon":120,
"rightlon":150,
"toplat":50,
"bottomlat":20
}
VARIABLES = [
"UGRD",
"VGRD",
"APCP",
"PRMSL",
"TMP",
"HTSGW",
"DIRPW",
"PERPW"
]
LEVELS = [
"lev_10_m_above_ground",
"lev_surface",
"lev_mean_sea_level"
]
os.makedirs(OUTPUT_DIR, exist_ok=True)
def get_cycle():
now = datetime.utcnow()
hour = (now.hour // 6) * 6
cycle = f"{hour:02d}"
date = now.strftime("%Y%m%d")
return date, cycle
def build_url(date, cycle, fh):
filename = f"gfs.t{cycle}z.pgrb2.0p25.f{fh}"
params = {
"file": filename,
"leftlon": REGION["leftlon"],
"rightlon": REGION["rightlon"],
"toplat": REGION["toplat"],
"bottomlat": REGION["bottomlat"],
"dir": f"/gfs.{date}/{cycle}/atmos"
}
for v in VARIABLES:
params[f"var_{v}"] = "on"
for l in LEVELS:
params[l] = "on"
return BASE_URL, params
def download_file(url, params, path):
if os.path.exists(path):
print("skip", path)
return
print("downloading", path)
r = requests.get(url, params=params, stream=True)
if r.status_code != 200:
print("failed", r.status_code)
return
with open(path, "wb") as f:
for chunk in r.iter_content(1024*1024):
f.write(chunk)
def main():
date, cycle = get_cycle()
print("cycle:", date, cycle)
for fh in FORECAST_HOURS:
fh_str = f"{fh:03d}"
url, params = build_url(date, cycle, fh_str)
output = os.path.join(
OUTPUT_DIR,
f"{date}_{cycle}_f{fh_str}.grib2"
)
download_file(url, params, output)
if __name__ == "__main__":
main()

56
weather_pipeline.py Normal file
View File

@@ -0,0 +1,56 @@
import os
import subprocess
import time
DOWNLOADER = "gfs_downloader.py"
GRID_BUILDER = "grid/grid_builder_v2.py"
TILE_GENERATOR = "tiles/vector_tile_generator.py"
def run_step(name, script):
print("\n==========================")
print("Running:", name)
print("==========================\n")
start = time.time()
result = subprocess.run(
["python", script],
capture_output=True,
text=True
)
print(result.stdout)
if result.returncode != 0:
print("ERROR:", result.stderr)
raise RuntimeError(name + " failed")
end = time.time()
print("\nFinished:", name)
print("Time:", round(end-start,2),"seconds")
def main():
print("Weather Pipeline Starting...")
print("Date:", time.strftime("%Y-%m-%d %H:%M:%S"))
try:
run_step("GFS Downloader", DOWNLOADER)
run_step("Grid Builder v2", GRID_BUILDER)
run_step("Vector Tile Generator", TILE_GENERATOR)
print("\n" + "="*50)
print("Weather Pipeline Completed Successfully!")
print("="*50)
except RuntimeError as e:
print("\n" + "!"*50)
print("Pipeline Failed:", str(e))
print("!"*50)
exit(1)
if __name__ == "__main__":
main()