528 lines
6.1 KiB
Markdown
528 lines
6.1 KiB
Markdown
# NavSea Data Classification & Validation Pipeline
|
||
Version: v1.0
|
||
|
||
|
||
Goal:
|
||
|
||
从当前数据库中的 PBF 数据自动：
|
||
|
||
1. 解析 at 字段
|
||
2. 构建语义属性表
|
||
3. 构建对象目录
|
||
4. 自动推导对象类型
|
||
5. 验证分类正确性
|
||
6. 统计 tile 密度
|
||
7. 输出 NavSea 分类报告
|
||
|
||
所有中间结果写入 MySQL 表。
|
||
|
||
数据库在 192.168.200.184. root/2chi9ks2
|
||
可以建立数据库，也可以使用python代码，必要的时候可以下载所需的包。
|
||
---
|
||
|
||
# TASK 1
|
||
Parse AT Attributes
|
||
|
||
Goal:
|
||
解析 properties 表中 k='at' 的 JSON 字符串。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE IF NOT EXISTS at_attributes (
|
||
feature_id BIGINT,
|
||
k VARCHAR(100),
|
||
v TEXT
|
||
);
|
||
```
|
||
|
||
Logic:
|
||
|
||
1. 查询
|
||
|
||
```sql
|
||
SELECT feature_id, v
|
||
FROM properties
|
||
WHERE k='at';
|
||
```
|
||
|
||
2. v 是 JSON array
|
||
|
||
Example:
|
||
|
||
```
|
||
[["レイヤ","航路標識点"],["形状分類","シーバース灯"],["灯色","W (白)"]]
|
||
```
|
||
|
||
3. 解析后写入：
|
||
|
||
```
|
||
feature_id | k | v
|
||
--------------------
|
||
2682088 | レイヤ | 航路標識点
|
||
2682088 | 形状分類 | シーバース灯
|
||
2682088 | 灯色 | W (白)
|
||
```
|
||
|
||
Implementation:
|
||
|
||
Python
|
||
Libraries:
|
||
|
||
```
|
||
pymysql
|
||
json
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
table: at_attributes
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 2
|
||
Build Feature Semantic Table
|
||
|
||
Goal:
|
||
|
||
将 feature + vt_layer + at 属性合并。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE feature_semantic AS
|
||
SELECT
|
||
f.id AS feature_id,
|
||
f.vt_layer,
|
||
f.geom_type,
|
||
MAX(CASE WHEN a.k='分類' THEN a.v END) AS class_name,
|
||
MAX(CASE WHEN a.k='形状分類' THEN a.v END) AS shape_name,
|
||
MAX(CASE WHEN a.k='レイヤ' THEN a.v END) AS layer_name
|
||
FROM features f
|
||
LEFT JOIN at_attributes a
|
||
ON f.id = a.feature_id
|
||
GROUP BY f.id;
|
||
```
|
||
|
||
Output table:
|
||
|
||
```
|
||
feature_semantic
|
||
```
|
||
|
||
Columns:
|
||
|
||
```
|
||
feature_id
|
||
vt_layer
|
||
geom_type
|
||
class_name
|
||
shape_name
|
||
layer_name
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 3
|
||
Generate Object Catalog
|
||
|
||
Goal:
|
||
|
||
统计所有语义对象。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE object_catalog AS
|
||
SELECT
|
||
layer_name,
|
||
class_name,
|
||
shape_name,
|
||
vt_layer,
|
||
geom_type,
|
||
COUNT(*) AS feature_count
|
||
FROM feature_semantic
|
||
GROUP BY
|
||
layer_name,
|
||
class_name,
|
||
shape_name,
|
||
vt_layer,
|
||
geom_type;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
object_catalog
|
||
```
|
||
|
||
Purpose:
|
||
|
||
得到完整对象目录。
|
||
|
||
Example:
|
||
|
||
```
|
||
魚礁 | p施設 | Point | 24683
|
||
灯台 | p航路標識群 | Point | 18000
|
||
等深線 | L等深線 | Line | 440000
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 4
|
||
Geometry Consistency Check
|
||
|
||
Goal:
|
||
|
||
检查对象是否使用一致 geometry。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE geometry_consistency AS
|
||
SELECT
|
||
class_name,
|
||
geom_type,
|
||
COUNT(*) AS feature_count
|
||
FROM feature_semantic
|
||
GROUP BY class_name, geom_type;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
geometry_consistency
|
||
```
|
||
|
||
Purpose:
|
||
|
||
发现异常对象。
|
||
|
||
Example anomaly:
|
||
|
||
```
|
||
灯台 | Polygon
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 5
|
||
Candidate Object Type Detection
|
||
|
||
Goal:
|
||
|
||
自动推导 object_type。
|
||
|
||
规则优先级：
|
||
|
||
1 class_name
|
||
2 shape_name
|
||
3 vt_layer
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE object_type_candidates AS
|
||
SELECT
|
||
feature_id,
|
||
COALESCE(class_name, shape_name, vt_layer) AS object_type,
|
||
geom_type
|
||
FROM feature_semantic;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
object_type_candidates
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 6
|
||
Object Type Statistics
|
||
|
||
Goal:
|
||
|
||
统计对象数量。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE object_type_stats AS
|
||
SELECT
|
||
object_type,
|
||
geom_type,
|
||
COUNT(*) AS feature_count
|
||
FROM object_type_candidates
|
||
GROUP BY object_type, geom_type;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
object_type_stats
|
||
```
|
||
|
||
Purpose:
|
||
|
||
识别主要对象。
|
||
|
||
---
|
||
|
||
# TASK 7
|
||
Style Cross Reference
|
||
|
||
Goal:
|
||
|
||
分析 style.json。
|
||
|
||
Extract:
|
||
|
||
```
|
||
layer_id
|
||
source-layer
|
||
icon-image
|
||
line-color
|
||
fill-color
|
||
```
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE style_layers (
|
||
layer_id VARCHAR(200),
|
||
source_layer VARCHAR(200),
|
||
icon VARCHAR(200),
|
||
line_color VARCHAR(200),
|
||
fill_color VARCHAR(200)
|
||
);
|
||
```
|
||
|
||
Join:
|
||
|
||
```sql
|
||
CREATE TABLE style_mapping AS
|
||
SELECT
|
||
s.layer_id,
|
||
s.icon,
|
||
o.object_type,
|
||
o.geom_type
|
||
FROM style_layers s
|
||
JOIN object_type_candidates o
|
||
ON s.source_layer = o.object_type;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
style_mapping
|
||
```
|
||
|
||
Purpose:
|
||
|
||
确认对象 → 图标关系。
|
||
|
||
---
|
||
|
||
# TASK 8
|
||
Tile Density Analysis
|
||
|
||
Goal:
|
||
|
||
统计 tile feature 密度。
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE tile_density AS
|
||
SELECT
|
||
z,
|
||
x,
|
||
y,
|
||
COUNT(*) AS feature_count
|
||
FROM features
|
||
GROUP BY z,x,y;
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
tile_density
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 9
|
||
Tile Density Top 100
|
||
|
||
Create Table:
|
||
|
||
```sql
|
||
CREATE TABLE tile_density_top100 AS
|
||
SELECT *
|
||
FROM tile_density
|
||
ORDER BY feature_count DESC
|
||
LIMIT 100;
|
||
```
|
||
|
||
Purpose:
|
||
|
||
识别高密度 tile。
|
||
|
||
---
|
||
|
||
# TASK 10
|
||
Spatial Sanity Checks
|
||
|
||
Goal:
|
||
|
||
发现明显错误。
|
||
|
||
Examples:
|
||
|
||
Navigation lights not point:
|
||
|
||
```sql
|
||
CREATE TABLE anomaly_navigation_geom AS
|
||
SELECT *
|
||
FROM feature_semantic
|
||
WHERE class_name='灯台'
|
||
AND geom_type!='Point';
|
||
```
|
||
|
||
Reef not point/polygon:
|
||
|
||
```sql
|
||
CREATE TABLE anomaly_reef_geom AS
|
||
SELECT *
|
||
FROM feature_semantic
|
||
WHERE class_name='魚礁'
|
||
AND geom_type NOT IN ('Point','Polygon');
|
||
```
|
||
|
||
---
|
||
|
||
# TASK 11
|
||
NavSea Classification Report
|
||
|
||
Generate markdown:
|
||
|
||
```
|
||
navsea_classification_report.md
|
||
```
|
||
|
||
Content:
|
||
|
||
## Dataset Summary
|
||
|
||
Total features:
|
||
|
||
```sql
|
||
SELECT COUNT(*) FROM features;
|
||
```
|
||
|
||
Total object types:
|
||
|
||
```sql
|
||
SELECT COUNT(DISTINCT object_type)
|
||
FROM object_type_candidates;
|
||
```
|
||
|
||
---
|
||
|
||
## Top Object Types
|
||
|
||
```sql
|
||
SELECT *
|
||
FROM object_type_stats
|
||
ORDER BY feature_count DESC
|
||
LIMIT 50;
|
||
```
|
||
|
||
---
|
||
|
||
## Geometry Consistency
|
||
|
||
```
|
||
geometry_consistency
|
||
```
|
||
|
||
---
|
||
|
||
## Style Mapping
|
||
|
||
```
|
||
style_mapping
|
||
```
|
||
|
||
---
|
||
|
||
## Tile Density
|
||
|
||
```
|
||
tile_density_top100
|
||
```
|
||
|
||
---
|
||
|
||
# Final Deliverables
|
||
|
||
Database Tables:
|
||
|
||
```
|
||
at_attributes
|
||
feature_semantic
|
||
object_catalog
|
||
geometry_consistency
|
||
object_type_candidates
|
||
object_type_stats
|
||
style_layers
|
||
style_mapping
|
||
tile_density
|
||
tile_density_top100
|
||
anomaly_navigation_geom
|
||
anomaly_reef_geom
|
||
```
|
||
|
||
Final Document:
|
||
|
||
```
|
||
navsea_classification_report.md
|
||
```
|
||
|
||
---
|
||
|
||
# Execution
|
||
|
||
Single command:
|
||
|
||
```
|
||
python navsea_audit.py
|
||
```
|
||
|
||
Pipeline:
|
||
|
||
```
|
||
parse_at
|
||
→ semantic_table
|
||
→ object_catalog
|
||
→ classification
|
||
→ validation
|
||
→ report
|
||
```
|
||
|
||
---
|
||
|
||
# Success Criteria
|
||
|
||
The system must allow answering:
|
||
|
||
1 What objects exist
|
||
2 How many features each object has
|
||
3 What geometry they use
|
||
4 How they are styled
|
||
5 Whether classification is consistent
|
||
6 Whether tile density is reasonable |