diff --git a/AGENTS.md b/AGENTS.md index 94311ba..8704ecd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,61 +1,354 @@ -# Codex / AI Agent 项目规则 +# AGENTS.md — CM95 / CM93 → MySQL Importer -## 当前目标 +## 1. 当前项目状态 -第一阶段已经具备 `CM93 → MySQL` 完整导入与审计代码。当前工作是用真实 `CM93JAPAN` 数据运行、诊断和修复实际兼容问题。 +当前 CM93 → MySQL importer 已完成 CM93JAPAN Dataset ID=1 的全量验证。 -**不要提前实现 PBF。** +冻结基线: -## 固定基线 - -- Windows 11 -- Python 3.12.10 -- MySQL 5.7.34-log -- CM93 根目录:`D:\Ship\CM93V2\CM93JAPAN` -- DB Schema:`sql/001_schema_mysql57.sql` - -## 不可破坏规则 - -1. 禁止 MySQL 8 专用 SQL。 -2. 禁止 float32 存最终经纬度。 -3. 禁止 importer simplify/snap/quantize/round geometry。 -4. 禁止自动 make_valid 后覆盖原 Polygon。 -5. raw decoded tables、raw feature、raw attribute 必须可保留。 -6. `_wgsox/_wgsoy` 必须保存。 -7. 每个 feature 必须可追溯 dataset/cell/source_record_index。 -8. 未知 object/attribute 不得静默跳过;无法确定长度时 cell 必须 FAILED。 -9. parser 任一 header/count/EOF 不一致时 cell 必须 FAILED。 -10. 每个 cell 独立事务;失败不得破坏其他 DONE cell。 -11. DONE cell 支持续导跳过。 -12. 不复制 OpenCPN GPL 源文件或类实现;保持独立实现。 -13. PBF 不属于当前阶段。 - -## 实际运行顺序 - -```powershell -cm93db env-check -cm93db db-check -cm93db scan -cm93db inspect-dicts -cm93db import --scale G --limit 10 -cm93db audit --scale G --limit 10 --precision-samples 32 +```text +commit: 9d56592cadf7e047fb059e5a8746188595a08b43 +tag: cm93-mysql-importer-baseline-v1 ``` -Pilot PASS 后再全量。 +最终验收: -## 修复要求 +```text +Geometry Audit: OVERALL: PASS +Non-Geometry Audit: OVERALL: PASS +``` -发现失败时,优先保留一个具体失败 cell 作为回归输入,并报告: +PBF 当前继续冻结。 -- cell 路径 -- parser stage -- byte position/record index -- header source counts -- 实际异常 -- 修改前后审计结果 +--- -不得通过 catch-and-skip 使任务“看起来完成”。 +## 2. 核心原则 -## 文档 +1. 真实源码优先,不根据历史报告猜测当前实现。 +2. 修改旧文件前必须先读取当前实际文件。 +3. 不重新设计已经通过验收的 importer 主流程,除非任务明确要求。 +4. 生产数据链必须保持可追溯、可回归。 +5. MySQL 目标环境按 MySQL 5.7 处理,不假设 MySQL 8 Spatial 行为。 +6. PBF 不属于当前 importer 维护任务,除非明确启动 PBF 阶段。 -项目说明、报告、设计记录使用中文;代码稳定标识符、数据库字段使用英文。 +--- + +## 3. 冻结生产文件 + +以下是当前已验证的核心文件: + +```text +src/cm93db/geometry.py +src/cm93db/importer.py +sql/001_schema_mysql57.sql +``` + +没有明确任务时,不得重写 geometry / importer 主流程。 + +如必须修改,完成后必须运行长期回归验收。 + +--- + +## 4. Geometry Contract + +### 4.1 Valid geometry + +已经 OGC-valid 的 Area / Coverage: + +```text +PASS-THROUGH +``` + +不得为了 canonical 风格、ring 顺序或代码统一而重建。 + +### 4.2 Invalid polygon normalization + +只有 OGC-invalid Area / Coverage 才允许: + +```text +boundary noding +→ polygonize_full +→ odd-even face selection +→ union selected faces +→ GEOS normalize +→ valid Polygon / MultiPolygon +``` + +禁止用下列方法直接覆盖原始 CM93 Polygon: + +```text +make_valid() +ST_MakeValid() +buffer(0) +``` + +Normalization 必须满足: + +- 不创造原始 boundary linework 之外的新边界; +- 最终必须是 non-empty valid Polygon/MultiPolygon; +- 无法安全完成时保留 issue / 拒绝错误 Coverage; +- raw source 永远保留。 + +### 4.3 raw_geotype=16 + +`raw_geotype=16` 是正常 CM93 parent/no-geometry。 + +必须保持: + +```text +geometry_type = META +geom = None / NULL +geometry_issue = None / NULL +geometry_repaired = False / 0 +``` + +禁止重新生成: + +```text +geometry_issue = "geometry missing" +``` + +禁止增加: + +```text +geometry_invalid_count +``` + +不得人为创建空 geometry。 + +### 4.4 bbox + +normalized Area 的 bbox 必须来自最终保存的 geometry.bounds。 + +### 4.5 Coverage + +Coverage 参与 WGS offset 判断,因此 unresolved invalid Coverage 不允许静默落库并继续参与 offset 选择。 + +--- + +## 5. Raw Source Contract + +必须保留: + +```text +raw_feature_blob +raw_attribute_blob +chart_raw_record +source_record_index +``` + +任何 normalization 都只改变标准化 geometry 表达,不改变原始 CM93 二进制证据。 + +Feature 必须可追溯: + +```text +dataset_id +cell_id +source_record_index +``` + +--- + +## 6. Database Contract + +目标: + +```text +MySQL 5.7 +``` + +Normalized geometry: + +```text +SRID 4326 +WKB double precision +``` + +不要: + +- 把 MySQL 8 的 SRID column constraint 当成 MySQL 5.7; +- 依赖 MySQL 8-only spatial behavior; +- 擅自修改 schema 来迁就临时测试; +- 把 geometry 直接塞回 `chart_feature` 代替现有独立 geometry tables。 + +--- + +## 7. Import Contract + +正常 import: + +- DONE + source size 相同 + SHA-256 相同 → SKIP; +- FAILED 默认允许普通 import 重试; +- `--reimport` 才强制重新解析并写入。 + +不要为了测试方便修改 SHA-256 skip 逻辑。 + +单 Cell 写库必须保持事务语义;失败时不得留下半个 Cell payload。 + +--- + +## 8. Counter Contract + +`geometry_invalid_count`: + +- 只统计实际 unresolved geometry issue; +- 正常 META/no-geometry 不统计。 + +`geometry_repaired_count`: + +- 统计通过冻结 normalization 得到最终有效 polygon 的 Feature; +- 与 Coverage 表自身 `geometry_repaired` 的记录数不是简单独立相加关系。 + +当前基线: + +```text +Feature repaired = 55 +Coverage repaired = 3 +Geometry issues = 0 +``` + +--- + +## 9. 当前验收基线 + +CM93JAPAN / Dataset ID=1: + +```text +Cells 784 +Features 322465 +Attributes 405153 +Soundings 275700 +Coverages 1156 + +Area 81080 +Area valid 81080 +Coverage valid 1156 +Area bbox mismatch 0 + +Feature repaired 55 +Coverage repaired 3 +Feature geometry issues 0 +Coverage geometry issues 0 + +raw_geotype=16 4406 +geotype16 issues 0 + +Source files checked 784 +Source missing 0 +Source size mismatch 0 +Source SHA256 mismatch 0 +``` + +--- + +## 10. 必做回归 + +长期工具: + +```text +tools/cm93_mysql_final_geometry_audit.py +tools/cm93_mysql_non_geometry_final_audit.py +``` + +修改以下任一项后,必须重新运行相关全库验收: + +- parser; +- edge / point / sounding decode; +- geometry build; +- Coverage; +- WGS offset; +- odd-even normalization; +- bbox; +- DB mapping; +- raw preservation; +- schema; +- Feature / Attribute / Sounding / Coverage 关系。 + +验收目标: + +```text +OVERALL: PASS +``` + +不能用“抽样看起来正常”代替全库 audit。 + +--- + +## 11. 历史审计脚本 + +仓库根目录现有 V 系列脚本属于调查历史。 + +允许: + +- 保留; +- 人工复查; +- 用于追溯决策来源。 + +禁止: + +- 让生产代码 import 它们; +- 把它们当运行时依赖; +- 以旧 V 输出覆盖当前数据库事实; +- 因为历史脚本报告过旧问题,就假设当前 importer 仍存在同一问题。 + +长期回归以 `tools/` 下两个 final audit 为准。 + +--- + +## 12. 文档规则 + +修改行为后同步更新: + +```text +README.md +AGENTS.md +docs/CM93_Importer冻结基线_v1.md +``` + +文档必须区分: + +- 当前生产行为; +- 历史调查结论; +- 已关闭问题; +- 尚未启动的未来阶段。 + +不得继续把已经解决的问题写成“当前待处理”。 + +--- + +## 13. Git 规则 + +当前稳定 tag: + +```text +cm93-mysql-importer-baseline-v1 +``` + +不得移动或重写该 tag。 + +后续变更: + +1. 在当前 `main` 基础上修改; +2. 验证; +3. commit; +4. push; +5. 必要时再创建新的版本 tag。 + +不要把与当前任务无关的本地文件一起提交。 + +--- + +## 14. PBF + +当前: + +```text +PBF = OUT OF SCOPE / FROZEN +``` + +除非用户明确启动 PBF 阶段: + +- 不新增 PBF schema; +- 不新增 PBF exporter; +- 不修改 importer 为 PBF 服务; +- 不以 PBF 需求反向破坏当前 MySQL/raw 数据模型。 diff --git a/README.md b/README.md index c334866..49113c0 100644 --- a/README.md +++ b/README.md @@ -1,236 +1,467 @@ -# CM93 → MySQL 高精度母数据库导入器 v0.2.0 +# CM95 — CM93 → MySQL Importer -## 目标 +CM95 当前仓库的已验证基线,是一个将 CM93 数据解析、标准化并保存到 MySQL 5.7 的 importer。 -本工程完成第一阶段: +当前基线: -**CM93 Z/A/B/C/D/E/F/G → 完整解析 → MySQL 5.7.34 母数据库 → 源/库双重审计。** +```text +commit: 9d56592cadf7e047fb059e5a8746188595a08b43 +tag: cm93-mysql-importer-baseline-v1 +date: 2026-09-02 +``` -本阶段不生成 PBF。PBF 是第二阶段,可从本母数据库反复生成。 +本阶段只覆盖: -## 固定环境 +```text +CM93 source +→ binary parse +→ dictionary decode +→ geometry build / normalization +→ MySQL 5.7 +→ audit / traceability +``` -- Windows 11 -- pyenv-win -- Python **3.12.10** -- MySQL **5.7.34** -- 当前 CM93 日本数据目录:`D:\Ship\CM93V2\CM93JAPAN` -- 标准化几何:经纬度/WGS84 语义,SRID 4326 -- 坐标计算:Python double / MySQL WKB double - -## 已实现 - -- CM93 256-byte substitution decode -- 10-byte prolog + 128-byte header -- 文件长度完整性检查 -- vector table -- 2D point table -- 3D sounding table -- feature table -- object dictionary -- typed attribute dictionary / attribute blob -- Point / Line / Area / Sounding -- edge reverse flag -- Area contour/ring 组装 -- `_m_sor` coverage -- `_wgsox/_wgsoy` 保存与标准化坐标修正 -- CM93 6378388.0 m Mercator -- typed MySQL attributes -- raw feature / raw attribute -- 解码后的全部 CM93 table 分块保存 -- 每 cell 独立事务 -- DONE 自动跳过 -- FAILED 默认重试 -- 全目录断点续导 -- import job / event 日志 -- 源文件重新解析审计 -- 数据库逐 cell 数量审计 -- raw 数据字节完整性审计 -- SHA-256 审计 -- MySQL WKB 双精度回读审计 - -## 已验证的真实 CM93 小样本 - -开发时对用户提供的 147 个真实 CM93 cell 执行完整 parser 回归: - -- Cell: 147/147 PASS -- Feature: 100,621 -- Attribute: 112,746 -- 未发生 record 越界或文件尾剩余 -- header attribute-byte count 与解析结果一致 -- header geometry-ref count 与解析结果一致 -- header related-pointer count 与解析结果一致 -- decoded raw table 字节合计与原文件长度一致 - -MySQL 写入需要在目标 Windows/MySQL 5.7.34 上做 pilot 验收;本包提供对应命令和审计。 +PBF 当前继续冻结,不属于本阶段范围。 --- -# 1. 升级已有 v0.1.x 工程 +## 1. 当前状态 -如果原目录已经有正确的 `.venv` 和 `.env`,把 v0.2.0 ZIP 中整个 `cm93-mysql-importer-starter` 目录覆盖到原工程即可。 +CM93JAPAN / Dataset ID=1 已完成两套独立全库验收: -ZIP 不包含 `.env`,所以你已有的数据库密码不会被覆盖。 +```text +Geometry Audit: OVERALL: PASS +Non-Geometry Audit: OVERALL: PASS +``` -然后执行: +最终基线: + +| 项目 | 数量 / 结果 | +|---|---:| +| Cells | 784 | +| Features | 322465 | +| Attributes | 405153 | +| Sounding points | 275700 | +| Coverages | 1156 | +| Area | 81080 | +| Area valid | 81080 / 81080 | +| Coverage valid | 1156 / 1156 | +| Feature repaired | 55 | +| Coverage repaired | 3 | +| Feature geometry issues | 0 | +| Coverage geometry issues | 0 | +| raw_geotype=16 | 4406 | +| raw_geotype=16 issues | 0 | +| Area bbox mismatch | 0 | +| Source files checked | 784 | +| Source files missing | 0 | +| Source size mismatch | 0 | +| Source SHA-256 mismatch | 0 | + +--- + +## 2. 主要能力 + +Importer 当前处理并保存: + +- CM93 Cell 扫描; +- CM93OBJ / CM93ATTR 字典; +- Cell header; +- vector / edge; +- 2D point; +- 3D sounding; +- Feature; +- Attribute; +- Point / Line / Area; +- Coverage; +- WGS offset; +- Sounding point; +- Feature bbox; +- raw feature / raw attribute; +- Cell 原始 table 分块; +- source SHA-256; +- import job / event; +- geometry issue / repaired 状态。 + +每个 Feature 必须保持: + +```text +dataset_id +cell_id +source_record_index +``` + +从数据库记录可追溯回原始 CM93 Cell 和 source record。 + +--- + +## 3. 生产代码 + +核心生产文件: + +```text +src/cm93db/cm93_binary.py +src/cm93db/cell_parser.py +src/cm93db/geometry.py +src/cm93db/importer.py +src/cm93db/audit.py +sql/001_schema_mysql57.sql +``` + +Geometry 和 importer 的当前行为已经通过全量数据验收。后续没有明确任务时,不应重写其主流程。 + +--- + +## 4. 数据库 + +目标数据库: + +```text +MySQL 5.7 +``` + +Schema: + +```text +sql/001_schema_mysql57.sql +``` + +Importer 的 normalized SRID 为: + +```text +EPSG:4326 +``` + +MySQL geometry 通过 WKB 写入,坐标使用 double precision。 + +不要假设 MySQL 8 的 Spatial / SRID 行为适用于本项目。 + +数据库连接配置以项目现有 `.env.example` / 配置模块为准。 + +--- + +## 5. 导入 + +Windows 项目环境示例: ```powershell cd D:\React\cm93-mysql-importer-starter -& ".\.venv\Scripts\python.exe" -m pip install -e . -& ".\.venv\Scripts\cm93db.exe" --help + +& ".\.venv\Scripts\cm93db.exe" import D:\Ship\CM93V2\CM93JAPAN ``` -确认出现 `import` 和 `audit`。 - -# 2. .env - -确保: +普通导入会根据: ```text -CM93_DB_HOST=127.0.0.1 -CM93_DB_PORT=3306 -CM93_DB_NAME=cm93_chart -CM93_DB_USER=你的用户 -CM93_DB_PASSWORD=你的密码 -CM93_DB_CHARSET=utf8mb4 -CM93_SOURCE_DIR=D:\Ship\CM93V2\CM93JAPAN +chart_cell.import_status +source_size +source_sha256 ``` -`scan / inspect-dicts / import / audit` 现在都可以省略路径,自动读取 `CM93_SOURCE_DIR`。 +跳过已经成功导入且源文件 SHA-256 未变化的 Cell。 -# 3. 环境/数据库检查 - -```powershell -& ".\.venv\Scripts\cm93db.exe" env-check -& ".\.venv\Scripts\cm93db.exe" db-check -& ".\.venv\Scripts\cm93db.exe" scan -& ".\.venv\Scripts\cm93db.exe" inspect-dicts -``` - -# 4. 先做 G 层 10-cell pilot - -不要第一次就直接跑完整日本数据。先验证 MySQL 实际写入: - -```powershell -& ".\.venv\Scripts\cm93db.exe" import --scale G --limit 10 -``` - -然后立即审计同一批: - -```powershell -& ".\.venv\Scripts\cm93db.exe" audit --scale G --limit 10 --precision-samples 32 -``` - -目标: +典型输出: ```text -source_parse_failed : 0 -db_missing_cells : 0 -db_not_done_cells : 0 -count_mismatches : 0 -hash_mismatches : 0 -raw_preservation_bad : 0 -unknown_objects : 0 -unknown_attributes : 0 -precision_failed : 0 -RESULT : PASS +SKIP ... (SHA-256 unchanged) ``` -# 5. 全量导入日本数据 - -Pilot PASS 后: +需要强制重新解析并写入所有选中 Cell 时: ```powershell -& ".\.venv\Scripts\cm93db.exe" import +& ".\.venv\Scripts\cm93db.exe" import D:\Ship\CM93V2\CM93JAPAN --reimport ``` -程序逐 cell 工作。已经 `DONE` 且 **SHA-256 未变化** 的 cell 自动跳过,因此中断后重新执行同一命令即可续导。 +`--reimport` 会清理每个 Cell 已有 payload 后重新生成,因此只应在明确需要全量重导时使用。 -FAILED cell 默认在下一次执行时重试。 - -# 6. 全量审计 - -```powershell -& ".\.venv\Scripts\cm93db.exe" audit --precision-samples 256 -``` - -这会重新读取 CM93 源文件,并与数据库逐 cell 对比: - -- SHA-256 -- Feature 数 -- Attribute 数 -- Point geometry 数 -- Line geometry 数 -- Area geometry 数 -- `_m_sor` coverage 数 -- Sounding point 数 -- raw table 保存字节数 -- 未知 object / attribute -- 抽样 WKB 双精度 round-trip - -## 更严格的 geometry validity 检查 - -```powershell -& ".\.venv\Scripts\cm93db.exe" audit --precision-samples 256 --validate-geometry -``` - -`--validate-geometry` 会额外用 Shapely 全量检查拓扑,明显更重。CM93 中存在自相交/边界类 contour 并不自动等于解析错误,因此这项结果作为信息单独显示,不会静默修复原 geometry。 - -# 7. 常用控制 - -只处理 F/G: - -```powershell -cm93db import --scale F --scale G -``` - -只处理一个 cell: - -```powershell -cm93db import --cell 04501075.G -cm93db audit --cell 04501075.G --precision-samples 16 -``` - -强制重导: - -```powershell -cm93db import --cell 04501075.G --reimport -``` - -只审计源文件 parser,不访问数据库内容: - -```powershell -cm93db audit --source-only -``` - -正式母库**不建议**使用 `--no-raw-tables`。如果有意不保存 raw table,审计时需显式: - -```powershell -cm93db audit --allow-missing-raw -``` +如果单个 Cell 因临时数据库连接错误等原因处于 `FAILED`,普通 import 默认会重试失败 Cell,同时跳过其余未变化的 DONE Cell。 --- -# 精度原则 +## 6. Geometry 语义 -1. CM93 raw X/Y 保持 16-bit 原始语义。 -2. Cell easting/northing header 值使用 double。 -3. CM93 Mercator 半长轴固定 `6378388.0`。 -4. `_wgsox/_wgsoy` 以米为单位,在 Mercator→经纬度之前按 CM93 Transform 语义减去。 -5. MySQL geometry 由 WKB 写入;WKB 坐标为 IEEE-754 8-byte double。 -6. importer 阶段不 simplify、不 quantize、不 snap、不 round。 -7. Polygon 不自动 `make_valid`。 -8. 原始 decoded table + raw feature/attribute 与标准化 geometry 同时保留。 -9. PBF 将来是可重新生成的发布格式,不能反向影响母数据库精度。 +### 6.1 坐标 -详见 `docs/精度规范.md` 和 `docs/CM93解析与审计设计.md`。 +CM93 raw coordinate 按 Cell header 参数转换到 Mercator 坐标,再转换为经纬度。 -# Codex 接手 +Coverage 的 `_wgsox / _wgsoy` 用于最终 geometry 的 WGS offset 修正。 -Codex 不需要重新实现 parser。先读: +最终数据库 geometry 使用 SRID 4326。 -- `AGENTS.md` -- `codex/ARCHITECTURE.md` -- `codex/20_导入与审计接手.md` +### 6.2 Line -后续主要职责是:在你的 Win11 + MySQL 5.7.34 + CM93JAPAN 实际环境运行 pilot、分析失败 cell、修正真实兼容问题,并保留精度/审计边界。 +Line 必须保留断开的多个 line parts,不得为了得到单条 LineString 而跨 discontinuity 人为连接。 + +### 6.3 Area / Coverage + +当前冻结规则: + +#### 已经 OGC-valid + +```text +PASS-THROUGH +``` + +已经 valid 的 Polygon / MultiPolygon 不重新 normalization,不为了统一 ring 顺序而重建。 + +#### OGC-invalid + +只有 invalid Area / Coverage 才执行: + +```text +boundary noding +→ polygonize_full +→ odd-even face selection +→ union selected faces +→ GEOS normalize +→ valid Polygon / MultiPolygon +``` + +该规则来自 CM93 实际 topology / fill semantics 的完整审计。 + +生产规则明确禁止直接使用: + +```text +make_valid() +ST_MakeValid() +buffer(0) +``` + +来覆盖原始 CM93 Polygon 语义。 + +Normalization 不允许创造原始 CM93 boundary linework 之外的新边界。 + +open ring、断裂 contour 等真实结构异常不能静默“修好”;无法安全 normalization 的 Coverage 必须阻止错误 Cell 落库。 + +### 6.4 raw_geotype=16 + +`raw_geotype=16` 是 CM93 正常的 parent/no-geometry 对象。 + +当前正确语义: + +```text +geometry_type = META +geometry = NULL +geometry_issue = NULL +geometry_repaired = 0 +``` + +它不是 `"geometry missing"` 错误,也不进入 `geometry_invalid_count`。 + +当前 Dataset ID=1 共验证: + +```text +4406 / 4406 +``` + +全部为正常 parent/no-geometry。 + +### 6.5 bbox + +Area 发生 normalization 后: + +```text +feature bbox = final normalized geometry.bounds +``` + +不能继续使用 normalization 前可能包含 retraced connector 的 bounds。 + +--- + +## 7. geometry_repaired / geometry_issue + +`geometry_repaired=1` 表示 importer 对原 OGC-invalid CM93 polygon 执行冻结 normalization,并得到了最终 valid Polygon/MultiPolygon。 + +`geometry_issue` 只用于: + +- geometry 真正缺失但该 primitive 应存在 geometry; +- geometry empty; +- open ring / 结构异常; +- normalization 无法安全完成; +- 其他真实 geometry 异常。 + +正常 META parent/no-geometry 不属于 issue。 + +当前已验证: + +```text +Feature repaired = 55 +Coverage repaired = 3 +Feature issues = 0 +Coverage issues = 0 +``` + +Coverage 同时也是 `_m_sor` Feature,所以不要简单把 `55 + 3` 当作 58 个独立 CM93 source Feature。 + +--- + +## 8. Raw source 保留 + +标准化 geometry 不得破坏原始输入证据。 + +必须长期保留: + +```text +chart_feature.raw_feature_blob +chart_feature.raw_attribute_blob +chart_raw_record +source_record_index +``` + +`chart_raw_record` 使用分块保存原始 Cell table,并在 metadata 中记录: + +```text +chunk_index +chunk_count +total_bytes +``` + +这保证后续仍可重新核对 parser 与 CM93 原始二进制。 + +--- + +## 9. 长期回归验收 + +正式长期验收工具: + +```text +tools/cm93_mysql_final_geometry_audit.py +tools/cm93_mysql_non_geometry_final_audit.py +``` + +### Geometry 全库验收 + +```powershell +& ".\.venv\Scripts\python.exe" .\tools\cm93_mysql_final_geometry_audit.py ` + --host 127.0.0.1 ` + --port 3306 ` + --user root ` + --database cm93_chart ` + --dataset-id 1 +``` + +当前 CM93JAPAN 基线应得到: + +```text +Cells : 784 +Features : 322465 +Feature repaired : 55 +Coverage repaired : 3 +Feature issues : 0 +Coverage issues : 0 +Geotype16 total : 4406 +Geotype16 issues : 0 +Area total : 81080 +Area valid : 81080 +Area bbox mismatch : 0 +Coverage total : 1156 +Coverage valid : 1156 +Failure count : 0 +OVERALL: PASS +``` + +### 非 Geometry 全库验收 + +```powershell +& ".\.venv\Scripts\python.exe" .\tools\cm93_mysql_non_geometry_final_audit.py ` + --host 127.0.0.1 ` + --port 3306 ` + --user root ` + --database cm93_chart ` + --dataset-id 1 ` + --source-root "D:\Ship\CM93V2\CM93JAPAN" +``` + +当前 CM93JAPAN 基线应得到: + +```text +Cells : 784 +Features : 322465 +Attributes : 405153 +Soundings : 275700 +Coverages : 1156 + +Cell count mismatch : 0 +Feature check failures : 0 +Dictionary failures : 0 +Attribute failures : 0 +Coverage failures : 0 +Sounding failures : 0 +JSON/relation failures : 0 +Raw failures : 0 + +Source files checked : 784 +Source files missing : 0 +Source size mismatch : 0 +Source SHA256 mismatch : 0 + +Failure count : 0 +OVERALL: PASS +``` + +任何实质性修改以下部分后,都应重新执行对应验收: + +- binary parser; +- dictionary decode; +- geometry build; +- Area/Coverage normalization; +- WGS offset; +- importer DB mapping; +- raw preservation; +- Feature / Attribute / Sounding / Coverage 关系; +- bbox; +- schema。 + +--- + +## 10. 历史调查脚本 + +仓库中保留了本次 importer 调查过程中产生的 V 系列审计脚本。 + +它们用于记录问题定位过程,不再作为日常回归入口。 + +日常回归只使用: + +```text +tools/cm93_mysql_final_geometry_audit.py +tools/cm93_mysql_non_geometry_final_audit.py +``` + +历史调查脚本不要被新的生产代码 import,也不要把历史审计结果作为运行时依赖。 + +--- + +## 11. Git 冻结基线 + +当前正式可回退点: + +```text +commit 9d56592cadf7e047fb059e5a8746188595a08b43 +tag cm93-mysql-importer-baseline-v1 +``` + +该 tag 表示: + +```text +CM93 → MySQL importer validated baseline +``` + +tag 应保持不可变。 + +后续改动在 `main` 上正常提交;需要对比时,以此 tag 作为 importer normalization 前后已验证基线。 + +--- + +## 12. 当前不做的事情 + +当前 importer 阶段不包括: + +- PBF 生成; +- PBF schema; +- PBF renderer; +- PBF API; +- 为 PBF 改写 MySQL importer; +- 为了下游显示而损失 raw CM93 数据; +- 对原始 CM93 文件进行写回。 + +PBF 只有在单独明确启动下一阶段后才进入设计和实现。 diff --git a/docs/CM93_Importer冻结基线_v1.md b/docs/CM93_Importer冻结基线_v1.md new file mode 100644 index 0000000..328f5f7 --- /dev/null +++ b/docs/CM93_Importer冻结基线_v1.md @@ -0,0 +1,112 @@ +# CM93 Importer 冻结基线 v1 + +日期:2026-09-02 +仓库:CM95 +基线 commit:`9d56592cadf7e047fb059e5a8746188595a08b43` +基线 tag:`cm93-mysql-importer-baseline-v1` + +## 结论 + +CM93JAPAN Dataset ID=1 的 CM93 → MySQL importer 已完成: + +- Geometry normalization 全量验收; +- 非 Geometry 数据完整性全量验收; +- 原始 CM93 source file SHA-256 对账。 + +结果: + +```text +Geometry Audit: OVERALL: PASS +Non-Geometry Audit: OVERALL: PASS +``` + +## 数据基线 + +| 项目 | 数量 / 结果 | +|---|---:| +| Cells | 784 | +| Features | 322465 | +| Attributes | 405153 | +| Soundings | 275700 | +| Coverages | 1156 | +| Area | 81080 | +| Area valid | 81080 | +| Coverage valid | 1156 | +| Feature repaired | 55 | +| Coverage repaired | 3 | +| Feature geometry issues | 0 | +| Coverage geometry issues | 0 | +| raw_geotype=16 | 4406 | +| raw_geotype=16 issues | 0 | +| Area bbox mismatch | 0 | +| Source files checked | 784 | +| Source files missing | 0 | +| Source size mismatch | 0 | +| Source SHA-256 mismatch | 0 | + +## 冻结 Geometry 规则 + +Valid Area / Coverage: + +```text +PASS-THROUGH +``` + +Invalid Area / Coverage: + +```text +boundary noding +→ polygonize_full +→ odd-even face selection +→ union selected faces +→ GEOS normalize +→ valid Polygon / MultiPolygon +``` + +禁止生产写回策略: + +```text +make_valid() +ST_MakeValid() +buffer(0) +``` + +`raw_geotype=16` 是正常 parent/no-geometry,不属于 geometry issue。 + +normalized Area 的 bbox 必须来自最终 geometry.bounds。 + +## Raw source + +必须保留: + +```text +raw_feature_blob +raw_attribute_blob +chart_raw_record +source_record_index +``` + +## 长期回归入口 + +```text +tools/cm93_mysql_final_geometry_audit.py +tools/cm93_mysql_non_geometry_final_audit.py +``` + +## 核心生产文件 + +```text +src/cm93db/geometry.py +src/cm93db/importer.py +sql/001_schema_mysql57.sql +``` + +本次文档收尾不修改上述生产逻辑。 + +## PBF + +```text +OUT OF SCOPE / FROZEN +``` + +PBF 必须作为独立后续阶段明确启动。 diff --git a/tools/cm93_mysql_final_geometry_audit.py b/tools/cm93_mysql_final_geometry_audit.py new file mode 100644 index 0000000..61edee2 --- /dev/null +++ b/tools/cm93_mysql_final_geometry_audit.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +from __future__ import annotations +import argparse, csv, getpass, json, sys +from datetime import datetime +from pathlib import Path + +try: + import pymysql + from pymysql.cursors import DictCursor +except ImportError: + print("缺少 pymysql:pip install pymysql") + sys.exit(2) + +try: + try: + from shapely import from_wkb + def load_wkb(x): return from_wkb(x) + except ImportError: + from shapely import wkb + def load_wkb(x): return wkb.loads(x) +except ImportError: + print("缺少 shapely:pip install shapely") + sys.exit(2) + +def scalar(cur, sql, params=()): + cur.execute(sql, params) + r = cur.fetchone() + return None if not r else next(iter(r.values())) + +def as_bytes(v): + if isinstance(v, memoryview): return v.tobytes() + if isinstance(v, bytearray): return bytes(v) + return v + +def wkb_fn(cur, table): + for fn in ("ST_AsBinary","ST_AsWKB","AsBinary","AsWKB"): + try: + cur.execute(f"SELECT {fn}(geom) AS g FROM {table} LIMIT 1") + cur.fetchone() + return fn + except Exception: + pass + raise RuntimeError(f"{table}: 无可用 Geometry->WKB 函数") + +def main(): + ap = argparse.ArgumentParser(description="CM93 normalization 最终全库 Geometry 验收") + ap.add_argument("--host", default="127.0.0.1") + ap.add_argument("--port", type=int, default=3306) + ap.add_argument("--user", default="root") + ap.add_argument("--password") + ap.add_argument("--database", default="cm93_chart") + ap.add_argument("--dataset-id", type=int, default=1) + ap.add_argument("--bbox-tolerance", type=float, default=5e-6) + a = ap.parse_args() + if a.password is None: + a.password = getpass.getpass("MySQL password: ") + + stamp = datetime.now().strftime("%Y%m%d_%H%M%S") + outdir = Path(f"cm93_final_geometry_audit_{stamp}") + outdir.mkdir(exist_ok=True) + + conn = pymysql.connect( + host=a.host, port=a.port, user=a.user, password=a.password, + database=a.database, charset="utf8mb4", + cursorclass=DictCursor, autocommit=False, + connect_timeout=20, read_timeout=900 + ) + + failures = [] + report = {"dataset_id": a.dataset_id, "generated_at": datetime.now().isoformat()} + + try: + with conn.cursor() as cur: + try: cur.execute("SET SESSION TRANSACTION READ ONLY") + except Exception: pass + try: cur.execute("START TRANSACTION READ ONLY") + except Exception: cur.execute("START TRANSACTION") + + counts = { + "cells": int(scalar(cur,"SELECT COUNT(*) FROM chart_cell WHERE dataset_id=%s",(a.dataset_id,)) or 0), + "features": int(scalar(cur,"SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s",(a.dataset_id,)) or 0), + "feature_repaired": int(scalar(cur,"SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s AND geometry_repaired=1",(a.dataset_id,)) or 0), + "coverage_repaired": int(scalar(cur,"SELECT COUNT(*) FROM chart_coverage WHERE dataset_id=%s AND geometry_repaired=1",(a.dataset_id,)) or 0), + "feature_issues": int(scalar(cur,"SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s AND geometry_issue IS NOT NULL",(a.dataset_id,)) or 0), + "coverage_issues": int(scalar(cur,"SELECT COUNT(*) FROM chart_coverage WHERE dataset_id=%s AND geometry_issue IS NOT NULL",(a.dataset_id,)) or 0), + "geotype16_total": int(scalar(cur,"SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s AND raw_geotype=16",(a.dataset_id,)) or 0), + "geotype16_issues": int(scalar(cur,"SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s AND raw_geotype=16 AND geometry_issue IS NOT NULL",(a.dataset_id,)) or 0), + } + report["counts"] = counts + + expected = { + "cells": 784, "features": 322465, + "feature_repaired": 55, "coverage_repaired": 3, + "feature_issues": 0, "coverage_issues": 0, + "geotype16_total": 4406, "geotype16_issues": 0, + } + for k,v in expected.items(): + if counts[k] != v: + failures.append({"check":k,"expected":v,"actual":counts[k]}) + + afn = wkb_fn(cur, "chart_area_geometry") + cur.execute(f""" + SELECT f.id,f.object_class,f.geometry_repaired, + f.bbox_min_lon,f.bbox_min_lat,f.bbox_max_lon,f.bbox_max_lat, + {afn}(g.geom) AS wkb + FROM chart_feature f + JOIN chart_area_geometry g ON g.feature_id=f.id + WHERE f.dataset_id=%s + """,(a.dataset_id,)) + area_total=area_valid=bbox_bad=0 + for r in cur: + area_total += 1 + try: + g = load_wkb(as_bytes(r["wkb"])) + if g.is_empty or g.geom_type not in ("Polygon","MultiPolygon") or not g.is_valid: + failures.append({"check":"area_invalid","feature_id":r["id"],"object_class":r["object_class"],"geom_type":g.geom_type,"is_valid":bool(g.is_valid)}) + continue + area_valid += 1 + dbb=(float(r["bbox_min_lon"]),float(r["bbox_min_lat"]),float(r["bbox_max_lon"]),float(r["bbox_max_lat"])) + gb=tuple(float(x) for x in g.bounds) + if any(abs(x-y)>a.bbox_tolerance for x,y in zip(dbb,gb)): + bbox_bad += 1 + failures.append({"check":"area_bbox","feature_id":r["id"],"db_bbox":dbb,"geom_bbox":gb}) + except Exception as e: + failures.append({"check":"area_wkb","feature_id":r["id"],"error":repr(e)}) + + cfn = wkb_fn(cur, "chart_coverage") + cur.execute(f""" + SELECT id,cell_id,source_record_index,geometry_repaired,{cfn}(geom) AS wkb + FROM chart_coverage WHERE dataset_id=%s + """,(a.dataset_id,)) + cov_total=cov_valid=0 + for r in cur: + cov_total += 1 + try: + g = load_wkb(as_bytes(r["wkb"])) + if g.is_empty or g.geom_type not in ("Polygon","MultiPolygon") or not g.is_valid: + failures.append({"check":"coverage_invalid","coverage_id":r["id"],"cell_id":r["cell_id"],"geom_type":g.geom_type,"is_valid":bool(g.is_valid)}) + continue + cov_valid += 1 + except Exception as e: + failures.append({"check":"coverage_wkb","coverage_id":r["id"],"error":repr(e)}) + + report["area"]={"total":area_total,"valid":area_valid,"bbox_mismatch":bbox_bad} + report["coverage"]={"total":cov_total,"valid":cov_valid} + conn.rollback() + finally: + conn.close() + + report["failure_count"]=len(failures) + report["overall"]="PASS" if not failures else "FAIL" + + with (outdir/"失败明细_v1.csv").open("w",encoding="utf-8-sig",newline="") as f: + keys=["check","expected","actual","feature_id","coverage_id","cell_id","object_class","geom_type","is_valid","db_bbox","geom_bbox","error"] + w=csv.DictWriter(f,fieldnames=keys,extrasaction="ignore"); w.writeheader(); w.writerows(failures) + + (outdir/"最终验收报告_v1.json").write_text(json.dumps(report,ensure_ascii=False,indent=2),encoding="utf-8") + summary = f"""CM93 normalization 最终全库 Geometry 验收 v1 +Dataset ID : {a.dataset_id} +Cells : {report['counts']['cells']} +Features : {report['counts']['features']} +Feature repaired : {report['counts']['feature_repaired']} +Coverage repaired : {report['counts']['coverage_repaired']} +Feature issues : {report['counts']['feature_issues']} +Coverage issues : {report['counts']['coverage_issues']} +Geotype16 total : {report['counts']['geotype16_total']} +Geotype16 issues : {report['counts']['geotype16_issues']} +Area total : {report['area']['total']} +Area valid : {report['area']['valid']} +Area bbox mismatch : {report['area']['bbox_mismatch']} +Coverage total : {report['coverage']['total']} +Coverage valid : {report['coverage']['valid']} +Failure count : {report['failure_count']} +OVERALL: {report['overall']} +""" + (outdir/"最终验收摘要_v1.txt").write_text(summary,encoding="utf-8") + print(summary) + print("报告目录:", outdir.resolve()) + +if __name__=="__main__": + main() diff --git a/tools/cm93_mysql_non_geometry_final_audit.py b/tools/cm93_mysql_non_geometry_final_audit.py new file mode 100644 index 0000000..f4ef342 --- /dev/null +++ b/tools/cm93_mysql_non_geometry_final_audit.py @@ -0,0 +1,1272 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +""" +CM93 -> MySQL 非 Geometry 数据完整性全库最终验收 v1 + +目标 +==== +在 Geometry normalization 已通过后,对同一 Dataset 的“非 Geometry”数据链做最终只读验收。 + +检查: +1. Dataset / Cell 完整性 +2. 每个 Cell 的 feature_count / attribute_count 与实际落库一致 +3. Feature 的 dataset/cell/source_record_index 可追溯性 +4. Feature source_record_index 在 Cell 内连续、唯一 +5. Attribute 外键、ordinal、dictionary 映射、类型值通道 +6. Coverage 与 _m_sor Feature 一一对应且同 Cell +7. Sounding 外键、point_index 连续性、depth 公式 +8. raw_feature_blob / raw_attribute_blob 保留 +9. chart_raw_record 分块索引、chunk_count、total_bytes、JSON metadata +10. Object / Attribute dictionary 映射完整性 +11. attributes_json 可解析;_cm93_related_indices 若存在,必须指向同 Cell Feature +12. 可选:源 CM93 文件存在、source_size、SHA-256 与数据库一致 +13. orphan / duplicate 统计必须为 0 + +不检查 Geometry validity(该项已经由独立 Geometry audit 验收)。 +不修改数据库。 +不依赖历史 V 系列测试输出。 +不涉及 PBF。 + +当前 CM93JAPAN 固定基线默认: +- Cells 784 +- Features 322465 +- Attributes 405153 +- Soundings 275700 + +如果以后换 Dataset,可以传 --no-fixed-baseline。 +""" + +from __future__ import annotations + +import argparse +import csv +import getpass +import hashlib +import json +import math +import sys +import zipfile +from collections import defaultdict +from datetime import datetime +from pathlib import Path +from typing import Any, Iterable + +try: + import pymysql + from pymysql.cursors import DictCursor, SSCursor +except ImportError: + print("缺少 pymysql,请执行: pip install pymysql") + sys.exit(2) + + +EXPECTED_COUNTS = { + "cells": 784, + "features": 322465, + "attributes": 405153, + "soundings": 275700, +} + + +def parse_args(): + p = argparse.ArgumentParser( + description="CM93 -> MySQL 非 Geometry 数据完整性全库最终验收 v1" + ) + p.add_argument("--host", default="127.0.0.1") + p.add_argument("--port", type=int, default=3306) + p.add_argument("--user", default="root") + p.add_argument("--password", default=None) + p.add_argument("--database", default="cm93_chart") + p.add_argument("--dataset-id", type=int, default=1) + p.add_argument( + "--source-root", + default=None, + help="可选:CM93 根目录,例如 D:\\Ship\\CM93V2\\CM93JAPAN;提供后校验文件存在/大小/SHA256", + ) + p.add_argument( + "--no-fixed-baseline", + action="store_true", + help="不要求当前 CM93JAPAN 的 784/322465/405153/275700 固定数量", + ) + p.add_argument( + "--output-dir", + default=None, + ) + return p.parse_args() + + +def scalar(cur, sql: str, params=()) -> Any: + cur.execute(sql, params) + row = cur.fetchone() + if row is None: + return None + if isinstance(row, dict): + return next(iter(row.values())) + return row[0] + + +def fetchall(cur, sql: str, params=()): + cur.execute(sql, params) + return list(cur.fetchall()) + + +def add_failure( + failures: list[dict[str, Any]], + check: str, + detail: Any = None, + **kwargs, +): + row = {"check": check} + if detail is not None: + row["detail"] = detail + row.update(kwargs) + failures.append(row) + + +def json_load(value): + if value is None: + return None + if isinstance(value, (dict, list)): + return value + if isinstance(value, (bytes, bytearray, memoryview)): + value = bytes(value).decode("utf-8") + return json.loads(value) + + +def file_sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +def write_csv(path: Path, rows: Iterable[dict[str, Any]]): + rows = list(rows) + if not rows: + path.write_text("", encoding="utf-8-sig") + return + + fields = [] + seen = set() + for row in rows: + for key in row: + if key not in seen: + seen.add(key) + fields.append(key) + + with path.open("w", encoding="utf-8-sig", newline="") as f: + w = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore") + w.writeheader() + for row in rows: + out = {} + for key in fields: + v = row.get(key) + if isinstance(v, (dict, list, tuple)): + v = json.dumps(v, ensure_ascii=False, default=str) + out[key] = v + w.writerow(out) + + +def table_exists(cur, database: str, table: str) -> bool: + return bool(scalar( + cur, + """ + SELECT COUNT(*) + FROM information_schema.tables + WHERE table_schema=%s AND table_name=%s + """, + (database, table), + )) + + +def begin_read_only(cur, warnings: list[str]): + try: + cur.execute("SET SESSION TRANSACTION READ ONLY") + except Exception as e: + warnings.append(f"SET SESSION TRANSACTION READ ONLY: {e}") + + try: + cur.execute("START TRANSACTION READ ONLY") + except Exception as e: + warnings.append(f"START TRANSACTION READ ONLY: {e}") + cur.execute("START TRANSACTION") + + +def main(): + args = parse_args() + + if args.password is None: + args.password = getpass.getpass("MySQL password: ") + + source_root = ( + Path(args.source_root).expanduser().resolve() + if args.source_root + else None + ) + + stamp = datetime.now().strftime("%Y%m%d_%H%M%S") + outdir = Path( + args.output_dir + or f"cm93_non_geometry_final_audit_dataset_{args.dataset_id}_{stamp}" + ).resolve() + outdir.mkdir(parents=True, exist_ok=True) + + failures: list[dict[str, Any]] = [] + warnings: list[str] = [] + + report: dict[str, Any] = { + "version": "v1.0", + "generated_at": datetime.now().isoformat(timespec="seconds"), + "database": args.database, + "dataset_id": args.dataset_id, + "read_only": True, + "pbf": "OUT_OF_SCOPE", + "source_root": str(source_root) if source_root else None, + } + + conn = pymysql.connect( + host=args.host, + port=args.port, + user=args.user, + password=args.password, + database=args.database, + charset="utf8mb4", + cursorclass=DictCursor, + autocommit=False, + connect_timeout=20, + read_timeout=900, + write_timeout=30, + ) + + try: + with conn.cursor() as cur: + begin_read_only(cur, warnings) + + print("[1/10] 基础表 / Dataset / 总数 ...") + + required_tables = [ + "chart_dataset", + "chart_cell", + "chart_feature", + "chart_feature_attribute", + "chart_coverage", + "chart_sounding_point", + "chart_raw_record", + "chart_dictionary_object", + "chart_dictionary_attribute", + ] + missing_tables = [ + t for t in required_tables + if not table_exists(cur, args.database, t) + ] + report["missing_tables"] = missing_tables + if missing_tables: + add_failure( + failures, + "missing_tables", + missing_tables, + ) + raise RuntimeError( + "缺少必要表:" + ", ".join(missing_tables) + ) + + dataset_count = int(scalar( + cur, + "SELECT COUNT(*) FROM chart_dataset WHERE id=%s", + (args.dataset_id,), + ) or 0) + + if dataset_count != 1: + add_failure( + failures, + "dataset_exists", + expected=1, + actual=dataset_count, + ) + + counts = { + "cells": int(scalar( + cur, + "SELECT COUNT(*) FROM chart_cell WHERE dataset_id=%s", + (args.dataset_id,), + ) or 0), + "features": int(scalar( + cur, + "SELECT COUNT(*) FROM chart_feature WHERE dataset_id=%s", + (args.dataset_id,), + ) or 0), + "attributes": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + """, + (args.dataset_id,), + ) or 0), + "soundings": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_sounding_point s + JOIN chart_feature f ON f.id=s.feature_id + WHERE f.dataset_id=%s + """, + (args.dataset_id,), + ) or 0), + "coverages": int(scalar( + cur, + "SELECT COUNT(*) FROM chart_coverage WHERE dataset_id=%s", + (args.dataset_id,), + ) or 0), + "raw_records": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_raw_record r + JOIN chart_cell c ON c.id=r.cell_id + WHERE c.dataset_id=%s + """, + (args.dataset_id,), + ) or 0), + } + report["counts"] = counts + + if not args.no_fixed_baseline: + for key, expected in EXPECTED_COUNTS.items(): + if counts[key] != expected: + add_failure( + failures, + f"baseline_{key}", + expected=expected, + actual=counts[key], + ) + + done_cells = int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_cell + WHERE dataset_id=%s AND import_status='DONE' + """, + (args.dataset_id,), + ) or 0) + non_done_cells = counts["cells"] - done_cells + + report["cell_status"] = { + "done": done_cells, + "non_done": non_done_cells, + } + + if non_done_cells: + add_failure( + failures, + "cell_status_non_done", + actual=non_done_cells, + ) + + print("[2/10] Cell header counts vs 实际落库 ...") + + cell_mismatch_rows = fetchall( + cur, + """ + SELECT + c.id AS cell_id, + c.source_relative_path, + c.feature_count AS expected_features, + COALESCE(f.actual_features,0) AS actual_features, + c.attribute_count AS expected_attributes, + COALESCE(a.actual_attributes,0) AS actual_attributes + FROM chart_cell c + LEFT JOIN ( + SELECT cell_id,COUNT(*) AS actual_features + FROM chart_feature + WHERE dataset_id=%s + GROUP BY cell_id + ) f ON f.cell_id=c.id + LEFT JOIN ( + SELECT f.cell_id,COUNT(*) AS actual_attributes + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + GROUP BY f.cell_id + ) a ON a.cell_id=c.id + WHERE c.dataset_id=%s + AND ( + c.feature_count<>COALESCE(f.actual_features,0) + OR c.attribute_count<>COALESCE(a.actual_attributes,0) + ) + ORDER BY c.id + """, + (args.dataset_id, args.dataset_id, args.dataset_id), + ) + + report["cell_count_mismatch"] = len(cell_mismatch_rows) + for row in cell_mismatch_rows: + add_failure( + failures, + "cell_header_count_mismatch", + **row, + ) + + print("[3/10] Feature traceability / source_record_index ...") + + feature_checks = { + "feature_cell_dataset_mismatch": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature f + JOIN chart_cell c ON c.id=f.cell_id + WHERE f.dataset_id=%s + AND c.dataset_id<>f.dataset_id + """, + (args.dataset_id,), + ) or 0), + "duplicate_source_record_index": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT cell_id,source_record_index + FROM chart_feature + WHERE dataset_id=%s + GROUP BY cell_id,source_record_index + HAVING COUNT(*)>1 + ) x + """, + (args.dataset_id,), + ) or 0), + "source_record_index_not_contiguous": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT + cell_id, + COUNT(*) AS n, + COUNT(DISTINCT source_record_index) AS nd, + MIN(source_record_index) AS min_i, + MAX(source_record_index) AS max_i + FROM chart_feature + WHERE dataset_id=%s + GROUP BY cell_id + HAVING + nd<>n + OR min_i<>0 + OR max_i<>n-1 + ) x + """, + (args.dataset_id,), + ) or 0), + "raw_feature_blob_null": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature + WHERE dataset_id=%s + AND raw_feature_blob IS NULL + """, + (args.dataset_id,), + ) or 0), + "raw_attribute_blob_null": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature + WHERE dataset_id=%s + AND raw_attribute_blob IS NULL + """, + (args.dataset_id,), + ) or 0), + "attributes_json_null": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature + WHERE dataset_id=%s + AND attributes_json IS NULL + """, + (args.dataset_id,), + ) or 0), + } + report["feature_checks"] = feature_checks + + for key, value in feature_checks.items(): + if value: + add_failure( + failures, + key, + expected=0, + actual=value, + ) + + print("[4/10] Object / Attribute dictionary 映射 ...") + + dictionary_checks = { + "feature_unknown_object_id": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature f + LEFT JOIN chart_dictionary_object d + ON d.dataset_id=f.dataset_id + AND d.object_class_id=f.object_class_id + WHERE f.dataset_id=%s + AND d.object_class_id IS NULL + """, + (args.dataset_id,), + ) or 0), + "feature_object_name_mismatch": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature f + JOIN chart_dictionary_object d + ON d.dataset_id=f.dataset_id + AND d.object_class_id=f.object_class_id + WHERE f.dataset_id=%s + AND f.object_class<>d.object_class + """, + (args.dataset_id,), + ) or 0), + "attribute_unknown_id": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + LEFT JOIN chart_dictionary_attribute d + ON d.dataset_id=f.dataset_id + AND d.attribute_id=a.attribute_id + WHERE f.dataset_id=%s + AND d.attribute_id IS NULL + """, + (args.dataset_id,), + ) or 0), + "attribute_name_mismatch": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + JOIN chart_dictionary_attribute d + ON d.dataset_id=f.dataset_id + AND d.attribute_id=a.attribute_id + WHERE f.dataset_id=%s + AND a.attribute_name<>d.attribute_name + """, + (args.dataset_id,), + ) or 0), + } + report["dictionary_checks"] = dictionary_checks + + for key, value in dictionary_checks.items(): + if value: + add_failure( + failures, + key, + expected=0, + actual=value, + ) + + print("[5/10] Attribute orphan / ordinal / value_type ...") + + attribute_checks = { + "attribute_orphan": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature_attribute a + LEFT JOIN chart_feature f ON f.id=a.feature_id + WHERE f.id IS NULL + """, + ) or 0), + "attribute_duplicate_ordinal": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT a.feature_id,a.ordinal_no + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + GROUP BY a.feature_id,a.ordinal_no + HAVING COUNT(*)>1 + ) x + """, + (args.dataset_id,), + ) or 0), + "attribute_ordinal_not_contiguous": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT + a.feature_id, + COUNT(*) AS n, + COUNT(DISTINCT a.ordinal_no) AS nd, + MIN(a.ordinal_no) AS min_o, + MAX(a.ordinal_no) AS max_o + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + GROUP BY a.feature_id + HAVING + nd<>n + OR min_o<>0 + OR max_o<>n-1 + ) x + """, + (args.dataset_id,), + ) or 0), + "attribute_unknown_value_type": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + AND a.value_type NOT IN + ('BYTE','INT','LONG','REAL','WORD10','STRING','LIST','COMPLEX','UNKNOWN') + """, + (args.dataset_id,), + ) or 0), + } + report["attribute_checks"] = attribute_checks + + for key, value in attribute_checks.items(): + if value: + add_failure( + failures, + key, + expected=0, + actual=value, + ) + + print("[6/10] Coverage <-> _m_sor Feature 映射 ...") + + coverage_checks = { + "coverage_duplicate_source_record_index": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT cell_id,source_record_index + FROM chart_coverage + WHERE dataset_id=%s + GROUP BY cell_id,source_record_index + HAVING COUNT(*)>1 + ) x + """, + (args.dataset_id,), + ) or 0), + "coverage_without_m_sor_feature": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_coverage c + LEFT JOIN chart_feature f + ON f.dataset_id=c.dataset_id + AND f.cell_id=c.cell_id + AND f.source_record_index=c.source_record_index + AND f.object_class='_m_sor' + WHERE c.dataset_id=%s + AND f.id IS NULL + """, + (args.dataset_id,), + ) or 0), + "m_sor_without_coverage": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature f + LEFT JOIN chart_coverage c + ON c.dataset_id=f.dataset_id + AND c.cell_id=f.cell_id + AND c.source_record_index=f.source_record_index + WHERE f.dataset_id=%s + AND f.object_class='_m_sor' + AND c.id IS NULL + """, + (args.dataset_id,), + ) or 0), + "feature_coverage_cross_cell": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_feature f + JOIN chart_coverage c ON c.id=f.coverage_id + WHERE f.dataset_id=%s + AND ( + c.dataset_id<>f.dataset_id + OR c.cell_id<>f.cell_id + ) + """, + (args.dataset_id,), + ) or 0), + } + report["coverage_checks"] = coverage_checks + + for key, value in coverage_checks.items(): + if value: + add_failure( + failures, + key, + expected=0, + actual=value, + ) + + print("[7/10] Sounding 数据链 ...") + + sounding_checks = { + "sounding_orphan": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_sounding_point s + LEFT JOIN chart_feature f ON f.id=s.feature_id + WHERE f.id IS NULL + """, + ) or 0), + "sounding_wrong_feature_type": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_sounding_point s + JOIN chart_feature f ON f.id=s.feature_id + WHERE f.dataset_id=%s + AND f.geometry_type<>'SOUNDING' + """, + (args.dataset_id,), + ) or 0), + "sounding_duplicate_point_index": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT s.feature_id,s.point_index + FROM chart_sounding_point s + JOIN chart_feature f ON f.id=s.feature_id + WHERE f.dataset_id=%s + GROUP BY s.feature_id,s.point_index + HAVING COUNT(*)>1 + ) x + """, + (args.dataset_id,), + ) or 0), + "sounding_index_not_contiguous": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT + s.feature_id, + COUNT(*) AS n, + COUNT(DISTINCT s.point_index) AS nd, + MIN(s.point_index) AS min_i, + MAX(s.point_index) AS max_i + FROM chart_sounding_point s + JOIN chart_feature f ON f.id=s.feature_id + WHERE f.dataset_id=%s + GROUP BY s.feature_id + HAVING + nd<>n + OR min_i<>0 + OR max_i<>n-1 + ) x + """, + (args.dataset_id,), + ) or 0), + "sounding_depth_formula_mismatch": int(scalar( + cur, + """ + SELECT COUNT(*) + FROM chart_sounding_point s + JOIN chart_feature f ON f.id=s.feature_id + WHERE f.dataset_id=%s + AND ABS( + s.depth - + CASE + WHEN s.raw_z>=12000 THEN s.raw_z-12000 + ELSE s.raw_z/10.0 + END + ) > 0.000001 + """, + (args.dataset_id,), + ) or 0), + } + report["sounding_checks"] = sounding_checks + + for key, value in sounding_checks.items(): + if value: + add_failure( + failures, + key, + expected=0, + actual=value, + ) + + print("[8/10] attributes_json / relation indices 全量解析 ...") + + relation_bad = 0 + attributes_json_bad = 0 + attributes_name_missing = 0 + + cur.execute( + """ + SELECT + id, + cell_id, + source_record_index, + attributes_json + FROM chart_feature + WHERE dataset_id=%s + ORDER BY cell_id,source_record_index + """, + (args.dataset_id,), + ) + feature_json_rows = list(cur.fetchall()) + + indices_by_cell: dict[int, set[int]] = defaultdict(set) + parsed_by_feature: dict[int, Any] = {} + + for row in feature_json_rows: + cell_id = int(row["cell_id"]) + src_idx = int(row["source_record_index"]) + indices_by_cell[cell_id].add(src_idx) + + try: + obj = json_load(row["attributes_json"]) + if not isinstance(obj, dict): + raise ValueError("attributes_json 不是 JSON object") + parsed_by_feature[int(row["id"])] = obj + except Exception as e: + attributes_json_bad += 1 + add_failure( + failures, + "attributes_json_parse", + feature_id=row["id"], + cell_id=cell_id, + source_record_index=src_idx, + detail=f"{type(e).__name__}: {e}", + ) + + for row in feature_json_rows: + fid = int(row["id"]) + obj = parsed_by_feature.get(fid) + if obj is None: + continue + + related = obj.get("_cm93_related_indices") + if related is not None: + if not isinstance(related, list): + relation_bad += 1 + add_failure( + failures, + "related_indices_not_list", + feature_id=fid, + actual_type=type(related).__name__, + ) + else: + cell_indices = indices_by_cell[int(row["cell_id"])] + for target in related: + try: + ti = int(target) + except Exception: + relation_bad += 1 + add_failure( + failures, + "related_index_not_int", + feature_id=fid, + target=repr(target), + ) + continue + + if ti not in cell_indices: + relation_bad += 1 + add_failure( + failures, + "related_index_missing_target", + feature_id=fid, + cell_id=row["cell_id"], + source_record_index=row["source_record_index"], + target_source_record_index=ti, + ) + + # Attribute row 的 attribute_name 应在对应 attributes_json 顶层出现。 + cur.execute( + """ + SELECT + a.feature_id, + a.attribute_name + FROM chart_feature_attribute a + JOIN chart_feature f ON f.id=a.feature_id + WHERE f.dataset_id=%s + """, + (args.dataset_id,), + ) + + for row in cur: + obj = parsed_by_feature.get(int(row["feature_id"])) + if obj is None: + continue + if row["attribute_name"] not in obj: + attributes_name_missing += 1 + if attributes_name_missing <= 1000: + add_failure( + failures, + "attribute_name_missing_from_attributes_json", + feature_id=row["feature_id"], + attribute_name=row["attribute_name"], + ) + + report["json_relation_checks"] = { + "attributes_json_bad": attributes_json_bad, + "related_indices_bad": relation_bad, + "attribute_name_missing_from_attributes_json": attributes_name_missing, + } + + if attributes_name_missing > 1000: + add_failure( + failures, + "attribute_name_missing_from_attributes_json_overflow", + detail=f"总数={attributes_name_missing},明细只记录前 1000 条", + ) + + print("[9/10] chart_raw_record 分块完整性 ...") + + cur.execute( + """ + SELECT + r.cell_id, + r.record_kind, + r.source_record_index, + OCTET_LENGTH(r.raw_blob) AS raw_bytes, + r.raw_metadata_json + FROM chart_raw_record r + JOIN chart_cell c ON c.id=r.cell_id + WHERE c.dataset_id=%s + ORDER BY r.cell_id,r.record_kind,r.source_record_index + """, + (args.dataset_id,), + ) + + raw_groups: dict[tuple[int, str], list[dict[str, Any]]] = defaultdict(list) + raw_json_bad = 0 + raw_empty_blob = 0 + + for row in cur: + if int(row["raw_bytes"] or 0) <= 0: + raw_empty_blob += 1 + add_failure( + failures, + "raw_record_empty_blob", + cell_id=row["cell_id"], + record_kind=row["record_kind"], + source_record_index=row["source_record_index"], + ) + + try: + meta = json_load(row["raw_metadata_json"]) + if not isinstance(meta, dict): + raise ValueError("raw_metadata_json 不是 JSON object") + except Exception as e: + raw_json_bad += 1 + add_failure( + failures, + "raw_metadata_json_parse", + cell_id=row["cell_id"], + record_kind=row["record_kind"], + source_record_index=row["source_record_index"], + detail=f"{type(e).__name__}: {e}", + ) + meta = None + + raw_groups[ + (int(row["cell_id"]), str(row["record_kind"])) + ].append({ + "index": int(row["source_record_index"]), + "raw_bytes": int(row["raw_bytes"] or 0), + "meta": meta, + }) + + raw_group_bad = 0 + + for (cell_id, kind), rows in raw_groups.items(): + indices = [r["index"] for r in rows] + expected_indices = list(range(len(rows))) + + if indices != expected_indices: + raw_group_bad += 1 + add_failure( + failures, + "raw_chunk_index_not_contiguous", + cell_id=cell_id, + record_kind=kind, + indices=indices[:50], + count=len(indices), + ) + + metas = [r["meta"] for r in rows if isinstance(r["meta"], dict)] + if len(metas) != len(rows): + continue + + chunk_counts = {m.get("chunk_count") for m in metas} + total_bytes_set = {m.get("total_bytes") for m in metas} + chunk_indices_meta = [m.get("chunk_index") for m in metas] + actual_bytes = sum(r["raw_bytes"] for r in rows) + + if chunk_counts != {len(rows)}: + raw_group_bad += 1 + add_failure( + failures, + "raw_chunk_count_mismatch", + cell_id=cell_id, + record_kind=kind, + metadata_chunk_counts=list(chunk_counts), + actual_chunk_count=len(rows), + ) + + if chunk_indices_meta != expected_indices: + raw_group_bad += 1 + add_failure( + failures, + "raw_metadata_chunk_index_mismatch", + cell_id=cell_id, + record_kind=kind, + metadata_indices=chunk_indices_meta[:50], + ) + + if len(total_bytes_set) != 1: + raw_group_bad += 1 + add_failure( + failures, + "raw_total_bytes_inconsistent", + cell_id=cell_id, + record_kind=kind, + values=list(total_bytes_set), + ) + else: + expected_total = next(iter(total_bytes_set)) + try: + expected_total = int(expected_total) + except Exception: + expected_total = -1 + + if expected_total != actual_bytes: + raw_group_bad += 1 + add_failure( + failures, + "raw_total_bytes_mismatch", + cell_id=cell_id, + record_kind=kind, + expected=expected_total, + actual=actual_bytes, + ) + + # 每个 Cell 至少应有 HEADER 和 FEATURE 原始表。 + raw_required_missing = int(scalar( + cur, + """ + SELECT COUNT(*) + FROM ( + SELECT c.id + FROM chart_cell c + LEFT JOIN chart_raw_record h + ON h.cell_id=c.id AND h.record_kind='HEADER' + LEFT JOIN chart_raw_record f + ON f.cell_id=c.id AND f.record_kind='FEATURE' + WHERE c.dataset_id=%s + GROUP BY c.id + HAVING COUNT(DISTINCT h.record_kind)=0 + OR COUNT(DISTINCT f.record_kind)=0 + ) x + """, + (args.dataset_id,), + ) or 0) + + report["raw_checks"] = { + "raw_metadata_json_bad": raw_json_bad, + "raw_empty_blob": raw_empty_blob, + "raw_group_bad": raw_group_bad, + "required_header_or_feature_missing_cells": raw_required_missing, + "raw_group_count": len(raw_groups), + } + + if raw_required_missing: + add_failure( + failures, + "raw_required_kind_missing", + expected=0, + actual=raw_required_missing, + ) + + print("[10/10] 可选源文件 SHA256 + 最终判定 ...") + + source_checks = { + "enabled": source_root is not None, + "files_checked": 0, + "files_missing": 0, + "size_mismatch": 0, + "sha256_mismatch": 0, + } + + if source_root is not None: + if not source_root.is_dir(): + add_failure( + failures, + "source_root_not_found", + path=str(source_root), + ) + else: + cells = fetchall( + cur, + """ + SELECT + id, + source_relative_path, + source_size, + source_sha256 + FROM chart_cell + WHERE dataset_id=%s + ORDER BY id + """, + (args.dataset_id,), + ) + + for cell in cells: + rel = str(cell["source_relative_path"]).replace("\\", "/") + path = source_root / Path(rel) + source_checks["files_checked"] += 1 + + if not path.is_file(): + source_checks["files_missing"] += 1 + add_failure( + failures, + "source_file_missing", + cell_id=cell["id"], + path=str(path), + ) + continue + + actual_size = path.stat().st_size + expected_size = int(cell["source_size"]) + + if actual_size != expected_size: + source_checks["size_mismatch"] += 1 + add_failure( + failures, + "source_size_mismatch", + cell_id=cell["id"], + path=str(path), + expected=expected_size, + actual=actual_size, + ) + + expected_hash = str(cell["source_sha256"] or "").lower() + actual_hash = file_sha256(path).lower() + + if expected_hash != actual_hash: + source_checks["sha256_mismatch"] += 1 + add_failure( + failures, + "source_sha256_mismatch", + cell_id=cell["id"], + path=str(path), + expected=expected_hash, + actual=actual_hash, + ) + + report["source_checks"] = source_checks + + conn.rollback() + + finally: + conn.close() + + report["warnings"] = warnings + report["failure_count"] = len(failures) + report["overall"] = "PASS" if not failures else "FAIL" + + write_csv(outdir / "失败明细_v1.csv", failures) + write_csv(outdir / "Cell计数不一致_v1.csv", cell_mismatch_rows) + + (outdir / "最终验收报告_v1.json").write_text( + json.dumps( + report, + ensure_ascii=False, + indent=2, + default=str, + ), + encoding="utf-8", + ) + + c = report["counts"] + lines = [ + "CM93 -> MySQL 非 Geometry 数据完整性全库最终验收 v1", + "=" * 82, + f"Dataset ID : {args.dataset_id}", + "", + f"Cells : {c['cells']}", + f"Features : {c['features']}", + f"Attributes : {c['attributes']}", + f"Soundings : {c['soundings']}", + f"Coverages : {c['coverages']}", + f"Raw records : {c['raw_records']}", + "", + f"Cell count mismatch : {report['cell_count_mismatch']}", + f"Feature check failures : {sum(report['feature_checks'].values())}", + f"Dictionary failures : {sum(report['dictionary_checks'].values())}", + f"Attribute failures : {sum(report['attribute_checks'].values())}", + f"Coverage failures : {sum(report['coverage_checks'].values())}", + f"Sounding failures : {sum(report['sounding_checks'].values())}", + f"JSON/relation failures : {sum(report['json_relation_checks'].values())}", + f"Raw failures : " + f"{report['raw_checks']['raw_metadata_json_bad'] + report['raw_checks']['raw_empty_blob'] + report['raw_checks']['raw_group_bad'] + report['raw_checks']['required_header_or_feature_missing_cells']}", + "", + ] + + if source_root is not None: + s = report["source_checks"] + lines += [ + f"Source files checked : {s['files_checked']}", + f"Source files missing : {s['files_missing']}", + f"Source size mismatch : {s['size_mismatch']}", + f"Source SHA256 mismatch : {s['sha256_mismatch']}", + "", + ] + + lines += [ + f"Failure count : {report['failure_count']}", + f"OVERALL: {report['overall']}", + ] + + summary = "\n".join(lines) + + (outdir / "最终验收摘要_v1.txt").write_text( + summary, + encoding="utf-8", + ) + + zip_path = outdir.with_suffix(".zip") + with zipfile.ZipFile( + zip_path, + "w", + compression=zipfile.ZIP_DEFLATED, + ) as z: + for p in sorted(outdir.rglob("*")): + if p.is_file(): + z.write(p, p.relative_to(outdir.parent)) + + print("") + print(summary) + print("") + print(f"报告目录: {outdir}") + print(f"报告 ZIP : {zip_path}") + print("把 ZIP 上传给 ChatGPT。") + + +if __name__ == "__main__": + main() diff --git a/tools/update_manifest_sha256.ps1 b/tools/update_manifest_sha256.ps1 new file mode 100644 index 0000000..b062722 --- /dev/null +++ b/tools/update_manifest_sha256.ps1 @@ -0,0 +1,31 @@ +# CM95 tracked-file SHA256 manifest generator +# 在仓库根目录执行: +# powershell -ExecutionPolicy Bypass -File .\tools\update_manifest_sha256.ps1 + +$ErrorActionPreference = "Stop" + +$root = (Get-Location).Path +$out = Join-Path $root "MANIFEST.sha256" + +$files = git ls-files +if ($LASTEXITCODE -ne 0) { + throw "git ls-files failed" +} + +$lines = foreach ($rel in $files) { + if ($rel -eq "MANIFEST.sha256") { + continue + } + + $path = Join-Path $root $rel + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + continue + } + + $hash = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash.ToLowerInvariant() + "$hash $($rel -replace '\\','/')" +} + +$lines | Set-Content -LiteralPath $out -Encoding UTF8 +Write-Host "Updated: $out" +Write-Host "Files: $($lines.Count)"