feat: add WeatherV2 wind canonical converter

This commit is contained in:
OpenAI Codex
2026-09-04 21:08:07 +08:00
parent 5fc922785d
commit 79d4aa6385
17 changed files with 965 additions and 0 deletions

View File

@@ -0,0 +1,58 @@
# STEP-054 真实 WeatherV2 风数据提取与 Canonical 测试数据生成
- 状态:`IMPLEMENTED / WAITING_NEXT_ANDROID_INTEGRATION_VALIDATION`
- Starting HEAD:`5fc922785d8ec007127d48cd3ab5dadce270b947`
- 分支:`particle/main`
- 目标:将 WeatherV2 ECMWF HRES 的真实 `WVECBIN1` 风场无损转换为已冻结的 `maplibre-particle-field/v1` Android 测试 asset;不实现网络、Provider、React Native Binding 或 Android 综合绘制。
## 实际数据与格式依据
读取 WeatherV2 源目录:
`/root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600`
按既定时次生成 T0~T3:`2026080700`、`2026080706`、`2026080712`、`2026080718`。四帧均实际存在,均为日本及周边的完整区域 grid:`513×769`、bounds `112.5, 0.0, 157.5, 55.77657301866769`,单源文件 `5,128,989` bytes。
WeatherV2 `src/dev/core/bin_export.py` 的 `_export_vector_target()` 先以 `decode_wind()` 从原 Windy `wm_grid_257` tile 解码 U/V,再按 `WVECBIN1 + uint32-le header length + JSON header + U float32-le plane + V float32-le plane + speed float32-le plane + valid uint8 plane` 写入 `wind_vectors.bin`。数组 row-major;Web Mercator tile y 向南增加,因此 row 是 north-to-south、column 是 west-to-east。
`src/dev/layers/wind.py` 明确读取 `decoded_r` 为 U、`decoded_g` 为 V;writer metadata 说明 U eastward-positive、V northward-positive、screen y 使用 `-v`。QBIN exporter 和 NSFT exporter 都重新使用同一 `decode_wind()` Field:QBIN `u_q/v_q` 的 quantization metadata 明确为 `m/s`(`scale=0.01`),NSFT wind 也从该 Field 量化。因此本次原始 U/V 单位已由 writer/下游 decoder 确认为 `m/s`,没有 kt 换算、scale/offset 还原或额外量化。
`wind_vectors.bin` 是 WeatherV2 pipeline 对完整已拼接 U/V grid 的 particle/vector export;QBIN 是同源单 tile量化分发格式,NSFT 是同源单 tile量化 PMTiles 格式。三者共享 decoded U/V、valid 和 grid/bounds 语义,但本任务直接读取完整 WVECBIN1,不经 QBIN/NSFT 拼瓦片。
## 转换工具与输出
新增独立工具:
- `tools/weatherv2_wind_canonical/convert_weatherv2_wind.py`
- `tools/weatherv2_wind_canonical/README.md`
工具参数为 `--run-dir`、`--valid-time`、`--output`;源文件仅只读。它严格校验 magic、embedded header 与 sidecar 一致性、fields/dtype、长度、bounds 和有效 U/V,再输出既有 `maplibre-particle-field/v1` metadata 与 `float32-le interleaved-uv` payload。输出 `validTime` 是真实 UTC epoch milliseconds;`generation` 和 `dataVersion` 同为 producer-owned epoch milliseconds,四帧严格递增。
输出目录:
`platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/`
其中 `dataset.json` 只是开发期四帧索引,不是 Weather Manifest,也不进入 Snapshot。每个 validTime 子目录包含 `metadata.json`、`field.f32`、`conversion_summary.json`。
未重采样、未裁剪;每个 canonical payload 保留原完整 `513×769` 范围,`3,155,976` bytes。有效 U/V 逐 float32 bit-exact 复制;source `valid=0` 或非有限 U/V 会映射为既有 `noData=-9999` 的 U/V pair,绝不写成 `(0,0)`。当前四个实际 frame 均为 `394,497` valid、`0` invalid,但转换器仍保留该 fail-closed mapping。
## 生成结果与验证
| validTime | SHA-256 | U min/max (m/s) | V min/max (m/s) | speed min/max (m/s) |
| --- | --- | --- | --- | --- |
| 2026080700 | `363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283` | -35.1724 / 22.5011 | -25.4723 / 28.2009 | 0.0037 / 35.2783 |
| 2026080706 | `ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451` | -37.1826 / 24.1255 | -28.2276 / 25.0581 | 0.0105 / 37.7632 |
| 2026080712 | `d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0` | -37.3931 / 24.7693 | -28.2666 / 27.8099 | 0.0244 / 37.4291 |
| 2026080718 | `970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b` | -31.9560 / 28.5575 | -26.1394 / 27.1316 | 0.0210 / 33.6781 |
- 四帧 metadata 可解析、payload length 与 `width×height×8` 一致,validTime 严格递增,四个 SHA-256 均不同:PASS。
- 每帧中心、四角附近和内部有效样本 source→canonical U/V 的最大绝对误差均为 `0.0`:PASS。
- T0 重复转换的 SHA-256 与已生成 payload 相同:PASS。
- 转换器 Python syntax、正常转换和无效 source 的非零失败路径:PASS。
- 既有 Core canonical/Adapter/Host ingress 定向测试:5/5 PASS。
## 冻结边界与未执行项
未修改 `ParticleFieldDescriptor`、`ParticleFieldSnapshot`、`ParticleFieldDataAdapter`、Timeline、Playback、Runtime Field Window、Renderer、Android Kotlin/JNI 或 WeatherV2 原始数据。未将 Weather Service、HTTP、Provider、QBIN/NSFT 网络、缓存或目录扫描加入 MapLibre Runtime。
未执行 Android Gradle、APK、ADB、设备/模拟器测试或 Jest。下一阶段才使用本 asset 做 MapLibre map + WeatherColorLayer + RasterParticleLayer 综合 Android 验收;本 Step 不提前进入该范围。

View File

@@ -0,0 +1,124 @@
{
"source": {
"format": "weather-v2-particle-bin-v1",
"magic": "WVECBIN1",
"targetTime": "2026080700",
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
"uVUnit": "m/s",
"validCount": 394497,
"invalidCount": 0,
"bytes": 5128989
},
"output": {
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"bytes": 3155976,
"sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283",
"validCount": 394497,
"invalidCount": 0,
"resampled": false
},
"statistics": {
"u": {
"min": -35.172367095947266,
"max": 22.501087188720703,
"mean": 0.7285217046737671
},
"v": {
"min": -25.472291946411133,
"max": 28.200902938842773,
"mean": 1.1300384998321533
},
"speed": {
"min": 0.0036890381015837193,
"max": 35.27834701538086,
"mean": 6.401810169219971
}
},
"sourceToCanonical": {
"lossless": true,
"maxAbsErrorU": 0.0,
"maxAbsErrorV": 0.0,
"samples": [
{
"row": 384,
"column": 256,
"sourceU": -10.921929359436035,
"sourceV": 4.515348434448242,
"canonicalU": -10.921929359436035,
"canonicalV": 4.515348434448242
},
{
"row": 1,
"column": 1,
"sourceU": -0.980097770690918,
"sourceV": 0.4984455108642578,
"canonicalU": -0.980097770690918,
"canonicalV": 0.4984455108642578
},
{
"row": 1,
"column": 511,
"sourceU": 1.2943105697631836,
"sourceV": -0.6254873275756836,
"canonicalU": 1.2943105697631836,
"canonicalV": -0.6254873275756836
},
{
"row": 767,
"column": 1,
"sourceU": -2.738156318664551,
"sourceV": 1.1959524154663086,
"canonicalU": -2.738156318664551,
"canonicalV": 1.1959524154663086
},
{
"row": 767,
"column": 511,
"sourceU": -5.0848894119262695,
"sourceV": -0.26341915130615234,
"canonicalU": -5.0848894119262695,
"canonicalV": -0.26341915130615234
},
{
"row": 0,
"column": 0,
"sourceU": -1.1939868927001953,
"sourceV": 1.7741403579711914,
"canonicalU": -1.1939868927001953,
"canonicalV": 1.7741403579711914
},
{
"row": 0,
"column": 1,
"sourceU": -1.051393985748291,
"sourceV": 1.4262237548828125,
"canonicalU": -1.051393985748291,
"canonicalV": 1.4262237548828125
},
{
"row": 0,
"column": 2,
"sourceU": -0.6949124336242676,
"sourceV": 0.9043483734130859,
"canonicalU": -0.6949124336242676,
"canonicalV": 0.9043483734130859
}
]
}
}

View File

@@ -0,0 +1,41 @@
{
"schema": "maplibre-particle-field/v1",
"fieldId": "ecmwf-hres-wind-2026080700",
"fieldType": "WIND",
"displayDomain": "GLOBAL",
"generation": 1786060800000,
"validTime": 1786060800000,
"dataVersion": 1786060800000,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"grid": {
"width": 513,
"height": 769
},
"vector": {
"layout": "interleaved-uv",
"encoding": "float32-le",
"noData": -9999.0,
"unit": "m/s",
"origin": "north-west",
"rowOrder": "north-to-south",
"columnOrder": "west-to-east",
"uPositive": "east",
"vPositive": "north"
},
"payload": {
"byteLength": 3155976,
"sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283"
},
"source": {
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
"referenceTime": "2026080600",
"targetTime": "2026080700",
"sourceBytes": 5128989,
"resampled": false
}
}

View File

@@ -0,0 +1,124 @@
{
"source": {
"format": "weather-v2-particle-bin-v1",
"magic": "WVECBIN1",
"targetTime": "2026080706",
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
"uVUnit": "m/s",
"validCount": 394497,
"invalidCount": 0,
"bytes": 5128989
},
"output": {
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"bytes": 3155976,
"sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451",
"validCount": 394497,
"invalidCount": 0,
"resampled": false
},
"statistics": {
"u": {
"min": -37.18255615234375,
"max": 24.125526428222656,
"mean": 0.9443698525428772
},
"v": {
"min": -28.227636337280273,
"max": 25.05806541442871,
"mean": 1.0168105363845825
},
"speed": {
"min": 0.010480343364179134,
"max": 37.76322555541992,
"mean": 6.7594804763793945
}
},
"sourceToCanonical": {
"lossless": true,
"maxAbsErrorU": 0.0,
"maxAbsErrorV": 0.0,
"samples": [
{
"row": 384,
"column": 256,
"sourceU": -10.2057466506958,
"sourceV": 4.789729118347168,
"canonicalU": -10.2057466506958,
"canonicalV": 4.789729118347168
},
{
"row": 1,
"column": 1,
"sourceU": -0.45672082901000977,
"sourceV": 3.2181167602539062,
"canonicalU": -0.45672082901000977,
"canonicalV": 3.2181167602539062
},
{
"row": 1,
"column": 511,
"sourceU": 2.235017776489258,
"sourceV": -0.7363739013671875,
"canonicalU": 2.235017776489258,
"canonicalV": -0.7363739013671875
},
{
"row": 767,
"column": 1,
"sourceU": -1.2302308082580566,
"sourceV": 1.5909242630004883,
"canonicalU": -1.2302308082580566,
"canonicalV": 1.5909242630004883
},
{
"row": 767,
"column": 511,
"sourceU": -5.034683704376221,
"sourceV": -0.15012121200561523,
"canonicalU": -5.034683704376221,
"canonicalV": -0.15012121200561523
},
{
"row": 0,
"column": 0,
"sourceU": -0.15583133697509766,
"sourceV": 3.617917060852051,
"canonicalU": -0.15583133697509766,
"canonicalV": 3.617917060852051
},
{
"row": 0,
"column": 1,
"sourceU": -0.45672082901000977,
"sourceV": 3.551283836364746,
"canonicalU": -0.45672082901000977,
"canonicalV": 3.551283836364746
},
{
"row": 0,
"column": 2,
"sourceU": -0.6974325180053711,
"sourceV": 3.2181167602539062,
"canonicalU": -0.6974325180053711,
"canonicalV": 3.2181167602539062
}
]
}
}

View File

@@ -0,0 +1,41 @@
{
"schema": "maplibre-particle-field/v1",
"fieldId": "ecmwf-hres-wind-2026080706",
"fieldType": "WIND",
"displayDomain": "GLOBAL",
"generation": 1786082400000,
"validTime": 1786082400000,
"dataVersion": 1786082400000,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"grid": {
"width": 513,
"height": 769
},
"vector": {
"layout": "interleaved-uv",
"encoding": "float32-le",
"noData": -9999.0,
"unit": "m/s",
"origin": "north-west",
"rowOrder": "north-to-south",
"columnOrder": "west-to-east",
"uPositive": "east",
"vPositive": "north"
},
"payload": {
"byteLength": 3155976,
"sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451"
},
"source": {
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
"referenceTime": "2026080600",
"targetTime": "2026080706",
"sourceBytes": 5128989,
"resampled": false
}
}

View File

@@ -0,0 +1,124 @@
{
"source": {
"format": "weather-v2-particle-bin-v1",
"magic": "WVECBIN1",
"targetTime": "2026080712",
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
"uVUnit": "m/s",
"validCount": 394497,
"invalidCount": 0,
"bytes": 5128989
},
"output": {
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"bytes": 3155976,
"sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0",
"validCount": 394497,
"invalidCount": 0,
"resampled": false
},
"statistics": {
"u": {
"min": -37.39310836791992,
"max": 24.769317626953125,
"mean": 0.7698453664779663
},
"v": {
"min": -28.266582489013672,
"max": 27.809932708740234,
"mean": 1.1808899641036987
},
"speed": {
"min": 0.024363698437809944,
"max": 37.42909240722656,
"mean": 6.384326934814453
}
},
"sourceToCanonical": {
"lossless": true,
"maxAbsErrorU": 0.0,
"maxAbsErrorV": 0.0,
"samples": [
{
"row": 384,
"column": 256,
"sourceU": -10.180034637451172,
"sourceV": 4.663511276245117,
"canonicalU": -10.180034637451172,
"canonicalV": 4.663511276245117
},
{
"row": 1,
"column": 1,
"sourceU": -0.5417494773864746,
"sourceV": 0.767613410949707,
"canonicalU": -0.5417494773864746,
"canonicalV": 0.767613410949707
},
{
"row": 1,
"column": 511,
"sourceU": -0.5780353546142578,
"sourceV": 0.26222801208496094,
"canonicalU": -0.5780353546142578,
"canonicalV": 0.26222801208496094
},
{
"row": 767,
"column": 1,
"sourceU": -1.8420743942260742,
"sourceV": 0.3505821228027344,
"canonicalU": -1.8420743942260742,
"canonicalV": 0.3505821228027344
},
{
"row": 767,
"column": 511,
"sourceU": -4.581581115722656,
"sourceV": -1.1498479843139648,
"canonicalU": -4.581581115722656,
"canonicalV": -1.1498479843139648
},
{
"row": 0,
"column": 0,
"sourceU": -0.5939555168151855,
"sourceV": 1.1400480270385742,
"canonicalU": -0.5939555168151855,
"canonicalV": 1.1400480270385742
},
{
"row": 0,
"column": 1,
"sourceU": -0.48954296112060547,
"sourceV": 1.0159034729003906,
"canonicalU": -0.48954296112060547,
"canonicalV": 1.0159034729003906
},
{
"row": 0,
"column": 2,
"sourceU": -0.48954296112060547,
"sourceV": 0.767613410949707,
"canonicalU": -0.48954296112060547,
"canonicalV": 0.767613410949707
}
]
}
}

View File

@@ -0,0 +1,41 @@
{
"schema": "maplibre-particle-field/v1",
"fieldId": "ecmwf-hres-wind-2026080712",
"fieldType": "WIND",
"displayDomain": "GLOBAL",
"generation": 1786104000000,
"validTime": 1786104000000,
"dataVersion": 1786104000000,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"grid": {
"width": 513,
"height": 769
},
"vector": {
"layout": "interleaved-uv",
"encoding": "float32-le",
"noData": -9999.0,
"unit": "m/s",
"origin": "north-west",
"rowOrder": "north-to-south",
"columnOrder": "west-to-east",
"uPositive": "east",
"vPositive": "north"
},
"payload": {
"byteLength": 3155976,
"sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0"
},
"source": {
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
"referenceTime": "2026080600",
"targetTime": "2026080712",
"sourceBytes": 5128989,
"resampled": false
}
}

View File

@@ -0,0 +1,124 @@
{
"source": {
"format": "weather-v2-particle-bin-v1",
"magic": "WVECBIN1",
"targetTime": "2026080718",
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
"uVUnit": "m/s",
"validCount": 394497,
"invalidCount": 0,
"bytes": 5128989
},
"output": {
"width": 513,
"height": 769,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"bytes": 3155976,
"sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b",
"validCount": 394497,
"invalidCount": 0,
"resampled": false
},
"statistics": {
"u": {
"min": -31.956016540527344,
"max": 28.557456970214844,
"mean": 1.212062120437622
},
"v": {
"min": -26.13941192626953,
"max": 27.131576538085938,
"mean": 1.1284888982772827
},
"speed": {
"min": 0.0210099034011364,
"max": 33.67807388305664,
"mean": 6.403453826904297
}
},
"sourceToCanonical": {
"lossless": true,
"maxAbsErrorU": 0.0,
"maxAbsErrorV": 0.0,
"samples": [
{
"row": 384,
"column": 256,
"sourceU": -10.21538257598877,
"sourceV": 2.3842287063598633,
"canonicalU": -10.21538257598877,
"canonicalV": 2.3842287063598633
},
{
"row": 1,
"column": 1,
"sourceU": -0.18732452392578125,
"sourceV": 0.2720670700073242,
"canonicalU": -0.18732452392578125,
"canonicalV": 0.2720670700073242
},
{
"row": 1,
"column": 511,
"sourceU": 0.460113525390625,
"sourceV": 0.4729623794555664,
"canonicalU": 0.460113525390625,
"canonicalV": 0.4729623794555664
},
{
"row": 767,
"column": 1,
"sourceU": -1.4576325416564941,
"sourceV": -0.03371381759643555,
"canonicalU": -1.4576325416564941,
"canonicalV": -0.03371381759643555
},
{
"row": 767,
"column": 511,
"sourceU": -2.9916043281555176,
"sourceV": 0.8597030639648438,
"canonicalU": -2.9916043281555176,
"canonicalV": 0.8597030639648438
},
{
"row": 0,
"column": 0,
"sourceU": -1.020096778869629,
"sourceV": 0.9125776290893555,
"canonicalU": -1.020096778869629,
"canonicalV": 0.9125776290893555
},
{
"row": 0,
"column": 1,
"sourceU": -0.49961376190185547,
"sourceV": 0.6990737915039062,
"canonicalU": -0.49961376190185547,
"canonicalV": 0.6990737915039062
},
{
"row": 0,
"column": 2,
"sourceU": -0.08322811126708984,
"sourceV": 0.41440296173095703,
"canonicalU": -0.08322811126708984,
"canonicalV": 0.41440296173095703
}
]
}
}

View File

@@ -0,0 +1,41 @@
{
"schema": "maplibre-particle-field/v1",
"fieldId": "ecmwf-hres-wind-2026080718",
"fieldType": "WIND",
"displayDomain": "GLOBAL",
"generation": 1786125600000,
"validTime": 1786125600000,
"dataVersion": 1786125600000,
"bounds": {
"west": 112.5,
"south": 0.0,
"east": 157.5,
"north": 55.77657301866769
},
"grid": {
"width": 513,
"height": 769
},
"vector": {
"layout": "interleaved-uv",
"encoding": "float32-le",
"noData": -9999.0,
"unit": "m/s",
"origin": "north-west",
"rowOrder": "north-to-south",
"columnOrder": "west-to-east",
"uPositive": "east",
"vPositive": "north"
},
"payload": {
"byteLength": 3155976,
"sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b"
},
"source": {
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
"referenceTime": "2026080600",
"targetTime": "2026080718",
"sourceBytes": 5128989,
"resampled": false
}
}

View File

@@ -0,0 +1,14 @@
{
"schema": "maplibre-particle-test-dataset/v1",
"datasetId": "ecmwf-hres-wind-20260807",
"source": "WeatherV2 ECMWF HRES WVECBIN1",
"referenceTime": "2026080600",
"validTimes": [
"2026080700",
"2026080706",
"2026080712",
"2026080718"
],
"snapshotLayout": "<validTime>/metadata.json + <validTime>/field.f32",
"notes": "开发期离线 canonical 测试索引;不是 Weather Manifest,也不进入 ParticleFieldSnapshot。"
}

View File

@@ -0,0 +1,37 @@
# WeatherV2 风场转 MapLibre canonical 测试数据工具
此工具只用于开发期离线转换:将 WeatherV2 的完整 `wind_vectors.bin`(`WVECBIN1`)转成 STEP-053 已冻结的 `maplibre-particle-field/v1`,不属于 MapLibre Runtime,也不访问网络、不扫描下载目录、更不修改 WeatherV2 原始数据。
## 输入
输入为一个 WeatherV2 reference/run 目录及其中一个 `YYYYMMDDHH` valid time:
```text
<run-dir>/<valid-time>/wind_vectors.bin
<run-dir>/<valid-time>/wind_vectors.bin.header.json
```
当前已核对的 `WVECBIN1` 布局为:`magic(8)`、little-endian `uint32 headerLength`、UTF-8 JSON header、row-major `u float32-le` 平面、`v float32-le` 平面、`speed float32-le` 平面、`valid uint8` 平面。WeatherV2 `bin_export.py` 将同一 `decode_wind()` U/V grid 写出;QBIN 和 NSFT 也从同一 decoder U/V 生成。QBIN metadata 明确其 U/V 单位为 m/s,因此本工具保持 m/s,不进行 kt 换算。
源 grid 的 y 来自 Web Mercator tile y,向南增加;因此输出的 row-major rows 是 north-to-south,columns 是 west-to-east。U=eastward-positive,V=northward-positive。
## 输出
每个输出目录包含:
- `metadata.json`:现有 `LocalParticleFieldFeed` 读取的 `maplibre-particle-field/v1` metadata。
- `field.f32`:`float32-le interleaved-uv`,顺序为 `U0,V0,U1,V1,...`。
- `conversion_summary.json`:源/输出 bounds、尺寸、统计值、valid/noData 计数、抽样一致性与 SHA-256。
本工具不重采样。每个源 `valid=0` 或非有限 U/V 的 cell 会写为既有 canonical `noData=-9999.0` 的 U/V 对;不会错误变成静风 `(0,0)`。有效样本按 float32 bit-exact 复制。
## 命令示例
```bash
python3 tools/weatherv2_wind_canonical/convert_weatherv2_wind.py \
--run-dir /root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600 \
--valid-time 2026080700 \
--output platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700
```
相同输入会产生相同 payload SHA-256。错误会输出明确消息并以非零状态退出。

View File

@@ -0,0 +1,190 @@
#!/usr/bin/env python3
"""Convert one WeatherV2 WVECBIN1 wind snapshot to maplibre-particle-field/v1."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import struct
import sys
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
MAGIC = b"WVECBIN1"
NO_DATA = np.float32(-9999.0)
def fail(message: str) -> "None":
raise ValueError(message)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run-dir", required=True, type=Path, help="WeatherV2 reference-time directory")
parser.add_argument("--valid-time", required=True, help="target time in YYYYMMDDHH UTC")
parser.add_argument("--output", required=True, type=Path, help="canonical snapshot output directory")
return parser.parse_args()
def require_mapping(value: object, name: str) -> dict:
if not isinstance(value, dict):
fail(f"{name} must be an object")
return value
def read_source(run_dir: Path, valid_time: str) -> tuple[dict, np.ndarray, np.ndarray, np.ndarray, np.ndarray, int]:
source_dir = run_dir / valid_time
payload_path = source_dir / "wind_vectors.bin"
sidecar_path = source_dir / "wind_vectors.bin.header.json"
if not payload_path.is_file() or not sidecar_path.is_file():
fail(f"missing WeatherV2 snapshot for {valid_time}: {source_dir}")
data = payload_path.read_bytes()
if len(data) < 12 or data[:8] != MAGIC:
fail(f"unsupported wind_vectors.bin magic in {payload_path}")
header_length = struct.unpack_from("<I", data, 8)[0]
header_end = 12 + header_length
if header_end > len(data):
fail("WVECBIN1 header length exceeds file length")
header = json.loads(data[12:header_end].decode("utf-8"))
sidecar = json.loads(sidecar_path.read_text(encoding="utf-8"))
if header != sidecar:
fail("embedded WVECBIN1 header differs from .header.json sidecar")
if header.get("format") != "weather-v2-particle-bin-v1" or header.get("layer") != "wind":
fail("source is not a WeatherV2 wind particle-bin-v1 snapshot")
if header.get("target_time") != valid_time:
fail(f"source target_time mismatch: {header.get('target_time')!r} != {valid_time!r}")
width, height = int(header.get("width", 0)), int(header.get("height", 0))
if width <= 0 or height <= 0:
fail("source width/height must be positive")
expected_fields = [
{"name": "u", "dtype": "float32"},
{"name": "v", "dtype": "float32"},
{"name": "speed", "dtype": "float32"},
{"name": "valid", "dtype": "uint8"},
]
if header.get("fields") != expected_fields:
fail(f"unsupported WVECBIN1 fields: {header.get('fields')!r}")
count = width * height
expected_bytes = header_end + count * 4 * 3 + count
if len(data) != expected_bytes:
fail(f"WVECBIN1 byte length mismatch: expected {expected_bytes}, got {len(data)}")
offset = header_end
u = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
offset += count * 4
v = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
offset += count * 4
speed = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
offset += count * 4
valid = np.frombuffer(data, dtype=np.uint8, count=count, offset=offset).copy().reshape(height, width).astype(bool)
bounds = require_mapping(header.get("bounds"), "source bounds")
if not all(math.isfinite(float(bounds[name])) for name in ("west", "south", "east", "north")):
fail("source bounds must be finite")
if not float(bounds["west"]) < float(bounds["east"]) or not float(bounds["south"]) < float(bounds["north"]):
fail("source bounds are invalid")
if not np.isfinite(u[valid]).all() or not np.isfinite(v[valid]).all():
fail("valid WeatherV2 samples contain non-finite U/V")
if not np.isfinite(speed[valid]).all():
fail("valid WeatherV2 samples contain non-finite speed")
return header, u, v, speed, valid, len(data)
def epoch_millis(valid_time: str) -> int:
try:
return int(datetime.strptime(valid_time, "%Y%m%d%H").replace(tzinfo=timezone.utc).timestamp() * 1000)
except ValueError as error:
fail(f"valid time must be YYYYMMDDHH UTC: {error}")
def stats(values: np.ndarray, valid: np.ndarray) -> dict:
chosen = values[valid]
if not chosen.size:
fail("source has no valid samples")
return {"min": float(chosen.min()), "max": float(chosen.max()), "mean": float(chosen.mean())}
def sample_indices(valid: np.ndarray) -> list[tuple[int, int]]:
height, width = valid.shape
candidates = [(height // 2, width // 2), (1, 1), (1, width - 2), (height - 2, 1), (height - 2, width - 2)]
selected: list[tuple[int, int]] = []
for row, col in candidates:
if valid[row, col] and (row, col) not in selected:
selected.append((row, col))
for row, col in zip(*np.nonzero(valid)):
if (int(row), int(col)) not in selected:
selected.append((int(row), int(col)))
if len(selected) == 8:
break
return selected
def write_output(header: dict,
u: np.ndarray,
v: np.ndarray,
speed: np.ndarray,
valid: np.ndarray,
source_bytes: int,
reference_time: str,
output: Path) -> dict:
width, height = int(header["width"]), int(header["height"])
valid = valid & np.isfinite(u) & np.isfinite(v)
payload = np.empty((height, width, 2), dtype="<f4")
payload[:, :, 0] = np.where(valid, u, NO_DATA)
payload[:, :, 1] = np.where(valid, v, NO_DATA)
payload_bytes = payload.tobytes(order="C")
payload_hash = hashlib.sha256(payload_bytes).hexdigest()
decoded = np.frombuffer(payload_bytes, dtype="<f4").reshape(height, width, 2)
if not np.array_equal(decoded[:, :, 0][valid], u[valid]) or not np.array_equal(decoded[:, :, 1][valid], v[valid]):
fail("source-to-canonical U/V conversion is not lossless")
if not np.all(decoded[:, :, 0][~valid] == NO_DATA) or not np.all(decoded[:, :, 1][~valid] == NO_DATA):
fail("invalid samples were not preserved as canonical noData")
bounds = header["bounds"]
timestamp = epoch_millis(str(header["target_time"]))
metadata = {
"schema": "maplibre-particle-field/v1",
"fieldId": f"ecmwf-hres-wind-{header['target_time']}",
"fieldType": "WIND",
"displayDomain": "GLOBAL",
"generation": timestamp,
"validTime": timestamp,
"dataVersion": timestamp,
"bounds": {name: float(bounds[name]) for name in ("west", "south", "east", "north")},
"grid": {"width": width, "height": height},
"vector": {"layout": "interleaved-uv", "encoding": "float32-le", "noData": float(NO_DATA), "unit": "m/s", "origin": "north-west", "rowOrder": "north-to-south", "columnOrder": "west-to-east", "uPositive": "east", "vPositive": "north"},
"payload": {"byteLength": len(payload_bytes), "sha256": payload_hash},
"source": {"kind": "WeatherV2 ECMWF HRES WVECBIN1", "referenceTime": reference_time, "targetTime": header["target_time"], "sourceBytes": source_bytes, "resampled": False},
}
samples = []
for row, col in sample_indices(valid):
samples.append({"row": row, "column": col, "sourceU": float(u[row, col]), "sourceV": float(v[row, col]), "canonicalU": float(decoded[row, col, 0]), "canonicalV": float(decoded[row, col, 1])})
summary = {
"source": {"format": header["format"], "magic": header["magic"], "targetTime": header["target_time"], "width": width, "height": height, "bounds": metadata["bounds"], "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", "uVUnit": "m/s", "validCount": int(valid.sum()), "invalidCount": int(valid.size - valid.sum()), "bytes": source_bytes},
"output": {"width": width, "height": height, "bounds": metadata["bounds"], "bytes": len(payload_bytes), "sha256": payload_hash, "validCount": int(valid.sum()), "invalidCount": int(valid.size - valid.sum()), "resampled": False},
"statistics": {"u": stats(u, valid), "v": stats(v, valid), "speed": stats(speed, valid)},
"sourceToCanonical": {"lossless": True, "maxAbsErrorU": 0.0, "maxAbsErrorV": 0.0, "samples": samples},
}
output.mkdir(parents=True, exist_ok=True)
(output / "field.f32").write_bytes(payload_bytes)
(output / "metadata.json").write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
(output / "conversion_summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
return summary
def main() -> int:
args = parse_args()
try:
header, u, v, speed, valid, source_bytes = read_source(args.run_dir, args.valid_time)
summary = write_output(header, u, v, speed, valid, source_bytes, args.run_dir.name, args.output)
print(json.dumps({"ok": True, "targetTime": header["target_time"], "output": str(args.output), "sha256": summary["output"]["sha256"], "validCount": summary["output"]["validCount"], "invalidCount": summary["output"]["invalidCount"]}, ensure_ascii=False))
return 0
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"error: {error}", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -398,3 +398,9 @@
- `fieldUploadCount=4 → 5 → 6` 证明正常 boundary 每次仅上传一个 NEW_T1。唯一 `simulationResetCount=1` 属于旧 fixture 切到 Host dataset 的允许初始化,不是正常 production boundary 回归。 - `fieldUploadCount=4 → 5 → 6` 证明正常 boundary 每次仅上传一个 NEW_T1。唯一 `simulationResetCount=1` 属于旧 fixture 切到 Host dataset 的允许初始化,不是正常 production boundary 回归。
- `HOST DUPLICATE` 返回 `DUPLICATE_NOOP` 且 upload/reset 不增长;`HOST INVALID` 经 Adapter reject fail-closed,last valid field 与 Renderer continuity(`stateSeed=CONTINUE`)保持,无黑屏、无 crash。 - `HOST DUPLICATE` 返回 `DUPLICATE_NOOP` 且 upload/reset 不增长;`HOST INVALID` 经 Adapter reject fail-closed,last valid field 与 Renderer continuity(`stateSeed=CONTINUE`)保持,无黑屏、无 crash。
- 用户正式风场数据集仍未提供、未验证,但不阻塞 contract 阶段关闭;后续可在同一 canonical contract 下追加。MIX 文本滞后及 WAITING/INVALID 视觉为 `NON-BLOCKING observability`,本次未修改功能代码或执行 Android/Jest。 - 用户正式风场数据集仍未提供、未验证,但不阻塞 contract 阶段关闭;后续可在同一 canonical contract 下追加。MIX 文本滞后及 WAITING/INVALID 视觉为 `NON-BLOCKING observability`,本次未修改功能代码或执行 Android/Jest。
## 2026-09-04 — STEP-054 真实 WeatherV2 风数据 Canonical 测试 asset
- 只读核对 WeatherV2 `bin_export.py`、`decode_wind()`、QBIN/NSFT export 后,确认 `wind_vectors.bin` 为完整拼接 `WVECBIN1` grid:row-major U/V/speed float32-le planes + valid uint8 plane,U=eastward、V=northward,单位 m/s;QBIN/NSFT 均由同一 decoded U/V Field 生成。
- 从 run `2026080600` 生成真实 ECMWF HRES `2026080700/06/12/18` 四帧。原始日本及周边 bounds 为 112.5/0.0/157.5/55.776573,`513×769`,不裁剪、不重采样;输出 canonical interleaved U/V Android assets,每帧 3,155,976 bytes。
- 转换器独立放在 `tools/weatherv2_wind_canonical/`,保留 valid/noData:invalid 写既有 -9999 sentinel,不伪造为 0 风;四帧 hash 不同,重复 T0 hash 相同,source→canonical 抽样误差 0。未改 Core/Renderer/Android,也未执行 Android/Jest;等待下一阶段集中 Android 集成验收。