feat: add WeatherV2 wind canonical converter
This commit is contained in:
@@ -0,0 +1,58 @@
|
|||||||
|
# STEP-054 真实 WeatherV2 风数据提取与 Canonical 测试数据生成
|
||||||
|
|
||||||
|
- 状态:`IMPLEMENTED / WAITING_NEXT_ANDROID_INTEGRATION_VALIDATION`
|
||||||
|
- Starting HEAD:`5fc922785d8ec007127d48cd3ab5dadce270b947`
|
||||||
|
- 分支:`particle/main`
|
||||||
|
- 目标:将 WeatherV2 ECMWF HRES 的真实 `WVECBIN1` 风场无损转换为已冻结的 `maplibre-particle-field/v1` Android 测试 asset;不实现网络、Provider、React Native Binding 或 Android 综合绘制。
|
||||||
|
|
||||||
|
## 实际数据与格式依据
|
||||||
|
|
||||||
|
读取 WeatherV2 源目录:
|
||||||
|
|
||||||
|
`/root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600`
|
||||||
|
|
||||||
|
按既定时次生成 T0~T3:`2026080700`、`2026080706`、`2026080712`、`2026080718`。四帧均实际存在,均为日本及周边的完整区域 grid:`513×769`、bounds `112.5, 0.0, 157.5, 55.77657301866769`,单源文件 `5,128,989` bytes。
|
||||||
|
|
||||||
|
WeatherV2 `src/dev/core/bin_export.py` 的 `_export_vector_target()` 先以 `decode_wind()` 从原 Windy `wm_grid_257` tile 解码 U/V,再按 `WVECBIN1 + uint32-le header length + JSON header + U float32-le plane + V float32-le plane + speed float32-le plane + valid uint8 plane` 写入 `wind_vectors.bin`。数组 row-major;Web Mercator tile y 向南增加,因此 row 是 north-to-south、column 是 west-to-east。
|
||||||
|
|
||||||
|
`src/dev/layers/wind.py` 明确读取 `decoded_r` 为 U、`decoded_g` 为 V;writer metadata 说明 U eastward-positive、V northward-positive、screen y 使用 `-v`。QBIN exporter 和 NSFT exporter 都重新使用同一 `decode_wind()` Field:QBIN `u_q/v_q` 的 quantization metadata 明确为 `m/s`(`scale=0.01`),NSFT wind 也从该 Field 量化。因此本次原始 U/V 单位已由 writer/下游 decoder 确认为 `m/s`,没有 kt 换算、scale/offset 还原或额外量化。
|
||||||
|
|
||||||
|
`wind_vectors.bin` 是 WeatherV2 pipeline 对完整已拼接 U/V grid 的 particle/vector export;QBIN 是同源单 tile量化分发格式,NSFT 是同源单 tile量化 PMTiles 格式。三者共享 decoded U/V、valid 和 grid/bounds 语义,但本任务直接读取完整 WVECBIN1,不经 QBIN/NSFT 拼瓦片。
|
||||||
|
|
||||||
|
## 转换工具与输出
|
||||||
|
|
||||||
|
新增独立工具:
|
||||||
|
|
||||||
|
- `tools/weatherv2_wind_canonical/convert_weatherv2_wind.py`
|
||||||
|
- `tools/weatherv2_wind_canonical/README.md`
|
||||||
|
|
||||||
|
工具参数为 `--run-dir`、`--valid-time`、`--output`;源文件仅只读。它严格校验 magic、embedded header 与 sidecar 一致性、fields/dtype、长度、bounds 和有效 U/V,再输出既有 `maplibre-particle-field/v1` metadata 与 `float32-le interleaved-uv` payload。输出 `validTime` 是真实 UTC epoch milliseconds;`generation` 和 `dataVersion` 同为 producer-owned epoch milliseconds,四帧严格递增。
|
||||||
|
|
||||||
|
输出目录:
|
||||||
|
|
||||||
|
`platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/`
|
||||||
|
|
||||||
|
其中 `dataset.json` 只是开发期四帧索引,不是 Weather Manifest,也不进入 Snapshot。每个 validTime 子目录包含 `metadata.json`、`field.f32`、`conversion_summary.json`。
|
||||||
|
|
||||||
|
未重采样、未裁剪;每个 canonical payload 保留原完整 `513×769` 范围,`3,155,976` bytes。有效 U/V 逐 float32 bit-exact 复制;source `valid=0` 或非有限 U/V 会映射为既有 `noData=-9999` 的 U/V pair,绝不写成 `(0,0)`。当前四个实际 frame 均为 `394,497` valid、`0` invalid,但转换器仍保留该 fail-closed mapping。
|
||||||
|
|
||||||
|
## 生成结果与验证
|
||||||
|
|
||||||
|
| validTime | SHA-256 | U min/max (m/s) | V min/max (m/s) | speed min/max (m/s) |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| 2026080700 | `363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283` | -35.1724 / 22.5011 | -25.4723 / 28.2009 | 0.0037 / 35.2783 |
|
||||||
|
| 2026080706 | `ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451` | -37.1826 / 24.1255 | -28.2276 / 25.0581 | 0.0105 / 37.7632 |
|
||||||
|
| 2026080712 | `d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0` | -37.3931 / 24.7693 | -28.2666 / 27.8099 | 0.0244 / 37.4291 |
|
||||||
|
| 2026080718 | `970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b` | -31.9560 / 28.5575 | -26.1394 / 27.1316 | 0.0210 / 33.6781 |
|
||||||
|
|
||||||
|
- 四帧 metadata 可解析、payload length 与 `width×height×8` 一致,validTime 严格递增,四个 SHA-256 均不同:PASS。
|
||||||
|
- 每帧中心、四角附近和内部有效样本 source→canonical U/V 的最大绝对误差均为 `0.0`:PASS。
|
||||||
|
- T0 重复转换的 SHA-256 与已生成 payload 相同:PASS。
|
||||||
|
- 转换器 Python syntax、正常转换和无效 source 的非零失败路径:PASS。
|
||||||
|
- 既有 Core canonical/Adapter/Host ingress 定向测试:5/5 PASS。
|
||||||
|
|
||||||
|
## 冻结边界与未执行项
|
||||||
|
|
||||||
|
未修改 `ParticleFieldDescriptor`、`ParticleFieldSnapshot`、`ParticleFieldDataAdapter`、Timeline、Playback、Runtime Field Window、Renderer、Android Kotlin/JNI 或 WeatherV2 原始数据。未将 Weather Service、HTTP、Provider、QBIN/NSFT 网络、缓存或目录扫描加入 MapLibre Runtime。
|
||||||
|
|
||||||
|
未执行 Android Gradle、APK、ADB、设备/模拟器测试或 Jest。下一阶段才使用本 asset 做 MapLibre map + WeatherColorLayer + RasterParticleLayer 综合 Android 验收;本 Step 不提前进入该范围。
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
{
|
||||||
|
"source": {
|
||||||
|
"format": "weather-v2-particle-bin-v1",
|
||||||
|
"magic": "WVECBIN1",
|
||||||
|
"targetTime": "2026080700",
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
|
||||||
|
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
|
||||||
|
"uVUnit": "m/s",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"bytes": 5128989
|
||||||
|
},
|
||||||
|
"output": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"bytes": 3155976,
|
||||||
|
"sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"resampled": false
|
||||||
|
},
|
||||||
|
"statistics": {
|
||||||
|
"u": {
|
||||||
|
"min": -35.172367095947266,
|
||||||
|
"max": 22.501087188720703,
|
||||||
|
"mean": 0.7285217046737671
|
||||||
|
},
|
||||||
|
"v": {
|
||||||
|
"min": -25.472291946411133,
|
||||||
|
"max": 28.200902938842773,
|
||||||
|
"mean": 1.1300384998321533
|
||||||
|
},
|
||||||
|
"speed": {
|
||||||
|
"min": 0.0036890381015837193,
|
||||||
|
"max": 35.27834701538086,
|
||||||
|
"mean": 6.401810169219971
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"sourceToCanonical": {
|
||||||
|
"lossless": true,
|
||||||
|
"maxAbsErrorU": 0.0,
|
||||||
|
"maxAbsErrorV": 0.0,
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"row": 384,
|
||||||
|
"column": 256,
|
||||||
|
"sourceU": -10.921929359436035,
|
||||||
|
"sourceV": 4.515348434448242,
|
||||||
|
"canonicalU": -10.921929359436035,
|
||||||
|
"canonicalV": 4.515348434448242
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.980097770690918,
|
||||||
|
"sourceV": 0.4984455108642578,
|
||||||
|
"canonicalU": -0.980097770690918,
|
||||||
|
"canonicalV": 0.4984455108642578
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": 1.2943105697631836,
|
||||||
|
"sourceV": -0.6254873275756836,
|
||||||
|
"canonicalU": 1.2943105697631836,
|
||||||
|
"canonicalV": -0.6254873275756836
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -2.738156318664551,
|
||||||
|
"sourceV": 1.1959524154663086,
|
||||||
|
"canonicalU": -2.738156318664551,
|
||||||
|
"canonicalV": 1.1959524154663086
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": -5.0848894119262695,
|
||||||
|
"sourceV": -0.26341915130615234,
|
||||||
|
"canonicalU": -5.0848894119262695,
|
||||||
|
"canonicalV": -0.26341915130615234
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 0,
|
||||||
|
"sourceU": -1.1939868927001953,
|
||||||
|
"sourceV": 1.7741403579711914,
|
||||||
|
"canonicalU": -1.1939868927001953,
|
||||||
|
"canonicalV": 1.7741403579711914
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -1.051393985748291,
|
||||||
|
"sourceV": 1.4262237548828125,
|
||||||
|
"canonicalU": -1.051393985748291,
|
||||||
|
"canonicalV": 1.4262237548828125
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 2,
|
||||||
|
"sourceU": -0.6949124336242676,
|
||||||
|
"sourceV": 0.9043483734130859,
|
||||||
|
"canonicalU": -0.6949124336242676,
|
||||||
|
"canonicalV": 0.9043483734130859
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Binary file not shown.
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"schema": "maplibre-particle-field/v1",
|
||||||
|
"fieldId": "ecmwf-hres-wind-2026080700",
|
||||||
|
"fieldType": "WIND",
|
||||||
|
"displayDomain": "GLOBAL",
|
||||||
|
"generation": 1786060800000,
|
||||||
|
"validTime": 1786060800000,
|
||||||
|
"dataVersion": 1786060800000,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"grid": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769
|
||||||
|
},
|
||||||
|
"vector": {
|
||||||
|
"layout": "interleaved-uv",
|
||||||
|
"encoding": "float32-le",
|
||||||
|
"noData": -9999.0,
|
||||||
|
"unit": "m/s",
|
||||||
|
"origin": "north-west",
|
||||||
|
"rowOrder": "north-to-south",
|
||||||
|
"columnOrder": "west-to-east",
|
||||||
|
"uPositive": "east",
|
||||||
|
"vPositive": "north"
|
||||||
|
},
|
||||||
|
"payload": {
|
||||||
|
"byteLength": 3155976,
|
||||||
|
"sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
|
||||||
|
"referenceTime": "2026080600",
|
||||||
|
"targetTime": "2026080700",
|
||||||
|
"sourceBytes": 5128989,
|
||||||
|
"resampled": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
{
|
||||||
|
"source": {
|
||||||
|
"format": "weather-v2-particle-bin-v1",
|
||||||
|
"magic": "WVECBIN1",
|
||||||
|
"targetTime": "2026080706",
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
|
||||||
|
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
|
||||||
|
"uVUnit": "m/s",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"bytes": 5128989
|
||||||
|
},
|
||||||
|
"output": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"bytes": 3155976,
|
||||||
|
"sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"resampled": false
|
||||||
|
},
|
||||||
|
"statistics": {
|
||||||
|
"u": {
|
||||||
|
"min": -37.18255615234375,
|
||||||
|
"max": 24.125526428222656,
|
||||||
|
"mean": 0.9443698525428772
|
||||||
|
},
|
||||||
|
"v": {
|
||||||
|
"min": -28.227636337280273,
|
||||||
|
"max": 25.05806541442871,
|
||||||
|
"mean": 1.0168105363845825
|
||||||
|
},
|
||||||
|
"speed": {
|
||||||
|
"min": 0.010480343364179134,
|
||||||
|
"max": 37.76322555541992,
|
||||||
|
"mean": 6.7594804763793945
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"sourceToCanonical": {
|
||||||
|
"lossless": true,
|
||||||
|
"maxAbsErrorU": 0.0,
|
||||||
|
"maxAbsErrorV": 0.0,
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"row": 384,
|
||||||
|
"column": 256,
|
||||||
|
"sourceU": -10.2057466506958,
|
||||||
|
"sourceV": 4.789729118347168,
|
||||||
|
"canonicalU": -10.2057466506958,
|
||||||
|
"canonicalV": 4.789729118347168
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.45672082901000977,
|
||||||
|
"sourceV": 3.2181167602539062,
|
||||||
|
"canonicalU": -0.45672082901000977,
|
||||||
|
"canonicalV": 3.2181167602539062
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": 2.235017776489258,
|
||||||
|
"sourceV": -0.7363739013671875,
|
||||||
|
"canonicalU": 2.235017776489258,
|
||||||
|
"canonicalV": -0.7363739013671875
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -1.2302308082580566,
|
||||||
|
"sourceV": 1.5909242630004883,
|
||||||
|
"canonicalU": -1.2302308082580566,
|
||||||
|
"canonicalV": 1.5909242630004883
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": -5.034683704376221,
|
||||||
|
"sourceV": -0.15012121200561523,
|
||||||
|
"canonicalU": -5.034683704376221,
|
||||||
|
"canonicalV": -0.15012121200561523
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 0,
|
||||||
|
"sourceU": -0.15583133697509766,
|
||||||
|
"sourceV": 3.617917060852051,
|
||||||
|
"canonicalU": -0.15583133697509766,
|
||||||
|
"canonicalV": 3.617917060852051
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.45672082901000977,
|
||||||
|
"sourceV": 3.551283836364746,
|
||||||
|
"canonicalU": -0.45672082901000977,
|
||||||
|
"canonicalV": 3.551283836364746
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 2,
|
||||||
|
"sourceU": -0.6974325180053711,
|
||||||
|
"sourceV": 3.2181167602539062,
|
||||||
|
"canonicalU": -0.6974325180053711,
|
||||||
|
"canonicalV": 3.2181167602539062
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Binary file not shown.
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"schema": "maplibre-particle-field/v1",
|
||||||
|
"fieldId": "ecmwf-hres-wind-2026080706",
|
||||||
|
"fieldType": "WIND",
|
||||||
|
"displayDomain": "GLOBAL",
|
||||||
|
"generation": 1786082400000,
|
||||||
|
"validTime": 1786082400000,
|
||||||
|
"dataVersion": 1786082400000,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"grid": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769
|
||||||
|
},
|
||||||
|
"vector": {
|
||||||
|
"layout": "interleaved-uv",
|
||||||
|
"encoding": "float32-le",
|
||||||
|
"noData": -9999.0,
|
||||||
|
"unit": "m/s",
|
||||||
|
"origin": "north-west",
|
||||||
|
"rowOrder": "north-to-south",
|
||||||
|
"columnOrder": "west-to-east",
|
||||||
|
"uPositive": "east",
|
||||||
|
"vPositive": "north"
|
||||||
|
},
|
||||||
|
"payload": {
|
||||||
|
"byteLength": 3155976,
|
||||||
|
"sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
|
||||||
|
"referenceTime": "2026080600",
|
||||||
|
"targetTime": "2026080706",
|
||||||
|
"sourceBytes": 5128989,
|
||||||
|
"resampled": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
{
|
||||||
|
"source": {
|
||||||
|
"format": "weather-v2-particle-bin-v1",
|
||||||
|
"magic": "WVECBIN1",
|
||||||
|
"targetTime": "2026080712",
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
|
||||||
|
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
|
||||||
|
"uVUnit": "m/s",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"bytes": 5128989
|
||||||
|
},
|
||||||
|
"output": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"bytes": 3155976,
|
||||||
|
"sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"resampled": false
|
||||||
|
},
|
||||||
|
"statistics": {
|
||||||
|
"u": {
|
||||||
|
"min": -37.39310836791992,
|
||||||
|
"max": 24.769317626953125,
|
||||||
|
"mean": 0.7698453664779663
|
||||||
|
},
|
||||||
|
"v": {
|
||||||
|
"min": -28.266582489013672,
|
||||||
|
"max": 27.809932708740234,
|
||||||
|
"mean": 1.1808899641036987
|
||||||
|
},
|
||||||
|
"speed": {
|
||||||
|
"min": 0.024363698437809944,
|
||||||
|
"max": 37.42909240722656,
|
||||||
|
"mean": 6.384326934814453
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"sourceToCanonical": {
|
||||||
|
"lossless": true,
|
||||||
|
"maxAbsErrorU": 0.0,
|
||||||
|
"maxAbsErrorV": 0.0,
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"row": 384,
|
||||||
|
"column": 256,
|
||||||
|
"sourceU": -10.180034637451172,
|
||||||
|
"sourceV": 4.663511276245117,
|
||||||
|
"canonicalU": -10.180034637451172,
|
||||||
|
"canonicalV": 4.663511276245117
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.5417494773864746,
|
||||||
|
"sourceV": 0.767613410949707,
|
||||||
|
"canonicalU": -0.5417494773864746,
|
||||||
|
"canonicalV": 0.767613410949707
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": -0.5780353546142578,
|
||||||
|
"sourceV": 0.26222801208496094,
|
||||||
|
"canonicalU": -0.5780353546142578,
|
||||||
|
"canonicalV": 0.26222801208496094
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -1.8420743942260742,
|
||||||
|
"sourceV": 0.3505821228027344,
|
||||||
|
"canonicalU": -1.8420743942260742,
|
||||||
|
"canonicalV": 0.3505821228027344
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": -4.581581115722656,
|
||||||
|
"sourceV": -1.1498479843139648,
|
||||||
|
"canonicalU": -4.581581115722656,
|
||||||
|
"canonicalV": -1.1498479843139648
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 0,
|
||||||
|
"sourceU": -0.5939555168151855,
|
||||||
|
"sourceV": 1.1400480270385742,
|
||||||
|
"canonicalU": -0.5939555168151855,
|
||||||
|
"canonicalV": 1.1400480270385742
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.48954296112060547,
|
||||||
|
"sourceV": 1.0159034729003906,
|
||||||
|
"canonicalU": -0.48954296112060547,
|
||||||
|
"canonicalV": 1.0159034729003906
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 2,
|
||||||
|
"sourceU": -0.48954296112060547,
|
||||||
|
"sourceV": 0.767613410949707,
|
||||||
|
"canonicalU": -0.48954296112060547,
|
||||||
|
"canonicalV": 0.767613410949707
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Binary file not shown.
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"schema": "maplibre-particle-field/v1",
|
||||||
|
"fieldId": "ecmwf-hres-wind-2026080712",
|
||||||
|
"fieldType": "WIND",
|
||||||
|
"displayDomain": "GLOBAL",
|
||||||
|
"generation": 1786104000000,
|
||||||
|
"validTime": 1786104000000,
|
||||||
|
"dataVersion": 1786104000000,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"grid": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769
|
||||||
|
},
|
||||||
|
"vector": {
|
||||||
|
"layout": "interleaved-uv",
|
||||||
|
"encoding": "float32-le",
|
||||||
|
"noData": -9999.0,
|
||||||
|
"unit": "m/s",
|
||||||
|
"origin": "north-west",
|
||||||
|
"rowOrder": "north-to-south",
|
||||||
|
"columnOrder": "west-to-east",
|
||||||
|
"uPositive": "east",
|
||||||
|
"vPositive": "north"
|
||||||
|
},
|
||||||
|
"payload": {
|
||||||
|
"byteLength": 3155976,
|
||||||
|
"sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
|
||||||
|
"referenceTime": "2026080600",
|
||||||
|
"targetTime": "2026080712",
|
||||||
|
"sourceBytes": 5128989,
|
||||||
|
"resampled": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
{
|
||||||
|
"source": {
|
||||||
|
"format": "weather-v2-particle-bin-v1",
|
||||||
|
"magic": "WVECBIN1",
|
||||||
|
"targetTime": "2026080718",
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane",
|
||||||
|
"arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south",
|
||||||
|
"uVUnit": "m/s",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"bytes": 5128989
|
||||||
|
},
|
||||||
|
"output": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"bytes": 3155976,
|
||||||
|
"sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b",
|
||||||
|
"validCount": 394497,
|
||||||
|
"invalidCount": 0,
|
||||||
|
"resampled": false
|
||||||
|
},
|
||||||
|
"statistics": {
|
||||||
|
"u": {
|
||||||
|
"min": -31.956016540527344,
|
||||||
|
"max": 28.557456970214844,
|
||||||
|
"mean": 1.212062120437622
|
||||||
|
},
|
||||||
|
"v": {
|
||||||
|
"min": -26.13941192626953,
|
||||||
|
"max": 27.131576538085938,
|
||||||
|
"mean": 1.1284888982772827
|
||||||
|
},
|
||||||
|
"speed": {
|
||||||
|
"min": 0.0210099034011364,
|
||||||
|
"max": 33.67807388305664,
|
||||||
|
"mean": 6.403453826904297
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"sourceToCanonical": {
|
||||||
|
"lossless": true,
|
||||||
|
"maxAbsErrorU": 0.0,
|
||||||
|
"maxAbsErrorV": 0.0,
|
||||||
|
"samples": [
|
||||||
|
{
|
||||||
|
"row": 384,
|
||||||
|
"column": 256,
|
||||||
|
"sourceU": -10.21538257598877,
|
||||||
|
"sourceV": 2.3842287063598633,
|
||||||
|
"canonicalU": -10.21538257598877,
|
||||||
|
"canonicalV": 2.3842287063598633
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.18732452392578125,
|
||||||
|
"sourceV": 0.2720670700073242,
|
||||||
|
"canonicalU": -0.18732452392578125,
|
||||||
|
"canonicalV": 0.2720670700073242
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 1,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": 0.460113525390625,
|
||||||
|
"sourceV": 0.4729623794555664,
|
||||||
|
"canonicalU": 0.460113525390625,
|
||||||
|
"canonicalV": 0.4729623794555664
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -1.4576325416564941,
|
||||||
|
"sourceV": -0.03371381759643555,
|
||||||
|
"canonicalU": -1.4576325416564941,
|
||||||
|
"canonicalV": -0.03371381759643555
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 767,
|
||||||
|
"column": 511,
|
||||||
|
"sourceU": -2.9916043281555176,
|
||||||
|
"sourceV": 0.8597030639648438,
|
||||||
|
"canonicalU": -2.9916043281555176,
|
||||||
|
"canonicalV": 0.8597030639648438
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 0,
|
||||||
|
"sourceU": -1.020096778869629,
|
||||||
|
"sourceV": 0.9125776290893555,
|
||||||
|
"canonicalU": -1.020096778869629,
|
||||||
|
"canonicalV": 0.9125776290893555
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 1,
|
||||||
|
"sourceU": -0.49961376190185547,
|
||||||
|
"sourceV": 0.6990737915039062,
|
||||||
|
"canonicalU": -0.49961376190185547,
|
||||||
|
"canonicalV": 0.6990737915039062
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"row": 0,
|
||||||
|
"column": 2,
|
||||||
|
"sourceU": -0.08322811126708984,
|
||||||
|
"sourceV": 0.41440296173095703,
|
||||||
|
"canonicalU": -0.08322811126708984,
|
||||||
|
"canonicalV": 0.41440296173095703
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
Binary file not shown.
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"schema": "maplibre-particle-field/v1",
|
||||||
|
"fieldId": "ecmwf-hres-wind-2026080718",
|
||||||
|
"fieldType": "WIND",
|
||||||
|
"displayDomain": "GLOBAL",
|
||||||
|
"generation": 1786125600000,
|
||||||
|
"validTime": 1786125600000,
|
||||||
|
"dataVersion": 1786125600000,
|
||||||
|
"bounds": {
|
||||||
|
"west": 112.5,
|
||||||
|
"south": 0.0,
|
||||||
|
"east": 157.5,
|
||||||
|
"north": 55.77657301866769
|
||||||
|
},
|
||||||
|
"grid": {
|
||||||
|
"width": 513,
|
||||||
|
"height": 769
|
||||||
|
},
|
||||||
|
"vector": {
|
||||||
|
"layout": "interleaved-uv",
|
||||||
|
"encoding": "float32-le",
|
||||||
|
"noData": -9999.0,
|
||||||
|
"unit": "m/s",
|
||||||
|
"origin": "north-west",
|
||||||
|
"rowOrder": "north-to-south",
|
||||||
|
"columnOrder": "west-to-east",
|
||||||
|
"uPositive": "east",
|
||||||
|
"vPositive": "north"
|
||||||
|
},
|
||||||
|
"payload": {
|
||||||
|
"byteLength": 3155976,
|
||||||
|
"sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b"
|
||||||
|
},
|
||||||
|
"source": {
|
||||||
|
"kind": "WeatherV2 ECMWF HRES WVECBIN1",
|
||||||
|
"referenceTime": "2026080600",
|
||||||
|
"targetTime": "2026080718",
|
||||||
|
"sourceBytes": 5128989,
|
||||||
|
"resampled": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"schema": "maplibre-particle-test-dataset/v1",
|
||||||
|
"datasetId": "ecmwf-hres-wind-20260807",
|
||||||
|
"source": "WeatherV2 ECMWF HRES WVECBIN1",
|
||||||
|
"referenceTime": "2026080600",
|
||||||
|
"validTimes": [
|
||||||
|
"2026080700",
|
||||||
|
"2026080706",
|
||||||
|
"2026080712",
|
||||||
|
"2026080718"
|
||||||
|
],
|
||||||
|
"snapshotLayout": "<validTime>/metadata.json + <validTime>/field.f32",
|
||||||
|
"notes": "开发期离线 canonical 测试索引;不是 Weather Manifest,也不进入 ParticleFieldSnapshot。"
|
||||||
|
}
|
||||||
37
tools/weatherv2_wind_canonical/README.md
Normal file
37
tools/weatherv2_wind_canonical/README.md
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
# WeatherV2 风场转 MapLibre canonical 测试数据工具
|
||||||
|
|
||||||
|
此工具只用于开发期离线转换:将 WeatherV2 的完整 `wind_vectors.bin`(`WVECBIN1`)转成 STEP-053 已冻结的 `maplibre-particle-field/v1`,不属于 MapLibre Runtime,也不访问网络、不扫描下载目录、更不修改 WeatherV2 原始数据。
|
||||||
|
|
||||||
|
## 输入
|
||||||
|
|
||||||
|
输入为一个 WeatherV2 reference/run 目录及其中一个 `YYYYMMDDHH` valid time:
|
||||||
|
|
||||||
|
```text
|
||||||
|
<run-dir>/<valid-time>/wind_vectors.bin
|
||||||
|
<run-dir>/<valid-time>/wind_vectors.bin.header.json
|
||||||
|
```
|
||||||
|
|
||||||
|
当前已核对的 `WVECBIN1` 布局为:`magic(8)`、little-endian `uint32 headerLength`、UTF-8 JSON header、row-major `u float32-le` 平面、`v float32-le` 平面、`speed float32-le` 平面、`valid uint8` 平面。WeatherV2 `bin_export.py` 将同一 `decode_wind()` U/V grid 写出;QBIN 和 NSFT 也从同一 decoder U/V 生成。QBIN metadata 明确其 U/V 单位为 m/s,因此本工具保持 m/s,不进行 kt 换算。
|
||||||
|
|
||||||
|
源 grid 的 y 来自 Web Mercator tile y,向南增加;因此输出的 row-major rows 是 north-to-south,columns 是 west-to-east。U=eastward-positive,V=northward-positive。
|
||||||
|
|
||||||
|
## 输出
|
||||||
|
|
||||||
|
每个输出目录包含:
|
||||||
|
|
||||||
|
- `metadata.json`:现有 `LocalParticleFieldFeed` 读取的 `maplibre-particle-field/v1` metadata。
|
||||||
|
- `field.f32`:`float32-le interleaved-uv`,顺序为 `U0,V0,U1,V1,...`。
|
||||||
|
- `conversion_summary.json`:源/输出 bounds、尺寸、统计值、valid/noData 计数、抽样一致性与 SHA-256。
|
||||||
|
|
||||||
|
本工具不重采样。每个源 `valid=0` 或非有限 U/V 的 cell 会写为既有 canonical `noData=-9999.0` 的 U/V 对;不会错误变成静风 `(0,0)`。有效样本按 float32 bit-exact 复制。
|
||||||
|
|
||||||
|
## 命令示例
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 tools/weatherv2_wind_canonical/convert_weatherv2_wind.py \
|
||||||
|
--run-dir /root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600 \
|
||||||
|
--valid-time 2026080700 \
|
||||||
|
--output platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700
|
||||||
|
```
|
||||||
|
|
||||||
|
相同输入会产生相同 payload SHA-256。错误会输出明确消息并以非零状态退出。
|
||||||
190
tools/weatherv2_wind_canonical/convert_weatherv2_wind.py
Normal file
190
tools/weatherv2_wind_canonical/convert_weatherv2_wind.py
Normal file
@@ -0,0 +1,190 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Convert one WeatherV2 WVECBIN1 wind snapshot to maplibre-particle-field/v1."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
MAGIC = b"WVECBIN1"
|
||||||
|
NO_DATA = np.float32(-9999.0)
|
||||||
|
|
||||||
|
|
||||||
|
def fail(message: str) -> "None":
|
||||||
|
raise ValueError(message)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--run-dir", required=True, type=Path, help="WeatherV2 reference-time directory")
|
||||||
|
parser.add_argument("--valid-time", required=True, help="target time in YYYYMMDDHH UTC")
|
||||||
|
parser.add_argument("--output", required=True, type=Path, help="canonical snapshot output directory")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def require_mapping(value: object, name: str) -> dict:
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
fail(f"{name} must be an object")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def read_source(run_dir: Path, valid_time: str) -> tuple[dict, np.ndarray, np.ndarray, np.ndarray, np.ndarray, int]:
|
||||||
|
source_dir = run_dir / valid_time
|
||||||
|
payload_path = source_dir / "wind_vectors.bin"
|
||||||
|
sidecar_path = source_dir / "wind_vectors.bin.header.json"
|
||||||
|
if not payload_path.is_file() or not sidecar_path.is_file():
|
||||||
|
fail(f"missing WeatherV2 snapshot for {valid_time}: {source_dir}")
|
||||||
|
data = payload_path.read_bytes()
|
||||||
|
if len(data) < 12 or data[:8] != MAGIC:
|
||||||
|
fail(f"unsupported wind_vectors.bin magic in {payload_path}")
|
||||||
|
header_length = struct.unpack_from("<I", data, 8)[0]
|
||||||
|
header_end = 12 + header_length
|
||||||
|
if header_end > len(data):
|
||||||
|
fail("WVECBIN1 header length exceeds file length")
|
||||||
|
header = json.loads(data[12:header_end].decode("utf-8"))
|
||||||
|
sidecar = json.loads(sidecar_path.read_text(encoding="utf-8"))
|
||||||
|
if header != sidecar:
|
||||||
|
fail("embedded WVECBIN1 header differs from .header.json sidecar")
|
||||||
|
if header.get("format") != "weather-v2-particle-bin-v1" or header.get("layer") != "wind":
|
||||||
|
fail("source is not a WeatherV2 wind particle-bin-v1 snapshot")
|
||||||
|
if header.get("target_time") != valid_time:
|
||||||
|
fail(f"source target_time mismatch: {header.get('target_time')!r} != {valid_time!r}")
|
||||||
|
width, height = int(header.get("width", 0)), int(header.get("height", 0))
|
||||||
|
if width <= 0 or height <= 0:
|
||||||
|
fail("source width/height must be positive")
|
||||||
|
expected_fields = [
|
||||||
|
{"name": "u", "dtype": "float32"},
|
||||||
|
{"name": "v", "dtype": "float32"},
|
||||||
|
{"name": "speed", "dtype": "float32"},
|
||||||
|
{"name": "valid", "dtype": "uint8"},
|
||||||
|
]
|
||||||
|
if header.get("fields") != expected_fields:
|
||||||
|
fail(f"unsupported WVECBIN1 fields: {header.get('fields')!r}")
|
||||||
|
count = width * height
|
||||||
|
expected_bytes = header_end + count * 4 * 3 + count
|
||||||
|
if len(data) != expected_bytes:
|
||||||
|
fail(f"WVECBIN1 byte length mismatch: expected {expected_bytes}, got {len(data)}")
|
||||||
|
offset = header_end
|
||||||
|
u = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
|
||||||
|
offset += count * 4
|
||||||
|
v = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
|
||||||
|
offset += count * 4
|
||||||
|
speed = np.frombuffer(data, dtype="<f4", count=count, offset=offset).copy().reshape(height, width)
|
||||||
|
offset += count * 4
|
||||||
|
valid = np.frombuffer(data, dtype=np.uint8, count=count, offset=offset).copy().reshape(height, width).astype(bool)
|
||||||
|
bounds = require_mapping(header.get("bounds"), "source bounds")
|
||||||
|
if not all(math.isfinite(float(bounds[name])) for name in ("west", "south", "east", "north")):
|
||||||
|
fail("source bounds must be finite")
|
||||||
|
if not float(bounds["west"]) < float(bounds["east"]) or not float(bounds["south"]) < float(bounds["north"]):
|
||||||
|
fail("source bounds are invalid")
|
||||||
|
if not np.isfinite(u[valid]).all() or not np.isfinite(v[valid]).all():
|
||||||
|
fail("valid WeatherV2 samples contain non-finite U/V")
|
||||||
|
if not np.isfinite(speed[valid]).all():
|
||||||
|
fail("valid WeatherV2 samples contain non-finite speed")
|
||||||
|
return header, u, v, speed, valid, len(data)
|
||||||
|
|
||||||
|
|
||||||
|
def epoch_millis(valid_time: str) -> int:
|
||||||
|
try:
|
||||||
|
return int(datetime.strptime(valid_time, "%Y%m%d%H").replace(tzinfo=timezone.utc).timestamp() * 1000)
|
||||||
|
except ValueError as error:
|
||||||
|
fail(f"valid time must be YYYYMMDDHH UTC: {error}")
|
||||||
|
|
||||||
|
|
||||||
|
def stats(values: np.ndarray, valid: np.ndarray) -> dict:
|
||||||
|
chosen = values[valid]
|
||||||
|
if not chosen.size:
|
||||||
|
fail("source has no valid samples")
|
||||||
|
return {"min": float(chosen.min()), "max": float(chosen.max()), "mean": float(chosen.mean())}
|
||||||
|
|
||||||
|
|
||||||
|
def sample_indices(valid: np.ndarray) -> list[tuple[int, int]]:
|
||||||
|
height, width = valid.shape
|
||||||
|
candidates = [(height // 2, width // 2), (1, 1), (1, width - 2), (height - 2, 1), (height - 2, width - 2)]
|
||||||
|
selected: list[tuple[int, int]] = []
|
||||||
|
for row, col in candidates:
|
||||||
|
if valid[row, col] and (row, col) not in selected:
|
||||||
|
selected.append((row, col))
|
||||||
|
for row, col in zip(*np.nonzero(valid)):
|
||||||
|
if (int(row), int(col)) not in selected:
|
||||||
|
selected.append((int(row), int(col)))
|
||||||
|
if len(selected) == 8:
|
||||||
|
break
|
||||||
|
return selected
|
||||||
|
|
||||||
|
|
||||||
|
def write_output(header: dict,
|
||||||
|
u: np.ndarray,
|
||||||
|
v: np.ndarray,
|
||||||
|
speed: np.ndarray,
|
||||||
|
valid: np.ndarray,
|
||||||
|
source_bytes: int,
|
||||||
|
reference_time: str,
|
||||||
|
output: Path) -> dict:
|
||||||
|
width, height = int(header["width"]), int(header["height"])
|
||||||
|
valid = valid & np.isfinite(u) & np.isfinite(v)
|
||||||
|
payload = np.empty((height, width, 2), dtype="<f4")
|
||||||
|
payload[:, :, 0] = np.where(valid, u, NO_DATA)
|
||||||
|
payload[:, :, 1] = np.where(valid, v, NO_DATA)
|
||||||
|
payload_bytes = payload.tobytes(order="C")
|
||||||
|
payload_hash = hashlib.sha256(payload_bytes).hexdigest()
|
||||||
|
decoded = np.frombuffer(payload_bytes, dtype="<f4").reshape(height, width, 2)
|
||||||
|
if not np.array_equal(decoded[:, :, 0][valid], u[valid]) or not np.array_equal(decoded[:, :, 1][valid], v[valid]):
|
||||||
|
fail("source-to-canonical U/V conversion is not lossless")
|
||||||
|
if not np.all(decoded[:, :, 0][~valid] == NO_DATA) or not np.all(decoded[:, :, 1][~valid] == NO_DATA):
|
||||||
|
fail("invalid samples were not preserved as canonical noData")
|
||||||
|
bounds = header["bounds"]
|
||||||
|
timestamp = epoch_millis(str(header["target_time"]))
|
||||||
|
metadata = {
|
||||||
|
"schema": "maplibre-particle-field/v1",
|
||||||
|
"fieldId": f"ecmwf-hres-wind-{header['target_time']}",
|
||||||
|
"fieldType": "WIND",
|
||||||
|
"displayDomain": "GLOBAL",
|
||||||
|
"generation": timestamp,
|
||||||
|
"validTime": timestamp,
|
||||||
|
"dataVersion": timestamp,
|
||||||
|
"bounds": {name: float(bounds[name]) for name in ("west", "south", "east", "north")},
|
||||||
|
"grid": {"width": width, "height": height},
|
||||||
|
"vector": {"layout": "interleaved-uv", "encoding": "float32-le", "noData": float(NO_DATA), "unit": "m/s", "origin": "north-west", "rowOrder": "north-to-south", "columnOrder": "west-to-east", "uPositive": "east", "vPositive": "north"},
|
||||||
|
"payload": {"byteLength": len(payload_bytes), "sha256": payload_hash},
|
||||||
|
"source": {"kind": "WeatherV2 ECMWF HRES WVECBIN1", "referenceTime": reference_time, "targetTime": header["target_time"], "sourceBytes": source_bytes, "resampled": False},
|
||||||
|
}
|
||||||
|
samples = []
|
||||||
|
for row, col in sample_indices(valid):
|
||||||
|
samples.append({"row": row, "column": col, "sourceU": float(u[row, col]), "sourceV": float(v[row, col]), "canonicalU": float(decoded[row, col, 0]), "canonicalV": float(decoded[row, col, 1])})
|
||||||
|
summary = {
|
||||||
|
"source": {"format": header["format"], "magic": header["magic"], "targetTime": header["target_time"], "width": width, "height": height, "bounds": metadata["bounds"], "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", "uVUnit": "m/s", "validCount": int(valid.sum()), "invalidCount": int(valid.size - valid.sum()), "bytes": source_bytes},
|
||||||
|
"output": {"width": width, "height": height, "bounds": metadata["bounds"], "bytes": len(payload_bytes), "sha256": payload_hash, "validCount": int(valid.sum()), "invalidCount": int(valid.size - valid.sum()), "resampled": False},
|
||||||
|
"statistics": {"u": stats(u, valid), "v": stats(v, valid), "speed": stats(speed, valid)},
|
||||||
|
"sourceToCanonical": {"lossless": True, "maxAbsErrorU": 0.0, "maxAbsErrorV": 0.0, "samples": samples},
|
||||||
|
}
|
||||||
|
output.mkdir(parents=True, exist_ok=True)
|
||||||
|
(output / "field.f32").write_bytes(payload_bytes)
|
||||||
|
(output / "metadata.json").write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
(output / "conversion_summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parse_args()
|
||||||
|
try:
|
||||||
|
header, u, v, speed, valid, source_bytes = read_source(args.run_dir, args.valid_time)
|
||||||
|
summary = write_output(header, u, v, speed, valid, source_bytes, args.run_dir.name, args.output)
|
||||||
|
print(json.dumps({"ok": True, "targetTime": header["target_time"], "output": str(args.output), "sha256": summary["output"]["sha256"], "validCount": summary["output"]["validCount"], "invalidCount": summary["output"]["invalidCount"]}, ensure_ascii=False))
|
||||||
|
return 0
|
||||||
|
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||||
|
print(f"error: {error}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -398,3 +398,9 @@
|
|||||||
- `fieldUploadCount=4 → 5 → 6` 证明正常 boundary 每次仅上传一个 NEW_T1。唯一 `simulationResetCount=1` 属于旧 fixture 切到 Host dataset 的允许初始化,不是正常 production boundary 回归。
|
- `fieldUploadCount=4 → 5 → 6` 证明正常 boundary 每次仅上传一个 NEW_T1。唯一 `simulationResetCount=1` 属于旧 fixture 切到 Host dataset 的允许初始化,不是正常 production boundary 回归。
|
||||||
- `HOST DUPLICATE` 返回 `DUPLICATE_NOOP` 且 upload/reset 不增长;`HOST INVALID` 经 Adapter reject fail-closed,last valid field 与 Renderer continuity(`stateSeed=CONTINUE`)保持,无黑屏、无 crash。
|
- `HOST DUPLICATE` 返回 `DUPLICATE_NOOP` 且 upload/reset 不增长;`HOST INVALID` 经 Adapter reject fail-closed,last valid field 与 Renderer continuity(`stateSeed=CONTINUE`)保持,无黑屏、无 crash。
|
||||||
- 用户正式风场数据集仍未提供、未验证,但不阻塞 contract 阶段关闭;后续可在同一 canonical contract 下追加。MIX 文本滞后及 WAITING/INVALID 视觉为 `NON-BLOCKING observability`,本次未修改功能代码或执行 Android/Jest。
|
- 用户正式风场数据集仍未提供、未验证,但不阻塞 contract 阶段关闭;后续可在同一 canonical contract 下追加。MIX 文本滞后及 WAITING/INVALID 视觉为 `NON-BLOCKING observability`,本次未修改功能代码或执行 Android/Jest。
|
||||||
|
|
||||||
|
## 2026-09-04 — STEP-054 真实 WeatherV2 风数据 Canonical 测试 asset
|
||||||
|
|
||||||
|
- 只读核对 WeatherV2 `bin_export.py`、`decode_wind()`、QBIN/NSFT export 后,确认 `wind_vectors.bin` 为完整拼接 `WVECBIN1` grid:row-major U/V/speed float32-le planes + valid uint8 plane,U=eastward、V=northward,单位 m/s;QBIN/NSFT 均由同一 decoded U/V Field 生成。
|
||||||
|
- 从 run `2026080600` 生成真实 ECMWF HRES `2026080700/06/12/18` 四帧。原始日本及周边 bounds 为 112.5/0.0/157.5/55.776573,`513×769`,不裁剪、不重采样;输出 canonical interleaved U/V Android assets,每帧 3,155,976 bytes。
|
||||||
|
- 转换器独立放在 `tools/weatherv2_wind_canonical/`,保留 valid/noData:invalid 写既有 -9999 sentinel,不伪造为 0 风;四帧 hash 不同,重复 T0 hash 相同,source→canonical 抽样误差 0。未改 Core/Renderer/Android,也未执行 Android/Jest;等待下一阶段集中 Android 集成验收。
|
||||||
|
|||||||
Reference in New Issue
Block a user