diff --git a/docs/50_Step记录/STEP-054_真实WeatherV2风数据提取与Canonical测试数据生成_v1.md b/docs/50_Step记录/STEP-054_真实WeatherV2风数据提取与Canonical测试数据生成_v1.md new file mode 100644 index 000000000..a52df121f --- /dev/null +++ b/docs/50_Step记录/STEP-054_真实WeatherV2风数据提取与Canonical测试数据生成_v1.md @@ -0,0 +1,58 @@ +# STEP-054 真实 WeatherV2 风数据提取与 Canonical 测试数据生成 + +- 状态:`IMPLEMENTED / WAITING_NEXT_ANDROID_INTEGRATION_VALIDATION` +- Starting HEAD:`5fc922785d8ec007127d48cd3ab5dadce270b947` +- 分支:`particle/main` +- 目标:将 WeatherV2 ECMWF HRES 的真实 `WVECBIN1` 风场无损转换为已冻结的 `maplibre-particle-field/v1` Android 测试 asset;不实现网络、Provider、React Native Binding 或 Android 综合绘制。 + +## 实际数据与格式依据 + +读取 WeatherV2 源目录: + +`/root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600` + +按既定时次生成 T0~T3:`2026080700`、`2026080706`、`2026080712`、`2026080718`。四帧均实际存在,均为日本及周边的完整区域 grid:`513×769`、bounds `112.5, 0.0, 157.5, 55.77657301866769`,单源文件 `5,128,989` bytes。 + +WeatherV2 `src/dev/core/bin_export.py` 的 `_export_vector_target()` 先以 `decode_wind()` 从原 Windy `wm_grid_257` tile 解码 U/V,再按 `WVECBIN1 + uint32-le header length + JSON header + U float32-le plane + V float32-le plane + speed float32-le plane + valid uint8 plane` 写入 `wind_vectors.bin`。数组 row-major;Web Mercator tile y 向南增加,因此 row 是 north-to-south、column 是 west-to-east。 + +`src/dev/layers/wind.py` 明确读取 `decoded_r` 为 U、`decoded_g` 为 V;writer metadata 说明 U eastward-positive、V northward-positive、screen y 使用 `-v`。QBIN exporter 和 NSFT exporter 都重新使用同一 `decode_wind()` Field:QBIN `u_q/v_q` 的 quantization metadata 明确为 `m/s`(`scale=0.01`),NSFT wind 也从该 Field 量化。因此本次原始 U/V 单位已由 writer/下游 decoder 确认为 `m/s`,没有 kt 换算、scale/offset 还原或额外量化。 + +`wind_vectors.bin` 是 WeatherV2 pipeline 对完整已拼接 U/V grid 的 particle/vector export;QBIN 是同源单 tile量化分发格式,NSFT 是同源单 tile量化 PMTiles 格式。三者共享 decoded U/V、valid 和 grid/bounds 语义,但本任务直接读取完整 WVECBIN1,不经 QBIN/NSFT 拼瓦片。 + +## 转换工具与输出 + +新增独立工具: + +- `tools/weatherv2_wind_canonical/convert_weatherv2_wind.py` +- `tools/weatherv2_wind_canonical/README.md` + +工具参数为 `--run-dir`、`--valid-time`、`--output`;源文件仅只读。它严格校验 magic、embedded header 与 sidecar 一致性、fields/dtype、长度、bounds 和有效 U/V,再输出既有 `maplibre-particle-field/v1` metadata 与 `float32-le interleaved-uv` payload。输出 `validTime` 是真实 UTC epoch milliseconds;`generation` 和 `dataVersion` 同为 producer-owned epoch milliseconds,四帧严格递增。 + +输出目录: + +`platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/` + +其中 `dataset.json` 只是开发期四帧索引,不是 Weather Manifest,也不进入 Snapshot。每个 validTime 子目录包含 `metadata.json`、`field.f32`、`conversion_summary.json`。 + +未重采样、未裁剪;每个 canonical payload 保留原完整 `513×769` 范围,`3,155,976` bytes。有效 U/V 逐 float32 bit-exact 复制;source `valid=0` 或非有限 U/V 会映射为既有 `noData=-9999` 的 U/V pair,绝不写成 `(0,0)`。当前四个实际 frame 均为 `394,497` valid、`0` invalid,但转换器仍保留该 fail-closed mapping。 + +## 生成结果与验证 + +| validTime | SHA-256 | U min/max (m/s) | V min/max (m/s) | speed min/max (m/s) | +| --- | --- | --- | --- | --- | +| 2026080700 | `363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283` | -35.1724 / 22.5011 | -25.4723 / 28.2009 | 0.0037 / 35.2783 | +| 2026080706 | `ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451` | -37.1826 / 24.1255 | -28.2276 / 25.0581 | 0.0105 / 37.7632 | +| 2026080712 | `d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0` | -37.3931 / 24.7693 | -28.2666 / 27.8099 | 0.0244 / 37.4291 | +| 2026080718 | `970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b` | -31.9560 / 28.5575 | -26.1394 / 27.1316 | 0.0210 / 33.6781 | + +- 四帧 metadata 可解析、payload length 与 `width×height×8` 一致,validTime 严格递增,四个 SHA-256 均不同:PASS。 +- 每帧中心、四角附近和内部有效样本 source→canonical U/V 的最大绝对误差均为 `0.0`:PASS。 +- T0 重复转换的 SHA-256 与已生成 payload 相同:PASS。 +- 转换器 Python syntax、正常转换和无效 source 的非零失败路径:PASS。 +- 既有 Core canonical/Adapter/Host ingress 定向测试:5/5 PASS。 + +## 冻结边界与未执行项 + +未修改 `ParticleFieldDescriptor`、`ParticleFieldSnapshot`、`ParticleFieldDataAdapter`、Timeline、Playback、Runtime Field Window、Renderer、Android Kotlin/JNI 或 WeatherV2 原始数据。未将 Weather Service、HTTP、Provider、QBIN/NSFT 网络、缓存或目录扫描加入 MapLibre Runtime。 + +未执行 Android Gradle、APK、ADB、设备/模拟器测试或 Jest。下一阶段才使用本 asset 做 MapLibre map + WeatherColorLayer + RasterParticleLayer 综合 Android 验收;本 Step 不提前进入该范围。 diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/conversion_summary.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/conversion_summary.json new file mode 100644 index 000000000..2ce6e357f --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/conversion_summary.json @@ -0,0 +1,124 @@ +{ + "source": { + "format": "weather-v2-particle-bin-v1", + "magic": "WVECBIN1", + "targetTime": "2026080700", + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", + "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", + "uVUnit": "m/s", + "validCount": 394497, + "invalidCount": 0, + "bytes": 5128989 + }, + "output": { + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "bytes": 3155976, + "sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283", + "validCount": 394497, + "invalidCount": 0, + "resampled": false + }, + "statistics": { + "u": { + "min": -35.172367095947266, + "max": 22.501087188720703, + "mean": 0.7285217046737671 + }, + "v": { + "min": -25.472291946411133, + "max": 28.200902938842773, + "mean": 1.1300384998321533 + }, + "speed": { + "min": 0.0036890381015837193, + "max": 35.27834701538086, + "mean": 6.401810169219971 + } + }, + "sourceToCanonical": { + "lossless": true, + "maxAbsErrorU": 0.0, + "maxAbsErrorV": 0.0, + "samples": [ + { + "row": 384, + "column": 256, + "sourceU": -10.921929359436035, + "sourceV": 4.515348434448242, + "canonicalU": -10.921929359436035, + "canonicalV": 4.515348434448242 + }, + { + "row": 1, + "column": 1, + "sourceU": -0.980097770690918, + "sourceV": 0.4984455108642578, + "canonicalU": -0.980097770690918, + "canonicalV": 0.4984455108642578 + }, + { + "row": 1, + "column": 511, + "sourceU": 1.2943105697631836, + "sourceV": -0.6254873275756836, + "canonicalU": 1.2943105697631836, + "canonicalV": -0.6254873275756836 + }, + { + "row": 767, + "column": 1, + "sourceU": -2.738156318664551, + "sourceV": 1.1959524154663086, + "canonicalU": -2.738156318664551, + "canonicalV": 1.1959524154663086 + }, + { + "row": 767, + "column": 511, + "sourceU": -5.0848894119262695, + "sourceV": -0.26341915130615234, + "canonicalU": -5.0848894119262695, + "canonicalV": -0.26341915130615234 + }, + { + "row": 0, + "column": 0, + "sourceU": -1.1939868927001953, + "sourceV": 1.7741403579711914, + "canonicalU": -1.1939868927001953, + "canonicalV": 1.7741403579711914 + }, + { + "row": 0, + "column": 1, + "sourceU": -1.051393985748291, + "sourceV": 1.4262237548828125, + "canonicalU": -1.051393985748291, + "canonicalV": 1.4262237548828125 + }, + { + "row": 0, + "column": 2, + "sourceU": -0.6949124336242676, + "sourceV": 0.9043483734130859, + "canonicalU": -0.6949124336242676, + "canonicalV": 0.9043483734130859 + } + ] + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/field.f32 b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/field.f32 new file mode 100644 index 000000000..21fb3645d Binary files /dev/null and b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/field.f32 differ diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/metadata.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/metadata.json new file mode 100644 index 000000000..8b7d63d68 --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700/metadata.json @@ -0,0 +1,41 @@ +{ + "schema": "maplibre-particle-field/v1", + "fieldId": "ecmwf-hres-wind-2026080700", + "fieldType": "WIND", + "displayDomain": "GLOBAL", + "generation": 1786060800000, + "validTime": 1786060800000, + "dataVersion": 1786060800000, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "grid": { + "width": 513, + "height": 769 + }, + "vector": { + "layout": "interleaved-uv", + "encoding": "float32-le", + "noData": -9999.0, + "unit": "m/s", + "origin": "north-west", + "rowOrder": "north-to-south", + "columnOrder": "west-to-east", + "uPositive": "east", + "vPositive": "north" + }, + "payload": { + "byteLength": 3155976, + "sha256": "363e2cd2e77501f6bceecbb4aaa68c4f9ef46998a60319ae5b7aba3548995283" + }, + "source": { + "kind": "WeatherV2 ECMWF HRES WVECBIN1", + "referenceTime": "2026080600", + "targetTime": "2026080700", + "sourceBytes": 5128989, + "resampled": false + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/conversion_summary.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/conversion_summary.json new file mode 100644 index 000000000..56fabac06 --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/conversion_summary.json @@ -0,0 +1,124 @@ +{ + "source": { + "format": "weather-v2-particle-bin-v1", + "magic": "WVECBIN1", + "targetTime": "2026080706", + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", + "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", + "uVUnit": "m/s", + "validCount": 394497, + "invalidCount": 0, + "bytes": 5128989 + }, + "output": { + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "bytes": 3155976, + "sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451", + "validCount": 394497, + "invalidCount": 0, + "resampled": false + }, + "statistics": { + "u": { + "min": -37.18255615234375, + "max": 24.125526428222656, + "mean": 0.9443698525428772 + }, + "v": { + "min": -28.227636337280273, + "max": 25.05806541442871, + "mean": 1.0168105363845825 + }, + "speed": { + "min": 0.010480343364179134, + "max": 37.76322555541992, + "mean": 6.7594804763793945 + } + }, + "sourceToCanonical": { + "lossless": true, + "maxAbsErrorU": 0.0, + "maxAbsErrorV": 0.0, + "samples": [ + { + "row": 384, + "column": 256, + "sourceU": -10.2057466506958, + "sourceV": 4.789729118347168, + "canonicalU": -10.2057466506958, + "canonicalV": 4.789729118347168 + }, + { + "row": 1, + "column": 1, + "sourceU": -0.45672082901000977, + "sourceV": 3.2181167602539062, + "canonicalU": -0.45672082901000977, + "canonicalV": 3.2181167602539062 + }, + { + "row": 1, + "column": 511, + "sourceU": 2.235017776489258, + "sourceV": -0.7363739013671875, + "canonicalU": 2.235017776489258, + "canonicalV": -0.7363739013671875 + }, + { + "row": 767, + "column": 1, + "sourceU": -1.2302308082580566, + "sourceV": 1.5909242630004883, + "canonicalU": -1.2302308082580566, + "canonicalV": 1.5909242630004883 + }, + { + "row": 767, + "column": 511, + "sourceU": -5.034683704376221, + "sourceV": -0.15012121200561523, + "canonicalU": -5.034683704376221, + "canonicalV": -0.15012121200561523 + }, + { + "row": 0, + "column": 0, + "sourceU": -0.15583133697509766, + "sourceV": 3.617917060852051, + "canonicalU": -0.15583133697509766, + "canonicalV": 3.617917060852051 + }, + { + "row": 0, + "column": 1, + "sourceU": -0.45672082901000977, + "sourceV": 3.551283836364746, + "canonicalU": -0.45672082901000977, + "canonicalV": 3.551283836364746 + }, + { + "row": 0, + "column": 2, + "sourceU": -0.6974325180053711, + "sourceV": 3.2181167602539062, + "canonicalU": -0.6974325180053711, + "canonicalV": 3.2181167602539062 + } + ] + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/field.f32 b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/field.f32 new file mode 100644 index 000000000..5f143c03f Binary files /dev/null and b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/field.f32 differ diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/metadata.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/metadata.json new file mode 100644 index 000000000..b307a05f2 --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080706/metadata.json @@ -0,0 +1,41 @@ +{ + "schema": "maplibre-particle-field/v1", + "fieldId": "ecmwf-hres-wind-2026080706", + "fieldType": "WIND", + "displayDomain": "GLOBAL", + "generation": 1786082400000, + "validTime": 1786082400000, + "dataVersion": 1786082400000, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "grid": { + "width": 513, + "height": 769 + }, + "vector": { + "layout": "interleaved-uv", + "encoding": "float32-le", + "noData": -9999.0, + "unit": "m/s", + "origin": "north-west", + "rowOrder": "north-to-south", + "columnOrder": "west-to-east", + "uPositive": "east", + "vPositive": "north" + }, + "payload": { + "byteLength": 3155976, + "sha256": "ef7a8e4b1c47f652f8f7012de16a87c16e3b25556a8f4686629f80514426c451" + }, + "source": { + "kind": "WeatherV2 ECMWF HRES WVECBIN1", + "referenceTime": "2026080600", + "targetTime": "2026080706", + "sourceBytes": 5128989, + "resampled": false + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/conversion_summary.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/conversion_summary.json new file mode 100644 index 000000000..35db277ca --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/conversion_summary.json @@ -0,0 +1,124 @@ +{ + "source": { + "format": "weather-v2-particle-bin-v1", + "magic": "WVECBIN1", + "targetTime": "2026080712", + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", + "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", + "uVUnit": "m/s", + "validCount": 394497, + "invalidCount": 0, + "bytes": 5128989 + }, + "output": { + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "bytes": 3155976, + "sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0", + "validCount": 394497, + "invalidCount": 0, + "resampled": false + }, + "statistics": { + "u": { + "min": -37.39310836791992, + "max": 24.769317626953125, + "mean": 0.7698453664779663 + }, + "v": { + "min": -28.266582489013672, + "max": 27.809932708740234, + "mean": 1.1808899641036987 + }, + "speed": { + "min": 0.024363698437809944, + "max": 37.42909240722656, + "mean": 6.384326934814453 + } + }, + "sourceToCanonical": { + "lossless": true, + "maxAbsErrorU": 0.0, + "maxAbsErrorV": 0.0, + "samples": [ + { + "row": 384, + "column": 256, + "sourceU": -10.180034637451172, + "sourceV": 4.663511276245117, + "canonicalU": -10.180034637451172, + "canonicalV": 4.663511276245117 + }, + { + "row": 1, + "column": 1, + "sourceU": -0.5417494773864746, + "sourceV": 0.767613410949707, + "canonicalU": -0.5417494773864746, + "canonicalV": 0.767613410949707 + }, + { + "row": 1, + "column": 511, + "sourceU": -0.5780353546142578, + "sourceV": 0.26222801208496094, + "canonicalU": -0.5780353546142578, + "canonicalV": 0.26222801208496094 + }, + { + "row": 767, + "column": 1, + "sourceU": -1.8420743942260742, + "sourceV": 0.3505821228027344, + "canonicalU": -1.8420743942260742, + "canonicalV": 0.3505821228027344 + }, + { + "row": 767, + "column": 511, + "sourceU": -4.581581115722656, + "sourceV": -1.1498479843139648, + "canonicalU": -4.581581115722656, + "canonicalV": -1.1498479843139648 + }, + { + "row": 0, + "column": 0, + "sourceU": -0.5939555168151855, + "sourceV": 1.1400480270385742, + "canonicalU": -0.5939555168151855, + "canonicalV": 1.1400480270385742 + }, + { + "row": 0, + "column": 1, + "sourceU": -0.48954296112060547, + "sourceV": 1.0159034729003906, + "canonicalU": -0.48954296112060547, + "canonicalV": 1.0159034729003906 + }, + { + "row": 0, + "column": 2, + "sourceU": -0.48954296112060547, + "sourceV": 0.767613410949707, + "canonicalU": -0.48954296112060547, + "canonicalV": 0.767613410949707 + } + ] + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/field.f32 b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/field.f32 new file mode 100644 index 000000000..1cd3ac253 Binary files /dev/null and b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/field.f32 differ diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/metadata.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/metadata.json new file mode 100644 index 000000000..cc1efe4fb --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080712/metadata.json @@ -0,0 +1,41 @@ +{ + "schema": "maplibre-particle-field/v1", + "fieldId": "ecmwf-hres-wind-2026080712", + "fieldType": "WIND", + "displayDomain": "GLOBAL", + "generation": 1786104000000, + "validTime": 1786104000000, + "dataVersion": 1786104000000, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "grid": { + "width": 513, + "height": 769 + }, + "vector": { + "layout": "interleaved-uv", + "encoding": "float32-le", + "noData": -9999.0, + "unit": "m/s", + "origin": "north-west", + "rowOrder": "north-to-south", + "columnOrder": "west-to-east", + "uPositive": "east", + "vPositive": "north" + }, + "payload": { + "byteLength": 3155976, + "sha256": "d86a6a51fd5216cf9368756962a5366fd32a627afd40cfde235ba7358c02c5a0" + }, + "source": { + "kind": "WeatherV2 ECMWF HRES WVECBIN1", + "referenceTime": "2026080600", + "targetTime": "2026080712", + "sourceBytes": 5128989, + "resampled": false + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/conversion_summary.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/conversion_summary.json new file mode 100644 index 000000000..d1d606279 --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/conversion_summary.json @@ -0,0 +1,124 @@ +{ + "source": { + "format": "weather-v2-particle-bin-v1", + "magic": "WVECBIN1", + "targetTime": "2026080718", + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "dtype": "float32 little-endian u plane, v plane, speed plane; uint8 valid plane", + "arrayOrder": "row-major; WeatherV2 tile y increases southward, so rows are north-to-south", + "uVUnit": "m/s", + "validCount": 394497, + "invalidCount": 0, + "bytes": 5128989 + }, + "output": { + "width": 513, + "height": 769, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "bytes": 3155976, + "sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b", + "validCount": 394497, + "invalidCount": 0, + "resampled": false + }, + "statistics": { + "u": { + "min": -31.956016540527344, + "max": 28.557456970214844, + "mean": 1.212062120437622 + }, + "v": { + "min": -26.13941192626953, + "max": 27.131576538085938, + "mean": 1.1284888982772827 + }, + "speed": { + "min": 0.0210099034011364, + "max": 33.67807388305664, + "mean": 6.403453826904297 + } + }, + "sourceToCanonical": { + "lossless": true, + "maxAbsErrorU": 0.0, + "maxAbsErrorV": 0.0, + "samples": [ + { + "row": 384, + "column": 256, + "sourceU": -10.21538257598877, + "sourceV": 2.3842287063598633, + "canonicalU": -10.21538257598877, + "canonicalV": 2.3842287063598633 + }, + { + "row": 1, + "column": 1, + "sourceU": -0.18732452392578125, + "sourceV": 0.2720670700073242, + "canonicalU": -0.18732452392578125, + "canonicalV": 0.2720670700073242 + }, + { + "row": 1, + "column": 511, + "sourceU": 0.460113525390625, + "sourceV": 0.4729623794555664, + "canonicalU": 0.460113525390625, + "canonicalV": 0.4729623794555664 + }, + { + "row": 767, + "column": 1, + "sourceU": -1.4576325416564941, + "sourceV": -0.03371381759643555, + "canonicalU": -1.4576325416564941, + "canonicalV": -0.03371381759643555 + }, + { + "row": 767, + "column": 511, + "sourceU": -2.9916043281555176, + "sourceV": 0.8597030639648438, + "canonicalU": -2.9916043281555176, + "canonicalV": 0.8597030639648438 + }, + { + "row": 0, + "column": 0, + "sourceU": -1.020096778869629, + "sourceV": 0.9125776290893555, + "canonicalU": -1.020096778869629, + "canonicalV": 0.9125776290893555 + }, + { + "row": 0, + "column": 1, + "sourceU": -0.49961376190185547, + "sourceV": 0.6990737915039062, + "canonicalU": -0.49961376190185547, + "canonicalV": 0.6990737915039062 + }, + { + "row": 0, + "column": 2, + "sourceU": -0.08322811126708984, + "sourceV": 0.41440296173095703, + "canonicalU": -0.08322811126708984, + "canonicalV": 0.41440296173095703 + } + ] + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/field.f32 b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/field.f32 new file mode 100644 index 000000000..6b948f5ae Binary files /dev/null and b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/field.f32 differ diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/metadata.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/metadata.json new file mode 100644 index 000000000..592327710 --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080718/metadata.json @@ -0,0 +1,41 @@ +{ + "schema": "maplibre-particle-field/v1", + "fieldId": "ecmwf-hres-wind-2026080718", + "fieldType": "WIND", + "displayDomain": "GLOBAL", + "generation": 1786125600000, + "validTime": 1786125600000, + "dataVersion": 1786125600000, + "bounds": { + "west": 112.5, + "south": 0.0, + "east": 157.5, + "north": 55.77657301866769 + }, + "grid": { + "width": 513, + "height": 769 + }, + "vector": { + "layout": "interleaved-uv", + "encoding": "float32-le", + "noData": -9999.0, + "unit": "m/s", + "origin": "north-west", + "rowOrder": "north-to-south", + "columnOrder": "west-to-east", + "uPositive": "east", + "vPositive": "north" + }, + "payload": { + "byteLength": 3155976, + "sha256": "970785b0334a9d1511b8897874a81f681814a32b1e0b90e37cf64b77ca9fa39b" + }, + "source": { + "kind": "WeatherV2 ECMWF HRES WVECBIN1", + "referenceTime": "2026080600", + "targetTime": "2026080718", + "sourceBytes": 5128989, + "resampled": false + } +} diff --git a/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/dataset.json b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/dataset.json new file mode 100644 index 000000000..d32d8d4dc --- /dev/null +++ b/platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/dataset.json @@ -0,0 +1,14 @@ +{ + "schema": "maplibre-particle-test-dataset/v1", + "datasetId": "ecmwf-hres-wind-20260807", + "source": "WeatherV2 ECMWF HRES WVECBIN1", + "referenceTime": "2026080600", + "validTimes": [ + "2026080700", + "2026080706", + "2026080712", + "2026080718" + ], + "snapshotLayout": "/metadata.json + /field.f32", + "notes": "开发期离线 canonical 测试索引;不是 Weather Manifest,也不进入 ParticleFieldSnapshot。" +} diff --git a/tools/weatherv2_wind_canonical/README.md b/tools/weatherv2_wind_canonical/README.md new file mode 100644 index 000000000..8b1769393 --- /dev/null +++ b/tools/weatherv2_wind_canonical/README.md @@ -0,0 +1,37 @@ +# WeatherV2 风场转 MapLibre canonical 测试数据工具 + +此工具只用于开发期离线转换:将 WeatherV2 的完整 `wind_vectors.bin`(`WVECBIN1`)转成 STEP-053 已冻结的 `maplibre-particle-field/v1`,不属于 MapLibre Runtime,也不访问网络、不扫描下载目录、更不修改 WeatherV2 原始数据。 + +## 输入 + +输入为一个 WeatherV2 reference/run 目录及其中一个 `YYYYMMDDHH` valid time: + +```text +//wind_vectors.bin +//wind_vectors.bin.header.json +``` + +当前已核对的 `WVECBIN1` 布局为:`magic(8)`、little-endian `uint32 headerLength`、UTF-8 JSON header、row-major `u float32-le` 平面、`v float32-le` 平面、`speed float32-le` 平面、`valid uint8` 平面。WeatherV2 `bin_export.py` 将同一 `decode_wind()` U/V grid 写出;QBIN 和 NSFT 也从同一 decoder U/V 生成。QBIN metadata 明确其 U/V 单位为 m/s,因此本工具保持 m/s,不进行 kt 换算。 + +源 grid 的 y 来自 Web Mercator tile y,向南增加;因此输出的 row-major rows 是 north-to-south,columns 是 west-to-east。U=eastward-positive,V=northward-positive。 + +## 输出 + +每个输出目录包含: + +- `metadata.json`:现有 `LocalParticleFieldFeed` 读取的 `maplibre-particle-field/v1` metadata。 +- `field.f32`:`float32-le interleaved-uv`,顺序为 `U0,V0,U1,V1,...`。 +- `conversion_summary.json`:源/输出 bounds、尺寸、统计值、valid/noData 计数、抽样一致性与 SHA-256。 + +本工具不重采样。每个源 `valid=0` 或非有限 U/V 的 cell 会写为既有 canonical `noData=-9999.0` 的 U/V 对;不会错误变成静风 `(0,0)`。有效样本按 float32 bit-exact 复制。 + +## 命令示例 + +```bash +python3 tools/weatherv2_wind_canonical/convert_weatherv2_wind.py \ + --run-dir /root/sourceserver/weatherV2/src/dev/_runtime/dev_pipeline_data/datasets/ecmwf-hres/wind/2026080600 \ + --valid-time 2026080700 \ + --output platform/android/MapLibreAndroidTestApp/src/main/assets/integration/particle-weatherv2/ecmwf-hres-wind-20260807/2026080700 +``` + +相同输入会产生相同 payload SHA-256。错误会输出明确消息并以非零状态退出。 diff --git a/tools/weatherv2_wind_canonical/convert_weatherv2_wind.py b/tools/weatherv2_wind_canonical/convert_weatherv2_wind.py new file mode 100644 index 000000000..efe3eae30 --- /dev/null +++ b/tools/weatherv2_wind_canonical/convert_weatherv2_wind.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +"""Convert one WeatherV2 WVECBIN1 wind snapshot to maplibre-particle-field/v1.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import struct +import sys +from datetime import datetime, timezone +from pathlib import Path + +import numpy as np + + +MAGIC = b"WVECBIN1" +NO_DATA = np.float32(-9999.0) + + +def fail(message: str) -> "None": + raise ValueError(message) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--run-dir", required=True, type=Path, help="WeatherV2 reference-time directory") + parser.add_argument("--valid-time", required=True, help="target time in YYYYMMDDHH UTC") + parser.add_argument("--output", required=True, type=Path, help="canonical snapshot output directory") + return parser.parse_args() + + +def require_mapping(value: object, name: str) -> dict: + if not isinstance(value, dict): + fail(f"{name} must be an object") + return value + + +def read_source(run_dir: Path, valid_time: str) -> tuple[dict, np.ndarray, np.ndarray, np.ndarray, np.ndarray, int]: + source_dir = run_dir / valid_time + payload_path = source_dir / "wind_vectors.bin" + sidecar_path = source_dir / "wind_vectors.bin.header.json" + if not payload_path.is_file() or not sidecar_path.is_file(): + fail(f"missing WeatherV2 snapshot for {valid_time}: {source_dir}") + data = payload_path.read_bytes() + if len(data) < 12 or data[:8] != MAGIC: + fail(f"unsupported wind_vectors.bin magic in {payload_path}") + header_length = struct.unpack_from(" len(data): + fail("WVECBIN1 header length exceeds file length") + header = json.loads(data[12:header_end].decode("utf-8")) + sidecar = json.loads(sidecar_path.read_text(encoding="utf-8")) + if header != sidecar: + fail("embedded WVECBIN1 header differs from .header.json sidecar") + if header.get("format") != "weather-v2-particle-bin-v1" or header.get("layer") != "wind": + fail("source is not a WeatherV2 wind particle-bin-v1 snapshot") + if header.get("target_time") != valid_time: + fail(f"source target_time mismatch: {header.get('target_time')!r} != {valid_time!r}") + width, height = int(header.get("width", 0)), int(header.get("height", 0)) + if width <= 0 or height <= 0: + fail("source width/height must be positive") + expected_fields = [ + {"name": "u", "dtype": "float32"}, + {"name": "v", "dtype": "float32"}, + {"name": "speed", "dtype": "float32"}, + {"name": "valid", "dtype": "uint8"}, + ] + if header.get("fields") != expected_fields: + fail(f"unsupported WVECBIN1 fields: {header.get('fields')!r}") + count = width * height + expected_bytes = header_end + count * 4 * 3 + count + if len(data) != expected_bytes: + fail(f"WVECBIN1 byte length mismatch: expected {expected_bytes}, got {len(data)}") + offset = header_end + u = np.frombuffer(data, dtype=" int: + try: + return int(datetime.strptime(valid_time, "%Y%m%d%H").replace(tzinfo=timezone.utc).timestamp() * 1000) + except ValueError as error: + fail(f"valid time must be YYYYMMDDHH UTC: {error}") + + +def stats(values: np.ndarray, valid: np.ndarray) -> dict: + chosen = values[valid] + if not chosen.size: + fail("source has no valid samples") + return {"min": float(chosen.min()), "max": float(chosen.max()), "mean": float(chosen.mean())} + + +def sample_indices(valid: np.ndarray) -> list[tuple[int, int]]: + height, width = valid.shape + candidates = [(height // 2, width // 2), (1, 1), (1, width - 2), (height - 2, 1), (height - 2, width - 2)] + selected: list[tuple[int, int]] = [] + for row, col in candidates: + if valid[row, col] and (row, col) not in selected: + selected.append((row, col)) + for row, col in zip(*np.nonzero(valid)): + if (int(row), int(col)) not in selected: + selected.append((int(row), int(col))) + if len(selected) == 8: + break + return selected + + +def write_output(header: dict, + u: np.ndarray, + v: np.ndarray, + speed: np.ndarray, + valid: np.ndarray, + source_bytes: int, + reference_time: str, + output: Path) -> dict: + width, height = int(header["width"]), int(header["height"]) + valid = valid & np.isfinite(u) & np.isfinite(v) + payload = np.empty((height, width, 2), dtype=" int: + args = parse_args() + try: + header, u, v, speed, valid, source_bytes = read_source(args.run_dir, args.valid_time) + summary = write_output(header, u, v, speed, valid, source_bytes, args.run_dir.name, args.output) + print(json.dumps({"ok": True, "targetTime": header["target_time"], "output": str(args.output), "sha256": summary["output"]["sha256"], "validCount": summary["output"]["validCount"], "invalidCount": summary["output"]["invalidCount"]}, ensure_ascii=False)) + return 0 + except (OSError, ValueError, json.JSONDecodeError) as error: + print(f"error: {error}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/粒子开发记录.md b/粒子开发记录.md index 63821985d..84f4f7e2a 100644 --- a/粒子开发记录.md +++ b/粒子开发记录.md @@ -398,3 +398,9 @@ - `fieldUploadCount=4 → 5 → 6` 证明正常 boundary 每次仅上传一个 NEW_T1。唯一 `simulationResetCount=1` 属于旧 fixture 切到 Host dataset 的允许初始化,不是正常 production boundary 回归。 - `HOST DUPLICATE` 返回 `DUPLICATE_NOOP` 且 upload/reset 不增长;`HOST INVALID` 经 Adapter reject fail-closed,last valid field 与 Renderer continuity(`stateSeed=CONTINUE`)保持,无黑屏、无 crash。 - 用户正式风场数据集仍未提供、未验证,但不阻塞 contract 阶段关闭;后续可在同一 canonical contract 下追加。MIX 文本滞后及 WAITING/INVALID 视觉为 `NON-BLOCKING observability`,本次未修改功能代码或执行 Android/Jest。 + +## 2026-09-04 — STEP-054 真实 WeatherV2 风数据 Canonical 测试 asset + +- 只读核对 WeatherV2 `bin_export.py`、`decode_wind()`、QBIN/NSFT export 后,确认 `wind_vectors.bin` 为完整拼接 `WVECBIN1` grid:row-major U/V/speed float32-le planes + valid uint8 plane,U=eastward、V=northward,单位 m/s;QBIN/NSFT 均由同一 decoded U/V Field 生成。 +- 从 run `2026080600` 生成真实 ECMWF HRES `2026080700/06/12/18` 四帧。原始日本及周边 bounds 为 112.5/0.0/157.5/55.776573,`513×769`,不裁剪、不重采样;输出 canonical interleaved U/V Android assets,每帧 3,155,976 bytes。 +- 转换器独立放在 `tools/weatherv2_wind_canonical/`,保留 valid/noData:invalid 写既有 -9999 sentinel,不伪造为 0 风;四帧 hash 不同,重复 T0 hash 相同,source→canonical 抽样误差 0。未改 Core/Renderer/Android,也未执行 Android/Jest;等待下一阶段集中 Android 集成验收。