diff --git a/MindEarth/README.md b/MindEarth/README.md index e8aa7d91e877ea9d2b744b063ce5ab856eaa4e87..79bb12fe6bd73827e1ce70c617f1d38283a3de48 100644 --- a/MindEarth/README.md +++ b/MindEarth/README.md @@ -17,7 +17,6 @@ - [Version Dependency](#Version-Dependency) - [Install Dependency](#Install-Dependency) - [Hardware](#Hardware) - - [pip install](#pip-install) - [source code install](#source-code-install) - [Community](#Community) - [Core Contributor](#Core-Contributor) @@ -44,23 +43,27 @@ MindEarth is an earth science suite developed based on [MindSpore](https://www.m ### Precipitation Nowcasting -| Case | Dataset | Network | GPU | NPU | -|:----------------------:|:--------------------------:|:---------------:|:-------:|:------:| -|DGMs | Radar dataset | GAN、ConvGRU | ✔️ | ✔️ | +| Case | Dataset | Network | NPU | +|:----------------------:|:--------------------------:|:---------------:|:------:| +|DGMs | Radar dataset | GAN、ConvGRU | ✔️ | +| NowcastNet | USA-MRMS dataset | GAN、two-path U-Net | ✔️ | +| PreDiff | SEVIR_LR dataset | LDM、 Earthformer | ✔️ | ### Medium-range Forecast -| Case | Dataset | Network | GPU | NPU | -|:----------------------:|:-------------------------------------:|:---------------:|:-------:|:------:| -|FourCastNet | ERA5 Reanalysis Dataset | AFNO | ✔️ | ✔️ | -|ViT-KNO | ERA5 Reanalysis Dataset | ViT | ✔️ | ✔️ | -|GraphCast | ERA5 Reanalysis Dataset | GNN | ✔️ | ✔️ | +| Case | Dataset | Network | NPU | +|:----------------------:|:-------------------------------------:|:---------------:|:------:| +|FourCastNet | ERA5 Reanalysis Dataset | AFNO | ✔️ | +|ViT-KNO | ERA5 Reanalysis Dataset | ViT | ✔️ | +|GraphCast | ERA5 Reanalysis Dataset | GNN | ✔️ | +|FuXi | ERA5 Reanalysis Dataset | CNN、Swin Transformer V2 | ✔️ | +|SKNO | ERA5 Reanalysis Dataset | SKNO | ✔️ | -### Data Preprocessing +### Earthquake Early Warning -| Case | Dataset | Network | GPU | NPU | -|:--------------------------:|:--------------------------:|:---------------:|:-------:|:------:| -| DEM Super-resolution | NASADEM、GEBCO_2021 | SRGAN | ✔️ | ✔️ | +| Case | Dataset | Network | NPU | +|:----------------------:|:--------------------------:|:---------------:|:------:| +|G-TEAM | Diting 2.0 Dataset | CNN、Transformer | ✔️ | ## **Installation** @@ -70,7 +73,9 @@ Because MindEarth is dependent on MindSpore, please click [MindSpore Download Pa | MindEarth | Branch | MindSpore |Python | |:--------:|:----------------------------------------------------------------------:|:-----------:|:-------:| -| master | [master](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | \ | \>=3.7 | +| master | [master](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | >=2.5.0(please read model's readme to choose the suitable version) | \>=3.11 | +| 0.3.0 | [r0.7](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | >=2.5.0 | \>=3.11 | +| 0.2.0 | [r0.6](https://gitee.com/mindspore/mindscience/tree/r0.6/MindEarth) | >=2.2.0 | \>=3.9 | | 0.1.0 | [r0.5](https://gitee.com/mindspore/mindscience/tree/r0.5/MindEarth) | >=1.8.1 | \>=3.7 | ### Install Dependency @@ -83,20 +88,11 @@ pip install -r requirements.txt | Hardware | OS | Status | |:--------------| :-------------- | :--- | -| Ascend 910 | Ubuntu-x86 | ✔️ | +| Ascend, 64G | Ubuntu-x86 | ✔️ | | | Ubuntu-aarch64 | ✔️ | | | EulerOS-aarch64 | ✔️ | | | CentOS-x86 | ✔️ | | | CentOS-aarch64 | ✔️ | -| GPU CUDA 11.1 | Ubuntu-x86 | ✔️ | - -### **pip install** - -```bash -# gpu and ascend are supported -export DEVICE_NAME=gpu -pip install mindearth_${DEVICE_NAME} -``` ### **source code install** @@ -113,13 +109,6 @@ cd {PATH}/mindscience/MindEarth bash build.sh -e ascend -j8 ``` -- Compile in GPU backend. - -```bash -export CUDA_PATH={your_cuda_path} -bash build.sh -e gpu -j8 -``` - - Install the compiled .whl file. ```bash @@ -133,7 +122,7 @@ pip install mindearth_*.whl Thanks goes to these wonderful people 🧑‍🤝‍🧑: -yufan, wangzidong, liuhongsheng, zhouhongye, liulei, libokai, chengqiang, dongyonghan, zhouchuansai +yufan, wangzidong, liuhongsheng, zhouhongye, liulei, libokai, chengqiang, dongyonghan, zhouchuansai, liuruoyan ## **Contribution Guide** diff --git a/MindEarth/README_CN.md b/MindEarth/README_CN.md index 0765f5a30eb86cd33c242b4397a49861e29b24d4..e300ecf9f5c2ffba57add5416a70627a1aa678b4 100644 --- a/MindEarth/README_CN.md +++ b/MindEarth/README_CN.md @@ -16,7 +16,6 @@ - [安装教程](#安装教程) - [依赖安装](#依赖安装) - [硬件支持情况](#硬件支持情况) - - [pip安装](#pip安装) - [源码安装](#源码安装) - [社区](#社区) - [核心贡献者](#核心贡献者) @@ -43,23 +42,27 @@ MindEarth是基于[昇思MindSpore](https://www.mindspore.cn/)开发的地球科 ### 短临降水 -| 案例 | 数据集 | 模型架构 | GPU | NPU | -|:----------------------:|:--------------------------:|:---------------:|:-------:|:------:| -| DGMs | 雷达数据 | GAN、ConvGRU | ✔️ | ✔️ | +| 案例 | 数据集 | 模型架构 | NPU | +|:----------------------:|:--------------------------:|:---------------:|:------:| +| DGMs | 雷达数据 | GAN、ConvGRU | ✔️ | +| NowcastNet | USA-MRMS数据集 | GAN、two-path U-Net | ✔️ | +| PreDiff | SEVIR_LR数据集 | LDM、 Earthformer | ✔️ | ### 中期气象预报 -| 案例 | 数据集 | 模型架构 | GPU | NPU | -|:----------------------:|:--------------------------:|:---------------:|:-------:|:------:| -|FourCastNet | ERA5再分析数据集 | AFNO | ✔️ | ✔️ | -|ViT-KNO | ERA5再分析数据集 | ViT | ✔️ | ✔️ | -|GraphCast | ERA5再分析数据集 | GNN | ✔️ | ✔️ | +| 案例 | 数据集 | 模型架构 | NPU | +|:----------------------:|:--------------------------:|:---------------:|:------:| +|FourCastNet | ERA5再分析数据集 | AFNO | ✔️ | +|ViT-KNO | ERA5再分析数据集 | ViT | ✔️ | +|GraphCast | ERA5再分析数据集 | GNN | ✔️ | +|FuXi | ERA5再分析数据集 | CNN、Swin Transformer V2 | ✔️ | +|SKNO | ERA5再分析数据集 | SKNO | ✔️ | -### 数据前处理 +### 地震预警 -| 案例 | 数据集 | 模型架构 | GPU | NPU | -|:-------------------------:|:--------------------------:|:---------------:|:-------:|:------:| -| DEM超分辨率 | NASADEM、GEBCO_2021 | SRGAN | ✔️ | ✔️ | +| 案例 | 数据集 | 模型架构 | NPU | +|:----------------------:|:--------------------------:|:---------------:|:------:| +|G-TEAM | diting 2.0数据集 | CNN、Transformer | ✔️ | ## **安装教程** @@ -69,8 +72,10 @@ MindEarth是基于[昇思MindSpore](https://www.mindspore.cn/)开发的地球科 | MindEarth | 分支 | MindSpore | Python | |:--------:|:----------------------------------------------------------------------:|:-----------:|:------:| -| master | [master](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | \ | \>=3.7 | -| 0.1.0 | [r0.5](https://gitee.com/mindspore/mindscience/tree/r0.5/MindEarth) | >=1.8.1 | \>=3.7 | +| master | [master](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | >=2.5.0(在模型readme查看具体版本需求) | >=3.11 | +| 0.3.0 | [r0.7](https://gitee.com/mindspore/mindscience/tree/master/MindEarth) | >=2.5.0 | >=3.11 | +| 0.2.0 | [r0.6](https://gitee.com/mindspore/mindscience/tree/r0.6/MindEarth) | >=2.2.0 | \>=3.9 | +| 0.1.0 | [r0.5](https://gitee.com/mindspore/mindscience/tree/r0.5/MindEarth) | >=1.8.1 | \>=3.9 | ### 依赖安装 @@ -82,20 +87,11 @@ pip install -r requirements.txt | 硬件平台 | 操作系统 | 状态 | | :------------ | :-------------- | :--- | -| Ascend 910 | Ubuntu-x86 | ✔️ | +| Ascend, 64G | Ubuntu-x86 | ✔️ | | | Ubuntu-aarch64 | ✔️ | | | EulerOS-aarch64 | ✔️ | | | CentOS-x86 | ✔️ | | | CentOS-aarch64 | ✔️ | -| GPU CUDA 11.1 | Ubuntu-x86 | ✔️ | - -### pip安装 - -```bash -# gpu and ascend are supported -export DEVICE_NAME=gpu -pip install mindearth_${DEVICE_NAME} -``` ### 源码安装 @@ -112,13 +108,6 @@ cd {PATH}/mindscience/MindEarth bash build.sh -e ascend -j8 ``` -- 编译GPU后端源码。 - -```bash -export CUDA_PATH={your_cuda_path} -bash build.sh -e gpu -j8 -``` - - 安装编译所得whl包。 ```bash @@ -132,7 +121,7 @@ pip install mindearth_*.whl 感谢以下开发者做出的贡献 🧑‍🤝‍🧑： -yufan, wangzidong, liuhongsheng, zhouhongye, liulei, libokai, chengqiang, dongyonghan, zhouchuansai +yufan, wangzidong, liuhongsheng, zhouhongye, liulei, libokai, chengqiang, dongyonghan, zhouchuansai, liuruoyan ## **贡献指南** diff --git a/MindEarth/RELEASE.md b/MindEarth/RELEASE.md index 9d69dee3b5b014c217d108eaddab0084fe2f3773..408107e012fe52bbe946cab6509994b472eded39 100644 --- a/MindEarth/RELEASE.md +++ b/MindEarth/RELEASE.md @@ -4,6 +4,31 @@ ENGLISH | [简体中文](RELEASE_CN.md) MindSpore Earth is a earth science suite developed based on MindSpore. It supports tasks such as multi-temporal-spatial-scale meteorological forecasting and data preprocessing, committed to efficiently enabling AI+meteorological/ocean fusion research. +## MindSpore Earth 0.3.0 Release Notes + +### Major Feature and Improvements + +#### Short-Term Precipitation Forecast + +- [STABLE] [PreDiff](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/nowcasting/PreDiff): PreDiff is a novel latent diffusion model (LDM)-based method for short-term precipitation nowcasting. This approach proposes a two-stage pipeline specifically designed for data-driven Earth system forecasting models. Additionally, it introduces a knowledge control mechanism to guide the denoising diffusion process in the latent space. This mechanism ensures that the generated predictions align more closely with domain-specific knowledge, thereby enhancing the credibility of the model's forecasts. + +#### Earthquake Early Warning + +- [RESEARCH]: [G-TEAM](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/earthquake/G-TEAM) is a data-driven earthquake early warning model whose core architecture combines the sequence modeling capability of Transformer and the spatial information extraction capability of GNN, so that the model can not only capture the time sequence characteristics of seismic waveforms, but also use the propagation relationship of seismic waves in the station network to improve the prediction precision of magnitude and epicenter location. It can quickly provide epicenter location, magnitude and seismic intensity distribution within 3 seconds after an earthquake occurs. + +#### Medium-Range Global Predictions + +- [STABLE] [GraphCast](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): The graphcastTp mode has been newly added. In the downstream medium-term precipitation cases, a grid of 0.5°×0.5° is adopted, and ERA5 data with an input resolution of 360×720 is used for training. +- [STABLE] [SKNO](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/skno): A new SKNO model has been added, which integrates the KNO model and the SHT operator. The SKNO model is developed based on the 16-year assimilation reanalysis dataset ERA5. It can predict global meteorological conditions with a temporal resolution of 6 hours and a spatial resolution of 1.4 degrees. The prediction results include indicators such as temperature, humidity, and wind speed. + +### Contributors + +Thanks to the following developers for their contributions: + +hsliu_ustc, hong-ye-zhou, liulei277, kevinli123, Zhou Chuansai, alancheng511, Cui Yinghao, xingzhongfan, cmy_melody, Liu Ruoyan + +Contributions to the project in any form are welcome! + ## MindSpore Earth 0.2.0 Release Notes ### Major Feature and Improvements @@ -31,19 +56,19 @@ Contributions to the project in any form are welcome! ### Major Feature and Improvements -#### Nowcasting precipitation predictions +#### Nowcasting Precipitation Predictions - [STABLE] [Dgmr](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/nowcasting/dgmr): Dgmr (Deep Generative Model of Radar Network) is a deep generative model for the probabilistic nowcasting of precipitation from radar developed by researchers from DeepMind. The main body of the model is a generator, which is trained adversarially with temporal and spatial discriminator losses and additional regularization terms. The model learns contextual representations from the first four frames of the radar sequence, which are used as input to the sampler. The sampler is a recurrent network composed of convolutional gated recurrent units (GRU), which takes context representations and latent vectors sampled from a Gaussian distribution as input to predict 18 future radar fields. -#### Medium-range global predictions +#### Medium-range Global Predictions - [STABLE] [FourCastNet](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/fourcastnet): FourCastNet adopts the adaptive Fourier neural operator AFNO, which is an improvement on the Vision Transformer model. It constructs a continuous global convolution of the mixing operation steps and effectively implements it through FFT in the Fourier domain, reducing the spatial mixing complexity to O(NlogN). This model is the first AI prediction model that can compare its prediction accuracy with the high-resolution integrated prediction system (IFS) model of the European Centre for Medium Range Weather Forecasting (ECMWF). - [STABLE] [ViT-KNO](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/koopman_vit): Koopman Neural Operator is a lightweight, grid independent model designed based on Koopman's global linearization theory and combined with neural operator ideas. This model was developed in collaboration between Huawei ACS lab and Tsinghua University. By embedding complex dynamics into the linear structure to constrain the reconstruction process, this model can capture complex nonlinear behavior while maintaining its lightweight and computational effectiveness. Compared with FNO, KNO has more efficient training performance and better prediction accuracy. - [STABLE] [GraphCast](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): GraphCast was proposed by Google DeepMind, which uses GNN to autoregressively generate prediction results in the "encoding-processing-decoding" architecture. The encoder maps the latitude-longitude input grid of meteorological elements at historical times to a multi-scale icosahedral grid representation. The processor performs multiple rounds of message passing on a multi grid representation. The decoder maps the multi grid representation back to the latitude-longitude grid as the prediction for the next time step. In addition, MindEarth has implemented multi-step iterative training to reduce model error accumulation in response to the attenuation of multiple prediction accuracy. -#### Super-resolution reconstruction of global DEM +#### Super-resolution Reconstruction of Global DEM -- [STABLE] [DEM-SRNet](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/dem-super-resolution): DEM-SRNet is a super-resolution model of a digital elevation model. The model is based on 30m resolution NASADEM satellite images, 450m resolution GEBCO_2021 public data of the United Nations Intergovernmental Oceanographic Commission and high-resolution ocean terrain data of some areas, using a combination of deep residual pre-trained neural network and transfer learning technology to generate a global 90m high-resolution DEM. This data set can provide more accurate basic geographical information and plays a vital role in research fields such as global climate change, ocean tidal movements, and material exchange in the geosphere. +- [STABLE] [DEM-SRNet](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/dem-super-resolution): DEM-SRNet is a super-resolution model of a digital elevation model. The model is based on 30m resolution NASADEM satellite images, 450m resolution GEBCO_2021 public data of the United Nations Intergovernmental Oceanographic Commission and high-resolution ocean terrain data of some areas, using a combination of deep residual pre-trained neural network and transfer learning technology to generate a global 90m high-resolution DEM. This dataset can provide more accurate basic geographical information and plays a vital role in research fields such as global climate change, ocean tidal movements, and material exchange in the geosphere. ### Contributors diff --git a/MindEarth/RELEASE_CN.md b/MindEarth/RELEASE_CN.md index 5d7f42d1574a6838e46f3b10f0492ccf6731bdd0..6330538fad0b43f651fec6bd81d0648f6372f983 100644 --- a/MindEarth/RELEASE_CN.md +++ b/MindEarth/RELEASE_CN.md @@ -4,6 +4,31 @@ MindSpore Earth是基于昇思MindSpore开发的地球科学套件，支持多时空尺度气象预报、数据前后处理等任务，致力于高效使能AI+气象海洋的融合研究。 +## MindSpore Earth 0.3.0 Release Notes + +### 主要特性和增强 + +#### 短临降水预测 + +- [STABLE] [PreDiff](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/nowcasting/PreDiff): 新增PreDiff，是一种新颖的基于潜在扩散模型（LDM）的方法，用于降水短临预测。该方法提出了一个两阶段的pipeline，专门用于数据驱动的地球系统预测模型。同时，开发了知识控制（Knowledge control）机制，用于引导 PreDiff 在潜在空间中的去噪扩散过程。这一机制保证了生成的预测结果能更好地匹配领域相关知识，从而提高了模型预测的可信度。 + +#### 地震预警 + +- [RESEARCH]: 新增[G-TEAM](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/earthquake/G-TEAM)，是一种数据驱动的全国地震预警模型，其核心架构结合了Transformer 的序列建模能力和 GNN 的空间信息提取能力，使得模型既能捕捉地震波形的时序特征，又能利用地震波在台站网络中的传播关系，提升震级和震中位置的预测精度，能够在地震发生后 3 秒内迅速提供震中位置、震级及地震强度分布。 + +#### 中期天气预报预测 + +- [STABLE] [GraphCast](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): 新增graphcastTp模式，在下游的中期降水案例中，采用0.5°×0.5°，输入分辨率为360×720的ERA5数据进行训练。 +- [STABLE] [SKNO](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/skno): 新增SKNO模型，融合了KNO模型和SHT算子。SKNO基于16年同化再分析数据集ERA5开发，能够对6小时时间分辨率和1.4度空间分辨率的全球气象进行预测，预测结果包括温度、湿度、风度等指标。 + +### 贡献者 + +感谢以下开发者做出的贡献: + +hsliu_ustc, hong-ye-zhou, liulei277, kevinli123, Zhou Chuansai, alancheng511, Cui Yinghao, xingzhongfan, cmy_melody, Liu Ruoyan + +欢迎任何形式的贡献！ + ## MindSpore Earth 0.2.0 Release Notes ### 主要特性和增强 @@ -16,7 +41,7 @@ MindSpore Earth是基于昇思MindSpore开发的地球科学套件，支持多 - [STABLE] [GraphCast](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): 新增支持0.25°高分辨率数据，GraphCast全尺度模型训练与推理。 - [STABLE] [FuXi](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/fuxi): 新增气象模型伏羲，支持中期预报模型的训练和推理。 -- [STABLE] [Medium Precipitation](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): 新增中期降水模块案例，支持中期的降水量预报，需要基于预训练的中期大模型做为backbone。 +- [STABLE] [Medium Precipitation](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/medium-range/graphcast): 新增中期降水模块案例，支持中期的降水量预报，需要基于预训练的中期大模型作为backbone。 - [RESEARCH] [CTEFNet](https://gitee.com/mindspore/mindscience/tree/master/MindEarth/applications/climate-prediction/ensoforecast): 新增基于 CNN 与迁移学习的厄尔尼诺预测模型，将ENSO有效预报时长延长至19个月。 ### 贡献者 diff --git a/MindEarth/applications/earthquake/G-TEAM/G-TEAM.ipynb b/MindEarth/applications/earthquake/G-TEAM/G-TEAM.ipynb index 211fcfb2a527dede9e1ddee928cf4005200397fe..35a4cd2f98d59a6c17a890886682d8300c7c4973 100644 --- a/MindEarth/applications/earthquake/G-TEAM/G-TEAM.ipynb +++ b/MindEarth/applications/earthquake/G-TEAM/G-TEAM.ipynb @@ -41,7 +41,7 @@ "id": "ced04ef1-9f41-429c-817d-9ed87ad67209", "metadata": {}, "source": [ - "下述src可在[GTEAM/src](./src)下载" + "下述src可在[GTEAM/src](./src)下载。" ] }, { @@ -93,7 +93,7 @@ "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-05 08:57:36,391 - utils.py[line:179] - INFO: {'hidden_dim': 1000, 'hidden_dropout': 0.0, 'n_heads': 10, 'n_pga_targets': 15, 'output_location_dims': [150, 100, 50, 30, 3], 'output_mlp_dims': [150, 100, 50, 30, 1], 'transformer_layers': 6, 'waveform_model_dims': [500, 500, 500], 'wavelength': [[0.01, 15], [0.01, 15], [0.01, 10]], 'times': [5], 'run_with_less_data': False, 'pga': True, 'mode': 'test', 'no_event_token': False}\n", @@ -136,7 +136,7 @@ "source": [ "## 数据集准备\n", "\n", - "根据地震后发生时间选择不同台站检测的数据" + "根据地震后发生时间，选择不同台站检测的数据。" ] }, { @@ -271,7 +271,9 @@ "id": "e45e95cf-a0e8-4ce6-a093-88cc4d871578", "metadata": {}, "source": [ - "## 开始推理" + "## 开始推理\n", + "\n", + "Mag震级误差(R², RMSE, MSE), Loc震中距离误差(与震源误差（三维）RMSE, 与震源误差（三维）MAE, 与震中误差（二维）RMSE, 与震中误差（二维）MAE), Pga峰值地面加速度误差(R², RMSE, MSE)。" ] }, { @@ -288,7 +290,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-05 08:57:42,398 - forcast.py[line:115] - INFO: times: [5]\n", diff --git a/MindEarth/applications/nowcasting/PreDiff/prediff.ipynb b/MindEarth/applications/nowcasting/PreDiff/prediff.ipynb index 777d8c62a0b11c7ba569b0cdbffc6aaefcb548b1..9130e01f92d9bbb23f09ab536e2ea5336b0605a6 100644 --- a/MindEarth/applications/nowcasting/PreDiff/prediff.ipynb +++ b/MindEarth/applications/nowcasting/PreDiff/prediff.ipynb @@ -38,7 +38,7 @@ "\n", "5.模型评估与可视化\n", "\n", - "数据集可以在[PreDiff/dataset](https://deep-earth.s3.amazonaws.com/datasets/sevir_lr.zip)下载数据并保存" + "数据集可以在[PreDiff/dataset](https://deep-earth.s3.amazonaws.com/datasets/sevir_lr.zip)下载数据并保存。" ] }, { @@ -48,7 +48,7 @@ "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "/data/zmmVol1/miniconda3/envs/lryms25py311/lib/python3.11/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for type is zero.\n", @@ -82,7 +82,7 @@ "id": "415eb386-b9ef-42af-9ab6-f98c9d8151da", "metadata": {}, "source": [ - "下述src可以在[PreDiff/src](./src)下载" + "下述src可以在[PreDiff/src](./src)下载。" ] }, { @@ -147,7 +147,7 @@ "source": [ "## 模型构建\n", "\n", - "模型初始化主要包括vae模块load ckpt以及earthformer部分初始化" + "模型初始化主要包括vae模块load ckpt以及earthformer部分初始化。" ] }, { @@ -157,11 +157,9 @@ "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ - "[WARNING] ME(2231351:281473150263328,MainProcess):2025-04-07-10:32:09.988.000 [mindspore/train/serialization.py:1956] For 'load_param_into_net', remove parameter prefix name: net., continue to load.\n", - "[WARNING] ME(2231351:281473150263328,MainProcess):2025-04-07-10:32:11.431.000 [mindspore/train/serialization.py:1956] For 'load_param_into_net', remove parameter prefix name: main_model., continue to load.\n", "2025-04-07 10:32:11,466 - utils.py[line:820] - INFO: Process ID: 2231351\n", "2025-04-07 10:32:11,467 - utils.py[line:821] - INFO: {'summary_dir': './summary/prediff/single_device0', 'eval_interval': 10, 'save_ckpt_epochs': 1, 'keep_ckpt_max': 100, 'ckpt_path': '/home/lry/202542测试/PreDiff/ckpt/diffusion.ckpt', 'load_ckpt': False}\n" ] @@ -342,7 +340,7 @@ "source": [ "## 损失函数\n", "\n", - "PreDiff训练中使用mse作为loss计算，采用了梯度裁剪，并将过程封装在了DiffusionTrainer中" + "PreDiff训练中使用mse作为loss计算，采用了梯度裁剪，并将过程封装在了DiffusionTrainer中。" ] }, { @@ -513,7 +511,7 @@ "source": [ "## 模型训练\n", "\n", - "在本教程中，我们使用DiffusionTrainer对模型进行训练" + "在本教程中，我们使用DiffusionTrainer对模型进行训练。" ] }, { @@ -523,7 +521,7 @@ "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:32:36,351 - 4106154625.py[line:46] - INFO: Initializing training process...\n" @@ -537,7 +535,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:09,378 - 4106154625.py[line:64] - INFO: epoch: 0 step: 0, loss: 1.0008465\n" @@ -551,7 +549,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:16,871 - 4106154625.py[line:64] - INFO: epoch: 0 step: 1, loss: 1.0023363\n", @@ -566,7 +564,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:20,513 - 4106154625.py[line:64] - INFO: epoch: 0 step: 3, loss: 0.99787366\n", @@ -585,7 +583,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:31,432 - 4106154625.py[line:64] - INFO: epoch: 0 step: 9, loss: 1.0003254\n", @@ -603,7 +601,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:40,962 - 4106154625.py[line:64] - INFO: epoch: 0 step: 14, loss: 1.0007098\n", @@ -621,7 +619,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:34:50,720 - 4106154625.py[line:64] - INFO: epoch: 0 step: 19, loss: 0.9934994\n", @@ -640,7 +638,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:01,964 - 4106154625.py[line:64] - INFO: epoch: 0 step: 25, loss: 0.99265075\n", @@ -659,7 +657,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:12,883 - 4106154625.py[line:64] - INFO: epoch: 0 step: 31, loss: 0.98547524\n", @@ -678,7 +676,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:24,417 - 4106154625.py[line:64] - INFO: epoch: 0 step: 37, loss: 0.98045284\n", @@ -696,7 +694,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:33,513 - 4106154625.py[line:64] - INFO: epoch: 0 step: 42, loss: 0.9717276\n", @@ -715,7 +713,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:45,249 - 4106154625.py[line:64] - INFO: epoch: 0 step: 48, loss: 0.9612762\n", @@ -733,7 +731,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:35:55,202 - 4106154625.py[line:64] - INFO: epoch: 0 step: 53, loss: 0.9575224\n", @@ -751,7 +749,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:04,717 - 4106154625.py[line:64] - INFO: epoch: 0 step: 58, loss: 0.94447565\n", @@ -770,7 +768,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:15,759 - 4106154625.py[line:64] - INFO: epoch: 0 step: 64, loss: 0.94463414\n", @@ -789,7 +787,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:26,527 - 4106154625.py[line:64] - INFO: epoch: 0 step: 70, loss: 0.9313204\n", @@ -807,7 +805,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:35,917 - 4106154625.py[line:64] - INFO: epoch: 0 step: 75, loss: 0.9491191\n", @@ -826,7 +824,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:46,501 - 4106154625.py[line:64] - INFO: epoch: 0 step: 81, loss: 0.92167306\n", @@ -844,7 +842,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:36:55,996 - 4106154625.py[line:64] - INFO: epoch: 0 step: 86, loss: 0.92873365\n", @@ -863,7 +861,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:06,972 - 4106154625.py[line:64] - INFO: epoch: 0 step: 92, loss: 0.9185343\n", @@ -881,7 +879,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:16,919 - 4106154625.py[line:64] - INFO: epoch: 0 step: 97, loss: 0.92682004\n", @@ -899,7 +897,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:26,159 - 4106154625.py[line:64] - INFO: epoch: 0 step: 102, loss: 0.9083508\n", @@ -919,7 +917,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:37,562 - 4106154625.py[line:64] - INFO: epoch: 0 step: 109, loss: 0.89940923\n", @@ -938,7 +936,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:47,338 - 4106154625.py[line:64] - INFO: epoch: 0 step: 115, loss: 0.88197625\n", @@ -957,7 +955,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:37:57,735 - 4106154625.py[line:64] - INFO: epoch: 0 step: 121, loss: 0.878676\n", @@ -976,7 +974,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:07,435 - 4106154625.py[line:64] - INFO: epoch: 0 step: 127, loss: 0.87677616\n", @@ -995,7 +993,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:17,224 - 4106154625.py[line:64] - INFO: epoch: 0 step: 133, loss: 0.8630925\n", @@ -1015,7 +1013,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:28,139 - 4106154625.py[line:64] - INFO: epoch: 0 step: 140, loss: 0.8921677\n", @@ -1034,7 +1032,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:37,836 - 4106154625.py[line:64] - INFO: epoch: 0 step: 146, loss: 0.86497414\n", @@ -1053,7 +1051,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:47,621 - 4106154625.py[line:64] - INFO: epoch: 0 step: 152, loss: 0.8443006\n", @@ -1071,7 +1069,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:38:57,390 - 4106154625.py[line:64] - INFO: epoch: 0 step: 157, loss: 0.82819533\n", @@ -1090,7 +1088,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:07,130 - 4106154625.py[line:64] - INFO: epoch: 0 step: 163, loss: 0.82220745\n", @@ -1110,7 +1108,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:19,382 - 4106154625.py[line:64] - INFO: epoch: 0 step: 170, loss: 0.82164574\n", @@ -1128,7 +1126,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:28,879 - 4106154625.py[line:64] - INFO: epoch: 0 step: 175, loss: 0.8163141\n", @@ -1147,7 +1145,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:38,946 - 4106154625.py[line:64] - INFO: epoch: 0 step: 181, loss: 0.80203724\n", @@ -1166,7 +1164,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:49,513 - 4106154625.py[line:64] - INFO: epoch: 0 step: 187, loss: 0.85867965\n", @@ -1184,7 +1182,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:39:59,182 - 4106154625.py[line:64] - INFO: epoch: 0 step: 192, loss: 0.8151839\n", @@ -1202,7 +1200,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:40:08,369 - 4106154625.py[line:64] - INFO: epoch: 0 step: 197, loss: 0.8161787\n", @@ -1221,7 +1219,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:40:19,189 - 4106154625.py[line:64] - INFO: epoch: 0 step: 203, loss: 0.7799623\n", @@ -1239,7 +1237,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:40:28,204 - 4106154625.py[line:64] - INFO: epoch: 0 step: 208, loss: 0.78541696\n", @@ -1258,7 +1256,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:40:39,729 - 4106154625.py[line:64] - INFO: epoch: 0 step: 214, loss: 0.79589576\n", @@ -1276,7 +1274,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:40:49,338 - 4106154625.py[line:64] - INFO: epoch: 0 step: 219, loss: 0.76227266\n", @@ -1295,7 +1293,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:41:00,458 - 4106154625.py[line:64] - INFO: epoch: 0 step: 225, loss: 0.7822351\n", @@ -1313,7 +1311,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:41:09,631 - 4106154625.py[line:64] - INFO: epoch: 0 step: 230, loss: 0.7710381\n", @@ -1332,7 +1330,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:41:20,764 - 4106154625.py[line:64] - INFO: epoch: 0 step: 236, loss: 0.7427261\n", @@ -1351,7 +1349,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:41:32,130 - 4106154625.py[line:64] - INFO: epoch: 0 step: 242, loss: 0.7901791\n", @@ -1369,7 +1367,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 10:41:41,763 - 4106154625.py[line:64] - INFO: epoch: 0 step: 247, loss: 0.76385504\n", @@ -1387,15 +1385,14 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "......" ] }, - { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 13:39:55,859 - 4106154625.py[line:64] - INFO: epoch: 4 step: 1247, loss: 0.021378823\n", @@ -1414,7 +1411,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 13:40:06,770 - 4106154625.py[line:64] - INFO: epoch: 4 step: 1253, loss: 0.026906993\n", @@ -1707,7 +1704,7 @@ "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 14:04:16,558 - 2610859736.py[line:66] - INFO: ============== Start Test ==============\n" @@ -1722,7 +1719,7 @@ ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ "2025-04-07 14:10:31,931 - 2610859736.py[line:201] - INFO: test cost: 375371.60 ms\n", diff --git a/MindEarth/applications/nowcasting/PreDiff/src/diffusion/latent_diffusion.py b/MindEarth/applications/nowcasting/PreDiff/src/diffusion/latent_diffusion.py index 1454a67a0b2ec0174fe110309dda2ed40c78819f..657dd45c4d56f0c1d60a46a5e0b87fde7d859f2d 100644 --- a/MindEarth/applications/nowcasting/PreDiff/src/diffusion/latent_diffusion.py +++ b/MindEarth/applications/nowcasting/PreDiff/src/diffusion/latent_diffusion.py @@ -1045,7 +1045,7 @@ class PreDiffModule(LatentDiffusion): ) self._in_slice = in_slice self._out_slice = out_slice - return self._in_slices + return self._in_slice @property def out_slice(self): diff --git a/MindEarth/docs/mindearth_archi_cn.png b/MindEarth/docs/mindearth_archi_cn.png index 7ed07ae03bc73dd5bbdd5fbcf7e14923f1eb3b39..6797f2282e37dc4b9ed8aefc12143d4baeb10a96 100644 Binary files a/MindEarth/docs/mindearth_archi_cn.png and b/MindEarth/docs/mindearth_archi_cn.png differ diff --git a/MindEarth/docs/mindearth_archi_en.png b/MindEarth/docs/mindearth_archi_en.png index 1d99c101a38ba43e3af7e40d5f078a54717d8149..2f119437870e119bde3ed6822b4b999824507cd8 100644 Binary files a/MindEarth/docs/mindearth_archi_en.png and b/MindEarth/docs/mindearth_archi_en.png differ