Posts in Infrastructure
YAXT (Yet Another eXchange Tool)
- 23 May 2024
YAXT is a communication layer on top of the MPI library. At starting, the library generates a mapping table between source and target decomposition (communication pattern) and a specific redistribution objects, based on MPI derived datatypes, to perform the exchanges. YAXT is used by YAC for both interpolation weight generation and the coupling field exchange. To calculate the communication patterns of exchanges, more efficient than a simple MPI “all to all”, the libraries rely on a rendezvous algorithm based on a distributed directory of global indices. YAXT not only optimises the exchange pattern but also reduces the communication number, grouping the exchanged arrays on MPI derived datatypes, supposed to reduce the array copies and pack-unpack operations.
YAC - Yet Another Coupler
- 23 May 2024
Couplers allow the data exchange – e.g. of energy, momentum, water and important trace gases such as carbon dioxide – between the Earth system model components for ocean, atmosphere and land, and are essential for representing the climate system’s complex processes and feedbacks in the models.
ComIn - Community Interface
- 23 May 2024
The Community Interface (ComIn) organizes the data exchange and simulation events between the ICON model and “3rd party modules”. The concept can be logically divided into an Adapter Library and a Callback Register.
CDI-PIO (Climate Data Interface with parallel writing )
- 23 May 2024
CDI-PIO is currently used for parallelized GRIB1/GRIB2 and NetCDF output in ECHAM and ICON models. CDI-PIO is the parallel I/O component of the Climate Data Interface (CDI) that is developed and maintained by the Max-Planck-Institute for Meteorology and DKRZ. It is used by ICON, MPIOM, ECHAM, and the Climate Data Operator (CDO) toolkit. The two main I/O paths for output data are writing GRIB files using MPI-IO, and writing NetCDF4 files using HDF5 (which may then also use MPI-IO,or other VOL plugins).
HeAT (The Helmholtz Analytics Toolkit)
- 22 May 2024
HeAT is a distributed tensor framework for high performance data analytics. It provides highly optimized algorithms and data structures for tensor computations using CPUs, GPUs and distributed cluster systems on top of MPI. HeAT builds on PyTorch and mpi4py to provide high-performance computing infrastructure for memory-intensive applications within the NumPy/SciPy ecosystem.
ESMValTool (Earth System Model Evaluation Tool)
- 22 May 2024
ESMValTool is a diagnostics and performance metrics tool for the evaluation and analysis of Earth System Models (ESMs). ESMValTool allows for a comparison of single or multiple models against predecessor versions and observations. It includes many diagnostics and performance metrics covering different aspects of the Earth system and is high flexible.
ESM-Tools
- 22 May 2024
ESM-Tools is a modular software infrastructure that allows for seamlessly building, configuration and running of Earth System Models across different High Performance Computing (HPC) platforms (DKRZ-Levante, Jülich-Juwels, HLRN-4’s Lise and Emmy, ECMWF-Atos, ICCP-Aleph, etc.).
Xarray-Simlab
- 16 May 2024
A Xarray-simlab is a Python library that provides both a generic framework for building computational models in a modular fashion and a xarray extension for setting and running simulations using the “xarray.Dataset” structure. It is designed for fast, interactive and exploratory modeling.
PDAF - Parallel Data Assimilation Framework
- 16 May 2024
The Parallel Data Assimilation Framework - PDAF - is a software environment for data assimilation. PDAF simplifies the implementation of the data assimilation system with existing numerical models. With this, users can obtain a data assimilation system with less work and can focus on applying data assimilation.
Infrastructure
- 27 June 2023
Tools and software libraries are regarded as infrastructure components in natESM, which are distributed independently but only show their full benefit in combination with climate models or climate data. These include, for example, libraries for improved MPI communication on HPC systems, I/O libraries, coupler, analysis tools or orchestration approaches for model runs.