============================== What's new in Development ============================== Thread-Safe Get with GetContext ------------------------------- A new ``GetContext`` API extends the Selection-based ``Get()`` to support concurrent read pipelines on a single engine. Each caller allocates its own ``GetContext`` via ``Engine::NewGetContext()`` and uses it as the first argument to ``Get()`` and ``PerformGets()``; per-context queues mean two threads can drive independent ``Get`` batches against the same open engine without racing. .. code-block:: c++ auto ctx = engine.NewGetContext(); if (ctx) { engine.Get(*ctx, var, data, adios2::Selection::BoundingBox(start, count)); engine.PerformGets(*ctx); } ``NewGetContext()`` returns ``nullptr`` when the engine does not support concurrent contexts; callers feature-probe with the null check. Currently only the **BP5** reader supports them, and only when opened in ``Mode::ReadRandomAccess`` with a reentrant file transport (POSIX). See the Selection chapter for full details. SST Mercury Data Plane ----------------------- A new ``Mercury`` data plane for SST uses the Mercury/margo RPC framework to carry data, supporting multiple network backends through a single implementation: ``cxi://`` for HPE-Cray Slingshot (Frontier, Perlmutter), ``ofi+tcp`` / ``ofi+verbs`` for libfabric-based fabrics, ``bmi+tcp``, and ``na+sm`` for shared memory. The backend is selected at runtime via the ``MercuryProtocol`` engine parameter, with an ``SST_MERCURY_PROTOCOL`` environment-variable fallback. On Slingshot, the required OFI auth key is supplied automatically from ``SLINGSHOT_VNIS``. See the SST engine documentation for details. ====================== What's new in v2.12? ====================== C API Improvements ------------------ New functions for safely retrieving string data of unknown length using a two-call pattern (first call with NULL buffer to query length, then call again with allocated buffer to retrieve data): - ``adios2_get_string()`` - for string variables - ``adios2_attribute_string_data()`` - for single-value string attributes - ``adios2_attribute_string_data_array()`` - for string array attributes This addresses a gap in the C API where string lengths could not be queried before reading. Selection API for Get() Operations ----------------------------------- A new ``Selection`` class provides an explicit, self-contained way to specify what data to read in ``Get()`` calls. Instead of mutating the ``Variable`` object with ``SetSelection()``, ``SetBlockSelection()``, ``SetStepSelection()``, etc., all selection parameters are bundled into a ``Selection`` object that is passed directly to ``Get()``. This avoids relying on potentially stale variable state and makes the intent of each ``Get()`` call clear at the call site. A selection has two independent aspects: a spatial selection — ``Selection::All()`` (the default — reads the entire variable) or ``Selection::BoundingBox(start, count)`` (hyperslab) — and an optional block selection via ``Selection::Block(blockID)`` or ``.WithBlock(blockID)``. These are orthogonal: a block ID can be combined with a bounding box to read a sub-region of a specific write block. Additional parameters (steps, memory layout, accuracy) can be added via fluent ``With*()`` methods or mutable ``Set*()`` methods. A ``ToString()`` method is provided for debugging. .. code-block:: c++ // Select everything engine.Get(var, data, adios2::Selection::All()); // Fluent style — bounding box with step range auto sel = adios2::Selection::BoundingBox({0, 0}, {10, 20}).WithSteps(0, 5); engine.Get(var, data, sel); // Mutable style — reuse across iterations adios2::Selection sel; sel.SetBoundingBox({0, 0}, {10, 20}); for (size_t step = 0; step < nsteps; ++step) { sel.SetSteps(step, 1); engine.Get(var, buffers[step], sel); } // Debug output std::cout << sel.ToString() << std::endl; Existing ``SetSelection()``-based APIs remain available and are not deprecated. See :ref:`Selection` for full documentation. Plugin Interface v2 (Breaking Change) -------------------------------------- Both the engine and operator plugin interfaces have been redesigned as standalone classes that no longer inherit from internal core types (``core::Engine`` and ``core::Operator``). This is a **breaking change** for existing plugins written against the v1 interface (ADIOS2 2.11 and earlier). **What changed:** - **Engine plugins** now inherit from ``PluginEngineInterface`` (in ``adios2/plugin/PluginEngineInterface.h``) which uses public C++ API types (``adios2::IO``, ``adios2::Variable``) instead of internal core types. The ``EngineCreate()`` signature now takes ``adios2::IO`` and ``adios2::Mode`` instead of ``core::IO &`` and ``helper::Comm``. - **Operator plugins** now inherit from ``PluginOperatorInterface`` (in ``adios2/plugin/PluginOperatorInterface.h``) which is fully self-contained with no dependencies on ADIOS internals. - Plugin shared libraries now only need to link against ``adios2::cxx`` (not ``adios2::core``). **Migrating from v1:** Update your plugin class to inherit from the new interface, replace any ``core::IO`` / ``core::Variable`` usage with the public ``adios2::IO`` / ``adios2::Variable`` types, and update the ``EngineCreate()`` / ``EngineDestroy()`` signatures accordingly. See the examples in ``examples/plugins/`` for reference implementations. **New engine plugin capabilities:** The v2 engine interface has also been extended with optional methods for more advanced functionality: - **Selection-based Get**: ``DoGetSync()`` and ``DoGetDeferred()`` overloads that accept a ``Selection`` parameter, enabling sub-array and block reads. - **Span-based Put**: ``DoPut()`` with a ``Span`` parameter for zero-copy writes of primitive types. - **Step queries**: ``Steps()`` to report the total number of available steps. - **Block introspection**: ``MinBlocksInfo()`` to return block decomposition metadata for a given variable and step. All new methods have empty default implementations, so they are optional. See :ref:`Plugins` for full documentation. XRootD Server Resource Management ----------------------------------- The XRootD server plugin used for remote campaign access now includes resource-aware cache management: - **FD and metadata eviction**: The file pool tracks per-file file descriptor costs and metadata memory usage, evicting idle entries when configurable limits are approached. Limits can be set via environment variables (``ADIOS_POOL_FD_LIMIT``, ``ADIOS_POOL_METADATA_LIMIT``) or at runtime through the admin HTTP interface. - **Shared tar FD cache**: When multiple datasets are packed into a single tar file (common in campaign archives), a process-wide singleton shares one file descriptor across all engines reading from the same tar, reducing FD consumption from N to 1 per tar file. - **Batched data requests**: The remote reader now batches multiple variable reads into a single network round-trip, reducing latency for workloads that read many variables per step. - **Per-request access log**: An optional server-side log records each remote read (variable, step, selection, byte count) as JSON lines, for analyzing access patterns. It is off by default; set ``ADIOS2_XROOTD_ACCESSLOG`` to a file path to enable it, with size-based rotation and retention configurable via environment variables. See the XRootD plugin ``README.md`` for details. Cross-Endian Interoperability ----------------------------- The ``ADIOS2_USE_Endian_Reverse`` CMake option has been removed. Cross-endian file interoperability (reading files written on a machine with different byte order) is now always enabled. This means files written on big-endian systems can be read on little-endian systems and vice versa without any special build configuration. S3 Object Storage for BP5 Data ------------------------------ BP5 can now write data files to S3-compatible object storage (Amazon S3, MinIO, Ceph, etc.) while keeping metadata on the local filesystem. This hybrid model provides cheap bulk storage for data with fast local metadata access. Set ``DataFileTransport=awssdk``, ``S3Endpoint``, and ``S3Bucket`` as engine parameters. Credentials are accepted as parameters or via standard AWS environment variables (``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``). Key features: - **Multi-object mode** (default): each data segment is a separate S3 object, providing automatic crash recovery - **Async write**: automatically enabled for S3 transports so uploads overlap with computation - **Append and AppendAfterSteps**: supported via segment discovery and selective object deletion - **Stale object cleanup**: previous data objects are deleted on ``Open(Write)`` - **s3.json sidecar**: written locally so readers auto-detect the S3 location See :ref:`BP5 S3 Object Storage` for configuration details. C++17 Minimum Requirement -------------------------- The minimum C++ standard required to build ADIOS2 has been raised from C++14 to C++17. Python Bindings: Nanobind Migration ------------------------------------- The Python bindings have been migrated from pybind11 to nanobind. Nanobind produces smaller, faster extension modules and enables stable-ABI (``abi3``) wheels for Python 3.12 and later, so a single wheel binary works across multiple Python versions without recompilation. Asymmetric Encryption Operator -------------------------------- A new ``Encryption`` operator supports asymmetric encryption of variable data using libsodium's sealed-box construction (X25519 key exchange + XSalsa20-Poly1305). Writers encrypt data using the recipient's public key; only the holder of the corresponding private key can decrypt. No writer keypair is required, so the data producer does not need to be trusted with decryption credentials. SZ3 Compression Operator -------------------------- A new ``SZ3`` operator is available for lossy data compression using the SZ3 library, extending the available compression options alongside SZ, ZFP, MGARD, and BigWhoop. RefactorProDM Operator ----------------------- A new ``RefactorProDM`` operator integrates ADIOS2 with ProDM, a data decomposition library by Dr. Xin Liang. When ProDM is available at build time, this operator provides decomposition-based data reduction as an alternative to traditional compressors. Python Free-Threading Support ------------------------------ The Python bindings are now compatible with free-threaded Python (PEP 703, ``python3.13t``), allowing ADIOS2 to operate without the GIL in environments that support it. HTTPS Transport for Remote Access ----------------------------------- An HTTPS-based remote transport (``XrootdHttpsRemote``) using libcurl has been added as an alternative to the native XRootD protocol. This enables reading ADIOS2 datasets over standard HTTPS URLs from XRootD SSI services, useful in environments where native XRootD connectivity is unavailable. Campaign Reader Improvements ------------------------------ The campaign reader received several improvements: host aliases in ``~/.config/hpc-campaign/hosts.yaml`` allow symbolic machine names to be defined once and reused across dataset entries; XRootD connections can now be declared as host entries; in-memory metadata caching reduces repeated file access overhead; and the reader now strictly accepts ACA version 0.7, rejecting unsupported versions with a clear error. For large archives the SQLite query was rewritten as a single join across the dataset, replica, file, and resolution tables, reducing open time from ~27 seconds to under one second for collections of thousands of images. TAR File Reading ----------------- The BP5 reader now supports datasets packaged inside TAR archives. When a TAR index is provided, BP5 datasets, text files, and images can be read from a local TAR, an HTTPS-hosted TAR, or a TAR accessible via the remote server. This is primarily used by the campaign reader to access archived datasets. Python Stream: minmax() ------------------------ ``adios2.Stream`` gains a new ``minmax(name, [step], [block_info_list])`` method that returns the minimum and maximum values of a variable. In read mode or when a step is specified it returns per-step statistics; in random-access mode without a step argument it returns the global min/max. Inline Engine Buffering ------------------------ The inline engine now supports basic output buffering, a step toward full feature parity with file-based engines for in-situ workflows. Windows and macOS pip Wheel Support ------------------------------------- Automated integration tests for pip ``sdist`` and wheel builds on Windows and macOS have been added to CI, verifying correct library loading and packaging under ``pip install`` workflows. REUSE License Compliance ------------------------- ADIOS2 now complies with the REUSE specification, ensuring every source file carries a machine-readable SPDX license and copyright notice. =================== What's new in 2.11? =================== This is a major release with new features and lots of bug fixes. The main highlights include enhanced derived variables, remote data access, GPU improvements, campaign management overhaul, and advanced compression operators. Derived Variables ----------------- Support for reader-side computed variables with expression evaluation. This release adds comprehensive functionality including new mathematical operations (trigonometric functions, multiplication, division, power) and the ability to apply derived variables to aggregated arrays. Remote Data Access Infrastructure ---------------------------------- Implementation of XRootD and remote server capabilities enabling distributed file access. This includes support for remote streaming, metadata caching, and optimized query handling across networked systems. GPU and Memory Space Improvements ---------------------------------- Enhanced GPU backend support with memory selection adjustments, Kokkos integration improvements, and bindings for cuPy pointers and torch tensors. Addresses layout mismatches between user code and device memory. Campaign Management System Overhaul ------------------------------------ Significant updates to campaign tracking using SQLite databases replacing JSON files, configuration file support, and attribute handling. Includes multi-host support and improved metadata organization. Compression and Data Format Advances ------------------------------------- New operators including BigWhoop compression and Complex MGARD support. ZFP now supports 4D arrays, and improved MGARD efficiency through zstd compression of lossless data portions. OpenSSF Best Practices ----------------------- ADIOS2 now adopts OpenSSF (Open Source Security Foundation) best practices including enhanced security measures, automated vulnerability scanning, and adherence to industry-standard development practices. =================== What's new in 2.10? =================== This is a major release with new features and lots of bug fixes. The main new feature is the new Python API. Python ------ Before, ADIOS had two separate APIs for Python. The low-level ("Full") API was written with Pybind11 and directly mimicked the C++ API. The high-level API was another, smaller, and more pythonesque API that allowed for easier scripting with Python. The main problems with these two were that they were independent, and that the high-level API was not complete. Once a developer needed a feature only available in the full API, they had to start from scratch writing a script with the full API. In 2.10, there is officially one Python API, written in Python, which in turn uses the old Pybind11 classes. The new API combines the high-level features of the old high-level API -- hopefully in a more consistent and likeable way, -- and the full feature set of the low-level bindings. .. note:: Old scripts that used the full API can still run without almost any modification, just change the import line from ``import adios2`` to ``import adios2.bindings as adios2`` Old scripts that used the high-level API must be modified to make them work with the new API, see :ref:`Transition from old API to new API` u See :ref:`Python APIs` New/updated features -------------------- - BP5 is supported on Windows now - SST and DataMan staging engines are GPU-Aware now - SYCL support added for Intel GPUs (besides CUDA and HIP for NVidia and AMD GPUs) - the SST/libfabric data transport now works on Frontier (besides the MPI data transport) Packaging ---------- - adios2 package is now on `PyPi `_ ================== What's new in 2.9? ================== This is a major release with new features and lots of bug fixes. General ------- - GPU-Aware I/O enabled by using Kokkos. Device pointers can be passed to Put()/Get() calls directly. Kokkos 3.7.x required for this release. Works with CUDA, HIP and Kokkos applications. https://adios2.readthedocs.io/en/latest/advanced/gpu_aware.html#gpu-aware-i-o - GPU-compression. MGARD and ZFP operators can compress data on GPU if they are built for GPU. MGARD operator can be fed with host/device pointers and will move data automaticaly. ZFP operator requires matching data and compressor location. - Joined Array concept (besides Global Array and Local Array), which lets writers dump Local Arrays (no offsets no global shape) that are put together into a Global Array by the reader. One dimension of the arrays is selected for this join operation, while other dimensions must be the same for all writers. https://adios2.readthedocs.io/en/latest/components/components.html?highlight=Joined#shapes File I/O -------- - Default File engine is now BP5. If for some reason this causes problems, manually specify using "BP4" for your application. - BP5 engine supports multithreaded reading to accelerate read performance for low-core counts. - BP5 Two level metadata aggregation and reduction reduced memory impact of collecting metadata and therefore is more scalable in terms of numbers of variables and writers than BP4. - Uses Blosc-2 instead of Blosc for lossless compression. The new compression operator is backward compatible with old files compressed with blosc. The name of the operator remains "blosc". Staging ------- - UCX dataplane added for SST staging engine to support networks under the UCX consortium - MPI dataplane added for SST staging engine. It relies on MPI intercommunicators to connect multiple independent MPI applications for staging purposes. Applications must enable multithreaded MPI for this dataplane. Experimental features --------------------- - Preliminary support for data structs. A struct can have single variables of basic types, and 1D fixed size arrays of basic types. Supported by BP5, SST and SSC engines.