Supported Operators

The Operator abstraction allows ADIOS2 to act upon the user application data, either from a adios2::Variable or a set of Variables in an adios2::IO object. Current supported operations are:

  1. Data compression/decompression, lossy and lossless.

This section provides a description of the supported operators in ADIOS2 and their specific parameters to allow extra-control from the user. Parameters are passed in key-value pairs for:

  1. Operator general supported parameters.

  2. Operator specific supported parameters.

Parameters are passed at:

  1. Compile time: using the second parameter of the method ADIOS2::DefineOperator

  2. Runtime Configuration Files in the ADIOS component.

CompressorZFP

The CompressorZFP Operator is compressor that uses a lossy but optionally error-bounded compression to achieve high compression ratios.

ZFP provides compressed-array classes that support high throughput read and write random access to individual array elements. ZFP also supports serial and parallel (OpenMP and CUDA) compression of whole arrays, e.g., for applications that read and write large data sets to and from disk.

ADIOS2 provides a CompressorZFP operator that lets you compress an decompress variables. Below there is an example of how to invoke CompressorZFP operator:

adios2::IO io = adios.DeclareIO("Output");
auto ZFPOp    = adios.DefineOperator("CompressorZFP", adios2::ops::LossyZFP);

auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
var_r32.AddOperation(ZFPOp, {{adios2::ops::zfp::key::rate, rate}});

CompressorZFP Specific parameters

The CompressorZFP operator accepts the following operator specific parameters:

CompressorZFP available parameters

accuracy

Fixed absolute error tolerance

rate

Fixed number of bits in a compression unit

precision

Fixed number of uncompressed bits per value

backend

Backend device: cuda omp serial

CompressorZFP Execution Policy

CompressorZFP can run in multiple backend devices: GPUs (CUDA), OpenMP, and in the host CPU. By default CompressorZFP will choose its backend following the above order upon the availability of the device adapter.

Exceptionally, if its corresponding ADIOS2 variable contains a CUDA memory address, this is a CUDA buffer, the CUDA backend will be called if available.

In any case, the user can manually set the backend using the ZFPOperator specific parameter backend.

CompressorSZ3

The CompressorSZ3 Operator is a compressor that uses lossy error-bounded compression to achieve high compression ratios for scientific floating-point data.

SZ3 is a modular error-bounded lossy compression framework for scientific datasets. It provides various compression modes and error control mechanisms, making it suitable for a wide range of scientific applications that require both high compression ratios and controlled data accuracy.

ADIOS2 provides a CompressorSZ3 operator that lets you compress and decompress variables. Below there is an example of how to invoke the CompressorSZ3 operator:

adios2::IO io = adios.DeclareIO("Output");
auto sz3Op = adios.DefineOperator("SZ3Compressor", "sz3");

auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
var_r32.AddOperation(sz3Op, {{"accuracy", "0.001"}});

CompressorSZ3 Specific parameters

The CompressorSZ3 operator accepts the following operator specific parameters:

CompressorSZ3 available parameters

accuracy

Alias for absolute, fixed absolute error tolerance

absolute

Alias for abs, fixed absolute error tolerance

abs

Fixed absolute error tolerance

abserrbound

Fixed absolute error tolerance

relative

Alias for rel, relative error tolerance

rel

Relative error tolerance

relerrbound

Relative error tolerance

psnr

Peak signal-to-noise ratio error bound

psnrerrbound

Peak signal-to-noise ratio error bound

norm

Alias for l2norm, L2 norm error bound

l2norm

L2 norm error bound

l2normerrbound

L2 norm error bound

mode

Error bound mode: ABS, REL, PSNR, L2NORM, ABS_AND_REL, ABS_OR_REL

errorboundmode

Error bound mode (same options as mode)

CompressorSZ3 Supported Data Types

CompressorSZ3 supports the following data types:

  • float - 32-bit floating point

  • double - 64-bit floating point

  • std::complex<float> - Complex 32-bit floating point

  • std::complex<double> - Complex 64-bit floating point

CompressorSZ3 Dimension Support

CompressorSZ3 supports arrays with 1 to 4 dimensions. The SZ3 library currently has a maximum dimension limit of 4.

CompressorSZ3 Error Modes

The CompressorSZ3 operator supports multiple error control modes:

  • Absolute Error (ABS): Guarantees that the absolute difference between the original and decompressed data is within the specified bound.

  • Relative Error (REL): Guarantees that the relative difference between the original and decompressed data is within the specified bound.

  • PSNR Error: Controls compression based on peak signal-to-noise ratio.

  • L2 Norm Error: Controls compression based on the L2 norm of the error.

  • ABS_AND_REL: Applies both absolute and relative error bounds simultaneously.

  • ABS_OR_REL: Uses whichever error bound is less restrictive.

Note

When using accuracy parameter without specifying a mode, the default error bound mode is ABS (absolute error).

CompressorMGARD

The CompressorMGARD Operator is a compressor that uses lossy error-bounded compression to achieve high compression ratios for scientific floating-point data.

MGARD (MultiGrid Adaptive Reduction of Data) is a multilevel compression framework based on a hierarchical (multigrid) decomposition of the input. It supports a rigorous error bound in either an absolute or relative sense and lets the user tune how that error is measured through a smoothness parameter.

ADIOS2 provides a CompressorMGARD operator that lets you compress and decompress variables. Below there is an example of how to invoke the CompressorMGARD operator:

adios2::IO io = adios.DeclareIO("Output");
auto mgardOp = adios.DefineOperator("MGARDCompressor", adios2::ops::LossyMGARD);

auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
var_r32.AddOperation(mgardOp, {{adios2::ops::mgard::key::tolerance, "0.001"},
                               {adios2::ops::mgard::key::s, "inf"}});

CompressorMGARD Specific parameters

The CompressorMGARD operator accepts the following operator specific parameters:

CompressorMGARD available parameters

tolerance

Error tolerance (mandatory). Interpreted according to mode.

accuracy

Alias for tolerance.

mode

Error bound mode: ABS or REL (default REL).

s

Smoothness / norm parameter (default 0). Use inf for the L-infinity (max) norm, 0 for the L2 norm, or any finite real for the corresponding Sobolev norm.

threshold

Minimum input size in bytes below which compression is skipped and the block is stored uncompressed (default 100000).

lossless_type

Lossless stage applied after MGARD’s quantization. One of: huffman, huffman_lz4, huffman_zstd, cpu_lossless (default huffman_zstd).

CompressorMGARD Supported Data Types

CompressorMGARD supports the following data types:

  • float - 32-bit floating point

  • double - 64-bit floating point

  • std::complex<float> - complex 32-bit floating point

  • std::complex<double> - complex 64-bit floating point

For complex inputs, the operator presents the data to MGARD as a real-valued array of the underlying scalar type (float or double) with the trailing dimension doubled. MGARD therefore operates on the real and imaginary parts interleaved along that axis, which may affect compression quality compared to a purely real-valued input of the same shape.

CompressorMGARD Dimension Support

CompressorMGARD supports arrays with 1 to 5 dimensions. ADIOS2 pads 1D and 2D blocks up to 3 dimensions before handing them to MGARD-x.

CompressorMGARD Error Bound Semantics

The error bound is specified by two orthogonal parameters: mode selects how tolerance is interpreted, and s selects the norm in which the error is measured.

``mode`` controls the scale of the bound:

  • ``ABS``: tolerance is an absolute value. The error in the selected norm is bounded by tolerance.

  • ``REL`` (default): tolerance is relative to the magnitude of the input block. The guaranteed error is tolerance * ||input||.

``s`` selects the norm used to measure the error:

  • s=inf (L-infinity) bounds the worst-case pointwise error.

  • s=0 (L2, default) bounds the aggregate error in the least-squares sense.

  • Finite real values select the corresponding Sobolev norms, which balance pointwise accuracy against smoothness.

CompressorMGARD GPU Interaction

The ADIOS2 MGARD operator passes the input buffer directly to the underlying MGARD-x library without staging it through host memory. Whether a GPU device pointer can be used as input therefore depends on the MGARD-x build that ADIOS2 is linked against:

  • If MGARD-x is built with a GPU backend (CUDA, HIP, or SYCL), it consumes a matching device pointer in place.

  • If MGARD-x is built SERIAL-only, the input must reside in host memory.

The output buffer that ADIOS2 hands to the operator is host-resident, so the decompressed result is always returned to host memory regardless of the MGARD-x backend.

Plugin Operator

For details on using the Plugin Operator, see the Plugins documentation.

Encryption

The Encryption Operator uses the Plugins interface and requires libsodium. If ADIOS2 finds libsodium at configure time the plugin is built automatically.

The operator supports two modes. Mode is selected by which key sources are present (parameters or environment variables); asymmetric mode takes priority.

Symmetric Mode

Uses XSalsa20-Poly1305 (crypto_secretbox). A single 32-byte secret key is shared between writer and reader. For each variable chunk a fresh random 24-byte nonce is generated; the ciphertext carries a 16-byte Poly1305 MAC. On-disk layout:

[ uint64 original_size | 24 B nonce | ciphertext + 16 B MAC ]

Key lookup order (first match wins):

  1. ADIOS2_SECRET_KEY environment variable (hex string)

  2. SecretKeyFile parameter — path to a 32-byte binary key file. On the writer side the file is created if it does not exist.

  3. SecretKey parameter (hex string)

Asymmetric Mode

Uses a hybrid scheme: Curve25519 key agreement + XSalsa20-Poly1305 bulk cipher.

For each variable chunk:

  1. A random 32-byte session key is generated.

  2. The session key is encrypted with crypto_box_seal — an anonymous ECDH construction using an ephemeral sender keypair. Only the recipient’s public key is required to seal; both keys are required to unseal.

  3. The chunk data is encrypted with the session key via crypto_secretbox.

  4. The session key is wiped from memory.

On-disk layout:

[ uint64 original_size | 48 B sealed_session_key | 24 B nonce | ciphertext + 16 B MAC ]

The 48-byte sealed session key is crypto_box_SEALBYTES (32) + 32.

Writer only needs the public key. Reader needs the secret key (public key can be derived automatically).

Key pairs can be generated with the adios2_seal_keygen utility.

Public key lookup order (first match wins):

  1. ADIOS2_PUBLIC_KEY_FILE environment variable (file path)

  2. ADIOS2_PUBLIC_KEY environment variable (64-char hex)

  3. PublicKeyFile parameter

  4. PublicKey parameter (hex)

Secret key lookup order (first match wins):

  1. ADIOS2_ASYM_SECRET_KEY_FILE environment variable (file path)

  2. ADIOS2_ASYM_SECRET_KEY environment variable (64-char hex)

  3. SecretKeyFile parameter

  4. SecretKey parameter (hex)

If a secret key is provided without a public key, the public key is derived automatically via crypto_scalarmult_base (Curve25519 base-point multiply). Setting any of the four asymmetric environment variables activates asymmetric mode even when no parameters are passed.