Supported Operators

The Operator abstraction allows ADIOS2 to act upon the user application data, either from a adios2::Variable or a set of Variables in an adios2::IO object. Current supported operations are:

Data compression/decompression, lossy and lossless.

This section provides a description of the supported operators in ADIOS2 and their specific parameters to allow extra-control from the user. Parameters are passed in key-value pairs for:

Operator general supported parameters.
Operator specific supported parameters.

Parameters are passed at:

Compile time: using the second parameter of the method ADIOS2::DefineOperator
Runtime Configuration Files in the ADIOS component.

CompressorZFP

The CompressorZFP Operator is compressor that uses a lossy but optionally error-bounded compression to achieve high compression ratios.

ZFP provides compressed-array classes that support high throughput read and write random access to individual array elements. ZFP also supports serial and parallel (OpenMP and CUDA) compression of whole arrays, e.g., for applications that read and write large data sets to and from disk.

ADIOS2 provides a CompressorZFP operator that lets you compress an decompress variables. Below there is an example of how to invoke CompressorZFP operator:

adios2::IO io = adios.DeclareIO("Output");
auto ZFPOp    = adios.DefineOperator("CompressorZFP", adios2::ops::LossyZFP);

auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
var_r32.AddOperation(ZFPOp, {{adios2::ops::zfp::key::rate, rate}});

CompressorZFP Specific parameters

The CompressorZFP operator accepts the following operator specific parameters:

`CompressorZFP` available parameters
`accuracy`	Fixed absolute error tolerance
`rate`	Fixed number of bits in a compression unit
`precision`	Fixed number of uncompressed bits per value
`backend`	Backend device: `cuda` `omp` `serial`

CompressorZFP Execution Policy

CompressorZFP can run in multiple backend devices: GPUs (CUDA), OpenMP, and in the host CPU. By default CompressorZFP will choose its backend following the above order upon the availability of the device adapter.

Exceptionally, if its corresponding ADIOS2 variable contains a CUDA memory address, this is a CUDA buffer, the CUDA backend will be called if available.

In any case, the user can manually set the backend using the ZFPOperator specific parameter backend.

CompressorSZ3

The CompressorSZ3 Operator is a compressor that uses lossy error-bounded compression to achieve high compression ratios for scientific floating-point data.

SZ3 is a modular error-bounded lossy compression framework for scientific datasets. It provides various compression modes and error control mechanisms, making it suitable for a wide range of scientific applications that require both high compression ratios and controlled data accuracy.

ADIOS2 provides a CompressorSZ3 operator that lets you compress and decompress variables. Below there is an example of how to invoke the CompressorSZ3 operator:

adios2::IO io = adios.DeclareIO("Output");
auto sz3Op = adios.DefineOperator("SZ3Compressor", "sz3");

auto var_r32 = io.DefineVariable<float>("r32", shape, start, count);
var_r32.AddOperation(sz3Op, {{"accuracy", "0.001"}});

CompressorSZ3 Specific parameters

The CompressorSZ3 operator accepts the following operator specific parameters:

`CompressorSZ3` available parameters
`accuracy`	Alias for `absolute`, fixed absolute error tolerance
`absolute`	Alias for `abs`, fixed absolute error tolerance
`abs`	Fixed absolute error tolerance
`abserrbound`	Fixed absolute error tolerance
`relative`	Alias for `rel`, relative error tolerance
`rel`	Relative error tolerance
`relerrbound`	Relative error tolerance
`psnr`	Peak signal-to-noise ratio error bound
`psnrerrbound`	Peak signal-to-noise ratio error bound
`norm`	Alias for `l2norm`, L2 norm error bound
`l2norm`	L2 norm error bound
`l2normerrbound`	L2 norm error bound
`mode`	Error bound mode: `ABS`, `REL`, `PSNR`, `L2NORM`, `ABS_AND_REL`, `ABS_OR_REL`
`errorboundmode`	Error bound mode (same options as `mode`)

CompressorSZ3 Supported Data Types

CompressorSZ3 supports the following data types:

float - 32-bit floating point
double - 64-bit floating point
std::complex<float> - Complex 32-bit floating point
std::complex<double> - Complex 64-bit floating point

CompressorSZ3 Dimension Support

CompressorSZ3 supports arrays with 1 to 4 dimensions. The SZ3 library currently has a maximum dimension limit of 4.

CompressorSZ3 Error Modes

The CompressorSZ3 operator supports multiple error control modes:

Absolute Error (ABS): Guarantees that the absolute difference between the original and decompressed data is within the specified bound.
Relative Error (REL): Guarantees that the relative difference between the original and decompressed data is within the specified bound.
PSNR Error: Controls compression based on peak signal-to-noise ratio.
L2 Norm Error: Controls compression based on the L2 norm of the error.
ABS_AND_REL: Applies both absolute and relative error bounds simultaneously.
ABS_OR_REL: Uses whichever error bound is less restrictive.

Note

When using accuracy parameter without specifying a mode, the default error bound mode is ABS (absolute error).

Plugin Operator

For details on using the Plugin Operator, see the Plugins documentation.

Encryption

The Encryption Operator uses the Plugins interface. This operator uses libsodium for encrypting and decrypting data. If ADIOS can find libsodium at configure time, this plugin will be built.

The encryption mode is selected automatically based on the parameters supplied.

Symmetric Mode

In symmetric mode, the operator generates a secret key and encrypts the data with the key and a nonce as described in the libsodium secret key cryptography docs. The key is saved to the specified SecretKeyFile and will be used for decryption. The key should be kept confidential since it is used to both encrypt and decrypt the data.

Symmetric mode is activated when SecretKeyFile is provided without any public-key parameters.

Key	Value Format	Explanation
PluginName	string	Required. Name to refer to plugin, e.g., `MyOperator`
PluginLibrary	string	Required. Name of shared library, `EncryptionOperator`
SecretKeyFile	string	Required. Path to secret key file

Asymmetric Mode

In asymmetric mode, a random per-write session key is sealed with the recipient’s Curve25519 public key using crypto_box_seal, and the bulk data is encrypted with that session key via crypto_secretbox. Only the public key is needed for writing; both the public and secret keys are needed for reading.

Asymmetric mode is activated when PublicKeyFile or PublicKey is provided. Key pairs can be generated with the adios2_seal_keygen utility.

The secret key lookup order for reading is: SecretKeyFile parameter, SecretKey parameter, ADIOS2_SECRET_KEY_FILE environment variable, ADIOS2_SECRET_KEY environment variable. If only the secret key is supplied, the public key is derived automatically via Curve25519.

Key	Value Format	Explanation
PluginName	string	Required. Name to refer to plugin, e.g., `MyOperator`
PluginLibrary	string	Required. Name of shared library, `EncryptionOperator`
PublicKeyFile	string	Path to a 32-byte Curve25519 public key file
PublicKey	string	Hex-encoded public key (64 hex characters)
SecretKeyFile	string	Path to a 32-byte Curve25519 secret key file (reading)
SecretKey	string	Hex-encoded secret key (64 hex characters, reading only)