High-Level APIs

The high-level APIs are designed for simple tasks for which performance is not critical. Unlike the Full APIs, the high-level APIs only require a single object handler resembling a C++ fstream or a Python file I/O idiom. The high-level APIs are recommended to both first-time and advanced users; the low-level APIs being recommended only when performance testing identifies a bottleneck or when more control is needed.

Typical scenarios for using the simple high-level APIs are:

Reading a file to perform data analysis with libraries (matplotlib, scipy, etc.)
Interactive: few calls make interactive usage easier.
Saving data to files is small or personal projects
Online frameworks: e.g. Jupyter notebooks, see python-mpi examples running on MyBinder

The designed functionality syntax is closely related to the native language IO bindings for formatted text files e.g. C++ fstream getline, and Python file IO. The main function calls are: open (or constructor in C++), write, read and close (or destructor in C++). In addition, ADIOS2 borrows the corresponding language native syntax for advancing lines to advance the step in write mode, and for a “step-by-step” streaming basis in read mode. See each language section in this chapter for a write/read example.

Note

The simplified APIs are based on language native file IO interface. Hence write and read calls are always synchronized and variables data memory is ready to use immediately after these calls.

Currently ADIOS2 support bindings for the following languages and their minimum standards:

Language	Standard	Interface	Based on
C++	11/newer	`#include adios2.h`	`fstream`
Matlab

The following sections provide a summary of the API calls on each language and links to Write and Read examples to put it all together.

C++ High-Level API

C++11 High-Level APIs are based on a single object adios2::fstream

Caution

DO NOT place use namespace adios2 in your C++ code. Use adios2::fstream directly to prevent conflicts with std::fstream.

C++11 Write example

#include <adios2.h>
...

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

// Nx, Ny from application, std::size_t
const adios2::Dims shape{Nx, Ny * static_cast<std::size_t>(size)};
const adios2::Dims start{0, Ny * static_cast<std::size_t>(rank)};
const adios2::Dims count{Nx, Ny};

adios2::fstream oStream("cfd.bp", adios2::fstream::out, MPI_COMM_WORLD);

// NSteps from application
for (std::size_t step = 0; step < NSteps; ++step)
{
    if(rank == 0 && step == 0) // global variable
    {
        oStream.write<int32_t>("size", size);
    }

    // physicalTime double, <double> is optional
    oStream.write<double>( "physicalTime", physicalTime );
    // T and P are std::vector<float>
    oStream.write( "temperature", T.data(), shape, start, count );
    // adios2::endl will advance the step after writing pressure
    oStream.write( "pressure", P.data(), shape, start, count, adios2::end_step );

}

// Calling close is mandatory!
oStream.close();

C++11 Read “step-by-step” example

#include <adios2.h>
...

int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

// Selection Window from application, std::size_t
const adios2::Dims start{0, 0};
const adios2::Dims count{SelX, SelY};

if( rank == 0)
{
   // if only one rank is active use MPI_COMM_SELF
   adios2::fstream iStream("cfd.bp", adios2::fstream::in, MPI_COMM_SELF);

   adios2::fstep iStep;
   while (adios2::getstep(iStream, iStep))
   {
       if( iStep.currentstep() == 0 )
       {
           const std::size_t sizeOriginal = iStep.read<std::size_t>("size");
       }
       const double physicalTime = iStream.read<double>( "physicalTime");
       const std::vector<float> temperature = iStream.read<float>( "temperature", start, count );
       const std::vector<float> pressure = iStream.read<float>( "pressure", start, count );
   }
   // Don't forget to call close!
   iStream.close();
}

`adios2::fstream` API documentation

class fstream

Public Types

enum openmode

Available open modes for adios2::fstream constructor or open calls

Values:

enumerator out: write

enumerator in: read

enumerator in_random_access: read_random_access

enumerator app: append, not yet supported

Public Functions

fstream(const std::string &name, adios2::fstream::openmode mode, MPI_Comm comm, const std::string engineType = "BPFile")

High-level API MPI constructor, based on C++11 fstream. Allows for passing parameters in source code.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
engineType – available adios2 engine

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

fstream(const std::string &name, const adios2::fstream::openmode mode, MPI_Comm comm, const std::string &configFile, const std::string ioInConfigFile)

High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

fstream(const std::string &name, const adios2::fstream::openmode mode, const std::string engineType = "BPFile")

High-level API non-MPI constructor, based on C++11 fstream. Allows for passing parameters in source code.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
engineType – available adios2 engine

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

fstream(const std::string &name, const adios2::fstream::openmode mode, const std::string &configFile, const std::string ioInConfigFile)

High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

fstream() = default: Empty constructor, allows the use of open later in the code

~fstream() = default: Using RAII STL containers only

explicit operator bool() const noexcept: Checks if fstream object is valid

void open(const std::string &name, const openmode mode, MPI_Comm comm, const std::string engineType = "BPFile")

High-level API MPI open, based on C++11 fstream. Allows for passing parameters in source code. Used after empty constructor.

Parameters:

name – stream name
mode – adios2::fstream::in (Read), adios2::fstream::out (Write), adios2::fstream::app (Append)
comm – MPI communicator establishing domain for fstream
engineType – available adios2 engine

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

void open(const std::string &name, const openmode mode, MPI_Comm comm, const std::string configFile, const std::string ioInConfigFile)

High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file. Used after empty constructor.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

void open(const std::string &name, const openmode mode, const std::string engineType = "BPFile")

High-level API non-MPI open, based on C++11 fstream. Allows for passing parameters in source code. Used after empty constructor.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
engineType – available adios2 engine

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

void open(const std::string &name, const openmode mode, const std::string configFile, const std::string ioInConfigFile)

High-level API non-MPI constructor, based on C++11 fstream. Allows for runtime config file. Used after empty constructor.

Parameters:

name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

void set_parameter(const std::string key, const std::string value) noexcept

Set a single stream parameter based on Engine supported parameters. MUST be passed before the first call to write or read. See: https://adios2.readthedocs.io/en/latest/engines/engines.html

Parameters:

key – input parameter key
value – input parameter value

void set_parameters(const adios2::Params &parameters) noexcept

Set stream parameters based on Engine supported parameters. MUST be passed before the first call to write or read. See: https://adios2.readthedocs.io/en/latest/engines/engines.html

Parameters:: parameters – input key/value parameters

template<class T> void write_attribute(const std::string &name, const T &value, const std::string &variableName = "", const std::string separator = "/", const bool endStep = false)

Define attribute inside fstream or for a variable after write. Single value input version.

Parameters:

name – unique attribute identifier IO object or for a Variable if variableName is not empty (associated to a variable)
value – single data value
variableName – default is empty, if not empty attributes is associated to a variable after a write
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.

template<class T> void write_attribute(const std::string &name, const T *data, const size_t size, const std::string &variableName = "", const std::string separator = "/", const bool endStep = false)

Define attribute inside fstream or for a variable after write. Array input version.

Parameters:

name – unique attribute identifier IO object or for a Variable if variableName is not empty (associated to a variable)
data – pointer to user data
size – number of data elements
variableName – default is empty, if not empty attributes is associated to a variable after a write
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.

template<class T> void write(const std::string &name, const T *data, const adios2::Dims &shape = adios2::Dims(), const adios2::Dims &start = adios2::Dims(), const adios2::Dims &count = adios2::Dims(), const bool endStep = false)

writes a self-describing array variable

Parameters:

name – variable name
data – variable data data
shape – variable global MPI dimensions. Pass empty for local variables.
start – variable offset for current MPI rank. Pass empty for local variables.
count – variable dimension for current MPI rank. Local variables only have count.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep if true.

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

template<class T> void write(const std::string &name, const T *data, const adios2::Dims &shape, const adios2::Dims &start, const adios2::Dims &count, const adios2::vParams &operations, const bool endStep = false)

write overload that allows passing supported operations (e.g. lossy compression “zfp”, “mgard”, “sz”) to a self-described array variable

Parameters:

name – variable name
data – variable data data
shape – variable global MPI dimensions. Pass empty for local variables.
start – variable offset for current MPI rank. Pass empty for local variables.
count – variable dimension for current MPI rank. Local variables only have count.
operations – vector of operations, each entry is a std::pair:
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep if true.

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

template<class T> void write(const std::string &name, const T &value, const bool isLocalValue = false, const bool endStep = false)

Write a self-describing single-value variable

Parameters:

name – variable name
value – variable data value (can be r-value)
isLocalValue – true: local value (returned as GlobalArray), false: global value (returned as global value)
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.

Throws:

std::invalid_argument – (user input error) or std::runtime_error (system error)

template<class T> void read(const std::string &name, T *data, const size_t blockID = 0)

Reads into a pre-allocated pointer. When used with adios2::getstep reads current step

Parameters:

name – variable name
data – pre-allocated pointer to hold read data
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> void read(const std::string &name, T &value, const size_t blockID = 0)

Reads a value. When used with adios2::getstep reads current step value

Parameters:

name – variable name
value – output value, if variable is not found (name and type don’t match) the returned value address becomes nullptr
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> void read(const std::string &name, T *data, const size_t stepsStart, const size_t stepsCount = 1, const size_t blockID = 0)

Read accessing steps in random access mode. Not be used with adios2::getstep as it throw an exception when reading in stepping mode.

Parameters:

name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes nullptr
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> void read(const std::string &name, T &value, const size_t step, const size_t blockID = 0)

Reads into a single value for a single step. Not be used with adios2::getstep as it throws an exception when reading in stepping mode.

Parameters:

name – variable name
value – filled with value, if variable is not found (name, type and step don’t match) the returned value address becomes nullptr
step – selected single step
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> void read(const std::string &name, T *data, const adios2::Dims &start, const adios2::Dims &count, const size_t blockID = 0)

Reads into a pre-allocated pointer a selection piece in dimension. When used with adios2::getstep reads current step

Parameters:

name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes nullptr
start – variable local offset selection
count – variable local dimension selection from start
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> void read(const std::string &name, T *data, const adios2::Dims &start, const adios2::Dims &count, const size_t stepsStart, const size_t stepsCount, const size_t blockID = 0)

Reads into a pre-allocated pointer a selection piece in dimensions and steps. Not be used with adios2::getstep as it throws an exception when reading in stepping mode.

Parameters:

name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes a nullptr
start – variable local offset selection
count – variable local dimension selection from start
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be necessarily contiguous
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

template<class T> std::vector<T> read(const std::string &name, const size_t blockID = 0)

Reads entire variable for current step (streaming mode: step by step)

Parameters:

name – variable name
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

Returns:

data of variable name for current step. Single data will have a size=1 vector

template<class T> std::vector<T> read(const std::string &name, const size_t stepsStart, const size_t stepsCount = 1, const size_t blockID = 0)

Returns a vector with full variable dimensions for the current step selection. Not be used with adios2::getstep as it throw an exception when reading in stepping mode.

Parameters:

name – variable name
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

Returns:

data of variable name for current step, empty if exception is thrown

template<class T> std::vector<T> read(const std::string &name, const Dims &start, const Dims &count, const size_t blockID = 0)

Reads a selection piece in dimension for current step (streaming mode: step by step)

Parameters:

name – variable name
start – variable local offset selection
count – variable local dimension selection from start
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

Returns:

data of variable name for current step, empty if exception is thrown

template<class T> std::vector<T> read(const std::string &name, const Dims &start, const Dims &count, const size_t stepsStart, const size_t stepsCount, const size_t blockID = 0)

Reads a selection piece in dimension and a selection piece in steps (non-streaming mode). Not be used with adios2::getstep as it throw an exception when reading in stepping mode.

Parameters:

name – variable name
start – variable local offset selection
count – variable local dimension selection from start
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected

Throws:

throws – exception if variable name, dimensions or step not found

Returns:

variable data, empty if exception is thrown

template<class T> std::vector<T> read_attribute(const std::string &name, const std::string &variableName = "", const std::string separator = "/")

Reads an attribute returning a vector For single data vector size = 1

Parameters:

name – attribute name
variableName – default is empty, if not empty look for an attribute associated to a variable
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.

Returns:

vector containing attribute data

void end_step(): At write: ends the current step At read: use it in streaming mode to inform the writer that the reader is done consuming the step. No effect for file engines.

void close(): close current stream becoming inaccessible

size_t current_step() const noexcept

Return current step when getstep is called in a loop, read mode only

Returns:: current step

Friends

friend bool getstep(adios2::fstream &stream, adios2::fstep &step)

Gets step from stream Based on std::getline, enables reading on a step-by-step basis in a while or for loop. Read mode only

Parameters:

stream – input stream containing steps
step – output object current step, adios2::fstep in an alias to adios2::fstream with scope narrowed to one step

Returns:

true: step is valid, false: step is invalid (end of stream).

Matlab simple bindings

The ADIOS Matlab API supports reading data from ADIOS BP files with a simplified API that consists of three functions:

ADIOSOPEN returns a structure with information on an ADIOS BP File (variables and attributes).

ADIOSREAD reads in a variable from the file. It expects the info structure returned by ADIOSOPEN.

ADIOSCLOSE closes the file.

Organization of an ADIOS BP file

An ADIOS BP file contains a set of variables and attributes. Each variable in the group has a path, which defines a logical hierarchy of the variables within the file.

Time dimension of a variable

Variables can be written several times from a program, if they have a time dimension. The reader exposes the variables with an extra dimension, i.e. a 2D variable written over time is seen as a 3D variable. In MATLAB, the extra dimension is the last dimension (the slowest changing dimension). Since the reader allows reading an arbitrary slice of a variable, data for one timestep can be read in with slicing.

Min/max of arrays

The ADIOS BP format stores the min/max values in each variable. The info structure therefore contains these min/max values. There is practically no overhead to provide this information (along with the values of all attributes) even for file sizes of several terabytes.

In the Matlab console use help for these functions

>>> help adiosopen
>>> help adiosread
>>> help adiosclose

`ADIOSOPEN`

FILE = adiosopen(PATH): Open a file for reading pointed by PATH and return an information structure (FILE).

The returned FILE structure contains the following information

Name              File path

Handlers          Object handlers to pass on to ADIOS functions
  FileHandler        uint64 file handler
  GroupHandler       uint64 IO group object handler
  ADIOSHandler       uint64 ADIOS object handler

Variables         Structure array of variables
     Name            Path of variable
     Type            Matlab type class of data
     Dims            Array of dimensions
     StepsStart      First step's index for this variable in file, always at least 1
     StepsCount      Number of steps for this variable in file, always at least 1
     GlobalMin       Global minimum  of the variable (1-by-1 mxArray)
     GlobalMax       Global maximum of the variable

Attribute         Structure array of attributes
     Name            Path of attribute
     Type            Matlab type class of data
     Value           Attribute value

`ADIOSREAD`

Read data from a BP file opened with adiosopen. Provide the structure returned by adiosopen as the first input argument, and the path to a variable. Inspect file.Variables and file.Attributes for the list of variables and attributes available in a file.

data = adiosread(file, VARPATH)

Read the entire variable VARPATH from a BP file. file is the output of ADIOSOPEN. VARPATH is a string to a variable or attribute. If an N-dimensional array variable has multiple steps in the file this function reads all steps and returns an N+1 dimensional array where the last dimension equals the number of steps.

data = adiosread(file, INDEX)

Read the entire variable from a BP file. INDEX points to a variable in the file.Variables array.

data = adiosread(..., START, COUNT, STEPSTART, STEPCOUNT)

Read a portion of a variable.

START and COUNT:
A slice is defined as two arrays of N integers, where N is the
number of dimensions of the variable, describing the
"start" and "count" values. The "start" values start from 1.
    E.g. [1 5], [10 2] reads the first 10 values in the first dimension
and 2 values from the 5th position in the second dimension resulting in
a 10-by-2 array.
    You can use negative numbers to index from the end of the array
as in python. -1 refers to the last element of the array, -2 the one
before and so on.
    E.g. [-1], [1] reads in the last value of a 1D array.
         [1], [-1] reads in the complete 1D array.

STEPSTART and STEPCOUNT:
Similarly, the number of steps from a specific step can be read instead
of all data. Steps start from 1. Negative index can be used as well.
    E.g. -1, 1  will read in the last step from the file
         n, -1  will read all steps from 'n' to the last one

`ADIOSCLOSE`

adiosclose(file): Close file and free internal data structures. file is the structure returned by adiosopen.

High-Level APIs

C++ High-Level API

C++11 Write example

C++11 Read “step-by-step” example

adios2::fstream API documentation

Matlab simple bindings

Organization of an ADIOS BP file

Time dimension of a variable

Min/max of arrays

ADIOSOPEN

ADIOSREAD

ADIOSCLOSE

`adios2::fstream` API documentation

`ADIOSOPEN`

`ADIOSREAD`

`ADIOSCLOSE`