High-Level APIs
The high-level APIs are designed for simple tasks for which performance is not critical. Unlike the Full APIs, the high-level APIs only require a single object handler resembling a C++ fstream or a Python file I/O idiom. The high-level APIs are recommended to both first-time and advanced users; the low-level APIs being recommended only when performance testing identifies a bottleneck or when more control is needed.
Typical scenarios for using the simple high-level APIs are:
Reading a file to perform data analysis with libraries (matplotlib, scipy, etc.)
Interactive: few calls make interactive usage easier.
Saving data to files is small or personal projects
Online frameworks: e.g. Jupyter notebooks, see python-mpi examples running on MyBinder
The designed functionality syntax is closely related to the native language IO bindings for formatted text files e.g. C++ fstream
getline
, and Python file IO.
The main function calls are: open
(or constructor in C++), write
, read
and close
(or destructor in C++).
In addition, ADIOS2 borrows the corresponding language native syntax for advancing lines to advance the step in write mode, and for a “step-by-step” streaming basis in read mode.
See each language section in this chapter for a write/read example.
Note
The simplified APIs are based on language native file IO interface. Hence write
and read
calls are always synchronized and variables data memory is ready to use immediately after these calls.
Currently ADIOS2 support bindings for the following languages and their minimum standards:
Language |
Standard |
Interface |
Based on |
C++ |
11/newer |
|
|
Python |
2.7/3 |
|
Python IO |
Matlab |
The following sections provide a summary of the API calls on each language and links to Write and Read examples to put it all together.
C++ High-Level API
C++11 High-Level APIs are based on a single object adios2::fstream
Caution
DO NOT place use namespace adios2
in your C++ code.
Use adios2::fstream
directly to prevent conflicts with std::fstream
.
C++11 Write example
#include <adios2.h>
...
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// Nx, Ny from application, std::size_t
const adios2::Dims shape{Nx, Ny * static_cast<std::size_t>(size)};
const adios2::Dims start{0, Ny * static_cast<std::size_t>(rank)};
const adios2::Dims count{Nx, Ny};
adios2::fstream oStream("cfd.bp", adios2::fstream::out, MPI_COMM_WORLD);
// NSteps from application
for (std::size_t step = 0; step < NSteps; ++step)
{
if(rank == 0 && step == 0) // global variable
{
oStream.write<int32_t>("size", size);
}
// physicalTime double, <double> is optional
oStream.write<double>( "physicalTime", physicalTime );
// T and P are std::vector<float>
oStream.write( "temperature", T.data(), shape, start, count );
// adios2::endl will advance the step after writing pressure
oStream.write( "pressure", P.data(), shape, start, count, adios2::end_step );
}
// Calling close is mandatory!
oStream.close();
C++11 Read “step-by-step” example
#include <adios2.h>
...
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// Selection Window from application, std::size_t
const adios2::Dims start{0, 0};
const adios2::Dims count{SelX, SelY};
if( rank == 0)
{
// if only one rank is active use MPI_COMM_SELF
adios2::fstream iStream("cfd.bp", adios2::fstream::in, MPI_COMM_SELF);
adios2::fstep iStep;
while (adios2::getstep(iStream, iStep))
{
if( iStep.currentstep() == 0 )
{
const std::size_t sizeOriginal = iStep.read<std::size_t>("size");
}
const double physicalTime = iStream.read<double>( "physicalTime");
const std::vector<float> temperature = iStream.read<float>( "temperature", start, count );
const std::vector<float> pressure = iStream.read<float>( "pressure", start, count );
}
// Don't forget to call close!
iStream.close();
}
adios2::fstream
API documentation
-
class adios2::fstream
Public Types
Public Functions
-
fstream(const std::string &name, adios2::fstream::openmode mode, MPI_Comm comm, const std::string engineType = "BPFile")
High-level API MPI constructor, based on C++11 fstream. Allows for passing parameters in source code.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
engineType – available adios2 engine
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
fstream(const std::string &name, const adios2::fstream::openmode mode, MPI_Comm comm, const std::string &configFile, const std::string ioInConfigFile)
High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
fstream(const std::string &name, const adios2::fstream::openmode mode, const std::string engineType = "BPFile")
High-level API non-MPI constructor, based on C++11 fstream. Allows for passing parameters in source code.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
engineType – available adios2 engine
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
fstream(const std::string &name, const adios2::fstream::openmode mode, const std::string &configFile, const std::string ioInConfigFile)
High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
fstream() = default
Empty constructor, allows the use of open later in the code
-
~fstream() = default
Using RAII STL containers only
-
explicit operator bool() const noexcept
Checks if fstream object is valid
-
void open(const std::string &name, const openmode mode, MPI_Comm comm, const std::string engineType = "BPFile")
High-level API MPI open, based on C++11 fstream. Allows for passing parameters in source code. Used after empty constructor.
- Parameters
name – stream name
mode – adios2::fstream::in (Read), adios2::fstream::out (Write), adios2::fstream::app (Append)
comm – MPI communicator establishing domain for fstream
engineType – available adios2 engine
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
void open(const std::string &name, const openmode mode, MPI_Comm comm, const std::string configFile, const std::string ioInConfigFile)
High-level API MPI constructor, based on C++11 fstream. Allows for runtime config file. Used after empty constructor.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
comm – MPI communicator establishing domain for fstream
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
void open(const std::string &name, const openmode mode, const std::string engineType = "BPFile")
High-level API non-MPI open, based on C++11 fstream. Allows for passing parameters in source code. Used after empty constructor.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
engineType – available adios2 engine
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
void open(const std::string &name, const openmode mode, const std::string configFile, const std::string ioInConfigFile)
High-level API non-MPI constructor, based on C++11 fstream. Allows for runtime config file. Used after empty constructor.
- Parameters
name – stream name
mode – fstream::in (Read), fstream::out (Write), fstream::app (Append)
configFile – adios2 runtime configuration file
ioInConfigFile – specific io name in configFile
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
void set_parameter(const std::string key, const std::string value) noexcept
Set a single stream parameter based on Engine supported parameters. MUST be passed before the first call to write or read. See: https://adios2.readthedocs.io/en/latest/engines/engines.html
- Parameters
key – input parameter key
value – input parameter value
-
void set_parameters(const adios2::Params ¶meters) noexcept
Set stream parameters based on Engine supported parameters. MUST be passed before the first call to write or read. See: https://adios2.readthedocs.io/en/latest/engines/engines.html
- Parameters
parameters – input key/value parameters
-
template<class T>
void write_attribute(const std::string &name, const T &value, const std::string &variableName = "", const std::string separator = "/", const bool endStep = false) Define attribute inside fstream or for a variable after write. Single value input version.
- Parameters
name – unique attribute identifier IO object or for a Variable if variableName is not empty (associated to a variable)
value – single data value
variableName – default is empty, if not empty attributes is associated to a variable after a write
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.
-
template<class T>
void write_attribute(const std::string &name, const T *data, const size_t size, const std::string &variableName = "", const std::string separator = "/", const bool endStep = false) Define attribute inside fstream or for a variable after write. Array input version.
- Parameters
name – unique attribute identifier IO object or for a Variable if variableName is not empty (associated to a variable)
data – pointer to user data
size – number of data elements
variableName – default is empty, if not empty attributes is associated to a variable after a write
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.
-
template<class T>
void write(const std::string &name, const T *data, const adios2::Dims &shape = adios2::Dims(), const adios2::Dims &start = adios2::Dims(), const adios2::Dims &count = adios2::Dims(), const bool endStep = false) writes a self-describing array variable
- Parameters
name – variable name
data – variable data data
shape – variable global MPI dimensions. Pass empty for local variables.
start – variable offset for current MPI rank. Pass empty for local variables.
count – variable dimension for current MPI rank. Local variables only have count.
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep if true.
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
template<class T>
void write(const std::string &name, const T *data, const adios2::Dims &shape, const adios2::Dims &start, const adios2::Dims &count, const adios2::vParams &operations, const bool endStep = false) write overload that allows passing supported operations (e.g. lossy compression “zfp”, “mgard”, “sz”) to a self-described array variable
- Parameters
name – variable name
data – variable data data
shape – variable global MPI dimensions. Pass empty for local variables.
start – variable offset for current MPI rank. Pass empty for local variables.
count – variable dimension for current MPI rank. Local variables only have count.
operations – vector of operations, each entry is a std::pair:
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep if true.
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
template<class T>
void write(const std::string &name, const T &value, const bool isLocalValue = false, const bool endStep = false) Write a self-describing single-value variable
- Parameters
name – variable name
value – variable data value (can be r-value)
isLocalValue – true: local value (returned as GlobalArray), false: global value (returned as global value)
endStep – similar to std::endStep, end current step and flush (default). Use adios2::endStep for true.
- Throws
std::invalid_argument – (user input error) or std::runtime_error (system error)
-
template<class T>
void read(const std::string &name, T *data, const size_t blockID = 0) Reads into a pre-allocated pointer. When used with adios2::getstep reads current step
- Parameters
name – variable name
data – pre-allocated pointer to hold read data
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
void read(const std::string &name, T &value, const size_t blockID = 0) Reads a value. When used with adios2::getstep reads current step value
- Parameters
name – variable name
value – output value, if variable is not found (name and type don’t match) the returned value address becomes nullptr
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
void read(const std::string &name, T *data, const size_t stepsStart, const size_t stepsCount = 1, const size_t blockID = 0) Read accessing steps in random access mode. Not be used with adios2::getstep as it throw an exception when reading in stepping mode.
- Parameters
name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes nullptr
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
void read(const std::string &name, T &value, const size_t step, const size_t blockID = 0) Reads into a single value for a single step. Not be used with adios2::getstep as it throws an exception when reading in stepping mode.
- Parameters
name – variable name
value – filled with value, if variable is not found (name, type and step don’t match) the returned value address becomes nullptr
step – selected single step
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
void read(const std::string &name, T *data, const adios2::Dims &start, const adios2::Dims &count, const size_t blockID = 0) Reads into a pre-allocated pointer a selection piece in dimension. When used with adios2::getstep reads current step
- Parameters
name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes nullptr
start – variable local offset selection
count – variable local dimension selection from start
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
void read(const std::string &name, T *data, const adios2::Dims &start, const adios2::Dims &count, const size_t stepsStart, const size_t stepsCount, const size_t blockID = 0) Reads into a pre-allocated pointer a selection piece in dimensions and steps. Not be used with adios2::getstep as it throws an exception when reading in stepping mode.
- Parameters
name – variable name
data – pre-allocated pointer to hold read data, if variable is not found (name and type don’t match) it becomes a nullptr
start – variable local offset selection
count – variable local dimension selection from start
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be necessarily contiguous
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
-
template<class T>
std::vector<T> read(const std::string &name, const size_t blockID = 0) Reads entire variable for current step (streaming mode: step by step)
- Parameters
name – variable name
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
- Returns
data of variable name for current step. Single data will have a size=1 vector
-
template<class T>
std::vector<T> read(const std::string &name, const size_t stepsStart, const size_t stepsCount = 1, const size_t blockID = 0) Returns a vector with full variable dimensions for the current step selection. Not be used with adios2::getstep as it throw an exception when reading in stepping mode.
- Parameters
name – variable name
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
- Returns
data of variable name for current step, empty if exception is thrown
-
template<class T>
std::vector<T> read(const std::string &name, const Dims &start, const Dims &count, const size_t blockID = 0) Reads a selection piece in dimension for current step (streaming mode: step by step)
- Parameters
name – variable name
start – variable local offset selection
count – variable local dimension selection from start
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
- Returns
data of variable name for current step, empty if exception is thrown
-
template<class T>
std::vector<T> read(const std::string &name, const Dims &start, const Dims &count, const size_t stepsStart, const size_t stepsCount, const size_t blockID = 0) Reads a selection piece in dimension and a selection piece in steps (non-streaming mode). Not be used with adios2::getstep as it throw an exception when reading in stepping mode.
- Parameters
name – variable name
start – variable local offset selection
count – variable local dimension selection from start
stepsStart – variable initial step (relative to the variable first appearance, not absolute step in stream)
stepsCount – variable number of steps form step_start, don’t have to be contiguous, necessarily
blockID – required for local variables, specify current block to be selected
- Throws
throws – exception if variable name, dimensions or step not found
- Returns
variable data, empty if exception is thrown
-
template<class T>
std::vector<T> read_attribute(const std::string &name, const std::string &variableName = "", const std::string separator = "/") Reads an attribute returning a vector For single data vector size = 1
- Parameters
name – attribute name
variableName – default is empty, if not empty look for an attribute associated to a variable
separator – default is “/”, hierarchy between variable name and attribute, e.g. variableName/attribute1, variableName::attribute1. Not used if variableName is empty.
- Returns
vector containing attribute data
-
void end_step()
At write: ends the current step At read: use it in streaming mode to inform the writer that the reader is done consuming the step. No effect for file engines.
-
void close()
close current stream becoming inaccessible
-
size_t current_step() const noexcept
Return current step when getstep is called in a loop, read mode only
- Returns
current step
Friends
-
friend bool getstep(adios2::fstream &stream, adios2::fstep &step)
Gets step from stream Based on std::getline, enables reading on a step-by-step basis in a while or for loop. Read mode only
- Parameters
stream – input stream containing steps
step – output object current step, adios2::fstep in an alias to adios2::fstream with scope narrowed to one step
- Returns
true: step is valid, false: step is invalid (end of stream).
-
fstream(const std::string &name, adios2::fstream::openmode mode, MPI_Comm comm, const std::string engineType = "BPFile")
Python High-Level API
Python simple bindings follow closely Python style directives. Just like the full APIs, they rely on numpy and, optionally, on mpi4py
, if the underlying ADIOS2 library is compiled with MPI.
For online examples on MyBinder :
Python Write example
from mpi4py import MPI
import numpy as np
import adios2
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
...
shape = [size * nx]
start = [rank * nx]
count = [nx]
# with-as will call adios2.close on fh at the end
with adios2.open("cfd.bp", "w", comm) as fh:
# NSteps from application
for i in range(0, NSteps):
if(rank == 0 and i == 0):
fh.write("size", np.array([size]))
fh.write("physical_time", np.array([physical_time]) )
# temperature and pressure are numpy arrays
fh.write("temperature", temperature, shape, start, count)
# advances to next step
fh.write("pressure", pressure, shape, start, count, end_step=True)
Python Read “step-by-step” example
from mpi4py import MPI
import numpy as np
import adios2
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
...
shape = [size * nx]
start = [rank * nx]
count = [nx]
if( rank == 0 ):
# with-as will call adios2.close on fh at the end
# if only one rank is active pass MPI.COMM_SELF
with adios2.open("cfd.bp", "r", MPI.COMM_SELF) as fh:
for fstep in fh:
# inspect variables in current step
step_vars = fstep.available_variables()
# print variables information
for name, info in step_vars.items():
print("variable_name: " + name)
for key, value in info.items():
print("\t" + key + ": " + value)
print("\n")
# track current step
step = fstep.current_step()
if( step == 0 ):
size_in = fstep.read("size")
# read variables return a numpy array with corresponding selection
physical_time = fstep.read("physical_time")
temperature = fstep.read("temperature", start, count)
pressure = fstep.read("pressure", start, count)
Caution
When reading in stepping mode with the for-in directive, as in the example above, use the step handler (fstep
) inside the loop rather than the global handler (fh
)
File class API
- adios2.open(*args, **kwargs)
Overloaded function.
open(name: str, mode: str, comm: MPI4PY_Comm, engine_type: str=’BPFile’) -> adios2::py11::File
Simple API MPI open, based on python IO. Allows for passing parameters in source code.
- Parameters
- name
stream name
- mode
“w” : write, “r” : read, “a” : append (append not yet supported)
- comm (mpi4py)
MPI communicator
- engine_type
adios2 engine type, default=BPFile
- Returns
- file (adios2 stream)
handler to adios File for the simple Python API
open(name: str, mode: str, comm: MPI4PY_Comm, config_file: str, io_in_config_file: str) -> adios2::py11::File
Simple API MPI open, based on python IO. Allows for passing a runtime configuration file in xml format and the name of the io element related to the returning File.
- Parameters
- name
stream name
- mode
“w” : write, “r” : read, “a” : append (append not yet supported)
- comm (mpi4py)
MPI communicator
- config_file
adios2 runtime configuration file name, in xml format
- io_in_config_file
io element in configfile related to returning File
- Returns
- file (adios2 stream)
handler to adios File for the simple Python API
open(name: str, mode: str, engine_type: str=’BPFile’) -> adios2::py11::File
High-level API, file object open
open(name: str, mode: str, config_file: str, io_in_config_file: str) -> adios2::py11::File
High-level API, file object open with a runtime config file
- class adios2.File
- add_transport(self: adios2.File, type: str, parameters: Dict[str, str] = {}) int
Adds a transport and its parameters to current IO. Must be supported by current engine type.
- Parameters
- type
must be a supported transport type for current engine.
- parameters
acceptable parameters for a particular transport CAN’T use the keywords “Transport” or “transport” in key
- Returns
- transport_index
handler to added transport
- available_attributes(self: adios2.File) Dict[str, Dict[str, str]]
Returns a 2-level dictionary with attribute information. Read mode only.
- Returns
- attributes dictionary
- key
attribute name
- value
attribute information dictionary
- available_variables(self: adios2.File) Dict[str, Dict[str, str]]
Returns a 2-level dictionary with variable information. Read mode only.
- Returns
- variables dictionary
- key
variable name
- value
variable information dictionary
- close(self: adios2.File) None
Closes file, thus becoming unreachable. Not required if using open in a with-as statement. Required in all other cases per-open to avoid resource leaks.
- current_step(self: adios2.File) int
Inspect current step when using for-in loops, read mode only
- Returns
current step
- end_step(self: adios2.File) None
Write mode: advances to the next step. Convenient when declaring variable attributes as advancing to the next step is not attached to any variable.
Read mode: in streaming mode releases the current step (no effect in file based engines)
- read(*args, **kwargs)
Overloaded function.
read(self: adios2.File, name: str, block_id: int=0) -> array
Reads entire variable for current step (streaming mode step by step)
- Parameters
- name
variable name
- block_id
required for local array variables
- Returns
- array
values of variable name for current step. Single values will have a shape={1} numpy array
read(self: adios2.File, name: str, start: List[int]=[], count: List[int]=[], block_id: int=0) -> array
Reads a selection piece in dimension for current step (streaming mode step by step)
- Parameters
- name
variable name
- start
variable local offset selection (defaults to (0, 0, …)
- count
variable local dimension selection from start defaults to whole array for GlobalArrays, or selected Block size for LocalArrays
- block_id
required for local array variables
- Returns
- array
values of variable name for current step empty if exception is thrown
read(self: adios2.File, name: str, start: List[int], count: List[int], step_start: int, step_count: int, block_id: int=0) -> array
Random access read allowed to select steps, only valid with File Engines
- Parameters
- name
variable to be read
- start
variable offset dimensions
- count
variable local dimensions from offset
- step_start
variable step start
- step_count
variable number of steps to read from step_start
- block_id
required for local array variables
- Returns
- array
resulting array from selection
- read_attribute(self: adios2.File, name: str, variable_name: str = '', separator: str = '/') array
Reads a numpy based attribute
- Parameters
- name
attribute name
- variable_name
if attribute is associated with a variable
- separator
concatenation string between variable_name and attribute e.g. variable_name + separator + name (var/attr) Not used if variable_name is empty
- Returns
- array
resulting array attribute data
- read_attribute_string(self: adios2.File, name: str, variable_name: str = '', separator: str = '/') List[str]
Read a string attribute
- Parameters
- name
attribute name
- variable_name
if attribute is associated with a variable
- separator
concatenation string between variable_name and attribute e.g. variable_name + separator + name (var/attr) Not used if variable_name is empty
- Returns
- list
resulting string list attribute data
- read_string(*args, **kwargs)
Overloaded function.
read_string(self: adios2.File, name: str, block_id: int=0) -> List[str]
Reads string value for current step (use for streaming mode step by step)
- Parameters
- name
string variable name
- block_id
required for local variables
Returns
- list
data string values. For global values: returns 1 element For local values: returns n-block elements
read_string(self: adios2.File, name: str, step_start: int, step_count: int, block_id: int=0) -> List[str]
Reads string value for a certain step (random access mode)
- Parameters
- name
string variable name
- step_start
variable step start
- step_count
variable number of steps to read from step_start
- block_id
required for local variables
- Returns
- string
data string values for a certain step range.
- set_parameter(self: adios2.File, key: str, value: str) None
Sets a single parameter. Overwrites value if key exists.
- Parameters
- key
input parameter key
- value
parameter value
- set_parameters(self: adios2.File, parameters: Dict[str, str]) None
Sets parameters using a dictionary. Removes any previous parameter.
- Parameters
- parameters
input key/value parameters
- value
parameter value
- steps(self: adios2.File) int
Inspect available number of steps, for file engines, read mode only
- Returns
steps
- write(*args, **kwargs)
Overloaded function.
write(self: adios2.File, name: str, array: array, shape: List[int]=[], start: List[int]=[], count: List[int]=[], end_step: bool=False) -> None
writes a self-describing array (numpy) variable
- Parameters
- name
variable name
- array
variable data values
- shape
variable global MPI dimensions. Pass empty numpy array for local variables.
- start
variable offset for current MPI rank. Pass empty numpy array for local variables.
- count
variable dimension for current MPI rank. Pass a numpy array for local variables.
- end_step
end current step, begin next step and flush (default = false).
write(self: adios2.File, name: str, array: array, shape: List[int], start: List[int], count: List[int], operations: List[Tuple[str, Dict[str, str]]], end_step: bool=False) -> None
writes a self-describing array (numpy) variable with operations e.g. compression: ‘zfp’, ‘mgard’, ‘sz’
- Parameters
- name
variable name
- array
variable data values
- shape
variable global MPI dimensions. Pass empty numpy array for local variables.
- start
variable offset for current MPI rank. Pass empty numpy array for local variables.
- count
variable dimension for current MPI rank. Pass a numpy array for local variables.
- end_step
end current step, begin next step and flush (default = false).
write(self: adios2.File, name: str, array: array, local_value: bool=False, end_step: bool=False) -> None
writes a self-describing single value array (numpy) variable
- Parameters
- name
variable name
- array
variable data single value
- local_value
true: local value, false: global value
- end_step
end current step, begin next step and flush (default = false).
write(self: adios2.File, name: str, string: str, local_value: bool=False, end_step: bool=False) -> None
writes a self-describing single value string variable
- Parameters
- name
variable name
- string
variable data single value
- local_value
true: local value, false: global value
- end_step
end current step, begin next step and flush (default = false).
- write_attribute(*args, **kwargs)
Overloaded function.
write_attribute(self: adios2.File, name: str, array: array, variable_name: str=’’, separator: str=’/’, end_step: bool=False) -> None
writes a self-describing single value array (numpy) variable
- Parameters
- name
attribute name
- array
attribute numpy array data
- variable_name
if attribute is associated with a variable
- separator
concatenation string between variable_name and attribute e.g. variable_name + separator + name (“var/attr”) Not used if variable_name is empty
- end_step
end current step, begin next step and flush (default = false).
write_attribute(self: adios2.File, name: str, string_value: str, variable_name: str=’’, separator: str=’/’, end_step: bool=False) -> None
writes a self-describing single value array (numpy) variable
- Parameters
- name
attribute name
- string_value
attribute single string
- variable_name
if attribute is associated with a variable
- separator
concatenation string between variable_name and attribute e.g. variable_name + separator + name (“var/attr”) Not used if variable_name is empty
- end_step
end current step, begin next step and flush (default = false).
write_attribute(self: adios2.File, name: str, string_array: List[str], variable_name: str=’’, separator: str=’/’, end_step: bool=False) -> None
writes a self-describing single value array (numpy) variable
- Parameters
- name
attribute name
- string_array
attribute string array
- variable_name
if attribute is associated with a variable
- separator
concatenation string between variable_name and attribute e.g. variable_name + separator + name (“var/attr”) Not used if variable_name is empty
- end_step
end current step, begin next step and flush (default = false).
Matlab simple bindings
The ADIOS Matlab API supports reading data from ADIOS BP files with a simplified API that consists of three functions:
ADIOSOPEN
returns a structure with information on an ADIOS BP File (variables and attributes).
ADIOSREAD
reads in a variable from the file. It expects the info structure returned byADIOSOPEN
.
ADIOSCLOSE
closes the file.
Organization of an ADIOS BP file
An ADIOS BP file contains a set of variables and attributes. Each variable in the group has a path, which defines a logical hierarchy of the variables within the file.
Time dimension of a variable
Variables can be written several times from a program, if they have a time dimension. The reader exposes the variables with an extra dimension, i.e. a 2D variable written over time is seen as a 3D variable. In MATLAB, the extra dimension is the last dimension (the slowest changing dimension). Since the reader allows reading an arbitrary slice of a variable, data for one timestep can be read in with slicing.
Min/max of arrays
The ADIOS BP format stores the min/max values in each variable. The info structure therefore contains these min/max values. There is practically no overhead to provide this information (along with the values of all attributes) even for file sizes of several terabytes.
In the Matlab console use help for these functions
>>> help adiosopen
>>> help adiosread
>>> help adiosclose
ADIOSOPEN
FILE = adiosopen(PATH)
Open a file for reading pointed by
PATH
and return an information structure (FILE
).
The returned FILE structure contains the following information
Name File path Handlers Object handlers to pass on to ADIOS functions FileHandler uint64 file handler GroupHandler uint64 IO group object handler ADIOSHandler uint64 ADIOS object handler Variables Structure array of variables Name Path of variable Type Matlab type class of data Dims Array of dimensions StepsStart First step's index for this variable in file, always at least 1 StepsCount Number of steps for this variable in file, always at least 1 GlobalMin Global minimum of the variable (1-by-1 mxArray) GlobalMax Global maximum of the variable Attribute Structure array of attributes Name Path of attribute Type Matlab type class of data Value Attribute value
ADIOSREAD
Read data from a BP file opened with adiosopen
.
Provide the structure returned by adiosopen
as the first input argument,
and the path to a variable.
Inspect file.Variables
and file.Attributes
for the list of variables
and attributes available in a file.
data = adiosread(file, VARPATH)
Read the entire variable VARPATH
from a BP file. file
is the output of ADIOSOPEN
.
VARPATH
is a string to a variable or attribute.
If an N-dimensional array variable has multiple steps in the file
this function reads all steps and returns an N+1 dimensional array
where the last dimension equals the number of steps.
data = adiosread(file, INDEX)
Read the entire variable from a BP file.
INDEX
points to a variable in the file.Variables
array.
data = adiosread(..., START, COUNT, STEPSTART, STEPCOUNT)
Read a portion of a variable.
START and COUNT: A slice is defined as two arrays of N integers, where N is the number of dimensions of the variable, describing the "start" and "count" values. The "start" values start from 1. E.g. [1 5], [10 2] reads the first 10 values in the first dimension and 2 values from the 5th position in the second dimension resulting in a 10-by-2 array. You can use negative numbers to index from the end of the array as in python. -1 refers to the last element of the array, -2 the one before and so on. E.g. [-1], [1] reads in the last value of a 1D array. [1], [-1] reads in the complete 1D array. STEPSTART and STEPCOUNT: Similarly, the number of steps from a specific step can be read instead of all data. Steps start from 1. Negative index can be used as well. E.g. -1, 1 will read in the last step from the file n, -1 will read all steps from 'n' to the last one
ADIOSCLOSE
adiosclose(file)
Close file and free internal data structures.
file
is the structure returned byadiosopen
.