Install from Source

ADIOS2 uses CMake version 3.6 or above, for building, testing and installing the library and utilities.

Building, Testing, and Installing ADIOS 2

Caution

Always do a fresh build from scratch if your source is updated with considerable changes e.g. git pull

To build ADIOS v2.x, clone the repository and invoke the canonical CMake build sequence:

$ git clone https://github.com/ornladios/ADIOS2.git ADIOS2
$ mkdir adios2-build && cd adios2-build
$ cmake ../ADIOS2
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
...

ADIOS2 build configuration:
  ...
  Features:
    Library Type: shared
    Build Type:   Release
    Testing: ON
    Build Options:
      BZip2    : ON
      ZFP      : OFF
      SZ       : OFF
      MGARD    : OFF
      MPI      : ON
      DataMan  : ON
      SST      : ON
      ZeroMQ   : ON
      HDF5     : ON
      Python   : ON
      Fortran  : ON
      SysVShMem: ON
      Endian_Reverse: OFF

Then compile using

$ make -j 16

Optionally, run the tests:

$ ctest
Test project /home/wgodoy/workspace/build
        Start   1: ADIOSInterfaceWriteTest.DefineVar_int8_t_1x10
  1/295 Test   #1: ADIOSInterfaceWriteTest.DefineVar_int8_t_1x10 .........................   Passed    0.16 sec
        Start   2: ADIOSInterfaceWriteTest.DefineVar_int16_t_1x10
  2/295 Test   #2: ADIOSInterfaceWriteTest.DefineVar_int16_t_1x10 ........................   Passed    0.06 sec
        Start   3: ADIOSInterfaceWriteTest.DefineVar_int32_t_1x10

      ...

        Start 294: ADIOSBZip2Wrapper.WrongParameterValue
294/295 Test #294: ADIOSBZip2Wrapper.WrongParameterValue .................................   Passed    0.00 sec
        Start 295: ADIOSBZip2Wrapper.WrongBZip2Name
295/295 Test #295: ADIOSBZip2Wrapper.WrongBZip2Name ......................................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 295

Total Test time (real) =   95.95 sec

And finally, use the standard invocation to install:

$ make install

CMake Options

The following options can be specified with CMake’s -DVAR=VALUE syntax. The default option is highlighted.

VAR

VALUE

Description

ADIOS2_USE_MPI

ON/OFF

MPI or non-MPI (serial) build.

ADIOS2_USE_ZeroMQ

ON/OFF

ZeroMQ for the DataMan engine.

ADIOS2_USE_HDF5

ON/OFF

HDF5 engine. If HDF5 is not on the syspath, it can be set using -DHDF5_ROOT=/path/to/hdf5

ADIOS2_USE_Python

ON/OFF

Python bindings. Python 3 will be used if found. If you want to specify a particular python version use -DPYTHON_EXECUTABLE=/path/to/interpreter/python

ADIOS2_USE_Fortran

ON/OFF

Bindings for Fortran 90 or above.

ADIOS2_USE_SST

ON/OFF

Simplified Staging Engine (SST) and its dependencies, requires MPI. Can optionally use LibFabric for RDMA transport. Specify the LibFabric install manually with the -DLIBFABRIC_ROOT=… option.

ADIOS2_USE_BZip2

ON/OFF

BZIP2 compression.

ADIOS2_USE_ZFP

ON/OFF

ZFP compression (experimental).

ADIOS2_USE_SZ

ON/OFF

SZ compression (experimental).

ADIOS2_USE_MGARD

ON/OFF

MGARD compression (experimental).

ADIOS2_USE_PNG

ON/OFF

PNG compression (experimental).

ADIOS2_USE_Blosc

ON/OFF

Blosc compression (experimental).

ADIOS2_USE_Endian_Reverse

ON/OFF

Enable endian conversion if a different endianness is detected between write and read.

ADIOS2_USE_IME

ON/OFF

DDN IME transport.

In addition to the ADIOS2_USE_Feature options, the following options are also available to control how the library gets built:

CMake VAR Options

Values

Description |

BUILD_SHARED_LIBS

ON/OFF

Build shared libraries.

ADIOS2_BUILD_EXAMPLES

ON/OFF

Build examples.

ADIOS2_BUILD_TESTING

ON/OFF

Build test code.

CMAKE_INSTALL_PREFIX

/path/to/install (/usr/local)

Installation location.

CMAKE_BUILD_TYPE

Debug/Release/RelWithDebInfo/MinSizeRel

Compiler optimization levels.

Example: Enable Fortran, disable Python bindings and ZeroMQ functionality

$ cmake -DADIOS2_USE_Fortran=ON -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF ../ADIOS2

Notes:

To provide search paths to CMake for dependency searching:

  • Use a PackageName_ROOT variable to provide the location of a specific package.

  • Add an install prefix to the CMAKE_PREFIX_PATH which is searched for all packages.

  • Both the PackageName_ROOT and CMAKE_PREFIX_PATH can be used as either environment variables or CMake variables (passed via -D), where the CMake variable takes prescedence.

# Several dependencies are installed under /opt/foo/bar and then a
# single dependency (HDF5 in this case) is installed in /opt/hdf5/1.12.0
$ export CMAKE_PREFIX_PATH=/opt/foo/bar
$ cmake -DHDF5_ROOT=/opt/hdf5/1.12.0 ../ADIOS2

Example: the following configuration will build, test and install under /opt/adios2/2.6.0 an optimized (Release) version of ADIOS2.

$ cd build
$ cmake -DADIOS2_USE_Fortran=ON -DCMAKE_INSTALL_PREFIX=/opt/adios2/2.6.0 -DCMAKE_BUILD_Type=Release ../ADIOS2
$ make -j16
$ ctest
$ make install

For a fully configurable build script, click here.

Building on HPC Systems

  1. Modules: Make sure all “module” dependencies are loaded and that minimum requirements are satisfied. Load the latest CMake module as many HPC systems default to an outdated version. Build with a C++11-compliant compiler, such as gcc >= 4.8.1, Intel >= 15, and PGI >= 15.

  2. Static/Dynamic build: On Cray systems such as Titan, the default behavior is static linkage, thus CMake builds ADIOS2 creates the static library libadios2.a by default. Read the system documentation to enable dynamic compilation, usually by setting an environment variable such as CRAYPE_LINK_TYPE=dynamic. Click here for a fully configurable script example on OLCF systems.

  3. Big Endian and 32-bit systems: ADIOS2 hasn’t been tested on big endian or 32-bit systems. Please be aware before attempting to run.

  4. CMake minimum version: The ADIOS2 build system requires a minimum CMake version of 3.6.0. However, IBM XL, Cray, and PGI compilers require version 3.9.0 or newer.

  5. PGI compilers and C++11 support: Version 15 of the PGI compiler is C++11 compliant. However it relies on the C++ standard library headers supplied by the system version of GCC, which may or may support all the C++11 features used in ADIOS2. On many systems (Titan at OLCF, for example) even though the PGI compiler supports C++11, the configured GCC and its headers do not (4.3.x on Cray Linux Environment, and v5 systems like Titan). To configure the PGI compiler to use a newer GCC, you must create a configuration file in your home directory that overrides the PGI compiler’s default configuration. On Titan, the following steps will re-configure the PGI compiler to use GCC 6.3.0 provided by a module:

$ module load gcc/6.3.0
$ makelocalrc $(dirname $(which pgc++)) -gcc $(which gcc) -gpp $(which g++) -g77 $(which gfortran) -o -net 1>${HOME}/.mypgirc 2>/dev/null

Installing the ADIOS2 library and the C++ and C bindings

By default, ADIOS2 will build the C++11 libadios2 library and the C and C++ bindings.

  1. Minimum requirements:

    • A C++11 compliant compiler

    • An MPI C implementation on the syspath, or in a location identifiable by CMake.

  2. Linking make install will copy the required headers and libraries into the directory specified by CMAKE_INSTALL_PREFIX:

    • Libraries:

      • lib/libadios2.* C++11 and C bindings

    • Headers:

      • include/adios2.h C++11 namespace adios2

      • include/adios2_c.h C prefix adios2_

    • Config file: run this command to get installation info

      • bin/adios2-config

Enabling the Python bindings

To enable the Python bindings in ADIOS2, based on PyBind11, make sure to follow these guidelines:

  • Minimum requirements:

    • Python 2.7 and above.

    • numpy

    • mpi4py

  • Running: If CMake enables Python compilation, an adios2.so library containing the Python module is generated in the build directory under lib/pythonX.X/site-packages/

    • make sure your PYTHONPATH environment variable contains the path to adios2.so.

    • make sure the Python interpreter is compatible with the version used for compilation via python --version.

    • Run the Python tests with ctest -R Python

    • Run helloBPWriter.py and helloBPTimeWriter.py

    • In Python 2.7:

    $ mpirun -n 4 python helloBPWriter.py
    $ python helloBPWriter.py
    
    • In Python 3:

    $ mpirun -n 4 python3 helloBPWriter.py
    $ python3 helloBPWriter.py
    

Enabling the Fortran bindings

  1. Minimum requirements:

    • A Fortran 90 compliant compiler

    • A Fortran MPI implementation

  2. Linking the Fortran bindings: make install will copy the required library and modules into the directory specified by CMAKE_INSTALL_PREFIX

    • Library (note that libadios2 must also be linked) - lib/libadios2_f.* - lib/libadios2.*

    • Modules - include/adios2/fortran/*.mod

  3. Module adios2: only module required to be used in an application use adios

Running Tests

ADIOS2 uses googletest to enable automatic testing after a CMake build. To run tests just type after building with make, run:

$ ctest
  or
$ make test

The following screen will appear providing information on the status of each finalized test:

Test project /home/wfg/workspace/build
Start  1: ADIOSInterfaceWriteTest.DefineVarChar1x10
1/46 Test  #1: ADIOSInterfaceWriteTest.DefineVarChar1x10 .......................   Passed    0.06 sec
Start  2: ADIOSInterfaceWriteTest.DefineVarShort1x10
2/46 Test  #2: ADIOSInterfaceWriteTest.DefineVarShort1x10 ......................   Passed    0.04 sec
Start  3: ADIOSInterfaceWriteTest.DefineVarInt1x10
3/46 Test  #3: ADIOSInterfaceWriteTest.DefineVarInt1x10 ........................   Passed    0.04 sec
Start  4: ADIOSInterfaceWriteTest.DefineVarLong1x10
...
128/130 Test #128: ADIOSZfpWrapper.UnsupportedCall ..........................................   Passed    0.05 sec
    Start 129: ADIOSZfpWrapper.MissingMandatoryParameter
129/130 Test #129: ADIOSZfpWrapper.MissingMandatoryParameter ................................   Passed    0.05 sec
    Start 130: */TestManyVars.DontRedefineVars/*
130/130 Test #130: */TestManyVars.DontRedefineVars/* ........................................   Passed    0.08 sec

100% tests passed, 0 tests failed out of 130

Total Test time (real) = 204.82 sec

Running Examples

ADIOS2 is best learned by examples.

A few very basic examples are described below:

Directory

Description

ADIOS2/examples/hello

very basic “hello world”-style examples for reading and writing .bp files.

ADIOS2/examples/heatTransfer

2D Poisson solver for transients in Fourier’s modeo of heat transfer. Outputs bp.dir or HDF5.

ADIOS2/examples/basics

covers different Variable use cases classified by the dimension.

As Package

Conda

Currently ADIOS 2 can be obtained from anaconda cloud:

Spack

ADIOS 2 is fully supported in the latest Spack adios2 package

Docker

Docker images including building and installation of ADIOS 2 in a container: * Ubuntu 18.04: under scripts/docker/images/ubuntu/Dockerfile

Linking ADIOS 2

From CMake

ADIOS exports a CMake package configuration file that allows its targets to be directly imported into another CMake project via the find_package command:

cmake_minimum_required(VERSION 3.6)
project(MySimulation C CXX)

find_package(MPI REQUIRED)
find_package(ADIOS2 REQUIRED)
#...
add_library(my_library src1.cxx src2.cxx)
target_link_libraries(my_library PRIVATE adios2::cxx11_mpi MPI::MPI_C)

When configuring your project you can then set the ADIOS2_ROOT or ADIOS2_DIR environment variables to the install prefix of ADIOS2.

From non-CMake build systems

If you’re not using CMake then you can manually get the necessary compile and link flags for your project using adios2-config:

$ /path/to/install-prefix/bin/adios2-config --cxxflags
ADIOS2_DIR: /path/to/install-prefix
-isystem /path/to/install-prefix/include -isystem /opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/include -pthread -std=gnu++11
$ /path/to/install-prefix/bin/adios2-config --cxxlibs
ADIOS2_DIR: /path/to/install-prefix
-Wl,-rpath,/path/to/install-prefix/lib:/opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib /path/to/install-prefix/lib/libadios2.so.2.4.0 -pthread -Wl,-rpath -Wl,/opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib -Wl,--enable-new-dtags -pthread /opt/ohpc/pub/mpi/openmpi3-gnu7/3.1.0/lib/libmpi.so -Wl,-rpath-link,/path/to/install-prefix/lib