Variables

In the previous tutorial we learned how to define a simple string variable, write it, and read it back.

In this tutorial we will go two steps further:

1. We will define variables which include arrays, and we will write them and read them back.

2. We will use MPI to write and read the above variables in parallel.

Let’s start with the writing part.

Start editing the skeleton file ADIOS2/examples/hello/bpWriter/bpWriter_tutorialSkeleton.cpp.

1. In an MPI application first we need to always initialize MPI. We do that with the following lines:

int rank, size;
int provided;

// MPI_THREAD_MULTIPLE is only required if you enable the SST MPI_DP
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

2. Now we need to create some application variables which will be used to define ADIOS2 variables.

// Application variable
std::vector<float> myFloats = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
std::vector<int> myInts = {0, -1, -2, -3, -4, -5, -6, -7, -8, -9};
const std::size_t Nx = myFloats.size();
const std::string myString("Hello Variable String from rank " + std::to_string(rank));

3. Now we need to define an ADIOS2 instance and the ADIOS2 variables.

adios2::ADIOS adios(MPI_COMM_WORLD);
adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");

adios2::Variable<float> bpFloats = bpIO.DefineVariable<float>(
    "bpFloats", {size * Nx}, {rank * Nx}, {Nx}, adios2::ConstantDims);

adios2::Variable<int> bpInts = bpIO.DefineVariable<int>("bpInts", {size * Nx}, {rank * Nx},
                                                        {Nx}, adios2::ConstantDims);

// For the sake of the tutorial we create an unused variable
adios2::Variable<std::string> bpString = bpIO.DefineVariable<std::string>("bpString");

Note

The above int/float variables are global arrays. The 1st argument of the DefineVariable function is the variable name, the 2nd are the global dimensions, the 3rd is the start index for a rank, the 4th are the rank/local dimensions, and the 5th is a boolean variable to indicate if the dimensions are constant or not over multiple steps, where adios2::ConstantDims == true We will explore other tutorials that don’t use constant dimensions.

4. Now we need to open the ADIOS2 engine and write the variables.

adios2::Engine bpWriter = bpIO.Open("myVector_cpp.bp", adios2::Mode::Write);

bpWriter.BeginStep();
bpWriter.Put(bpFloats, myFloats.data());
bpWriter.Put(bpInts, myInts.data());
// bpWriter.Put(bpString, myString);
bpWriter.EndStep();

bpWriter.Close();

5. Finally we need to finalize MPI.

MPI_Finalize();

6. The final code should look as follows (excluding try/catch and the optional usage of MPI), and it was derived from the example ADIOS2/examples/hello/bpWriter/bpWriter.cpp.

/*
 * Distributed under the OSI-approved Apache License, Version 2.0.  See
 * accompanying file Copyright.txt for details.
 *
 * bpWriter.cpp: Simple self-descriptive example of how to write a variable
 * to a BP File that lives in several MPI processes.
 *
 *  Created on: Feb 16, 2017
 *      Author: William F Godoy godoywf@ornl.gov
 */

#include <ios>       //std::ios_base::failure
#include <iostream>  //std::cout
#include <stdexcept> //std::invalid_argument std::exception
#include <vector>

#include <adios2.h>
#if ADIOS2_USE_MPI
#include <mpi.h>
#endif

int main(int argc, char *argv[])
{
    int rank, size;
#if ADIOS2_USE_MPI
    int provided;

    // MPI_THREAD_MULTIPLE is only required if you enable the SST MPI_DP
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
#else
    rank = 0;
    size = 1;
#endif

    /** Application variable */
    std::vector<float> myFloats = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
    std::vector<int> myInts = {0, -1, -2, -3, -4, -5, -6, -7, -8, -9};
    const std::size_t Nx = myFloats.size();

    const std::string myString("Hello Variable String from rank " + std::to_string(rank));

    try
    {
        /** ADIOS class factory of IO class objects */
#if ADIOS2_USE_MPI
        adios2::ADIOS adios(MPI_COMM_WORLD);
#else
        adios2::ADIOS adios;
#endif

        /*** IO class object: settings and factory of Settings: Variables,
         * Parameters, Transports, and Execution: Engines */
        adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");

        /** global array : name, { shape (total) }, { start (local) }, {
         * count
         * (local) }, all are constant dimensions */
        adios2::Variable<float> bpFloats = bpIO.DefineVariable<float>(
            "bpFloats", {size * Nx}, {rank * Nx}, {Nx}, adios2::ConstantDims);

        adios2::Variable<int> bpInts = bpIO.DefineVariable<int>("bpInts", {size * Nx}, {rank * Nx},
                                                                {Nx}, adios2::ConstantDims);

        adios2::Variable<std::string> bpString = bpIO.DefineVariable<std::string>("bpString");
        (void)bpString; // For the sake of the example we create an unused
                        // variable

        std::string filename = "myVector_cpp.bp";
        /** Engine derived class, spawned to start IO operations */
        adios2::Engine bpWriter = bpIO.Open(filename, adios2::Mode::Write);

        bpWriter.BeginStep();
        /** Put variables for buffering, template type is optional */
        bpWriter.Put(bpFloats, myFloats.data());
        bpWriter.Put(bpInts, myInts.data());
        // bpWriter.Put(bpString, myString);
        bpWriter.EndStep();

        /** Create bp file, engine becomes unreachable after this*/
        bpWriter.Close();
        if (rank == 0)
        {
            std::cout << "Wrote file " << filename
                      << " to disk. It can now be read by running "
                         "./bin/adios2_hello_bpReader.\n";
        }
    }
    catch (std::invalid_argument &e)
    {
        std::cerr << "Invalid argument exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::ios_base::failure &e)
    {
        std::cerr << "IO System base failure exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::exception &e)
    {
        std::cerr << "Exception: " << e.what() << "\n";
#if ADIOS2_USE_MPI
        std::cerr << "STOPPING PROGRAM from rank " << rank << "\n";
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }

#if ADIOS2_USE_MPI
    MPI_Finalize();
#endif

    return 0;
}

7. You can compile and run it as follows:

cd Path-To-ADIOS2/examples/hello/bpWriter
mkdir build
cd build
cmake -DADIOS2_DIR=Path-To-ADIOS2/build/ ..
cmake --build .
mpirun -np 2 ./adios2_hello_bpWriter_mpi

8. You can check the content of the output file “myVector_cpp.bp” using bpls as follows:

Path-To-ADIOS2/build/bin/bpls ./myVector_cpp.bp

  float    bpFloats  {10}
  int32_t  bpInts    {10}

Now let’s move to the reading part.

Start editing the skeleton file ADIOS2/examples/hello/bpReader/bpReader_tutorialSkeleton.cpp.

9. In an MPI application first we need to always initialize MPI. We do that with the following line:

int rank, size;
int provided;

// MPI_THREAD_MULTIPLE is only required if you enable the SST MPI_DP
MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

10. Now we need to define an ADIOS2 instance and open the ADIOS2 engine.

adios2::ADIOS adios(MPI_COMM_WORLD);

adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");

adios2::Engine bpReader = bpIO.Open("myVector_cpp.bp", adios2::Mode::Read);

11. Now we need to read the variables. In this case we know the variables that we need to inquire, so we can use the InquireVariable function immediately. But let’s explore how to check the available variables in a file first, and then we will use the InquireVariable function.

bpReader.BeginStep();
const std::map<std::string, adios2::Params> variables = bpIO.AvailableVariables();

for (const auto &variablePair : variables)
{
    std::cout << "Name: " << variablePair.first;
    for (const auto &parameter : variablePair.second)
    {
        std::cout << "\t" << parameter.first << ": " << parameter.second << "\n";
    }
}

adios2::Variable<float> bpFloats = bpIO.InquireVariable<float>("bpFloats");
adios2::Variable<int> bpInts = bpIO.InquireVariable<int>("bpInts");

12. Now we need to read the variables from each rank. We will use the SetSelection to set the start index and rank dimensions, then Get function to read the variables, and print the contents from rank 0.

const std::size_t Nx = 10;
if (bpFloats) // means found
{
   std::vector<float> myFloats;

   // read only the chunk corresponding to our rank
   bpFloats.SetSelection({{Nx * rank}, {Nx}});
   bpReader.Get(bpFloats, myFloats, adios2::Mode::Sync);

   if (rank == 0)
   {
       std::cout << "MyFloats: \n";
       for (const auto number : myFloats)
       {
           std::cout << number << " ";
       }
       std::cout << "\n";
   }
}

if (bpInts) // means not found
{
   std::vector<int> myInts;
   // read only the chunk corresponding to our rank
   bpInts.SetSelection({{Nx * rank}, {Nx}});

   bpReader.Get(bpInts, myInts, adios2::Mode::Sync);

   if (rank == 0)
   {
       std::cout << "myInts: \n";
       for (const auto number : myInts)
       {
           std::cout << number << " ";
       }
       std::cout << "\n";
   }
}

Note

While using the Get function, we used the third parameter named Mode. The mode parameter can also be used for the Put function.

For the Put function, there are three modes: Deferred (default), Sync, and Span. and for the Get there are two modes: Deferred (default) and Sync.

1. The Deferred mode is the default mode, because it is the fastest mode, as it allows Put / Get to be grouped before potential data transport at the first encounter of PerformPuts / PerformGets, EndStep or Close.

2. The Sync mode forces Put / Get to be performed immediately so that the data are available immediately.

3. The Span mode is special mode of Deferred that allows population from non-contiguous memory structures.

For more information about the Mode parameter for both Put and Get functions, and when you should use each option see Basics: Interface Components: Engine.

13. Now we need close the ADIOS2 engine.

bpReader.EndStep();
bpReader.Close();

14. Finally we need to finalize MPI.

MPI_Finalize();

15. The final code should look as follows (excluding try/catch), and it was derived from the example ADIOS2/examples/hello/bpWriter/bpWriter.cpp.

/*
 * Distributed under the OSI-approved Apache License, Version 2.0.  See
 * accompanying file Copyright.txt for details.
 *
 * bpReader.cpp: Simple self-descriptive example of how to read a variable
 * from a BP File.
 *
 *  Created on: Feb 16, 2017
 *      Author: William F Godoy godoywf@ornl.gov
 */
#include <ios>      //std::ios_base::failure
#include <iostream> //std::cout
#if ADIOS2_USE_MPI
#include <mpi.h>
#endif
#include <stdexcept> //std::invalid_argument std::exception
#include <vector>

#include <adios2.h>

int main(int argc, char *argv[])
{
    int rank, size;

#if ADIOS2_USE_MPI
    int provided;
    // MPI_THREAD_MULTIPLE is only required if you enable the SST MPI_DP
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
#else
    rank = 0;
    size = 1;
#endif
    std::cout << "rank " << rank << " size " << size << "\n";
    try
    {
#if ADIOS2_USE_MPI
        /** ADIOS class factory of IO class objects */
        adios2::ADIOS adios(MPI_COMM_WORLD);
#else
        adios2::ADIOS adios;
#endif

        /*** IO class object: settings and factory of Settings: Variables,
         * Parameters, Transports, and Execution: Engines */
        adios2::IO bpIO = adios.DeclareIO("BPFile_N2N");

        /** Engine derived class, spawned to start IO operations */
        adios2::Engine bpReader = bpIO.Open("myVector_cpp.bp", adios2::Mode::Read);

        bpReader.BeginStep();
        const std::map<std::string, adios2::Params> variables = bpIO.AvailableVariables();

        for (const auto &variablePair : variables)
        {
            std::cout << "Name: " << variablePair.first;

            for (const auto &parameter : variablePair.second)
            {
                std::cout << "\t" << parameter.first << ": " << parameter.second << "\n";
            }
        }

        /** Write variable for buffering */
        adios2::Variable<float> bpFloats = bpIO.InquireVariable<float>("bpFloats");
        adios2::Variable<int> bpInts = bpIO.InquireVariable<int>("bpInts");

        const std::size_t Nx = 10;
        if (bpFloats) // means found
        {
            std::vector<float> myFloats;

            // read only the chunk corresponding to our rank
            bpFloats.SetSelection({{Nx * rank}, {Nx}});
            // myFloats.data is pre-allocated
            bpReader.Get(bpFloats, myFloats, adios2::Mode::Sync);

            if (rank == 0)
            {
                std::cout << "MyFloats: \n";
                for (const auto number : myFloats)
                {
                    std::cout << number << " ";
                }
                std::cout << "\n";
            }
        }

        if (bpInts) // means not found
        {
            std::vector<int> myInts;
            // read only the chunk corresponding to our rank
            bpInts.SetSelection({{Nx * rank}, {Nx}});

            bpReader.Get(bpInts, myInts, adios2::Mode::Sync);

            if (rank == 0)
            {
                std::cout << "myInts: \n";
                for (const auto number : myInts)
                {
                    std::cout << number << " ";
                }
                std::cout << "\n";
            }
        }
        bpReader.EndStep();

        /** Close bp file, engine becomes unreachable after this*/
        bpReader.Close();
    }
    catch (std::invalid_argument &e)
    {
        if (rank == 0)
        {
            std::cerr << "Invalid argument exception, STOPPING PROGRAM from rank " << rank << "\n";
            std::cerr << e.what() << "\n";
        }
#if ADIOS2_USE_MPI
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::ios_base::failure &e)
    {
        if (rank == 0)
        {
            std::cerr << "IO System base failure exception, STOPPING PROGRAM "
                         "from rank "
                      << rank << "\n";
            std::cerr << e.what() << "\n";
            std::cerr << "The file myVector_cpp.bp does not exist."
                      << " Presumably this is because adios2_hello_bpWriter has not "
                         "been run."
                      << " Run ./adios2_hello_bpWriter before running this program.\n";
        }
#if ADIOS2_USE_MPI
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }
    catch (std::exception &e)
    {
        if (rank == 0)
        {
            std::cerr << "Exception, STOPPING PROGRAM from rank " << rank << "\n";
            std::cerr << e.what() << "\n";
        }
#if ADIOS2_USE_MPI
        MPI_Abort(MPI_COMM_WORLD, 1);
#endif
    }

#if ADIOS2_USE_MPI
    MPI_Finalize();
#endif

    return 0;
}

16. You can compile and run it as follows:

cd Path-To-ADIOS2/examples/hello/bpReader
mkdir build
cd build
cmake -DADIOS2_DIR=Path-To-ADIOS2/build/ ..
cmake --build .
mpirun -np 2 ./adios2_hello_bpReader_mpi