Visualizing Data

Note

As of version 2.6.0 this work is experimental. As it builds on top of the library in third party ecosystems, and doesn’t ship with ADIOS 2 source code.

Certain ADIOS 2 bp files, and in-memory streams in the future), can be recognized by third party products to enable data visualization. The expectation is to keep adding support to the list of products enabling ADIOS 2 for visualization purposes. The goal of this section is to describe the currently cover cases and how to create an ADIOS 2 BP file to accomodate the visualization product requirements.

Using VTK and Paraview

ADIOS BP files can now be seamlessly integrated into the Visualization Toolkit (VTK) and Paraview. Datasets can be described with an additional attribute that conforms to the VTK XML data model formats as high-level descriptors that will allow interpretation of ADIOS 2 variables into a hierarchy understood by the VTK infrastructure. This XML data format is saved in ADIOS as either an attribute, and optionally, as an additional vtk.xml file inside file.bp.dir/vtk.xml from the default engine. For more details, see BP3 (Default).

Current limitations:

  • Only works with MPI builds of VTK and Paraview

  • Support only one Block per ADIOS dataset

  • Only supports BP Files, streams are not supported

  • Currently working up to 3D (and linearized 1D) variables for scalars and vectors.

  • Image Data, vti, is supported with ADIOS2 Global Array Variables only

  • Unstructured Grid, vtu, is supported with ADIOS2 Local Arrays Variables only

Two “VTK File” types are supported:

  1. Image data (.vti)

  2. Unstructured Grid (.vtu)

The main idea is to populate the above XML format contents describing the extent and the data arrays with ADIOS variables in the BP data set. The result is a more-than-compact typical VTK data file since extra information about the variable, such as dimensions and type, as they already exist in the BP data set.

A typical VTK image data XML descriptor (.vti):

<?xml version="1.0"?>
<VTKFile type="ImageData">
  <ImageData WholeExtent="x1 x2 y1 y2 z1 z2" Origin="x0 y0 z0" Spacing="dx dy dz">
    <Piece Extent="x1 x2 y1 y2 z1 z2">
      <PointData>
        <DataArray Name="p1"/>
        <DataArray Name="p2"/>
      </PointData>
      <CellData>
        <DataArray Name="c1"/>
        <DataArray Name="c2"/>
      </CellData>
    </Piece>
  </ImageData>
</VTKFile>

A typical VTK unstructured grid XML descriptor (.vtu):

<?xml version="1.0"?>
<VTKFile type="ImageData">
  <ImageData WholeExtent="x1 x2 y1 y2 z1 z2" Origin="x0 y0 z0" Spacing="dx dy dz">
    <Piece Extent="x1 x2 y1 y2 z1 z2">
      <PointData>
        <DataArray Name="p1"/>
        <DataArray Name="p2"/>
      </PointData>
      <CellData>
        <DataArray Name="c1"/>
        <DataArray Name="c2"/>
      </CellData>
    </Piece>
  </ImageData>
</VTKFile>

In addition, VTK can interpret physical-time or output-step varying data stored with ADIOS by resusing the special “TIME” tag. This is better illustrated in the following section.

Saving the vtk.xml data model

For the full source code of the following illustration example see the gray-scott adios2 tutorial

To incorporate the data model in a BP data file, the application has two options:

  1. Adding a string attribute called “vtk.xml” in code. “TIME” is a special tag for adding physical time variables.

const std::string imageData = R"(
     <?xml version="1.0"?>
     <VTKFile type="ImageData" version="0.1" byte_order="LittleEndian">
       <ImageData WholeExtent=")" + extent + R"(" Origin="0 0 0" Spacing="1 1 1">
         <Piece Extent=")" + extent + R"(">
           <CellData Scalars="U">
               <DataArray Name="U" />
               <DataArray Name="V" />
               <DataArray Name="TIME">
                 step
               </DataArray>
           </CellData>
         </Piece>
       </ImageData>
     </VTKFile>)";

io.DefineAttribute<std::string>("vtk.xml", imageData);

Tip

C++11 users should take advantage C++11 string literals (R”( xml_here )”) to simplify escaping characters in the XML.

The resulting bpls output should contain the “vtk.xml” attribute and the variables in the model:

> bpls gs.bp -lav
File info:
  of variables:  3
  of attributes: 7
  statistics:    Min / Max

  double   Du       attr   = 0.2
  double   Dv       attr   = 0.1
  double   F        attr   = 0.02
  double   U        24*{48, 48, 48} = 0.107439 / 1.04324
  double   V        24*{48, 48, 48} = 0 / 0.672232
  double   dt       attr   = 1
  double   k        attr   = 0.048
  double   noise    attr   = 0.01
  int32_t  step     24*scalar = 0 / 575
  string   vtk.xml  attr   =
<VTKFile type="ImageData" version="0.1" byte_order="LittleEndian">
  <ImageData WholeExtent="0 49 0 49 0 49" Origin="0 0 0" Spacing="1 1 1">
    <Piece Extent="0 49 0 49 0 49">
      <CellData Scalars="U">
        <DataArray Name="U" />
        <DataArray Name="V" />
        <DataArray Name="TIME">
          step
        </DataArray>
      </CellData>
    </Piece>
  </ImageData>
</VTKFile>
  1. Saving a “vtk.xml” file inside the file.bp.dir to describe the data after is created

> cat gs.bp.dir/vtk.xml

  <?xml version="1.0"?>
  <VTKFile type="ImageData" version="0.1" byte_order="LittleEndian">
    <ImageData WholeExtent=")" + extent + R"(" Origin="0 0 0" Spacing="1 1 1">
      <Piece Extent=")" + extent + R"(">
        <CellData Scalars="U">
          <DataArray Name="U" />
          <DataArray Name="V" />
          <DataArray Name="TIME">
            step
          </DataArray>
        </CellData>
      </Piece>
    </ImageData>
  </VTKFile>

The result is that the generated BP file should be recognize by a branch of Paraview/VTK that must be built from source:

https://i.imgur.com/ap3l9Z5.png:alt:my-picture2

Similarly, unstructured grid (.vtu) support can be added with the limitations of using specific labels for the variable names setting the “connectivity”, “vertices”, and cell “types”.

The following example is taken from example 2 of the MFEM product examples website using ADIOS 2:

The resulting bpls output for unstructured grid data types:

File info:
  of variables:  6
  of attributes: 4
  statistics:    Min / Max

  uint32_t  NumOfElements       {4} = 1024 / 1024
  uint32_t  NumOfVertices       {4} = 1377 / 1377
  string    app                 attr   = "MFEM"
  uint64_t  connectivity        [4]*{1024, 9} = 0 / 1376
  uint32_t  dimension           attr   = 3
  string    glvis_mesh_version  attr   = "1.0"
  double    sol                 [4]*{1377, 3} = -0.201717 / 1.19304
  uint32_t  types               scalar = 11
  double    vertices            [4]*{1377, 3} = -1.19304 / 8.20172
  string    vtk.xml             attr   =
<VTKFile type="UnstructuredGrid" version="0.1" byte_order="LittleEndian">
  <UnstructuredGrid>
    <Piece NumberOfPoints="NumOfVertices" NumberOfCells="NumOfElements">
      <Points>
        <DataArray Name="vertices" />
      </Points>
      <Cells>
        <DataArray Name="connectivity" />
        <DataArray Name="types" />
      </Cells>
      <PointData>
        <DataArray Name="sol" />
      </PointData>
    </Piece>
  </UnstructuredGrid>
</VTKFile>

and resulting visualization in Paraview for different “cell” types:

https://i.imgur.com/ke1xiNh.png:alt:my-picture3

Build VTK and Paraview with ADIOS 2 Support

Note

Currently the implementation for ADIOS 2 readers exist in VTK and Paraview nightly releases and master branches. We expect this to be part of the VTK and Paraview release cycle with their upcoming releases. Users must build from source and point to these branches until formal merge into their master branches is done.

Paraview and its VTK dependency must be built with the following CMake options:

  1. -DVTK_MODULE_ENABLE_VTK_IOADIOS2=YES

  2. -DVTK_USE_MPI=ON

  3. -DPARAVIEW_USE_MPI=ON

For comprehensive build instructions see the documentation for VTK (VTK) and Paraview.