ADIOS 2: The Adaptable Input/Output System version 2

Funded by the Exascale Computing Project (ECP), U.S. Department of Energy

Introduction

• What’s new in 2.10?
  • Python
  • New/updated features
  • Packaging
• What’s new in 2.9?
  • General
  • File I/O
  • Staging
  • Experimental features
• Introduction
  • What ADIOS2 is and isn’t
  • Adaptable IO beyond files in Scientific Data Lifecycles

Setting Up

• Install from Source
  • Building, Testing, and Installing ADIOS 2
  • CMake Options
  • Building on HPC Systems
  • Installing the ADIOS2 library and the C++ and C bindings
  • Enabling the Python bindings
  • Enabling the Fortran bindings
  • Running Tests
  • Running Examples
• As Package
  • Conda
  • PyPI
  • Spack
  • Docker
• Linking ADIOS 2
  • From CMake
  • From non-CMake build systems
• Use on DOE machines
  • NERSC Perlmutter
  • OLCF Frontier
  • ALCF Aurora

Basics

• Interface Components
  • Components Overview
  • ADIOS
  • IO
    • Setting a Particular Engine and its Parameters
    • Adding Supported Transports with Parameters
    • Defining, Inquiring and Removing Variables and Attributes
    • Opening an Engine
  • Variable
    • Data Types
    • Shapes
    • Global Array Capabilities and Limitations
  • Attribute
  • Engine
    • BeginStep
    • EndStep
    • Close
    • Put: modes and memory contracts
    • PerformPuts
    • PerformDataWrite
    • Get: modes and memory contracts
    • PerformGets
    • Engine usage example
    • Available Engines
  • Operator
  • Runtime Configuration Files
    • XML
    • YAML
  • Anatomy of an ADIOS Program
    • Anatomy of an ADIOS Output
    • Anatomy of an ADIOS Input
    • Anatomy of an ADIOS File-only Input
• Supported Virtual Engine Names
  • Virtual Engine Setups
• Supported Engines
  • BP5
  • BP4
  • BP3
  • HDF5
  • SST Sustainable Staging Transport
  • SSC Strong Staging Coupler
  • DataMan for Wide Area Network Data Staging
  • DataSpaces
  • Inline for zero-copy
  • Null
  • Plugin Engine
• Supported Operators
  • CompressorZFP
    • CompressorZFP Specific parameters
    • CompressorZFP Execution Policy
  • Plugin Operator
  • Encryption
• Full APIs
  • C++11 bindings
    • ADIOS2 components classes
    • ADIOS class
    • IO class
    • Variable <T> class
    • Attribute <T> class
    • Engine class
    • Operator class
  • Fortran bindings
    • ADIOS2 typed handlers
    • ADIOS subroutines
    • IO subroutines
    • Variable subroutines
    • Engine subroutines
    • Operator subroutines
    • Attribute subroutines
  • C bindings
    • adios2_adios handler functions
    • adios2_io handler functions
    • adios2_variable handler functions
    • adios2_attribute handler functions
    • adios2_engine handler functions
    • adios2_operator handler functions
• High-Level APIs
  • C++ High-Level API
    • C++11 Write example
    • C++11 Read “step-by-step” example
    • adios2::fstream API documentation
  • Matlab simple bindings
    • Organization of an ADIOS BP file
    • Time dimension of a variable
    • Min/max of arrays
    • ADIOSOPEN
    • ADIOSREAD
    • ADIOSCLOSE
• Python APIs
  • Python Example Code
    • Examples in the ADIOS2 repository
    • Python Write example
    • Python Read “step-by-step” example
    • Python Read Random Access example
  • adios2 classes for Python
    • Stream
      • Stream.adios
      • Stream.all_blocks_info()
      • Stream.available_attributes()
      • Stream.available_variables()
      • Stream.begin_step()
      • Stream.close()
      • Stream.current_step()
      • Stream.define_variable()
      • Stream.end_step()
      • Stream.engine
      • Stream.inquire_attribute()
      • Stream.inquire_variable()
      • Stream.io
      • Stream.loop_index()
      • Stream.mode
      • Stream.num_steps()
      • Stream.read()
      • Stream.read_attribute()
      • Stream.read_attribute_string()
      • Stream.set_parameters()
      • Stream.step_status()
      • Stream.steps()
      • Stream.write()
      • Stream.write_attribute()
    • FileReader
    • Adios
      • Adios.at_io()
      • Adios.declare_io()
      • Adios.define_operator()
      • Adios.flush_all()
      • Adios.impl
      • Adios.inquire_operator()
      • Adios.remove_all_ios()
      • Adios.remove_io()
    • IO
      • IO.add_transport()
      • IO.adios()
      • IO.available_attributes()
      • IO.available_variables()
      • IO.define_attribute()
      • IO.define_variable()
      • IO.engine_type()
      • IO.flush_all()
      • IO.impl
      • IO.inquire_attribute()
      • IO.inquire_variable()
      • IO.open()
      • IO.parameters()
      • IO.remove_all_attributes()
      • IO.remove_all_variables()
      • IO.remove_attribute()
      • IO.remove_variable()
      • IO.set_engine()
      • IO.set_parameter()
      • IO.set_parameters()
    • Engine
      • Engine.all_blocks_info()
      • Engine.begin_step()
      • Engine.between_step_pairs()
      • Engine.blocks_info()
      • Engine.close()
      • Engine.current_step()
      • Engine.end_step()
      • Engine.flush()
      • Engine.get()
      • Engine.impl
      • Engine.lock_reader_selections()
      • Engine.lock_writer_definitions()
      • Engine.perform_data_write()
      • Engine.perform_gets()
      • Engine.perform_puts()
      • Engine.put()
      • Engine.steps()
    • Variable
      • Variable.add_operation()
      • Variable.add_operation_string()
      • Variable.block_id()
      • Variable.count()
      • Variable.impl
      • Variable.name()
      • Variable.operations()
      • Variable.remove_operations()
      • Variable.selection_size()
      • Variable.set_block_selection()
      • Variable.set_selection()
      • Variable.set_shape()
      • Variable.set_step_selection()
      • Variable.shape()
      • Variable.shape_id()
      • Variable.single_value()
      • Variable.sizeof()
      • Variable.start()
      • Variable.steps()
      • Variable.steps_start()
      • Variable.type()
    • Attribute
      • Attribute.data()
      • Attribute.data_string()
      • Attribute.impl
      • Attribute.name()
      • Attribute.single_value()
      • Attribute.type()
    • Operator
      • Operator.get_parameters()
      • Operator.impl
      • Operator.set_parameter()
  • Python bindings to C++
    • Examples using the Python bindings in the ADIOS2 repository
    • ADIOS class
      • ADIOS
        • ADIOS.AtIO()
        • ADIOS.DeclareIO()
        • ADIOS.DefineOperator()
        • ADIOS.FlushAll()
        • ADIOS.InquireOperator()
        • ADIOS.RemoveAllIOs()
        • ADIOS.RemoveIO()
    • IO class
      • IO
        • IO.AddTransport()
        • IO.AvailableAttributes()
        • IO.AvailableVariables()
        • IO.DefineAttribute()
        • IO.DefineVariable()
        • IO.EngineType()
        • IO.FlushAll()
        • IO.InquireAttribute()
        • IO.InquireVariable()
        • IO.Open()
        • IO.Parameters()
        • IO.RemoveAllAttributes()
        • IO.RemoveAllVariables()
        • IO.RemoveAttribute()
        • IO.RemoveVariable()
        • IO.SetEngine()
        • IO.SetParameter()
        • IO.SetParameters()
    • Variable class
      • Variable
        • Variable.AddOperation()
        • Variable.BlockID()
        • Variable.Count()
        • Variable.Name()
        • Variable.Operations()
        • Variable.RemoveOperations()
        • Variable.SelectionSize()
        • Variable.SetBlockSelection()
        • Variable.SetSelection()
        • Variable.SetShape()
        • Variable.SetStepSelection()
        • Variable.Shape()
        • Variable.ShapeID()
        • Variable.SingleValue()
        • Variable.Sizeof()
        • Variable.Start()
        • Variable.Steps()
        • Variable.StepsStart()
        • Variable.Type()
    • Attribute class
      • Attribute
        • Attribute.Data()
        • Attribute.DataString()
        • Attribute.Name()
        • Attribute.SingleValue()
        • Attribute.Type()
    • Engine class
      • Engine
        • Engine.BeginStep()
        • Engine.BetweenStepPairs()
        • Engine.BlocksInfo()
        • Engine.Close()
        • Engine.CurrentStep()
        • Engine.EndStep()
        • Engine.Flush()
        • Engine.Get()
        • Engine.LockReaderSelections()
        • Engine.LockWriterDefinitions()
        • Engine.Name()
        • Engine.PerformDataWrite()
        • Engine.PerformGets()
        • Engine.PerformPuts()
        • Engine.Put()
        • Engine.Steps()
        • Engine.Type()
    • Operator class
      • Operator
        • Operator.Parameters()
        • Operator.SetParameter()
        • Operator.Type()
    • Query class
      • Query
        • Query.GetBlockIDs()
        • Query.GetResult()
  • Transition from old API to new API

Advanced Topics

• Aggregation
  • Aggregation in BP5
• Memory Management
  • BP4 buffering
  • BP5 buffering
  • Span object in internal buffer
• GPU-aware I/O
  • Building ADIOS2 with a GPU backend
    • Building with CUDA enabled
    • Building with Kokkos enabled
  • Writing GPU buffers
    • Example using a Device buffer
    • Kokkos applications
  • Reading GPU buffers
  • Build scripts
• ADIOS2 query API
  • The interface
    • A Sample Compound Query
  • Code EXAMPLES:
    • C++:
    • Python:
• Plugins
  • Writing Your Plugin Class
    • Engine Plugin
    • Operator Plugin
  • Build Shared Library
  • Using Your Plugin in an Application
    • Engine Plugins
    • Operator Plugins
• Campaign Management
  • The idea
    • Example
  • Setup
  • Remote access
  • Requirements
  • Limitations
• ADIOS2 in ECP hardware
  • OLCF CRUSHER
    • SST MPI Data Transport

Tutorials

• Overview
• Download And Build
• Basic Tutorials
  • Hello World
  • Variables
  • Attributes
  • Operators
  • Steps

Ecosystem Tools

• HDF5 API Support through VOL
  • Disclaimer
  • External
  • Internal
• Command Line Utilities
  • bpls : Inspecting Data
  • adios_reorganize
  • adios2-config
  • sst_conn_tool : SST network connectivity tool
• Visualizing Data
  • Using VTK and Paraview
    • Saving the vtk.xml data model

Additional Information

• FAQ
  • MPI vs Non-MPI
  • APIs
  • Building on Titan
  • Building and Running on Fujitsu FX100
  • FAQs Answered
    • Can I use the same library for MPI and non-MPI code?
    • Can I use ADIOS 2 C++11 library with C++98 codes?
    • Why are C and Fortran APIs missing functionality?
    • C++11: Why are std::string arguments passed sometimes by value and sometimes by reference?
    • C++11: Should I pass adios2:: objects by value or by reference?
    • Fortran: Can I pass slices and temporary arrays to adios2_put?
    • My application uses PGI compilers on Titan, can I link ADIOS 2?
    • How do I enable the Python bindings on Titan?
    • How do I build ADIOS 2 on Fujitsu FX100?
    • SST engine hangs on Fujitsu FX100. Why?
• Advice

• Search Page