ADIOS 2: The Adaptable Input/Output System version 2

Funded by the Exascale Computing Project (ECP), U.S. Department of Energy

Introduction

• What’s new in 2.11?
  • Summary
    • Derived Variables
    • Remote Data Access Infrastructure
    • GPU and Memory Space Improvements
    • Campaign Management System Overhaul
    • Compression and Data Format Advances
    • OpenSSF Best Practices
• What’s new in 2.10?
• What’s new in 2.9?
  • Summary
    • General
    • File I/O
    • Staging
    • Experimental features
• Introduction
  • What ADIOS2 is and isn’t
  • Adaptable IO beyond files in Scientific Data Lifecycles

Setting Up

• Install from Source
  • Building, Testing, and Installing ADIOS 2
  • CMake Options
  • Building on HPC Systems
  • Installing the ADIOS2 library and the C++ and C bindings
  • Enabling the Python bindings
  • Enabling the Fortran bindings
  • Running Tests
  • Running Examples
• As Package
  • Conda
  • PyPI
  • Spack
  • Docker
• Linking ADIOS 2
  • From CMake
  • From non-CMake build systems
• Use on DOE machines
  • NERSC Perlmutter
  • OLCF Frontier
  • ALCF Aurora

Basics

• Interface Components
  • Components Overview
  • ADIOS
  • IO
    • Setting a Particular Engine and its Parameters
    • Adding Supported Transports with Parameters
    • Defining, Inquiring and Removing Variables and Attributes
    • Opening an Engine
  • Variable
    • Data Types
    • Shapes
    • Global Array Capabilities and Limitations
  • Attribute
  • Engine
    • BeginStep
    • EndStep
    • Close
    • Put: modes and memory contracts
    • PerformPuts
    • PerformDataWrite
    • Get: modes and memory contracts
    • PerformGets
    • Engine usage example
    • Available Engines
  • Operator
  • Runtime Configuration Files
    • XML
    • YAML
  • Anatomy of an ADIOS Program
    • Anatomy of an ADIOS Output
    • Anatomy of an ADIOS Input
    • Anatomy of an ADIOS File-only Input
• Supported Virtual Engine Names
  • Virtual Engine Setups
• Supported Engines
  • BP5
  • BP4
  • BP3
  • HDF5
  • SST Sustainable Staging Transport
  • SSC Strong Staging Coupler
  • DataMan for Wide Area Network Data Staging
  • DataSpaces
  • Inline for zero-copy
  • Null
  • Plugin Engine
• Supported Operators
  • CompressorZFP
    • CompressorZFP Specific parameters
    • CompressorZFP Execution Policy
  • Plugin Operator
  • Encryption
• Full APIs
  • C++11 bindings
    • ADIOS2 components classes
    • ADIOS class
      • adios2::ADIOS
        • ADIOS()
        • ADIOS()
        • ADIOS()
        • ADIOS()
        • ADIOS()
        • ADIOS()
        • operator bool()
        • ADIOS()
        • ADIOS()
        • ~ADIOS()
        • operator=()
        • operator=()
        • DeclareIO()
        • AtIO()
        • DefineOperator()
        • DefineOperator()
        • InquireOperator()
        • FlushAll()
        • RemoveIO()
        • RemoveAllIOs()
        • EnterComputationBlock()
        • ExitComputationBlock()
    • IO class
      • adios2::IO
        • IO()
        • ~IO()
        • operator bool()
        • Name()
        • InConfigFile()
        • SetEngine()
        • SetParameter()
        • SetParameters()
        • SetParameters()
        • ClearParameters()
        • Parameters()
        • AddTransport()
        • SetTransportParameter()
        • DefineVariable()
        • InquireVariable()
        • InquireVariableType()
        • DefineAttribute()
        • DefineAttribute()
        • InquireAttribute()
        • RemoveVariable()
        • RemoveAllVariables()
        • RemoveAttribute()
        • RemoveAllAttributes()
        • Open()
        • InquireGroup()
        • Open()
        • Open()
        • FlushAll()
        • AvailableVariables()
        • AvailableAttributes()
        • VariableType()
        • AttributeType()
        • AddOperation()
        • EngineType()
    • Variable <T> class
      • adios2::Variable
        • Variable()
        • ~Variable()
        • operator bool()
        • StoreStatsOnly()
        • SetMemorySpace()
        • GetMemorySpace()
        • SetShape()
        • SetBlockSelection()
        • SetSelection()
        • SetMemorySelection()
        • SetStepSelection()
        • SetAccuracy()
        • SelectionSize()
        • Name()
        • Type()
        • Sizeof()
        • ShapeID()
        • Shape()
        • Start()
        • Count()
        • Steps()
        • StepsStart()
        • BlockID()
        • AddOperation()
        • Operations()
        • RemoveOperations()
        • MinMax()
        • Min()
        • Max()
        • GetAccuracy()
        • AllStepsBlocksInfo()
        • adios2::Variable::Info
          • Data()
          • Start
          • Count
          • Min
          • Max
          • Value
          • WriterID
          • BlockID
          • Step
          • IsReverseDims
          • IsValue
    • Attribute <T> class
      • adios2::Attribute
        • Attribute()
        • operator bool()
        • Name()
        • Type()
        • Data()
        • IsValue()
    • Engine class
      • adios2::Engine
        • Engine()
        • ~Engine()
        • operator bool()
        • Name()
        • Type()
        • OpenMode()
        • GetMetadata()
        • BeginStep()
        • BeginStep()
        • CurrentStep()
        • Put()
        • Put()
        • Put()
        • Put()
        • Put()
        • Put()
        • Put()
        • PerformPuts()
        • PerformDataWrite()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • Get()
        • PerformGets()
        • EndStep()
        • BetweenStepPairs()
        • Flush()
        • Close()
        • AllStepsBlocksInfo()
        • BlocksInfo()
        • GetAbsoluteSteps()
        • Steps()
        • LockWriterDefinitions()
        • LockReaderSelections()
    • Operator class
      • adios2::Operator
        • Operator()
        • operator bool()
        • Type()
        • SetParameter()
        • Parameters()
      • adios2::Group
        • AvailableGroups()
        • AvailableVariables()
        • AvailableAttributes()
        • InquirePath()
        • setPath()
        • InquireGroup()
        • InquireVariable()
        • InquireAttribute()
        • VariableType()
        • AttributeType()
  • Fortran bindings
    • ADIOS2 typed handlers
    • ADIOS subroutines
    • IO subroutines
    • Variable subroutines
    • Engine subroutines
    • Operator subroutines
    • Attribute subroutines
  • C bindings
    • adios2_adios handler functions
      • adios2_init
      • adios2_init_config
      • adios2_init_mpi()
      • adios2_init_config_mpi()
      • adios2_init_serial()
      • adios2_init_config_serial()
      • adios2_declare_io()
      • adios2_declare_io_order()
      • adios2_at_io()
      • adios2_define_operator()
      • adios2_inquire_operator()
      • adios2_flush_all()
      • adios2_finalize()
      • adios2_remove_io()
      • adios2_remove_all_ios()
      • adios2_enter_computation_block()
      • adios2_exit_computation_block()
    • adios2_io handler functions
      • adios2_in_config_file()
      • adios2_set_engine()
      • adios2_set_parameters()
      • adios2_set_parameter()
      • adios2_get_parameter()
      • adios2_clear_parameters()
      • adios2_add_transport()
      • adios2_set_transport_parameter()
      • adios2_define_variable()
      • adios2_inquire_variable()
      • adios2_inquire_all_variables()
      • adios2_inquire_group_variables()
      • adios2_define_attribute()
      • adios2_define_attribute_array()
      • adios2_define_variable_attribute()
      • adios2_define_variable_attribute_array()
      • adios2_inquire_attribute()
      • adios2_inquire_variable_attribute()
      • adios2_inquire_all_attributes()
      • adios2_inquire_group_attributes()
      • adios2_inquire_subgroups()
      • adios2_remove_variable()
      • adios2_remove_all_variables()
      • adios2_available_variables()
      • adios2_available_attributes()
      • adios2_remove_attribute()
      • adios2_remove_all_attributes()
      • adios2_open()
      • adios2_open_with_metadata()
      • adios2_open_new_comm()
      • adios2_flush_all_engines()
      • adios2_engine_type()
      • adios2_get_engine()
    • adios2_variable handler functions
      • adios2_set_shape()
      • adios2_store_stats_only()
      • adios2_set_memory_space()
      • adios2_get_memory_space()
      • adios2_set_block_selection()
      • adios2_set_selection()
      • adios2_set_memory_selection()
      • adios2_set_step_selection()
      • adios2_variable_name()
      • adios2_variable_type()
      • adios2_variable_type_string()
      • adios2_variable_shapeid()
      • adios2_variable_ndims()
      • adios2_variable_shape()
      • adios2_variable_shape_with_memory_space()
      • adios2_variable_start()
      • adios2_variable_count()
      • adios2_variable_steps_start()
      • adios2_variable_steps()
      • adios2_selection_size()
      • adios2_add_operation()
      • adios2_set_operation_parameter()
      • adios2_remove_operations()
      • adios2_variable_min()
      • adios2_variable_max()
    • adios2_attribute handler functions
      • adios2_attribute_name()
      • adios2_attribute_type()
      • adios2_attribute_type_string()
      • adios2_attribute_is_value()
      • adios2_attribute_size()
      • adios2_attribute_data()
    • adios2_engine handler functions
      • adios2_engine_name()
      • adios2_engine_get_type()
      • adios2_engine_openmode()
      • adios2_engine_get_metadata()
      • adios2_begin_step()
      • adios2_current_step()
      • adios2_between_step_pairs()
      • adios2_steps()
      • adios2_put()
      • adios2_put_by_name()
      • adios2_perform_puts()
      • adios2_perform_data_write()
      • adios2_get()
      • adios2_get_by_name()
      • adios2_perform_gets()
      • adios2_end_step()
      • adios2_flush()
      • adios2_flush_by_index()
      • adios2_close()
      • adios2_close_by_index()
      • adios2_lock_writer_definitions()
      • adios2_lock_reader_selections()
      • adios2_inquire_blockinfo()
      • adios2_free_blockinfo()
    • adios2_operator handler functions
      • adios2_operator_type()
• High-Level APIs
  • C++ High-Level API
    • C++11 Write example
    • C++11 Read “step-by-step” example
    • adios2::fstream API documentation
      • adios2::fstream
        • openmode
          • out
          • in
          • in_random_access
          • app
        • fstream()
        • fstream()
        • fstream()
        • fstream()
        • fstream()
        • ~fstream()
        • operator bool()
        • open()
        • open()
        • open()
        • open()
        • set_parameter()
        • set_parameters()
        • write_attribute()
        • write_attribute()
        • write()
        • write()
        • write()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read()
        • read_attribute()
        • end_step()
        • close()
        • current_step()
        • getstep
  • Matlab simple bindings
    • Organization of an ADIOS BP file
    • Time dimension of a variable
    • Min/max of arrays
    • ADIOSOPEN
    • ADIOSREAD
    • ADIOSCLOSE
• Python APIs
  • Python Example Code
    • Examples in the ADIOS2 repository
    • Python Write example
    • Python Read “step-by-step” example
    • Python Read Random Access example
  • adios2 classes for Python
    • Stream
      • Stream.adios
      • Stream.all_blocks_info()
      • Stream.available_attributes()
      • Stream.available_variables()
      • Stream.begin_step()
      • Stream.close()
      • Stream.current_step()
      • Stream.define_variable()
      • Stream.end_step()
      • Stream.engine
      • Stream.get_metadata()
      • Stream.inquire_attribute()
      • Stream.inquire_variable()
      • Stream.io
      • Stream.loop_index()
      • Stream.mode
      • Stream.num_steps()
      • Stream.read()
      • Stream.read_attribute()
      • Stream.read_attribute_string()
      • Stream.read_complete()
      • Stream.read_in_buffer()
      • Stream.set_parameters()
      • Stream.step_status()
      • Stream.steps()
      • Stream.write()
      • Stream.write_attribute()
    • FileReader
    • Adios
      • Adios.at_io()
      • Adios.declare_io()
      • Adios.define_operator()
      • Adios.flush_all()
      • Adios.impl
      • Adios.inquire_operator()
      • Adios.remove_all_ios()
      • Adios.remove_io()
    • IO
      • IO.add_transport()
      • IO.adios()
      • IO.available_attributes()
      • IO.available_variables()
      • IO.define_attribute()
      • IO.define_derived_variable()
      • IO.define_variable()
      • IO.engine_type()
      • IO.flush_all()
      • IO.impl
      • IO.inquire_attribute()
      • IO.inquire_variable()
      • IO.open()
      • IO.open_with_metadata()
      • IO.parameters()
      • IO.remove_all_attributes()
      • IO.remove_all_variables()
      • IO.remove_attribute()
      • IO.remove_variable()
      • IO.set_engine()
      • IO.set_parameter()
      • IO.set_parameters()
    • Engine
      • Engine.all_blocks_info()
      • Engine.begin_step()
      • Engine.between_step_pairs()
      • Engine.blocks_info()
      • Engine.close()
      • Engine.current_step()
      • Engine.end_step()
      • Engine.flush()
      • Engine.get()
      • Engine.get_metadata()
      • Engine.impl
      • Engine.lock_reader_selections()
      • Engine.lock_writer_definitions()
      • Engine.perform_data_write()
      • Engine.perform_gets()
      • Engine.perform_puts()
      • Engine.put()
      • Engine.steps()
    • Variable
      • Variable.add_operation()
      • Variable.add_operation_string()
      • Variable.block_id()
      • Variable.count()
      • Variable.get_accuracy()
      • Variable.impl
      • Variable.name()
      • Variable.operations()
      • Variable.remove_operations()
      • Variable.selection_size()
      • Variable.set_accuracy()
      • Variable.set_block_selection()
      • Variable.set_selection()
      • Variable.set_shape()
      • Variable.set_step_selection()
      • Variable.shape()
      • Variable.shape_id()
      • Variable.single_value()
      • Variable.sizeof()
      • Variable.start()
      • Variable.steps()
      • Variable.steps_start()
      • Variable.store_stats_only()
      • Variable.type()
    • Attribute
      • Attribute.data()
      • Attribute.data_string()
      • Attribute.impl
      • Attribute.name()
      • Attribute.single_value()
      • Attribute.type()
    • Operator
      • Operator.get_parameters()
      • Operator.impl
      • Operator.set_parameter()
  • Python bindings to C++
    • Examples using the Python bindings in the ADIOS2 repository
    • ADIOS class
      • ADIOS
        • ADIOS.AtIO()
        • ADIOS.DeclareIO()
        • ADIOS.DefineOperator()
        • ADIOS.FlushAll()
        • ADIOS.InquireOperator()
        • ADIOS.RemoveAllIOs()
        • ADIOS.RemoveIO()
    • IO class
      • IO
        • IO.AddTransport()
        • IO.AvailableAttributes()
        • IO.AvailableVariables()
        • IO.DefineAttribute()
        • IO.DefineDerivedVariable()
        • IO.DefineVariable()
        • IO.EngineType()
        • IO.FlushAll()
        • IO.InquireAttribute()
        • IO.InquireVariable()
        • IO.Open()
        • IO.Parameters()
        • IO.RemoveAllAttributes()
        • IO.RemoveAllVariables()
        • IO.RemoveAttribute()
        • IO.RemoveVariable()
        • IO.SetEngine()
        • IO.SetParameter()
        • IO.SetParameters()
    • Variable class
      • Variable
        • Variable.AddOperation()
        • Variable.BlockID()
        • Variable.Count()
        • Variable.GetAccuracy()
        • Variable.GetAccuracyRequested()
        • Variable.Name()
        • Variable.Operations()
        • Variable.RemoveOperations()
        • Variable.SelectionSize()
        • Variable.SetAccuracy()
        • Variable.SetBlockSelection()
        • Variable.SetSelection()
        • Variable.SetShape()
        • Variable.SetStepSelection()
        • Variable.Shape()
        • Variable.ShapeID()
        • Variable.SingleValue()
        • Variable.Sizeof()
        • Variable.Start()
        • Variable.Steps()
        • Variable.StepsStart()
        • Variable.StoreStatsOnly()
        • Variable.Type()
    • Attribute class
      • Attribute
        • Attribute.Data()
        • Attribute.DataString()
        • Attribute.Name()
        • Attribute.SingleValue()
        • Attribute.Type()
    • Engine class
      • Engine
        • Engine.BeginStep()
        • Engine.BetweenStepPairs()
        • Engine.BlocksInfo()
        • Engine.Close()
        • Engine.CurrentStep()
        • Engine.EndStep()
        • Engine.Flush()
        • Engine.Get()
        • Engine.GetMetadata()
        • Engine.LockReaderSelections()
        • Engine.LockWriterDefinitions()
        • Engine.Name()
        • Engine.PerformDataWrite()
        • Engine.PerformGets()
        • Engine.PerformPuts()
        • Engine.Put()
        • Engine.Steps()
        • Engine.Type()
    • Operator class
      • Operator
        • Operator.Parameters()
        • Operator.SetParameter()
        • Operator.Type()
    • Query class
      • Query
        • Query.GetBlockIDs()
        • Query.GetResult()
  • Transition from old API to new API

Advanced Topics

• Aggregation
  • Aggregation in BP5
• Memory Management
  • BP4 buffering
  • BP5 buffering
  • Span object in internal buffer
• GPU-aware I/O
  • Building ADIOS2 with a GPU backend
    • Building with CUDA enabled
    • Building with Kokkos enabled
  • Writing GPU buffers
    • Example using a Device buffer
    • Kokkos applications
  • Reading GPU buffers
  • Build scripts
• ADIOS2 query API
  • The interface
    • A Sample Compound Query
  • Code EXAMPLES:
    • C++:
    • Python:
• Plugins
  • Writing Your Plugin Class
    • Engine Plugin
    • Operator Plugin
  • Build Shared Library
  • Using Your Plugin in an Application
    • Engine Plugins
    • Operator Plugins
• Campaign Management
  • Requirements
  • Limitations
    • Example
  • Remote access
• ADIOS2 in ECP hardware
  • OLCF CRUSHER
    • SST MPI Data Transport
• Derived variables
  • Build ADIOS2 with support for derived variables
  • Supported derived operations
• Perfromance profiling
  • JSON Performance File
  • External Profiling Libraries
  • Real-time Performance Monioring

Tutorials

• Overview
• Download And Build
• Basic Tutorials
  • Hello World
  • Variables
  • Attributes
  • Operators
  • Steps
  • SST Engine Example:
    • 0. Notes:
    • 1. WAN (default)
      • 2. RDMA
      • 3. MPI
      • 4. UCX

Ecosystem Tools

• HDF5 API Support through VOL
  • Disclaimer
  • External
  • Internal
• Command Line Utilities
  • bpls : Inspecting Data
  • adios_reorganize
  • adios2-config
  • sst_conn_tool : SST network connectivity tool
• Visualizing Data
  • Using VTK and Paraview
    • Saving the vtk.xml data model

Additional Information

• FAQ
  • MPI vs Non-MPI
  • APIs
  • Building on Titan
  • Building and Running on Fujitsu FX100
  • FAQs Answered
    • Can I use the same library for MPI and non-MPI code?
    • Can I use ADIOS 2 C++11 library with C++98 codes?
    • Why are C and Fortran APIs missing functionality?
    • C++11: Why are std::string arguments passed sometimes by value and sometimes by reference?
    • C++11: Should I pass adios2:: objects by value or by reference?
    • Fortran: Can I pass slices and temporary arrays to adios2_put?
    • My application uses PGI compilers on Titan, can I link ADIOS 2?
    • How do I enable the Python bindings on Titan?
    • How do I build ADIOS 2 on Fujitsu FX100?
    • SST engine hangs on Fujitsu FX100. Why?
• Advice

• Search Page