Struct ErasureCoding 

pub struct ErasureCoding;

Erasure coding abstraction.

Supports creation of parity shards and recovery of missing data.

Erasure coding processes each 64-byte block of shards independently from other blocks, so large shards are split into narrower columns that are processed one by one, which keeps the working set in CPU cache and is much faster than processing whole shards at once. With parallel feature enabled, par_* methods additionally process columns concurrently.

Implementations

impl ErasureCoding

pub fn new() -> Self

Create new erasure coding instance

pub fn extend<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &[[u8; SHARD_BYTES]; NUM_SHARDS], parity: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

Extend contiguously stored source shards with parity shards

pub fn par_extend<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &[[u8; SHARD_BYTES]; NUM_SHARDS], parity: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

Parallel version of Self::extend().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::extend() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::extend() does.

pub fn extend_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], parity: [ParityShard; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

where SourceShard: AsRef<[u8; SHARD_BYTES]>, ParityShard: AsMut<[u8; SHARD_BYTES]>,

Extend source shards with parity shards, where shards are not stored contiguously

pub fn par_extend_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], parity: [ParityShard; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

where SourceShard: AsRef<[u8; SHARD_BYTES]>, ParityShard: AsMut<[u8; SHARD_BYTES]>,

Parallel version of Self::extend_scattered().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::extend_scattered() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::extend_scattered() does.

pub fn recover_source<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], parity: &[[u8; SHARD_BYTES]; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

Recover missing source shards in place, with everything stored contiguously.

Parity shards are inputs only, missing ones are simply not used. Prefer this over Self::recover_all() when parity shards are not needed.

pub fn par_recover_source<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], parity: &[[u8; SHARD_BYTES]; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

Parallel version of Self::recover_source().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::recover_source() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::recover_source() does.

pub fn recover_source_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], source_present: &ShardsBitmap<NUM_SHARDS>, parity: [Option<ParityShard>; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

where SourceShard: AsMut<[u8; SHARD_BYTES]>, ParityShard: AsRef<[u8; SHARD_BYTES]>,

Recover missing source shards in place, where shards are not stored contiguously.

Parity shards are inputs only, so missing ones simply have no memory.

pub fn par_recover_source_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], source_present: &ShardsBitmap<NUM_SHARDS>, parity: [Option<ParityShard>; NUM_SHARDS], ) -> Result<(), ErasureCodingError>

where SourceShard: AsMut<[u8; SHARD_BYTES]>, ParityShard: AsRef<[u8; SHARD_BYTES]>,

Parallel version of Self::recover_source_scattered().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::recover_source_scattered() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::recover_source_scattered() does.

pub fn recover_all<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], parity: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

Recover missing source and parity shards in place, with everything stored contiguously.

Use Self::recover_source() instead if parity shards are not needed.

pub fn par_recover_all<const NUM_SHARDS: usize, const SHARD_BYTES: usize>( &self, source: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], parity: &mut [[u8; SHARD_BYTES]; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

Parallel version of Self::recover_all().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::recover_all() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::recover_all() does.

pub fn recover_all_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], parity: [ParityShard; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

where SourceShard: AsMut<[u8; SHARD_BYTES]>, ParityShard: AsMut<[u8; SHARD_BYTES]>,

Recover missing source and parity shards in place, where shards are not stored contiguously

pub fn par_recover_all_scattered<const NUM_SHARDS: usize, const SHARD_BYTES: usize, SourceShard, ParityShard>( &self, source: [SourceShard; NUM_SHARDS], parity: [ParityShard; NUM_SHARDS], present: &ShardsPresent<NUM_SHARDS>, ) -> Result<(), ErasureCodingError>

where SourceShard: AsMut<[u8; SHARD_BYTES]>, ParityShard: AsMut<[u8; SHARD_BYTES]>,

Parallel version of Self::recover_all_scattered().

Columns of large shards are processed using up to 4 threads, which is beneficial when erasure coding is the only thing running. Prefer Self::recover_all_scattered() when calling from code that is already parallel to avoid contention.

Small shards (like many record chunks) are not split into columns and are processed by a single thread, the same way as Self::recover_all_scattered() does.

Trait Implementations

impl Clone for ErasureCoding

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ErasureCoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ErasureCoding

fn default() -> Self

Returns the “default value” for a type.