Struct ParquetLogger 

pub struct ParquetLogger<T>
where
    for<'a> &'a [T]: RecordWriter<T>,
{ /* private fields */ }

Write log records to a Parquet data file.

Use ParquetLogger to open a parquet file and write one log record at a time. ParquetLogger buffers up to maximum_buffer_size log records in memory and then synchronizes the buffer to the file.

Derive ParquetRecordWriter on your log record struct, create a ParquetLogger, and log records to the file.

Example

use hoomd_utility::data::ParquetLogger;
use parquet_derive::ParquetRecordWriter;

#[derive(ParquetRecordWriter)]
pub struct LogRecord {
    /// The simulation step.
    step: u64,

    /// Total system potential energy.
    potential_energy: f64,
}

// let path = "log.parquet";
let mut parquet_logger = ParquetLogger::<LogRecord>::create(path)?;
parquet_logger.log(LogRecord {
    step: 0,
    potential_energy: 1.0,
})?;
parquet_logger.log(LogRecord {
    step: 1,
    potential_energy: 2.0,
})?;

Implementations

impl<T> ParquetLogger<T>

where for<'a> &'a [T]: RecordWriter<T>,

pub fn create<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Create a new parquet file.

create overwrites any existing file at path. The default maximum_buffer_size is $2^{17}$ records.

Errors

Returns Error when there is I/O error opening the file or parquet fails to initialize the file.

Example

use hoomd_utility::data::ParquetLogger;
use parquet_derive::ParquetRecordWriter;

#[derive(ParquetRecordWriter)]
pub struct LogRecord {
    /// The simulation step.
    step: u64,

    /// Total system potential energy.
    potential_energy: f64,
}

// let path = "log.parquet";
let mut parquet_logger = ParquetLogger::<LogRecord>::create(path)?;

pub fn create_unique<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Create a unique parquet file.

Parquet files cannot be appended to once written. The accepted solution by the Parquet developers is to create many files (file.parquet.0, file.parquet.1, and so on) that you concatenate on read.

create_unique facilitates this process by appending .0, .1, … .{N}. to the given path. create_unique attempts all extensions in order, continuing to the next increment each time it finds an existing file.

The default maximum_buffer_size is $2^{17}$ records.

Errors

Returns Error when there is I/O error opening the file or parquet fails to initialize the file.

Example

use hoomd_utility::data::ParquetLogger;
use parquet_derive::ParquetRecordWriter;

#[derive(ParquetRecordWriter)]
pub struct LogRecord {
    /// The simulation step.
    step: u64,

    /// Total system potential energy.
    potential_energy: f64,
}

// let path = "log.parquet";
let mut parquet_logger = ParquetLogger::<LogRecord>::create_unique(path)?;

pub fn log(&mut self, record: T) -> Result<(), Error>

Log a record to the file.

log buffers up to maximum_buffer_size records in memory before synchronizing them to the file. Call log once for each record you produce during a simulation.

Errors

Returns Error when there is I/O error writing to the file or parquet is unable to write the buffer.

Example

use hoomd_utility::data::ParquetLogger;
use parquet_derive::ParquetRecordWriter;

#[derive(ParquetRecordWriter)]
pub struct LogRecord {
    /// The simulation step.
    step: u64,

    /// Total system potential energy.
    potential_energy: f64,
}

// let path = "log.parquet";
let mut parquet_logger = ParquetLogger::<LogRecord>::create(path)?;
parquet_logger.log(LogRecord {
    step: 0,
    potential_energy: 1.0,
})?;
parquet_logger.log(LogRecord {
    step: 1,
    potential_energy: 2.0,
})?;

pub fn maximum_buffer_size(&self) -> usize

Maximum number of log records to buffer in memory.

pub fn maximum_buffer_size_mut(&mut self) -> &mut usize

Mutable reference to the maximum number of log records to buffer in memory.

pub fn sync(&mut self) -> Result<(), Error>

Synchronize the buffer to the file.

sync writes all currently buffered log records to a row group. A single parquet file can contain at most $2^{15}$ row groups, so call sync only at specific points when data must be on disk.

All buffered data is automatically synchronized when the logger is dropped.

Errors

Returns Error when there is I/O error writing to the file or parquet is unable to write the buffer.

Trait Implementations

impl<T> Drop for ParquetLogger<T>

where for<'a> &'a [T]: RecordWriter<T>,

Synchronize the buffer and close the parquet file.

fn drop(&mut self)

Executes the destructor for this type.