I/O routines

EC-specific IO routines

Use to store arrays in a file, and to load them back. Use memory-maps to store and load large arrays.

mioclosemmap(io::IO, array::AbstractArray)

Close memory-map file and flush to disk.

mioflushmmap(io::IO, array::AbstractArray)

Flush memory-map file to disk.

mioheadersize(io::IOStream) -> Int

Number of header bytes preceding the array data in an mio-format file (type code + array count + ndim + dims) — equivalently, the absolute byte offset at which the single mmapped array's data begins. Add it to a data-relative byte offset to address the data with miopread! / mioprefetch. Parses the header from the file start and restores the stream position.

mioload!(fname::String, arrs::AbstractArray{T,N}...; skip_error=false)

Load arrays from a file fname into pre-allocated arrays arrs. Return true if successful, false otherwise. If skip_error is set to true, the function will not throw an error if the type of the data/number of dimensions in the file does not match T/N.

mioload(fname::String; array_of_arrays = false)

Load arrays from a file fname.

Return an array of arrays. If there is only one array - return array itself (unless array_of_arrays is set to true).

mioload(fname::String, ::Val{N}, T::Type=Float64; skip_error=false) where {N}

Type-stable load arrays from a file fname.

Return an array of arrays. All arrays have the same type T and have N dimensions. For N = 1, return vectors even if the original array was a multi-dimensional array. If skip_error is set to true, the function will not throw an error if the type of the data/number of dimensions in the file does not match T/N and an array with one empty Array{T,N} will be returned.

miommap(fname::String)

Memory-map an existing file for reading. Return a pointer to the file and the mmaped array.

mionewmmap(fname::String, dims::Tuple{Vararg{Int}}, Type=Float64)

Create a new memory-map file for writing (overwrites existing file). Return a pointer to the file and the mmaped array.

miopread!(io::IOStream, ptr::Ptr, nbytes::Int, offset::Int)

Positional read: copy exactly nbytes bytes from io at absolute byte offset into ptr. offset is measured from the file start — add mioheadersize to address the array data. ptr must point to at least nbytes of live memory (root it with GC.@preserve). Uses pread(2) on Unix (positional, does not disturb the stream position); on other platforms falls back to seek+unsafe_read, which does move the stream position (fine for single-threaded use). Errors if fewer than nbytes bytes are available.

mioprefetch(io::IOStream, offset::Int, nbytes::Int)

Best-effort readahead hint: advise the kernel that [offset, offset+nbytes) of io (absolute bytes; add mioheadersize) will be read soon, letting it queue the I/O asynchronously ahead of a burst of miopread!s. Uses posix_fadvise(POSIX_FADV_WILLNEED) on Linux and is a silent no-op elsewhere — it is purely advisory (never required for correctness) and only Linux exposes the syscall portably (macOS would need fcntl(F_RDADVISE), most BSDs vary).

miosave(fname::String,arrs::AbstractArray{T}...) where T

Save arrays arrs in a file fname.