amep.utils.optimal_chunksize#

amep.utils.optimal_chunksize(data_length: int, number_of_elements: int, bit_per_element: int = 32, buffer: float = 0.25, maxmem: float = 1) → int#

Estimates an optimal chunksize for limiting the RAM usage.

Notes

The memory estimate in GB for the computation with one chunk can be written chunksize*number_of_elements*bit_per_element/8/1e9+buffer. Setting it equal to maxmem leads to the estimate of chunksize.

Parameters:

data_length (int) – Length of the data to be chunked.
number_of_elements (int) – Number of (array) elements to be stored for each chunk.
bit_per_element (int, optional) – Number of bits per element, e.g., 32 for float32 numbers or 64 for float64 numbers. The default is 32.
buffer (float, optional) – Additional buffer to avoid filling up the RAM. The default is 0.25.
maxmem (float, optional) – Maxmimum RAM usage per CPU core. The default is MAXMEM.

Returns:

Chunksize.

Return type:

int