Skip to content

Encoded chunk

zarrista.EncodedChunk

Encoded chunk bytes, together with the codec chain that decodes them.

This class splits a chunk read into two steps. The first step reads the encoded bytes, which is IO-bound. The second step decodes them, which is CPU-bound. You can then run each step where it is best: read the bytes with await, and decode them on a thread pool.

Array.retrieve_encoded_chunk returns an EncodedChunk. The object holds the data type, the fill value, the shape, and the codec chain, so you do not track which codec chain belongs to which bytes.

The object holds no reference to the array or to the store. It is therefore safe to send to another thread.

Examples:

Read a chunk, then decode it off the main thread:

chunk = array.retrieve_encoded_chunk([0, 0])
if chunk is not None:
    decoded = await chunk.decode_async()

buffer property 

buffer: Buffer

The raw, still-encoded chunk bytes.

codecs property 

codecs: CodecChain

The codec chain that decodes the bytes.

data_type property 

data_type: DataType

The Zarr data type of the decoded chunk.

fill_value property 

fill_value: FillValue

The fill value of the decoded chunk.

shape property 

shape: list[int]

The shape of the decoded chunk, in elements along each dimension.

decode 

decode(**codec_options: Unpack[CodecOptions]) -> Tensor

Decode the chunk bytes on the calling thread.

The method releases the GIL while it decodes. Other Python threads can therefore run at the same time.

Parameters:

Returns:

  • Tensor –

    The decoded chunk data.

Raises:

  • TypeError –

    If a keyword argument is not a known codec option.

decode_async async 

decode_async(
    *, pool: ThreadPool | None = None, **codec_options: Unpack[CodecOptions]
) -> Tensor

Decode the chunk bytes on a Rust thread pool.

The method does the work on a thread pool and does not hold the GIL. Use it to decode many chunks at the same time.

Every decode uses the full CPU by default, because concurrent_target defaults to the number of threads in the pool. To decode N chunks at the same time, pass concurrent_target=max(1, cores // N).

Parameters:

  • pool (ThreadPool | None, default: None ) –

    The thread pool that runs the decode. Defaults to the global Rust thread pool, which zarrista also uses for its other parallel work. Pass a ThreadPool to use a separate pool.

  • **codec_options (Unpack[CodecOptions], default: {} ) –

    The codec options, as CodecOptions.

Returns:

  • Tensor –

    The decoded chunk data.

Raises:

  • TypeError –

    If a keyword argument is not a known codec option.