We need to build more efficient tools faster than ever before.
Figure 1: NVIDIA GPUDirect Storage (GDS). Source: Wei Ji Leong, FOSS4G 2023 Oceania
Figure 2: xbatcher - slicing n-dimensional arrays using named variables. Illustration and code sample. Source: Wei Ji Leong, FOSS4G 2023 Oceania
Figure 3: zen3geo's chainable I/O readers and processors for geospatial data. Source: Wei Ji Leong, FOSS4G 2023 Oceania
Figure 4: The zen3geo DataPipe. Starting with a url to an ERA5 Zarr store, custom pre-processing function, sliced into subsets using xbatcher, converted from CuPy arrays into Torch tensors, and finally passed into a DataLoader. Source: Wei Ji Leong, FOSS4G 2023 Oceania
What we're doing.
Latest