High-Performance Machine Learning: Combining Cloud & GPU NativeBlogEstimated5 min readByWei Ji LeongonMar 19, 2024circle information iconImage by Midjourney v5.2, AI-generated using prompt 'Rocket launching straight through the clouds of the Earth's atmosphere at a 45 degree angle with orange flames from its booster --aspect 7:4' with seed 1342788268We need to build more efficient tools faster than ever before.circle information iconImage by Wei Ji LeongFigure 1: NVIDIA GPUDirect Storage (GDS). Source: Wei Ji Leong, FOSS4G 2023 Oceania circle information iconImage by Wei Ji LeongFigure 2: xbatcher - slicing n-dimensional arrays using named variables. Illustration and code sample. Source: Wei Ji Leong, FOSS4G 2023 Oceania circle information iconImage by Wei Ji LeongFigure 3: zen3geo's chainable I/O readers and processors for geospatial data. Source: Wei Ji Leong, FOSS4G 2023 Oceania circle information iconImage by Wei Ji LeongFigure 4: The zen3geo DataPipe. Starting with a url to an ERA5 Zarr store, custom pre-processing function, sliced into subsets using xbatcher, converted from CuPy arrays into Torch tensors, and finally passed into a DataLoader. Source: Wei Ji Leong, FOSS4G 2023 Oceania TopicsCloud Computing & InfrastructureData & AI/MLPreviousVisualizing the Future of Our ForestsNextWhat's New in LonboardWhat we're doing.Latest