In this notebook, we create Zarr stores for the CMIP6 TAS daily data available in NetCDF on S3. This method of creating Zarr stores uses pangeo-forge and it’s recipes pattern.
The test datasets produced are:
Chunked to optimize for time series analysis:
latitude: 252, longitude: 252, time: 365.
This dataset has larger chunks, but more timesteps are loaded into each chunk.
Chunked to optimize for visualization at a single time step.
latitude: 600, longitude: 1440, time: 1.
This dataset has small chunks, but will likely not work well for time series generation.
Chunked to optimize for both time series and visualization:
latitude: 600, longitude: 1440, time: 29.
This dataset has larger chunks, but more timesteps are loaded into each chunk.
1.1 Install and import libraries
%%capture!pip install loguru
import fsspecimport s3fsimport xarray as xrimport sys; sys.path.append('..')from helpers.profiler import Timerimport helpers.eodc_hub_role as eodc_hub_role
NASA Earth Exchange, NASA Ames Research Center, Moffett Field, CA 94035
variant_label :
r1i1p1f2
product :
output
realm :
atmos
source :
BCSD
scenario :
historical
references :
BCSD method: Thrasher et al., 2012, Hydrol. Earth Syst. Sci.,16, 3309-3314. Ref period obs: latest version of the Princeton Global Meteorological Forcings (http://hydrology.princeton.edu/data.php), based on Sheffield et al., 2006, J. Climate, 19 (13), 3088-3111.
version :
1.0
tracking_id :
25d6baa3-0404-4eba-a3f1-afddbf69d4cc
title :
GISS-E2-1-G, r1i1p1f2, historical, global downscaled CMIP6 climate projection data
resolution_id :
0.25 degree
history :
2021-10-04T18:41:40.796912+00:00: install global attributes
disclaimer :
This data is considered provisional and subject to change. This data is provided as is without any warranty of any kind, either express or implied, arising by law or otherwise, including but not limited to warranties of completeness, non-infringement, accuracy, merchantability, or fitness for a particular purpose. The user assumes all risk associated with the use of, or inability to use, this data.
This data is considered provisional and subject to change. This data is provided as is without any warranty of any kind, either express or implied, arising by law or otherwise, including but not limited to warranties of completeness, non-infringement, accuracy, merchantability, or fitness for a particular purpose. The user assumes all risk associated with the use of, or inability to use, this data.
downscalingModel :
BCSD
external_variables :
areacella
frequency :
day
history :
2021-10-04T18:41:40.796912+00:00: install global attributes
institution :
NASA Earth Exchange, NASA Ames Research Center, Moffett Field, CA 94035
product :
output
realm :
atmos
references :
BCSD method: Thrasher et al., 2012, Hydrol. Earth Syst. Sci.,16, 3309-3314. Ref period obs: latest version of the Princeton Global Meteorological Forcings (http://hydrology.princeton.edu/data.php), based on Sheffield et al., 2006, J. Climate, 19 (13), 3088-3111.
resolution_id :
0.25 degree
scenario :
historical
source :
BCSD
title :
GISS-E2-1-G, r1i1p1f2, historical, global downscaled CMIP6 climate projection data
This data is considered provisional and subject to change. This data is provided as is without any warranty of any kind, either express or implied, arising by law or otherwise, including but not limited to warranties of completeness, non-infringement, accuracy, merchantability, or fitness for a particular purpose. The user assumes all risk associated with the use of, or inability to use, this data.
downscalingModel :
BCSD
external_variables :
areacella
frequency :
day
history :
2021-10-04T18:41:40.796912+00:00: install global attributes
institution :
NASA Earth Exchange, NASA Ames Research Center, Moffett Field, CA 94035
product :
output
realm :
atmos
references :
BCSD method: Thrasher et al., 2012, Hydrol. Earth Syst. Sci.,16, 3309-3314. Ref period obs: latest version of the Princeton Global Meteorological Forcings (http://hydrology.princeton.edu/data.php), based on Sheffield et al., 2006, J. Climate, 19 (13), 3088-3111.
resolution_id :
0.25 degree
scenario :
historical
source :
BCSD
title :
GISS-E2-1-G, r1i1p1f2, historical, global downscaled CMIP6 climate projection data
This data is considered provisional and subject to change. This data is provided as is without any warranty of any kind, either express or implied, arising by law or otherwise, including but not limited to warranties of completeness, non-infringement, accuracy, merchantability, or fitness for a particular purpose. The user assumes all risk associated with the use of, or inability to use, this data.
downscalingModel :
BCSD
external_variables :
areacella
frequency :
day
history :
2021-10-04T18:41:40.796912+00:00: install global attributes
institution :
NASA Earth Exchange, NASA Ames Research Center, Moffett Field, CA 94035
product :
output
realm :
atmos
references :
BCSD method: Thrasher et al., 2012, Hydrol. Earth Syst. Sci.,16, 3309-3314. Ref period obs: latest version of the Princeton Global Meteorological Forcings (http://hydrology.princeton.edu/data.php), based on Sheffield et al., 2006, J. Climate, 19 (13), 3088-3111.
resolution_id :
0.25 degree
scenario :
historical
source :
BCSD
title :
GISS-E2-1-G, r1i1p1f2, historical, global downscaled CMIP6 climate projection data
tracking_id :
25d6baa3-0404-4eba-a3f1-afddbf69d4cc
variant_label :
r1i1p1f2
version :
1.0
# Write output to json fileimport jsondatasets = {}for chunk_set in chunk_sets: chunk_prefix =str(("_").join(map(str, chunk_set.values()))) dataset_id =f"{chunk_prefix}_CMIP6_daily_{model}_{variable}.zarr" dataset_url =f"s3://{bucket}/{zarr_directory}/{dataset_id}" datasets[dataset_id] = {"dataset_url": dataset_url,"variable": variable }withopen("cmip6-zarr-datasets.json", "w") as f: f.write(json.dumps(datasets)) f.close()