Time series API limits¶

Last updated: 2025-01-02

The titiler-cmr API can be deployed as a Lambda function in AWS. Since requests to the time series endpoints will make recursive requests to the Lambda function for the lower-level time step operations, there are some limits in place to avoid making large requests that are likely to overwhelm the API.

Terminology¶

Grid cells refer to the smallest spatial units in a gridded dataset, representing the area over which a single data value is aggregated or measured. These cells collectively form a regular grid that defines the spatial structure of the data.

Highlights¶

• Maximum of 995 discrete points or intervals in a time series request (due to AWS Lambda concurrency limits)
• You can use the length of the time series, the AOI size, and the resolution of the dataset to calculate the number of total grid cells (x_dimension * y_dimension * n_time) which is helpful for determining if a request will succeed
• The /timeseries/bbox endpoint for generating GIFs for a bounding box will struggle on requests for a large AOI and/or a lengthy time series for high spatial resolution datasets. Based on a coarse evaluation of the API, requests are limited to 100,000,000 (1e8) total grid cells. There is a limit in place that will cause requests that exceed this limit to fail fast without firing hundreds of doomed Lambda invocations.
• The /timeseries/statistics endpoint can handle larger requests than the /timeseries/bbox endpoint Based on a coarse evaluation of the API, requests are limited to 15,000,000,000 (1.5e10) total grid cells as long as the individual images read by the /statistics endpoint are smaller than 56,250,000 (5.625e7) grid cells.

Tips¶

• If you hit an error because the total size of the request is too large, try reducing the temporal resolution of the time series, e.g. from daily (P1D) to weekly (P7D) or greater (P10D)
• If you need higher temporal resolution but the full request is not able handle it, split the request into multiple smaller requests and merge the results yourself!

Background¶

The time series API provides rapid access to time series analysis and visualization of collections in the CMR catalog, but there are some limitations to the API deployment that require some care when making large requests.

There are several factors that must be considered in order to make a successful time series request:

• Spatial resolution of the dataset (especially for the xarray backend)
• Request AOI size
• Number of points/intervals in the time series

These factors all influence the runtime and memory footprint of the initial /timeseries request and requests that are too large in any of those dimensions can result in an API failure.

Details¶

Number of points/intervals in the time series¶

The top factor that determines if a request will succeed or fail is the number of points in the time series. In the default deployment, there is a hard cap of 995 time points in any time series request. This cap is in place because there is a concurrency limit of 1000 on the Lambda function that executes the API requests.

Spatial resolution and AOI size¶

For datasets that use the rasterio backend, there will be very few limitations on maximum array size as long as the data are COGs and you specify a reasonable output image size (or use the max_size parameter) in your request.

For datasets without overviews/pyramids, titiler-cmr will need to read all of the bytes that overlap the request AOI even if the resulting image is going to be downsampled for a GIF. Therefore, if the area of interest for a /timeseries/statistics or /timeseries/bbox request will create a large array that is likely to exceed the capacity of the Lambda function, the request will fail fast.

The limits for the xarray backend are:

• /timeseries/bbox
  • individual image size: 5.6e7 grid cells (~7500x7500)
  • total image size (x_dimension * y_dimension * n_time): 1e8 grid cells
• /timeseries/statistics
  • individual image size: 5.6e7 grid cells (~7500x7500)
  • total image size: 1.5e10 grid cells

For low-resolution datasets (e.g. 28km or 0.25 degree) you will not run into any issues (unless you request too many time points!) because a request for the full dataset will be reading arrays that are ~1440x720 grid cells.

For higher-resolution datasets (e.g. 1km or 0.01 degree), you will start to run into problems as the size of the raw arrays that titiler-cmr is processing increases (and the number of discrete points or intervals increases).

Examples¶

The MUR SST dataset is good for demonstrating the limits of the time series endpoints with the xarray backend. MUR SST delivers high resolution (1 km, 0.01 degree) daily global sea surface temperature observations! With this resolution it is easy to craft a request that will break the /timeseries endpoints. Here are some examples of how to manipulate the time series parameters to achieve success with the /timeseries/bbox endpoint.

from datetime import datetime, timedelta

import httpx

Here is a request that will succeed (if the lambda is warmed up):

• 5x5 degree bounding box (500 x 500 pixels)
• 180 daily observations (180 / P1D)
• total size: 500 * 500 * 180 = 4.5e7

bounds = (-5, -5, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=180)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P1D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

That request is about half of the maximum request size for the /timeseries/bbox endpoint. We can push it to the limit by doubling the length of the time series:

• 5x5 degree bounding box (500 x 500 grid cells)
• 360 daily observations (360 / P1D)
• total size: 500 * 500 * 360 = 9.0e7

bounds = (-5, -5, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=360)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P1D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

If we increase the length of the time series such that the request exceeds the maximum size, the API will return an error:

• 5x5 degree bounding box (500 x 500 grid cells)
• 540 daily observations (540 / P1D)
• total size: 500 * 500 * 540 = 1.35e8 (greater than maximum of 1.0e8!)

bounds = (-5, -5, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=540)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P1D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

We can get get a successful response for the larger time window if we reduce the temporal resolution:

• 5x5 degree bounding box (500 x 500 grid cells)
• 77 weekly observations (540 / P7D)
• total size: 500 * 500 * 77 = 1.925e7

bounds = (-5, -5, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=540)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P7D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

With the weekly temporal resolution we have some room to increase the size of the bounding box!

• 10x10 degree bounding box (1000 x 1000 grid cells)
• 77 weekly observations (540 / P7D)
• total size: 1000 * 1000 * 77 = 7.7e7

bounds = (-10, -10, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=540)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P7D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

If we double the AOI size again, we will break exceed the request size limit:

• 20x20 degree bounding box (1000 x 1000 grid cells)
• 77 weekly observations (540 / P7D)
• total size: 2000 * 2000 * 77 = 3.08e8 (greater than maximum of 1e8

bounds = (-20, -20, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=540)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P7D",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

But if we reduce the temporal resolution from weekly to monthly, it will work!

• 20x20 degree bounding box (1000 x 1000 grid cells)
• 18 monthly observations (540 / P1M)
• total size: 2000 * 2000 * 18 = 3.08e8

bounds = (-20, -20, 0, 0)
bbox_str = ",".join(str(x) for x in bounds)

start_datetime = datetime(year=2011, month=1, day=1, hour=0, minute=0, second=1)
end_datetime = start_datetime + timedelta(days=540)

response = httpx.get(
    f"https://dev-titiler-cmr.delta-backend.com/timeseries/bbox/{bbox_str}.gif",
    params={
        "concept_id": "C1996881146-POCLOUD",
        "datetime": "/".join(dt.isoformat() for dt in [start_datetime, end_datetime]),
        "step": "P1M",
        "variable": "analysed_sst",
        "backend": "xarray",
        "rescale": "273,315",
        "colormap_name": "viridis",
        "temporal_mode": "point",
    },
    timeout=None,
)

However, there is a maximum image size that we can read with the xarray backend, so we cannot increase the bounding box indefinitely. The limit imposed on the API at this time is 5.6e7 grid cells (7500 x 7500 grid cells). In the case of MUR-SST, that is a bounding box of roughly 75 x 75 degrees.