Benchmarking tile generation¶

This notebook walks through benchmarking performance of TiTiler-CMR for a given Earthdata CMR dataset.

In this notebook, you'll learn:

How to benchmark tile rendering performance across zoom levels
What factors impact tile generation performance in TiTiler-CMR.

In [1]:

Copied!





import os

import earthaccess
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.lines import Line2D


from datacube_benchmark import (
    DatasetParams,
    benchmark_viewport,
    tiling_benchmark_summary,
)
import os

import earthaccess
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from matplotlib.lines import Line2D


from datacube_benchmark import (
    DatasetParams,
    benchmark_viewport,
    tiling_benchmark_summary,
)

TiTiler-CMR¶

For this walkthrough, we will use the Titiler-CMR staging endpoint https://staging.openveda.cloud/api/titiler-cmr/.

Titiler-CMR supports two different backends:

xarray → for gridded/cloud-native datasets (e.g., NetCDF4/HDF5/GRIB), typically exposed as variables.
rasterio → for COG/raster imagery-style datasets exposed as bands (optionally via a regex).

Tip: Explore data granules with earthaccess

You can use earthaccess to search and inspect the individual granules used in your query. This helps you validate which files were accessed, their sizes, and the temporal range.

In [2]:

Copied!

concept_id = "C2723754864-GES_DISC"
time_range = ("2022-03-01T00:00:01Z", "2022-03-02T23:59:59Z")

# Authenticate if needed
earthaccess.login()  # or use "interactive" if needed

results = earthaccess.search_data(concept_id=concept_id, temporal=time_range)

print(f"Found {len(results)} granules between {time_range[0]} and {time_range[1]}")
concept_id = "C2723754864-GES_DISC"
time_range = ("2022-03-01T00:00:01Z", "2022-03-02T23:59:59Z")

# Authenticate if needed
earthaccess.login()  # or use "interactive" if needed

results = earthaccess.search_data(concept_id=concept_id, temporal=time_range)

print(f"Found {len(results)} granules between {time_range[0]} and {time_range[1]}")

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connection.py:204, in HTTPConnection._new_conn(self)
    203 try:
--> 204     sock = connection.create_connection(
    205         (self._dns_host, self.port),
    206         self.timeout,
    207         source_address=self.source_address,
    208         socket_options=self.socket_options,
    209     )
    210 except socket.gaierror as e:

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
     84 try:
---> 85     raise err
     86 finally:
     87     # Break explicitly a reference cycle

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
     72     sock.bind(source_address)
---> 73 sock.connect(sa)
     74 # Break explicitly a reference cycle

OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

NewConnectionError                        Traceback (most recent call last)
File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    786 # Make the request on the HTTPConnection object
--> 787 response = self._make_request(
    788     conn,
    789     method,
    790     url,
    791     timeout=timeout_obj,
    792     body=body,
    793     headers=headers,
    794     chunked=chunked,
    795     retries=retries,
    796     response_conn=response_conn,
    797     preload_content=preload_content,
    798     decode_content=decode_content,
    799     **response_kw,
    800 )
    802 # Everything went great!

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py:488, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    487         new_e = _wrap_proxy_error(new_e, conn.proxy.scheme)
--> 488     raise new_e
    490 # conn.request() calls http.client.*.request, not the method in
    491 # urllib3.request. It also calls makefile (recv) on the socket.

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py:464, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
    463 try:
--> 464     self._validate_conn(conn)
    465 except (SocketTimeout, BaseSSLError) as e:

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py:1093, in HTTPSConnectionPool._validate_conn(self, conn)
   1092 if conn.is_closed:
-> 1093     conn.connect()
   1095 # TODO revise this, see https://github.com/urllib3/urllib3/issues/2791

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connection.py:759, in HTTPSConnection.connect(self)
    758 sock: socket.socket | ssl.SSLSocket
--> 759 self.sock = sock = self._new_conn()
    760 server_hostname: str = self.host

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connection.py:219, in HTTPConnection._new_conn(self)
    218 except OSError as e:
--> 219     raise NewConnectionError(
    220         self, f"Failed to establish a new connection: {e}"
    221     ) from e
    223 sys.audit("http.client.connect", self, self.host, self.port)

NewConnectionError: HTTPSConnection(host='urs.earthdata.nasa.gov', port=443): Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/requests/adapters.py:645, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    644 try:
--> 645     resp = conn.urlopen(
    646         method=request.method,
    647         url=url,
    648         body=request.body,
    649         headers=request.headers,
    650         redirect=False,
    651         assert_same_host=False,
    652         preload_content=False,
    653         decode_content=False,
    654         retries=self.max_retries,
    655         timeout=timeout,
    656         chunked=chunked,
    657     )
    659 except (ProtocolError, OSError) as err:

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py:841, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
    839     new_e = ProtocolError("Connection aborted.", new_e)
--> 841 retries = retries.increment(
    842     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    843 )
    844 retries.sleep()

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/urllib3/util/retry.py:535, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    534     reason = error or ResponseError(cause)
--> 535     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    537 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPSConnectionPool(host='urs.earthdata.nasa.gov', port=443): Max retries exceeded with url: /api/users/find_or_create_token (Caused by NewConnectionError("HTTPSConnection(host='urs.earthdata.nasa.gov', port=443): Failed to establish a new connection: [Errno 101] Network is unreachable"))

During handling of the above exception, another exception occurred:

ConnectionError                           Traceback (most recent call last)
Cell In[2], line 5
      2 time_range = ("2022-03-01T00:00:01Z", "2022-03-02T23:59:59Z")
      4 # Authenticate if needed
----> 5 earthaccess.login()  # or use "interactive" if needed
      7 results = earthaccess.search_data(concept_id=concept_id, temporal=time_range)
      9 print(f"Found {len(results)} granules between {time_range[0]} and {time_range[1]}")

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/earthaccess/api.py:360, in login(strategy, persist, system)
    358 for strategy in ["environment", "netrc", "interactive"]:
    359     try:
--> 360         earthaccess.__auth__.login(
    361             strategy=strategy,
    362             persist=persist,
    363             system=system,
    364         )
    365     except LoginStrategyUnavailable as err:
    366         logger.debug(err)

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/earthaccess/auth.py:148, in Auth.login(self, strategy, persist, system)
    146     self._netrc()
    147 elif strategy == "environment":
--> 148     self._environment()
    150 return self

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/earthaccess/auth.py:300, in Auth._environment(self)
    293     raise LoginStrategyUnavailable(
    294         "Either the environment variables EARTHDATA_USERNAME and "
    295         "EARTHDATA_PASSWORD must both be set, or EARTHDATA_TOKEN must be set for "
    296         "the 'environment' login strategy."
    297     )
    299 logger.debug("Using environment variables for EDL")
--> 300 return self._get_credentials(username, password, token)

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/earthaccess/auth.py:314, in Auth._get_credentials(self, username, password, user_token)
    312 self.username = username
    313 self.password = password
--> 314 token_resp = self._find_or_create_token()
    316 if not (token_resp.ok):  # type: ignore
    317     msg = f"Authentication with Earthdata Login failed with:\n{token_resp.text}"

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/earthaccess/auth.py:332, in Auth._find_or_create_token(self)
    330 def _find_or_create_token(self) -> requests.Response:
    331     with self.get_session() as session:
--> 332         return session.post(
    333             self.EDL_FIND_OR_CREATE_TOKEN_URL,
    334             headers={"Accept": "application/json"},
    335             timeout=10,
    336         )

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/requests/sessions.py:640, in Session.post(self, url, data, json, **kwargs)
    629 def post(self, url, data=None, json=None, **kwargs):
    630     r"""Sends a POST request. Returns :class:`Response` object.
    631 
    632     :param url: URL for the new :class:`Request` object.
   (...)
    637     :rtype: requests.Response
    638     """
--> 640     return self.request("POST", url, data=data, json=json, **kwargs)

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/requests/sessions.py:592, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    587 send_kwargs = {
    588     "timeout": timeout,
    589     "allow_redirects": allow_redirects,
    590 }
    591 send_kwargs.update(settings)
--> 592 resp = self.send(prep, **send_kwargs)
    594 return resp

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/requests/sessions.py:706, in Session.send(self, request, **kwargs)
    703 start = preferred_clock()
    705 # Send the request
--> 706 r = adapter.send(request, **kwargs)
    708 # Total elapsed time of the request (approximately)
    709 elapsed = preferred_clock() - start

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/requests/adapters.py:678, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    674     if isinstance(e.reason, _SSLError):
    675         # This branch is for urllib3 v1.22 and later.
    676         raise SSLError(e, request=request)
--> 678     raise ConnectionError(e, request=request)
    680 except ClosedPoolError as e:
    681     raise ConnectionError(e, request=request)

ConnectionError: HTTPSConnectionPool(host='urs.earthdata.nasa.gov', port=443): Max retries exceeded with url: /api/users/find_or_create_token (Caused by NewConnectionError("HTTPSConnection(host='urs.earthdata.nasa.gov', port=443): Failed to establish a new connection: [Errno 101] Network is unreachable"))

Tile Generation Benchmarking¶

We are going to measure the tile generation performance across different zoom levels using titiler_cmr_benchmark.benchmark_viewport function. This function simulates the load of a typical viewport render in a slippy map, where multiple adjacent tiles must be fetched in parallel to draw a single view.

Step 1: Specify API Parameters¶

First, we have to define the parameters for the CMR dataset we want to benchmark. The DatasetParams class encapsulates all the necessary information to interact with a specific dataset via TiTiler-CMR.

Note this first example is for a dataset where each file has global coverage. This is important to evaluating the results of benchmarking.

In [3]:

Copied!





endpoint = os.getenv("TITILER_CMR_ENDPOINT", "https://openveda.cloud/api/titiler-cmr")

ds_xarray = DatasetParams(
    concept_id="C2723754864-GES_DISC",
    backend="xarray",
    datetime_range="2022-04-01T00:00:01Z/2022-04-02T23:59:59Z",
    variable="precipitation",
    step="P1D",
    temporal_mode="point",
)
endpoint = os.getenv("TITILER_CMR_ENDPOINT", "https://openveda.cloud/api/titiler-cmr")

ds_xarray = DatasetParams(
    concept_id="C2723754864-GES_DISC",
    backend="xarray",
    datetime_range="2022-04-01T00:00:01Z/2022-04-02T23:59:59Z",
    variable="precipitation",
    step="P1D",
    temporal_mode="point",
)

Step 2: Specifiy Zoom Levels¶

Zoom levels determine the detail and extent of the area being rendered. At lower zoom levels, a single tile covers a large spatial area and may intersect many granules. This usually translates to more I/O, more resampling/mosaic work, higher latency, and higher chance of timeouts errors.

As you increase the zoom level, each tile covers a smaller area, reducing the number of intersecting granules and the amount of work per request.

We'll define a range of zoom levels to test to see how performance varies.

In [4]:

Copied!





min_zoom = 3
max_zoom = 20

# Define the viewport parameters
viewport_width = 4
viewport_height = 4
lng = 25.0
lat = 29.0
min_zoom = 3
max_zoom = 20

# Define the viewport parameters
viewport_width = 4
viewport_height = 4
lng = 25.0
lat = 29.0

Step 3: Run the Benchmark¶

Now, let's run the benchmark across the specified zoom levels and visualize the results.

Under the hood, benchmark_viewport computes the center tile for each zoom level, selects its neighboring tiles to approximate a viewport, and requests them concurrently from the TiTiler-CMR endpoint. This function returns a pandas.DataFrame containing the response times for each tile request.

In [5]:

Copied!





df_viewport = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_xarray,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=60.0,
)
df_viewport = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_xarray,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=60.0,
)

=== TiTiler-CMR Tile Benchmark ===
Client: 2 physical / 4 logical cores | RAM: 15.61 GiB
Dataset: C2723754864-GES_DISC (xarray)
Query params: 8 parameters
  concept_id: C2723754864-GES_DISC
  backend: xarray
  datetime: 2022-04-01T00:00:01Z/2022-04-02T23:59:59Z
  variable: precipitation
  step: P1D
  temporal_mode: point
  tile_format: png
  tile_scale: 1

~~~~~~~~~~~~~~~~ ERROR JSON REQUEST ~~~~~~~~~~~~~~~~
URL: https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2723754864-GES_DISC&backend=xarray&datetime=2022-04-01T00%3A00%3A01Z%2F2022-04-02T23%3A59%3A59Z&variable=precipitation&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1
Error: 301 Moved Permanently
Body:

---------------------------------------------------------------------------
HTTPStatusError                           Traceback (most recent call last)
Cell In[5], line 1
----> 1 df_viewport = await benchmark_viewport(
      2     endpoint=endpoint,
      3     dataset=ds_xarray,
      4     lng=lng,
      5     lat=lat,
      6     viewport_width=viewport_width,
      7     viewport_height=viewport_height,
      8     min_zoom=min_zoom,
      9     max_zoom=max_zoom,
     10     timeout_s=60.0,
     11 )

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:189, in benchmark_viewport(endpoint, dataset, lng, lat, viewport_width, viewport_height, tms_id, tile_format, tile_scale, min_zoom, max_zoom, timeout_s, max_connections, max_connections_per_host, max_concurrent, **kwargs)
    180     center = tms.tile(lng=lng, lat=lat, zoom=zoom)
    181     return get_surrounding_tiles(
    182         center_x=center.x,
    183         center_y=center.y,
   (...)
    186         height=viewport_height,
    187     )
--> 189 return await benchmarker.benchmark_tiles(
    190     dataset, viewport_strategy, warmup_per_zoom=1, **kwargs
    191 )

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:399, in TiTilerCMRBenchmarker.benchmark_tiles(self, dataset, tiling_strategy, warmup_per_zoom, **kwargs)
    396     print(f"  {k}: {v}")
    398 async with self._create_http_client() as client:
--> 399     tilejson_info = await self._get_tilejson_info(client, tile_params)
    400     tiles_endpoints = tilejson_info["tiles_endpoints"]
    401     tms = morecantile.tms.get(self.tms_id)

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:725, in TiTilerCMRBenchmarker._get_tilejson_info(self, client, params)
    709 """
    710 Query TiTiler-CMR TileJSON and return parsed tiles endpoints, and bounds.
    711 
   (...)
    722     Dictionary with entries, tilejson, tile endpoints, and bounds.
    723 """
    724 url = f"{self.endpoint.rstrip('/')}/{self.tms_id}/tilejson.json"
--> 725 ts_json, _, _ = await self._request_json(
    726     client,
    727     method="GET",
    728     url=url,
    729     params=dict(params),
    730     timeout_s=self.timeout_s,
    731 )
    732 tiles_endpoints = ts_json.get("tiles", [])
    734 if not tiles_endpoints:

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:770, in TiTilerCMRBenchmarker._request_json(self, client, method, url, timeout_s, params, json_payload)
    768 else:
    769     raise ValueError(f"Unsupported HTTP method: {method!r}")
--> 770 response.raise_for_status()
    771 elapsed = time.perf_counter() - t0
    772 data = response.json()

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/httpx/_models.py:829, in Response.raise_for_status(self)
    827 error_type = error_types.get(status_class, "Invalid status code")
    828 message = message.format(self, error_type=error_type)
--> 829 raise HTTPStatusError(message, request=request, response=self)

HTTPStatusError: Redirect response '301 Moved Permanently' for url 'https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2723754864-GES_DISC&backend=xarray&datetime=2022-04-01T00%3A00%3A01Z%2F2022-04-02T23%3A59%3A59Z&variable=precipitation&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1'
Redirect location: 'https://openveda.cloud/api/titiler-cmr/xarray/WebMercatorQuad/tilejson.json?collection_concept_id=C2723754864-GES_DISC&temporal=2022-04-01T00%3A00%3A01Z%2F2022-04-02T23%3A59%3A59Z&variables=precipitation&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301

In [6]:

Copied!

df_viewport.head()
df_viewport.head()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 df_viewport.head()

NameError: name 'df_viewport' is not defined

The output includes the following columns:

zoom, x, y — XYZ tile indices
status_code — HTTP code (200 = success, 204 = no-data, 4xx/5xx = errors)
response_time_sec — wall time in seconds
response_size_bytes — payload size
ok, is_error, has_data — convenience flags

Now, let's use a convenience function to summarize the benchmark results.

In [7]:

Copied!

df_summary = tiling_benchmark_summary(df_viewport)
df_summary.head()
df_summary = tiling_benchmark_summary(df_viewport)
df_summary.head()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 df_summary = tiling_benchmark_summary(df_viewport)
      2 df_summary.head()

NameError: name 'df_viewport' is not defined

Step 4: Plot the results¶

You may notice there is little variation in performance across zoom levels. This collection's files have global extent. Regardless of the zoom level of the tile request, the same file or files must be opened and read. (Multiple files in the case that the datetime parameter returns multiple granules).

This situation is in contrast to the next example which uses files without global extent.

In [8]:

Copied!





def summarize_and_plot_tiles_from_df(
    df: pd.DataFrame,
    *,
    jitter=0.08,
    alpha=0.35,
    figsize=(9, 5),
    title_lines=None,
):
    """Generate summary and plot from tile benchmark DataFrame."""
    summary = tiling_benchmark_summary(df)

    fig, ax = plt.subplots(figsize=figsize)
    fig.subplots_adjust(right=0.72, top=0.80)

    zoom_levels = sorted(
        int(z) for z in pd.to_numeric(df["zoom"], errors="coerce").dropna().unique()
    )
    ax.set_xticks(zoom_levels)
    if zoom_levels:
        ax.set_xlim(min(zoom_levels) - 0.6, max(zoom_levels) + 0.6)

    for z in zoom_levels:
        sub = df[df["zoom"] == z]
        if sub.empty:
            continue

        x = np.random.normal(loc=z, scale=jitter, size=len(sub))
        ok_mask = sub["ok"].astype(bool).values
        err_mask = sub["is_error"].astype(bool).values

        ax.scatter(
            x[ok_mask],
            sub.loc[ok_mask, "response_time_sec"],
            alpha=alpha,
            edgecolor="none",
            label=None,
        )
        ax.scatter(
            x[err_mask],
            sub.loc[err_mask, "response_time_sec"],
            marker="x",
            alpha=min(0.85, alpha + 0.25),
            label=None,
        )

        med = pd.to_numeric(sub["response_time_sec"], errors="coerce").median()
        if np.isfinite(med):
            ax.hlines(med, z - 0.45, z + 0.45, linestyles="--")

    ax.set_xlabel("Zoom level")
    ax.set_ylabel("Tile response time (s)")

    ok_proxy = Line2D([], [], linestyle="none", marker="o", label="200 OK")
    err_proxy = Line2D(
        [], [], linestyle="none", marker="x", label="error (≥400 or failure)"
    )
    ax.legend(
        [ok_proxy, err_proxy],
        ["200 OK", "error"],
        frameon=False,
        loc="upper left",
        bbox_to_anchor=(1.02, 1.00),
    )

    if title_lines:
        ax.set_title("\n".join(title_lines), fontsize=9, loc="left", pad=12)

    ax.grid(True, axis="y", alpha=0.2)
    plt.tight_layout()

    return summary, (fig, ax)


summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport,
    title_lines=[
        "concept_id: C2723754864-GES_DISC",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)
plt.show()
def summarize_and_plot_tiles_from_df(
    df: pd.DataFrame,
    *,
    jitter=0.08,
    alpha=0.35,
    figsize=(9, 5),
    title_lines=None,
):
    """Generate summary and plot from tile benchmark DataFrame."""
    summary = tiling_benchmark_summary(df)

    fig, ax = plt.subplots(figsize=figsize)
    fig.subplots_adjust(right=0.72, top=0.80)

    zoom_levels = sorted(
        int(z) for z in pd.to_numeric(df["zoom"], errors="coerce").dropna().unique()
    )
    ax.set_xticks(zoom_levels)
    if zoom_levels:
        ax.set_xlim(min(zoom_levels) - 0.6, max(zoom_levels) + 0.6)

    for z in zoom_levels:
        sub = df[df["zoom"] == z]
        if sub.empty:
            continue

        x = np.random.normal(loc=z, scale=jitter, size=len(sub))
        ok_mask = sub["ok"].astype(bool).values
        err_mask = sub["is_error"].astype(bool).values

        ax.scatter(
            x[ok_mask],
            sub.loc[ok_mask, "response_time_sec"],
            alpha=alpha,
            edgecolor="none",
            label=None,
        )
        ax.scatter(
            x[err_mask],
            sub.loc[err_mask, "response_time_sec"],
            marker="x",
            alpha=min(0.85, alpha + 0.25),
            label=None,
        )

        med = pd.to_numeric(sub["response_time_sec"], errors="coerce").median()
        if np.isfinite(med):
            ax.hlines(med, z - 0.45, z + 0.45, linestyles="--")

    ax.set_xlabel("Zoom level")
    ax.set_ylabel("Tile response time (s)")

    ok_proxy = Line2D([], [], linestyle="none", marker="o", label="200 OK")
    err_proxy = Line2D(
        [], [], linestyle="none", marker="x", label="error (≥400 or failure)"
    )
    ax.legend(
        [ok_proxy, err_proxy],
        ["200 OK", "error"],
        frameon=False,
        loc="upper left",
        bbox_to_anchor=(1.02, 1.00),
    )

    if title_lines:
        ax.set_title("\n".join(title_lines), fontsize=9, loc="left", pad=12)

    ax.grid(True, axis="y", alpha=0.2)
    plt.tight_layout()

    return summary, (fig, ax)


summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport,
    title_lines=[
        "concept_id: C2723754864-GES_DISC",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)
plt.show()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[8], line 75
     69     plt.tight_layout()
     71     return summary, (fig, ax)
     74 summary, (fig, ax) = summarize_and_plot_tiles_from_df(
---> 75     df_viewport,
     76     title_lines=[
     77         "concept_id: C2723754864-GES_DISC",
     78         "endpoint: https://openveda.cloud/api/titiler-cmr",
     79     ],
     80 )
     81 plt.show()

NameError: name 'df_viewport' is not defined

HLS Example¶

In this example, we will benchmark a CMR dataset that is structured as Cloud-Optimized GeoTIFFs (COGs) with individual bands. We will use the rasterio backend for this dataset.

In contrast to the first example, HLS is much higher spatial resolution (30 meters vs 0.1 degrees)and each granule has a small spatial footprint. In general, the lower the zoom level (more zoomed out), the more files need to be opened to render a tile, which can lead to increased latency.

A more in-depth HLS benchmark report is provided in the Harmonized Landsat Sentinel 2 (HLS): tiling configuration and rendering performance documentation.

In [9]:

Copied!





ds_hls_day = DatasetParams(
    concept_id="C2021957295-LPCLOUD",
    backend="rasterio",
    datetime_range="2023-10-01T00:00:01Z/2023-10-07T00:00:01Z",
    bands=["B04", "B03", "B02"],
    bands_regex="B[0-9][0-9]",
    step="P1D",
    temporal_mode="point",
)
ds_hls_week = DatasetParams(
    concept_id="C2021957657-LPCLOUD",
    backend="rasterio",
    datetime_range="2023-10-01T00:00:01Z/2023-10-20T00:00:01Z",
    bands=["B04", "B03", "B02"],
    bands_regex="B[0-9][0-9]",
    step="P1W",
    temporal_mode="point",
)

min_zoom = 3
max_zoom = 20
viewport_width = 3
viewport_height = 3
timeout_s = 60.0
ds_hls_day = DatasetParams(
    concept_id="C2021957295-LPCLOUD",
    backend="rasterio",
    datetime_range="2023-10-01T00:00:01Z/2023-10-07T00:00:01Z",
    bands=["B04", "B03", "B02"],
    bands_regex="B[0-9][0-9]",
    step="P1D",
    temporal_mode="point",
)
ds_hls_week = DatasetParams(
    concept_id="C2021957657-LPCLOUD",
    backend="rasterio",
    datetime_range="2023-10-01T00:00:01Z/2023-10-20T00:00:01Z",
    bands=["B04", "B03", "B02"],
    bands_regex="B[0-9][0-9]",
    step="P1W",
    temporal_mode="point",
)

min_zoom = 3
max_zoom = 20
viewport_width = 3
viewport_height = 3
timeout_s = 60.0

In [10]:

Copied!





df_viewport_day = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_hls_day,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=timeout_s,
)

df_viewport_day_summary = tiling_benchmark_summary(df_viewport_day)
df_viewport_day_summary.head()
df_viewport_day = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_hls_day,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=timeout_s,
)

df_viewport_day_summary = tiling_benchmark_summary(df_viewport_day)
df_viewport_day_summary.head()

=== TiTiler-CMR Tile Benchmark ===
Client: 2 physical / 4 logical cores | RAM: 15.61 GiB
Dataset: C2021957295-LPCLOUD (rasterio)
Query params: 11 parameters
  concept_id: C2021957295-LPCLOUD
  backend: rasterio
  datetime: 2023-10-01T00:00:01Z/2023-10-07T00:00:01Z
  bands: B04
  bands: B03
  bands: B02
  bands_regex: B[0-9][0-9]
  step: P1D
  temporal_mode: point
  tile_format: png
  tile_scale: 1

~~~~~~~~~~~~~~~~ ERROR JSON REQUEST ~~~~~~~~~~~~~~~~
URL: https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2021957295-LPCLOUD&backend=rasterio&datetime=2023-10-01T00%3A00%3A01Z%2F2023-10-07T00%3A00%3A01Z&bands=B02&bands_regex=B%5B0-9%5D%5B0-9%5D&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1
Error: 301 Moved Permanently
Body:

---------------------------------------------------------------------------
HTTPStatusError                           Traceback (most recent call last)
Cell In[10], line 1
----> 1 df_viewport_day = await benchmark_viewport(
      2     endpoint=endpoint,
      3     dataset=ds_hls_day,
      4     lng=lng,
      5     lat=lat,
      6     viewport_width=viewport_width,
      7     viewport_height=viewport_height,
      8     min_zoom=min_zoom,
      9     max_zoom=max_zoom,
     10     timeout_s=timeout_s,
     11 )
     13 df_viewport_day_summary = tiling_benchmark_summary(df_viewport_day)
     14 df_viewport_day_summary.head()

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:189, in benchmark_viewport(endpoint, dataset, lng, lat, viewport_width, viewport_height, tms_id, tile_format, tile_scale, min_zoom, max_zoom, timeout_s, max_connections, max_connections_per_host, max_concurrent, **kwargs)
    180     center = tms.tile(lng=lng, lat=lat, zoom=zoom)
    181     return get_surrounding_tiles(
    182         center_x=center.x,
    183         center_y=center.y,
   (...)
    186         height=viewport_height,
    187     )
--> 189 return await benchmarker.benchmark_tiles(
    190     dataset, viewport_strategy, warmup_per_zoom=1, **kwargs
    191 )

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:399, in TiTilerCMRBenchmarker.benchmark_tiles(self, dataset, tiling_strategy, warmup_per_zoom, **kwargs)
    396     print(f"  {k}: {v}")
    398 async with self._create_http_client() as client:
--> 399     tilejson_info = await self._get_tilejson_info(client, tile_params)
    400     tiles_endpoints = tilejson_info["tiles_endpoints"]
    401     tms = morecantile.tms.get(self.tms_id)

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:725, in TiTilerCMRBenchmarker._get_tilejson_info(self, client, params)
    709 """
    710 Query TiTiler-CMR TileJSON and return parsed tiles endpoints, and bounds.
    711 
   (...)
    722     Dictionary with entries, tilejson, tile endpoints, and bounds.
    723 """
    724 url = f"{self.endpoint.rstrip('/')}/{self.tms_id}/tilejson.json"
--> 725 ts_json, _, _ = await self._request_json(
    726     client,
    727     method="GET",
    728     url=url,
    729     params=dict(params),
    730     timeout_s=self.timeout_s,
    731 )
    732 tiles_endpoints = ts_json.get("tiles", [])
    734 if not tiles_endpoints:

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:770, in TiTilerCMRBenchmarker._request_json(self, client, method, url, timeout_s, params, json_payload)
    768 else:
    769     raise ValueError(f"Unsupported HTTP method: {method!r}")
--> 770 response.raise_for_status()
    771 elapsed = time.perf_counter() - t0
    772 data = response.json()

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/httpx/_models.py:829, in Response.raise_for_status(self)
    827 error_type = error_types.get(status_class, "Invalid status code")
    828 message = message.format(self, error_type=error_type)
--> 829 raise HTTPStatusError(message, request=request, response=self)

HTTPStatusError: Redirect response '301 Moved Permanently' for url 'https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2021957295-LPCLOUD&backend=rasterio&datetime=2023-10-01T00%3A00%3A01Z%2F2023-10-07T00%3A00%3A01Z&bands=B02&bands_regex=B%5B0-9%5D%5B0-9%5D&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1'
Redirect location: 'https://openveda.cloud/api/titiler-cmr/rasterio/WebMercatorQuad/tilejson.json?collection_concept_id=C2021957295-LPCLOUD&temporal=2023-10-01T00%3A00%3A01Z%2F2023-10-07T00%3A00%3A01Z&assets=B02&assets_regex=B%5B0-9%5D%5B0-9%5D&step=P1D&temporal_mode=point&tile_format=png&tile_scale=1'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301

In [11]:

Copied!





df_viewport_week = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_hls_week,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=timeout_s,
)

df_viewport_week_summary = tiling_benchmark_summary(df_viewport_week)
df_viewport_week_summary.head()
df_viewport_week = await benchmark_viewport(
    endpoint=endpoint,
    dataset=ds_hls_week,
    lng=lng,
    lat=lat,
    viewport_width=viewport_width,
    viewport_height=viewport_height,
    min_zoom=min_zoom,
    max_zoom=max_zoom,
    timeout_s=timeout_s,
)

df_viewport_week_summary = tiling_benchmark_summary(df_viewport_week)
df_viewport_week_summary.head()

=== TiTiler-CMR Tile Benchmark ===
Client: 2 physical / 4 logical cores | RAM: 15.61 GiB
Dataset: C2021957657-LPCLOUD (rasterio)
Query params: 11 parameters
  concept_id: C2021957657-LPCLOUD
  backend: rasterio
  datetime: 2023-10-01T00:00:01Z/2023-10-20T00:00:01Z
  bands: B04
  bands: B03
  bands: B02
  bands_regex: B[0-9][0-9]
  step: P1W
  temporal_mode: point
  tile_format: png
  tile_scale: 1

~~~~~~~~~~~~~~~~ ERROR JSON REQUEST ~~~~~~~~~~~~~~~~
URL: https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2021957657-LPCLOUD&backend=rasterio&datetime=2023-10-01T00%3A00%3A01Z%2F2023-10-20T00%3A00%3A01Z&bands=B02&bands_regex=B%5B0-9%5D%5B0-9%5D&step=P1W&temporal_mode=point&tile_format=png&tile_scale=1
Error: 301 Moved Permanently
Body:

---------------------------------------------------------------------------
HTTPStatusError                           Traceback (most recent call last)
Cell In[11], line 1
----> 1 df_viewport_week = await benchmark_viewport(
      2     endpoint=endpoint,
      3     dataset=ds_hls_week,
      4     lng=lng,
      5     lat=lat,
      6     viewport_width=viewport_width,
      7     viewport_height=viewport_height,
      8     min_zoom=min_zoom,
      9     max_zoom=max_zoom,
     10     timeout_s=timeout_s,
     11 )
     13 df_viewport_week_summary = tiling_benchmark_summary(df_viewport_week)
     14 df_viewport_week_summary.head()

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:189, in benchmark_viewport(endpoint, dataset, lng, lat, viewport_width, viewport_height, tms_id, tile_format, tile_scale, min_zoom, max_zoom, timeout_s, max_connections, max_connections_per_host, max_concurrent, **kwargs)
    180     center = tms.tile(lng=lng, lat=lat, zoom=zoom)
    181     return get_surrounding_tiles(
    182         center_x=center.x,
    183         center_y=center.y,
   (...)
    186         height=viewport_height,
    187     )
--> 189 return await benchmarker.benchmark_tiles(
    190     dataset, viewport_strategy, warmup_per_zoom=1, **kwargs
    191 )

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:399, in TiTilerCMRBenchmarker.benchmark_tiles(self, dataset, tiling_strategy, warmup_per_zoom, **kwargs)
    396     print(f"  {k}: {v}")
    398 async with self._create_http_client() as client:
--> 399     tilejson_info = await self._get_tilejson_info(client, tile_params)
    400     tiles_endpoints = tilejson_info["tiles_endpoints"]
    401     tms = morecantile.tms.get(self.tms_id)

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:725, in TiTilerCMRBenchmarker._get_tilejson_info(self, client, params)
    709 """
    710 Query TiTiler-CMR TileJSON and return parsed tiles endpoints, and bounds.
    711 
   (...)
    722     Dictionary with entries, tilejson, tile endpoints, and bounds.
    723 """
    724 url = f"{self.endpoint.rstrip('/')}/{self.tms_id}/tilejson.json"
--> 725 ts_json, _, _ = await self._request_json(
    726     client,
    727     method="GET",
    728     url=url,
    729     params=dict(params),
    730     timeout_s=self.timeout_s,
    731 )
    732 tiles_endpoints = ts_json.get("tiles", [])
    734 if not tiles_endpoints:

File ~/work/titiler-cmr/titiler-cmr/docs/packages/datacube_benchmark/src/datacube_benchmark/cmr/benchmark.py:770, in TiTilerCMRBenchmarker._request_json(self, client, method, url, timeout_s, params, json_payload)
    768 else:
    769     raise ValueError(f"Unsupported HTTP method: {method!r}")
--> 770 response.raise_for_status()
    771 elapsed = time.perf_counter() - t0
    772 data = response.json()

File ~/work/titiler-cmr/titiler-cmr/.venv/lib/python3.12/site-packages/httpx/_models.py:829, in Response.raise_for_status(self)
    827 error_type = error_types.get(status_class, "Invalid status code")
    828 message = message.format(self, error_type=error_type)
--> 829 raise HTTPStatusError(message, request=request, response=self)

HTTPStatusError: Redirect response '301 Moved Permanently' for url 'https://openveda.cloud/api/titiler-cmr/WebMercatorQuad/tilejson.json?concept_id=C2021957657-LPCLOUD&backend=rasterio&datetime=2023-10-01T00%3A00%3A01Z%2F2023-10-20T00%3A00%3A01Z&bands=B02&bands_regex=B%5B0-9%5D%5B0-9%5D&step=P1W&temporal_mode=point&tile_format=png&tile_scale=1'
Redirect location: 'https://openveda.cloud/api/titiler-cmr/rasterio/WebMercatorQuad/tilejson.json?collection_concept_id=C2021957657-LPCLOUD&temporal=2023-10-01T00%3A00%3A01Z%2F2023-10-20T00%3A00%3A01Z&assets=B02&assets_regex=B%5B0-9%5D%5B0-9%5D&step=P1W&temporal_mode=point&tile_format=png&tile_scale=1'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301

In [12]:

Copied!





summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport_day,
    title_lines=[
        "concept_id: C2036881735-POCLOUD",
        "Viewport: 3x3 tiles -- daily",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)

plt.show()
summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport_day,
    title_lines=[
        "concept_id: C2036881735-POCLOUD",
        "Viewport: 3x3 tiles -- daily",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)

plt.show()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[12], line 2
      1 summary, (fig, ax) = summarize_and_plot_tiles_from_df(
----> 2     df_viewport_day,
      3     title_lines=[
      4         "concept_id: C2036881735-POCLOUD",
      5         "Viewport: 3x3 tiles -- daily",
      6         "endpoint: https://openveda.cloud/api/titiler-cmr",
      7     ],
      8 )
     10 plt.show()

NameError: name 'df_viewport_day' is not defined

In [13]:

Copied!





summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport_week,
    title_lines=[
        "concept_id: C2036881735-POCLOUD",
        "Viewport: 3x3 tiles -- weekly",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)

plt.show()
summary, (fig, ax) = summarize_and_plot_tiles_from_df(
    df_viewport_week,
    title_lines=[
        "concept_id: C2036881735-POCLOUD",
        "Viewport: 3x3 tiles -- weekly",
        "endpoint: https://openveda.cloud/api/titiler-cmr",
    ],
)

plt.show()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[13], line 2
      1 summary, (fig, ax) = summarize_and_plot_tiles_from_df(
----> 2     df_viewport_week,
      3     title_lines=[
      4         "concept_id: C2036881735-POCLOUD",
      5         "Viewport: 3x3 tiles -- weekly",
      6         "endpoint: https://openveda.cloud/api/titiler-cmr",
      7     ],
      8 )
     10 plt.show()

NameError: name 'df_viewport_week' is not defined

Conclusion¶

In this notebook, we explored how to check the performance of tile rendering performance in TiTiler-CMR using different datasets and backends. We observed how factors such as zoom levels, temporal intervals, and dataset structures impact the latency of tile requests.

In general, performance depends on:

zoom level and spatial resolution of the dataset
the width of the datetime interval and the temporal resolution of the dataset
how many granules intersect the tile footprint

Takeaways:

Consider specifying a minzoom and maximum datetime interval given a specific datasets temporal and spatial resolution.