Skip to content

AWS S3

obstore.store.S3Store

Interface to an Amazon S3 bucket.

All constructors will check for environment variables. All environment variables starting with AWS_ will be evaluated. Names must match keys from S3ConfigInput. Only upper-case environment variables are accepted.

Some examples of variables extracted from environment:

Examples:

Using requester-pays buckets:

Pass request_payer=True as a keyword argument or have AWS_REQUESTER_PAYS=True set in the environment.

Anonymous requests:

Pass skip_signature=True as a keyword argument or have AWS_SKIP_SIGNATURE=True set in the environment.

client_options property 

client_options: ClientConfig | None

Get the store's client configuration.

config property 

config: S3Config

Get the underlying S3 config parameters.

prefix property 

prefix: str | None

Get the prefix applied to all operations in this store, if any.

retry_config property 

retry_config: RetryConfig | None

Get the store's retry configuration.

__init__ 

__init__(
    bucket: str | None = None,
    *,
    prefix: str | None = None,
    config: S3Config | S3ConfigInput | None = None,
    client_options: ClientConfig | None = None,
    retry_config: RetryConfig | None = None,
    credential_provider: S3CredentialProvider | None = None,
    **kwargs: Unpack[S3ConfigInput],
) -> None

Create a new S3Store.

Parameters:

  • bucket (str | None, default: None ) –

    The AWS bucket to use.

Other Parameters:

  • prefix (str | None) –

    A prefix within the bucket to use for all operations.

  • config (S3Config | S3ConfigInput | None) –

    AWS configuration. Values in this config will override values inferred from the environment. Defaults to None.

  • client_options (ClientConfig | None) –

    HTTP Client options. Defaults to None.

  • retry_config (RetryConfig | None) –

    Retry configuration. Defaults to None.

  • credential_provider (S3CredentialProvider | None) –

    A callback to provide custom S3 credentials.

  • kwargs (Unpack[S3ConfigInput]) –

    AWS configuration values. Supports the same values as config, but as named keyword args.

Returns:

  • None –

    S3Store

from_url classmethod 

from_url(
    url: str,
    *,
    config: S3Config | S3ConfigInput | None = None,
    client_options: ClientConfig | None = None,
    retry_config: RetryConfig | None = None,
    credential_provider: S3CredentialProvider | None = None,
    **kwargs: Unpack[S3ConfigInput],
) -> S3Store

Parse available connection info from a well-known storage URL.

The supported url schemes are:

  • s3://<bucket>/<path>
  • s3a://<bucket>/<path>
  • https://s3.<region>.amazonaws.com/<bucket>
  • https://<bucket>.s3.<region>.amazonaws.com
  • https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket

Parameters:

  • url (str) –

    well-known storage URL.

Other Parameters:

  • config (S3Config | S3ConfigInput | None) –

    AWS Configuration. Values in this config will override values inferred from the url. Defaults to None.

  • client_options (ClientConfig | None) –

    HTTP Client options. Defaults to None.

  • retry_config (RetryConfig | None) –

    Retry configuration. Defaults to None.

  • credential_provider (S3CredentialProvider | None) –

    A callback to provide custom S3 credentials.

  • kwargs (Unpack[S3ConfigInput]) –

    AWS configuration values. Supports the same values as config, but as named keyword args.

Returns:

obstore.store.S3ConfigInput

Bases: TypedDict

Configuration parameters for S3Store.

There are duplicates of many parameters, and parameters can be either upper or lower case. Not all parameters are required.

Not importable at runtime

To use this type hint in your code, import it within a TYPE_CHECKING block:

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from obstore.store import S3ConfigInput

ACCESS_KEY_ID instance-attribute 

ACCESS_KEY_ID: str

AWS Access Key

AWS_ACCESS_KEY_ID instance-attribute 

AWS_ACCESS_KEY_ID: str

AWS Access Key

AWS_BUCKET instance-attribute 

AWS_BUCKET: str

Bucket name

AWS_BUCKET_NAME instance-attribute 

AWS_BUCKET_NAME: str

Bucket name

AWS_CHECKSUM_ALGORITHM instance-attribute 

AWS_CHECKSUM_ALGORITHM: S3ChecksumAlgorithm | str

Sets the checksum algorithm which has to be used for object integrity check during upload.

AWS_CONDITIONAL_PUT instance-attribute 

AWS_CONDITIONAL_PUT: str

Configure how to provide conditional put support

Supported values:

  • "etag" (default): Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headers If-Match and If-None-Match.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>": The name of a DynamoDB table to use for coordination.

    This will use the same region, credentials and endpoint as configured for S3.

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI instance-attribute 

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: str

Set the container credentials relative URI

docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

AWS_COPY_IF_NOT_EXISTS instance-attribute 

AWS_COPY_IF_NOT_EXISTS: Literal['multipart'] | str

Configure how to provide "copy if not exists".

Supported values:

  • "multipart":

    Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.

    Warning

    When using this mode, copy_if_not_exists does not copy tags or attributes from the source object.

    Warning

    When using this mode, copy_if_not_exists makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads.

  • "header:<HEADER_NAME>:<HEADER_VALUE>":

    Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.

    If set, copy_if_not_exists will perform a normal copy operation with the provided header pair, and expect the store to fail with 412 Precondition Failed if the destination file already exists.

    For example header: cf-copy-destination-if-none-match: *, would set the header cf-copy-destination-if-none-match to *.

  • "header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>":

    The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>":

    The name of a DynamoDB table to use for coordination.

    The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.

AWS_DEFAULT_REGION instance-attribute 

AWS_DEFAULT_REGION: S3Regions | str

Default region

AWS_DISABLE_TAGGING instance-attribute 

AWS_DISABLE_TAGGING: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

AWS_ENDPOINT instance-attribute 

AWS_ENDPOINT: str

Sets custom endpoint for communicating with AWS S3.

AWS_ENDPOINT_URL instance-attribute 

AWS_ENDPOINT_URL: str

Sets custom endpoint for communicating with AWS S3.

AWS_IMDSV1_FALLBACK instance-attribute 

AWS_IMDSV1_FALLBACK: bool

Fall back to ImdsV1

AWS_METADATA_ENDPOINT instance-attribute 

AWS_METADATA_ENDPOINT: str

Set the instance metadata endpoint

AWS_REGION instance-attribute 

AWS_REGION: S3Regions | str

Region

AWS_REQUEST_PAYER instance-attribute 

AWS_REQUEST_PAYER: bool

If True, enable operations on requester-pays buckets.

AWS_S3_EXPRESS instance-attribute 

AWS_S3_EXPRESS: str

Enable Support for S3 Express One Zone

AWS_SECRET_ACCESS_KEY instance-attribute 

AWS_SECRET_ACCESS_KEY: str

Secret Access Key

AWS_SERVER_SIDE_ENCRYPTION instance-attribute 

AWS_SERVER_SIDE_ENCRYPTION: S3EncryptionAlgorithm | str

Type of encryption to use.

If set, must be one of:

  • "AES256" (SSE-S3)
  • "aws:kms" (SSE-KMS)
  • "aws:kms:dsse" (DSSE-KMS)
  • "sse-c"

AWS_SESSION_TOKEN instance-attribute 

AWS_SESSION_TOKEN: str

Token to use for requests (passed to underlying provider)

AWS_SKIP_SIGNATURE instance-attribute 

AWS_SKIP_SIGNATURE: bool

If True, S3Store will not fetch credentials and will not sign requests.

AWS_SSE_BUCKET_KEY_ENABLED instance-attribute 

AWS_SSE_BUCKET_KEY_ENABLED: bool

If set to True, will use the bucket's default KMS key for server-side encryption. If set to False, will disable the use of the bucket's default KMS key for server-side encryption.

AWS_SSE_CUSTOMER_KEY_BASE64 instance-attribute 

AWS_SSE_CUSTOMER_KEY_BASE64: str

The base64 encoded, 256-bit customer encryption key to use for server-side encryption. If set, the server side encryption config value must be "sse-c".

AWS_SSE_KMS_KEY_ID instance-attribute 

AWS_SSE_KMS_KEY_ID: str

The KMS key ID to use for server-side encryption.

If set, the server side encrypting config value must be "aws:kms" or "aws:kms:dsse".

AWS_TOKEN instance-attribute 

AWS_TOKEN: str

Token to use for requests (passed to underlying provider)

AWS_UNSIGNED_PAYLOAD instance-attribute 

AWS_UNSIGNED_PAYLOAD: bool

Avoid computing payload checksum when calculating signature.

AWS_VIRTUAL_HOSTED_STYLE_REQUEST instance-attribute 

AWS_VIRTUAL_HOSTED_STYLE_REQUEST: bool

If virtual hosted style request has to be used.

BUCKET instance-attribute 

BUCKET: str

Bucket name

BUCKET_NAME instance-attribute 

BUCKET_NAME: str

Bucket name

CHECKSUM_ALGORITHM instance-attribute 

CHECKSUM_ALGORITHM: S3ChecksumAlgorithm | str

Sets the checksum algorithm which has to be used for object integrity check during upload.

CONDITIONAL_PUT instance-attribute 

CONDITIONAL_PUT: str

Configure how to provide conditional put support

Supported values:

  • "etag" (default): Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headers If-Match and If-None-Match.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>": The name of a DynamoDB table to use for coordination.

    This will use the same region, credentials and endpoint as configured for S3.

COPY_IF_NOT_EXISTS instance-attribute 

COPY_IF_NOT_EXISTS: Literal['multipart'] | str

Configure how to provide "copy if not exists".

Supported values:

  • "multipart":

    Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.

    Warning

    When using this mode, copy_if_not_exists does not copy tags or attributes from the source object.

    Warning

    When using this mode, copy_if_not_exists makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads.

  • "header:<HEADER_NAME>:<HEADER_VALUE>":

    Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.

    If set, copy_if_not_exists will perform a normal copy operation with the provided header pair, and expect the store to fail with 412 Precondition Failed if the destination file already exists.

    For example header: cf-copy-destination-if-none-match: *, would set the header cf-copy-destination-if-none-match to *.

  • "header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>":

    The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>":

    The name of a DynamoDB table to use for coordination.

    The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.

DEFAULT_REGION instance-attribute 

DEFAULT_REGION: S3Regions | str

Default region

DISABLE_TAGGING instance-attribute 

DISABLE_TAGGING: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

ENDPOINT instance-attribute 

ENDPOINT: str

Sets custom endpoint for communicating with AWS S3.

ENDPOINT_URL instance-attribute 

ENDPOINT_URL: str

Sets custom endpoint for communicating with AWS S3.

IMDSV1_FALLBACK instance-attribute 

IMDSV1_FALLBACK: bool

Fall back to ImdsV1

METADATA_ENDPOINT instance-attribute 

METADATA_ENDPOINT: str

Set the instance metadata endpoint

REGION instance-attribute 

REGION: S3Regions | str

Region

REQUEST_PAYER instance-attribute 

REQUEST_PAYER: bool

If True, enable operations on requester-pays buckets.

S3_EXPRESS instance-attribute 

S3_EXPRESS: str

Enable Support for S3 Express One Zone

SECRET_ACCESS_KEY instance-attribute 

SECRET_ACCESS_KEY: str

Secret Access Key

SESSION_TOKEN instance-attribute 

SESSION_TOKEN: str

Token to use for requests (passed to underlying provider)

SKIP_SIGNATURE instance-attribute 

SKIP_SIGNATURE: bool

If True, S3Store will not fetch credentials and will not sign requests.

TOKEN instance-attribute 

TOKEN: str

Token to use for requests (passed to underlying provider)

UNSIGNED_PAYLOAD instance-attribute 

UNSIGNED_PAYLOAD: bool

Avoid computing payload checksum when calculating signature.

VIRTUAL_HOSTED_STYLE_REQUEST instance-attribute 

VIRTUAL_HOSTED_STYLE_REQUEST: bool

If virtual hosted style request has to be used.

access_key_id instance-attribute 

access_key_id: str

AWS Access Key

aws_access_key_id instance-attribute 

aws_access_key_id: str

AWS Access Key

aws_bucket instance-attribute 

aws_bucket: str

Bucket name

aws_bucket_name instance-attribute 

aws_bucket_name: str

Bucket name

aws_checksum_algorithm instance-attribute 

aws_checksum_algorithm: S3ChecksumAlgorithm | str

Sets the checksum algorithm which has to be used for object integrity check during upload.

aws_conditional_put instance-attribute 

aws_conditional_put: str

Configure how to provide conditional put support

Supported values:

  • "etag" (default): Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headers If-Match and If-None-Match.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>": The name of a DynamoDB table to use for coordination.

    This will use the same region, credentials and endpoint as configured for S3.

aws_container_credentials_relative_uri instance-attribute 

aws_container_credentials_relative_uri: str

Set the container credentials relative URI

docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

aws_copy_if_not_exists instance-attribute 

aws_copy_if_not_exists: Literal['multipart'] | str

Configure how to provide "copy if not exists".

Supported values:

  • "multipart":

    Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.

    Warning

    When using this mode, copy_if_not_exists does not copy tags or attributes from the source object.

    Warning

    When using this mode, copy_if_not_exists makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads.

  • "header:<HEADER_NAME>:<HEADER_VALUE>":

    Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.

    If set, copy_if_not_exists will perform a normal copy operation with the provided header pair, and expect the store to fail with 412 Precondition Failed if the destination file already exists.

    For example header: cf-copy-destination-if-none-match: *, would set the header cf-copy-destination-if-none-match to *.

  • "header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>":

    The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>":

    The name of a DynamoDB table to use for coordination.

    The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.

aws_default_region instance-attribute 

aws_default_region: S3Regions | str

Default region

aws_disable_tagging instance-attribute 

aws_disable_tagging: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

aws_endpoint instance-attribute 

aws_endpoint: str

Sets custom endpoint for communicating with AWS S3.

aws_endpoint_url instance-attribute 

aws_endpoint_url: str

Sets custom endpoint for communicating with AWS S3.

aws_imdsv1_fallback instance-attribute 

aws_imdsv1_fallback: bool

Fall back to ImdsV1

aws_metadata_endpoint instance-attribute 

aws_metadata_endpoint: str

Set the instance metadata endpoint

aws_region instance-attribute 

aws_region: S3Regions | str

Region

aws_request_payer instance-attribute 

aws_request_payer: bool

If True, enable operations on requester-pays buckets.

aws_s3_express instance-attribute 

aws_s3_express: bool

Enable Support for S3 Express One Zone

aws_secret_access_key instance-attribute 

aws_secret_access_key: str

Secret Access Key

aws_server_side_encryption instance-attribute 

aws_server_side_encryption: S3EncryptionAlgorithm | str

Type of encryption to use.

If set, must be one of:

  • "AES256" (SSE-S3)
  • "aws:kms" (SSE-KMS)
  • "aws:kms:dsse" (DSSE-KMS)
  • "sse-c"

aws_session_token instance-attribute 

aws_session_token: str

Token to use for requests (passed to underlying provider)

aws_skip_signature instance-attribute 

aws_skip_signature: bool

If True, S3Store will not fetch credentials and will not sign requests.

aws_sse_bucket_key_enabled instance-attribute 

aws_sse_bucket_key_enabled: bool

If set to True, will use the bucket's default KMS key for server-side encryption. If set to False, will disable the use of the bucket's default KMS key for server-side encryption.

aws_sse_customer_key_base64 instance-attribute 

aws_sse_customer_key_base64: str

The base64 encoded, 256-bit customer encryption key to use for server-side encryption. If set, the server side encryption config value must be "sse-c".

aws_sse_kms_key_id instance-attribute 

aws_sse_kms_key_id: str

The KMS key ID to use for server-side encryption.

If set, the server side encryption config value must be "aws:kms" or "aws:kms:dsse".

aws_token instance-attribute 

aws_token: str

Token to use for requests (passed to underlying provider)

aws_unsigned_payload instance-attribute 

aws_unsigned_payload: bool

Avoid computing payload checksum when calculating signature.

aws_virtual_hosted_style_request instance-attribute 

aws_virtual_hosted_style_request: bool

If virtual hosted style request has to be used.

bucket_name instance-attribute 

bucket_name: str

Bucket name

checksum_algorithm instance-attribute 

checksum_algorithm: S3ChecksumAlgorithm | str

Sets the checksum algorithm which has to be used for object integrity check during upload.

conditional_put instance-attribute 

conditional_put: str

Configure how to provide conditional put support

Supported values:

  • "etag" (default): Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headers If-Match and If-None-Match.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>": The name of a DynamoDB table to use for coordination.

    This will use the same region, credentials and endpoint as configured for S3.

copy_if_not_exists instance-attribute 

copy_if_not_exists: Literal['multipart'] | str

Configure how to provide "copy if not exists".

Supported values:

  • "multipart":

    Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.

    Warning

    When using this mode, copy_if_not_exists does not copy tags or attributes from the source object.

    Warning

    When using this mode, copy_if_not_exists makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads.

  • "header:<HEADER_NAME>:<HEADER_VALUE>":

    Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.

    If set, copy_if_not_exists will perform a normal copy operation with the provided header pair, and expect the store to fail with 412 Precondition Failed if the destination file already exists.

    For example header: cf-copy-destination-if-none-match: *, would set the header cf-copy-destination-if-none-match to *.

  • "header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>":

    The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.

  • "dynamo:<TABLE_NAME>" or "dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>":

    The name of a DynamoDB table to use for coordination.

    The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.

default_region instance-attribute 

default_region: S3Regions | str

Default region

disable_tagging instance-attribute 

disable_tagging: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

endpoint instance-attribute 

endpoint: str

Sets custom endpoint for communicating with AWS S3.

endpoint_url instance-attribute 

endpoint_url: str

Sets custom endpoint for communicating with AWS S3.

imdsv1_fallback instance-attribute 

imdsv1_fallback: bool

Fall back to ImdsV1

metadata_endpoint instance-attribute 

metadata_endpoint: str

Set the instance metadata endpoint

region instance-attribute 

region: S3Regions | str

Region

request_payer instance-attribute 

request_payer: bool

If True, enable operations on requester-pays buckets.

s3_express instance-attribute 

s3_express: bool

Enable Support for S3 Express One Zone

secret_access_key instance-attribute 

secret_access_key: str

Secret Access Key

session_token instance-attribute 

session_token: str

Token to use for requests (passed to underlying provider)

skip_signature instance-attribute 

skip_signature: bool

If True, S3Store will not fetch credentials and will not sign requests.

token instance-attribute 

token: str

Token to use for requests (passed to underlying provider)

unsigned_payload instance-attribute 

unsigned_payload: bool

Avoid computing payload checksum when calculating signature.

virtual_hosted_style_request instance-attribute 

virtual_hosted_style_request: bool

If virtual hosted style request has to be used.

obstore.store.S3Config

Bases: TypedDict

Configuration parameters returned from S3Store.config.

Note that this is a strict subset of the keys allowed for input into the store, see S3ConfigInput.

Not importable at runtime

To use this type hint in your code, import it within a TYPE_CHECKING block:

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from obstore.store import S3Config

aws_access_key_id instance-attribute 

aws_access_key_id: str

AWS Access Key

aws_bucket instance-attribute 

aws_bucket: str

Bucket name

aws_checksum_algorithm instance-attribute 

aws_checksum_algorithm: S3ChecksumAlgorithm | str

Sets the checksum algorithm which has to be used for object integrity check during upload.

aws_conditional_put instance-attribute 

aws_conditional_put: str

See S3ConfigInput.aws_conditional_put.

aws_container_credentials_relative_uri instance-attribute 

aws_container_credentials_relative_uri: str

See S3ConfigInput.aws_container_credentials_relative_uri.

aws_copy_if_not_exists instance-attribute 

aws_copy_if_not_exists: Literal['multipart'] | str

See S3ConfigInput.aws_copy_if_not_exists.

aws_default_region instance-attribute 

aws_default_region: S3Regions | str

Default region

aws_disable_tagging instance-attribute 

aws_disable_tagging: bool

Disable tagging objects. This can be desirable if not supported by the backing store.

aws_endpoint instance-attribute 

aws_endpoint: str

Sets custom endpoint for communicating with AWS S3.

aws_imdsv1_fallback instance-attribute 

aws_imdsv1_fallback: bool

Fall back to ImdsV1

aws_metadata_endpoint instance-attribute 

aws_metadata_endpoint: str

Set the instance metadata endpoint

aws_region instance-attribute 

aws_region: S3Regions | str

Region

aws_request_payer instance-attribute 

aws_request_payer: bool

If True, enable operations on requester-pays buckets.

aws_s3_express instance-attribute 

aws_s3_express: bool

Enable Support for S3 Express One Zone

aws_secret_access_key instance-attribute 

aws_secret_access_key: str

Secret Access Key

aws_server_side_encryption instance-attribute 

aws_server_side_encryption: S3EncryptionAlgorithm | str

See S3ConfigInput.aws_server_side_encryption.

aws_session_token instance-attribute 

aws_session_token: str

Token to use for requests (passed to underlying provider)

aws_skip_signature instance-attribute 

aws_skip_signature: bool

If True, S3Store will not fetch credentials and will not sign requests.

aws_sse_bucket_key_enabled instance-attribute 

aws_sse_bucket_key_enabled: bool

If set to True, will use the bucket's default KMS key for server-side encryption. If set to False, will disable the use of the bucket's default KMS key for server-side encryption.

aws_sse_customer_key_base64 instance-attribute 

aws_sse_customer_key_base64: str

The base64 encoded, 256-bit customer encryption key to use for server-side encryption. If set, the server side encryption config value must be "sse-c".

aws_sse_kms_key_id instance-attribute 

aws_sse_kms_key_id: str

The KMS key ID to use for server-side encryption.

If set, the server side encryption config value must be "aws:kms" or "aws:kms:dsse".

aws_token instance-attribute 

aws_token: str

Token to use for requests (passed to underlying provider)

aws_unsigned_payload instance-attribute 

aws_unsigned_payload: bool

Avoid computing payload checksum when calculating signature.

aws_virtual_hosted_style_request instance-attribute 

aws_virtual_hosted_style_request: bool

If virtual hosted style request has to be used.

obstore.store.S3Credential

Bases: TypedDict

An S3 credential.

Not importable at runtime

To use this type hint in your code, import it within a TYPE_CHECKING block:

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from obstore.store import S3Credential

access_key_id instance-attribute 

access_key_id: str

AWS access key ID.

expires_at instance-attribute 

expires_at: datetime | None

Expiry datetime of credential. The datetime should have time zone set.

If None, the credential will never expire.

secret_access_key instance-attribute 

secret_access_key: str

AWS secret access key

token instance-attribute 

token: NotRequired[str | None]

AWS token.

obstore.store.S3CredentialProvider

Bases: Protocol

A type hint for a synchronous or asynchronous callback to provide custom S3 credentials.

This should be passed into the credential_provider parameter of S3Store.

Examples:

Return static credentials that don't expire: 

def get_credentials() -> S3Credential:
    return {
        "access_key_id": "...",
        "secret_access_key": "...",
        "token": None,
        "expires_at": None,
    }

Return static credentials that are valid for 5 minutes: 

from datetime import datetime, timedelta, UTC

async def get_credentials() -> S3Credential:
    return {
        "access_key_id": "...",
        "secret_access_key": "...",
        "token": None,
        "expires_at": datetime.now(UTC) + timedelta(minutes=5),
    }

A class-based credential provider with state:

from __future__ import annotations

from typing import TYPE_CHECKING

import boto3
import botocore.credentials

if TYPE_CHECKING:
    from obstore.store import S3Credential


class Boto3CredentialProvider:
    credentials: botocore.credentials.Credentials

    def __init__(self, session: boto3.session.Session) -> None:
        credentials = session.get_credentials()
        if credentials is None:
            raise ValueError("Received None from session.get_credentials")

        self.credentials = credentials

    def __call__(self) -> S3Credential:
        frozen_credentials = self.credentials.get_frozen_credentials()
        return {
            "access_key_id": frozen_credentials.access_key,
            "secret_access_key": frozen_credentials.secret_key,
            "token": frozen_credentials.token,
            "expires_at": None,
        }

Not importable at runtime

To use this type hint in your code, import it within a TYPE_CHECKING block:

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from obstore.store import S3CredentialProvider

__call__ staticmethod 

__call__() -> S3Credential | Coroutine[Any, Any, S3Credential]

Return an S3Credential.