AWS S3¶
obstore.store.S3Store ¶
Configure a connection to Amazon S3 using the specified credentials in the specified Amazon region and bucket.
Examples:
Using requester-pays buckets:
Pass request_payer=True
as a keyword argument. Or, if you're using
S3Store.from_env
, have AWS_REQUESTER_PAYS=True
set in the environment.
Anonymous requests:
Pass skip_signature=True
as a keyword argument. Or, if you're using
S3Store.from_env
, have AWS_SKIP_SIGNATURE=True
set in the environment.
__init__ ¶
__init__(
bucket: str,
*,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
**kwargs: Unpack[S3Config],
) -> None
Create a new S3Store
Parameters:
-
bucket
(str
) –The AWS bucket to use.
Other Parameters:
-
config
(S3Config | None
) –AWS Configuration. Values in this config will override values inferred from the environment. Defaults to None.
-
client_options
(ClientConfig | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
None
–S3Store
from_env
classmethod
¶
from_env(
bucket: str | None = None,
*,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
**kwargs: Unpack[S3Config],
) -> S3Store
Construct a new S3Store with regular AWS environment variables
All environment variables starting with AWS_
will be evaluated. Names must
match items from S3ConfigKey
. Only upper-case environment variables are
accepted.
Some examples of variables extracted from environment:
AWS_ACCESS_KEY_ID
-> access_key_idAWS_SECRET_ACCESS_KEY
-> secret_access_keyAWS_DEFAULT_REGION
-> regionAWS_ENDPOINT
-> endpointAWS_SESSION_TOKEN
-> tokenAWS_CONTAINER_CREDENTIALS_RELATIVE_URI
-> docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.htmlAWS_REQUEST_PAYER
-> set to "true" to permit requester-pays connections.
Parameters:
-
bucket
(str | None
, default:None
) –The AWS bucket to use.
Other Parameters:
-
config
(S3Config | None
) –AWS Configuration. Values in this config will override values inferred from the environment. Defaults to None.
-
client_options
(ClientConfig | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
from_session
classmethod
¶
from_session(
session: Session | Session,
bucket: str,
*,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
**kwargs: Unpack[S3Config],
) -> S3Store
Construct a new S3Store with credentials inferred from a boto3 Session
This can be useful to read S3 credentials from disk-based credentials sources.
Note
This is a convenience function for users who are already using boto3
or
botocore
. If you're not already using boto3
or botocore
, use other
constructors, which do not need boto3
or botocore
to be installed.
Examples:
import boto3
session = boto3.Session()
store = S3Store.from_session(session, "bucket-name", region="us-east-1")
Parameters:
-
session
(Session | Session
) –The boto3.Session or botocore.session.Session to infer credentials from.
-
bucket
(str
) –The AWS bucket to use.
Other Parameters:
-
config
(S3Config | None
) –AWS Configuration. Values in this config will override values inferred from the session. Defaults to None.
-
client_options
(ClientConfig | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
from_url
classmethod
¶
from_url(
url: str,
*,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
**kwargs: Unpack[S3Config],
) -> S3Store
Parse available connection info from a well-known storage URL.
The supported url schemes are:
s3://<bucket>/<path>
s3a://<bucket>/<path>
https://s3.<region>.amazonaws.com/<bucket>
https://<bucket>.s3.<region>.amazonaws.com
https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket
Note
Note that from_url
will not use any additional parts of the path as a
bucket prefix. It will only extract the bucket, region, and endpoint. If you
wish to use a path prefix, consider wrapping this with PrefixStore
.
Parameters:
-
url
(str
) –well-known storage URL.
Other Parameters:
-
config
(S3Config | None
) –AWS Configuration. Values in this config will override values inferred from the url. Defaults to None.
-
client_options
(ClientConfig | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
obstore.store.S3Config ¶
Bases: TypedDict
Configuration parameters for S3Store.
There are duplicates of many parameters, and parameters can be either upper or lower case. Not all parameters are required.
AWS_CHECKSUM_ALGORITHM
instance-attribute
¶
AWS_CHECKSUM_ALGORITHM: str
Sets the checksum algorithm which has to be used for object integrity check during upload.
AWS_CONDITIONAL_PUT
instance-attribute
¶
AWS_CONDITIONAL_PUT: str
Configure how to provide conditional put support
Supported values:
-
"etag"
: Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headersIf-Match
andIf-None-Match
. -
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
: The name of a DynamoDB table to use for coordination.This will use the same region, credentials and endpoint as configured for S3.
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
instance-attribute
¶
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: str
Set the container credentials relative URI
docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
AWS_COPY_IF_NOT_EXISTS
instance-attribute
¶
AWS_COPY_IF_NOT_EXISTS: str
Configure how to provide "copy if not exists".
Supported values:
-
"multipart"
:Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.
Warning
When using this mode,
copy_if_not_exists
does not copy tags or attributes from the source object.Warning
When using this mode,
copy_if_not_exists
makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads. -
"header:<HEADER_NAME>:<HEADER_VALUE>"
:Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.
If set,
copy_if_not_exists
will perform a normal copy operation with the provided header pair, and expect the store to fail with412 Precondition Failed
if the destination file already exists.For example
header: cf-copy-destination-if-none-match: *
, would set the headercf-copy-destination-if-none-match
to*
. -
"header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>"
:The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.
-
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
:The name of a DynamoDB table to use for coordination.
The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.
AWS_DISABLE_TAGGING
instance-attribute
¶
AWS_DISABLE_TAGGING: bool
Disable tagging objects. This can be desirable if not supported by the backing store.
AWS_ENDPOINT
instance-attribute
¶
AWS_ENDPOINT: str
Sets custom endpoint for communicating with AWS S3.
AWS_ENDPOINT_URL
instance-attribute
¶
AWS_ENDPOINT_URL: str
Sets custom endpoint for communicating with AWS S3.
AWS_METADATA_ENDPOINT
instance-attribute
¶
AWS_METADATA_ENDPOINT: str
Set the instance metadata endpoint
AWS_REQUEST_PAYER
instance-attribute
¶
AWS_REQUEST_PAYER: bool
If True
, enable operations on requester-pays buckets.
AWS_SERVER_SIDE_ENCRYPTION
instance-attribute
¶
AWS_SERVER_SIDE_ENCRYPTION: str
Type of encryption to use.
If set, must be one of:
"AES256"
(SSE-S3)"aws:kms"
(SSE-KMS)"aws:kms:dsse"
(DSSE-KMS)"sse-c"
AWS_SESSION_TOKEN
instance-attribute
¶
AWS_SESSION_TOKEN: str
Token to use for requests (passed to underlying provider)
AWS_SKIP_SIGNATURE
instance-attribute
¶
AWS_SKIP_SIGNATURE: bool
If True
, S3Store will not fetch credentials and will not sign requests.
AWS_SSE_BUCKET_KEY_ENABLED
instance-attribute
¶
AWS_SSE_BUCKET_KEY_ENABLED: bool
If set to True
, will use the bucket's default KMS key for server-side encryption.
If set to False
, will disable the use of the bucket's default KMS key for server-side encryption.
AWS_SSE_CUSTOMER_KEY_BASE64
instance-attribute
¶
AWS_SSE_CUSTOMER_KEY_BASE64: str
The base64 encoded, 256-bit customer encryption key to use for server-side
encryption. If set, the server side encryption config value must be "sse-c"
.
AWS_SSE_KMS_KEY_ID
instance-attribute
¶
AWS_SSE_KMS_KEY_ID: str
The KMS key ID to use for server-side encryption.
If set, the server side encrypting config value must be "aws:kms"
or "aws:kms:dsse"
.
AWS_TOKEN
instance-attribute
¶
AWS_TOKEN: str
Token to use for requests (passed to underlying provider)
AWS_UNSIGNED_PAYLOAD
instance-attribute
¶
AWS_UNSIGNED_PAYLOAD: bool
Avoid computing payload checksum when calculating signature.
AWS_VIRTUAL_HOSTED_STYLE_REQUEST
instance-attribute
¶
AWS_VIRTUAL_HOSTED_STYLE_REQUEST: bool
If virtual hosted style request has to be used.
CHECKSUM_ALGORITHM
instance-attribute
¶
CHECKSUM_ALGORITHM: str
Sets the checksum algorithm which has to be used for object integrity check during upload.
CONDITIONAL_PUT
instance-attribute
¶
CONDITIONAL_PUT: str
Configure how to provide conditional put support
Supported values:
-
"etag"
: Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headersIf-Match
andIf-None-Match
. -
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
: The name of a DynamoDB table to use for coordination.This will use the same region, credentials and endpoint as configured for S3.
COPY_IF_NOT_EXISTS
instance-attribute
¶
COPY_IF_NOT_EXISTS: str
Configure how to provide "copy if not exists".
Supported values:
-
"multipart"
:Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.
Warning
When using this mode,
copy_if_not_exists
does not copy tags or attributes from the source object.Warning
When using this mode,
copy_if_not_exists
makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads. -
"header:<HEADER_NAME>:<HEADER_VALUE>"
:Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.
If set,
copy_if_not_exists
will perform a normal copy operation with the provided header pair, and expect the store to fail with412 Precondition Failed
if the destination file already exists.For example
header: cf-copy-destination-if-none-match: *
, would set the headercf-copy-destination-if-none-match
to*
. -
"header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>"
:The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.
-
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
:The name of a DynamoDB table to use for coordination.
The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.
DISABLE_TAGGING
instance-attribute
¶
DISABLE_TAGGING: bool
Disable tagging objects. This can be desirable if not supported by the backing store.
ENDPOINT_URL
instance-attribute
¶
ENDPOINT_URL: str
Sets custom endpoint for communicating with AWS S3.
REQUEST_PAYER
instance-attribute
¶
REQUEST_PAYER: bool
If True
, enable operations on requester-pays buckets.
SESSION_TOKEN
instance-attribute
¶
SESSION_TOKEN: str
Token to use for requests (passed to underlying provider)
SKIP_SIGNATURE
instance-attribute
¶
SKIP_SIGNATURE: bool
If True
, S3Store will not fetch credentials and will not sign requests.
UNSIGNED_PAYLOAD
instance-attribute
¶
UNSIGNED_PAYLOAD: bool
Avoid computing payload checksum when calculating signature.
VIRTUAL_HOSTED_STYLE_REQUEST
instance-attribute
¶
VIRTUAL_HOSTED_STYLE_REQUEST: bool
If virtual hosted style request has to be used.
aws_checksum_algorithm
instance-attribute
¶
aws_checksum_algorithm: str
Sets the checksum algorithm which has to be used for object integrity check during upload.
aws_conditional_put
instance-attribute
¶
aws_conditional_put: str
Configure how to provide conditional put support
Supported values:
-
"etag"
: Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headersIf-Match
andIf-None-Match
. -
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
: The name of a DynamoDB table to use for coordination.This will use the same region, credentials and endpoint as configured for S3.
aws_container_credentials_relative_uri
instance-attribute
¶
aws_container_credentials_relative_uri: str
Set the container credentials relative URI
docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
aws_copy_if_not_exists
instance-attribute
¶
aws_copy_if_not_exists: str
Configure how to provide "copy if not exists".
Supported values:
-
"multipart"
:Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.
Warning
When using this mode,
copy_if_not_exists
does not copy tags or attributes from the source object.Warning
When using this mode,
copy_if_not_exists
makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads. -
"header:<HEADER_NAME>:<HEADER_VALUE>"
:Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.
If set,
copy_if_not_exists
will perform a normal copy operation with the provided header pair, and expect the store to fail with412 Precondition Failed
if the destination file already exists.For example
header: cf-copy-destination-if-none-match: *
, would set the headercf-copy-destination-if-none-match
to*
. -
"header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>"
:The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.
-
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
:The name of a DynamoDB table to use for coordination.
The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.
aws_disable_tagging
instance-attribute
¶
aws_disable_tagging: bool
Disable tagging objects. This can be desirable if not supported by the backing store.
aws_endpoint
instance-attribute
¶
aws_endpoint: str
Sets custom endpoint for communicating with AWS S3.
aws_endpoint_url
instance-attribute
¶
aws_endpoint_url: str
Sets custom endpoint for communicating with AWS S3.
aws_metadata_endpoint
instance-attribute
¶
aws_metadata_endpoint: str
Set the instance metadata endpoint
aws_request_payer
instance-attribute
¶
aws_request_payer: bool
If True
, enable operations on requester-pays buckets.
aws_server_side_encryption
instance-attribute
¶
aws_server_side_encryption: str
Type of encryption to use.
If set, must be one of:
"AES256"
(SSE-S3)"aws:kms"
(SSE-KMS)"aws:kms:dsse"
(DSSE-KMS)"sse-c"
aws_session_token
instance-attribute
¶
aws_session_token: str
Token to use for requests (passed to underlying provider)
aws_skip_signature
instance-attribute
¶
aws_skip_signature: bool
If True
, S3Store will not fetch credentials and will not sign requests.
aws_sse_bucket_key_enabled
instance-attribute
¶
aws_sse_bucket_key_enabled: bool
If set to True
, will use the bucket's default KMS key for server-side encryption.
If set to False
, will disable the use of the bucket's default KMS key for server-side encryption.
aws_sse_customer_key_base64
instance-attribute
¶
aws_sse_customer_key_base64: str
The base64 encoded, 256-bit customer encryption key to use for server-side
encryption. If set, the server side encryption config value must be "sse-c"
.
aws_sse_kms_key_id
instance-attribute
¶
aws_sse_kms_key_id: str
The KMS key ID to use for server-side encryption.
If set, the server side encryption config value must be "aws:kms"
or "aws:kms:dsse"
.
aws_token
instance-attribute
¶
aws_token: str
Token to use for requests (passed to underlying provider)
aws_unsigned_payload
instance-attribute
¶
aws_unsigned_payload: bool
Avoid computing payload checksum when calculating signature.
aws_virtual_hosted_style_request
instance-attribute
¶
aws_virtual_hosted_style_request: bool
If virtual hosted style request has to be used.
checksum_algorithm
instance-attribute
¶
checksum_algorithm: str
Sets the checksum algorithm which has to be used for object integrity check during upload.
conditional_put
instance-attribute
¶
conditional_put: str
Configure how to provide conditional put support
Supported values:
-
"etag"
: Supported for S3-compatible stores that support conditional put using the standard HTTP precondition headersIf-Match
andIf-None-Match
. -
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
: The name of a DynamoDB table to use for coordination.This will use the same region, credentials and endpoint as configured for S3.
copy_if_not_exists
instance-attribute
¶
copy_if_not_exists: str
Configure how to provide "copy if not exists".
Supported values:
-
"multipart"
:Native Amazon S3 supports copy if not exists through a multipart upload where the upload copies an existing object and is completed only if the new object does not already exist.
Warning
When using this mode,
copy_if_not_exists
does not copy tags or attributes from the source object.Warning
When using this mode,
copy_if_not_exists
makes only a best effort attempt to clean up the multipart upload if the copy operation fails. Consider using a lifecycle rule to automatically clean up abandoned multipart uploads. -
"header:<HEADER_NAME>:<HEADER_VALUE>"
:Some S3-compatible stores, such as Cloudflare R2, support copy if not exists semantics through custom headers.
If set,
copy_if_not_exists
will perform a normal copy operation with the provided header pair, and expect the store to fail with412 Precondition Failed
if the destination file already exists.For example
header: cf-copy-destination-if-none-match: *
, would set the headercf-copy-destination-if-none-match
to*
. -
"header-with-status:<HEADER_NAME>:<HEADER_VALUE>:<STATUS>"
:The same as the header variant above but allows custom status code checking, for object stores that return values other than 412.
-
"dynamo:<TABLE_NAME>"
or"dynamo:<TABLE_NAME>:<TIMEOUT_MILLIS>"
:The name of a DynamoDB table to use for coordination.
The default timeout is used if not specified. This will use the same region, credentials and endpoint as configured for S3.
disable_tagging
instance-attribute
¶
disable_tagging: bool
Disable tagging objects. This can be desirable if not supported by the backing store.
endpoint_url
instance-attribute
¶
endpoint_url: str
Sets custom endpoint for communicating with AWS S3.
request_payer
instance-attribute
¶
request_payer: bool
If True
, enable operations on requester-pays buckets.
session_token
instance-attribute
¶
session_token: str
Token to use for requests (passed to underlying provider)
skip_signature
instance-attribute
¶
skip_signature: bool
If True
, S3Store will not fetch credentials and will not sign requests.