AWS S3¶
obstore.store.S3Store ¶
Configure a connection to Amazon S3 using the specified credentials in the specified Amazon region and bucket.
from_env
classmethod
¶
from_env(
bucket: str,
*,
config: Dict[S3ConfigKey | str, str] | None = None,
client_options: Dict[ClientConfigKey, str | bool] | None = None,
retry_config: RetryConfig | None = None
) -> S3Store
Construct a new S3Store with regular AWS environment variables
Variables extracted from environment:
AWS_ACCESS_KEY_ID
-> access_key_idAWS_SECRET_ACCESS_KEY
-> secret_access_keyAWS_DEFAULT_REGION
-> regionAWS_ENDPOINT
-> endpointAWS_SESSION_TOKEN
-> tokenAWS_CONTAINER_CREDENTIALS_RELATIVE_URI
-> docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.htmlAWS_ALLOW_HTTP
-> set to "true" to permit HTTP connections without TLS
Parameters:
-
bucket
(str
) –The AWS bucket to use.
Other Parameters:
-
config
(Dict[S3ConfigKey | str, str] | None
) –AWS Configuration. Values in this config will override values inferred from the environment. Defaults to None.
-
client_options
(Dict[ClientConfigKey, str | bool] | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
from_session
classmethod
¶
from_session(
session: Session | Session,
bucket: str,
*,
config: Dict[S3ConfigKey | str, str] | None = None,
client_options: Dict[ClientConfigKey, str | bool] | None = None,
retry_config: RetryConfig | None = None
) -> S3Store
Construct a new S3Store with credentials inferred from a boto3 Session
Parameters:
-
session
(Session | Session
) –The boto3.Session or botocore.session.Session to infer credentials from.
-
bucket
(str
) –The AWS bucket to use.
Other Parameters:
-
config
(Dict[S3ConfigKey | str, str] | None
) –AWS Configuration. Values in this config will override values inferred from the session. Defaults to None.
-
client_options
(Dict[ClientConfigKey, str | bool] | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
from_url
classmethod
¶
from_url(
url: str,
*,
config: Dict[S3ConfigKey | str, str] | None = None,
client_options: Dict[ClientConfigKey, str | bool] | None = None,
retry_config: RetryConfig | None = None
) -> S3Store
Parse available connection info from a well-known storage URL.
The supported url schemes are:
s3://<bucket>/<path>
s3a://<bucket>/<path>
https://s3.<region>.amazonaws.com/<bucket>
https://<bucket>.s3.<region>.amazonaws.com
https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket
Parameters:
-
url
(str
) –well-known storage URL.
Other Parameters:
-
config
(Dict[S3ConfigKey | str, str] | None
) –AWS Configuration. Values in this config will override values inferred from the url. Defaults to None.
-
client_options
(Dict[ClientConfigKey, str | bool] | None
) –HTTP Client options. Defaults to None.
-
retry_config
(RetryConfig | None
) –Retry configuration. Defaults to None.
Returns:
-
S3Store
–S3Store
obstore.store.S3ConfigKey
module-attribute
¶
S3ConfigKey = Literal[
"access_key_id",
"aws_access_key_id",
"aws_allow_http",
"aws_bucket_name",
"aws_bucket",
"aws_checksum_algorithm",
"aws_conditional_put",
"aws_container_credentials_relative_uri",
"aws_copy_if_not_exists",
"aws_default_region",
"aws_disable_tagging",
"aws_endpoint_url",
"aws_endpoint",
"aws_imdsv1_fallback",
"aws_metadata_endpoint",
"aws_region",
"aws_s3_express",
"aws_secret_access_key",
"aws_server_side_encryption",
"aws_session_token",
"aws_skip_signature",
"aws_sse_bucket_key_enabled",
"aws_sse_kms_key_id",
"aws_token",
"aws_unsigned_payload",
"aws_virtual_hosted_style_request",
"bucket_name",
"bucket",
"checksum_algorithm",
"conditional_put",
"copy_if_not_exists",
"default_region",
"disable_tagging",
"endpoint_url",
"endpoint",
"imdsv1_fallback",
"metadata_endpoint",
"region",
"s3_express",
"secret_access_key",
"session_token",
"skip_signature",
"token",
"unsigned_payload",
"virtual_hosted_style_request",
"ACCESS_KEY_ID",
"AWS_ACCESS_KEY_ID",
"AWS_ALLOW_HTTP",
"AWS_BUCKET_NAME",
"AWS_BUCKET",
"AWS_CHECKSUM_ALGORITHM",
"AWS_CONDITIONAL_PUT",
"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI",
"AWS_COPY_IF_NOT_EXISTS",
"AWS_DEFAULT_REGION",
"AWS_DISABLE_TAGGING",
"AWS_ENDPOINT_URL",
"AWS_ENDPOINT",
"AWS_IMDSV1_FALLBACK",
"AWS_METADATA_ENDPOINT",
"AWS_REGION",
"AWS_S3_EXPRESS",
"AWS_SECRET_ACCESS_KEY",
"AWS_SERVER_SIDE_ENCRYPTION",
"AWS_SESSION_TOKEN",
"AWS_SKIP_SIGNATURE",
"AWS_SSE_BUCKET_KEY_ENABLED",
"AWS_SSE_KMS_KEY_ID",
"AWS_TOKEN",
"AWS_UNSIGNED_PAYLOAD",
"AWS_VIRTUAL_HOSTED_STYLE_REQUEST",
"BUCKET_NAME",
"BUCKET",
"CHECKSUM_ALGORITHM",
"CONDITIONAL_PUT",
"COPY_IF_NOT_EXISTS",
"DEFAULT_REGION",
"DISABLE_TAGGING",
"ENDPOINT_URL",
"ENDPOINT",
"IMDSV1_FALLBACK",
"METADATA_ENDPOINT",
"REGION",
"S3_EXPRESS",
"SECRET_ACCESS_KEY",
"SESSION_TOKEN",
"SKIP_SIGNATURE",
"TOKEN",
"UNSIGNED_PAYLOAD",
"VIRTUAL_HOSTED_STYLE_REQUEST",
]
Valid AWS S3 configuration keys.
Either lower case or upper case strings are accepted.
aws_access_key_id
,access_key_id
: AWS Access Keyaws_secret_access_key
,secret_access_key
: Secret Access Keyaws_region
,region
: Regionaws_default_region
,default_region
: Default regionaws_bucket
,aws_bucket_name
,bucket
,bucket_name
: Bucket nameaws_endpoint
,aws_endpoint_url
,endpoint
,endpoint_url
: Sets custom endpoint for communicating with AWS S3.aws_session_token
,aws_token
,session_token
,token
: Token to use for requests (passed to underlying provider)aws_imdsv1_fallback
,imdsv1_fallback
: Fall back to ImdsV1aws_virtual_hosted_style_request
,virtual_hosted_style_request
: If virtual hosted style request has to be usedaws_unsigned_payload
,unsigned_payload
: Avoid computing payload checksum when calculating signature.aws_metadata_endpoint
,metadata_endpoint
: Set the instance metadata endpointaws_disable_tagging
,disable_tagging
: Disable tagging objects. This can be desirable if not supported by the backing store.aws_s3_express
,s3_express
: Enable Support for S3 Express One Zone