Skip to content

obstore.fsspec

obstore.fsspec

fsspec integration.

The underlying object_store Rust crate cautions against relying too strongly on stateful filesystem representations of object stores:

The ObjectStore interface is designed to mirror the APIs of object stores and not filesystems, and thus has stateless APIs instead of cursor based interfaces such as Read or Seek available in filesystems.

This design provides the following advantages:

  • All operations are atomic, and readers cannot observe partial and/or failed writes
  • Methods map directly to object store APIs, providing both efficiency and predictability
  • Abstracts away filesystem and operating system specific quirks, ensuring portability
  • Allows for functionality not native to filesystems, such as operation preconditions and atomic multipart uploads

Where possible, implementations should use the underlying obstore APIs directly. Only where this is not possible should users fall back to this fsspec integration.

AsyncFsspecStore

Bases: AsyncFileSystem

An fsspec implementation based on a obstore Store.

You should be able to pass an instance of this class into any API that expects an fsspec-style object.

__init__ 

__init__(
    store: ObjectStore,
    *args,
    asynchronous: bool = False,
    loop=None,
    batch_size: int | None = None
)

Construct a new AsyncFsspecStore

Parameters:

  • store (ObjectStore) –

    a configured instance of one of the store classes in obstore.store.

  • asynchronous (bool, default: False ) –

    Set to True if this instance is meant to be be called using the fsspec async API. This should only be set to true when running within a coroutine.

  • loop –

    since both fsspec/python and tokio/rust may be using loops, this should be kept None for now, and will not be used.

  • batch_size (int | None, default: None ) –

    some operations on many files will batch their requests; if you are seeing timeouts, you may want to set this number smaller than the defaults, which are determined in fsspec.asyn._get_batch_size.

Example:

from obstore.fsspec import AsyncFsspecStore
from obstore.store import HTTPStore

store = HTTPStore.from_url("https://example.com")
fsspec_store = AsyncFsspecStore(store)
resp = fsspec_store.cat("/")
assert resp.startswith(b"<!doctype html>")

BufferedFileSimple

Bases: AbstractBufferedFile

read 

read(length: int = -1)

Return bytes from the remote file

if positive, returns up to this many bytes; if negative, return all

remaining byets.