CachingFileSystem implicitly implies that the file object of the wrapped fs uses the property `.cache`

[ZhengKeli]

I found that the current implementation of CachingFileSystem implicitly requires that the file object of wrapped fs uses the property .cache.

https://github.com/fsspec/filesystem_spec/blob/f84b99f0d1f079f990db1a219b74df66ab3e7160/fsspec/implementations/cached.py#L388-L390

However, not all implementation uses the property .cache. I am currently facing an issue that the file object of WebdavFileSystem (from https://github.com/skshetry/webdav4) does overrides method .read() ignoring .cache. Because of that, nothing is cached when I wrap a WebdavFileSystem object with CachingFileSystem.

webdav4/src/webdav4/fsspec.py

Lines 428 to 433 in e9b8fbd

	def read(self, length: int = -1) -> Union[str, bytes, None]:
	"""Read chunk of bytes."""
	chunk = self.reader.read(length)
	if chunk:
	self.loc += len(chunk)
	return chunk

In my opinion, a wrapper like CachingFileSystem should not imply how the wrapped object works. Perhaps a more general solution should be implemented here?

[hexxawax]

I've encountered this issue as well. I'm currently looking for a temporary local workaround.
I agree that a long-term solution is preferable. Implementing caching by inheriting from fsspec.AbstractBufferedFile, similar to how it's done in s3fs and gcsfs, seems like a possible solution.

[hexxawax]

Here is my temporary workaround by creating a custom class that explicitly handles the caching. It inherits from WebdavFileSystem and overrides the _open method to manage a local file cache. This solution works correctly and is memory-safe for large files.

Here is the implementation we're using:

"""
This custom class provides a robust and cached file system for WebDAV
storage, which significantly speeds up subsequent access to the same files.
It is a temporary workaround for a known bug in the `webdav4` library
(see https://github.com/skshetry/webdav4/issues/209).
"""
import os
import fsspec
from fsspec.implementations.local import LocalFileSystem
from webdav4.fsspec import WebdavFileSystem
from typing import Any, cast, Optional
import time

# Create a local cache directory
CACHE_DIR = os.path.expanduser("~/.fsspec-cache")
os.makedirs(CACHE_DIR, exist_ok=True)

class CachedWebdavFileSystem(WebdavFileSystem):
    """
    A custom WebDAV file system that adds an explicit cache with expiration.
    It overrides the `_open` method to serve a local version if it's
    cached and not expired.
    """

    def __init__(self, *args, cache_expiry_seconds: Optional[int] = 86400, **kwargs):
        super().__init__(*args, **kwargs)
        self.local_fs = LocalFileSystem()
        self.cache_expiry_seconds = cache_expiry_seconds

    def _open(self, path, mode='rb', block_size=None, autocommit=True, **kwargs):
        """
        Overrides the open method to manage the cache.
        If the file is already in the local cache and not expired, it reads
        it locally. Otherwise, it downloads the file, caches it, and then
        returns it.
        """
        if 'r' not in mode or 'b' not in mode:
            raise NotImplementedError("Only binary read mode ('rb') is supported for the cache.")

        # Create a simple filename for the local cache
        local_file_name = path.replace('/', '_').replace('..', '')
        local_path = os.path.join(CACHE_DIR, local_file_name)

        # Check for expired cache
        cache_expired = False
        if self.local_fs.exists(local_path):
            file_mod_time = os.path.getmtime(local_path)
            # Check if cache expiry is set and the file is older than the expiry time
            if self.cache_expiry_seconds is not None and (time.time() - file_mod_time) > self.cache_expiry_seconds:
                self.local_fs.rm(local_path)
                cache_expired = True

        if self.local_fs.exists(local_path) and not cache_expired:
            return self.local_fs.open(local_path, mode)
        else:
            temp_path = local_path + ".downloading"
            
            # Read and write in chunks to avoid memory issues with large files
            try:
                with super()._open(path, 'rb', block_size=block_size, autocommit=autocommit, **kwargs) as remote_file:
                    with open(temp_path, 'wb') as local_temp_file:
                        chunk_size = 10 * 1024 * 1024  # 10 MB
                        while True:
                            chunk = remote_file.read(chunk_size)
                            if not chunk:
                                break
                            local_temp_file.write(chunk)
                
                # Rename the temporary file once the download is complete
                os.rename(temp_path, local_path)
                return self.local_fs.open(local_path, mode)
            except Exception as e:
                # Remove the temporary file in case of an error
                if os.path.exists(temp_path):
                    os.remove(temp_path)
                raise e

Hopefully, this temporary fix can help others who encounter the same issue in the meantime.