create multiple separate memory filesystems

[milahu]

fsspec/implementations/memory.py

class MemoryFileSystem(AbstractFileSystem):
    """A filesystem based on a dict of BytesIO objects

    This is a global filesystem so instances of this class all point to the same
    in memory filesystem.
    """

    store: ClassVar[dict[str, Any]] = {}  # global, do not overwrite!
    pseudo_dirs = [""]  # global, do not overwrite!

... ok, but i need multiple different memory filesystems in parallel

assert (
    fsspec.filesystem("memory://").store !=
    fsspec.filesystem("memory://").store
)

this should be as simple as moving store and pseudo_dirs to def __init__

maybe add a new protocol localmemory

[martindurant]

This is a reasonable idea and something I thought of doing in the past. The reason it hasn't happened yet, is that MemoryFS is used extensively in tests - in fact, that is the main use case - and it if often useful not to have to carry variable handles around.

Note that:

assert (
    fsspec.filesystem("memory").store !=
    fsspec.filesystem("memory").store
)

would only be true if the filesystem is not cachable, otherwise you get the identical python object: fsspec.filesystem("memory") is fsspec.filesystem("memory").

fsspec.get_filesystem_class("memory").cachable is True

[martindurant]

In short: if you can make the change and no tests fail (or can be easily fixed), let's do it. Note that the documentation will need to be updated.