Getting a generator or iterator instead of lists of OpenFile objects or addresses.

[MalteEbner]

Functions like fsspec.open_files or FileSystem.ls return list-like objects when run on directories or with glob patterns. This has two main drawbacks:

• The functions only return once the entire directory has been listed. When listing cloud buckets with millions of entries, this can take many minutes. It leads to:
  • Higher failure risk due to long runtimes.
  • No way to add user feedback in the meantime, e.g. a progress bar.
  • Can't start processing the first files found while the rest are still being listed.
• All OpenFile objects or addresses are kept in memory at once.

Is it possible to get a generator or iterator instead of a list? I'm particularly interested in support for local, s3fs, and gcsfs.

[martindurant]

This has been discussed before, and some (internal/hidden) methods do allow it, in particular S3FileSystem._iterdir().
I agree that such a mechanism would be useful and that lists in general are not a perfect solution. However, list have the advantage of being simple! Exposing a generator would break/change the current API and cause implicit crossing of the sync/async barrier during iteration. So it would need to be implemented with care and perhaps with a different, explicit, method.

cc @kyleknap - I don't know offhand if abfs exposes a (async) generator for listings, but I thought you might find the idea interesting.

[kyleknap]

@martindurant yeah having some public interface for returning an iterator of items, instead of a list, back would definitely be useful. As @MalteEbner highlighted, especially for cloud storage, being able to process items as they are listed from the API becomes necessary for scaling when a large number of entries are present. Processing entries a page at a time is also an encouraged best practice in cloud SDKs so that your apps can properly scale.

That being said, I'd be hesitant to break the current API to default to iterators because of the amount of churn I'd suspect it would cause in both implementations and fsspec consumers. I like the idea of having separate methods (e.g. iter_ls()/iter_glob(), or ilist()/iglob()). If we were worried about polluting the method namespace, we could also consider adding an iter parameter to return an iterator instead of list (e.g. ls(iter=True)). Thoughts?

From the SDK perspective (at least for Azure Blob Storage), working in generators/iterators is not an issue; the default return type for list methods are iterators (and async iterators for the async methods). I think the more interesting part as you mentioned @martindurant is how it would look when implementing across the async/sync barrier. I'm not familiar enough with fsspec/adlfs yet to know off hand.