Skip to content

Deal with hidden dask usage #300

@dfulu

Description

@dfulu

We have moved to primarily use xarray-tensorstore as a backend, but in some places it is possible for dask to creep back in. I recently had this where there was an extra unused non-dimensional coordinate in our ECMWF data. This coordinate wa maintained up to the compute method and since it uses dask, even though the coordinate is small and has very few chunks, it slowed down my sampler by around 20-30%.

This is something that could keep coming up.

We should perhaps:

  • Remove non-dimensional coordinates in the load functions?
  • Add a warning to the user if any datasets are dask arrays

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions