Cache API responses

[jwodder]

Responses from the Dandi Archive API and S3 should be cached for a period of time in order to speed up browsing.

• The Archive API does not use ETags, so the caching cannot be done at the HTTP level. The best option may be to add a cache in the client that maps API request paths to the deserialized objects they return (and likewise for S3).

• Should there be some mechanism (e.g., a purge=1 query parameter) for a user browsing dandidav to force cache invalidation for a request?

[yarikoptic]

If we assume that dandisets do not change more frequently than once in a minute we can "compute" our own ETag based on dandiset version modification time + path/query parameters. That should then potentially provide solution for #189

[jwodder]

@yarikoptic If we're implementing caching for Archive responses, should the caching mechanism have the same logging, size-limiting, & expiration features as implemented for the Zarr manifest cache in #170?

[jwodder]

@yarikoptic Also, assuming caching covers all Archive requests and not just lists of assets in Dandisets, how should cache expiration be handled for requests that don't operate on a Dandiset version, i.e.:

• /dandisets/
• /dandisets/{dandiset_id}/
• /dandisets/{dandiset_id}/versions/

Note that, for the paths containing Dandiset IDs, we can't use the Dandiset-wide modified for the etag, as it can be earlier than the modified of the draft version (See Dandiset 000027 for an example).

[jwodder]

@yarikoptic Ping.

[yarikoptic]

@yarikoptic If we're implementing caching for Archive responses, should the caching mechanism have the same logging, size-limiting, & expiration features as implemented for the Zarr manifest cache in #170?

sounds ok to me to resort to the "same" as we do not know better ATM

@yarikoptic Also, assuming caching covers all Archive requests and not just lists of assets in Dandisets, how should cache expiration be handled for requests that don't operate on a Dandiset version, i.e.:

• /dandisets/
• /dandisets/{dandiset_id}/
• /dandisets/{dandiset_id}/versions/

IMHO doing time based alone (e.g. expire in 5 minutes) should be good enough unless forced somehow (e.g. via explicit query option e.g. ?force_refresh if possible; otherwise check if there is some more standard query option for webdav). I would not expect them to change frequently.