Skip to content

Defining a large dataset with several splits under it #199

@MaramHasanain

Description

@MaramHasanain

We have a recurrent format of some datasets where the same dataset will have multiple splits under each, where splits are different by language, subtask, train-dev-test, etc. but have the same file structure.
Our current implementation assumes we will have one dataset per split (especially with the metadata specifying dataset language for example), OR we will have an ad-hoc method of using the same dataset class but passing different splits file names with different assets.

I think we should find a more unified way to handle such cases (e.g., parent dataset and subsets under each, where subsets are different by metadata only for example).

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions