Skip to content

Setup zero-shot evaluation with EAI harness #4

@slippylolo

Description

@slippylolo

Description

For zero-shot evaluation, we would ideally want to use the EAI evaluation harness. Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or (2) connect the EAI harness to T5X.

We are currently thinking of going with (1), as it seems like an easier choice.

Action items

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions