forked from google-research/t5x
-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
Description
Description
For zero-shot evaluation, we would ideally want to use the EAI evaluation harness. Two strategies are possible: (1) convert T5X Jax checkpoints to HF Transformers PyTorch checkpoints, or (2) connect the EAI harness to T5X.
We are currently thinking of going with (1), as it seems like an easier choice.
Action items
- Write a converter from T5X checkpoints to HF Transformers;
- Reproduce evaluation results obtained with SeqIO (see Setup zero-shot evaluation in SeqIO #3).