Efficiently Serving Many Transformer Adapters

[smellslikeml]

Transformer adapters are used for efficient fine-tuning, can we optimize tritonserver for serving inference for many adapters in a way which efficiently shares memory for models?

[smellslikeml]

Recently found Batched LoRAs

For those interested in running with Triton, here is an implementation

[smellslikeml]

Now we have S-LoRA
https://github.com/S-LoRA/S-LoRA