Has anyone tried deploying Flash attention v2 with a BERT model on the NVIDIA Triton Inference Server?

[loredunk]

I'm not sure how to enable BERT with flash attention during the start-up of the Triton server in order to accelerate inference.

[MatthieuToulemont]

It's probably not what you are looking for, but compiling your model to TensorRT will use Flash Attention.
Another solution if you use BLS model to run your model in python is to install torch>2 (assuming you use torch) and use scaled_dot_product_attention