Skip to content

Encoder-Decoder example doesn't actually use encoder? #786

@srinath2022

Description

@srinath2022

Hi Team, I am trying to follow the guide here to use mBART model on triton inference server - https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/encoder_decoder.md

and the output from my model is empty. On further debug, I realized that the example triton server configs provided here - https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/encoder_decoder.md#4-prepare-tritonserver-configs-

from tensorrt_llm/triton_backend/all_models/inflight_batcher_llm doesn't actually use the encoder anywhere. The inputs to the tensorrt_llm model are input_ids from the preprocessor, neither preprocessor, nor tensorrt_llm use the parameter for encoder from config.pbtxt.

Am I missing something?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions