API

The API system allows the user to query the model in a pre-defined manner through applications such as Postman.

General

/get_response: Get response using natural language with a routing prompt that routes to one of describe_objects(object_ids), rank_objects(object_ids), describe_rank_objects(object_ids), guess_touch_given_objects(object_candidates), describe_rgb(prompt), and ask(query)

• query={query}

/reset: Delete any extracted tactile frames, chat history and chat embedding history

/add_new_rag_item: Add a new item to new RAG memory and save its frames, data and embedding in rag_new_sample_dir and rag_new_embedding_dir, as specified in configs/demo.yaml

• object_name=kiwi
• object_descriptions=fuzzy,flat (omit values to use predicted tactile descriptions from Octopi-1.5 )
• object_properties=3.33,4.44 (omit values to use predicted values from Octopi-1.5 ' regression module)

/reload_new_rag_items: Reload frame paths, data and embeddings from all new RAG items back into new RAG memory

/remove_new_rag_item: Remove most recently added item from new RAG memory

/reset_new_rag_items: Delete all new items from new RAG memory

Troubleshooting

/describe: Describe the tactile videos corresponding to the numbers given and in order (separated by commas)

• object_ids=1,2,3

/rank: Rank the tactile videos corresponding to the numbers given and in order (separated by commas)

• object_ids=1,3,2

/describe_and_rank: Describe and rank the tactile videos corresponding to the numbers given and in order (separated by commas)

• object_ids=3,2,1

/describe_rgb: Describe the RGB image taken of the scene

• prompt=Identify each object held in the cardboard holder along with details necessary for tactile reasoning, from right to left. Format your answer as 'Object 1: details, object name.\nObject 2:...' with less than 5 words each.

/guess_from_objects: Guessing touched object from a list of object candidates' names (separated by commas)

• object_candidates=a baseball,a tennis ball,a soccer ball

/ask: Query Octopi-1.5 using text

• query={query}